Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the phenomenon that the predictive performance of a prediction function degrades when overtraining under mixup data augmentation. The authors refer to the loss landscape as the U-shaped curve, where the prediction performance on the test data degrades when the number of epochs is increased. After observing the phenomenon experimentally, the authors present a theory that explains the phenomenon. The authors consider both classification and regression problems, although they focus primarily on classification problems.
strength_and_weaknesses: This paper provides beneficial practical insights. Many practitioners will be interested in this issue related to the mixup and overtraining. The authors' contribution is to raise the issue based on experimental observations and present a simple theoretical framework.

The theoretical result is straightforward and may not be accurate in some points. Furthermore, I felt that the theoretical validation between the analysis of classification (Section 5.1) and the analysis of regression (Section 5.2) is not consistent. Therefore, there is much room for improvement.

However, the problem setting and the approach are interesting. Although I think that the paper's proposed framework is not complete, I am inclined to accept it because I appreciate the challenging attempts.
clarity,_quality,_novelty_and_reproducibility: Some ambiguity remains, but the results shown are interesting.
summary_of_the_review: ### Major comments:
The problem setting and the theoretical framework proposed in this paper are very simple. While the theoretical analysis could be improved, the problem posed and the experimental observations are intriguing.

The reason why I feel that the theoretical analysis is incomplete is that it is hard to find the consistency among some of the results presented. For example, the connection between Theorem 5.1 and Theorem 5.2 is not clear. Overall, the results seem to be pieced together with little relevance. Furthermore, there are some ambiguous descriptions and undefined terms.

Although I am dissatisfied with some of such details, I will not vote for rejection because it reports findings that are useful for practical use.

- On page 3: In Lemma 3.1, what is $\theta$ or what is a class of $\theta$? For example, can the authors write, "Then, for all $\theta \in ???$"?
- On page 3: In Lemma 3.1, $\mathbb{E}_{\lambda}\hat{R}(\theta)$ is a random variable regarding the randomness of $\lambda$? That is,  does the statement hold with probability one?
- The figures are hard to read because the characters are very small. It is not kind to people who read the printed paper. 
- While the y-axis of the experimental results for test data in Figure 1 is test error, it is test accuracy in Figure 2. I read a printed paper, so I could not notice the change in the y-axis and was confused.
- On page 5: In Theorem 5.1, does the bound hold with probability one?
- On page 6: Is the definition that "For example, if $f$ is strongly convex with some parameter..." applied for all $x$?
- I omit other details but feel that the definition and statement in the theoretical analysis are not rigorous. I would like the authors to clarify them in more detail.

### Minor comments:
- On page 3: Can we interpreted $\mathcal{Y}$ as being $\mathcal{Y} \in \mathcal{P}(\mathcal{Y})$. 
- On page 3: In Lemma 3.1, what is the definition of $\mathbb{E}_{\lambda}$?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper aims to understand Mixup training, and specifically the training dynamics if mixup is continued to run for much longer than is usual. The paper shows an interesting empirical phenomenon, that overtraining with mixup leads to a U-shaped curve for training error, i.e. training error increases after a point (and also becomes worse than the error of the ERM solution). To demonstrate this, the paper considers ResNet models trained on CIFAR10 and SVHN datasets. 

The paper then provides an intuitive explanation for this phenomenon for the simple regression setting on Gaussian data. Though the setup is simplistic, it brings out the same phenomenon as in the more realistic experiments. The key idea behind why Mixup overfits when overtrained is intuitive too: if the ground truth function f is non-linear, then the Mixup labels do not necessarily correspond to the labels assigned by the ground-truth function (because Mixup does a linear interpolation to assign the labels of the Mixed-up datapoints which would be inconsistent with the labels if f is non-linear). Therefore, Mixup introduces label noise, and if overtrained the model will fit the noise and not generalize as well.


strength_and_weaknesses: Strengths:

1. The paper identifies an interesting empirical phenomenon, and the findings appear to be quite robust to different setups.
2. The theory is clean, and provides a clear explanation which fits the intuition.
3. The paper relates the observations to a number of recent empirical phenomenon identified in deep learning, such as sharp/flat minima and the gradient norms not converging to zero. This places the paper well in the broader literature, and it could contribute to a growing understanding of neural network training dynamics.

Weaknesses:

1. The finding might have slightly limited practical applicability since the #epochs at which overfitting starts to happen is quite large, and early-stopping is often employed when the model starts to overfit. The main takeaway is probably the more conceptual one of how mixup training behaves.
clarity,_quality,_novelty_and_reproducibility: The paper is generally quite well-written. However, there are a few issues which should be addressed:

1. The presentation of the theorem (particularly Thm 5.2) could be made clearer to have the takeaways be easier to extract. There also seems to be an error in the statement, I believe there should be a sum over all k in front of the C_2/\mu_k term. Due to all the \mu_k's and \sigma_k's floating around, it is not as easy to understand how the parameters interact. Perhaps having a corollary for some simple (e.g. isotropic?) settings would be useful. 
2. The text in the figures is too small to read.


summary_of_the_review: In summary, I think this is an interesting paper and provides valuable insight into a very popular training strategy (Mixup). While it may not directly lead to better training, I think it does provide useful insight into neural network training dynamics more broadly. This aspect is further strengthened by the papers analysis of the overtraining phenomenon in relation to other empirically observed phenomenon in deep learning.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors studied why training with the mixup loss for extra epochs often degrades test classification accuracy. They argued, through empirical observation, theoretical analysis and synthetic experiments, that the mixup loss introduces label noise, and a model may overfit to the label noise during extended training, hence increasing the generalization gap.

This paper makes several contributions:
1. it reports an interesting phenomenon in training classifiers and replicates it on multiple datasets;
2. it shows that extended training with the mixup loss increases the sharpness of the loss landscape;
3. it provides a possible explanation to the phenomenon by analyzing a regression problem;
4. it creates a synthetic regression problem which validates the theoretical analysis;
5. it suggests a practical solution to alleviate the observed problem.
strength_and_weaknesses: Strength:
- the paper studies an interesting phenomenon concerning a popular training loss (or data augmentation method), which has a wide audience base;
- the paper provides both empirical evidence and theoretical arguments to support its hypothesis;
- the paper is well-written and easy to follow.

Weakness:
- On the novelty and validity of the reported phenomenon. With some search I found a paper [1] reporting test accuracies during extended training with ERM or mixup, on benchmark datasets such as CIFAR-10 and CIFAR-100. In [1, Table 13 and 14] only one out of four settings (i.e. CIFAR-100 + RX-50) shows performance degradation with extended mixup training. One significant difference of the current work is that it turns off (other) data augmentation. Could this be the reason why the reported results seem contradictory? And if so, maybe it is proper to state that the observation may only be valid without data augmentation, and contrast with the previous report.

[1] AutoMix: Unveiling the Power of Mixup for Stronger Classifiers. ECCV 2022.

- On the interpretation of the theoretical analysis. If I understand correctly, the gradient flow characterization (eq. (3), Lemma 5.1) is basically the coordinates of $\theta_t$ as a parametric curve. While $\theta_t \neq \theta^*$, it does not imply that $||\theta_t - \theta^*||$ or $R_t$ is a U-shaped function. Theorem 5.2 is an upper bound result on $R_t - R^*$. Without a matching lower bound, or a direct calculation of $\frac{d R_t}{dt}$, or showing $R_t$ is smaller for some $t$ vs. $t \to \infty$, one still cannot justify that $R_t$ will **actually** increase as $t$ goes to infinity.

There are two additional points regarding the link between the authors' theory and observation that I would raise for discussion:

1. At the end of Sec. 5, the authors claim that their Theorem 5.2 "explains why reducing the learning rate will make the population risk again decrease in a certain interval". However, as I understand, in the gradient flow ODE interpretation (eq. (3), Lemma 5.1), changing $\eta$ simply corresponds to reparametrize the curve, (e.g. reducing $\eta$ by half corresponds to running $t$ 2x slower), hence a stretching or squeezing of the $R_t$ curve along the x-axis. Therefore it only scales $\frac{d R_t}{dt}$ but does not change its sign. Could you explain your reasoning and correct me if I am wrong?

2. At the end of Sec. 6.1, the authors claim that the minimum of the MSE loss for a fixed $\lambda=0.5$ come earlier in training due to increased noise level, which is consistent with their theory. However, as drawing $\lambda$ from a distribution introduces extra noise to the training, a better ablation study would be to compare the effects of different fixed $\lambda$, e.g. $\\lambda \\in \\{0.1, 0.2, 0.3, 0.4, 0.5\\}$.
clarity,_quality,_novelty_and_reproducibility: The paper has good clarity, makes a few novel contributions, and the authors have provided code for reproducibility.
Concerns regarding the quality of the paper have been discussed in the "weakness" section.
summary_of_the_review: To my best knowledge, I believe the paper studies an interesting and potentially important question; both the experiments and the theoretical analysis are sound per se; however, I have concerns regarding the novelty and scope of validity of the reported phenomenon, as well as whether the theoretical analysis actually justifies the authors' hypothesis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors point out a previously unknown behaviour of Mixup when overtraining: when the number of training epochs is extremely high, the Mixup performance is inferior to the one of ERM. The authors provide sufficient empirical evidence to show this phenomenon generalizes at least across datasets, and provide an explanation in terms of the labeling noise Mixup induces. They also provide a simplilfied theoretical explanation on a random feature model that exhibits a similar behaviour. As a fix to the problem, they suggest switching off mixup after a few epochs, which is empirically interesting, although further experiments would be required in order to understand its usefulness for more complex setups.
strength_and_weaknesses: Strengths:
- clearly written, easy to follow, good references
- the phenomenon is novel, although overtraining is not typically a problem in large-scale experimental settings
- most explanations provided are plausible and supported either by theory or experiments

Weaknesses:
- Q1: The experiments mostly use ResNets. Does this phenomenon emerge also for other types of architectures? The way overparametrisation plays a role might interact differently with different inductive biases. Could you try showing some results for other convolutional architectures (e.g. ConvNeXt, PyramidNet), some sequential tasks (e.g. using LSTMs)  or Transformers fine-tuning?
- Q2: The authors refer to the conjecture that flatter minima imply better generalization: for CIFAR-10 and CIFAR-100 covariate-shift datasets are available (e.g. CIFAR-10/100-C, CIFAR-10.1 and CIFAR-10.2). Would it be possible for them to validate their argument about the flatness inducing better generalization on these datasets by running their models and evaluating the accuracy on these datasets? Indeed, whether the Mixup loss is flat or sharp should not bee too relevant, what probably should matter is whether in the loss-landscape of the ERM cross-entropy loss the configuration reached by the optimisation algorithm is flat or not, and this will influence the actual generalization performance.
- Q3: I understand the authors probably switched off augmentations to disentangle the behaviour of Mixup from that of augmentations. However, the authors claim augmentations would further exasperate the phenomenon observed. Could they provide empirical evidence for this?
- Q4: It is not clear to me how the teacher-student experiment should be relevant. What is it supposed to tell? Maybe it's just a matter of writing or my own misunderstanding. Could you please elaborate further?
- Q5: Mixup is a technique used in many SOTA training procedures. The authors seem to suggest it could be useful to switch it off after a few epochs. This creates the issue of having to determine when it should be switched off: presumably, choosing different epochs will produce very different results. Is there a procedure the authors could suggest to do it? Additionally, the effectiveness of this procedure should be validated on more extensive experimental evidence (a few more architectures, and if possible larger datasets (e.g. ImageNet)). 
- Q6: The authors of [1] suggest using the Mixup loss as a regulariser of normal cross-entropy (i.e. the loss is CE(clean samples) + MixupLoss(interpolatedSamples)) in order to fix out-of-distribution detection issues given by the label-smoothing effect of Mixup induced by the randomness of the interpolation parameter.  Given the argument you make about the impact of the label noise and you suggest a fix by switching off Mixup, could you please either empirically or theoretically discuss whether using Mixup as a regularizer of cross-entropy could be a fix to the problem? Would the impact of the noise signal be alleviated or cancelled in this case?
- Q7: Overtraining is not really a problem for large-scale classification (since the resources requirements are already extremely high that no one would waste them overtraining), however your observations can be helpful in case practitioners observe failure cases of Mixup on smaller datasets. Instead of subsampling larger datasets, could you suggest real-world datasets in which overtraining might happen unintentionally and hence make practitioners actually observe this phenomenon? 
- Q8: Since Mixup already requires cross-validation (given the need of cross-validating the hyperparameter of the Beta distribution), what would prevent practitioners to just use Early Stopping or some other strategy to avoid over-training? Would Early Stopping outperform ERM training?
[1] RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness; Francesco Pinto, Harry Yang, Ser-Nam Lim, Philip H.S. Torr, Puneet K. Dokania


 
clarity,_quality,_novelty_and_reproducibility: The writing is clear, and the paper is structured logically. Maybe the figures could be positioned a bit better, and some visual diversity could be used to help parse them (e.g. aggregating some plots where reasonable and using colours better). 
The work is original enough and interesting. The experiments seem to be easily reproducible by others given Mixup is a well-established technique. 
summary_of_the_review: The paper is overall interesting, although its practical usefulness is not particularly emphasized. The idea is simple and clear, the explanations are plausible. If the authors could convincingly address the weaknesses I pointed out, I'd be happy to rise the score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper exposes an interesting phenomenon occurring when training deep neural networks with mixup regularization referred to as over-training with mixup. After a certain number of epochs, the network might start to overfit to the noise introduced by mixup regularization.

Reviewers were divided in their opinions of the paper, with some leaning towards acceptance and others leaning towards rejection. However, there was good agreement on the pros and cons of the paper, and the main discussions revolved around the relative importance of each pro and con.

Overall, reviewers agreed that the phenomenon exposed in the paper is interesting and that the experiments sufficiently corroborate it. However, there was some concern about the practical impact of the paper, as the overtrain with mixup phenomenon seems to require an unrealistic number of epochs to occur, and therefore may not have a significant impact on performance in real-world scenarios. Additionally, there were concerns about the theoretical explanation provided in the paper, with some reviewers questioning how well it directly explains the observed experimental results. 

Despite these concerns, reviewers were hopeful that the paper would stimulate further work in this direction, both practical and theoretical. Based on this grounds, it is my pleasure to recommend acceptance of the paper.

Please remember to address the reviewers' remarks. On top of that, please bring into the main text discussion and results related to overtraining in training with augmentation, and further elaborate on how the theory explains the observed experimental results.
justification_for_why_not_higher_score: Due to concerns related to impact of the paper.
Decision: Accept: poster