Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a simple modification of the well-known Mixup that uses the combination of vanilla training loss and Mixup training loss as the training objective. This modification helps to overcome the two issues of Mixup: 1) Mixup chooses extremely small $\alpha$ during cross-validation and 2) Mixup can not distinguish in-distribution and out-of-distribution samples. Experiment results show that RegMix outperforms Mixup and many other baselines in in-distribution accuracy, Covariate Shift accuracy, and out-of-distribution detections.
strengths_and_weaknesses: Strengths:
1. This paper suggests a simple modification of Mixup that fixes the two issues of Mixup and leads to better CS and OOD performance.
2. The paper is well-written. It first defines the issues of Mixup and shows how RegMix is able to overcome these issues.
3. Extensive experiments have been carried out on various image datasets and several tasks.

Weaknesses:
1. The proposed method is not that different from the original Mixup.
2. The proposed method is designed for the earliest mixup-based method. And we are not sure if it also works for the recent progress on mixup-based methods, e.g., CutMix, PuzzleMix.
3. There is no theoretical justification for why RegMix works.
questions: 1. This paper states that one of the limitations of Mixup is that it uses small $\alpha$ for good generalization. And this paper uses $\alpha=0.3$ for CiFAR-10 & CIFAR-100, and $\alpha=0.1$ for ImageNet. But as I know in the case of image classification, e.g., CIFAR-10, ImageNet-1k, the commonly used $\alpha$ is 1 which leads to a uniform distribution. Could the authors provide results of these datasets under the commonly used $\alpha$ and provide some explanation on why Mixup prefers small $\alpha$ in this case? I believe this will provide a more solid sound. 

2. Since the choice of $\alpha$ is the limitation of MIxup and benefit of RegMix, I would suggest putting the $\alpha$ used for these two methods in the main body instead of in the Appendix.

3. (Minor one) It would be better to provide the comparison of a failure case of Mixup and a success case of RegMix for OOD.
limitations: 1. Mixup was developed in 2018 and several better mixup-based methods have been proposed, e.g. the common training pipeline of ViT now uses a combination of Mixup and CutMix. I was wondering how these methods perform against CS and OOD and if RegMix together with these new mixup-based methods can still lead to better results.

2. It would be better to provide some theoretical justifications to show why RegMix can help overcome the issues of Mixup. Specifically, why Mixup prefers small $\alpha$ but RegMix prefers a large one.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents RegMixup, which employs Mixup training as an additional regularize of the standard training. It can be treated as a special case of employing Mixup, but with a possibility of 0.5.  Some interesting findings are presented, for instance, RegMixup scales well with a beta distribution of (10, 10), and RegMixup provides high entropy barrier separating in-distribution from out-of-distribution samples. Extensive experiments are conducted on CIFAR-10, CIFAR-100, and ImageNet.  Calibration, OOD, and distribution/covariance shift results are presented. Overall, the presented method is simple yet insightful. 
strengths_and_weaknesses: Strengths:
1: The proposed method is simple, and can be easily used for training robust image classifiers. 

2: Good presentation, well-written.

3: Experiments are extensive. Results are solid. 


Weaknesses:
1: No experiments on CutMix (and other variants, but CutMix should be enough). Does RegMixup also help CutMix?

2: The training of RegMixup seems to see 2x samples per iteration. Thus the running speed is slow (as authors claimed 1.5 x slower). When compared to other methods, RegMixup seeing 2x samples may lead to unfair comparison. 

3: Might be good to validate the performance of RegMixup on transformer and other training recipes (used in transformer models). 


questions: My main concerns are listed in weaknesses, especially points 1 and 2, here is one more question/suggestion:

1: In Figure 2 and lines 108-109, authors claimed that Mixup employs a small value of α. But I believe many papers [1,2,3] employ beta (1,1) as the default setting. Authors may need to provide some supporting references or consider changing this claim

Overall I'm happy to change my rating to Borderline Accept if authors can strongly address weaknesses 1 and 2, where weakness 2 limits the application of RegMixup and weakness 1 limits the generality. If all of my concerns are well-addressed, I'm OK to raise my rating further into Weak Accept. 

[1]: CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features,  ICCV 19.

[2]: You Only Cut Once: Boosting Data Augmentation with a Single Cut, ICML 22.

[3]: Lots of transformer papers and recent network architecture papers.
limitations: Limitations are discussed. Negative societal impact are not discussed. A potential negative societal impact is that the interpretability of RegMixup is still limited and hence needs to be carefully applied to sensitive applications.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a simple change to Mixup data augmentation to also train with unperturbed images in addition to augmented images. Making this small change induces alpha cross-validation to choose stronger Mixup (closer to 50/50 image interpolation), yielding more image diversity without losing access to in-distribution samples during training. Extensive experiments show improvement in both in-distribution and out-of-distribution (OOD) accuracy, and OOD detection. This method avoids the common tradeoff of in-distribution and OOD accuracy.
strengths_and_weaknesses: Strengths: The proposed method is very simple and easy to implement, as well as computationally inexpensive. It is validated with extensive experiments covering a wide set of models and datasets, and shows good results on multiple tasks.

Weaknesses: The paper does not read well due to many grammatical/writing issues. My first impression was negative because of this. As an example, the word "of" is missing in the title ("out of distribution"). Please copy edit the final version.
questions: Do you have further or more concrete directions for future work in mind?
limitations: The authors do acknowledge a few experiments in which their method is outperformed, and partial explanations for why (e.g. AugMix).
review_title: null

Meta-review: metareview: The paper presents a surprisingly simple modification to mixup regularization, which results in a consistent improvement over standard mixup both in the original dataset and in the out-of-distribution setting. All of the reviewers agree that the paper adds to the literature despite its simplicity, and accordingly, I recommend acceptance.
confidence: Less certain
Decision: Accept