Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extends the previous theoretical analysis on Mixup to all Mixup-based methods with data-agnostic mask selection methods. Inspired by the previous analysis on Mixup, this work also shows that Mixup-based methods help improve generalization and robustness performances. And this paper shows that Mixup-based methods can be viewed as the regularizer of input gradient and Hessian as well as the first layer parameters. From such a view, this work further investigates that different Mixup-based methods have different effects as the regularizer on input gradient and Hessian and hence there is no optimal choice for all the cases. Stand on that, this paper proposed two combinations of Mixup and CutMix that combine the advantages of both of them.
strengths_and_weaknesses: Strengths:

1. Extensive analysis to extend previous theoretical understanding of Mixup to most of the Mixup-based methods.
2. Empirical results to support their theoretical findings.
3. The overall flow is good, every time I have a question I see the answer.

Weakness:

1. The proposed new methods appear to have marginal improvements over current results, although the current best one - Stochastic Mixup & CutMix also supports their claims.
2. Some of the parts could be written in a better manner,  e.g., the explanation in Fig. 4.
questions: 1. I didn't find Theorem 4 in the Appendix. Do Theorem 3 (a) and (b) in the Appendix correspond to Theorem 4?

2. For understanding Fig. 4, could you add the meanings of the x & y-axes and the color bar? Are the x & y-axes mean the distances between pixels along a specific dimension instead of the location of a pixel?

limitations: 1. This paper claims that Mixup & CutMix are good for different cases. But it only provides the results of their proposed methods when CutMix is the better choice. It would be better to provide the case when Mixup is alternatively the better choice to further support that the proposed methods combine the advantages of both these two methods.

2. The captions of figures could be more detailed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors develop and unified theory to explain the various data augmentation (DA) schemes used in Deep network training literature. Particularly, taking motivation from mixed sample strategies (MSDA) and cropping based approaches such as CutMix, the authors propose a loss formulation that considers any non-DA loss, and can be used to generalize various masking and mixing approaches to DA. They show that the CutMix and MSDA losses can then be interpreted through first- and second-order gradient regularization, thereby providing a framework to incorporate/develop other DA schemes. They propose two alternatives DA schemes, HMix and Mix, that result in improved performance of certain classification tasks. 

While I did not explore these results completely, the inclusion of results on n-sample DA in the Supplementary helps broaden the scope of the future work/applications of the proposed results in the paper. 
strengths_and_weaknesses:  - While my expertise in DA is not specific to loss formulation and analysis topics covered in the paper, the results presented in the paper help improve the overall understanding of how various DA schemes affect the learning objective, which is clearly presented. 
 - To the best of my knowledge, the proposed general loss function proposed is a significant contribution and should help the community analyze other DA schemes as well, those not considered in the paper. 
 - The empirical results on HMix and GMix help validate the formulation developed.
 - The paper is generally well written, with a clear flow of thoughts and presentation, and is clear to read even for those with minimal expertise in the particular sub-field that the papers is about. 
 - Minor Issue: The paper could use a bit of proofreading to fix typos here and there, eg. L147 can be benefit -> can be beneficial. 

==== x ==== x ==== x ====

Post-Rebuttal comments: I have gone though the author's responses to my and the other reviewers' comments and as there is an overall consensus that the paper has good merit and is deserving of an acceptance, I will keep my score to accept the paper. 
questions: While I understand the space constraints present, some points in the Theorem’s proof could still be discussed in the Main paper, to help better understand the results — I’m not sure if this a standard assumption in the field, but it seemed unclear to me why one would need $E_{r_x}[r_x] = 0$, or if there are any specific practical implications to having this assumption. Could we guarantee this on the datasets? 


limitations: I found the paper to be self contained and the authors seems to have sufficiently discussed most topics that would appear to be open ends for future work, such as explore scenarios with negative M in MSDA, etc.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a unified theoretical analysis of mixed sample data augmentation (MSDA). The theory shows that the MSDA training strategy can improve adversarial robustness and generalization of training. Based on these results, this paper proposes generalized MSDAs, a Hybrid version of Mixup and CutMix (HMix) and Gaussian Mixup (GMix), which are simple extensions of Mixup and CutMix. Where HMix unifies Mixup and Cutmix and uses the Beta distribution to determine the hybrid strategy hyperparameters, GMix is a smoother hybrid strategy.
strengths_and_weaknesses: ### Strengths
- The paper is well written and organized.
-  Detailed theoretical derivations support the ideas of the paper.

### Weaknesses
- Although the paper draws some conclusions through theoretical derivation, the proposed method is too similar to SmoothMix [1].
- Some classical baselines, such as ManifoldMix [2], are missing from the experimental comparison methods. in addition, some new mixup methods of comparison are missing, such as SaliencyMix [3] (comparable in computational complexity to vanilla mixup), etc.
- Some problems of theoretical derivation (see the questions for details).

[1] Lee, Jin-Ha, et al. "Smoothmix: a simple yet effective data augmentation to train robust classifiers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020.

[2] Verma, Vikas, et al. "Manifold mixup: Better representations by interpolating hidden states." International Conference on Machine Learning. PMLR, 2019.

[3] Uddin, A. F. M., et al. "Saliencymix: A saliency guided data augmentation strategy for better regularization." International Conference on Learning Representations, 2021.
questions: - The advantage of the beta distribution in HMix is not explained, simply because the Beta distribution is more commonly used and the Beta distribution has a closed-form of conjugate representation for easy derivation?
- The proposed strategies do not consider the label mismatch problem [4]; they both construct hybrid strategies by randomly selecting a box or a point. If the mixed samples and labels do not match, does it affect the theoretical derivation of conclusions?
- Both proposed strategies do not take full advantage of the previous conclusions, for example, MSDA is data-dependent, but both hybrid strategies are data-independent.
- The effect of hyperparameters such as \alpha, r, \lambda, etc. in the hybrid strategy is not fully explored.

[4] Liu, Zicheng, et al. "Automix: Unveiling the power of mixup." European Conference on Computer Vision, 2022.
limitations: Based on previous theoretical works on mixup methods, this paper draws some interesting conclusions. However, the proposed method is very similar to the already published SmoothMix and the methods compared in the experimental section are not adequate. Therefore, I think this article is not up to the acceptance criteria.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes unified theoretical analysis of mixed sample data augmentation (MSDA) by extending Zhang et al. (2021) that analyzed theoretical model of Mixup. The authors show that MSDSA can improve performance by regularization effects in gradient and Hessian. Furthermore, they propose new augmentation methods, HMix and GMix, based on their theoretical analysis. Experimental results show that HMix and GMix outperform the previous MSDA methods in CIFAR-100 and ImageNet.
strengths_and_weaknesses: Strengths: 

- This is a first theoretical study that proposes unified theoretical analysis of MSDA. The authors show theoretical reason why MSDA works. Furthermore, they show that Mixup and CutMix have different regularization effects in gradient and Hessian.

- The proposed HMix and GMix are consistent with their theoretical analysis. HMix and GMix outperform the previous MSDA methods in CIFAR-100 and ImageNet.

- The paper is clearly written.

Weaknesses: 

- Theorem 1 requires second-order Taylor approximation. This paper does not consider the validity of this approximation.

- The authors do not provide $\alpha_{ij}$ of HMix, while they provide the one of Mixup, CutMix, and GMix.

- The authors argue that potential negative societal impacts are discussed in the appendix. However, I cannot find the discussion in the appendix.

questions: 1. Is second-order Taylor approximation of loss function valid? Zhang et al. (2021) suggest that the approximation is valid for Mixup. Can their result be generalized to other MSDA?

2. Can results of Theorem 1 are applied to other MSDA methods such as ResizeMix (Qin et al., 2020), Fmix (Harris et al., 2021), or PuzzleMix?

- Qin, J., Fang, J., Zhang, Q., Liu, W., Wang, X., and Wang, X. (2020). Resizemix: Mixing data with preserved object information and true labels. arXiv preprint arXiv:2012.11101. 
- Harris, E., Marcu, A., Painter, M., Niranjan, M., Prügel-Bennett, A., and Hare, J. Fmix: Enhancing mixed sample data augmentation. ICLR, 2021.

limitations: This paper does not consider online optimization of mask design $M$ or $N$. It may be an interested topic for future works.
review_title: null

Meta-review: metareview: This work proposes a theoretical analysis and unified specification for mixed sample data augmentation methods. The reviewers praise the extensive theoretical analysis as well as the strong empirical results in the paper. The authors and reviewers engaged in substantial discussion, which led multiple reviewers to revise their assessment of the paper upwards. I can therefore recommend accepting this paper.
confidence: Certain
Decision: Accept