Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a specific instance of Mixup data augmentation: Midpoint Mixup. It provides a theoretical characterization of the learning dynamics of neural networks with mixup-augmented data, and proved how it can improve feature learning by helping the neural network pick up diverse features, under the multi-view data framework proposed by Allen-Zhu and Li.


[1] Zeyuan Allen-Zhu, Yuanzhi Li, Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
strength_and_weaknesses: Strength: Most of the paper is clearly written and it provides a rigorous analysis of the subject it studies. Mixup is an interesting data augmentation technique and tries to explain why it can help in improving the feature learning of neural nets. The theory result goes beyond the previous works that are based on linear models (as they are based on nonlinear neural networks) and is reasonable due to the feature learning technique it uses. It might be an important paper if the authors can fix the issues I find below.

Weaknesses: 
1. The authors did not provide sufficient motivation why they specifically study Midpoint Mixup rather than the general mixup augmentations. Moreover, they did not provide an empirical comparison between the original mixup and the midpoint mixup, making it hard to justify their choice of study. A potential improvement is to explain the technical difficulty of analyzing the original mixup, and justifies that the midpoint mixup is competitive (or particularly interesting).
2. Assumption 3.10, which seems to be the key assumption to migrate the proof technique in [2] to the setting of this paper, is very technical, without intuitive explanation, and without support from empirical evidence. It is hard to make sense of this assumption as assembling any real-world structure. This weakens the validity of the proposed explanation as the theory result could be just an artificial construction.
3. The experimental results in this paper are not informative, as many of their metrics are too low. I do not understand why this paper constructs such special data by mixing images using Dirichlet distribution (see equation 4.1 and Fig 1) to compare ERM with Midpoint-Mixup. Moreover, the test errors often exceed 60%-90% for experiments on CIFAR-10/100 when L>1 (that is over the special dataset rather than the original Cifar10/100), meaning that the model struggles a lot to predict the mixup label. I don't think these experiments are sufficient to support the authors' choice of study or their theoretical claims.


[1] Ruoqi Shen, Sebastien Bubeck, Suriya Gunasekar, Data Augmentation as Feature Manipulation
[2] Zeyuan Allen-Zhu, Yuanzhi Li, Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
clarity,_quality,_novelty_and_reproducibility: The clarity is decent. the novelty is partially limited because the proof/data framework is largely inherited from prior work, while the original part (such as assumption 3.10) lacks enough explanation.
summary_of_the_review: This paper studies an interesting instance of Mixup augmentation and obtained some decent theory results with the help of an established theoretical framework, but the authors are not good at justifying their choice of study and their theoretical claims, and they fail at explaining the key technical ingredient of the paper, as well as how well their assumptions can connect to practical scenarios.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on mid-point mixup (a variant of mixup) and provides theoretical analysis on the effect of using mid-point mixup on multi-view data. The theoretical result is specific to the setting of the 2-layer convolutional networks and the datasets where samples at each class have 2 features. Under this setting, this paper shows that empricial risk minimization learn only single feature at each class, while mid-point mixup learns both features at each class. 
strength_and_weaknesses: - Strength
    - The direction of analyzing mid-point mixup on mulit-view data is new and interesting.

- Weakness

    - Theory
        - Why is the assumption of Lemma 2.1 (z_{i,j} is unique for all i,j pair) true in practice? Simple 2-dimensional dataset where each class follows Gaussian distribution cannot follow this assumption. (how about high-dimensional case?)
        - It seems like Lemma 2.1 is important to show that Midpoint Mixup is not poor, but the proof of Lemma 2.1 is not fully understood on my side. (I put the details in the questions section)

    - Experiments
        - This paper shows that when L is large, Mid-point Mixup is having less error than ERM. Of course if L is large, it will go to the ideal multi-view dataset this paper is assuming, but then why don’t we even start with real data? Instead, making a synthetic multi-view data to show the effectiveness of the theory seems more reasonable. I personally feel the experiments in this paper is similar to just “simulation” instead of real-world experiments.
        - In Table 1, the authors are comparing “test error 96%” versus “test error 97%”, when we have 100 classes. What does it mean to increase 1% test error from “nearly random guessing”? Does these empirical results have any positive impact on the practical deployment of mid-point mixup?

    - Writing
        - The assumptions and results are less organized (thus I have many questions below). The definition 3.4 has no intuitive explanation or visualization.
clarity,_quality,_novelty_and_reproducibility: The problem itself seems to be novel, but the theoretical results are unclear. Below I wrote some questions to clarify them. 

* I couldn't fully get the proof of Lemma 2.1. Why does the “existence of well-defined h simultaneously optimizing each term of eq.2.3” implies “RHS of (2.4) is achieving the minimum”, so that we have “$g^{y_i} (z_{i,j}) = g^{y_j} (z_{i,j})$ for all $z_{i,j}$”?

* I couldn't get the sentence "J(g, X) by just taking $⟨w_y, v_{y,1}⟩ \rightarrow \infty$ for every class y" in the paper. I thought setting $w_c = v_{y,1}$ was enough to make the empirical loss = 0?

* I couldn't get what footnote 1 means.

* Sec.3.1 title is “linear model” and Sec.3.2 title is “multi-view data setup”. But it seems like Def.3.1 in Sec.3.1 is also multi-view. What is the criteria differentiating Sec.3.1 and 3.2? Maybe because Sec.3.2 for general model, not limited to linear model? If so, the section name is misleading.


I am also adding minor comments
* The first sentence of Definition 3.4 is “Identically to Definition 3.4”. Maybe typo for “Identically to Definition 3.1”?
* Typo: themself → themselves
summary_of_the_review: This paper focuses on an interesting problem, but has much room for improvement in terms of organizing the results, validating the assumption, and explaining how the experimental results are helpful in practical settings.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work adopts the feature learning framework by Allen-Zhu and Li (2021), and further introduces latent orthogonal feature views. The authors prove that under certain assumptions on features, a two-layer smooth-ReLU network can learn all features with midpoint mixup, whereas ERM can only learn one feature.
strength_and_weaknesses: Strength:
1. On top of Allen-Zhu and Li (2021), the authors further decompose the input into multiple orthonormal feature views. This allows the authors to analyze the contribution of each individual feature and their gradient correlations.
2. Although the change of data settings (section 3.1) and assumptions (3.10) is simple, the authors provide significant efforts in proving the feature learning under the mixup loss.

Weakness:
* Theory:
1) In Eq. B.34, do you need a lower bound or dependence on the training iterations, such that the feature correlations will not be over-corrected?
2) Could you add more intuitive explanations of the meaning of function f in Assumption 3.10? Is it possible to analytically prove the monotonicity of f?

* Experiments:
1) Thm 3.9 and 3.11 did not directly imply better test accuracy (generalization). Instead, is it possible to empirically verify the one or all feature(s) (v) learned (defined by 3.8) by the Midpoint Mixup?
2) It would be better to add variances in Table 1.
3) Many test errors in Table 1 are far worse than random guess (90% error rate). From Eq. 4.1, I did not see any noise is introduced. What is the reason for this large test error?
clarity,_quality,_novelty_and_reproducibility: 1. Overall the paper writing is clear and easy to follow.
2. As mentioned above, the main novelty of this work is to prove the benefit of the mixup in feature learning under the multi-view data setting.
summary_of_the_review: In general, I believe this work contributes significantly to the feature learning topic.
I hope the authors could fix my concerns above and add more clarifications.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper analyzes mid-point mixup, specifically the learning dynamics of neural networks with midpoint mixup-augmented data, and show how it can improve feature learning by helping the neural network learn diverse features, under the multi-view data framework proposed by Allen-Zhu and Li (2021). Authors also experiment with synthetic features augmented datasets that show that midpoint mixup outperforms ERM. The paper is the first to analyze mixup for nonlinear models and from a multiview feature learning perspective. However, reviewers have unaddressed concerns on the paper, in particular about the experimental setup being impractical (test errors in the range of 90%) and less connected with theory. The authors have also made significant revisions to many parts of the original manuscript and there is a concern that this will require another careful review of the paper. Authors should address the concerns about the alignment between theory and experiments and submit to a future venue. 
justification_for_why_not_higher_score: Reviewers have unaddressed concerns on the paper, in particular about the experimental setup being impractical (test errors in the range of 90%) and less connected with theory. The authors have also made significant revisions to many parts of the original manuscript and there is a concern that this will require another careful review of the paper.
Decision: Reject