Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: I enjoyed reading this well-written and easy-to-follow paper. The paper builds on the rather old idea of minimizing the empirical vicinal risk (Chapelle et al., 2000) instead of the empirical risk. The authors' contribution is to provide a particular instance of vicinity distribution, which amounts to linear interpolation between samples. This idea of linear interpolation on the training sample to generate additional (adversarial, in the words of the authors) data is definitely appealing to prevent overfitting and improve generalization performance at a mild computational cost (note that this comment does not just apply to deep learning). This notion is definitely of interest to machine learning, and to the ICLR community in particular. I have several comments and remarks on the concept of mixup, listed below in no particular order. My overall opinion on the paper is positive and I stand for acceptance, provided the authors answer the points below. I would especially be interested in discussing those with the authors.

1 - While data augmentation literature is well acknowledged in the paper, I would also like to see a comment on domain adaptation, which is a very closely related topic and of particular interest to the ICLR community.

2 - Paragraph after Eq. (1), starting with "Learning" and ending with "(Szegedy et al., 2014)": I am not so familiar with the term memorization, is this just a fancy way of talking about overfitting? If so, you might want to rephrase this paragraph with terms more used in the machine learning community. When you write "one trivial way to minimize [the empirical risk] is to memorize the training data", do you mean output a predictor which only delivers predictions on $X_i$, equal to $Y_i$? If so, this is again not specific to deep learning and I feel this should be a bit more discussed.

3 - I have not found in the paper a clear heuristics about how pairs of training samples should be picked to create interpolations. Picking at random is the simplest however I feel that a proximity measure on the space $\mathcal{X}$ on which samples live would come in handy. For example, sampling with a probability decreasing as the Euclidean distance seems a natural idea. In any case, I strongly feel this discussion is missing in the paper.

4 - On a related note, I would like to see a discussion on how many "adversarial" examples should be used. Since the computational overhead cost of computing one new sample is reasonable (sampling from a Beta distribution + one addition), I wonder why $m$ is not taken very large, yielding more accurate estimates of the empirical risk. A related question: under what conditions does the vicinal risk converge (in expectation for example) to the empirical risk? I think some comments would be nice.

5 - I am intrigued by the last paragraph of Section 5. What do the authors exactly have in mind when they suggest that mixup could be generalized to regression problems? As far as I understood the paper, since $\tilde{y}$ is defined as a linear interpolation between $y_i$ and $y_j$, this formulation only works for continuous $y$s, like in regression. This formulation is not straightforwardly transposable to classification for example. I therefore am quite confused about the fact that the authors present experiments on classification tasks, with a method that writes for regression.

6 - Writing linear interpolations to generate new data points implicitly makes the assumption that the input and output spaces ($\mathcal{X}$ and $\mathcal{Y}$) are convex. I have no clear intuition wether this is a limitation of the authors' proposed method but I strongly feel this should be carefully addressed by a comment in Section 2.
review_title: Interesting strategy

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies an approach of data augmentation where a convex combination of multiple samples is used as a new sample.  While the use of such convex combination (mixing features) is not new, this paper proposes to use a convex combination of corresponding labels as the label of the new sample (mixing labels).  The authors motivate the proposed approach in the context of vicinal risk minimization, but the proposed approach is not well supported by theory.  Experimental results suggest that the proposed approach significantly outperforms the baseline of using only the standard data augmentation studied in Goyal et al. (2017).

While the idea of mixing not only features but also labels is new and interesting, its advantage over the existing approach of mixing only features is not shown.  The authors mention "interpolating only between inputs with equal label did not lead to the performance gains of mixup," but this is not shown in the experiments.  The authors cite recent work by DeVries & Taylor (2017) and Pereyra et al. (2017), but the technique of combining multiple samples for data augmentation have been a popular approach.  See for example a well cited paper by Chawla et al. (2002).  The baseline should thus be mixing only features, and this should be compared against the proposed approach of mixing both features and labels.

N. V. Chawla et al., SMOTE: Synthetic Minority Over-sampling Technique, JAIR 16: 321-357 (2002).

Minor comments:

Figure 1(b): How should I read this figure?  For example, what does the color represent?

Table 1: What is an epoch for mixup?  How does the per epoch complexity of mixup copare against that of ERM?

Table 2: The test error seems to be quite sensitive to the number of epochs.  Why not use validation to determine when to stop training?

Table 2: What is the performance of mixup + dropout?

===

I appreciate the thorough revision.  The empirical advantages over baselines including SMOTE and others are now well demonstrated in the experimental results.  It is also good to see that mixup is complementary to dropout, and the combined method works even better than either.

I understand and appreciate the authors' argument as to why mixup should work, but it is not sufficiently convincing to me why a convex combination in Euclidean space should produce good data distribution.  Convex combination certainly changes the manifold.  However, the lack of sufficient theoretical justification is now well complemented by extensive experiments, and it will motivate more theoretical work.

review_title: Baseline should be mixing features only

final_score: rating: 6: Marginally above acceptance threshold
review_content: Theoretical contributions: None. Moreover, there is no clear theoretical explanation for why this approach ought to work. The authors cite (Chapelle et al., 2000) and actually most of the equations are taken from there, but the authors do not justify why the proposed distribution is a good approximation for the true p(x, y). 

Practical contributions: The paper introduces a new technique for training DNNs by forming a convex combination between two training data instances, as well as changing the associated label to the corresponding convex combination of the original 2 labels. 

Experimental results. The authors show mixup provides improvement over baselines in the following settings:
  * Image Classification on Imagenet. CIFAR-10 and CIFAR-100, across architectures.
  * Speech data 
  * Memorization of corrupted labels
  * Adversarial robustness (white box and black box attacks)
  * GANs (though quite a limited example, it is hard to generalize from this setting to the standard problems that GANs are used for).
  * Tabular data.

Reproducibility: The provided website to access the source code is currently not loading. However, experiment hyperparameters are meticulously recorded in the paper. 

Key selling points:
  * Good results across the board.
  * Easy to implement.
  * Not computationally expensive. 

What is missing:
  * Convincing theoretical arguments for why combining data and labels this way is a good approach. Convex combinations of natural images does not result in natural images. 
 * Baseline in which the labels are not mixed, in order to ensure that the gains are not coming from the data augmentation only. Combining the proposed data augmentation with label smoothing should be another baseline.
  * A thorough discussion on mixing in feature space, as well as a baseline which mizes in feature space. 
  * A concrete strategy for obtaining good results using the proposed method. For example, for speech data the authors say that “For mixup, we use a warm-up period of five epochs where we train the network on original training examples, since we find it speeds up initial convergence.“ Would be good to see how this affects results and convergence speed. Apart from having to tune the lambda hyperparameter, one might also have to tune when to start mixup. 
  * Figure 2 seems like a test made to work for this method and does not add much to the paper. Yes, if one trains on convex combination between data, one expects the model to do better in that regime. 
  * Label smoothing baseline to put numbers into perspective, for example in Figure 4. 




review_title: Good results, but important baselines missing and no backing theory

Meta-review: The paper presents a simple but surprisingly effective data augmentation technique which is thoroughly evaluated on a variety of classification tasks, leading to improvement over state-of-the-art baselines. The paper is somewhat lacking a theoretical justification beyond intuitions, but extensive evaluation makes up for that.
Decision: Accept (Poster)