Towards Understanding the Data Dependency of Mixup-style Training

1 INTRODUCTION . Mixup ( Zhang et al. , 2018 ) is a modification to the standard supervised learning setup which involves training on convex combinations of pairs of data points and their labels instead of the original data itself . In the original paper , Zhang et al . ( 2018 ) demonstrated that training deep neural networks using Mixup leads to better generalization performance , as well as greater robustness to adversarial attacks and label noise on image classification tasks . The empirical advantages of Mixup training have been affirmed by several follow-up works ( He et al. , 2019 ; Thulasidasan et al. , 2019 ; Lamb et al. , 2019 ; Arazo et al. , 2019 ; Guo , 2020 ) . The idea of Mixup has also been extended beyond the supervised learning setting , and been applied to semi-supervised learning ( Berthelot et al. , 2019 ; Sohn et al. , 2020 ) , contrastive learning ( Verma et al. , 2021 ; Lee et al. , 2020 ) , privacy-preserving learning ( Huang et al. , 2021 ) , and learning with fairness constraints ( Chuang & Mroueh , 2021 ) . However , from a theoretical perspective , Mixup training is still mysterious even in the basic multiclass classfication setting – why should the output of a linear mixture of two training samples be the same linear mixture of their labels , especially when considering highly nonlinear models ? Despite several recent theoretical results ( Guo et al. , 2019 ; Carratino et al. , 2020 ; Zhang et al. , 2020 ; 2021 ) , there is still not a complete understanding of why Mixup training actually works in practice . In this paper , we try to understand why Mixup works by first understanding when Mixup works : in particular , how the properties of Mixup training rely on the structure of the training data . We consider two properties for classifiers trained with Mixup . First , even though Mixup training does not observe many original data points during training , it usually can still correctly classify all of the original data points ( empirical risk minimization ( ERM ) ) . Second , the aforementioned empirical works have shown how classifiers trained with Mixup often have better adversarial robustness and generalization than standard training . In this work , we show that both of these properties can rely heavily on the data used for training , and that they need not hold in general . Main Contributions and Related Work . The idea that Mixup can potentially fail to minimize the original risk is not new ; Guo et al . ( 2019 ) provide examples of how Mixup labels can conflict with actual data point labels . However , their theoretical results do not characterize the data and Correspondence to Muthu Chidambaram ( muthu @ cs.duke.edu ) . model conditions under which this failure can provably happen when minimizing the Mixup loss . In Section 2 of this work , we provide a concrete classification dataset on which continuous approximateminimizers of the Mixup loss can fail to minimize the empirical risk . We also provide sufficient conditions for Mixup to minimize the original risk , and show that these conditions hold approximately on standard image classification benchmarks . With regards to generalization and robustness , the parallel works of Carratino et al . ( 2020 ) and Zhang et al . ( 2020 ) showed that Mixup training can be viewed as minimizing the empirical loss along with a data-dependent regularization term . Zhang et al . ( 2020 ) further relate this term to the adversarial robustness and Rademacher complexity of certain function classes learned with Mixup . In Section 3 , we take an alternative approach to understanding generalization and robustness by analyzing the margin of Mixup classifiers . Our perspective can be viewed as complementary to that of the aforementioned works , as we directly consider the properties exhibited by a Mixup-optimal classifier instead of considering what properties are encouraged by the regularization effects of the Mixup loss . In addition to our margin analysis , we also show that for the common setting of linear models trained on high-dimensional Gaussian features both Mixup ( for a large class of mixing distributions ) and ERM with gradient descent learn the same classifier with high probability . Finally , we note the related works that are beyond the scope of our paper ; namely the many Mixup-like training procedures such as Manifold Mixup ( Verma et al. , 2019 ) , Cut Mix ( Yun et al. , 2019 ) , Puzzle Mix ( Kim et al. , 2020 ) , and Co-Mixup ( Kim et al. , 2021 ) . 2 MIXUP AND EMPIRICAL RISK MINIMIZATION . The goal of this section is to understand when Mixup training can also minimize the empirical risk . Our main technique for doing so is to derive a closed-form for the Mixup-optimal classifier over a sufficiently powerful function class , which we do in Section 2.2 after introducing the basic setup in Section 2.1 . We use this closed form to motivate a concrete example on which Mixup training does not minimize the empirical risk in Section 2.3 , and show under mild nondegeneracy conditions that Mixup will minimize the emprical risk in Section 2.4 . 2.1 SETUP . We consider the problem of k-class classification where the classes 1 , ... , k correspond to compact disjoint sets X1 , ... , Xk ⊂ Rn with an associated probability measure PX supported on X =⋃k i=1Xi . We use C to denote the set of all functions g : Rn → [ 0 , 1 ] k satisfying the property that∑k i=1 g i ( x ) = 1 for all x ( where gi represents the i-th coordinate function of g ) . We refer to a function g ∈ C as a classifier , and say that g classifies x as class j if j = argmaxi gi ( x ) . The cross-entropy loss associated with such a classifier g is then : J ( g , PX ) = − k∑ i=1 ∫ Xi log gi ( x ) dPX ( x ) The goal of standard training is to learn a classifier h ∈ argming∈C J ( g , PX ) . Any such classifier h will necessarily satisfy hi ( x ) = 1 on Xi since the Xi are disjoint . Mixup . In the Mixup version of our setup , we are interested in minimizing the cross-entropy of convex combinations of the original data and their classes . These convex combinations are determined according to a probability measure Pf whose support is [ 0 , 1 ] , and we assume this measure has a density f . For two points s , t ∈ X , we let zst ( λ ) = λs+ ( 1−λ ) t ( and use zst when λ is understood ) and define the Mixup cross-entropy on s , t with respect to a classifier g as : ` mix ( g , s , t , λ ) = { − log gi ( zst ) s , t ∈ Xi − ( λ log gi ( zst ) + ( 1− λ ) log gj ( zst ) ) s ∈ Xi , t ∈ Xj Having defined ` mix as above , we may write the component of the full Mixup cross-entropy loss corresponding to mixing points from classes i and j as : J i , jmix ( g , PX , Pf ) = ∫ Xi×Xj× [ 0,1 ] ` mix ( g , s , t , λ ) d ( PX × PX × Pf ) ( s , t , λ ) The final Mixup cross-entropy loss is then the sum of J i , jmix over all i , j ∈ { 1 , ... , k } ( corresponding to all possible mixings between classes , including themselves ) : Jmix ( g , PX , Pf ) = k∑ i=1 k∑ j=1 J i , jmix ( g , PX , Pf ) Relation to Prior Work . We have opted for a more general definition of the Mixup loss ( at least when constrained to multi-class classification ) than prior works . This is not generality for generality ’ s sake , but rather because many of our results apply to any mixing distribution supported on [ 0 , 1 ] . One obtains the original Mixup formulation of Zhang et al . ( 2018 ) for multi-class classification on a finite dataset by taking the Xi to be finite sets , and choosing PX to be the normalized counting measure ( corresponding to a discrete uniform distribution ) . Additionally , Pf is chosen to have density Beta ( α , α ) , where α is a hyperparameter . 2.2 MIXUP-OPTIMAL CLASSIFIER . Given our setup , we now wish to characterize the behavior of a Mixup-optimal classifier at a point x ∈ Rn . However , if the optimization of Jmix is considered over the class of functions C , this is intractable ( to the best of our knowledge ) due to the lack of regularity conditions imposed on functions in C. We thus wish to constrain the optimization of Jmix to a class of functions that is sufficiently powerful ( so as to include almost all practical settings ) while still allowing for local analysis . To do so , we will need the following definitions , which will also be referenced throughout the results in this section and the next : Ai , jx , = { ( s , t , λ ) ∈ Xi ×Xj × [ 0 , 1 ] : λs+ ( 1− λ ) t ∈ B ( x ) } Ai , jx , , δ = { ( s , t , λ ) ∈ Xi ×Xj × [ 0 , 1− δ ] : λs+ ( 1− λ ) t ∈ B ( x ) } Xmix = x ∈ Rn : ⋃ i , j Ai , jx , has positive measure for every > 0 ξi , jx , = ∫ Ai , jx , d ( PX × PX × Pf ) ( s , t , λ ) ξi , jx , , λ = ∫ Ai , jx , λ d ( PX × PX × Pf ) ( s , t , λ ) The set Ai , jx , represents all points in Xi × Xj that have lines between them intersecting an - neighborhood of x , while the set Ai , jx , , δ represents the restriction of A i , j x , to only those points whose connecting line segments intersect an -neighborhood of x with λ values bounded by 1− δ ( used in Section 3 ) . The set Xmix corresponds to all points for which every neighborhood factors into Jmix . The ξi , jx , term represents the measure of the set A i , j x , while ξ i , j x , , λ represents the expectation of λ over the same set . To provide better intuition for these definitions , we provide visualizations in Section B of the appendix . We can now define the subset of C to which we will constrain our optimization of Jmix . Definition 2.1 . Let C∗ to be the subset of C for which every h ∈ C∗ satisfies h ( x ) = lim →0 argminθ∈ [ 0,1 ] k Jmix ( θ ) |B ( x ) for all x ∈ Xmix when the limit exists . Here Jmix ( θ ) |B ( x ) represents the Mixup loss for a constant function with value θ with the restriction of each term in Jmix to the set Ai , jx , . We immediately justify this definition with the following proposition . Proposition 2.2 . Any function h ∈ argming∈C∗ Jmix ( g , PX , Pf ) satisfies Jmix ( h ) ≤ Jmix ( g ) for any continuous g ∈ C. Proof Sketch . We can argue directly from definitions by considering points in Xmix for which h and g differ . Proposition 2.2 demonstrates that optimizing over C∗ is at least as good as optimizing over the subset of C consisting of continuous functions , so we cover most cases of practical interest ( i.e . optimizing deep neural networks ) . As such , the term “ Mixup-optimal ” is intended to mean optimal with respect to C∗ throughout the rest of the paper . We may now characterize the classification of a Mixup-optimal classifier on Xmix . Lemma 2.3 . For any point x ∈ Xmix and > 0 , there exists a continuous function h satisfying : hi ( x ) = ξi , ix , + ∑ j 6=i ( ξi , jx , , λ + ( ξ j , i x , − ξ j , i x , , λ ) ) ∑k q=1 ( ξq , qx , + ∑ j 6=q ( ξq , jx , , λ + ( ξ j , q x , − ξj , qx , , λ ) ) ) ( 1 ) With the property that lim →0 h ( x ) = h ( x ) for every h ∈ argming∈C∗ Jmix ( g , PX , Pf ) when the limit exists . Proof Sketch . We set h = argminθ∈ [ 0,1 ] k Jmix ( θ ) |B ( x ) and show that this is well-defined , continuous , and has the above form using the strict convexity of the minimization problem . Remark 2.4 . For the important case of finite datasets , it will be shown that the limit above always exists as part of the proof of Theorem 3.2 . The expression for hi just represents the expected location of the point x on all lines between class i and other classes , normalized by the sum of the expected locations for all classes . It can be simplified significantly if Pf is assumed to be symmetric ; we give this as a corollary after the proof in Section C of the Appendix . Importantly , we note that while h as defined in Lemma 2.3 is continuous for every > 0 , its pointwise limit h need not be , which we demonstrate below . Proposition 2.5 . Let X1 = { ( 0 , 1 ) , ( 0 , −1 ) } and let X2 = { ( 1 , 0 ) , ( −1 , 0 ) } , with PX being discrete uniform overX1∪X2 and Pf being continuous uniform over [ 0 , 1 ] . Then the Mixup-optimal classifier h is discontinuous at ( 0 , 0 ) . Proof Sketch . One may explicitly compute for x = ( 0 , 0 ) that h1 ( x ) = h2 ( x ) = 12 . Proposition 2.5 illustrates our first significant difference between Mixup training and standard training : there always exists a minimizer of the empirical cross-entropy J that can be extended to a continuous function ( since a minimizer is constant on the class supports and not constrained elsewhere ) , whereas depending on the data the minimizer of Jmix can be discontinuous .

Summary: This paper provides interesting theoretical tools that help us study the relationship between empirical risk minimization/ empirical cross-entropy minimzation and Mixup minimization. The contributions of the paper consist of the following:  1. Constructing a dataset on which Mixup training fails to obtain the empirical risk minimization. 2. Providing sufficient conditions for minimizing the original risk and estimates of these sufficient condition measures on natural datasets. 3. An observation that the rate of empirical risk minimization using Mixup and a theorem about why this should happen. 4. A study on when the Mixup optimial classifier fails to generalize and a demonstration on a toy dataset. 5. An example for when Mixup training leads to the same classifier, obtained from standard training.  The main difference with related works is that the paper attempts to understand *why* Mixup works by studying *when* Mixup works with concrete examples.