How Does Mixup Help With Robustness and Generalization?

1 INTRODUCTION . Mixup was introduced by Zhang et al . ( 2018 ) as a data augmentation technique . It has been empirically shown to substantially improve test performance and robustness to adversarial noise of state-of-the-art neural network architectures ( Zhang et al. , 2018 ; Lamb et al. , 2019 ; Thulasidasan et al. , 2019 ; Zhang et al. , 2018 ; Arazo et al. , 2019 ) . Despite the impressive empirical performance , it is still not fully understood why Mixup leads to such improvement across the different aspects mentioned above . We first provide more background about robustness and generalization properties of deep networks and Mixup . Then we give an overview of our main contributions . Adversarial robustness . Although neural networks have achieved remarkable success in many areas such as natural language processing ( Devlin et al. , 2018 ) and image recognition ( He et al. , 2016a ) , it has been observed that neural networks are very sensitive to adversarial examples — prediction can be easily flipped by human imperceptible perturbations ( Goodfellow et al. , 2014 ; Szegedy et al. , 2013 ) . Specifically , in Goodfellow et al . ( 2014 ) , the authors use fast gradient sign method ( FGSM ) to generate adversarial examples , which makes an image of panda to be classified as gibbon with high confidence . Although various defense mechanisms have been proposed against adversarial attacks , those mechanisms typically sacrifice test accuracy in turn for robustness ( Tsipras et al. , 2018 ) and many of them require a significant amount of additional computation time . In contrast , Mixup training tends to improve test accuracy and at the same time also exhibits a certain degree of resistance to adversarial examples , such as those generated by FGSM ( Lamb et al. , 2019 ) . Moreover , the corresponding training time is relatively modest . As an illustration , we compare the robust test ∗Equal contribution . accuracy between a model trained with Mixup and a model trained with standard empirical risk minimization ( ERM ) under adversarial attacks generated by FGSM ( Fig . 1a ) . The model trained with Mixup loss has much better robust accuracy . Robustness of Mixup under other attacks have also been empirically studied in Lamb et al . ( 2019 ) . Generalization . Generalization theory has been a central focus of learning theory ( Vapnik , 1979 ; 2013 ; Bartlett et al. , 2002 ; Bartlett & Mendelson , 2002 ; Bousquet & Elisseeff , 2002 ; Xu & Mannor , 2012 ) , but it still remains a mystery for many modern deep learning algorithms ( Zhang et al. , 2016 ; Kawaguchi et al. , 2017 ) . For Mixup , from Fig . ( 1b ) , we observe that Mixup training results in better test performance than the standard empirical risk minimization . That is mainly due to its good generalization property since the training errors are small for both Mixup training and empirical risk minimization ( experiments with training error results are included in the appendix ) . While there have been many enlightening studies trying to establish generalization theory for modern machine learning algorithms ( Sun et al. , 2015 ; Neyshabur et al. , 2015 ; Hardt et al. , 2016 ; Bartlett et al. , 2017 ; Kawaguchi et al. , 2017 ; Arora et al. , 2018 ; Neyshabur & Li , 2019 ) , few existing studies have illustrated the generalization behavior of Mixup training in theory . Our contributions . In this paper , we theoretically investigate how Mixup improves both adversarial robustness and generalization . We begin by relating the loss function induced by Mixup to the standard loss with additional adaptive regularization terms . Based on the derived regularization terms , we show that Mixup training minimizes an upper bound on the adversarial loss , which leads to the robustness against single-step adversarial attacks . For generalization , we show how the regularization terms can reduce over-fitting and lead to better generalization behaviors than those of standard training . Our analyses provides insights and framework to understand the impact of Mixup . Outline of the paper . Section 2 introduces the notations and problem setup . In Section 3 , we present our main theoretical results , including the regularization effect of Mixup and the subsequent analysis to show that such regularization improves adversarial robustness and generalization . Section 4 concludes with a discussion of future work . Proofs are deferred to the Appendix . 1.1 RELATED WORK . Since its advent , Mixup training ( Zhang et al. , 2018 ) has been shown to substantially improve generalization and single-step adversarial robustness among a wide rage of tasks , on both supervised ( Lamb et al. , 2019 ; Verma et al. , 2019a ; Guo et al. , 2019 ) , and semi-supervised settings ( Berthelot et al. , 2019 ; Verma et al. , 2019b ) . This has motivated a recent line of work for developing a number of variants of Mixup , including Manifold Mixup ( Verma et al. , 2019a ) , Puzzle Mix ( Kim et al. , 2020 ) , CutMix ( Yun et al. , 2019 ) , Adversarial Mixup Resynthesis ( Beckham et al. , 2019 ) , and PatchUp ( Faramarzi et al. , 2020 ) . However , theoretical understanding of the underlying mechanism of why Mixup and its variants perform well on generalization and adversarial robustness is still limited . Some of the theoretical tools we use in this paper are related to Wang & Manning ( 2013 ) and Wager et al . ( 2013 ) , where the authors use second-order Taylor approximation to derive a regularized loss function for Dropout training . This technique is then extended to drive more properties of Dropout , including the inductive bias of Dropout ( Helmbold & Long , 2015 ) , the regularization effect in matrix factorization ( Mianjy et al. , 2018 ) , and the implicit regularization in neural networks ( Wei et al. , 2020 ) . This technique has been recently applied to Mixup in a parallel and independent work ( Carratino et al. , 2020 ) to derive regularization terms . Compared with the results in Carratino et al . ( 2020 ) , our derived regularization enjoys a simpler form and therefore enables the subsequent analysis of adversarial robustness and generalization . We clarify the detailed differences in Section 3 . To the best of our knowledge , our paper is the first to provide a theoretical treatment to connect the regularization , adversarial robustness , and generalization for Mixup training . 2 PRELIMINARIES . In this section , we state our notations and briefly recap the definition of Mixup . Notations . We denote the general parameterized loss as l ( θ , z ) , where θ ∈ Θ ⊆ Rd and z = ( x , y ) is the input and output pair . We consider a training dataset S = { ( x1 , y1 ) , · · · , ( xn , yn ) } , where xi ∈ X ⊆ Rp and yi ∈ Y ⊆ Rm are i.i.d . drawn from a joint distribution Px , y . We further denote x̃i , j ( λ ) = λxi + ( 1 − λ ) xj , ỹi , j ( λ ) = λyi + ( 1 − λ ) yj for λ ∈ [ 0 , 1 ] and let z̃i , j ( λ ) = ( x̃i , j ( λ ) , ỹi , j ( λ ) ) . Let L ( θ ) = Ez∼Px , y l ( θ , z ) denote the standard population loss and Lstdn ( θ , S ) = ∑n i=1 l ( θ , zi ) /n denote the standard empirical loss . For the two distributions D1 and D2 , we use pD1 + ( 1− p ) D2 for p ∈ ( 0 , 1 ) to denote the mixture distribution such that a sample is drawn with probabilities p and ( 1 − p ) from D1 and D2 respectively . For a parameterized function fθ ( x ) , we use∇fθ ( x ) and∇θfθ ( x ) to respectively denote the gradient with respect to x and θ . For two vectors a and b , we use cos ( x , y ) to denote 〈x , y〉/ ( ‖x‖ · ‖y‖ ) . Mixup . Generally , for classification cases , the output yi is the embedding of the class of xi , i.e . the one-hot encoding by taking m as the total number of classes and letting yi ∈ { 0 , 1 } m be the binary vector with all entries equal to zero except for the one corresponding to the class of xi . In particular , if we take m = 1 , it degenerates to the binary classification . For regression cases , yi can be any real number/vector . The Mixup loss is defined in the following form : Lmixn ( θ , S ) = 1 n2 n∑ i , j=1 Eλ∼Dλ l ( θ , z̃ij ( λ ) ) , ( 1 ) where Dλ is a distribution supported on [ 0 , 1 ] . Throughout the paper , we consider the most commonly used Dλ – Beta distribution Beta ( α , β ) for α , β > 0 . 3 MAIN RESULTS . In this section , we first introduce a lemma that characterizes the regularization effect of Mixup . Based on this lemma , we then derive our main theoretical results on adversarial robustness and generalization error bound in Sections 3.2 and 3.3 respectively . 3.1 THE REGULARIZATION EFFECT OF MIXUP . As a starting point , we demonstrate how Mixup training is approximately equivalent to optimizing a regularized version of standard empirical loss Lstdn ( θ , S ) . Throughout the paper , we consider the following class of loss functions for the prediction function fθ ( x ) and target y : L = { l ( θ , ( x , y ) ) |l ( θ , ( x , y ) ) = h ( fθ ( x ) ) − yfθ ( x ) for some function h } . ( 2 ) This function class L includes many commonly used losses , including the loss function induced by Generalized Linear Models ( GLMs ) , such as linear regression and logistic regression , and also crossentropy for neural networks . In the following , we introduce a lemma stating that the Mixup training with λ ∼ Dλ = Beta ( α , β ) induces a regularized loss function with the weights of each regularization specified by a mixture of Beta distributions D̃λ = αα+βBeta ( α+ 1 , β ) + β α+βBeta ( β + 1 , α ) . Lemma 3.1 . Consider the loss function l ( θ , ( x , y ) ) = h ( fθ ( x ) ) − yfθ ( x ) , where h ( · ) and fθ ( · ) for all θ ∈ Θ are twice differentiable . We further denote D̃λ as a uniform mixture of two Beta distributions , i.e. , αα+βBeta ( α+ 1 , β ) + β α+βBeta ( β+ 1 , α ) , andDX as the empirical distribution of the training dataset S = ( x1 , · · · , xn ) , the corresponding Mixup loss Lmixn ( θ , S ) , as defined in Eq . ( 1 ) with λ ∼ Dλ = Beta ( α , β ) , can be rewritten as Lmixn ( θ , S ) = L std n ( θ , S ) + 3∑ i=1 Ri ( θ , S ) + Eλ∼D̃λ [ ( 1− λ ) 2ϕ ( 1− λ ) ] , where lima→0 ϕ ( a ) = 0 and R1 ( θ , S ) = Eλ∼D̃λ [ 1− λ ] n n∑ i=1 ( h′ ( fθ ( xi ) ) − yi ) ∇fθ ( xi ) > Erx∼DX [ rx − xi ] , R2 ( θ , S ) = Eλ∼D̃λ [ ( 1− λ ) 2 ] 2n n∑ i=1 h′′ ( fθ ( xi ) ) ∇fθ ( xi ) > Erx∼DX [ ( rx − xi ) ( rx − xi ) > ] ∇fθ ( xi ) , R3 ( θ , S ) = Eλ∼D̃λ [ ( 1− λ ) 2 ] 2n n∑ i=1 ( h′ ( fθ ( xi ) ) − yi ) Erx∼DX [ ( rx − xi ) ∇2fθ ( xi ) ( rx − xi ) > ] . By putting the higher order terms of approximation in ϕ ( · ) , this result shows that Mixup is related to regularizing ∇fθ ( xi ) and ∇2fθ ( xi ) , which are the first and second directional derivatives with respect to xi . Throughout the paper , our theory is mainly built upon analysis of the quadratic approximation of Lmixn ( θ , S ) , which we further denote as L̃mixn ( θ , S ) : = L std n ( θ , S ) + 3∑ i=1 Ri ( θ , S ) . ( 3 ) Comparison with related work . The result in Lemma 3.1 relies on the second-order Taylor expansion of the loss function Eq . ( 1 ) . Similar approximations have been proposed before to study the regularization effect of Dropout training , see Wang & Manning ( 2013 ) ; Wager et al . ( 2013 ) ; Mianjy et al . ( 2018 ) ; Wei et al . ( 2020 ) . Recently , Carratino et al . ( 2020 ) independently used similar approximation to study the regularization effect of Mixup . However , the regularization terms derived in Carratino et al . ( 2020 ) is much more complicated than those in Lemma 3.1 . For example , in GLM , our technique yields the regularization term as shown in Lemma 3.3 , which is much simpler than those in Corollaries 2 and 3 in Carratino et al . ( 2020 ) . One technical step we use here to simplify the regularization expression is to equalize Mixup with input perturbation , see more details in the proof in the Appendix . This simpler expression enables us to study the robustness and generalization of Mixup in the subsequent sections . Validity of the approximation . In the following , we present numerical experiments to support the approximation in Eq . ( 3 ) . Following the setup of numerical validations in Wager et al . ( 2013 ) ; Carratino et al . ( 2020 ) , we experimentally show that the quadratic approximation is generally very accurate . Specifically , we train a Logistic Regression model ( as one example of a GLM model , which we study later ) and a two layer neural network with ReLU activations . We use the two-moons dataset ( Buitinck et al. , 2013 ) . Fig . 2 shows the training and test data ’ s loss functions for training two models with different loss functions : the original Mixup loss and the approximate Mixup loss . Both models had the same random initialization scheme . Throughout training , we compute the test and training loss of each model using its own loss function . The empirical results shows the approximation of Mixup loss is quite close to the original Mixup loss .

Summary: The paper theoretically studies the beneficial effect of mixup on robustness and generalization of machine models. The mixup loss is  rewritten to be the sum of the original empirical loss and a regularization term (plus a high order term). For robustness, the regularization term is proven to be upper bound of first and second order terms of the adversarial loss's Taylor expansion. Hence, the mixup loss can upper bound the approximate adversarial loss. For generalization, the regularization term is used to control the hypothesis to have small Rademacher complexity. The paper is clearly written and well organized.