Noisy Feature Mixup

1 INTRODUCTION . Mitigating over-fitting and improving generalization on test data are central goals in machine learning . One approach to accomplish this is regularization , which can be either data-agnostic or data-dependent ( e.g. , explicitly requiring the use of domain knowledge or data ) . Noise injection is a typical example of data-agnostic regularization ( Bishop , 1995 ) , where noise can be injected into the input data ( An , 1996 ) , or the activation functions ( Gulcehre et al. , 2016 ) , or the hidden layers of deep neural networks ( Camuto et al. , 2020 ; Lim et al. , 2021 ) . Data augmentation constitutes a different class of regularization methods ( Baird , 1992 ; Chapelle et al. , 2001 ; DeCoste & Schölkopf , 2002 ) , which can also be either data-agnostic or data-dependent . Data augmentation involves training a model with not just the original data , but also with additional data that is properly transformed , and it has led to state-of-the-art results in image recognition ( Cireşan et al. , 2010 ; Krizhevsky et al. , 2012 ) . The recently-proposed data-agnostic method , mixup ( Zhang et al. , 2017 ) , trains a model on linear interpolations of a random pair of examples and their corresponding labels , thereby encouraging the model to behave linearly in-between training examples . Both noise injection and mixup have been shown to impose smoothness and increase model robustness to data perturbations ( Zhang et al. , 2020 ; Carratino et al. , 2020 ; Lim et al. , 2021 ) , which is critical for many safety and sensitive applications ( Goodfellow et al. , 2018 ; Madry et al. , 2017 ) . In this paper , we propose and study a simple yet effective data augmentation method , which we call Noisy Feature Mixup ( NFM ) . This method combines mixup and noise injection , thereby inheriting the benefits of both methods , and it can be seen as a generalization of input mixup ( Zhang et al. , 2017 ) and manifold mixup ( Verma et al. , 2019 ) . When compared to noise injection and mixup , NFM imposes regularization on the largest natural region surrounding the dataset ( see Fig . 1 ) , which may help improve robustness and generalization when predicting on out of distribution data . Conveniently , NFM can be implemented on top of manifold mixup , introducing minimal computation overhead . Contributions . Our main contributions in this paper are summarized as follows . • We study NFM via the lens of implicit regularization , showing that NFM amplifies the regularizing effects of manifold mixup and noise injection , implicitly reducing the feature-output Jacobians and Hessians according to the mixing level and noise levels ( see Theorem 1 ) . • We provide mathematical analysis to show that NFM can further improve model robustness when compared to manifold mixup and noise injection . In particular , we show that , under appropriate assumptions , NFM training approximately minimizes an upper bound on the sum of an adversarial loss and feature-dependent regularizers ( see Theorem 2 ) . • We provide empirical results in support of our theoretical findings , showing that NFM improves robustness with respect to various forms of data perturbation across a wide range of state-of-theart architectures on computer vision benchmark tasks . In the Supplementary Materials ( SM ) , we provide proofs for our theorems along with additional theoretical and empirical results to gain more insights into NFM . In particular , we show that NFM can implicitly increase classification margin ( see Proposition 1 in SM C ) and the noise injection procedure in NFM can robustify manifold mixup in a probabilistic sense ( see Theorem 5 in SM D ) . We also provide and discuss generalization bounds for NFM ( see Theorem 6 and 7 in SM E ) . Notation . I denotes identity matrix , [ K ] : = { 1 , . . . , K } , the superscript T denotes transposition , ◦ denotes composition , denotes Hadamard product , 1 denotes the vector with all components equal one . For a vector v , vk denotes its kth component and ‖v‖p denotes its lp norm for p > 0. conv ( X ) denote the convex hull of X . Mλ ( a , b ) : = λa+ ( 1−λ ) b , for random variables a , b , λ. δz denotes the Dirac delta function , defined as δz ( x ) = 1 if x = z and δz ( x ) = 0 otherwise . 1A denotes indicator function of the set A . For α , β > 0 , D̃λ : = αα+βBeta ( α + 1 , β ) + β α+βBeta ( β + 1 , α ) denotes a uniform mixture of two Beta distributions . For two vectors a , b , cos ( a , b ) : = 〈a , b〉/‖a‖2‖b‖2 denotes their cosine similarity . N ( a , b ) is a Gaussian distribution with mean a and covariance b . 2 RELATED WORK . Regularization . Regularization refers to any technique that reduces overfitting in machine learning ; see ( Mahoney & Orecchia , 2011 ; Mahoney , 2012 ) and references therein , in particular for a discussion of implicit regularization , a topic that has received attention recently in the context of stochastic gradient optimization applied to neural network models . Traditional regularization techniques such as ridge regression , weight decay and dropout do not make use of the training data to reduce the model capacity . A powerful class of techniques is data augmentation , which constructs additional examples from the training set , e.g. , by applying geometric transformations to the original data ( Shorten & Khoshgoftaar , 2019 ) . A recently proposed technique is mixup ( Zhang et al. , 2017 ) , where the examples are created by taking convex combinations of pairs of inputs and their labels . Verma et al . ( 2019 ) extends mixup to hidden representations in deep neural networks . Subsequent works by Greenewald et al . ( 2021 ) ; Yin et al . ( 2021 ) ; Engstrom et al . ( 2019 ) ; Kim et al . ( 2020a ) ; Yun et al . ( 2019 ) ; Hendrycks et al . ( 2019 ) introduce different variants and extensions of mixup . Regularization is also intimately connected to robustness ( Hoffman et al. , 2019 ; Sokolić et al. , 2017 ; Novak et al. , 2018 ; Elsayed et al. , 2018 ; Moosavi-Dezfooli et al. , 2019 ) . Adding to the list is NFM , a powerful regularization method that we propose to improve model robustness . Robustness . Model robustness is an increasingly important issue in modern machine learning . Robustness with respect to adversarial examples ( Kurakin et al. , 2016 ) can be achieved by adversarial training ( Goodfellow et al. , 2014 ; Madry et al. , 2017 ; Utrera et al. , 2020 ) . Several works present theoretical justifications to observed robustness and how data augmentation can improve it ( Hein & Andriushchenko , 2017 ; Yang et al. , 2020b ; Couellan , 2021 ; Pinot et al. , 2019a ; 2021 ; Zhang et al. , 2020 ; 2021 ; Carratino et al. , 2020 ; Kimura , 2020 ; Dao et al. , 2019 ; Wu et al. , 2020 ; Gong et al. , 2020 ; Chen et al. , 2020 ) . Relatedly , Fawzi et al . ( 2016 ) ; Franceschi et al . ( 2018 ) ; Lim et al . ( 2021 ) investigate how noise injection can be used to improve robustness . Parallel to this line of work , we provide theory to understand how NFM can improve robustness . Also related is the study of the trade-offs between robustness and accuracy ( Min et al. , 2020 ; Zhang et al. , 2019 ; Tsipras et al. , 2018 ; Schmidt et al. , 2018 ; Su et al. , 2018 ; Raghunathan et al. , 2020 ; Yang et al. , 2020a ) . 3 NOISY FEATURE MIXUP . Noisy Feature Mixup is a generalization of input mixup ( Zhang et al. , 2017 ) and manifold mixup ( Verma et al. , 2019 ) . The main novelty of NFM against manifold mixup lies in the injection of noise when taking convex combinations of pairs of input and hidden layer features . Fig . 1 illustrates , at a high level , how this modification alters the region in which the resulting augmented data resides . Fig . 2 shows that NFM is most effective at smoothing the decision boundary of the trained classifiers ; compared to noise injection and mixup alone , it imposes the strongest smoothness on this dataset . Formally , we consider multi-class classification with K labels . Denote the input space by X ⊂ Rd and the output space by Y = RK . The classifier , g , is constructed from a learnable map f : X → RK , mapping an input x to its label , g ( x ) = arg maxk fk ( x ) ∈ [ K ] . We are given a training set , Zn : = { ( xi , yi ) } ni=1 , consisting of n pairs of input and one-hot label , with each training pair zi : = ( xi , yi ) ∈ X × Y drawn i.i.d . from a ground-truth distribution D. We consider training a deep neural network f : = fk ◦ gk , where gk : X → gk ( X ) maps an input to a hidden representation at layer k , and fk : gk ( X ) → gL ( X ) : = Y maps the hidden representation to a one-hot label at layer L. Here , gk ( X ) ⊂ Rdk for k ∈ [ L ] , dL : = K , g0 ( x ) = x and f0 ( x ) = f ( x ) . Training f using NFM consists of the following steps : 1 . Select a random layer k from a set , S ⊂ { 0 } ∪ [ L ] , of eligible layers in the neural network . 2 . Process two random data minibatches ( x , y ) and ( x′ , y′ ) as usual , until reaching layer k. This gives us two immediate minibatches ( gk ( x ) , y ) and ( gk ( x′ ) , y′ ) . 3 . Perform mixup on these intermediate minibatches , producing the mixed minibatch : ( g̃k , ỹ ) : = ( Mλ ( gk ( x ) , gk ( x ′ ) ) , Mλ ( y , y ′ ) ) , ( 1 ) where the mixing level λ ∼ Beta ( α , β ) , with the hyper-parameters α , β > 0 . 4 . Produce noisy mixed minibatch by injecting additive and multiplicative noise : ( ˜̃gk , ỹ ) : = ( ( 1 + σmultξ mult k ) Mλ ( gk ( x ) , gk ( x′ ) ) + σaddξaddk , Mλ ( y , y′ ) ) , ( 2 ) where the ξaddk and ξ mult k are R dk -valued independent random variables modeling the additive and multiplicative noise respectively , and σadd , σmult ≥ 0 are pre-specified noise levels . 5 . Continue the forward pass from layer k until the output using the noisy mixed minibatch ( ˜̃gk , ỹ ) . 6 . Compute the loss and gradients that update all the parameters of the network . At the level of implementation , following ( Verma et al. , 2019 ) , we backpropagate gradients through the entire computational graph , including those layers before the mixup layer k. In the case where σadd = σmult = 0 , NFM reduces to manifold mixup ( Verma et al. , 2019 ) . If in addition S = { 0 } , it reduces to the original mixup method ( Zhang et al. , 2017 ) . The main difference between NFM and manifold mixup lies in the noise injection of the fourth step above . Note that NFM is equivalent to injecting noise into gk ( x ) , gk ( x′ ) first , then performing mixup on the resulting pair , i.e. , the order that the third and fourth steps occur does not change the resulting noisy mixed minibatch . For simplicity , we have used the same mixing level , noise distribution , and noise levels for all layers in S in our formulation . Within the above setting , we consider the expected NFM loss : LNFM ( f ) = E ( x , y ) , ( x′ , y′ ) ∼DEk∼SEλ∼Beta ( α , β ) Eξk∼Ql ( fk ( Mλ , ξk ( gk ( x ) , gk ( x ′ ) ) ) , Mλ ( y , y ′ ) ) , where l : RK × RK → [ 0 , ∞ ) is a loss function ( note that here we have suppressed the dependence of both l and f on the learnable parameter θ in the notation ) , ξk : = ( ξaddk , ξ mult k ) are drawn from some probability distribution Q with finite first two moments , and Mλ , ξk ( gk ( x ) , gk ( x ′ ) ) : = ( 1 + σmultξ mult k ) Mλ ( gk ( x ) , gk ( x′ ) ) + σaddξaddk . NFM seeks to minimize a stochastic approximation of LNFM ( f ) by sampling a finite number of k , λ , ξk values and using minibatch gradient descent to minimize this loss approximation .

Summary: The paper proposes a new and inexpensive mixup method named Noisy Feature Mixup (NFM) to mitigate over-fitting and improve generalization. NFM combines mixup and noise injection, which inherits the benefits of these methods. Due to its conciseness, it is convenient to apply NFM in model training. More importantly, the authors prove NFM's regularization effect on model optimization and robustness improvement with mathematical derivation. They build the implicit connection between the NFM empirical loss and the original loss. Then they identify the regularizing effects of the proposed NFM. To demonstrate that NFM helps robustness, the authors relate the NFM loss to the one used for adversarial training, which can be regarded as an example belonging to distributionally robust optimization. With integrated experiments, NFM shows its superiority on various models and datasets. The further discussion shows the interesting tradeoff between predictive accuracy on clean and perturbed test sets. The Supplementary Material provides detailed proof and additional experimental results to show NFM's firmed ground.