WeMix: How to Better Utilize Data Augmentation

1 INTRODUCTION . Data augmentation ( Baird , 1992 ; Schmidhuber , 2015 ) has been a key to the success of deep learning in image classification ( He et al. , 2019 ) , and is becoming increasingly common in other tasks such as natural language processing ( Zhang et al. , 2015 ) and object detection ( Zoph et al. , 2019 ) . The data augmentation expands training set by generating virtual instances through random augmentation to the original ones . This alleviates the overfitting ( Shorten & Khoshgoftaar , 2019 ) problem when training large deep neural networks . Despite many encouraging results , it is not the case that data augmentation will always improve generalization errors ( Min et al. , 2020 ; Raghunathan et al. , 2020 ) . In particular , Raghunathan et al . ( 2020 ) showed that training by augmented data will lead to a smaller robust error but potentially a larger standard error . Therefore , it is critical to answer the following two questions before applying data augmentation in deep learning : • When will the deep models benefit from data augmentation ? • How to better leverage augmented data during training ? Several previous works ( Raghunathan et al. , 2020 ; Wu et al. , 2020 ; Min et al. , 2020 ) tried to address the questions . Their analysis is limited to specific problems such as linear ridge regression therefore may not be applicable to deep learning . In this work , we aim to answer the two questions from a theoretical perspective under a more general non-convex setting . We address the first question in a more general form covering applications in deep learning . For the second question , we develop new approaches that are provably more effective than the conventional data augmentation approaches . Most data augmentation operations alter the data distribution during the training progress . This imposes a data distribution bias ( we simply use “ data bias ” in the rest of this paper ) between the augmented data and the original data , which may make it difficult to fully leverage the augmented data . To be more concrete , let us consider label-mixing augmentation ( e.g. , mixup ( Zhang et al. , 2018 ; Tokozume et al. , 2018 ) ) . Suppose we have n original data D = { ( xi , yi ) , i = 1 , . . . , n } , where the input-label pair ( xi , yi ) follows a distribution Pxy = ( Px , Py ( ·|x ) ) , Px is the marginal distribution of the inputs and Py ( ·|x ) is the conditional distribution of the labels given inputs ; we generate m augmented data D̃ = { ( x̃i , ỹi ) , i = 1 , . . . , m } , where ( x̃i , ỹi ) ∼ Px̃ỹ = ( Px̃ , Pỹ ( ·|x̃ ) ) , and Px = Px̃ but Py ( ·|x ) 6= Pỹ ( ·|x̃ ) . Given x ∼ Px , the data bias is defined as δy = maxy , ỹ ‖y − ỹ‖ . We will show that when the bias between D and D̃ is large , directly training on the augmented data will not be as effective as training on the original data . Given the fact that augmented data may hurt the performance , the next question is how to design better learning algorithms to leash out the power of augmented data . To this end , we develop two novel algorithms to alleviate the data bias . The first algorithm , termed AugDrop , corrects the data bias by introducing a constrained optimization problem . The second algorithm , termed MixLoss , corrects the data bias by introducing a modified loss function . We show that , both theoretically and empirically , even with a large data bias , the proposed algorithms can still improve the generalization performance by effectively leveraging the combination of augmented data and original data . We summarize the main contributions of this work as follows : • We prove that in a conventional training scheme , a deep model can benefit from augmented data when the data bias is small . • We design two algorithms termed AugDrop and MixLoss that can better leverage augmented data even when the data bias is large with theoretical guarantees . • Based on our theoretical findings , we empirically propose a new efficient algorithm WeMix by combining AugDrop and MixLoss , which has better performances without extra training cost . 2 RELATED WORK . A series of empirical works ( Cubuk et al. , 2019 ; Ho et al. , 2019 ; Lim et al. , 2019 ; Lin et al. , 2019a ; Cubuk et al. , 2020 ; Hataya et al. , 2019 ) on how to learn a good policy of using different data augmentations have been proposed without theoretical guarantees . In this section , we mainly focus on reviewing theoretical studies on data augmentation . For a survey of data augmentation , we refer readers to ( Shorten & Khoshgoftaar , 2019 ) and references therein for a comprehensive overview . Several works have attempted to establish theoretical understandings of data augmentation from different perspectives ( Dao et al. , 2019 ; Chen et al. , 2019 ; Rajput et al. , 2019 ) . Min et al . ( 2020 ) shown that , with more training data , weak augmentation can improve performance while strong augmentation always hurts the performance . Later on , Chen et al . ( 2020 ) study the gap between the generalization error ( please see the formal definition in ( Chen et al. , 2020 ) ) of adversarially-trained models and standard models . Both of their theoretical analyses were built on special linear binary classification model or linear regression model for label-preserving augmentation . Recently , Raghunathan et al . ( 2020 ) studied label-preserving transformation in data augmentation , which is identical to the first case in this paper . Their analysis is restricted to linear least square regression under noiseless setting , which is not applicable to training deep neural networks . Besides , their analysis requires infinite unlabeled data . By contrast , we do not require the original data to be unlimited . Wu et al . ( 2020 ) considered linear data augmentations . There are several major differences between their work and ours . First , they focus on the ridge linear regression problem which is strongly convex , while we consider non-convex optimization problems , which is more applicable in deep learning . Second , we study more general data augmentations beyond linear transformation . 3 PRELIMINARIES AND NOTATIONS . We study a learning problem for finding a classifier to map an input x ∈ X onto a label y ∈ Y ⊂ RK , where K is the number of classes . We assume the input-label pair ( x , y ) is drawn from a distribution Pxy = ( Px , Py ( ·|x ) ) . Since every augmented example ( x̃ , ỹ ) is generated by applying a certain transformation to either one or multiple examples , we will assume that ( x̃ , ỹ ) is drawn from a slightly different distribution Px̃ỹ = ( Px̃ , Pỹ ( ·|x̃ ) ) , where Px̃ is the marginal distribution on the inputs x̃ and Pỹ ( ·|x̃ ) ) ( we can write it as Pỹ for simplicity ) is the conditional distribution of the labels ỹ given inputs x̃ . We sample n training examples ( xi , yi ) , i = 1 , . . . , n from distribution Pxy and m training examples ( x̃i , ỹi ) , i = 1 , . . . , m from Px̃ỹ . We assume that m n due to the data augmentation . We denote by D = { ( xi , yi ) , i = 1 , . . . , n } and D̃ = ( x̃i , ỹi ) , i = 1 , . . . , m } the dataset sampled from Pxy and Px̃ỹ , respectively . We denote by T ( x ) the set of augmented data transformed from x . We use the notation E ( x , y ) ∼Pxy [ · ] to stand for the expectation that takes over a random variable ( x , y ) following a distribution Pxy . We denote by ∇wh ( w ) the gradient of a function h ( w ) in terms of variable w. When the variable to be taken a gradient is obvious , we use the notation ∇h ( w ) for simplicity . Let use ‖ · ‖ as the Euclidean norm for a vector or the Spectral norm for a matrix . The augmented data D̃ can be different from the original data D in two cases , according to ( Raghunathan et al. , 2020 ) . In the first case , often referred to as label-preserving , we consider Py ( ·|x ) = Pỹ ( ·|x̃ ) , ∀x̃ ∈ T ( x ) but Px 6= Px̃ . ( 1 ) In the second case , often referred to as label-mixing , we consider Px = Px̃ but Py ( ·|x ) 6= Pỹ ( ·|x̃ ) , ∃x̃ ∈ T ( x ) . ( 2 ) Examples of label-preserving augmentation include translation , adding noises , small rotation , and brightness or contrast changes ( Krizhevsky et al. , 2012 ; Raghunathan et al. , 2020 ) . One important example of label-mixing augmentation is mixup ( Zhang et al. , 2018 ; Tokozume et al. , 2018 ) . Due to the space limitation , we will focus on the label-mixing case , and the related studies and analysis for the label-preserving case can be found in Appendix A . To further quantify the difference between original data and augmented data when Px = Px̃ and Py 6= Pỹ , we introduce the data bias δy given x ∼ Px as following : δy : = max y , ỹ ‖y − ỹ‖ . ( 3 ) The equation in ( 3 ) measures the difference between the label from original data and the label from augmented data given input x . We aim to learn a prediction function f ( x ; w ) : RD × X → RK that is as close as possible to y , where w ∈ RD is the parameter and RD is a closed convex set . We respectively define two objective functions for optimization problems over the original data and the augmented data as L ( w ) = E ( x , y ) [ ` ( y , f ( x ; w ) ) ] , L̃ ( w ) = E ( x̃ , ỹ ) [ ` ( ỹ , f ( x̃ ; w ) ) ] , ( 4 ) where ` is a cross-entropy loss function which is given by ` ( y , f ( x ; w ) ) = K∑ i=1 yipi ( x ; w ) , where pi ( x ; w ) = − log ( exp ( fi ( x ; w ) ) ∑K j=1 exp ( fj ( x ; w ) ) ) . ( 5 ) We denote by w∗ and w̃∗ the optimal solutions to minw L ( w ) and minw L̃ ( w ) respectively , w∗ ∈ arg min w∈RD L ( w ) , w̃∗ ∈ arg min w∈RD L̃ ( w ) . ( 6 ) Taking L ( w ) as an example , we introduce some function properties used in our analysis . Definition 1 . The stochastic gradients of the objective functions L ( w ) is unbiased and bounded , if we have E ( x , y ) [ ∇w ` ( y , f ( x ; w ) ) ] = ∇L ( w ) , and there exists a constant G > 0 , such that ‖∇wp ( x ; w ) ‖ ≤ G , ∀x ∈ X , ∀w ∈ RD , where p ( x ; w ) = ( p1 ( x ; w ) , . . . , pK ( x ; w ) ) is a vector . Definition 2 . L ( w ) is smooth with an L-Lipchitz continuous gradient , if there exists a constant L > 0 such that ‖∇L ( w ) −∇L ( u ) ‖ ≤ L‖w− u‖ , ∀w , u ∈ RD , or equivalently , L ( w ) −L ( u ) ≤ 〈∇L ( u ) , w − u〉+ L2 ‖w − u‖ 2 , ∀w , u ∈ RD . The above properties are standard and widely used in the literature of non-convex optimization ( Ghadimi & Lan , 2013 ; Yan et al. , 2018 ; Yuan et al. , 2019 ; Wang et al. , 2019 ; Li et al. , 2020 ) . We introduce an important property termed Polyak-Łojasiewicz ( PL ) condition ( Polyak , 1963 ) on the objective function L ( w ) . Definition 3 . ( PL condition ) L ( w ) satisfies the PL condition , if there exists a constant µ > 0 such that 2µ ( L ( w ) − L ( w∗ ) ) ≤ ‖∇L ( w ) ‖2 , ∀w ∈ RD , where w∗ is defined in ( 6 ) . The PL condition has been observed in training deep and shallow neural networks ( Allen-Zhu et al. , 2019 ; Xie et al. , 2017 ) , and is widely used in many non-convex optimization studies ( Karimi et al. , 2016 ; Li & Li , 2018 ; Charles & Papailiopoulos , 2018 ; Yuan et al. , 2019 ; Li et al. , 2020 ) . It is also theoretically verified in ( Allen-Zhu et al. , 2019 ) and empirically estimated in ( Yuan et al. , 2019 ) for deep neural networks . It is worth noting that PL condition is weaker than many conditions such as strong convexity , restricted strong convexity and weak strong convexity ( Karimi et al. , 2016 ) . Finally , we will refer to κ = Lµ as condition number throughout this study .

Summary: This paper proposes a method to improve the generalization of deep networks when the input is applied strong augmentations, i.e. mixup, resulting in large data bias. The authors proposes theoretical foundation behind their two methods, AugDrop and MixLoss and show their effectiveness on CIFAR10/CIFAR100 datasets. Although the paper has technical novelty and it only provides incremental gains on relatively small-size dataset. Below, you can find some of my comments/questions about the paper.