Theoretical Analysis of Consistency Regularization with Limited Augmented Data

1 INTRODUCTION . Modern machine learning models , especially deep learning models , require abundant training samples . Since data collection and human annotation are expensive , data augmentation has been a ubiquitous practice in creating artificial labeled samples and improving generalization performance . This practice is corroborated by the fact that the semantics of images remain the same through simple translations like obscuring , flipping , rotation , color jitter , rescaling ( Shorten & Khoshgoftaar , 2019 ) . Conventional algorithms use data augmentation to expand the training data set ( Krizhevsky et al. , 2012 ; Simard et al. , 1998 ; Cubuk et al. , 2018 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ) . As an alternative , consistency regularization enforces the model to output similar predictions on the original and augmented samples and contributes to many recent state-of-the-art supervised or semisupervised algorithms . This idea was first proposed in ( Bachman et al. , 2014 ) and popularized by Laine & Aila ( 2016 ) ; Sajjadi et al . ( 2016 ) , and gained more attention recently with the success of FixMatch ( Sohn et al. , 2020 ) for semi-supervised few-shot learning , and AdaMatch ( Berthelot et al. , 2021 ) for domain adaptation . Several recent papers ( see e.g . ( Chen et al. , 2020a ; Mei et al. , 2021 ; Lyle et al. , 2019 ) ) attempt to provide a theoretical understanding of data augmentation ( DA ) ; they focus on establishing that augmenting data saves on the number of labeled samples needed for the same level of accuracy . However , none of these explicitly compare in an apples to apples way the efficacy ( in terms of the number of augmented samples ) of one algorithmic choice of how to use the augmented samples vs another algorithmic choice . Another dimension un-explored in previous work is any characterization of the quality of augmentation . In this paper , we hope to answer the following research question : Is it possible to develop a theoretical framework to compare the sample efficiency of different algorithms that use augmented data ? We present a new theoretical framework that casts consistency regularization as a way to reduce function class complexity , which immediately connects to the well-established theory on generalization and gives rise to a generalization bound for consistency regularization under a general bounded loss function . When specialized to linear regression , this new theoretical framework shows that the consistency regularization is strictly more sample efficient than empirical risk minimization ( ERM ) on the augmented dataset . In addition , using this framework , we also provide generalization bounds under consistency regularization for logistic regression , two-layer neural networks , and expansion-based data augmentations . As summary , our main contributions are : • A statistical framework of consistency regularization . We first present a simple new statistical framework to analyze data augmentation - with a formal theoretical definition of data augmentation and its strength . We then use this framework to give a generalization bound of consistency regularization and provide instantiations for linear regression , logistic regression , two-layer neural networks , and expansion-based data augmentations . • Theoretically proving the efficacy of consistency regularization . When specializing our framework with consistency regularization to linear/logistic regression , it yields a strictly smaller generalization error than ERM with the same augmented data . • Empirical comparisons between consistency regularization and ERM . We perform experiments that make a clean and apples-apples comparison ( i.e. , with no extra modeling or data tweaks ) between consistency regularization and ERM using CIFAR-100 and WideResNet . Our empirical results demonstrate the superior efficacy of consistency regularization . 2 RELATED WORK . Empirical findings . Data augmentation ( DA ) is an essential recipe for almost every state-of-theart supervised learning algorithm since the seminal work of ( Krizhevsky et al. , 2012 ) ( see reference therein ( Simard et al. , 1998 ; Cubuk et al. , 2018 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ; Kuchnik & Smith , 2018 ) ) . It started from adding augmented data to the training samples via ( random ) perturbations , distortions , scales , crops , rotations , and horizontal flips . More sophisticated variants were subsequently designed ; a non-exhaustive list includes Mixup ( Zhang et al. , 2017 ) , Cutout ( DeVries & Taylor , 2017 ) , and Cutmix ( Yun et al. , 2019 ) . The choice of data augmentation and their combinations require domain knowledge and experts ’ heuristics , which triggered some automated search algorithm to find the best augmentation strategies ( Lim et al. , 2019 ; Cubuk et al. , 2019 ) . The effects of different DAs have been systematically explored in ( Tensmeyer & Martinez , 2016 ) . Recent practices not only add augmented data to the training set but also enforce the predictor output to be similar by adding consistency regularization ( Bachman et al. , 2014 ; Laine & Aila , 2016 ; Sohn et al. , 2020 ) . One benefit of consistency regularization is the feasibility of exploiting unlabeled data . Therefore input consistency on augmented data also formed a major component to state-of-theart algorithms for semi-supervised learning ( Laine & Aila , 2016 ; Sajjadi et al. , 2016 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) , self-supervised learning ( Chen et al. , 2020b ) , and unsupervised domain adaptation ( French et al. , 2017 ; Berthelot et al. , 2021 ) . Theoretical studies . Many interpret the effect of DA as some form of regularization ( He et al. , 2019 ) . Some work focuses on linear transformations and linear models ( Wu et al. , 2020 ) or kernel classifiers ( Dao et al. , 2019 ) . Convolutional neural networks by design enforce translation equivariance symmetry ( Benton et al. , 2020 ; Li et al. , 2019 ) ; further studies have hard-coded CNN ’ s invariance or equivariance to rotation ( Cohen & Welling , 2016 ; Marcos et al. , 2017 ; Worrall et al. , 2017 ; Zhou et al. , 2017 ) , scaling ( Sosnovik et al. , 2019 ; Worrall & Welling , 2019 ) and other types of transformations . A line of work view data augmentation as invariant learning by averaging over group actions ( Chen et al. , 2020a ; Mei et al. , 2021 ; Lyle et al. , 2019 ) . They consider an ideal setting that is equivalent to ERM with all possible augmented data , bringing a clean mathematical interpretation . We are interested in a more realistic setting with limited augmented data . In this setting , it is crucial to utilize the limited data with proper training methods , the difference of which can not be revealed under the previous studied settings . Some more recent work investigates the feature representation learning procedure with DA for selfsupervised learning tasks ( Wen & Li , 2021 ; HaoChen et al. , 2021 ; von Kügelgen et al. , 2021 ; Garg & Liang , 2020 ) . Cai et al . ( 2021 ) ; Wei et al . ( 2021 ) studied the effect of data augmentation with label propagation . Data augmentation is also deployed to improve robustness ( Rajput et al. , 2019 ) , to facilitate domain adaptation and domain generalization ( Cai et al. , 2021 ; Sagawa et al. , 2019 ) . 3 DATA AUGMENTATION CONSISTENCY AND HOW IT LEARNS EFFICIENTLY . In this section , we first formally define data augmentation and introduce the problem setup . We then define data augmentation consistency ( DAC ) regularization and show how it effectively reduces the function class complexity , which connects to a generalization bound for bounded loss functions via Rademacher complexity . Subsequently , we specialize our general result to linear regression , which firmly shows that the DAC regularization provably learns more efficiently than minimizing the empirical risk on the augmented dataset . The following section will present more applications ( including logistic regression , neural network , etc . ) . 3.1 PROBLEM SETUP AND DATA AUGMENTATION . Consider the standard supervised learning problem setup : x ∈ X are input features , and y ∈ Y is its label ( or response ) . Let P ∗ be the true distribution of ( x , y ) ( i.e. , the label distribution follows y ∼ P ∗ ( y|x ) ) . We can then formally define data augmentation as : Definition 1 ( Data augmentation ) . For any sample x ∈ X , we say x′ ∈ X is its augmentation , if and only if P ∗ ( y|x ) = P ∗ ( y|x′ ) . The definition above specifies what it means for one input sample to be an augmentation of another . While the definition covers any x′ with the same label distribution as x , our results only use the augmented samples that can be achieved via certain transformations ( e.g. , random cropping , rotation ) . However , our definition does not cover augmentations that alter the labels ( e.g. , MixUp ( Zhang et al. , 2017 ) ) . Now we introduce the learning problem on an augmented dataset : Let ( X , y ) ∈ XN × YN be a training set consisting of N i.i.d . samples . Besides the original ( X , y ) , each training sample in it is provided with α augmented samples . The input features of the augmented dataset can be written as : Ã ( X ) = [ x1 ; · · · ; xN ; x1,1 ; · · · ; xN,1 ; · · · ; x1 , α ; · · · ; xN , α ] ∈ X ( 1+α ) N , where xi is in the original training set and xi , j , ∀j ∈ [ α ] are the augmentations of xi . The labels of the augmented samples are kept the same , which can be denoted as M̃y ∈ Y ( 1+α ) N , where M̃ ∈ R ( 1+α ) N×N is a vertical stack of ( 1 + α ) identity mappings . Specifically , when the input xis are d-dimensional real vectors , we have the following notion of augmentation strength daug : Definition 2 ( Strength of augmentations ) . For any δ ∈ ( 0 , 1 ) , let daug ( δ ) , argmax daug PÃ , X [ rank ( Ã ( X ) − M̃X ) < daug ] ≤ δ , daug , daug ( 1/N ) . Intuitively , strength of augmentations daug ( δ ) means that with probability at least 1 − δ , the augmentations perturb at least daug ( δ ) dimensions ; whereas daug can be intuitively understood as the minimum number of dimensions that the augmentations in Ã ( X ) perturbed with high probability . A lager daug corresponds to a stronger data augmentations . For instance , when Ã ( X ) = M̃X almost surely ( e.g. , when the augmentations are identical copies of the original samples , corresponding to the weakest augmentation – no augmentations at all ) , we have daug ( δ ) = daug = 0 for all δ ∈ ( 0 , 1 ) . On the other hand , if the augmentations are randomly generated , then it is more likely to see larger daug ( i.e. , more dimensions being perturbed ) with larger α ( i.e. , more augmentations ) . In the next subsection , we formally introduce “ data augmentation consistency regularization ” and present a generalization bound under bounded loss functions . We subsequently specialize the bound to linear regression and show that consistency regularization is strictly more sample efficient than empirical risk minimization ( ERM ) on the augmented dataset .

Summary: This paper aims to offer a theoretical analysis of the training with data augmentation and associated consistency loss. While it is intuitive that training with data augmentation and consistency loss will help, this paper offers a theoretical justification of the intuitions. The simple framework (to view DAC as a hypothesis space complexity reduction technique) is neat and intuitive.