Theoretical Analysis of Consistency Regularization with Limited Augmented Data

1 INTRODUCTION . Modern machine learning models , especially deep learning models , require abundant training samples . Since data collection and human annotation are expensive , data augmentation has been a ubiquitous practice in creating artificial labeled samples and improving generalization performance . This practice is corroborated by the fact that the semantics of images remain the same through simple translations like obscuring , flipping , rotation , color jitter , rescaling ( Shorten & Khoshgoftaar , 2019 ) . Conventional algorithms use data augmentation to expand the training data set ( Krizhevsky et al. , 2012 ; Simard et al. , 1998 ; Cubuk et al. , 2018 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ) . As an alternative , consistency regularization enforces the model to output similar predictions on the original and augmented samples and contributes to many recent state-of-the-art supervised or semisupervised algorithms . This idea was first proposed in ( Bachman et al. , 2014 ) and popularized by Laine & Aila ( 2016 ) ; Sajjadi et al . ( 2016 ) , and gained more attention recently with the success of FixMatch ( Sohn et al. , 2020 ) for semi-supervised few-shot learning , and AdaMatch ( Berthelot et al. , 2021 ) for domain adaptation . Several recent papers ( see e.g . ( Chen et al. , 2020a ; Mei et al. , 2021 ; Lyle et al. , 2019 ) ) attempt to provide a theoretical understanding of data augmentation ( DA ) ; they focus on establishing that augmenting data saves on the number of labeled samples needed for the same level of accuracy . However , none of these explicitly compare in an apples to apples way the efficacy ( in terms of the number of augmented samples ) of one algorithmic choice of how to use the augmented samples vs another algorithmic choice . Another dimension un-explored in previous work is any characterization of the quality of augmentation . In this paper , we hope to answer the following research question : Is it possible to develop a theoretical framework to compare the sample efficiency of different algorithms that use augmented data ? We present a new theoretical framework that casts consistency regularization as a way to reduce function class complexity , which immediately connects to the well-established theory on generalization and gives rise to a generalization bound for consistency regularization under a general bounded loss function . When specialized to linear regression , this new theoretical framework shows that the consistency regularization is strictly more sample efficient than empirical risk minimization ( ERM ) on the augmented dataset . In addition , using this framework , we also provide generalization bounds under consistency regularization for logistic regression , two-layer neural networks , and expansion-based data augmentations . As summary , our main contributions are : • A statistical framework of consistency regularization . We first present a simple new statistical framework to analyze data augmentation - with a formal theoretical definition of data augmentation and its strength . We then use this framework to give a generalization bound of consistency regularization and provide instantiations for linear regression , logistic regression , two-layer neural networks , and expansion-based data augmentations . • Theoretically proving the efficacy of consistency regularization . When specializing our framework with consistency regularization to linear/logistic regression , it yields a strictly smaller generalization error than ERM with the same augmented data . • Empirical comparisons between consistency regularization and ERM . We perform experiments that make a clean and apples-apples comparison ( i.e. , with no extra modeling or data tweaks ) between consistency regularization and ERM using CIFAR-100 and WideResNet . Our empirical results demonstrate the superior efficacy of consistency regularization . 2 RELATED WORK . Empirical findings . Data augmentation ( DA ) is an essential recipe for almost every state-of-theart supervised learning algorithm since the seminal work of ( Krizhevsky et al. , 2012 ) ( see reference therein ( Simard et al. , 1998 ; Cubuk et al. , 2018 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ; Kuchnik & Smith , 2018 ) ) . It started from adding augmented data to the training samples via ( random ) perturbations , distortions , scales , crops , rotations , and horizontal flips . More sophisticated variants were subsequently designed ; a non-exhaustive list includes Mixup ( Zhang et al. , 2017 ) , Cutout ( DeVries & Taylor , 2017 ) , and Cutmix ( Yun et al. , 2019 ) . The choice of data augmentation and their combinations require domain knowledge and experts ’ heuristics , which triggered some automated search algorithm to find the best augmentation strategies ( Lim et al. , 2019 ; Cubuk et al. , 2019 ) . The effects of different DAs have been systematically explored in ( Tensmeyer & Martinez , 2016 ) . Recent practices not only add augmented data to the training set but also enforce the predictor output to be similar by adding consistency regularization ( Bachman et al. , 2014 ; Laine & Aila , 2016 ; Sohn et al. , 2020 ) . One benefit of consistency regularization is the feasibility of exploiting unlabeled data . Therefore input consistency on augmented data also formed a major component to state-of-theart algorithms for semi-supervised learning ( Laine & Aila , 2016 ; Sajjadi et al. , 2016 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) , self-supervised learning ( Chen et al. , 2020b ) , and unsupervised domain adaptation ( French et al. , 2017 ; Berthelot et al. , 2021 ) . Theoretical studies . Many interpret the effect of DA as some form of regularization ( He et al. , 2019 ) . Some work focuses on linear transformations and linear models ( Wu et al. , 2020 ) or kernel classifiers ( Dao et al. , 2019 ) . Convolutional neural networks by design enforce translation equivariance symmetry ( Benton et al. , 2020 ; Li et al. , 2019 ) ; further studies have hard-coded CNN ’ s invariance or equivariance to rotation ( Cohen & Welling , 2016 ; Marcos et al. , 2017 ; Worrall et al. , 2017 ; Zhou et al. , 2017 ) , scaling ( Sosnovik et al. , 2019 ; Worrall & Welling , 2019 ) and other types of transformations . A line of work view data augmentation as invariant learning by averaging over group actions ( Chen et al. , 2020a ; Mei et al. , 2021 ; Lyle et al. , 2019 ) . They consider an ideal setting that is equivalent to ERM with all possible augmented data , bringing a clean mathematical interpretation . We are interested in a more realistic setting with limited augmented data . In this setting , it is crucial to utilize the limited data with proper training methods , the difference of which can not be revealed under the previous studied settings . Some more recent work investigates the feature representation learning procedure with DA for selfsupervised learning tasks ( Wen & Li , 2021 ; HaoChen et al. , 2021 ; von Kügelgen et al. , 2021 ; Garg & Liang , 2020 ) . Cai et al . ( 2021 ) ; Wei et al . ( 2021 ) studied the effect of data augmentation with label propagation . Data augmentation is also deployed to improve robustness ( Rajput et al. , 2019 ) , to facilitate domain adaptation and domain generalization ( Cai et al. , 2021 ; Sagawa et al. , 2019 ) . 3 DATA AUGMENTATION CONSISTENCY AND HOW IT LEARNS EFFICIENTLY . In this section , we first formally define data augmentation and introduce the problem setup . We then define data augmentation consistency ( DAC ) regularization and show how it effectively reduces the function class complexity , which connects to a generalization bound for bounded loss functions via Rademacher complexity . Subsequently , we specialize our general result to linear regression , which firmly shows that the DAC regularization provably learns more efficiently than minimizing the empirical risk on the augmented dataset . The following section will present more applications ( including logistic regression , neural network , etc . ) . 3.1 PROBLEM SETUP AND DATA AUGMENTATION . Consider the standard supervised learning problem setup : x ∈ X are input features , and y ∈ Y is its label ( or response ) . Let P ∗ be the true distribution of ( x , y ) ( i.e. , the label distribution follows y ∼ P ∗ ( y|x ) ) . We can then formally define data augmentation as : Definition 1 ( Data augmentation ) . For any sample x ∈ X , we say x′ ∈ X is its augmentation , if and only if P ∗ ( y|x ) = P ∗ ( y|x′ ) . The definition above specifies what it means for one input sample to be an augmentation of another . While the definition covers any x′ with the same label distribution as x , our results only use the augmented samples that can be achieved via certain transformations ( e.g. , random cropping , rotation ) . However , our definition does not cover augmentations that alter the labels ( e.g. , MixUp ( Zhang et al. , 2017 ) ) . Now we introduce the learning problem on an augmented dataset : Let ( X , y ) ∈ XN × YN be a training set consisting of N i.i.d . samples . Besides the original ( X , y ) , each training sample in it is provided with α augmented samples . The input features of the augmented dataset can be written as : Ã ( X ) = [ x1 ; · · · ; xN ; x1,1 ; · · · ; xN,1 ; · · · ; x1 , α ; · · · ; xN , α ] ∈ X ( 1+α ) N , where xi is in the original training set and xi , j , ∀j ∈ [ α ] are the augmentations of xi . The labels of the augmented samples are kept the same , which can be denoted as M̃y ∈ Y ( 1+α ) N , where M̃ ∈ R ( 1+α ) N×N is a vertical stack of ( 1 + α ) identity mappings . Specifically , when the input xis are d-dimensional real vectors , we have the following notion of augmentation strength daug : Definition 2 ( Strength of augmentations ) . For any δ ∈ ( 0 , 1 ) , let daug ( δ ) , argmax daug PÃ , X [ rank ( Ã ( X ) − M̃X ) < daug ] ≤ δ , daug , daug ( 1/N ) . Intuitively , strength of augmentations daug ( δ ) means that with probability at least 1 − δ , the augmentations perturb at least daug ( δ ) dimensions ; whereas daug can be intuitively understood as the minimum number of dimensions that the augmentations in Ã ( X ) perturbed with high probability . A lager daug corresponds to a stronger data augmentations . For instance , when Ã ( X ) = M̃X almost surely ( e.g. , when the augmentations are identical copies of the original samples , corresponding to the weakest augmentation – no augmentations at all ) , we have daug ( δ ) = daug = 0 for all δ ∈ ( 0 , 1 ) . On the other hand , if the augmentations are randomly generated , then it is more likely to see larger daug ( i.e. , more dimensions being perturbed ) with larger α ( i.e. , more augmentations ) . In the next subsection , we formally introduce “ data augmentation consistency regularization ” and present a generalization bound under bounded loss functions . We subsequently specialize the bound to linear regression and show that consistency regularization is strictly more sample efficient than empirical risk minimization ( ERM ) on the augmented dataset .

Summary: Data augmentation is a common technique to improve generalization, especially when data is scarce. This paper introduces a theoretical framework for analyzing the effectiveness of consistency regularization when data augmentation is employed. In the limit, consistency regularization is akin to solving a constrained optimization problem with consistency constraints. The paper theoretically studies this limit for linear regression, logistic regression, and a two-layer perceptron with ReLU activation, and tries to characterize the benefits of consistency regularization beyond that of vanilla data augmentation. The paper then continues to experiments where it is shown that consistency regularization outperforms data augmentation on three benchmarks, and the benefits are significant especially when labeled data is scarce.