Tradeoffs in Data Augmentation: An Empirical Study

1 INTRODUCTION . Models that achieve state-of-the-art in image classification often use heavy data augmentation strategies . The best techniques use various transforms applied sequentially and stochastically . Though the effectiveness of this is well-established , the mechanism through which these transformations work is not well-understood . ∗Equal contribution †Currently at Form Energy , Inc , Somerville , MA Since early uses of data augmentation , it has been assumed that augmentation works because it simulates realistic samples from the true data distribution : “ [ augmentation strategies are ] reasonable since the transformed reference data is now extremely close to the original data . In this way , the amount of training data is effectively increased '' ( Bellegarda et al. , 1992 ) . Because of this , augmentations have often been designed with the heuristic of incurring minimal distribution shift from the training data . This rationale does not explain why unrealistic distortions such as cutout ( DeVries & Taylor , 2017 ) , SpecAugment ( Park et al. , 2019 ) , and mixup ( Zhang et al. , 2017 ) significantly improve generalization performance . Furthermore , methods do not always transfer across datasets—Cutout , for example , is useful on CIFAR-10 and not on ImageNet ( Lopes et al. , 2019 ) . Additionally , many augmentation policies heavily modify images by stochastically applying multiple transforms to a single image . Based on this observation , some have proposed that augmentation strategies are effective because they increase the diversity of images seen by the model . In this complex landscape , claims about diversity and distributional similarity remain unverified heuristics . Without more precise data augmentation science , finding state-of-the-art strategies requires brute force that can cost thousands of GPU hours ( Cubuk et al. , 2018 ; Zhang et al. , 2019 ) . This highlights a need to specify and measure the relationship between the original training data and the augmented dataset , as relevant to a given model ’ s performance . In this paper , we quantify these heuristics . Seeking to understand the mechanisms of augmentation , we focus on single transforms as a foundation . We present an empirical study of 204 different augmentations on CIFAR-10 and 225 on ImageNet , varying both broad transform families and finer transform parameters . To better understand current state of the art augmentation policies , we additionally measure 58 composite augmentations on ImageNet and three state of the art augmentations on CIFAR-10 . Our contributions are : 1 . We introduce Affinity and Diversity : interpretable , easy-to-compute metrics for parametrizing augmentation performance . Affinity quantifies how much an augmentation shifts the training data distribution . Diversity quantifies the complexity of the augmented data with respect to the model and learning procedure . 2 . We find that performance is dependent on both metrics . In the Affinity-Diversity plane , the best augmentation strategies jointly optimize the two ( see Fig 1 ) . 3 . We connect augmentation to other familiar forms of regularization , such as ` 2 and learning rate scheduling , observing common features of the dynamics : performance can be improved and training accelerated by turning off regularization at an appropriate time . 4 . We find that performance is only improved when a transform increases the total number of unique training examples . The utility of these new training examples is informed by the augmentation ’ s Affinity and Diversity . 2 RELATED WORK . Since early uses of data augmentation in training neural networks , there has been an assumption that effective transforms for data augmentation are those that produce images from an “ overlapping but different '' distribution ( Bengio et al. , 2011 ; Bellegarda et al. , 1992 ) . Indeed , elastic distortions as well as distortions in the scale , position , and orientation of training images have been used on MNIST ( Ciregan et al. , 2012 ; Sato et al. , 2015 ; Simard et al. , 2003 ; Wan et al. , 2013 ) , while horizontal flips , random crops , and random distortions to color channels have been used on CIFAR-10 and ImageNet ( Krizhevsky et al. , 2012 ; Zagoruyko & Komodakis , 2016 ; Zoph et al. , 2017 ) . For object detection and image segmentation , one can also use object-centric cropping ( Liu et al. , 2016 ) or cut-and-paste new objects ( Dwibedi et al. , 2017 ; Fang et al. , 2019 ; Ngiam et al. , 2019 ) . In contrast , researchers have also successfully used less domain-specific transformations , such as Gaussian noise ( Ford et al. , 2019 ; Lopes et al. , 2019 ) , input dropout ( Srivastava et al. , 2014 ) , erasing random patches of the training samples ( DeVries & Taylor , 2017 ; Park et al. , 2019 ; Zhong et al. , 2017 ) , and adversarial noise ( Szegedy et al. , 2013 ) . Mixup ( Zhang et al. , 2017 ) and Sample Pairing ( Inoue , 2018 ) are two augmentation methods that use convex combinations of training samples . It is also possible to improve generalization by combining individual transformations . For example , reinforcement learning has been used to choose more optimal combinations of data augmentation transformations ( Ratner et al. , 2017 ; Cubuk et al. , 2018 ) . Follow-up research has lowered the computation cost of such optimization , by using population based training ( Ho et al. , 2019 ) , density matching ( Lim et al. , 2019 ) , adversarial policy-design that evolves throughout training ( Zhang et al. , 2019 ) , or a reduced search space ( Cubuk et al. , 2019 ) . Despite producing unrealistic outputs , such combinations of augmentations can be highly effective in different tasks ( Berthelot et al. , 2019 ; Tan & Le , 2019 ; Tan et al. , 2019 ; Xie et al. , 2019a ; b ) . Across these different examples , the role of distribution shift in training remains unclear . Lim et al . ( 2019 ) ; Hataya et al . ( 2020 ) have found augmentation policies by minimizing the distance between the distributions of augmented data and clean data . Recent work found that after training with augmented data , fine-tuning on clean training data can be beneficial ( He et al. , 2019 ) , while Touvron et al . ( 2019 ) found it beneficial to fine-tune with a test-set resolution that aligns with the training-set resolution . The true input-space distribution from which a training dataset is drawn remains elusive . To better understand the effect of distribution shift on performance , many works attempt to estimate it . Often these techniques require training secondary models , such as those based on variational methods ( Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ; Nowozin et al. , 2016 ; Blei et al. , 2017 ) . Others have augmented the training set by modelling the data distribution directly ( Tran et al. , 2017 ) . Recent work has suggested that even unrealistic distribution modelling can be beneficial ( Dai et al. , 2017 ) . These methods try to specify the distribution separately from the model they are trying to optimize . As a result , they are insensitive to any interaction between the model and data distribution . Instead , we are interested in a measure of how much the data shifts along directions that are most relevant to the model ’ s performance . 3 METHODS . We performed extensive experiments with various augmentations on CIFAR-10 and ImageNet . Experiments on CIFAR-10 used the WRN-28-2 model ( Zagoruyko & Komodakis , 2016 ) , trained for 78k steps with cosine learning rate decay . Results are the mean over 10 initializations and reported errors ( often too small to show on figures ) are the standard error on the mean . Details on the error analysis are in Sec . C. Experiments on ImageNet used the ResNet-50 model ( He et al. , 2016 ) , trained for 112.6k steps with a weight decay rate of 1e-4 , and a learning rate of 0.2 , which is decayed by 10 at epochs 30 , 60 , and 80 . Images were pre-processed by dividing each pixel value by 255 and normalizing by the dataset statistics . Random crop was also applied on all ImageNet models . These pre-processed data without further augmentation are “ clean data ” and a model trained on it is the “ clean baseline ” . We followed the same implementation details as Cubuk et al . ( 2018 ) 1 , including for most augmentation operations . Further implementation details are in Sec . A . Unless specified otherwise , data augmentation was applied following standard practice : each time an image is drawn , the given augmentation is applied with a given probability . We call this mode dynamic augmentation . Due to whatever stochasticity is in the transform itself ( such as randomly selecting the location for a crop ) or in the policy ( such as applying a flip only with 50 % probability ) , the augmented image could be different each time . Thus , most of the tested augmentations increase the number of possible distinct images that can be shown during training . We also performed select experiments by training with static augmentation . In static augmentation , the augmentation policy ( one or more transforms ) is applied once to the entire clean training set . Static augmentation does not change the number of unique images in the dataset . 3.1 AFFINITY : A SIMPLE METRIC FOR DISTRIBUTION SHIFT . Thus far , heuristics of distribution shift have motivated design of augmentation policies . Inspired by this focus , we introduce a simple metric to quantify how augmentation shifts data with respect to the decision boundary of the clean baseline model . 1Available at bit.ly/2v2FojN We start by noting that a trained model is often sensitive to the distribution of the training data . That is , model performance varies greatly between new samples from the true data distribution and samples from a shifted distribution . Importantly , the model ’ s sensitivity to distribution shift is not purely a function of the input data distribution , since training dynamics and the model ’ s implicit biases affect performance . Because the goal of augmentation is improving model performance , measuring shifts with respect to the distribution captured by the model is more meaningful than measuring shifts in the distribution of the input data alone . We thus define Affinity to be the ratio between the validation accuracy of a model trained on clean data and tested on an augmented validation set , and the accuracy of the same model tested on clean data . Here , the augmentation is applied to the validation dataset in one pass , as a static augmentation . More formally we define : Definition 1 . Let Dtrain and Dval be training and validation datasets drawn IID from the same clean data distribution , and let D′val be derived from Dval by applying a stochastic augmentation strategy , a , once to each image in Dval , D′val = { ( a ( x ) , y ) : ∀ ( x , y ) ∈ Dval } . Further let m be a model trained on Dtrain and A ( m , D ) denote the model ’ s accuracy when evaluated on dataset D. The Affinity , T [ a ; m ; Dval ] , is given by T [ a ; m ; Dval ] = A ( m , D′val ) /A ( m , Dval ) . ( 1 ) With this definition , Affinity of one represents no shift and a smaller number suggests that the augmented data is out-of-distribution for the model . In Fig . 2 we illustrate Affinity with a two-class classification task on a mixture of two Gaussians . Augmentation in this example comprises shift of the means of the Gaussians of the validation data compared to those used for training . Under this shift , we calculate both Affinity and KL divergence of the shifted data with respect to the original data . Affinity changes only when the shift in the data is with respect to the model ’ s decision boundary , whereas the KL divergence changes even when data is shifted in the direction that is irrelevant to the classification task . In this way , Affinity captures what is relevant to a model : shifts that impact predictions . This same metric has been used as a measure of a model ’ s robustness to image corruptions that do not change images ’ semantic content ( Azulay & Weiss , 2018 ; Dodge & Karam , 2017 ; Ford et al. , 2019 ; Hendrycks & Dietterich , 2019 ; Rosenfeld et al. , 2018 ; Yin et al. , 2019 ) . Here we , turn this around and use it to quantify the shift of augmented data compared to clean data . Affinity has the following advantages as a metric : 1 . It is easy to measure . It requires only clean training of the model in question . 2 . It is independent of any confounding interaction between the data augmentation and the training process , since augmentation is only used on the validation set and applied statically . 3 . It is a measure of distance sensitive to properties of both the data distribution and the model . We gain confidence in this metric by comparing it to other potential model-dependent measures of distribution shift . We consider the mean log likelihood of augmented test images ( Grathwohl et al. , 2019 ) , and the Watanabe–Akaike information criterion ( WAIC ) ( Watanabe , 2010 ) . These other metrics have high correlation with Affinity . Details can be found in Sec . F .

Summary: This paper studies the problem of data augmentation that obtains new training examples by modifying existing ones. Data augmentation is popular in machine learning and artificial intelligence since it enhances the number of training examples. However, its effect on model performance remains unknown in practice. An augmentation operator (e.g. image rotation) can be either helpful or harmful. This paper introduces two novel metrics, named affinity and diversity, to quantify the effect of any given augmentation operator. The authors find that an operator with high affinity score and high diversity score leads to the best performance improvement.