Deep AutoAugment

While recent automated data augmentation works lead to state-of-the-art results , their design spaces and the derived data augmentation strategies still incorporate strong human priors . In this work , instead of selecting a set of hand-picked default augmentations alongside the searched data augmentations , we propose a fully automated approach for data augmentation search called Deep AutoAugment ( DeepAA ) . DeepAA progressively builds a multi-layer data augmentation pipeline from scratch by adding augmentation layers one at a time until reaching convergence . For each augmentation layer , the policy is optimized to maximize the cosine similarity between the gradients of the original and augmented data along the direction with low variance . Our experiment show that even without default augmentations , we can learn an augmentation policy that achieves comparable performance with that of previous works . We show that the regularized gradient matching , i.e. , maximizing the gradient cosine similarity along the direction with low variance is effective in searching for data augmentation policy from scratch . 1 INTRODUCTION Data augmentation ( DA ) is a powerful technique for machine learning since it effectively regularizes the model by increasing the number and the diversity of data points ( Goodfellow et al. , 2016 ; Zhang et al. , 2017 ) . A large body of data augmentation transformations has been proposed ( Inoue , 2018 ; Zhang et al. , 2018 ; DeVries & Taylor , 2017 ; Yun et al. , 2019 ; Hendrycks et al. , 2020 ; Yan et al. , 2020 ) to improve the model performance . While applying a set of well designed augmentation transformations could help yield considerable performance enhancement especially in image recognition tasks , manually selecting high-quality augmentation transformations and determining how they should be combined still require strong domain expertise and prior knowledge of the dataset of interest . With the recent trend of automated machine learning , data augmentation search flourished in the image domain ( Cubuk et al. , 2019 ; 2020 ; Ho et al. , 2019 ; Lim et al. , 2019 ; Hataya et al. , 2020 ; Li et al. , 2020 ; Liu et al. , 2021 ) , which yields significant performance improvement over hand-crafted data augmentation methods . Although data augmentation policies in previous works ( Cubuk et al. , 2019 ; 2020 ; Ho et al. , 2019 ; Lim et al. , 2019 ; Hataya et al. , 2020 ; Li et al. , 2020 ) contain multiple transformations applied sequentially , only one or two transformations of each sub-policy are found through searching , while the rest transformation are hand-picked and applied by default in addition to the found policy ( Figure 1 ( A ) ) . From this perspective , we believe that the previous automated methods are not entirely automated , as they are still built upon hand-crafted default augmentations . In this work , we propose Deep AutoAugment ( DeepAA ) , a multi-layer data augmentation search method which aims to remove the need of hand-crafted default transformations ( Figure 1 ( B ) ) . DeepAA fully automates the data augmentation process by searching a deep data augmentation policy on an expanded set of transformations that includes the widely adopted search space as well as the default transformations ( e.g . flips , Cutout and crop ) . DeepAA formulates the search of the data augmentation policy as a regularized gradient matching problem by maximizing the cosine similarity of gradients between augmented data and the original data with regularization . As the augmentation layers go deep , the search space of augmentation policies can be prohibitively large . To efficiently search for a multi-layer augmentation policy , DeepAA incrementally adds augmentations based on the data distribution transformed by all the previous augmentation layers . We evaluate the performance of DeepAA on three datasets CIFAR-10 , CIFAR-100 , and ImageNet , and compare the results with automated data augmentation search methods including AutoAugment ( AA ) ( Cubuk et al. , 2019 ) , PBA ( Ho et al. , 2019 ) , Fast AutoAugment ( FastAA ) ( Lim et al. , 2019 ) , Faster AutoAugment ( Faster AA ) ( Hataya et al. , 2020 ) , DADA ( Li et al. , 2020 ) and RandAugment ( RA ) ( Cubuk et al. , 2020 ) , UniformAugment ( UA ) ( LingChen et al. , 2020 ) , TrivialAugment ( TA ) ( Müller & Hutter , 2021 ) , Adversarial AutoAugment ( AdvAA ) ( Zhang et al. , 2019 ) and Augmentationwise Weight Sharing ( AWS ) ( Tian et al. , 2020 ) . Our results show that , without using any default augmentations , DeepAA achieves comparable performance compared to existing automatic augmentation search methods on CIFAR-10 , CIFAR-100 on Wide-ResNet-28-10 and ImageNet on ResNet-50 and ResNet-200 with standard augmentation space and training procedure . We summarize our main contributions below : • We propose Deep AutoAugment ( DeepAA ) , a fully automated data augmentation search method that finds a multi-layer data augmentation policy from scratch . • We formulate such multi-layer data augmentation search as a regularized gradient matching problem . We show that maximizing cosine similarity along the direction of low variance is effective for data augmentation search when the augmentation layers go deep . • We address the issue of exponential growth of the dimensionality of the search space when more augmentation layers are added by incrementally adding augmentation layers based on the data distribution transformed by all the previous augmentation layers . • Our experiment results show that , without using any default augmentations , DeepAA achieves comparable performance with prior works . 2 RELATED WORK . Automated Data Augmentation . Automating data augmentation policy design has recently emerged as a promising paradigm for data augmentation . The pioneer work on automated data augmentation was proposed in AutoAugment ( Cubuk et al. , 2019 ) , where the search is performed under reinforcement learning framework . AutoAugment requires to train the neural network repeatedly , which takes thousands of GPU hours to converge . Subsequent works ( Lim et al. , 2019 ; Li et al. , 2020 ; Liu et al. , 2021 ) aim at reducing the computation cost . Fast AutoAugment ( Lim et al. , 2019 ) treats data augmentation as inference time density matching which can be implemented efficiently with Bayesian optimization . Differentiable Automatic Data Augmentation ( DADA ) ( Li et al. , 2020 ) further reduces the computation cost through a reparameterized Gumbel-softmax distribution ( Jang et al. , 2017 ) . RandAugment ( Cubuk et al. , 2020 ) introduces a simplified search space containing two interpretable hyperparameters , which can be optimized simply by grid search . Adversarial AutoAugment ( AdvAA ) ( Zhang et al. , 2019 ) searches for the augmentation policy in an adversarial and online manner . It also incorporates the concept of Batch Augmentaiton ( BA ) ( Berman et al. , 2019 ; Hoffer et al. , 2020 ) , where multiple adversarial policies run in parallel . Augmentation-wise weight sharing ( AWS ) ( Tian et al. , 2020 ) proposed a weight sharing strategy to efficiently search for data augmentation operations in image classification which outperforms previous arts by a large margin . Although many automated data augmentation methods have been proposed , the use of default augmentations still imposes strong domain knowledge . DeepAA is proposed to address this constraint that is shared across existing automated data augmentation methods . Gradient Matching . Our work is also related to gradient matching . In ( Du et al. , 2018 ) , the authors showed that the cosine similarity between the gradient of different tasks provides a signal to detect when an auxiliary loss is helpful to the main loss . In ( Wang et al. , 2020 ) , the authors proposed to use cosine similarity as the training signal to optimize the data usage via weighting data points . A similar approach was proposed in ( Müller et al. , 2021 ) , which uses the gradient inner product as a per-example reward for optimizing data distribution and data augmentation under the reinforcement learning framework . Our approach also utilizes the cosine similarity to guide the data augmentation search . However , our implementation of cosine similarity is different from the above from two aspects : we propose a Jacobian-vector product form to backpropagate through the cosine similarity , which is computational and memory efficient and does not require computing higher order derivative ; we also propose a sampling scheme that effectively allows the cosine similarity to grow with added augmentation stages . 3 DEEP AUTOAUGMENT . 3.1 MOTIVATION . Data augmentation can be viewed as a process of filling missing data points in the dataset with the same data distribution ( Hataya et al. , 2020 ) . By augmenting a single data point for multiple times , we expect the resulting data distribution to be close to the full dataset under a certain type of transformation . For example , by augmenting a single image with proper color jittering , we obtain a batch of augmented images which has similar distributions of lighting conditions as the full dataset . As the distribution of augmented data gets closer to the full dataset , the gradient of the augmented data should be steered towards a batch of original data sampled from the dataset . In DeepAA , we formulate the search of the data augmentation policy as a regularized gradient matching problem , which manages to steer the gradient to a batch of original data by augmenting a single image for multiple times . We construct the augmented training batch by augmenting a single training data point for multiple times following the augmentation policy . We construct a validation batch by sampling a batch of original data from the validation set . We expect that by augmentation , the gradient of augmented training batch can be steered towards the gradient of the validation batch . The intuition is that an effective data augmentation fills the missing data points in the training set , thus steering the training gradient towards the validation gradient . Another challenge for augmentation policy search is that the search space can be prohibitively large with deep augmentation layers ( K ≥ 5 ) . This was not a problem in previous works , where the augmentation policies is shallow ( K ≤ 2 ) . For example , in AutoAugment Cubuk et al . ( 2019 ) , each sub-policy contains K = 2 transformations to be applied sequentially , and the search space of AutoAugment contains 16 image operations and 10 discrete magnitude levels . The resulting number of combinations of transformations in AutoAugment is roughly ( 16 × 10 ) 2 = 25 , 600 , which is handled well in previous research . However , by discarding the default augmentation pipeline and searching for data augmentations from scratch , it requires deeper augmentation layers in order to perform well . For a data augmentation with K = 5 sequentially applied transformations , the number of sub-policies is ( 16× 10 ) 5 ≈ 1011 , which is prohibitively large for the following two reasons . First , it becomes less likely to encounter a good policy by exploration , as the good policy becomes more sparse on high dimensional search space . Second , the dimension of parameters in the policy also grows with K , making it more computational challenging to optimize . Instead of searching for subpolicies , we propose to build up the full data augmentation by progressively stacking augmentation layers , where each augmentation layer is optimized on top of the data distribution transformed by all previous layers . This avoids sampling sub-policies from a large search space , and the parameters of the policy is reduced to T for each augmentation layer .

Summary: This paper provides an interesting finding that instead of injecting experts prior to hand-pick augmentations at the top layers, the entire search can be formulated end-to-end in a multi-layer (deep) way without hand-picked transformation optimized with gradient matching. The independent sampling at each layer with estimated mean gradient of training data is used to stabilize the training and make the optimization process more efficient. Empirical results on CIFAR-10/100 and ImageNet outperform previous best performing DAS methods.