Year: 2021

Abstract:
While recent automated data augmentation works lead to state-of-the-art results, their design spaces and the derived data augmentation strategies still incorporate strong human priors. In this work, instead of selecting a set of hand-picked default augmentations alongside the searched data augmentations, we propose a fully automated approach for data augmentation search called Deep AutoAugment (DeepAA). DeepAA progressively builds a multi-layer data augmentation pipeline from scratch by adding augmentation layers one at a time until reaching convergence. For each augmentation layer, the policy is optimized to maximize the cosine similarity between the gradients of the original and augmented data along the direction with low variance. Our experiment show that even without default augmentations, we can learn an augmentation policy that achieves comparable performance with that of previous works. We show that the regularized gradient matching, i.e., maximizing the gradient cosine similarity along the direction with low variance is effective in searching for data augmentation policy from scratch.

text: While recent automated data augmentation works lead to state-of-the-art results, their design spaces and the derived data augmentation strategies still incorporate strong human priors. In this work, instead of selecting a set of hand-picked default augmentations alongside the searched data augmentations, we propose a fully automated approach for data augmentation search called Deep AutoAugment (DeepAA). DeepAA progressively builds a multi-layer data augmentation pipeline from scratch by adding augmentation layers one at a time until reaching convergence. For each augmentation layer, the policy is optimized to maximize the cosine similarity between the gradients of the original and augmented data along the direction with low variance. Our experiment show that even without default augmentations, we can learn an augmentation policy that achieves comparable performance with that of previous works. We show that the regularized gradient matching, i.e., maximizing the gradient cosine similarity along the direction with low variance is effective in searching for data augmentation policy from scratch.
1 INTRODUCTION
Data augmentation (DA) is a powerful technique for machine learning since it effectively regularizes the model by increasing the number and the diversity of data points (Goodfellow et al., 2016; Zhang et al., 2017). A large body of data augmentation transformations has been proposed (Inoue, 2018; Zhang et al., 2018; DeVries & Taylor, 2017; Yun et al., 2019; Hendrycks et al., 2020; Yan et al., 2020) to improve the model performance. While applying a set of well designed augmentation transformations could help yield considerable performance enhancement especially in image recognition tasks, manually selecting high-quality augmentation transformations and determining how they should be combined still require strong domain expertise and prior knowledge of the dataset of interest. With the recent trend of automated
machine learning, data augmentation search flourished in the image domain (Cubuk et al., 2019; 2020; Ho et al., 2019; Lim et al., 2019; Hataya et al., 2020; Li et al., 2020; Liu et al., 2021), which yields significant performance improvement over hand-crafted data augmentation methods.
Although data augmentation policies in previous works (Cubuk et al., 2019; 2020; Ho et al., 2019; Lim et al., 2019; Hataya et al., 2020; Li et al., 2020) contain multiple transformations applied sequentially, only one or two transformations of each sub-policy are found through searching, while the rest transformation are hand-picked and applied by default in addition to the found policy (Figure 1(A)). From this perspective, we believe that the previous automated methods are not entirely automated, as they are still built upon hand-crafted default augmentations.
In this work, we propose Deep AutoAugment (DeepAA), a multi-layer data augmentation search method which aims to remove the need of hand-crafted default transformations (Figure 1(B)). DeepAA fully automates the data augmentation process by searching a deep data augmentation policy on an expanded set of transformations that includes the widely adopted search space as well as the default transformations (e.g. flips, Cutout and crop). DeepAA formulates the search of the data augmentation policy as a regularized gradient matching problem by maximizing the cosine similarity of gradients between augmented data and the original data with regularization. As the augmentation layers go deep, the search space of augmentation policies can be prohibitively large. To efficiently search for a multi-layer augmentation policy, DeepAA incrementally adds augmentations based on the data distribution transformed by all the previous augmentation layers.
We evaluate the performance of DeepAA on three datasets CIFAR-10, CIFAR-100, and ImageNet, and compare the results with automated data augmentation search methods including AutoAugment (AA) (Cubuk et al., 2019), PBA (Ho et al., 2019), Fast AutoAugment (FastAA) (Lim et al., 2019), Faster AutoAugment (Faster AA) (Hataya et al., 2020), DADA (Li et al., 2020) and RandAugment (RA) (Cubuk et al., 2020), UniformAugment (UA) (LingChen et al., 2020), TrivialAugment (TA) (Müller & Hutter, 2021), Adversarial AutoAugment (AdvAA) (Zhang et al., 2019) and Augmentationwise Weight Sharing (AWS) (Tian et al., 2020). Our results show that, without using any default augmentations, DeepAA achieves comparable performance compared to existing automatic augmentation search methods on CIFAR-10, CIFAR-100 on Wide-ResNet-28-10 and ImageNet on ResNet-50 and ResNet-200 with standard augmentation space and training procedure.
We summarize our main contributions below:
• We propose Deep AutoAugment (DeepAA), a fully automated data augmentation search method that finds a multi-layer data augmentation policy from scratch.
• We formulate such multi-layer data augmentation search as a regularized gradient matching problem. We show that maximizing cosine similarity along the direction of low variance is effective for data augmentation search when the augmentation layers go deep.
• We address the issue of exponential growth of the dimensionality of the search space when more augmentation layers are added by incrementally adding augmentation layers based on the data distribution transformed by all the previous augmentation layers.
• Our experiment results show that, without using any default augmentations, DeepAA achieves comparable performance with prior works.

heading: 2 RELATED WORK
text: Automated Data Augmentation. Automating data augmentation policy design has recently emerged as a promising paradigm for data augmentation. The pioneer work on automated data augmentation was proposed in AutoAugment (Cubuk et al., 2019), where the search is performed under reinforcement learning framework. AutoAugment requires to train the neural network repeatedly, which takes thousands of GPU hours to converge. Subsequent works (Lim et al., 2019; Li et al., 2020; Liu et al., 2021) aim at reducing the computation cost. Fast AutoAugment (Lim et al., 2019) treats data augmentation as inference time density matching which can be implemented efficiently with Bayesian optimization. Differentiable Automatic Data Augmentation (DADA) (Li et al., 2020) further reduces the computation cost through a reparameterized Gumbel-softmax distribution (Jang et al., 2017). RandAugment (Cubuk et al., 2020) introduces a simplified search space containing two interpretable hyperparameters, which can be optimized simply by grid search. Adversarial AutoAugment (AdvAA) (Zhang et al., 2019) searches for the augmentation policy in an adversarial and online manner. It also incorporates the concept of Batch Augmentaiton (BA) (Berman et al., 2019; Hoffer et al., 2020), where multiple adversarial policies run in parallel. Augmentation-wise weight sharing (AWS) (Tian et al., 2020) proposed a weight sharing strategy to efficiently search for data augmentation operations in image classification which outperforms previous arts by a large margin. Although many automated data augmentation methods have been proposed, the use of default augmentations still imposes strong domain knowledge. DeepAA is proposed to address this constraint that is shared across existing automated data augmentation methods.
Gradient Matching. Our work is also related to gradient matching. In (Du et al., 2018), the authors showed that the cosine similarity between the gradient of different tasks provides a signal to detect when an auxiliary loss is helpful to the main loss. In (Wang et al., 2020), the authors proposed to use cosine similarity as the training signal to optimize the data usage via weighting data points. A similar approach was proposed in (Müller et al., 2021), which uses the gradient inner product as a per-example reward for optimizing data distribution and data augmentation under the reinforcement learning framework. Our approach also utilizes the cosine similarity to guide the data augmentation search. However, our implementation of cosine similarity is different from the above from two aspects: we propose a Jacobian-vector product form to backpropagate through the cosine similarity, which is computational and memory efficient and does not require computing higher order derivative; we also propose a sampling scheme that effectively allows the cosine similarity to grow with added augmentation stages.

heading: 3 DEEP AUTOAUGMENT

heading: 3.1 MOTIVATION
text: Data augmentation can be viewed as a process of filling missing data points in the dataset with the same data distribution (Hataya et al., 2020). By augmenting a single data point for multiple times, we expect the resulting data distribution to be close to the full dataset under a certain type of transformation. For example, by augmenting a single image with proper color jittering, we obtain a batch of augmented images which has similar distributions of lighting conditions as the full dataset. As the distribution of augmented data gets closer to the full dataset, the gradient of the augmented data should be steered towards a batch of original data sampled from the dataset. In DeepAA, we formulate the search of the data augmentation policy as a regularized gradient matching problem, which manages to steer the gradient to a batch of original data by augmenting a single image for multiple times. We construct the augmented training batch by augmenting a single training data point for multiple times following the augmentation policy. We construct a validation batch by sampling a batch of original data from the validation set. We expect that by augmentation, the gradient of augmented training batch can be steered towards the gradient of the validation batch. The intuition is that an effective data augmentation fills the missing data points in the training set, thus steering the training gradient towards the validation gradient.
Another challenge for augmentation policy search is that the search space can be prohibitively large with deep augmentation layers (K ≥ 5). This was not a problem in previous works, where the augmentation policies is shallow (K ≤ 2). For example, in AutoAugment Cubuk et al. (2019), each sub-policy contains K = 2 transformations to be applied sequentially, and the search space of AutoAugment contains 16 image operations and 10 discrete magnitude levels. The resulting number of combinations of transformations in AutoAugment is roughly (16 × 10)2 = 25, 600, which is handled well in previous research. However, by discarding the default augmentation pipeline and searching for data augmentations from scratch, it requires deeper augmentation layers in order to perform well. For a data augmentation with K = 5 sequentially applied transformations, the number of sub-policies is (16× 10)5 ≈ 1011, which is prohibitively large for the following two reasons. First, it becomes less likely to encounter a good policy by exploration, as the good policy becomes more sparse on high dimensional search space. Second, the dimension of parameters in the policy also grows with K, making it more computational challenging to optimize. Instead of searching for subpolicies, we propose to build up the full data augmentation by progressively stacking augmentation layers, where each augmentation layer is optimized on top of the data distribution transformed by all previous layers. This avoids sampling sub-policies from a large search space, and the parameters of the policy is reduced to T for each augmentation layer.

heading: 3.2 SEARCH SPACE
text: Let O denote the set of augmentation operations (e.g. identity, rotate, brightness), m denote an operation magnitude in the set M, x denote an image sampled from the space X . We define the set of transformations as the set of operations with a fixed magnitude, that is T := {t|t = o(· ; m), o ∈ O and m ∈M}. By this definition, every t is a map t : X → X , and there are |T| = |M| · |O| possible transformations. In previous works (Cubuk et al., 2019; Lim et al., 2019; Li et al., 2020; Hataya et al., 2020), a data augmentation policy P consists of several sub-policies. As explained above, the
size of candidate sub-policies grows exponentially with depth K. Therefore, in this work, instead of searching for sub-policies, we propose a practical method that builds up the full data augmentation by progressively stacking augmentation layers. The final data augmentation policy hence consists of K layers of sequentially applied policy P = {P1, · · · ,PK}. Policy Pk is optimized conditioned on the data distribution augmented by all previous (k − 1) layers of policies, thus we write the policy as a conditional distribution Pk := pθk(n|{P1, · · · ,Pk−1}) where n denotes the indices of transformations in T. With some abuse of notations, We use a simplified notation as pθk to replace pθk(n|{P1, · · · ,Pk−1}) for clarity.

heading: 3.3 AUGMENTATION POLICY SEARCH VIA REGULARIZED GRADIENT MATCHING
text: Assume that we augment a single data point x for multiple times following distribution pθ. The resulting average gradient of such augmentation is denoted as g(x, θ) which is a function of data x and distribution parameters θ. Let v denote the gradients of a batch of the original data. We optimize the policy by maximizing the cosine similarity between the gradients of the augmented data and a batch of the original data as follows:
θ = argmax θ cosineSimilarity(v, g(x, θ)) (1)
= argmax θ vT · g(x, θ) ∥v∥·∥g(x, θ)∥
where ∥·∥ denotes the L2-norm. The parameters of the distribution can be updated via gradient ascent:
θ ← θ + η∇θ cosineSimilarity(v, g(x, θ)), (2)
where η is the learning rate.

heading: 3.3.1 POLICY SEARCH FOR ONE LAYER
text: We start with the case where the data augmentation policy only contains a single augmentation layer, i.e., P = {pθ}. Let L(x;w) denote the classification loss of data point x where w ∈ RD represents the flattened weights of the neural network. Consider applying augmentation on a single data point x following the distribution pθ, the resulting averaged gradient can be calculated analytically by averaging all possible transformations in T with the corresponding probability p(θ):
g(x; θ) = |T|∑ n=1 pθ(n)∇wL(tn(x);w) (3)
= G(x) · pθ where G(x) = [ ∇wL(t1(x);w), · · · ,∇wL(t|T|(x);w) ] is a D × |T| Jacobian matrix, and pθ = [pθ(1), · · · , pθ(|T|)]T is a |T| dimensional Categorical distribution. The gradient w.r.t. the cosine similarity in Eqn. (2) can be derived as:
∇θ cosineSimilarity(v, g(x; θ)) = ∇θpθ · r (4)
where
r = G(x)T ( v
∥g(θ)∥ − v
T g(θ)
∥g(θ)∥2 · g(θ) ∥g(θ)∥
) (5)
which can be interpreted as a reward for each transformation.

heading: 3.3.2 POLICY SEARCH FOR MULTIPLE LAYERS
text: The above derivation is based on the assumption that g(θ) can be computed analytically by Eq. (3). However, when K ≥ 2, it becomes impractical to compute the average gradient of the augmented data given that the search space dimensionality grows exponentially and we need to average the gradient of all |T|K possible sub-policies.
To efficiently search for a multi-layer augmentation policy, we propose to incrementally add augmentations based on the data distribution transformed by all the previous augmentation layers. Specifically, let P = {P1, · · · ,PK} denote the K-layer policy. The policy Pk modifies the data distribution on top of the data distribution augmented by the previous (k− 1) layers. Therefore, the policy at the kth layer is a distribution Pk = pθk(n) conditioned on the policies {P1, · · · ,Pk−1} where each one is a |T|-dimensional Categorical distribution. Given that, the Jacobian matrix at the kth layer can be derived by averaging over the previous (k − 1) layers of policies as follows:
G(x)k = |T|∑ nk−1=1 · · · |T|∑ n1=1 pθk−1(nk−1) · · · pθ1(n1)[∇wL((t1 ◦ tnk−1 · · · ◦ tn1)(x);w), · · · ,
∇wL((t|T| ◦ tnk−1 ◦ · · · ◦ tn1)(x);w)]
(6)
where Gk can be estimated via the Monte Carlo method as: G̃k(x) = ∑
ñk−1∼pθk
· · · ∑
ñ1∼pθ1
[∇wL((t1 ◦ tñk−1 · · · ◦ tñ1)(x);w), · · · ,
∇wL((t|T| ◦ tñk−1 ◦ · · · ◦ tñ1)(x);w)] (7)
where ñk−1 ∼ pθk−1(n), · · · , ñ1 ∼ pθ1(n).
The average gradient at the kth layer can also be estimated by the Monte Carlo method as: g̃(x; θk) = ∑
ñk∼pθk
· · · ∑
ñ1∼pθ1
∇wL ((tñk ◦ · · · ◦ tñ1)(x);w) . (8)
Therefore, the reward at the kth layer is derived as:
r̃k(x) = ( G̃k(x) )T ( v ∥g̃k(x; θk)∥ − v T g̃k(x; θk) ∥g̃k(x; θk)∥2 · g̃k(x; θk) ∥g̃k(x; θk)∥ ) . (9)
To prevent the augmentation policy from overfitting, we regularize the optimization by avoiding optimizing towards the direction with high variance. Thus, we penalize the average reward with its standard deviations as
rk = Ex{r̃k(x)} − c · √ Ex{(r̃k(x)− Ex{r̃k(x)})2} , (10)
where we use 16 randomly sampled images to calculate the expectation. The c controls the degree of regularization, which is set to 1.0 throughout this paper. With such regularization, we prevent the policy from converging to the region with high variance.
Therefore the parameters of policy Pk (k ≥ 2) can be updated as:
θ ← θk + η∇θk cosineSimilarity(v, g(θk)) (11)
where
∇θ cosineSimilarity(v, gk(x; θ)) = ∇θpθk · rk. (12)

heading: 4 EXPERIMENTS AND ANALYSIS
text: Benchmarks and Baselines. We evaluate the performance of DeepAA on three standard benchmarks: CIFAR-10, CIFAR-100, ImageNet, and compare it against a baseline based on standard augmentations (i.e., flip left-righ, pad-and-crop for CIFAR-10/100, and Inception-stype preprocesing (Szegedy et al., 2015) for ImageNet) as well as ten existing automatic augmentation methods including (1) AutoAugment (AA) (Cubuk et al., 2019), (2) PBA (Ho et al., 2019), (3) Fast AutoAugment (Fast AA) (Lim et al., 2019), (4) Faster AutoAugment (Hataya et al., 2020), (5) DADA (Li et al., 2020), (6) RandAugment (RA) (Cubuk et al., 2020), (7) UniformAugment (UA) (LingChen et al., 2020), (8) TrivialAugment (TA) (Müller & Hutter, 2021), (9) Adversarial AutoAugment (AdvAA) (Zhang et al., 2019) and (10) Augmentation-wise Weight Sharing (AWS) (Tian et al., 2020).
Search Space. We set up the operation set O to include 16 commonly used operations (identity, shearx, shear-y, translate-x, translate-y, rotate, solarize, equalize, color, posterize, contrast, brightness, sharpness, autoContrast, invert, Cutout) as well as two operations (flips and crop) that are used as the default operations in the aforementioned existing automatic augmentation methods. Among the operations in O, 11 operations are associated with magnitudes. We then discretize the range of magnitudes into 12 uniformly spaced levels and treat each operation with a discrete magnitude as an independent transformation. Therefore, the policy in each layer is a 139-dimensional Categorical distribution corresponding to |T| = 139 {operation, magnitude} pairs. The list of operations and the range of magnitudes in the standard augmentation space are summarized in Appendix A.

heading: 4.1 PERFORMANCE ON CIFAR-10 AND CIFAR-100
text: Policy Search. Following (Cubuk et al., 2019), we conduct the augmentation policy search based on Wide-ResNet-40-2 (Zagoruyko & Komodakis, 2016). We train the network on a subset of 4,000 randomly selected samples from CIFAR-10. We then update the policy parameters θk (k = 1, 2, · · · ,K) for 512 iterations for each of the K augmentation layers. We use the Adam optimizer (Kingma & Ba, 2014) and set the learning rate to 0.025 for policy updating.
Policy Evaluation. Using the publicly available repository of Fast AutoAugment (Lim et al., 2019), we evaluate the found augmentation policy on both CIFAR-10 and CIFAR-100 using Wide-ResNet28-10 and Shake-Shake-2x96d models. The evaluation configurations are kept consistent with that of Fast AutoAugment.
Results. Table 1 reports the Top-1 test accuracy on CIFAR-10/100 for Wide-ResNet-28-10 and Shake-Shake-2x96d, respectively. The results of DeepAA are the average of four independent runs with different initializations. We also show the 95% confidence interval of the mean accuracy. As shown, DeepAA achieves similar performance compared against previous works even if it searched the policy from scratch without applying the default augmentation.

heading: 4.2 PERFORMANCE ON IMAGENET
text: Policy Search. We conduct the augmentation policy search based on ResNet-18 (He et al., 2016). We train the network on a subset of 200, 000 randomly selected samples from ImageNet for 30 epochs. We use the same search settings as in CIFAR-10.
Policy Evaluation. We evaluate the performance of the found augmentation policy on ResNet-50 and ResNet-200 based on the public repository of Fast AutoAugment (Lim et al., 2019). The parameters for training are identical to the configuration of (Lim et al., 2019). In particular, we train and evaluate with images of size 224x224.
Results. The performance of our proposed method on ImageNet is presented in Table 2. As shown, DeepAA achieves the second best performance compared with previous works without the use of default augmentation pipeline. In particular, DeepAA performs better on large models, as performance of on ResNet-200 is close to the best within the 95% confidence interval.
1On CIFAR-10/100, TA(Wide) uses a wider (stronger) augmentation space, while the other compared methods uses the standard augmentation space.

heading: 4.3 PERFORMANCE WITH BATCH AUGMENTATION
text: Batch Augmentation (BA) is a technique that draws multiple augmented-instance of the samples in one optimization batch. It has been shown to be able to improve the generalization performance of the network (Berman et al., 2019; Hoffer et al., 2020). We have conducted a grid search on the hyperparameters of both TA and DeepAA (details in Appendix D). As shown in Table 3, after tuning the hyperparameters, the performance of TA (Wide) + BA is better than the reported performance in the original TA paper. DeepAA + BA indeed enhances the performance of DeepAA, and it outperforms TA (Wide) + BA and AdvAA after hyperparameter tuning.
4.4 UNDERSTANDING DEEPAA
Effectiveness of Augmentation Policy Search. One advantage of DeepAA is the proposed augmentation policy search method based on gradient matching. To examine its effectiveness, we factor out the impact coming from multiple augmentation layers, and conduct an experiment to search for a single layer of augmentation policy based on it. When evaluating the searched policy, we apply the default augmentation in addition to the searched policy. We refer to this variant as DeepAA-simple. Figure 2 compares the Top-1 test accuracy on ImageNet using ResNet-50 between DeepAA-simple, DeepAA, and existing automatic
augmentation methods. While there is 0.22% performance drop compared to DeepAA, with a single augmentation layer, DeepAA-simple outperforms other methods and is able to achieve similar performance compared to TA (Wide) but with a smaller (weaker) augmentation space and trains on a smaller image size (224x224 vs 244x224).
Policy Search Cost. Table 4 compares the policy search time on CIFAR-10/100 and ImageNet in GPU hours. DeepAA searches for a 5-layer data augmentation policy that outperforms other automated data augmentation methods within accepted amount of time. Note that the multi-layer search space in DeepAA is much larger than the search space of existing automatic augmentation methods, since we search for the a deeper data augmentation policy from scratch.
Impact of the Number of Augmentation Layers. Another advantage of DeepAA is its multi-layer search space that can go beyond two layers which existing automatic augmentation methods are
1On ImageNet, TA(RA) achieves 77.85% and 77.55% top-1 accuracy with images of size 244x224 and 224x224 respectively.
2On ImageNet, TA(Wide) achieves 78.07% and 77.97% top-1 accuracy with images of size 244x224 and 224x224 respectively. In addition, TA(Wide) uses a wider (stronger) augmentation space.
designed upon. To examine the impact of the number of augmentation layers on the performance of DeepAA, we conduct an experiment with different numbers of augmentation layers. Table 5 and Table 6 show the performance on CIFAR-10/100 and ImageNet respectively with increasing number of augmentation layers. As shown, for CIFAR-10/100, the performance gradually improves when more augmentation layers are added until we reach five layers. The performance does not improve when the sixth layer is added. For ImageNet, we have the similar observation where the performance stops improving when more than five augmentation layers are included.
Identification of the Optimal Number of Augmentation Layers. Given the results obtained from the previous experiment, a natural question to ask is whether it is possible to automatically identify the optimal number of augmentation layers for DeepAA. To shed light on the answer to this question, Figure 3 illustrates the distributions of operations in the searched policy for CIFAR-10/100 and ImageNet respectively. Shown in Figure 3(a), the augmentation of CIFAR-10/100 converges to the identity transformation at the sixth augmentation layer, which is a natural indication of the end of the augmentation pipeline. Shown in Figure 3(b), we have the similar observation for ImageNet where the identity transformation dominates in the sixth augmentation layer. These observations match our Top-1 test accuracies listed in Table 5 and Table 6. Additionally, the distribution of the magnitude of each operation for CIFAR-10/100 and ImageNet can be found in Appendix B and Appendix C.
Validity of Optimizing Gradient Matching with Regularization. To demonstrate the validity of optimizing gradient matching with regularization, we have designed a baseline method named “DeepTA”. In DeepTA, we stack multiple layers of TA on the same augmentation space of DeepAA without default augmentations. In doing so, we can fairly compare the behavior of DeepAA and the baseline over the course of augmentation policy search. As stated in Eqn. (10) and Eqn. (12), we explicitly optimize the gradient similarities with the average reward minus its standard deviation. The first term, the average reward Ex{r̃k(x)}, encourages the direction of high cosine similarity. The second term, the standard deviation of the reward √ Ex{(r̃k(x)− Ex{r̃k(x)})2}, acts as a regularization that penalizes the directions with high variance. These two terms jointly maximize the gradient similarity along the direction with low variance. To illustrate the optimization trajectory, we design two metrics that closely related to the two terms in Eqn. (10): the mean value and the standard deviation of the improvement of gradient similarity. The improvement of gradient similarity is obtained by subtracting the baseline cosine similarity of the original image batch. In our experiment, the mean and standard deviation of the gradient similarity improvement are calculated over 256 independently sampled original images.
(a) Mean of the gradient similarity improvement (b) Standard deviation of the gradient similarity improvement (c) Mean accuracy over the course of optimization
Figure 4: Illustration of the search trajectory of DeepAA in comparison with DeepTA on CIFAR-10.
As shown in Figure 4(a), the cosine similarity of DeepTA reaches the peak at the 5th layer, and stacking more layers decreases the cosine similarity. In contrast, for DeepAA, the cosine similarity increases consistently until it converges to identity transformation at the 6th layer. In Figure 4(b), the standard deviation of DeepTA increases considerably when stacking more layers. In contrast, in DeepAA, as we optimize the gradient similarity along the direction of low variance, the standard deviation of DeepAA does not grow as fast as DeepTA. In Figure 4(c), both DeepAA and DeepTA reach peak performance with six augmentation layers, where DeepAA achieves better accuracy compared against DeepTA. These results indicate that maximizing the cosine similarity along the path with low variance can lead to higher performance.
Comparison with Policies Found by Other Methods. In Figure 7 in Appendix E, we compare the policy of DeepAA with the policy found by other automated data augmentation methods including AA, FastAA and DADA. We have three observations. First, AA, FastAA and DADA assign high probability (over 1.0) on flip, Cutout and crop, as those transformations are hand-picked and applied by default. DeepAA finds a similar pattern that assigns high probability on flip, Cutout and crop. Second, unlike AA which mainly focused on color transformations, DeepAA has high probability over both spatial and color transformations. Third, FastAA has evenly distributed magnitudes, while DADA has low magnitudes. In contrast, DeepAA assigns high probability to the stronger magnitudes.

heading: 5 CONCLUSION
text: In this work, we propose Deep AutoAugment (DeepAA), a fully automated multi-layer data augmentation search method without using hand-picked default transformations. DeepAA formulates data augmentation search as a regularized gradient matching problem, which maximizes the gradient similarity between augmented data and original data along the direction with low variance. Our experiment results show that DeepAA achieves comparable performance of previous works without using default augmentations, which indicates that regularized gradient matching is an effective way of searching for the full data augmentation policy from scratch.
Reproducibility Statement: We carefully explain our experiment settings to be as straightforward to readers as possible. The evaluation of the found data augmentation policy is based the public repository of Fast AutoAugment. We believe our results can be readily reproduced. Our implementation will be made public.

heading: A A LIST OF STANDARD AUGMENTATION SPACE

heading: B THE DISTRIBUTION OF MAGNITUDES FOR CIFAR-10/100

heading: C THE DISTRIBUTION OF MAGNITUDES FOR IMAGENET

heading: D HYPERPARAMETERS FOR BATCH AUGMENTATION
text: The performance of BA is sensitive to the training settings (Fort et al., 2021; Wightman et al., 2021). Therefore, we conduct a grid search on the learning rate, weight decay and number of epochs for TA and DeepAA with Batch Augmentation. The found best parameters are summarized in Table 8 in the Appendix. We did not tune the hyperparameters of AdvAA (Zhang et al., 2019) and AWS (Tian et al., 2020), since AdvAA claims to be adaptive to the training process and running AWS’s searching script to obtain the policy is too time consuming.

heading: E COMPARISON OF DATA AUGMENTATION POLICY

Review summaries:
The paper proposes a novel automatic augmentation method. That is a method that learns based on a dataset, how to augment that dataset during training of a classifier.  The proposed method works by matching gradients based on cosine similarity on fully trained networks and uses this approach to build augmentation policies with multiple augmentations applied after one another. Similar to the recent usage of very deep RandAugment for EfficientNetV2s for example.

A more "automated" augmentation search method is proposed. The main contributions are:  - Default transformations like flipping and cropping are regarded as new augmentation candidates which could be optimized, making the algorithm more automated. - The gradient matching formulation is borrowed and seems to be effective for augmentation search. - MC sampling and a progressive search pipeline are used to reduce the computational complexity.

This paper proposed a learned data augmentation method. The proposed method is able to search for the augmentation policy with multiple layers by searching the next transformation operation conditioned on all previous augmentations. The authors also proposed to use cosine similarity between the gradients of the original and augmented data as guidance rather than e.g. the more commonly used classification validation loss.

This paper provides an interesting finding that instead of injecting experts prior to hand-pick augmentations at the top layers, the entire search can be formulated end-to-end in a multi-layer (deep) way without hand-picked transformation optimized with gradient matching. The independent sampling at each layer with estimated mean gradient of training data is used to stabilize the training and make the optimization process more efficient. Empirical results on CIFAR-10/100 and ImageNet outperform previous best performing DAS methods.