Deep AutoAugment
Authors: Yu Zheng
Zhi Zhang
Shen Yan
Mi Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: automated machine learning
data augmentation

Abstract: While recent automated data augmentation methods lead to state-of-the-art results, their design spaces and the derived data augmentation strategies still incorporate strong human priors. In this work, instead of fixing a set of hand-picked default augmentations alongside the searched data augmentations, we propose a fully automated approach for data augmentation search named Deep AutoAugment (DeepAA). DeepAA progressively builds a multi-layer data augmentation pipeline from scratch by stacking augmentation layers one at a time until reaching convergence. For each augmentation layer, the policy is optimized to maximize the cosine similarity between the gradients of the original and augmented data along the direction with low variance. Our experiments show that even without default augmentations, we can learn an augmentation policy that achieves strong performance with that of previous works. Extensive ablation studies show that the regularized gradient matching is an effective search method for data augmentation policies. Our code is available at: https://github.com/MSU-MLSys-Lab/DeepAA .

Published as a conference paper at ICLR 2022
DEEP AUTOAUGMENT
Yu Zheng', Zhi Zhang", Shen Yan', Mi Zhang' 1 Michigan State University, AAmazon Web Services zhengy30emsu. edu,
com, (yanshen6, mi mizhang) @msu edu edu
ABSTRACT
While recent automated data augmentation methods lead state-of-the-ard results, their design spaces and the derived data augmentation strategies still incorporate strong human priors. In this work, instead of fixing a set of haad-picker default augmentations alongside the searched data augmentations, we propose a fully automated approach for data augmentation search named Deep AutoAugment (DeepAA). DeepAA progressively builds a multi-layer data augmentation pipeline from scratch by stacking augmentation layers one at a time until reaching conver- gence. For each augmentation layer, the policy is optimized to maximize the cosine similarity between the gradients the original and augmented data along the direction with low variance. Our experiments show that even without default aug- mentations, we can learn an augmentation policy that achieves strong performance with that of previous works. Extensive ablation studies show that the regularized gradient matching is an effective search method for data augmentation policies. Our code is available at: htts://gttuo.com///////M/SsLLLLD
INTRODUCTION
Data augmentation (DA) a powerful tech- nique for machine learning since it effec-
Trandfmmation
Tranefomation
tively regularizes the model by increas-
ing the number and the diversity of data
Augmentation Policy
points (Goodfellow et al.] 2016 Zhang al.) 2017). A large body of data aug- mentation transformations has been pro-
Tranfornaiion
Transformation
posed (Inoue) 2018: IZhang et 2018
DeVries & Taylor 2017: Yun et al. 2019
Hendryeks et al. 2020 Yan al. 2020
Debult Tonatomation
Transtomation
to improve model performance. While ap- plying a set of well-designed augmentation transformations could help yield consider- able performance enhancement especially in image recognition tasks, manually select- ing high-quality augmentation transforma- tions and determining how they should be combined still require strong domain exper- tise and prior knowledge of the dataset of interest. With the recent trend of automated machine learning (AutoML), data augmen- tation search flourishes in the image domain (Cubuk et al. 2019 2020 Ho et al. 2019 Lim et al.
Defuut Tonetomation 2
Tranffomation
K
(8)
Figure 1: (A) Existing automated data augmentation meth- ods with shallow augmentation policy followed by hand- picked transformations. (B) DeepAA with deep augmentation policy with no hand-picked transformations.
2019 Hataya et al., 2020 Li et al.) 2020 Liu et al. (021). which yields significant performance improvement over hand- -and-erafted data augmentation methods.
Although data augmentation policies in previous works (Cubuk et al. 2019 2020 Ho et al. 2019 Lim
al. 2019 Hataya et al. 2020 Li et al. 2020) contain multiple transformations applied sequentially. only one or two transformations of each sub-policy are found through searching whereas the rest transformations are hand-picked and applied by default in addition to the found policy (Figure A)). From this perspective, we believe that previous automated methods are not entirely automated as they are still built upon hand-crafted default augmentations.

---

Published as a conference paper at ICLR 2022
In this work, we propose Deep AutoAugment (DeepAA), multi-layer data augmentation search method which aims to remove the need of hand-crafted default transformations (Figure(T(B)). DeepAA fully automates the data augmentation process by searching deep data augmentation policy on an expanded set of transformations that includes the widely adopted search space and the default transformations (e.g. flips, Cutout, crop). We formulate the search of data augmentation policy regularized gradient matching problem by maximizing the cosine similarity of the gradients between augmented data and original data with regularization. To avoid exponential growth of dimensionality of the search space when more augmentation layers are used, we incrementally stack augmentation layers based on the data distribution transformed by all the previous augmentation layers.
We evaluate the performance of DeepAA on three datasets - CIFAR-10. 10, CIFAR-ION and ImageNet- and compare it with existing automated data augmentation search methods including AutoAugment
(AA) (Cubuk et al. 2019), PBA (Ho et al. 2019), Fast AutoAugment (FastAA) (Lim et al. 2019). Faster Autonnu (Faster AA) (Hataya et al. 2020), DADA (Li et al. 2020), RandAugment (RA)
(Cubuk et al. 2020), UniformAugment (UA) (LingChen et al. 2020). TrivialAugment (TA) (Müller & Hutter (021), and Adversarial AugoAugment (AdvAA) (Zhang et al. 2019). Our results show that,
without any default augmentations, DeepAA achieves the best performance compared to existing automatic augmentation search methods on CIFAR-10. 10, CIFAR- 100 on Wide-ResNet--2 28-12 10 and ImageNet ResNet-50 and ResNet-220 with standard augmentation space and training procedure.
We summarize our main contributions below:
We propose Deep AutoAugment (DeepAA), a fully automated data augmentation search method that finds a multi-layer data augmentation policy from scratch.
We formulate such multi-layer data augmentation search as regularized gradient matching problem. We show that maximizing cosine similarity along the direction of low variance is effective for data augmentation search when augmentation layers go deep.
We address the issue of exponential growth of the dimensionality of the search space when more augmentation layers are added by incrementally adding augmentation layers based on the data distribution transformed by all the previous augmentation layers.
Our experiment results show that, without using any default augmentations, DeepAA achieves stronger performance compared with prior works.
2 RELATED WORK
Automated Data Augmentation. Automating data augmentation policy design has recently emerged as a promising paradigm for data augmentation. The pioneer work on automated data augmentation was proposed in AutoAugment (Cubuk et al. 2019), where the search is performed under reinforce- ment learning framework. AutoAugment requires to train the neural network repeatedly, which
takes thousands of GPU hours to converge. Subsequent works (Lim al. 2019 Li et al. 2020 Liu
ct al. 2021) aim at reducing the computation cost. Fast AutoAugment (Lim et al. 2019) treats data augmentation inference time density matching which can be implemented efficienlly with Bayesian optimization. Differentiable Automatic Data Augmentation (DADA) (Li et af. 2020) further reduces the computation cost through a reparameterized Gumbel-softmax distribution (Jang al. (017). RandAugment (Cubuk et al. 2020) introduces a simplified search space containing two interpretable hyperparameters, which can be optimized simply by grid search. Adversarial AutoAugment (AdvAA) (Zhang et al. 2019) searches for the augmentation policy in an adversarial and online manner. It also incorporates the concept of Batch Augmentaiton (Berman al., 2019 Hoffer et al. (020). where multiple adversarial policies run in parallel. Although many automated data augmentation methods have been proposed, the use of default augmentations still imposes strong domain knowledge.
Gradient Matching, Our work is also related to gradient matching. In (Du et al., 2018), the authors showed that the cosine similarity between the gradients of different tasks provides a signal to detect when an auxiliary loss is helpful to the main loss. In (Wang et al. 2020). the authors proposed to use cosine similarity as the training signal to optimize the data usage via weighting data points. A similar approach was proposed in (Müller et al. 2021). l). which uses the gradient inner product per- example reward for optimizing data distribution and data augmentation under the reinforcement learning framework, Our approach also utilizes the cosine similarity to guide the data augmentation
2

---

Published as a conference paper at ICLR 2022
search. However, our implementation of cosine similarity is different from the above from two aspects: we propose a Jacobian-vectie product form to backpropagate through the cosine similarity, which is computational and memory efficient and does not require computing higher order derivative; we also propose a sampling scheme that effectively allows the cosine similarity to increase with added augmentation stages.
3
DEEP AUTOAUGMENT
3.1 OVERVIEW
Data augmentation can be viewed as process of filling missing data points in the dataset with the same data distribution (Hataya et al.) 2020). By augmenting single data point multiple times, we expect the resulting data distribution to be close to the full dataset under a certain type of transformation. For example, by augmenting single image with proper color jittering, batch of augmented images which has similar distribution of lighting conditions as the full dataset. As the distribution of augmented data gets closer the full dataset, the gradient of the augmented data should be steered towards a batch of original data sampled from the dataset. In DeepA we formulate the search of the data augmentation policy as a regularized gradient matching problem, which manages to steer the gradient to batch of original data by augmenting single image multiple times. Specifically, we construct the augmented training batch by augmenting a single training data point multiple times following the augmentation policy. We construct a validation batch by sampling a batch of original data from the validation set. We expect that by augmentation, the gradient of augmented training batch can be steered towards the gradient of the validation batch. To do so, we search for data augmentation that maximizes the cosine similarity between the gradients of the validation data and the augmented training data. The intuition is that effective data augmentation should preserve data distribution (Chen et al. 2020) where the distribution of the augmented images should align with the distribution of the validation set such that the training gradient direction is close to the validation gradient direction.
obtain
Another challenge for augmentation policy search is that the search space can be prohibitively large with deep augmentation layers (K 2 5). This was not a problem in previous works, where the augmentation policies is shallow (K < 2). For example, in AutoAugment (Cuuuk et al.) (2019). each sub-policy contains K = 2 transformations to be applied sequentially, and the search space of AutoAugment contains 16 image operations and 10 discrete magniude levels. The resulting number of combinations of transformations in AutoAugment is roughly (16 x 10) = 25, 600, which is handled well in previous works. However, when discarding the default augmentation pipeline and searching for data augmentations from scratch, it requires deeper augmentation layers in order to perform well. For data augmentation with K sequentially applied transformations, the number a
of sub-policies is (16 x 10)$ ~ 10 which is prohibitively large for the following two reasons. First, it becomes less likely to encounter good policy by exploration as good policies become more sparse on high dimensional search space. Second, the dimension of parameters in the policy also grows with K, making it more computational challenging to optimize. To tackle this challenge, we propose to build up the full data augmentation by progressively stacking augmentation layers, where each augmentation layer is optimized on top of the data distribution transformed by all previous layers. This avoids sampling sub-policies from such large search space, and the number of parameters of the policy is reduced from TT K to T for each augmentation layer.
3.2 SEARCH SPACE
Let O denote the set of augmentation operations (e.g. identity, rotate, brightness), m denote an operation magnitude in the set MI, and æ denote an image sampled from the space X'. We define the set of transformations as the set of operations with fixed magnitude as T := [t]t o(+ m)) o E O and m E M). Under this definition, every i is a map t: X + X, and there are T M [O] possible transformations. In previous works (Cubuk et al. 2019 Lim et al. 2019 Li et al. 2020 Hataya et al., 2020), a data augmentation policy P consists of several sub-policies. As explained above, the size of candidate sub-policies grows exponentially with depth K. Therefore, we propose a practical method that builds up the full data augmentation by progressively stacking augmentation layers. The final data augmentation policy hence consists of K layers of sequentially applied policy P = (Pi, .., Pr), where policy Ps is optimized conditioned on the data distribution augmented
3

---

Published as a conference paper at ICLR 2022
by all previous (I: 1) layers of policies. Thus write the policy conditional distribution Ps: := Po, (n (Pi.... , where n denotes the indices of transformations in T. For the purpose of clarity,
simplified notation Po, replace po, (n|(PI, Pk-1)).
3.3 AUGMENTATION POLICY SEARCH VIA REGULARIZED GRADIENT MATCHING
Assume that single data point z is augmented multiple times following the policy The resulting average gradient such augmentation denoted as g(z, 8). which function data and policy parameters 0. Let e denote the gradients of a batch of the original data. We optimize the policy by maximizing the cosine similarity between the gradients of the augmented data and a batch of the original data as follows:
0 = arg max cosineSimilarityye..fe 0))
(1)
v² g(x, 0)
arg max
| 0)
where ||-|| denotes the L2-norm. The parameters of the policy be updated via gradient ascent:
x- 0 + v cosineSimilarity(.. (z,
where 1) is the learning rate.
3.3.1 POLICY SEARCH FOR ONE LAYER
We start with the case where the data augmentation policy only contains a single augmentation layer, i.e., P {pe} }. Let L(x; w) denote the classification loss of data point z where IR represents the flattened weights the neural network. Consider applying augmentation on single data point following the distribution ps. The resulting averaged gradient can be calculated analytically by averaging all the possible transformations in with the corresponding probability p(0):
g(0;
Σ
(3)
G(x) pe
where G(x) = V (t)( ( is a D x
[T] Jacobian matrix, and Pe [po( , is [T] dimensional categorical distribution. The gradient w.r.t. the cosine
similarity in Eq. can be derived as:
V cosineSimilarityy g(2; 0))
opo
(4)
where
(0)
(
r G=
(5)
[g((0]]
|0)
which can be interpreted as a reward for each transformation. Therefore, pe T in Eq.44 represents the average reward under policy Pa-
3.3.2 POLICY SEARCH FOR MULTIPLE LAYERS
The above derivation is based on the assumption that g(0) can be computed analytically by (3.(3) However, when K 2 2, it becomes impractical to compute the average gradient of the augmented data given that the search space dimensionality grows exponentially with K. Consequently, need to average the gradient of all [T] K possible sub-policies.
To reduce the parameters of the policy to T for each augmentation layer, propose to incrementally stack augmentations based on the data distribution transformed by all the previous augmentation layers. Specifically, let P ........ PP} } denote the K K-layer policy. The policy Ps modifies the data distribution on top of the data distribution augmented by the previous (A 1) layers. Therefore, the policy at the 1tth layer is distribution Ps po, (n) conditioned on the policies "PI,...,PA. PP-11)
4

---

Published as a conference paper at ICLR 2022
where each one is a -immennnsi categorical distribution. Given that, the Jacobian matrix the hth layer can be derived by averaging over the previous (A: - 1)) layers of policies as follows:
[T|
G(z)*=
Σ Thk
Po. (mx-1) (n, L. tm (x);
(6)
( o tne-. o...o tn, (2);
where G can be estimated via the Monte Carlo method as:
(z)
Σ
Σ (uL((t o the_. ... ... thr
Po- Pox
Aimppo
V ((tpl th o...o tx,
where 7g-1 ~ p ... nu ~ pe, (n).
The average gradient at the hh layer can be estimated by the Monte Carlo method as:
a(a; 0k)
E
Σ
Vu ((tan tay (x);
(8)
Therefore, the reward the htth layer is derived as:
7 (x) G (x))) (1
2T (x; 0k)
(x; 0k)
(x;
1 (z; 0x)
1 (I; 0s)
To prevent the augmentation policy from overfitting, we regularize the optimization by avoiding optimizing towards the direction with high variance. Thus, we penalize the average reward with its standard deviation
=
E, Ex - c. -.
(10)
where 16 randomly sampled images to calculate the expectation. The hyperparameter controls the degree regularization, which set to 1.0. With such regularization, we prevent the policy from converging to the transformations with high variance.
Therefore the parameters of policy Pk (k 2 2) can be updated as: 0₂ 1 Ve, cosineSimilarity(.. ((,)
(11)
where
cosineSiimilarityy... (x; Vero.
(12)
EXPERIMENTS AND ANALYSIS
Benchmarks and Baselines. We evaluate the performance of DeepAA on three standard benchmarks:
CIFAR-10, CIFAR-100 ImageNet, and compare it against a baseline based on standard augmentations
(i.e., flip left-righ, pad-and-crop for CIFAR-10/100 and Inception-style preprocesing (Szegedy
jt al. 2015) for ImageNet) as well as nine existing automatic augmentation methods including (1)
AutoAugment (AA) (Cubuk al.) 2019), (2) PBA (Ho et 2019), (3) Fast AutoAugment (Fast
AA) (Lim et al. 2019). (4) Faster AutoAugment (Hataya et al. 2020), (5) DADA (Li et al. 2020). (6)
RandAugment (RA) (Cubuk al. 2020), (7) UniformAugment (UA) (LingChen al. 2020).
TuimialAugment (TA) Müüller & Hutter 2021), l), and (9) Adversarial AutoAugment (AdvAA) (Zhang
al. 2019).
Search Space. We set up the operation set O to include 16 commonly used operations (identity. shear- x, shear-y, translate-x, translate-y, rotate, solarize, equalize, color, posterize, contrast, brightness, sharpness, autoContrast, invert, Cutout) well as two operations (i.e., flips and crop) that are used as the default operations in the aforementioned methods. Among the operations in O, 11 operations are associated with magnitudes. We then discretize the range of magnitudes into 12 uniformly spaced levels and treat each operation with a discrete magnitude as an independent transformation. Therefore, the policy in each layer is a dimennennil categorical distribution corresponding to (T] 139 (operation, magnitude] pairs. The list operations and the range of magnitudes in the standard augmentation space summarized in Appendix A
5

---

Published as a conference paper at ICLR 2022
4.1 PERFORMANCE ON CIFAR- 10 AND CIFAR-100
Policy Search. Following (Cubuk et al. 2019). we conduct the augmentation policy search based on Wide-ResNer-40-22 (Zagoruyko & Komodakis 2016). We first train the network on a subset of 4, 000 randomly selected samples from CIFAR-TO. We then progressively update the policy network parameters 0% (k 1.2, KK) for 512 iterations for each of the augmentation layers. We use the Adam optimizer Kingmaa & Ba 2013 and set the learning rate to 0.025 for policy updating,
Policy Evaluation. Using the publicly available repository of Fast AutoAugment (Lim et al. 2019), we evaluate the found augmentation policy on both CIFAR- 10 and CIFAR- 100 using Wide ResNet- 28-10 and Shake- Shake-Shake 2296 models. The evaluation configurations are kept consistent with that of Fast AutoAugment.
Results. Table [ reports the Top-1 test accuracy on CIFAR- 10/100 for Wide-ResNet-288 and Saake--Saake2x999 respectively. The results of DeepAA are the average of four independent runs with different initializations. We also show the 95% confidence interval the mean accuracy. As shown, DeepAA achieves the best performance compared against previous works using the standard augmentation space. Note that TA (Aidd uses a wider (stronger) augmentation space on this dataset.
Baxeline
AA
PBA
FasIAA
FasterAA
DADA
RA
UA
TA(RA)
TA(Wide)
DeepAA
CIFAR-10
WRN-28-10
96.1
97.4
97.4
97.3
97.4
97.3
97.3
97.33
97.46
97.46 98.21
97.56 0.14
Shake-Shake (26 2x960)
97.1
98.0
98.0
98.0
98.0
98.0
98.0
98.1
98.05
98.11 0,12
CIFAR-100
WRN-28-110
81.2
82.9
83.3
82.7
82.7
82.5
83.3
82,82
83.54
84.33
84.02 0.18
Shake 2x96d)
85.7
84.7
86.19
Table I: Top-1 test accuracy CIFAR-10/100 for Wide-ResNet-28-10 10 and Shake-Shake-2x966 The results of A A are averaged over four independent runs with different initializations. The 95% confidence interval is denoted by +.
4.2 PERFORMANCE ON IMAGENET
Policy Search. We conduct the augmentation policy search based on ResNet-18 (He et al. 2016). We first train the network on a subset 200, 000 randomly selected samples from ImageNet for 30 epochs. We then use the same settings as in CIFAR-I0 for updating the policy parameters.
Policy Evaluation. We evaluate the performance the found augmentation policy on ResNet-50 and ResNet-200 based on the public repository of Fast AutoAugment (Lim et al. 2019). The parameters for training are the same as the ones (Lim et al. 2019). In particular, we use step learning rate scheduler with reduction factor of 0.1, and we train and evaluate with images of size 224x224.
The performance on ImageNet is presented Table 2 As shown, DeepAA achieves the best performance compared with previous methods without the use of default augmentation pipeline. In particular, DeepAA performs better on larger models (i.e. 2000), as the performance of DeepAA ResNet-200 is the best within the 95% confidence interval. Note that while we train DeepAA using the image resolution (224x224), we report the best results of RA and TA, which are trained with a larger image resolution (244x224) on this dataset.
Results.
Baseline
AA
Fast AA
Faster AA
DADA
RA
UA
TACRA
TA(Wide 2
DeepAA
ResNet-50 ResNet-200
76.3
77.6
77.6
76.5
77.5
77.6
77.63
77.85
78.07
78.30 1 0.14 81.32 0.17
80.0
80.6
80.4
Table 2: Top-1 test accuracy (%) on ImageNet for ResNet-50 and ResNet-200. The results DeepAA are averaged over four independent runs with different initializations. The 95% confidence interval is denoted by t.
4.3 PERFORMANCE WITH BATCH AUGMENTATION
Batch Augmentation (BA) is a technique that draws multiple augmented instances of the same sample in one mini-batch. It has been shown to be able to improve the generalization performance of the
On CIFAR-10/100, TA (Wide) uses wider (stronger) augmentation space, while the other methods including TA (RA) uses the standard augmentation space.

---

Published as a conference paper at ICLR 2022
Hoffer al. 2020). AdvAA Zhang et al. 2019) directly searches for the augmentation policy under the BA setting whereas for TA and DeepAA, we apply BA with the same augmentation policy used in Table @ Note that since the performance of BA is sensitive to the hyperparameters (Fort et al. 2021). we have conducted a grid search on the hyperparameters of both TA and DeepAA (details included in Appendix DD. As shown in Table after tuning the hyperparameters, the performance of TA (Wide) using BA is already better than the reported performance in the original paper. The performance of DeepAA with BA outperforms that of both AdvAA and TA (Wide) with BA.
network (Berman al., 2019;
TAdWide) (original paper)
TA(Wide) (ours)
AdvAA
DeepA
CIFAR-10 CIFAR- 100
98.1 = 0.15
98.04 + 0.06
98. 08.06-2 + 0.23
98.21 0.14
84.51 + 0.18
84.62 + 0.14
85.40 1 0.15
85.61 0.17
Table Top-1 test accuracy (%) on CIFAR- 10/100 dataset with WRN-28 200-128-100 with Batch Augmentation (BA), where eight augmented instances were drawn for each image. The results of DeepAA are averaged over four independent runs with different initializations. The 95% confidence interval is denoted by 1.
4.4 UNDERSTANDING DEEPAA
Effectiveness of Gradient Matching. One unique- ness of DeepAA is the regularized gradient match- ing objective. To examine its effectiveness, we remove the impact coming from multiple augmen- tation layers, and only conduct search for single layer of augmentation policy. When evaluating the searched policy, we apply the default augmentation in addition to the searched policy. We refer to this variant as DeepAA-simple Figure Seuureecompper compares the Top-1 test accuracy ImageNet using ResNet- 50 between AbepAA-simple DeepA A, and other automatic augmentation methods. While there is 0.22% performance drop compared to DeepAA, with a single augmentation layer, DeepAA--immp still outperforms other methods and able to achieve similar performance compared to TA (Wide) but with standard augmentation space and trains on a smaller image size (224x224 vs 244x224).
78.5%
77.5%
77.6 77.6
71.5
77.67265
77.0%
26.5
76.5%
76.3
76.0%
Pest. Farton
Despli Doc
[8A] (W/0k))
Figure 2: Top-1 test accuracy (%) on ImageNet of DeepAA-simple, DeepAA, and other automatic augmentation methods on ResNet-50.
Policy Search Cost. Table compares the policy search time on CIFAR- 10/100 and ImageNet in GPU hours. D has comparable search time as PBA, Fast AA, and RA, but is slower than Faster AA and DADA. Note that Faster AA and DADA relax the discrete search space to continuous one similar to DARTS (Liu et al., (2088). While such relaxation leads shorter searching time, inevitably introduces a discrepancy between the true and relaxed augmentation spaces.
Dataset
AA
PBA
Fast AA
Faster AA
DADA
RA
DeepAA
CIFAR-10/100
5000
3.5
0.23
0.1
25
9
ImageNet
15000
2.3
1.3
5000
96
Table 4: Policy search time on CIFAR-10/100 and ImageNet in GPU hours.
Impact of the Number of Augmentation Layers. Another uniqueness DeepAA is its multi-layer search space that can go beyond two layers which existing automatic augmentation methods were designed upon. We examine the impact of the number of augmentation layers on the performance of DeepAA. Table 5 Table 6 show the performance on CLFAR-10/100 and ImageNet respectively with increasing number of augmentation layers. As shown, for CIFAR- 10/100/0000000000000 the performance gradually improves when more augmentation layers are added until we reach five layers. The performance does not improve when the sixth layer is added. For ImageNet, we have similar
'TA (RA) achieves 77.55% top-1 accuracy with image resolution 224x224.
TA (Wide) achieves 77.97% top-1 accuracy with image resolution 224.
7

---

Published as a conference paper at ICLR 2022
identity
Shcar
shearey
translate-x
tranclate
rotate
autoContrent
Invert
equalize
solarize
posteniee
contrast
brightness
sharpness
fpp
Culout
crop
100%
100%
80%
80%
Relalive percent
Relative percent
60%
60%
60%
40%+
20%
Layer indlen
Layer index
Operation distribution for CIFAR- 10/100
(b) Operatiin distribution ImageNeet
Figure 3: The distribution of operations at each layer the policy for CIFAR-10/100 and ImageNet. The probability of each operation is summed up over all 12 discrete intensity levels (see Appendix BBandd C of the corresponding transformation.
observation where the performance stops improving when more than five augmentation layers are included.
1 layer
2 layers
layers
4 layers
5 layers
layers
CIFAR-10 CIFAR-100
96.3 0.21
96.6 + 0.18
96.9 0.12
97.4 + 0.14
97.56 + 0.14
97.6 0.12
80.9 + 0.31
81.7 + 0.24
82.2 + 0.21
83.7 + 0.24
84.02 + 0.18
84.0 0.19
Table 5: Top-1 test accuracy of DeepAA CIFAR- 10/100 for different numbers of augmentation layers. The results are averaged over 4 independent runs with different initializations with the 95% confidence interval denoted by 1.
layer
layers
5 layers
7 layers
ImageNet 75.27 + 0.19 78. 18 0.22 78.30 0.14 78.30 0.14
Table 6: Top-1 test accuracy of DeepAA on ImageNet with ResNet-50 for different numbers of augmentation layers. results are averaged over independent runs w/ different initializations with the 95% confidence interval denoted by +.
Figure 3 illustrates the distributions of operations in the policy for CIFAR-10/100 and ImageNet respectively. As shown in Figure a]) the augmentation of CIFAR-10/100 converges to identity transformation at the sixth augmentation layer, which is natural indication of the end of the augmentation pipeline. We have similar observation in Figure33b for ImageNet, where the identity transformation dominates in the sixth augmentation layer. These observations match results listed in Table and Table T6] We also include the distribution of the magnitude within each operation for CIFAR- 10/100 and ImageNet in Appendix B and Appendix C
Validity of Optimizing Gradient Matching with Regularization. To evaluate the validity of opti- mizing gradient matching with regularization, we designed search-free baseline named "DeepTA". In DeepTAA, we stack multiple layers of TA on the same augmentation space DeepAA without using default augmentations. As stated in Eq. (T0) and Eq. we explicitly optimize the gradient similarities with the average reward minus its standard deviation. The first term the average reward Ez (7* (x)} - encourages the direction of high cosine similarity. The second term the standard deviation of the reward /Fiz Fez ((p (x) Eez (it (f*(()) (x))))] acts as regularization that penalizes the direction with high variance. These two terms jointly maximize the gradient similarity along the direction with low variance. To illustrate the optimization trajectory, we design two metrics that closely related to the two terms in Eq.(10), the mean value, and the standard deviation of the improvement of gradient similarity. The improvement of gradient similarity is obtained by subtracting the cosine similarity of the original image batch from that of the augmented batch. In our experiment, the mean and standard deviation of the gradient similarity improvement are calculated over 256 independently sampled original images.
8

---

Published as a conference paper at ICLR 2022
Stundord davisticn grodienn
similanty improvmnant
simileity ingroveneet
0.155
97.6
DeEphA DeapiA
Mona gradient
0.18
0.150
Dep's
Mean aocuracy
97.4
0.145
97.2
14
0. 0.140
96.8
0.130
96.5
0.10
0.125
96
2
5
Layer index
milex
Layer index
(a) Mean gradient similarity (b) Standard deviation of gradi- (c) Mean accuracy over different aug- improvement
ent similarity improvement
mentation depth
Figure 4: Illustration of the search trajectory of DeepAA in comparison with DeepTA CIFAR-I0.
As shown in Figure 4a), a), the cosine similarity of DeepTA reaches the peak at the fifth layer, and stacking more layers decreases the cosine similarity. contrast, for DeepAA, the cosine similarity increases consistently until it converges to identity transformation at the sixth layer. In Figure bb). the standard deviation of DeepTA significantly increases when stacking more layers. In contrast, in DeepAA, as optimize the gradient similarity along the direction of low variance, the standard deviation of DeepA does not grow as fast as Dee Figure c), both DeepAA and Deep'TA reach peak performance at the sixth layer, but DeepA achieves better accuracy compared against Dee Therefore, we empirically show that DeepAA effectively scales up the augmentation depth by increasing cosine similarity along the direction with low variance, leading to better results.
Comparison with Other Policies. In Figure 7 in Appendix B we compare the policy of DeepAA with the policy found by other data augmentation search methods including AA, FastAA and DADA. We have three interesting observations:
AA, FastAA and DADA assign high probability (over 1.0) on flip, Cutout and crop, those transformations are hand-picked and applied by default. DeepAA finds a similar pattern that assigns high probability on flip, Cutout and crop.
Unlike AA, which mainly focused on color transformations, DeepAA has high probability over both spatial and color transformations.
FastAA has evenly distributed magnitudes, while DADA has low magnitudes (common issues in DARTS-like method). Interestingly, DeepAA assigns high probability to the stronger magnitudes.
5 CONCLUSION
In this work, we present Deep AutoAugment (DeepAA), multi-layer data augmentation search method that finds deep data augmentation policy without using any hand-picked default transforma- tions. We formulate data augmentation search as a regularized gradient matching problem, which maximizes the gradient similarity between augmented data and original data along the direction with low variance. Our experimental results show that DeepAA achieves strong performance without using default augmentations, indicating that regularized gradient matching an effective search method for data augmentation policies.
Reproducibility Statement: We have described our experiment settings in great details. The evaluation of the found data augmentation policy is based the public repository of Fast AutoAugment. believe that our results can be readily reproduced.
ACKNOWLEDGEMENT
We thank Yi Zhu, Hang Zhang, Haichen Shen, Mu Li, and Alexander Smola for their help with this work. This work was partially supported by NSF Award (I:BII-1632051 and Amazon AWS Machine Learning Research Award.

---

Published as a conference paper at ICLR 2022
REFERENCES
Maxim Berman, Hervé Jégou, Andrea Vedaldi, lasonas Kokkinos, and Matthijs Douze. Multigrain: a unified image embedding for classes and instances. arXiv preprint arXiv:1902.0550% 2019.
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21 21(25)):1-71, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space, Advances Neural Information Processing Systems, volume 33, pp. 702-703, 2020.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: (e0i::170..45550 04552, 2017.
Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan, Adapting auxiliary losses using gradient similarity, arXiv preprint arXiv:1812.00224., 2018.
Stanislav Fort, Andrew Brock, Razvan Pascanu, Soham De, and Samuel Smith. Drawing multiple augmentation samples per image during training efficiently decreases test enror. arXiv preprint arXiv:2/05.11344, 2021.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT press Cambridge, 2016.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. In European Conference on Computer Vision, pp. 1-16. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. International Conference Learning Representations, 2020.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp.
PMLR, 2019.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8129-8111 9138, 2020.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. arXiv preprint arXiv: 2018
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax. In International Conference Learning Representations, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Yonggang Li. Guosheng Hu, Yongtao Wang. Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In European Conference on Computer Vision, pp. 580-595. Springer, 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2022
Tom Ching LingChen, Ava Khonsari, Amireeza Lashkari, Mina Rafi Nazari, Jaspreet Singh Sambee, and Mario A Nascimento. Uniformaugment: A search-free probabilistic data augmentation approach. arXiv preprint atXiv:2003.14348 2020.
2020.
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang. Direct differentiable augmentation search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12219-12222 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference Learning Reprexentations, 2018.
Samuel Müller, André Biedenkapp, and Frank Hutter. In-loop meta-learning with gradient-alignment
reward. arXiv preprint arXiv:2102.03275, 2021.
Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-an data augmenta- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 774-782, October 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Xinyi Wang. Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. In International Conference on Machine Learning, pp. 9983-9995. PMLR, 2020.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. volume 34, 2021.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv: 2001.00677, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. International Conference on Learning Representations, 2019.

---

Published as a conference paper at ICLR 2022
A A LIST OF STANDARD AUGMENTATION SPACE
Operation
Magnitude
Identity
ShearX
[-0.3, 0.3]
ShearY
[-0.3, 0.3]
TranslateX
[-0.45, 0.45]
Translate
[-0.45, 0.45]
Rotate
[-30, 30]
AutoContrast
Invert
Equalize
Solarize
[0, 256]
Posterize
[4,
Contrast
1091
Color
[..1
Brightness
[0.1
Sharpness
[0.1
Flips
Cutout
16 (60)
Crop
Table 7: List of operations in the search space and the corresponding range of magnitudes in the standard augmentation space. Note that some operations do not use magnitude parameters. We add flip and crop to the search space which were found in the default augmentation pipeline in previous works. Flips operates by randomly flipping the images with 50% probability. In line with previous works, crop denotes pad-and-crop and resize-and-crog transforms for CIFAR10/ respectively. We set Cutout magniude to 16 for CIFAR10/100 dataset to be the same as the Cutout in the default augmentation pipeline. We set Cutout magnitude to 60 pixels for ImageNet which is the upper limit of the magnitude used in AA (Cubuk et 2019).
and ImageNet
12

---

Published as a conference paper at ICLR 2022
B
THE DISTRIBUTION OF MAGNITUDES FOR CIFAR-10/100
shear-x
0.5
0.5
0.25
94
shear-y
025
0.85
80
translate-x
80
translare-y
0.25
036
0.25
rolate
0.26
0.25
0.
60
solarize
0.
60
posterine
contrast
0.2
0.5
color
025
0.2
brightness
00
025
6.33
64
sharyness
035
00
Layer
Layer
Layer 3
shear-g
10
shear-y
30
translate-x
68
translate-
85
65
10.
00
rolate
65
65
1.0
00
solarize
posterizo
00
80
contrast
color
65
14
brighmees
86
sharpneas
0.5
0.0
Layer
Layer
Layer 6
Figure 5: The distribution of discrete magnitudes of each augmentation transformation in each layer of the policy for CIFAR-10/100. The x-axis represents the discrete magnitudes and the y-axis represents the probability. The magnitude is discretized to 12 levels with each transformation having its own range. large absolute value of the magnitude corresponds to high transformation intensity. Note that we do not show identity, autoContrast, invert, equalize, flips, Cutout and crop because they do have intensity parameters.
13

---

Published as a conference paper at ICLR 2022
C
THE DISTRIBUTION OF MAGNITUDES FOR IMAGENET
slcars
0.2
6.23
64
0.
shear-y
0.5
translate-x
translate-y
65
015
0.25
06
rotate
6.55
88
0.9
0.
solarize
0.5
0.8
0.0
posterize
0.5
11
contrast
color
0.25
6.25
brightness
0.25
66
0.
00
sharpness
38
0.0
0.8
Layer
Layer
Layer
shear-k
0.55
00
60
00
shear-y
0.5
translate-
translate-y
0.
0.25
rotate
0.5
0.25
0.5
0.
0.0
solarize
0.0
posterize
contrast
10
10
22
color
0.5
0.5
brightness
0.25
84
sharpness
0.25
64
Layer
Tayer
Layer 6
Figure 6: The distribution of discrete magnitudes of each augmentation transformation in each layer of the policy for ImageNet. The x-axis represents the discrete magnitudes and the y-axis represents the probability. The magnitude is discretized to 12 levels with each transformation having its own range. large absolute value of the magnitude corresponds to high transformation intensity. Note that we do not show identity, autoContrast. invert, equalize, flips. Cutout and crop because they do not have intensity parameters.
14

---

Published as a conference paper at ICLR 2022
D HYPERPARAMETERS FOR BATCH AUGMENTATION
The performance of BA is sensitive to the training settings (Fort et al. 2021; Wightman et al. 2021) Therefore, conduct grid search the learning rate, weight decay and number of epochs for TA and DeepAA with Batch Augmentation. The best found parameters are summarized in Table 8 in
Appendix. We did tune the hyperparameters of AdvAA (Zhang al. 2019) since AdvAA claims to be adaptive to the training process.
Dataset
Augmentation
Model
Batch Size
Learning Rate
Weight Decay
Epoch
TA (Wide)
WRN-28-10
0.2
0.0005
100
CIFAR-I0
DeepAA
WRN-28-10
128
x 8
0.2
0.001
100
TA (Wide)
WRN-28-10
128
8
0.4
0.0005
35
CIFAR-100
DeepAA
WRN-28-10
128
8
0.4
0.0005
35
Table 8: Model hyperparameters of Batch Augmentation on CIFARIO/100 for TA (Wide) and DeepAA. Learning rate, weight decay and number of epochs are found via grid search.
15

---

Published as a conference paper at ICLR 2022
E COMPARISON OF DATA AUGMENTATION POLICY
valty
shear-x
sheary
sheary
translate-
manndare-
translateyy
rotate
autaContrast
autoContrest
Invert
invert
equalize
equalize
solarize
solarize
pasterize
contrast
contrast
color
brightness
biighnees
sharpness
sharpress
flip
Cutout
crop
crop
0.0
0.1
0.2
1.0
1.4
0.0
C.2
1.0
1.4
(a) DeepAA
(b)
idereity
Idene 5y
shear-x
shear-i
sheary
sheary
translate-x
trnslate-x
translateyy
rotate
mtate
autooontrast
autoContract
invert
invet
equalize
equalize
posterize
poseeriee
contrast
contrast
color
color
brightness
brightness
sharpness
flp
Cutout
Cutcut
<0pp
crop
0.0
0.1
0.2
1.0
1.2
1.4
0.0
0.1
0.2
1.0
1.2
12
(c) FastAA
(d) DADA
Figure 7: Comparison of the policy of DeepAA and some publicly available augmentaiotn policy found by other methods including AA, FastAA and DADA on CIFAR-10. Since the compared methods have varied numbers of augmentation layers, we cumulate the probability of each operation over all the augmentation layers. Thus, the cumulative probability can be larger than I. For AA, Fast AA and DADA, we add additional 1.0 probability to flip, Cutout and Crop, since they are applied by default. In addition, we normalize the magnitude to the range [-5, 5], and use color to distinguish different magnitudes.
16


Reviews:
[{"review_id": "S65SBm1YxLd", "reviewer": "Reviewer_VFdc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n- The proposed method reduces the searching time for multi-layered augmentation policy by a large margin while achieving good performance.\n- It is interesting to see that the latter augmentation layers converge to identify mapping (Fig3), which is a good indicator for the optimal number of augmentation layers.\n\n\n## Weakness\n- It is not well justified in the experiment section why formulating data augmentation search as a gradient matching problem is favorable. Even though the authors provide the similarity scores after stacking more augmentation layers in Tab6, the comparison of the cosine similarity of gradient between the proposed method and other augmentation methods is missing. Furthermore, the reason that higher cosine similarity of gradients leads to better final performance is missing. There exist a correlation between these two but the reviewer expect more analysis and insights for the causality.\n- It is unclear whether the proposed method generalized to other tasks (eg. object detection)", "summary_of_the_paper": "This paper proposed a learned data augmentation method. The proposed method is able to search for the augmentation policy with multiple layers by searching the next transformation operation conditioned on all previous augmentations. The authors also proposed to use cosine similarity between the gradients of the original and augmented data as guidance rather than e.g. the more commonly used classification validation loss.", "summary_of_the_review": "One major weakness of this work is that the main idea of using cosine similarity of gradients as guidance for policy search is not well justified in the experiment section. Except for that, based on the experimental results that (1) the proposed method reduces the policy search time by a large margin while maintaining good performance compared to other SoTA augmentation methods, and that (2) the proposed method is applicable to multi-layer augmentation, overall the reviewer considers it a good paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LxUt51v4Ew0", "reviewer": "Reviewer_9qhP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n+ Relaxing the strong human assumption on data augmentation for specific data is novel and well motivated. \n+ The proposal of using cosine similarity between the augmented training data and validation data as a reward signal to optimize the sampling probability of the corresponding data augmentation transformations is technically sound. \n+ Table 1, 2 shows its better performance than SOTA DAS methods. Table 4, 5 further shows its increasing performance given an increasing number of layers and its stable performance given multiple runs. \n+ Figure 3 shows in the top (deep) layers the augmentations are not dataset-specific and therefore justify its motivation. \n\nWeakness: \n- Isn’t the optimal similarity between train/validation to transform validation data to training data? Could the author[s] provide some intuition on why using gradient matching for the search. If it will cause overfitting issues on training data? \n- Fig.3, 4 and 5 show different operation selections at different stages for DeepAA. Could the author[s] show the difference between previous works? \n- Since this work is to advocate less human-constraints on operator selections, it would make this work better if the author[s] can show experiments on data-diversity cases where there is no prior knowledge about the data transformation. Showing extra results on some transfer learning experiments on different datasets will be highly appreciated.", "summary_of_the_paper": "This paper provides an interesting finding that instead of injecting experts prior to hand-pick augmentations at the top layers, the entire search can be formulated end-to-end in a multi-layer (deep) way without hand-picked transformation optimized with gradient matching. The independent sampling at each layer with estimated mean gradient of training data is used to stabilize the training and make the optimization process more efficient. Empirical results on CIFAR-10/100 and ImageNet outperform previous best performing DAS methods.", "summary_of_the_review": "Nice motivation and an interesting approach. Results on standard benchmarks are strong compared to other DAS methods. Experiments can be further expanded to different datasets to show the effectiveness of the proposed method. Can also add more comparisons to the previous methods in Fig. 3/4/5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K1DMElV5LmC", "reviewer": "Reviewer_BGUU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Getting rid of default augmentations is an interesting step\n- Approach to get around the need of RL in previous work\n    - Problem: Little misalignment with final objective, 1 sample vs full mean\n- Nice visualization of the policy choice of DeepAA\n- Repeated experiments with confidence intervals\n\nNot Clear\n- Do you optimize all augmentation layers at once or in separate optimization steps?\n\nWeaknesses\n- The statement about SOTA in the abstract is _not true_. The SOTA for C-10 with WRN-28-10 is at least 84.33 as reported in TrivialAugment, which is stronger than the 84.02 of this method. Similarly, TrivialAugment is forgotten in the comparison for ImageNet, even though here Deep AA is stronger but not as much as stated.\n- I believe stating to use 139 augmentations is a little misleading, as you treat the same augmentation with different strengths as different augmentations\n- Another experiment on the relatively standard SVHN datasets would be interesting, as this has very different images.\n- Claim of not using default augmentations is weakened by the fact that the search space is changed for datasets that require different default augmentations\n- Comparison to TrivialAugment is a little unfair, as TrivialAugment’s standard setting is stronger than the results presented, and TA actually includes a run with CIFAR-100 and ShakeShake\n- Missing true baseline. I would be more sure of the method's performance if there was a simple baseline implemented on the same search space and in your codebase, either UA or TA: This could actually work out very well for you, too, if you include another baseline of a trivial baseline without standard augmentations, too.\n- A baseline for Table 6 would be good to answer whether the cosine similarity increase with any augmentation application? That could be done by applying RA with different numbers of layers or applying TA multiple times.\n\nVery minor: there are some typos here and there.\n", "summary_of_the_paper": "The paper proposes a novel automatic augmentation method. That is a method that learns based on a dataset, how to augment that dataset during training of a classifier.\n\nThe proposed method works by matching gradients based on cosine similarity on fully trained networks and uses this approach to build augmentation policies with multiple augmentations applied after one another. Similar to the recent usage of very deep RandAugment for EfficientNetV2s for example.", "summary_of_the_review": "Overall, I believe the proposed method is interesting: it builds interesting policies in an intuitive way and requires less per-dataset adaptions than previous work.\n\nI do believe, though, that the results simply are not strong enough for publication and a few comparisons in the paper are unfair to baselines. It might be possible to improve upon this method in some ways though, for example with Batch Augmentation (\"Augment Your Batch:..\" by Hoffer et al.), as this is closer to \\theta’s objective.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1hYDEKUz4CU", "reviewer": "Reviewer_3j37", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n- **P1.** **[more automated]** This work takes a further step towards automated machine learning by absorbing some default image transformations (flipping and cropping) into the operation set. The idea is natural, reasonable, and worth exploring.\n\n- **P2.** **[relatively novel; enlightening]** The formulation of gradient matching is radically different from previous bilevel-optimization based works. Though it is not a new idea to perform gradient matching in meta-learning, this opens the avenue for automated data augmentation search, and could potentially be useful for other scenarios like self-supervised learning.\n\n\n**Cons:**\n\n- **C1.** **[missing details on other default operations]** Currently there are only two default operations (flipping and cropping) being searched. But there are also some other defaults like pixel value scaling (normalization) and color jittering. Are they still used by default in DeepAA? Or are there maybe some existing experimental results that include these two (norm and colorjitter) default augmentations?\n\n- **C2.** **[correctness of statements]** A few statements are not well-supported, or require small changes to be made correct: The authors claim that DeepAA outperforms existing automated data augmentation methods, but it seems that some recent work is missing [1, 2]. The authors should add the comparisons to the paper for correctness. Though this might make the performances of DeepAA look weaker, it wouldn’t devalue the novelty and contributions.\n\n- **C3.** **[reliability of new metric]** This work uses a new metric rather than the validation accuracy for augmentation search. However, the reliability of this metric is not well assessed. A straightforward way is to analyze the *correlation* between gradient similarities and final performances, *e.g.*, to evaluate different augmentation policies and to see whether the policies with higher gradient similarities would have better performances.\n\n- **C4.** **[unclear experimental details]** I got confused with some details and have some questions regarding them:\n  \n  - **C4.1.** In Sec. 4, the paper introduces that sub-datasets are used to train the networks. Does this mean that the CNN networks are pretrained on these sub-datasets before conducting the augmentation search (like [3])? If this is true, why not to use the full set since the pretraining seems not so expensive?\n  \n  - **C4.2.** In Sec. 4.1, the paper indicates the policy is updated for 512 iterations. Does this mean that only 512 images are used to search augmentation policies? Since Eq. (12) shows only one single data item is used for each policy update. If it's true, why not to use more training samples to further boost the policy? It would be insightful if a discussion is included on the choice of this hyper-parameter.\n\n**Minor Problems:**\n\n- [Sec. 1.] \"an effective *metrics*\" -- \"*metric*\"\n- [Sec. 3.1.] \"where the augmentation policies *is* shallow\" -- \"*are* shallow\".\n- [Sec. 3.3.3.] \"transformations that *performs* good\" -- \"*perform*\"\n- Figure 2. and 3. are not vectorized and look blurry. Using vector graphics (like Figure 1.) could be better.\n\n---------------------------\n\n[1] Zhang, Xinyu, et al. \"Adversarial AutoAugment.\" International Conference on Learning Representations. 2019.\n\n[2] Tian, Keyu, et al. \"Improving Auto-Augment via Augmentation-Wise Weight Sharing.\" Advances in Neural Information Processing Systems 33 (2020).\n\n[3] Lim, Sungbin, et al. \"Fast autoaugment.\" Advances in Neural Information Processing Systems 32 (2019): 6665-6675.\n\n\n\n\n", "summary_of_the_paper": "A more \"automated\" augmentation search method is proposed. The main contributions are:\n\n- Default transformations like flipping and cropping are regarded as new augmentation candidates which could be optimized, making the algorithm more automated.\n- The gradient matching formulation is borrowed and seems to be effective for augmentation search.\n- MC sampling and a progressive search pipeline are used to reduce the computational complexity.", "summary_of_the_review": "I tend to give a rejection for now. This work is well motivated and has a natural goal. The formulation of gradient matching brings something new to this area. However, there are some flaws like claims with minor issues (**C2**) and missing details (**C1, C4**). Some claims needs to be updated and some details need further explanations. Furthermore, I have some questions about the reliability (**C3**) which may affect my recommendation as well. If all of these issues are solved, I'll raise my score.\n\n--------\n*After rebuttal: my main concerns are well addressed, hence I recommend to accept this work. Besides, the manuscript can be improved by providing more discussion and analysis. See the response to authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: We appreciate the authors for engaging in discussions with the reviewers and providing further experimental results to clarify and address the concerns raised by them in their original reviews, leading to changes in some of the recommendations.

While the (revised) paper with the clarifications and new results incorporated are more ready for publication, some outstanding concerns should preferably be addressed to further enhance its quality.

The authors are highly recommended to take into consideration all the comments and suggestions of the reviewers to further revise their paper to make it a scholarly work to contribute to the ICLR and the more general ML community.
Decision: Accept (Poster)