AdaAug: Learning Class- and Instance-adaptive Data Augmentation Policies
Authors: Tsz-Him Cheung
Dit-Yan Yeung
Venue: ICLR 2022 Poster (2022)
Keywords: Data Augmentation
Automated Data Augmentation

Abstract: Data augmentation is an effective way to improve the generalization capability of modern deep learning models. However, the underlying augmentation methods mostly rely on handcrafted operations. Moreover, an augmentation policy useful to one dataset may not transfer well to other datasets. Therefore, Automated Data Augmentation (AutoDA) methods, like \textit{AutoAugment} and \textit{Population-based Augmentation}, have been proposed recently to automate the process of searching for optimal augmentation policies. However, the augmentation policies found are not adaptive to the dataset used, hindering the effectiveness of these AutoDA methods. In this paper, we propose a novel AutoDA method called \texttt{AdaAug} to efficiently learn adaptive augmentation policies in a class-dependent and potentially instance-dependent manner. Our experiments show that the adaptive augmentation policies learned by our method transfer well to unseen datasets such as the Oxford Flowers, Oxford-IIT Pets, FGVC Aircraft, and Stanford Cars datasets when compared with other AutoDA baselines. In addition, our method also achieves state-of-the-art performance on the CIFAR-10, CIFAR-100, and SVHN datasets.

Published as a conference paper at ICLR 2022
ADAAUG: LEARNING CLASS- AND INSTANCE- ADAPTIVE DATA AUGMENTATION POLICIES
Tsz-Him Cheung & Dit Yeung
Department of Computer Science and Engineering The Hong Kong University of Science and Technology (thcheungae, dyyeung) e ccs.. www.e.s.s.t..... .ust.hk hk
ABSTRACT
Data augmentation is effective way improve the generalization capability of modern deep learning models. However, the underlying augmentation methods mostly rely handcrafted operations. Moreover, augmentation policy use- ful to one dataset may not transfer well to other datasets. Therefore, Automated Data Augmentation (AutoDA) methods, like AutoAugment and Population- Augmentation, have been proposed recently to automate the process of search- ing for optimal augmentation policies. However, the augmentation policies found are not adaptive to the dataset used, hindering the effectiveness of these AutoDA methods. In this paper, we propose a novel AutoDA method called AdaAug to ef- ficiently learn adaptive augmentation policies in a class-dependent and potentially instance-dependent manner. Our experiments show that the adaptive augmenta- tion policies learned by our method transfer well to unseen datasets such the Ox- ford Flowers, Oxford-IIT Pets, FGVC Aircraft, and Stanford Cars datasets when compared with other AutoDA baselines. In addition, our method also achieves a state-of- ff-th-eart art performance on the CIFAR-10 10, CIFAR- 100, and SVHN datasets
INTRODUCTION
Data augmentation is a common way to enhance the robustness of deep learning models by aug- menting the datasets used for model training. Applying popular data augmentation operations such as randomized cropping, horizontal flipping, and color shifting to image data has become a standard procedure in modern image recognition models (Krizhevsky al., 2012, Shorten & Khoshgoftaar 2019). Over the years, various augmentation methods using advanced operations have been pro- posed. Examples include occlusion-base operations like Cutout (Devries Taylor) 2017) that ran- domly masks part of an image avoid overfitting, label-mixing operations like CutMix Yun al. 2019) that replaces the occluded part in Cutout with a different image patch, and Mixup (Zhang al. 2018) that interpolates two images with their corresponding one-hot encoded labels.
While these hand-crafted data augmentation methods can improve model generalization, choosing the operations and their corresponding parameters is often decided manually to make the augmen- tation scheme effective for the task at hand. Despite the manual efforts involved, an augmentation policy that is useful for particular dataset does not often generalize well to other datasets (Cubuk et al., 2019). To tackle this problem, a series of recent studies has been conducted to automate the process of finding an effective data augmentation policy for a target dataset. These automated data augmentation (AutoDA) methodologies show impressive results on several benchmark image datasets (Cubuk et al.
2019
Ho al. 2019 Lim al., 2019 Cubuk et al., 2020) HHatay et al.) 2020 al. 2020 Cheung & Yeung 2021).
2020 Hendrycks et al.
The manual tuning of augmentation parameters can also be addressed by using generative ap- proaches, such as training generative adversarial network (GAN) to create new artificial images directly (Antoniou al. 2017 Tran et al., 2017; Jha & Cecotti 2020) Yorioka et al., 2020 Zhao et al. 2020). However, these generative models are often hard to implement and computationally expensive to train.
Coode available at

---

Published as a conference paper at ICLR 2022
In supervised learning, data augmentation is considered to be naive way to inject inductive biases, such as translational invariance, to a classifier. With the recent advances in representation learning, data augmentation has become major approach to the learning of good representations. For exam- ple, there is an increasing trend to replace convolutional neural networks (CNNs) with transformers in computer vision (Carion et al. 2020 Dosovitskiy al., 2021). Being more generic architecture, transformers do not come with additional inductive bias such as translational invariance in CNNs, thereby requiring more data to learn the effective invariance properties. Data augmentation serves as an effective way to achieve this goal (Touvron et al. 2020). In addition, recent self- self-supervised models rely heavily on data augmentation create different views of the same data and learn the robust representations through contrastive learning (Chen et al. 2020 Grill et al., 2020; He et al.) 2020). However, an improper choice of augmentation operations, especially with excessive strength, may impose wrong inductive bias to the models and lead to performance degradation (Chen et al.) 2020 Xiao ct al, 202I). Consequently, there is a need to enrich the current data augmentation meth- ods, especially with better choice of the data augmentation policy, to take advantage of the recent advances in representation learning.
Previous AutoDA methods attempt to find an optimal augmentation policy augment given dataset. However, most of the discovered policies are not adaptive to variations of the dataset. Even in a dataset in which an augmentation policy is found to be effective, the same augmentation scheme is applied equally to all classes. This can limit the potential of data diversity brought about by data augmentation. For example, in digit classification, flip-invariance is useful for the digits "0", "I" and "8" but not for the other digits; in shape classification, thearing-invariane is useful for "triangles" but not for "rectangles". None of the previous AutoDA methods can learn an adaptive deas-deepndent augmentation policy. To address this limitation, we propose AdaAug, as an Au- toDA method, to learn a class- desendennndd and potentially instance-dependent augmentation policy efficiently.
Despite the attractive potential of such an adaptive scheme if it can indeed be realized, actually learn- ing an adaptive augmentation policy poses at least two major technical challenges. First, the search space for the per-class, per-instance augmentation policies is very large, rendering it intractable when trying to maintain an individual policy for each class even each realization of the input data. Second, the gradient information of the augmentation parameters is hard to obtain as the operation selection process and the transformations icy efficiently a challenging problem to address. In this work, AdaAug employs a recognition model to learn the underlying augmentation policy for each data instance and takes an alternating procedure to update the augmentation policy using a differentiable workflow. In the exploitation pass, AdaAug trains a classifier for a number of steps, followed by the exploration pass which validates the classifier and updates the policy to minimize the validation loss. The intu- ition behind our design of AdaAug is that such an alternating procedure can learn augmentations that help generalize the trained model to unseen validation data. For example, rotational invariance would be found to be a desirable inductive bias that the model should learn if the validation set contains some similar but rotated versions of the training images. Our goal is to capture such infor- mation and assign a higher probability to use rotation for augmentation in this case. An application scenario would be a computer vision task for drones where the unseen data may contain different kinds of rotated images. To summarize, our contributions are listed as follows:
oonddffferent.. Optimizing the augmentation pol-
We introduce novel AutoDA method to learn a class-dependent and potentially instance- dependent augmentation policy for each data instance.
We propose a differentiable workflow to search for the augmentation policy efficiently.
We demonstrate that the policies learned by our method transfer better to unseen datasets, such as Oxford Flowers, Oxford-IIT Pets. FGVC Aireraft, and Stanford Cars, when com- pared to other AutoDA baselines.
We demonstrate a state-of-the performance on the 10, CIFAR- 100, and SVHN datasets.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Automated Data Augmentation. Several AutoDA methods have been proposed to compose the augmentation operations automatically. AutoAAument (Cubuk al. 2019) learns to generate the probability and magnitude when applying different augmentation operations as a policy using re- inforcement learning (RL). It alternately generates an augmentation policy to train a child model and updates the policy generator using the validation performance as a reward. Since it is com- putationally expensive train the child models repeatedly, several techniques have been proposed subsequently in an attempt to reduce the search effort. Fast AutoAugment (Lim et al.) 2019) uses Augmentation Bayesian optimization (BO) to tune the augmentation parameters. Population-bass
(PBA) (Ho et al. 2019) exploits population- based training (PBT) to search for an optimal augmen- tation policy schedule by training multiple parallel child models using an evolutionary approach. RandAugment (Cubuk et al. 2020) applies the augmentation operations uniformly and reduces the search space significantly by covering only the number of operators and the global augmentation magnitude. Instead of using the validation performance evaluate the augmentation quality, Ad- versarial AutoAugment (Zhang et al. (020)) uses an adversarial objective to learn the augmentation policy. MODALS (Cheung & Yeung 2021) utilizes PBA search for optimal latent space aug- mentation policy and augment data from any modality not limited to image data.
Differentiable Data Augmentation. In addition to using RL. BO and PBT optimize the augmen- tation parameters, there exist related methods that modify the otherwise discrete search procedures make them end-to-end differentiable. This results in more efficient optimization procedure and a more precise policy than RL and PBT as the search space is continuous. In AdaAug, part of our contribution to design differentiable workflow to learn the augmentation policy, For pre- vious differentiable augmentation approaches, Faster AutoAugment (Hataya et al. 2020) proposes a differentiable Relaxed Bernoulli distribution to sample the candidate augmentation functions and estimate the gradients of the non-differentiable augmentation magnitude using the Stop Gradient es- timator. Specifically, it optimizes a density matching loss between the training and validation data. DADA (Li et al. 2020) differentiates through the discrete policy sampling process using Gumbel- Softmax trick. While AutoDA and differentiable data augmentation have been shown to be success- ful in improving the generalization performance of deep learning models, the learned augmentation policy is often applied uniformly to the whole dataset, meaning that all classes and instances share the same augmentation policy. On the contrary, each class and even each data instance receives an adaptive policy to augment in our proposed method.
Adaptive Data Augmentation. Attempts have been made to apply adaptive data augmentation at class or subgroup level. [Hauberg et al. (2016) model the transformations within each class and use statistical models to augment the dataset. The approach shows improvement in the small MNIST dataset and its variants. However, the augmentation operations are limited to spatial transforma- tions. In addition, observations of the data must be locatable and alignable, making it difficult to extend to most other computer vision tasks. Recently, CAMEL (Goel et al., (021) adopts finer data- data-generation method by fixing the classifiers that fail on a subgroup of class. It uses CycleGAN to learn different variations of the same training data within a subgroup. However, CAMEL re- quires the specifying of the subgroup information manually and assumes that the subgroups only exist within the same class. MetaAugment (Zhou et al., (02T) learns sample-wise weighting scheme and a global probability parameter to control the sampling of augmentation transformations. In AdaAug, the learned policy can capture class deasndeennddnt transformation automatically and also instance-dependent information, such as the light intensity of an image, across different classes.
3 ADAAUG
3.1
SEARCH SPACE
Let T be set of augmentation operations where T, denotes the j-th operation (c.g., "rotation") in the set. We formulate an augmentation policy as the probability and magnitude À in applying the augmentation operations. Here, p is probability vector with each entry P3 [o. 1]: 11 Pj 1 and À is vector with each entry X; E [0, 1]. where pj and Xy are the probability and magnitude, respectively, of applying the operation Ty- Mathematicallyy Tj X' + X is a mapping from the input space X' to itself. For an image x É X, Tj transforms with the magnitude parameter X, that
3

---

Published as a conference paper at ICLR 2022
Friees
AV
traiiing fio
Ap + 6o
Valicuter
Figure I: Illustration of the AdaAug pipeline. In the exploitation pass (solid line), the training im- ages are augmented by the policy network (h., (x)) as new unseen data train the classifier (colored in blue). In the exploration pass (dotted line), each validation data instance passes through multiple augmentation paths. The corresponding latent representations are summed with the gen- erated augmentation probabilities as weights. The policy projection network (colored in green) is updated to minimize the validation loss. compute the probability and magnitude vectors, a and stn denote the sigmoid and softmax functions, respectively.
specifies the strength of the transformation (e.g., degree of rotation), i.c., x + (2; Xy). Note that some operations like flipping do not depend the magnitude parameter. In training pass, given an input data æ, an augmentation policy (p, A), and the number of operators k, we sample operations according p and apply them with their corresponding magnitudes specified by x:
T(::P,A) = T $ = o . T (x:p,A)
Here, T(*), S t I k. denotes applying the t-th operation. Our goal is to learn an augmentation policy function To (p, A) generate adaptive augmentation policy that optimizes the generalization performance and is dependent on the input data (see Figure] and Algorithm I).
3.2 SEARCH ALGORITHM
Exploitation. AdaAug uses a feature extraction network fa : X' + to map an input space to latent space, dense layer 93 Z + y to map latent space label space, and projection function hy + P x to map latent space (representation) to a probability and magnitude space, where = [0. 1 and |||| = 1, vp E P and X = [o. 1] FI, AA E A. In our case, the functions f.g, (with the subscripts dropped for notational simplicity) are implemented as neural networks with network weights a, B, respectively. The softmax operation is applied to the first half of the output from & to get the probabilities and the sigmoid function is applied the other half to get the magnitudes. The policy network Ta (z) can then connect the class and image information the augmentation policy via f(x) with parameters 0 a). an exploitation pass, given the training data x, the policy function generates the data-dependent augmentation policy (p, A) (xxx)) and augments to give using Equation (1). Here, å is treated as new unseen training example and is used to train the classification model f(ă) by minimizing the cross- entropy loss: min Ctrain (a, B). Note that there is a discrete sampling procedure in Equation (I). During the update, the gradient of does not involve the computations in the policy network.
Exploration. the exploration pass, AdaAug generates the augmentation policy (p, = (x) given the validation data The [T] operations applied to separately with the corresponding magnitudes in The augmented validation data are passed to the feature extraction network f individually to get the latent representations, which are then summed based their weights in the
4

---

Published as a conference paper at ICLR 2022
probability vector p. The mixed representation is passed to g for computing the predicted labels:
; /
T g = Σ Ps fo (o))))
(p, >) - TT(x)
(2)
AdaAug updates y minimize the validation loss: min, Cvalid(a, 3, 7). As we make no assump- tion that the augmentation operations are differentiable, we follow prior approaches (Bengio et al., 2013 Li et al., 2020) using straight-through gradient estimator to optimize the augmentation mag- nitudes. Specifically, the gradient of the magnitudes is estimated with respect to each input pixel
value 2/, of the augmented data, i.e., 0b;
8,,,= 1.
The gradient then be calculated by:
Daraisd 0i..h
Σ
Σ wih
(3)
Dew, Qxi
w.h
Algorithm Search algorithm
1: procedure SEARCU((D (rrani DaiiinDDaai Doatidy T, k, m, r)
2: Initialize 3,
3: for r steps do # Exploitation
4: Sample mini-batch divain E Drrain
5: for (x, y) E derain do
6:
p. X) = (2)
7:
f augment(x)
augment computes Eg.
8:
ÿ = g o f f())
9:
Strain (a, B) = CrossEntropyLosse (a, 3) f- arg min,,8 ferain (a, 3)
11:
if r is divisible by m then # Exploration Sample mini-batch dealid Dralid
13:
for (x, y) E deatid do
14:
(p, A) = (T(r)
15:
g explore(x)
explore computes Eq.
16:
Lvalid B, CrossEntropyLoss((.. 7 arg min, Lvatidla, B,
17:
return a,y
Relation to Neural Architecture Search (NAS). Tuning of the augmentation policy parameters bears some similarity to the optimization the network architecture weights DARTS from the NAS literature (Liu et al. 2019). DARTS prescribes different computation paths to different opera- tion cells and relaxes the computation to a mixture of the operations weighted by learnable weights for each path. With as the model parameters and the weights for the computation paths, the search algorithm solves a bi-level optimization problem:
min Leatia w" (a),
w² (a) arg min Livain (w., (w,a) 0:)
While DARTS optimizes the architecture weights, AdaAug optimizes the projection parameter that decides the augmentation weights and magnitudes the exploration. DARTS solves the opti- mization by first-order and finite-difference approximation of the architecture gradient. AdaAug samples the augmentation operations and treats the augmented data missing training data X Xtrain- By absorbing the augmentation parameter in the training dataset, we avoid the complex bi-level optimization and simplify the exploitation procedure into training standard classifier:
min Lvatialo" 8 7i X Xaatid)
a" "" arg min Ltrain (a, (a,33 8; Xtrain) a.3
Relation to Density Matching. Data augmentation can regarded as a density matching problem
between the training and validation data (Ratner et al. 2017 Tran et al. 2017 Hataya et al. 2020
Lim et al. 2019). From this perspective, AdaAug improves model generalization by matching the density of Dirain with the density of the augmented Dootid. In the outer optimization objective in Equation (5) Adaaug minimizes the classification loss with the augmentation parameters over the same optimal model parameters a", 8 learned from Derain. so doing, approximately reduces the distance between the densities of the augmented Dvatid and D Drrain-
5

---

Published as a conference paper at ICLR 2022
3.3 INFERENCE
Like most automated augmentation pipelines (Cubuk et al. 2019 Ho et al. 2019 Lim et al. 2019 Hataya et al. 2020 Li et al. 2020). AdaAug searches for the augmentation policy on a small dataset using small model and then applies the learned policy network xo to train with larger dataset or model. We call the process of applying a searched policy to augment a new dataset as inference time. Regarding this workflow, RandAugment argues that the different search spaces in search time and inference time makes the augmentation policy unable to adjust the regularization strength to different target datasets and models (Cubuk et al. 2020). Therefore, it proposes to search for the global magnitude and number of operators for each case. To address this concern, AdaAug utilizes three diversity parameters to fine-tune the regularization strength of the found policy for large datasets and models. First, the number of operators A: is set to at search time, but a larger value of l can be used at inference time. To control the selection of more diverse operations, a
temperature parameter T is introduced in the softmax function: p; E, exp [M(=;)/T] Setting a
exp [h(2,)/21
larger T value allows the operations with lower probability being sampled more often in the same order. Last, AdaAug perturbs the magnitude value with the parameter ó so the perturbed magnitude is given as Ā ~ Uniform(> (À- - 6, À + 5). This allows the augmented data to show slight variations even with the same operation on the same input data. In practice, we can perform grid search on these three diversity parameters with the other hyperparameters using holdout validation set.
4 EXPERIMENTS AND RESULTS
In this section, we explain experimental design followed by presenting the results. We evaluate the empirical performance of AdaAug in two experiments: AdaAug-trr and AdaAug-direct. We select the comparison baselines that use the validation performance the evaluation method to learn the augmentation policy. For all the experiments, we report the average test-set error rate as the performance metric. Each model evaluated three times with different random initializations.
Augmentation operations. We match the operations adopted by AutoAugment. In addition to the 16 operations proposed previously (ShearX, ShearY, TranslateX, Translate Y, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Cutout, and Sample Pairing), we add the Identity operation for not applying data augmentation. For the simple baseline, we apply random horizontal flip, color jittering, color normalization, and Cutout with a 16 x 16 patch size. Our method and other baselines apply the found policy on top of these standard augmentations.
Policy search. We follow the setup adopted by AutoAuument (Cubuk et al., 2019) to use 4,000 training images for CIFAR- 10 and CIFAR- 100, and 1,000 training images for SVHN. The remaining images used as the validation set. We use Wide-ResNet-40-2 (Zagoruyko & Komodakis 2016) as the feature extraction network for all searches. We implement h as a linear layer and update the policy parameter y after every 10 training steps using the Adam optimizer with learning rate of 0.001 and a batch size of 128.
AdaAAug-tansfer In the first experiment, investigate how well the learned augmentation policy can transfer to unseen datasets. We search for the optimal augmentation policy on the CIFAR-100
dataset and use the learned policy to train with four fine-grained classification datasets: Oxford 102 Flowers (Nilsback & Zisserman 20088. Oxford-IIIT Pets (Em et al. 2017). FGVC Aircraft (Maji al. 2013), and Stantord Cars (Krause al. 2013). We compare the test error rate with AutoAug-
ment, Fast Auto/Augmet, DADA, and Augment using their published policies for CIFAR- 100. For all the tested datasets, we compare the transfer results when training the ResNet-50 model (He) et al. 2016) for 180 epochs from scratch and fine-tuning the ResNet-50 model pretrained on Ima- geNet for 100 epochs. We use the cosine learning rate decay with one annealing cycle (Loshchilov & Hutter 2017), initial learning rate of 0. weight decay le-4 and gradient clipping parameter 5.
AdaAug-direct. In the second experiment, we search for the optimal augmentation policy on a small subset of the target dataset and use the learned policy to train the full dataset. The purpose of the experiment is to demonstrate that while the AdaAug policy can adapt to other unseen datasets, it can also achieve competitive performances AdaAug- direct with state-of-the -fhte---rrtt-ttttt-e AutoDA methods using the same evaluation datasets: CIFAR- 10, CIFAR-100 (Krizhevsky & Hinton 2009). and SVHN (Netzer al. 2011). We test our method
the seen datasets with more training data. We compare

---

Published as a conference paper at ICLR 2022
Table I: Test error (%) when training ResNet-50 with different augmentation methods.
Dataset
Train Size
Classes
Simple
AA
Fast AA
DADA
RA
AdaAug
Oxford 102 Flowers
6,552
102
5.87
3.71
3,99
5.06
4.28
3.63 1 0.14
Oxford-IIIT Pets
3,680
37
26.42
22.61
22.11
23.80
27.84
20.14 10112
FGVC Aireraft
6,667
100
19.87
17.72
17.80
18.84
17.70
17.50 0.19
Stanford Cars
8,144
196
13.64
11.96
12.81
12.86
12.21
11.51 0.21
Table 2: Test error (%) when fine-tuning pretrained ResNet-50 with different augmentation methods.
Dataset
Simple
AA
RA
Fast AA
AdaAug
Oxford 102 Flowers
5.02
6.12
4.77
3,92
2.81 + 0.13
Oxford-IIIT Pets
19.47
18.75
23.03
16.79
16.14 + 0.07
FGVC Aircraft
18.40
16.61
17.02
17.44
16.03 + 0.15
Stanford Cars
11.92
9.18
10.72
10.29
8.82 + 0.08
using Nedt-ResNeet40-2 and Wide-ResNet-28- At inference time, we set the temperature 7 = and magnitude perturbation = 0.3 and search for the number of operators (1, 4) using holdout validation set, like RandAugment (Cubuk al.) 2020). For the hyperparameters, we follow AutoAugment, PBA, and Fast AutoAugment if possible.
4.1 ADAAUG-TRANSEER
Table and Table 2 show that the AdaAug policy outperforms the other baselines when training and tinettunig the ResNet-50 model on the Flowers, Pets, Aircraft, and Cars datasets, respectively. These baselines apply the same augmentation policy to all datasets. Such a policy may not be optimal to the target domain. In contrast, AdaAug adapts the augmentation policy to individual image classes and instances automatically. The AdaAug policy network applies different augmentation policies to unseen images according to their similarity to the classes that AdaAug has seen in the search. To further justify our claim, show the distribution of the augmentation parameters for different tasks in Appendix AA.544 The four datasets have many classes but few training examples per class. The negative effect of using a non-adaptive data augmentation policy is likely be more imminent than a situation where its dataset has fewer classes but more examples per class.
4.2 ADAAUG-DIRECT
CIFAR-10 10 and CIFAR-100. The policies learned by AdaAug mostly achieve either comparable or better performances than the baselines for both WRN-40-2 and WRN-28-10 models on the CIFAR- 10 and CIFAR- 100 datasets (see Table 3 We visualize the augmentation policy of CIFAR-10 10 by applying the policy to the validation data and averaging the predicted augmentation probability of each image for each class (see Figure 2) The policy contains all types of augmentations some degree. Among the operations, Flip dominates the policy. is in line with the augmentation se- lected manually as people find applying horizontal flipping to CIFAR- 10 can improve the prediction accuracy. The policy learned by AdaAug puts higher emphasis on Invert and Equalize, which are also reported by PBA. The focus Brightness also aligned with the policy in AutoAugment, Al- though there are minor variations in the importance of some operations between the policies learned by AdaAug, AutoAugment, and PBA, their empirical performance similar.
SVHN. AdaAug performs comparably to the baselines on the core set of SVHN. We visualize the augmentation policy for SVHN Figure 2 We found that AutoContrast, Invert, and Solarize receive much attention in SVHN. This makes sense because the specific color of the number and background irrelevant to the prediction. This is consistent with the findings by AutoAugment and PBA. The Flip operation receives significantly higher probability in digits "0", "1", and "8". It is likely because these three digits appear similar after flipping. This shows that AdaAug captures not only dataset-specific augmentation policy, but also class-dependent augmentation, which cannot be achieved by other basclines. In addition to the augmentation probability, we visualize the learned augmentation magnitude in Appendix A.5.2

---

Published as a conference paper at ICLR 2022
Table 3: Test error (%) on Reduced CIFAR-IO, CIFAR-IO, CIFAR-I00, reduced SVHN and the SVHN core set ("evaluated using the policy released by the baseline).
Simple
AA
PBA
Fast AA
DADA
Faster AA
RA
AdaAug
Reduced CIEAR-10
WRN-28-10
17.1
14.1
12.8
14.6*
15.6
15.1"
13.6 1 0.03
CIFAR-10
WRN-40-2
5.3
3.7
3.9"
3.6
3.6
3.7
4.1"
3.6 1: 0.11
WRN-28-10
3.9
2.6
2.6
2.7
2.7
2.6
2.7
2.6 1: 0.10
CIFAR-100
WRN-40-2
26.0
20.6
22.3*
20.7
20.9
21.4
19.8 + 0,08
WRN-28-10
18.8
17.1
16.7
17.3
17.5
17.3
16.7
17.1 + 0.12
Reduced SVHN WRN-28-10
13.2
8.2
7.8
8.1"
7.6
9.4"
8.2 + 0.12
SVHN (core set) WRN-28-10
3.1
1.9
2.1
2.3"
2.4
1.7
2.3 + 0.13
Iraetty
044
3.4
30
3.0
Figure 2: Illustration of the learned augmentation probability for CIFAR-10 (left) and SVHN (right).
ImageNet. We also validate method large-scale ImageNet dataset. AdaAug improves the top-1 accuracy 1% over the ResNet-50 baseline (see Appendix [A.1I]]. Although some baselines like AutoAugment produce similar performance slightly outperform our method in the AdaAug- direct experiment, AdaAug uses less computational effort in searching for the policy. Specifically, AutoAugment takes 5,000 GPU hours search for the CIFAR-10 policy, while AdaAug takes only 3.3 GPU hours on an old GeForce GTX 1080 GPU card (see Appendix A.4
DISCUSSION
Instance-dependdet augmentation. The AdaAug -direct results show that the learned AdaAug pol- icy can capture dataset- and class-dependent augmentations on CIFAR-10, 10, CIFAR-100, and SVHN. We further investigate whether AdaAug learn instance-dependent information. According to the architecture, AdaAug takes the output from the last layer of a CNN network as the image rep- resentation and uses it predict the augmentation parameters. The image representation contains the class information and potentially some image features. Empirically, we first examine if AdaAug generates different augmentation policies for different instances even within the same class. In Ap- pendix A.3.3) we plot the standard deviations of the predicted augmentation probabilities of the image instances for each class. It is observed that even within the same class, the predicted aug- mentation policy is slightly different across instances. It is a clue that the AdaAug policy captures some instance-level augmentation information. Qualitatively, we show the augmented examples of flower image using the AdaAug policy in Figure 3 The input image is relatively darker than the other images. Apparently, AdaAug is aware of this property and applies more brightness- augmentation to lighten up the image. We observe similar behaviour of AdaAug in other classes, but there inadequate empirical support to conclude a general rule on how the network decides the instance-an aware augmentation policies are learned in data-driven way.
8

---

Published as a conference paper at ICLR 2022
6=0.1 8 0.5 -
=
1w4
Figure 3: Visualization of the augmented images under different augmentation strengths using: Ran- dAugment (left); and AdaAug (right). (A the number of operators, m the magnitude parameter in RandAuggda and is the magnitude perturbation parameter in AdaAug.)
Quality of augmented data. We compare the augmented images from AdaAug with Rand under different augmentation strengths in Figure B In terms augmentation diversity, RandAug- ment produces more variations of the imput image. However, not all augmented images produced are plausible. As RandAugment applies the augmentations uniformly, some augmented flower images show a strange color difference the original image. With an increasing number of operators and magnitude, the flower object is sometimes translated out of the frame and results in a black image. This may affect the learning performance as color can be an important feature in classifying different species of flowers. For AdaAug, the augmented images are more visually appealing.
Ablation study. In this section, we study the effects of AdaAug-transfer on Oxford 102 Flowers and AdaAug-direct on CIFAR-I4 10 using three alternative search configurations. First, does the use of a nonlinear projection deliver better performance? We replace the linear layer h with 2-layer MLP using a 128 hidden layer size with ReLU activation. Second, we study whether the class-dependent and instance-dependent configuration improves model aceuracy. We remove the class and instance information by replacing the policy network with a fixed vector, which decides the augmentation probabilities and magnitudes for the entire dataset. the third setting, we mix the augmented images in the input space instead of the latent space and observe the effects. Our results show that the use of class- and instance-adaptiv augmentation policies contribute larger improve- ments in performance. A possible reason is that the nonlinear projection is more likely to overfit the search dataset and fail to generalize unseen datasets. Moreover, combining the augmentation path in the latent space learns better policy (see Table
In AdaAug -tansssferrr using nonlinear projection harms the prediction
Table 4: Test error rate (%) on AltAuugtranssfe and AdaAug- direct with different search settings.
AdaAug w/ 2-layer MLP the projection function
AdaAug w/o class, instance information
AdaAug w/ input- space mixing
AdaAug
transfer
3.7
4.5
3,9
4.0
direct
3.6
3.7
4.4
3.9
CONCLUSION
In this work, we propose a novel AutoDA approach, AdaAug, to learn class- and instance-adaptive augmentation policies efficiently. We demonstrate that the found policy transfers well to unseen datasets while achieving state-of-the-art results on the seen datasets. We provide evidence to show that the learned adaptive policy captures class- and instance-level information. We think that AdaAug can show further gains over existing baselines when applied to datasets with more dis- similar underlying augmentation rules among the data classes, and with fewer training examples for each class. It is also promising to investigate in the future if the proposed adaptive augmentation method can improve the performance of other computer vision and representation learning tasks.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This research has been made possible by the Hong Kong PhD Fellowship provided to the first au- thor and research grants (General Research Fund project 16204720 and Collaborative Research Fund project C6030-18GF from the Research Grants Council of Hong Kong; Amazon Web Services Ma- chine Learning Research Award 2020) provided to the second author.
REFERENCES
Antreas Antoniou, Amos J. Storkey, and Harrison Edwards. Data augmentation generative adver- sarial networks. arXiv preprint arXiv: 17X1::711..044444/ 2017.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1oXKvv:308.344 2013.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In 16th European Confer- ence on Computer Vision, ECCV 2020, volume 12346, pp. 213-229. Springer, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 1 119, pp. 1597-1607. PMLR, 2020.
Tsz Him Cheung and Dit Yan Yeung. MODALS: Modality-agnostic automated data augmentation in the latent space. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. AutoAugment: Learning augmentation strategies from data. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pp. 113-123. IEEE, 2019.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data augmentation with reduced search space. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, pp. 3008-3017. IEEE, 2020.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /708. 04552, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Yan Em, Feng Gao, Yihang Lou, Shiqi Wang, Tiejun Huang, and Ling- Yu Duan. Incorporating intra-class variance to fine-grained visual recognition. In 2017 IEEE International Conference on Multimedia and Expo, ICME 2017, pp. 1452-1457. IEEE, 2017.
Karan Goel, Albert Gu. Yixuan Li, and Christopher Re. Model patching: Closing the subgroup performance gap with data augmentation. In 9th International Conference on Learning Repre- sentations, ICLR 2021, 2021.
Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster AutoAugment: learning augmentation strategies using backpropagation. In 16th European Conference on Com- puler Vision, ECCV 2020, volume 12370, pp. 1-16. Springer, 2020.
10

---

Published as a conference paper at ICLR 2022
Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John W. Fisher III, and Lars Kai Hansen. Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, volume 51 of JMLR Workshop and Conference Proceedings, pp. 342-350. JMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pp. 770- 778. IEEE, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2020, pp. 9726-9735. IEEE, 2020.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Picter Abbeel. Population Based Augmentation: efficient learning of augmentation policy schedules. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97, pp. 2731-2744 PMLR, 2019.
Ganesh Jha and Hubert Cecotti. Data augmentation for handwritten digit recognition using genera- tive adversarial networks. Multim. Tools Appl., 79(47):5505-33506 2020.
J. Krause, Jun Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. In Second Workshop on Fine-Grained Visual Categorization, 2013.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Hya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Anmual Conference on Neural Information Processing Systems 2012., pp. 1106-11144 2012.
Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferabil- ity of representations via augmentation-aware self-supervision. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021,
NeurlPS 2021, 2021.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy M. Hospedales, Neil Martin Robertson, and Yongxing Yang. DADA: Differentiable automatic data augmentation. 16th European Conference on Computer Vision, ECCV 2020, 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast AutoAugment, In Advances in Newral Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019. pp. 6662-6672, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5155 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Un-supervised Feature Learning, 2011.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large num- ber of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008 pp. 722-729. IEEF, 2008
11

---

Published as a conference paper at ICLR 2022
Alexander J. Ratner, Henry R. Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learning to compose domain speciiic transformations for data augmentation. In Advances in Neural Information Processing Systems 30: Anmual Conference Neural Information Process- ing Systems 2017, pp- 3236-3246, 2017.
Connor Shorten and Taghi M. Khoshgoftaar. survey on image data augmentation for deep learn- ing. Journal of Big Data, 6:60, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:20/2. /2777, 2020.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle J. Palmer, and lan D. Reid. Bayesian data augmentation approach for learning deep models. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 2797-2806, 2017.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Daiki Yorioka, Hyunho Kang, and Keiichi Iwamura. Data augmentation for deep learning using generative adversarial networks. In IEEE Global Conference on Consumer Electronics, GCCE 2020, pp. 516-518. IEEE, 2020.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE International Conference on Computer Vision, ICCV 2019, pp. 6022-6031. IEFE, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016. BMVA Press, 2016.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond empiri- cal risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial AutoAugment. In 8th Inter- national Conference on Learning Representations, ICLR 2020, 2020.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
Fengwei Zhou, Jiawei Li, Chuanlong Xie, Fei Chen, Lanqing Hong, Rui Sun, and Zhenguo Li. MetaAugment: Sample-aware data augmentation policy learning. In Thirty-Fisth AAAI Confer- ence on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAL, pp. 11097-11105. AAAI Press, 2021.
A APPENDIX
A.1 ADDITIONAL EXPERIMENTS
A.1.1 LARGE-SCALE DATASET
For large-scale dataset, AdaAug improves the top-1 accuracy 1% over the ImageNet ResNet-50 baseline (see T). The performance gain is similar to previous AutoDA methods and validates the positive effect of AdaAug complex dataset.
1.1.2 ARCHITECTURE TRANSFER
We also provide the experimental results when using the learned Augmentation policy to train the Shake-Shake (26 2x96d) model Reduced CIFAR-10 and Reduced SVHN in Table
12

---

Published as a conference paper at ICLR 2022
Table 5: Test error rates (%) on ImageNet using ResNet-50.
Simple
AA
Fast AA
DADA
Faster AA
RA
AdaAug
23.7
22.4
22.4
22.5
23.5
22.4
22.8
Table 6: Test ertor rates (%) of Shake-Shake (26 2x96d) Reduced CIFAR-I0 and reduced SVHN
Simple
AutoAugment
PBA
AdaAug
Reduced CIFAR-10
Shake-Shall (26 2x96d)
17.05
10.04
10.64
10.92 + 0.07
Reduced SVHN
Shake-Shake (26 2x96d)
13.32
5.92
6.46
6.44 +: 0.17
A.2 ADDITIONAL ABLATION STUDY
To clarify the improvements of AdaAug, compare the performance of AdaAug under different settings in T Simple: standard data augmentation is applied; Random: AdaAug is applied with randomly initialized h, while keeping the diversity parameters the same; AdaAug (w/o diversity): AdaAug is applicd without the diversity parameters; AdaAug: AdaAug is applied with learned h, and the diversity parameters.
Table 7: Test error rate (%) on AdaAug-transfer and AdaAug-direct with different augmentation configurations.
Task
Dataset
Simple
Random
AdaAug (w/o diversity)
AdaAug
transfer
Oxford 102 Flowers
5.87
4.65
3.70
3.63 0.14
Oxford-I1IT Pets
26.42
23.31
21.18
20.14 00.1
FGVC Aireraft
19.87
18.70
17.62
17.50 + 0.19
Stanford Cars
13.64
12.27
11.62
11.51 + 0.21
direct
CIFAR-10 -10
5.3
3.8
3.7
3.6 1 0.11
26.0
22.4
20.7
19.8 0.08
A.3 FINE-TUNING THE DIVERSITY PARAMETERS
The following experiments study the sensitivity of the temperature T and magnitude perturbation ó on the Flower dataset by changing the values of T' and around the default values used our original experiments. The fine-tuning of the diversity parameters reduces the test-set error rate of AdaAug-trance transfer on the Flower dataset from 3.63 to 3.49.
A.4 EFFICIENCY OF POLICY SEARCII
We compare the GPU hours needed to search the augmentation policy between different Automated Data Augmentation methods in Table / AutoAugment, PBA and Fast Autment.
Among the baselines, AdaAug is more efficient than
A.5
MORE ANALYSIS ON ADAAUG POLICY LEARNING
A.5.1
CONVERGENCE POLICY LEARNING
Here, we provide some insights and empirical evidences for the convergence policy training. In particular, Figure T shows the training and validation losses when learning the CIFAR-10 augmen- tation policy. The training and validation losses converge to some fixed values towards the end of training. In addition, also visualize the change of the augmentation parameters (p and j)) in Fig- ure () The magnitude parameters start at a smaller value and converge towards the end training. For the augmentation probability, most of the candidates stabilized after certain epochs while
13

---

Published as a conference paper at ICLR 2022
Table Test-set error rate (%) on AdaAug-transtrr on Flower dataset with different values of å and
T.
0.1
0.2
0.3
0.4
0.5
T
0.1
0.2
0.3
0.4
0.5
3.0
3.80
3.81
3.63
3.49
3.67
8
0.3
4.16
3.92
3.63
3.92
4.28
Table 9: GPU hours needed to find the augmentation policy using AA, PBA, Fast AA, DADA, Faster AA, and AdaAug.
AA
PBA
Fast AA
DADA
Faster AA
AdaAug
CIFAR-10
5,000
5
3,5
0.1
0.23 0.061
2.89
SVHN
1,000
1.5
0.1
1.63
some others updated more frequently. These are the evidences that show the convergence of our proposed policy training method. For a more thorough analysis for the policy convergence, we would like to leave it the future work.
1.6
1.2
1.3
0.8
1.1
0.4
0.9
0
120
160
200
40
80
120
160
200
Figure 4: Convergence of the training loss (left) and validation loss (right) during the policy learning on CIFAR-10 dataset.
A.5.2 ANALYSIS OF LEARNED AUGMENTATION MAGNITUDE
Complement with the augmentation probability in Figure siows the augmentation magni- tude for CIFAR- and SVHN. We observe that the policy magnitude also shows slight variations among different classes and instances. Although the observation is less prominent and is harder to interpret when compared to the augmentation probability p. the learned augmentation magnitude adapts to different data samples.
A.5.3 VARIANCE OF LEARNED AUGMENTATION POLICY GROUPED CLASS
we plot the standard deviations of the predicted augmentation probabilities of the image instances grouped by its class label. observed that even within the same class, the predicted augmentation policy is different across instances. It is a clue that the AdaAug policy captures some instance-level augmentation information.
In Figure 7
A.5.4 DISTRIBUTIONS OF THE AUGMENTATION PARAMETERS IN ADAAUG-TRANSTER
In Figure 8 we show the distributions of the augmentation parameters when transferring the learned policy to the Flower, Pet, Car and Aircraft datasets. The distributions of the augmentation pa- rameters show slight differences between different tasks. Apparently, we observe that the colour transformations, for example Colour, Invert and Solarize are less preferred in the Flower and Pet datasets, while shearing operations relatively
favourable in the Car and Aircraft dataset. The observation of using less color transformation for the Flower dataset aligns with the findings from Lee et al. (002)) Although some differences in the augmentation distributions may be less
14

---

Published as a conference paper at ICLR 2022
10
Identity
Sharpness
Brightness
Color
0.6
Contrast
Solarize
Invert
AutoContrast
Rotate
0.2
Translate
Sheary
ShearXX
Figure 5: Update of the augmentation probability (left) and magnitude (right) during the policy learning.
Mentiy
10
Sharpnees
Color
Pistenize
Satarzze
Equarize
Tranclater
Sheart
Q3T
800
Figure 6: Illustration of the learned augmentation magnitudes for CIFAR-10 10 (left) and SVHN (right).
prominent and harder interpret when compared the AdaAug-direct cases, the policy shows its adaptations to different tasks.
A.6 LIMITATIONS
Although AdæAug is differentiable and efficient in search, it requires forming different augmen- tation paths and passing the images through each path. Compared to standard training, where forward images in one mini-batch, AdaAug processes b- [T] images in the exploration pass. At inference time, AdaAug keeps pre-trane rre-trained policy network (Ta) augment the new dataset. This adds extra computational effort when compared to other AutoDA policies, which may be a concern if the image resolution and model size large but the computational resources are limited.
15

---

Published as a conference paper at ICLR 2022
Hentity
Cuostt
Brohhness
Contract
Equasiiee
AdtoCantrant
0.0
Trandatet
0.0
Sheary
000
Figure 7: Illustration of the policy probability variations for CIFAR-10 10 (left) and SVHN (right).
1.0
Shatpmess
Coler
Equalie
Transitoo
Sheart
80
Accot
Rover
Figure 8: Illustration the augmentation probability (left) and magnitude (right) when transferring the learned augmentation policy to the Flower, Pet, Car and Aircraft datasets.
16


Reviews:
[{"review_id": "yKTtaJQN_ZD", "reviewer": "Reviewer_xMkp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros.**\n\n- **Clarity**. Overall, the writing is clear and easy to follow\n- **Well motivated problem and novel approach**. Instance-awareness of AutoDA is an essential but not explored aspect for better performance on the learned dataset and transferability to other datasets. The proposed idea that utilizes the hidden feature of the original input seems to be reasonable and novel.\n\n**Cons.**\n\n- **Experiments/Ablation for diversity parameters**. The authors introduce the three diversity parameters for further adapting the learned policy. However, the proposed method itself is also possible to adapt the augmentation based on the feature map of each instance. Why these additional diversity parameters are required? Is it necessary? Also, for a precise ablation, the results without these diversity parameters should be provided in a direct setup (Table2), and the comparison of distribution for resulting augmentations is required (e.g., with/without the diversity parameters, how the learned augmentation is adapted from CIFAR-100 to each target dataset?)\n- **ImageNet experiments**.\n    - First, it is quite confusing as the experimental results on ImageNet are presented in Section 4.1 (Transfer setting). Is it right or should it be moved to Section 4.2.?\n    - Next, although the proposed method is quite worsen than other automatic augmentation learning methods in Table 2, I believe that the advantage of the proposed method is task-wise adaptability which more benefit from the transfer learning as done in Table 1 (as fine-tuning of the model from ImageNet pre-trained model is one of the most promising transfer learning nowadays) Can the proposed method outperform other fixed augmentation when we finetune the ImageNet pre-trained model?\n- **Weighted summation within exploration**. According to Equation (2), the weighted summation of differently augmented samples is used to generate a prediction for updating augmentation policy. Hence, as the one sampled augmentation is only used at the exploitation, there is a discrepancy between exploration and exploitation in the way for using augmentation. Is the weighted summation is essential for exploration, or can it be replaced by another method? For example, one can use a Gumbel-softmax trick at exploration.\n\n**Other comments.**\n\n- **Illustration of the augmentation policy magnitude $\\lambda$**. In Figure 2, the authors show the learned probability p to provide empirical evidence of class/instance-dependent augmentation. Since the magnitude $\\lambda$ is also an important learnable parameter, it would be helpful to validate the proposed method if the corresponding illustration is additionally provided.\n- **Qualitative results for better transferability with instance-awareness**. In Table 1, the proposed AdaAug significantly outperforms the other baselines by automatically adapting the augmentation. If the distribution of used augmentation can be compared qualitatively as done in Figure 2, it would enhance the empirical justification. (i.e., how the distribution of augmentation is **quantitatively** changed for the different tasks, instead of **qualitatively** as done in Figure 3?)", "summary_of_the_paper": "- This paper proposes AdaAug, an Automated Data Augmentation (AutoDA) method to learn a class/instance-dependent augmentation policy efficiently. The key ideas of AdaAug are two-fold. First, it uses a hidden feature of the original input to adapt the augmentation for each instance. Second, it alternates exploit-and-explore procedures for efficiently updating the augmentation policy. The authors demonstrate the empirical effectiveness on two scenarios; 1) transfer and 2) direct. In transfer setup, AdaAug significantly outperforms other AutoDA baselines. In direct setup, AdaAug shows comparable performance with the baselines.", "summary_of_the_review": "Although the target problem is important and the proposed method is novel, the empirical significances and analysis are quite insufficient currently. If the authors can provide that 1) experimental results fine-tuned from ImageNet model and 2) the empirical analysis how the augmentation parameters are actually changed by proposed method, especially in transfer scenario, I'll raise my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iIK69HrOEO", "reviewer": "Reviewer_xQTc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. Adaptive augmentation by learning a class-dependent and potentially instance-dependent augmentation policy for each data instance is a nice idea with strong possibility in augmentation\n2. Empirical studies has been conducted on different scenarios.\n3. The paper is well-written and easy to follow.\n\nWeaknesses\nNot enough experiment. From the experimental results, it gives me an impression that the results are happen to be successful. Multiple trials with mean and variants should be given. ", "summary_of_the_paper": "This paper illustrates an adaptive based data augmentation method named as AdaAug that searches adaptive augmentation policies in a class-dependent and potentially instance-dependent manner to improve the generalisation capability of deep learning models. This paper proposes an efficient exploition-exploration workflow to search for an augmentation policy that optimizes the generalization performance. The empirical studies on some datasets shows that the performance increase with AdaAug.", "summary_of_the_review": "I will give marginally acceptation for this paper. The idea is clear and novel but the novelty and contribution is not quite sounding. I can not decline that this is an interesting paper for this community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZMcA31rRBPU", "reviewer": "Reviewer_gmjs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n- **S1.** **[reasonable and well-motivated]** The basic idea of this work that makes the augmentation conditional on data itself seems reasonable and worth exploring. Examples like \"flip-invariance in digit classification\" discussed in the paper motivates this work well.\n\n- **S2.** **[relatively novel]** It is not a new idea to explore the class or subgroup conditional augmentations as introduced in the related work, but the originality may not be overwhelmed since this work may be the first trial to *automatically* learn instance-dependent augmentations.\n\n- **S3.** **[enlightening]** The policy search phase (the exploration pass) in this method is elegant and extends the augmentation search space: it mixes the activations of different augmented images so that it avoids discretizing the augmentation magnitudes (while much prior work did this). This is not a major strength but a favorable bonus that allows better exploration for augmentation search, thus is appreciated.\n\n\n**Weaknesses:**\n\n- **W1.** **[missing ablation]** Some prior work like AutoAug [1] presents their search-free results to prove the augmentation policy does benefit from the search phase. If it is possible, these results should be shown to substantiate the benefits of the augmentation search. A straightforward way is to evaluate a randomly initialized $h_\\gamma$ (not trained ever) on downstream experiments while keeping other *diversity parameters* the same.\n\n- **W2.** **[unclear performance improvements]** The performance reported in Table 5. shows that large $\\delta$ (*e.g.*, 0.4) would bring substantial improvements. The large-$\\delta$ setting makes the augmentation policy close to a random one given the augmentation magnitude ranges from 0 to 1. This can raise the suspicion that the performance gains mainly come from the rich randomness of augmentation magnitudes, not the augmentation search algorithm. Reporting the variance (*e.g.*, standard deviation) of the augmentation magnitudes $\\lambda$ and comparing it with the value of 0.4 would help further clarify this.\n\n\n**Open questions:** (not weakness)\n\n- **O1.** As a regularization technique, data augmentation almost always improves the loss, *i.e.*, augmented data have larger loss values than original ones. Thus, it is somewhat surprising that the augmentation policy does not collapse to \"no augmentation\" (*i.e.*, all magnitudes turn into zero) when we minimizing the loss of augmented data in the exploration pass. Could the authors explain this?\n\n\n- **O2.** The authors claim that this method approximately reduces the distance between the densities of **augmented $D_{valid}$** and **$D_{train}$**. In contrast, I feel \"reduces the distance between the densities of **augmented $D_{train}$** and **$D_{valid}$**\" could be more natural and reasonable, as in most deep learning cases we apply data augmentation only on training data and hope these augmented samples could be like validation data.\n    \n    - In order to \"reduce the distance between the densities of **augmented $D_{train}$** and **$D_{valid}$**\", we need to use $D_{valid}$ to train $f_{\\alpha}$ and $g_{\\beta}$, and use augmented $D_{train}$ to optimize $h_{\\gamma}$.\n    \n      - [use $D_{valid}$ to train $f_{\\alpha}$ and $g_{\\beta}$]: the exploitation pass does not perform any augmentations since we want to use $D_{valid}$ itself to train $f_{\\alpha}$ and $g_{\\beta}$. Thus, this pass could be understood as \"encouraging $f_{\\alpha}$ and $g_{\\beta}$ to memorize the distribution of $D_{valid}$\".\n    \n      - [use augmented $D_{train}$ to optimize $h_{\\gamma}$]: the exploration pass could be regarded as \"making augmented training data more close to validation data via minimizing the loss of augmented training data\".\n     \n    - How do the authors think of this view?\n\n\n-------------------\n\n[1] Cubuk, Ekin D., et al. \"Autoaugment: Learning augmentation strategies from data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n\n\n\n", "summary_of_the_paper": "This paper proposes a search algorithm for instance conditional data augmentation. The contributions are two-fold: 1) it could be the first trial to automatically learn class-dependent or instance-dependent augmentations; 2) an efficient workflow is designed that not only realizes gradient-based augmentation search but also extends the search space of augmentation magnitudes to continuous.", "summary_of_the_review": "I incline to give a marginal accept now given the originality and enlightening contributions. However, there are some questions towards the experiments that may affect my judgment. See main review for more details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LAMFsa94-EJ", "reviewer": "Reviewer_kS7y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The motivation of this work is clear and the idea of searching data-dependent augmentation policies seems reasonable.\n2. Empirical investigation is conducted on different scenarios. \n3. The paper is well-written and easy to follow.\n\n**Weaknesses**\n1. There are some related works this paper omitted. For example, [1] has also considered individual sample variations in augmentation policy searching and it proposed a meta-learning approach to learn the augmentation policies. This paper can compare with this in terms of the idea and performance, so that we will know which approach is better under which scenarios. Also, regarding the performance, the authors mention AdaAug has achieved state-of-the-art performance on the CIFAR-10, CIFAR-100, and SVHN datasets. However, according to [1], it seems AdvAA [2] performs better than the baselines in this paper. Did the authors ever compare with this method? What are the results?\n2. The performance improvement of the proposed approach seems marginal. Some prior works such as MetaAug [1] and AutoAug [3] run multiple times for the model and present the mean and standard deviation. Did the authors run multiple times for each model and what about presenting the mean and standard deviation etc?\n3. Results on different neural network architectures are missed. Table 3 only presents results on the Wide-Resnet model. What is the performance on different models such as Shake-Shake (26 2x96d) and PyramidNet+ShakeDrop as shown in [1,2,3]?\n\n**Other questions**\n* As this work automatically learns the policy, we are curious about the issue of convergence during training. Is the convergence guaranteed on both training and validation data? Is it possible to do a theoretical convergence analysis?\n\n**References**\n\n[1] Zhou et al., MetaAugment: Sample-Aware Data Augmentation Policy Learning, AAAI 2021\n\n[2] Zhang et al., Adversarial Autoaugment, ICLR 2020\n\n[3] Cubuk et al., AutoAugment: Learning Augmentation Strategies from Data, CVPR 2019\n", "summary_of_the_paper": "This paper introduces a data augmentation method AdaAug that learns adaptive augmentation policies in a class-dependent and potentially instance-dependent manner to improve the generalisation capability of deep learning models. Concretely, it proposes an efficient exploition-exploration workflow to search for an augmentation policy that optimizes the generalization performance. Experimental results on datasets with transfer and direct settings show the efficacy of this approach.", "summary_of_the_review": "The topic of this paper is interesting and worth exploring. Overall, the paper is well-written and the idea is novel. However, there are some questions regarding the experiments and analyses. We would like to see the feedback of the authors to the questions above. Also, if the performance improvement of the proposed model is not significant, we would like to see some theoretical analyses on the advantages of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that this work is well-motivated and presents a novel approach for data augmentation around the adaptive augmentation policies. There were some concerns around the lack of ablation studies and unclear performance improvements, which were addressed well by the authors’ responses. Thus, I recommend an acceptance.
Decision: Accept (Poster)