Tradeoffs in Data Augmentation: An Empirical Study
Authors: Raphael Gontijo-Lopes
Sylvia Smullin
Ekin Dogus Cubuk
Ethan Dyer
Venue: ICLR 2021 Poster (2021)
Keywords: Generalization
Interpretability
Understanding Data Augmentation

Abstract: Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly understood. In practice, augmentation policies are often chosen using heuristics of distribution shift or augmentation diversity. Inspired by these, we conduct an empirical study to quantify how data augmentation improves model generalization. We introduce two interpretable and easy-to-compute measures: Affinity and Diversity. We find that augmentation performance is predicted not by either of these alone but by jointly optimizing the two.

Published as a conference paper at ICLR 2021
TRADEOFFS IN DATA AUGMENTATION: AN EMPIRICAL STUDY
Raphael Gontijo-Lopes"
Sylvia J. Smullin
Google Brain
Blueshift, Alphabet
iraphae 1 goog com com
Ekin D. Cubuk
Ethan Dyer
Google Brain
Blueshift, Alphabet edyeregoogle. com com
cubuk@google.com
ABSTRACT
Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly understood. In practice. augmentation policies often chosen using heuristics of distribution shift or augmentation diversity. Inspired by these, conduct empirical study quantify how data augmentation improves model generalization. We introduce two interpretable and masy-to-compute mea- sures: Affinity and Diversity. We find that augmentation performance is predicted not by either of these alone but by jointly optimizing the two.
Relative Test Accuracy
1.6
0
Relative Accuracy
High Diversity
Data
Clean
-1
Diversity
3
Divers
Augmented
1.2
Low Diversitt
.25
.5
.75
Low Affinity
High Affinity
Affinity
Affinity
CIFAR-10
(b) ImageNet
(c) Model's View Data
Figure Affinity Diversity parameterize the performance of a model trained augmentation. (a, b) Each point represents different augmentation that yiclds test accuracy greater than (CIFAR- 10: 84.7%, ImageNet: 71.1%) 11). Color shows the final test accuracy relative to the bascline trained without augmentation (CIFAR- 10: 89.7%, ImageNet: 76.1 1%). (c) Representation of how clean data and augmented data are related in metrics. Higher diversity is represented by larger bubble while distributional similarity depicted through the overlap of bubbles. Test accuracy generally improves the upper right this space. Adding real new data to the training set is expected to be in the upper right corner.
the space of these
INTRODUCTION
Models that achieve sfate-of-the-art in image classification often use heavy data augmentation strategies. The best techniques use various transforms applied sequentially and stochastically. Though the effectiveness of this is well-established. the mechanism through which these transformations work is not well-understood.
Equual contribution
Currently Form Energy, Inc, Somerville, MA

---

Published as a conference paper at ICLR 2021
Since early uses of data augmentation, it has been assumed that augmentation works because it simulates realistic samples from the true data distribution: "[augmentation strategies are] reasonable since the transformed reference data is now extremely close to the original data. In this way, the amount of training data effectively increased" (Bellegarda et al. 1992). Because of this, augmentations have often been designed with the heuristic of incurring minimal distribution shift from the training data.
This rationale does not explain why unrealistic distortions such as cutout (DeVries & Taylor 2017). SpecAugment (Park et al. 2019), and mixup (Zhang et al. 2017) significantly improve generalization performance. Furthermore, methods do not always transfer across useful on CIFAR- 10 and not on ImageNet (Lopes et al., 2019). Additionally, many augmentation policies heavily modify images by stochastically applying multiple transforms to single image. Based on this observation, some have proposed that augmentation strategies are effective because they increase the diversity of images seen by the model.
for example, is
In this complex landscape, claims about diversity and distributional similarity remain unverified heuristics. Without more precise data augmentation science, finding state-of-the-art strategies requires brute force that can cost thousands of GPU hours (Cubuk et al., 2018: Zhang et al. 2019). This highlights& a need to specify and measure the relationship between the original training data and the augmented dataset, relevant to given model's 's performance.
In this paper, we quantify these heuristics. Seeking to understand the mechanisms of augmentation, focus on single transforms augmentations on CIFAR-10 and 225 on ImageNet, varying both broad transform families and finer transform parameters. To better understand current state of the augmentation policies, addi- tionally measure 58 composite augmentations on ImageNet and three state of the art augmentations on CIFAR-10. Our contributions are:
foundation. We present an empirical study of 204 different
1. We introduce Affinity and Diversity: interpretable, easy-to-compure metrics for parametriz- ing augmentation performance. Affinity quantifies how much an augmentation shifts the training data distribution. Diversity quantifies the complexity of the augmented data with respect to the model and learning procedure.
2. We find that performance dependent on both metrics. In the Affinity-Diversee plane, the best augmentation strategies jointly optimize the two (see Fig[]
3. We connect augmentation to other familiar forms of regularization, such as l2 and learning rate scheduling, observing common features of the dynamics: performance can be improved and training accelerated by tuming off regularization at appropriate time.
4. We find that performance is only improved when transform increases the total number of unique training examples. The utility of these new training examples is informed by the augmentation's 's Affinity and Diversity.
2 RELATED WORK
Since early uses of data augmentation in training neural networks, there has been an assumption that effective transforms for data augmentation are those that produce images from an "overlapping but different" distribution (Bengio et al. 2011 Bellegarda al., 1992). Indeed, elastic distortions
as well as distortions in the scale, position, and orientation of training images have been used on
MNIST [Ciregan et al. [2122] Sato al. 2013 Simard et al. 2003 Wan al. 2013), while horizontal flips, random crops, and random distortions to color channels have been used on CIFAR-10 10 and ImageNet (Krizhevsky et al. 2012, Zagoruyko & Komodakis 2016 Zoph et al. 2017). For object
detection and image segmentation, one also object-centric cropping (Liu et al. 2016) or cut-and-paste new objects (Dwibedi et al. 2017 Fang et al. 2019 Ngiam et al. 2019).
In contrast, researchers have also successfully used less domain-specific transformations, such as Gaussian noise (Ford et al. 2019 Lopes et al. 2019), input dropout (Srivastava et al.) 2014), erasing random patches of the training samples (De V Vries& & Taylor 2017 Park et al. 2219 Zhong et al. 2017]
and adversarial noise (Szegedy et al. (013). Mixup (Zhang et al.) 2011) and Sample Pairing (Inoue 2018) are two augmentation methods that use convex combinations of training samples.
2

---

Published as a conference paper at ICLR 2021
It is also possible to improve generalization by combining individual transformations. For example, reinforcement learning has been used to choose more optimal combinations of data augmentation transformations (Ratner al., 2017 Cubuk et al. 2018). Follow-up research has lowered the computation cost of such optimization, by using population based training (Ho et al.] 2019), density matching (Lim al., 2019), adversarial policy-design that evolves throughout training (Zhang et al..
2019), or a reduced search space (Cubuk et al. 2019). Despite producing unrealistic outputs, such
combinations of augmentations can be highly effective in different tasks (Berthelot et al.
2019 Tan
& Le 2019;
Tan et al., 2019: Xie et al., 2019a b).
Across these different examples, the role of distribution shift in training remains unclear. Lim et al. (2019); et al. (2020) have found augmentation policies by minimizing the distance between the distributions of augmented data and clean data. Recent work found that after training with augmented data, fine-tuning on clean training data can be beneficial (He et al., (019), while Touvroo et al. (2019) found it beneficial to fine-tune with test-set resolution that aligns with the training-se set resolution.
The true input-space distribution from which training dataset is drawn remains elusive. To better understand the effect of distribution shift on performance, many works attempt to estimate it. Often these techniques require training secondary models, such as those based on variational methods
Goodffellow et al., 2014; Kingma & Welling, 2014, Nowozin et al. 2016 Blei et al. 2017). J. Others
have augmented the training set by modelling the data distribution directly (Tann et al.
2017). Recent
work has suggested that even unrealistic distribution modelling can be beneficial (Dai et al. 2017).
These methods try specify the distribution separately from the model they are trying to optimize. As result, they are insensitive to any interaction between the model and data distribution. Instead, we are interested in a measure of how much the data shifts along directions that are most relevant to the model's performance.
METHODS
We performed extensive experiments with various augmentations on CIFAR-10 and ImageNet. Experiments on CIFAR-10 used the WRN-28-2 model (Zagoruyko & Komodakis 2016), trained for 78k steps with cosine learning rate decay. Results are the mean over 10 initializations and reported errors (often too small to show on figures) are the standard error on the mean. Details on the error analysis are in Sec. C
Experiments on ImageNet used the ResNet-50 model (He et al. 2016), trained for 112.6k steps with a weight decay rate of le-4, and a learning rate of 0.2, which is decayed by 10 at epochs 30, 60, and 80.
Images were pre-processed by dividing each pixel value by 255 and normalizing by the dataset statistics. Random crop was also applied on all ImageNet models. These pre-processed data without further augmentation are "clean data" and model trained on it is the "clean baseline". We followed the same implementation details as Cubuk et al. (2018 including for most augmentation operations. Further implementation details are in Sec. [A]
Unless specified otherwise, data augmentation was applied following standard practice: each time an image drawn, the given augmentation is applied with a given probability. We call this mode dynamic augmentation. Due to whatever stochasticity is in the transform itself (such as randomly selecting the location for a crop) or in the policy (such as applying a flip only with 50% probability), the augmented image could be different each time. Thus, most of the tested augmentations increase the number of possible distinct images that can be shown during training.
We also performed select experiments by training with static augmentation. In static augmentation, the augmentation policy (one or more transforms) is applied once to the entire clean training set. Static augmentation does not change the number of unique images in the dataset.
3.1 AFFINITY: A SIMPLE METRIC FOR DISTRIBUTION SHIFT
Thus far, heuristics of distribution shift have motivated design of augmentation policies. Inspired by this focus, we introduce simple metric to quantify how augmentation shifts data with respect the decision boundary of the clean baseline model.
Available bit. 1y/202rojM
3

---

Published as a conference paper at ICLR 2021
We start by noting that a trained model is often sensitive to the distribution of the training data. That is, model performance varies greatly between new samples from the true data distribution and samples from shifted distribution.
Importantly, the model's sensitivity to distribution shift is not purely a function of the imput data distribution, since training dynamics and the model's implicit biases affect performance. Because the goal of augmentation is improving model performance, measuring shifts with respect to the distribution captured by the model is more meaningful than measuring shifts in the distribution of the input data alone.
We thus define Affinity be the ratio between the validation accuracy of model trained on clean data and tested on an augmented validation set, and the accuracy of the same model tested on clean data. Here, the augmentation is applied the validation dataset in one pass, as a static augmentation. More formally we define:
Definition 1. Let Dunin and Dval be training and validation datasets drawn HID from the same clean data distribution, and let val be derived from Dvat by applying a stochastic augmentation strategy, a, once each image in Doat, Diutt = {[((z))) v) V (z,y) y) E Dvatt- Further let m be a model trained on Daiia and A(m, D) denote the model's accuracy when evaluated on dataset D. The Affinity, T(a: m; Drat), given by
Taa m; Drat A(m, Daa)/A(m, Doal)
(1)
With this definition, Affinity of one represents no shift and a smaller number suggests that the augmented data is out-of aut-of-distribution for the model.
Figure 2: Affinity model-sensitive measure of distribution shift. Contours indicate lines of equal (left) Affinity, or (right) KL Divergence between the joint distribution of the original data and targets and the shifted data. The two axes indicate the actual shifts that define the augmentation. Affinity captures model- dependent features, such as the decision boundary.
Yshn
Yshn
3
2
26
In Fig. 2 we illustrate Affinity with two-class classification task on a mixture of two Gaussians. Augmentation in this example comprises shift the means of the Gaussians of the validation data compared to those used for training. Under this shift, we calculate both Affinity and KL divergence of the shifted data with respect to the original data. Affinity changes only when the shift the data with respect to the model's decision boundary, whereas the KI. divergence changes even when data is shifted in the direction that is irrelevant to the classification task. this way, Affinity captures what is relevant to model: shifts that impact predictions.
This same metric has been used as a measure model's robustness to image corruptions that do not change images' semantic content (Azulay & Weiss 2018 Dodge & Karam 2017; Ford et al.
2019; Hendrycks & Dietterich] 2019 Rosenfeld et al.) 2018 Yin et al.) 2019). Here we, turn this around and use quantify the shift of augmented data compared to clean data.
Affinity has the following advantages as metric: 1. It is easy to measure. It requires only clean training of the model in question.
2. It is independent any confounding interaction between the data augmentation and the training process, since augmentation is only used on the validation set and applied statically.
3. It is measure of distance sensitive to properties both the data distribution and the model.
We gain confidence in this metric by comparing it to other potential model-depended measures of distribution shift. We consider the mean log likelihood of augmented test images (Grathwohl et al., 2019), and the Slaambbe-Akaile information criterion (WAIC) (Watanabe), 2010). These other metrics have high correlation with Affinity. Details can be found in Sec.
3.2 DIVERSITY: A MEASURE OF AUGMENTATION COMPLEXITY
Inspired by the observation that multi-facte augmentation policies such as F FlipIR+CroptCut and RandAugment
(aaigmeeentCuukkkk (Cubuk et al. 2019) greatly improve performance, propose another axis
4

---

Published as a conference paper at ICLR 2021
which to view augmentation policies, which we dub Diversity. This measure is intended to quantify the intuition that augmentations prevent models from over- fitting by increasing the number of samples in the training set; the importance of this is shown in Sec. 4.3
Based on the intuition that more diverse data should be more difficult for a model to fit, we propose a model-based measure. The Diversity metric in this paper is the ratio of the final training loss of model trained with a given augmentation, relative to the final training loss of the model trained on clean data:
Definition 2. Let a be augmentation and D Dirain be the augmented training data resulting from applying the augmentation, a, stochastically. Further, let Lininn be the training loss for a model, m, trained on D We define the Diversity, Dla; m; D Draiin,
Dia; m, Derain] = ED [Trraie] /RD Donn
[T
As with Affinity, this definition of Diversity has the advantage that can capture model-dependent elements, i.e. it is informed by the class of priors implicit in choosing model and optimization scheme as well as by the stopping criterion used in training.
One shortcoming of the above measure is the computational cost of evaluation. For each augmentation strategy of interest one must train an independent model and computing the training loss is as computationally costly as determining final test accuracy. For some purposes this cost may not be a problem. For instance, many successful augmentation strategies are usable across a wide range of models. This re-usability helps differ the initial search cost, Here we focus on this metric as tool for understanding.
Nonetheless, with computational cost in mind, in Sec. we consider two alternative measures of diversity that are substantially less costly to evaluate. The first of these is again using the training loss but at an earlier point during training. We find that even after ten epochs of training the early time loss correlates well with its value at the end of training.
Another potential diversity measure is the entropy of the transformed data, D This is inspired by the intuition that augmentations with more degrees of freedom perform better. For discrete transformations, we consider the conditional entropy the augmented data.
DEn X Ex [Ea' XX) (1(pp(((())))
Here X is clean training image and x' X' is augmented image. This measure has the property that it can be evaluated without any training or reference to model architecture. However, the appropriate entropy for continuously-varyinn
transforms is less straightforward.
Lastly, in Sec. D we consider a fourth proxy for Diversity, the training time needed for a model to reach a given training accuracy threshold. In Sec. we show that these metrics correlate well with each other.
In the remaining sections we describe how the complementary metries of Diversity and Affinity can be used to characterize and understand augmentation performance.
4 RESULTS
4.1 AUGMENTATION PERFORMANCE IS DETERMINED BY BOTH AFFINITY AND DIVERSITY
Despite the original inspiration to mimic realistic transformations and minimize distribution shift, many state-of-the augmentations yield unrealistic images. This suggests that distribution shift alone does not fully describe or predict augmentation performance.
Figure 3(a) (left) measures Affinity across 204 different augmentations for CIFAR-10 and 223 for ImageNet respectively. We find that for the most important augmentations--un those that help performance-Aiee Affinity is a poor predictor of accuracy. Furthermore, we find many successful augmentations with low Affinity. For example, Rotate (fixed, 45deg, 50%), Cutout (16), and combinations of F FlipIR, Crop (32), and Cutout (16) all have Affinity 0.83 and test accuracy> 2% above clean baseline CIFAR-10. Augmentation details are in Sec. B
As Affinity does not fully characterize the performance an augmentation, we seek another metric. To assess the importance of augmentation's complexity, measure Diversity across the same set
5

---

Published as a conference paper at ICLR 2021
Teot Accupac
40.6
Relaii Test Accuracy
1.6
Relative Test Accuracy
10
2
Diversity
Diversity
1.4
25
Test Accussry
-10
1.0
Amnny
Diversty
25
.75
Affinity
Affinity
(a) Top CIFAR, Bottom ImageNet
(b) CIFAR-10
(c) ImageNet
Augmentation performance determined by both Affinity and Diversity. Test accuracy plotted against each of Affiiity and Diversity for the two datasets, showing that neither metric alone predicts performance. the CIFAR-10 plots (top), blue highlights (also inset) are the augmentations that increase test accuracy above the clcan bascline. Dashed lines indicate the clean bascline. (b) and (c) show test accuracy relative to clean baseline on the color scale in the plane of Affinity and Diversity. The three star markers in (b) are (left right) RandAugment, Aut ouugeen. and mixup. For fixed values of Affinity, test accuracy fixed values of Diversity, test accuracy generally increases with higher values of Affinity. To quantify this tendency we measured what fraction of points satisfy Inequality 3
Figure 3:
generally increases with higher values of Diversity.
For CIFAR-I0 99.1 % of point pairs satisfy the inequality, while for ImageNet 97.5% satisfy it.
of augmentations. We find that Diversity is complementary explaining how augmentations can increase test performance. As shown in Fig. 3(b) and (c) Affinity and Diversity together provide much clearer parameterization of an augmentation policy's benefit to performance. For fixed level of Diversity, augmentations with higher Affinity are consistently better. Similarly, for fixed Affinity, is generally better to have higher Diversity. To make this more quantitative we consider pairs of augmentation strategies (a, a') and measure the fraction of pairs satisfying
{(a, a(a,a'): A(a) A(a') and T(a) T(a') D(a) D(a')
simple case study is presented in Fig. The probability of the transform Rotate (fixed, 60deg) is varied. The accuracy and Affinity are not monotonically related, with the peak accuracy falling at an intermediate value of Affinity. Similarly, accuracy is correlated with Diversity for low probability transformations, but does not track for higher probabilities. The optimal probability for Rotate (fixed, 60deg) lies at an intermediate value of Affinity and Diversity.
91.5
3.4
Test Accuracy
Test Test Accuracy
3.2
Diversity
3.0
Accuracy
2.8
2.6
89.7*
2.75
3.00
3.25
4
8
Affinity
Diversity
Affinity
Figure 4: Test accuracy varies differently than either Affinity Diversity. Here, the probability of Rotate (fixed, 60deg) on CIFAR-I0 varied from 10% (cyan) to 90% (pink). Left: as probability increases, Affinity decreases linearly while the accuracy changes non-mone non-moootonically Center: accuracy and Diversity vary differently from each other as probability is changed. Right: test accuracy is maximized at intermediate values.
To situate the tested augmentations mostly single transforms- within the context of the state-of the- art, we tested three nigh-performance augmentations from literature: mixup (Zhang et al. 2017). AutcAugment (Cubuk et al. 2018), and RandAugment (Cubuk al. 2019). These are highlighted with a star marker in Fig. 3(b)
More than either of the metrics alone, Affinity and Diversity together provide useful parameterization of an augmentation's 's performance. We now turn to investigating the utility of this tool for explaining other observed phenomena of data augmentations.

---

Published as a conference paper at ICLR 2021
4.2 TURNING AUGMENTATIONS OFF MAY ADJUST AFFINITY, DIVERSITY, AND PERFORMANCE
The term "regularizer" ill-defined the literature, often referring to any technique used to re- duce generalization error without necessarily reducing training error (Goodfellow et al. 2016). With this definition, is widely acknowledged that commonly- augmentations act as regular- izers (aermandez-Careia & König 201 8a b Zhang et al., 2016 Dao et al., 2019). Though this is broad definition, we notice another commonality across seemingly different kinds of regularizers: various regularization techniques yield boosts in performance (or at least no degradation) if the regularization is turned off at the right time during training. For instance:
1. Decaying a large learning rate on an appropriate schedule can be better than maintaining a large learning rate throughout training (agoruyko & Komodakis 2016).
2. Turning off l at the right time does not hurt performance (Golatkar et al. 2019).
3. Relaxing model constraints mid-training can boost performance (d' Ascooli et al. 2019).
4. Fine-tuning clean data can improve final accuracy (He et al. 2019).
To further study augmentation as regularizer, we compare the constant augmentation case (with the same augmentation throughout) to the case where the augmentation is turned off partway through training and training is completed with clean data. For each transform, we test over a range of switch- off points and select the one that yields the best final validation or test accuracy on CIFAR 10 and ImageNet respectively. The Switch-off Lift is the resulting increase in final test accuracy, compared to training with augmented data the entire time.
Figure 5: aa) Switching off regularizers yields a performance boost: Three examples of how turn- ing off regularizer increases the validation accu- racy. This slingshot effect can speed up training and improve the hest validation accuracy. Top: train- ing with no augmentation (clean baseline), compared constant augmentation, and augmentation that turned off at 55k steps. Here, the augmentation Rotate (fixed, 20deg, 100%). ) Middle; Base- line with constant fz. This is compared to turning off regularization part way through training. Bottom: Constant learning rate of 0.1 compared to training where the learning rate is decayed in one step by a factor of 10. (b) Bad augmentations can become helpful if switched off: Colored lines connect the test accuracy with augmentation applied throughout training (top) to the test accuracy with switching mid- training. Color indicates the amount of Switch-off Lift; blue is positive and orange is negative. The dashed purple line indicates accuracy of clean model. Crossing it from the left means the switched augmen- tation outperforms the clean model, Dashed grey lines indicate the threshold for Syml Log scaling. The stan- dard error on the mean over the ten independent runs are small to not be visible.
90
Off Baseline Constant Aug
85
85
30
49
50
50
70
80
Valid Accuracy
00.0
Tum J, Baseline
85.0
20
30
43
53
60
70
60
90
Stepped UR Constant LR
85
20
40
52
Training Steps 11000
(a) Slingshot effect on CIFAR-10
Aug On
Turning
Aug Off
34
7B
160
Test Accuracy
(b) Switch-off Lift on CIFAR-10
For some poor-pes poor-performin augmentations, this gain can actually bring the test accuracy above the baseline, shown in ((b) We additionally observe ((a) that this test accuracy improvement can happen quite rapidly for both augmentations and for the other regularizers tested. This suggests an opportunity to accelerate training without hurting performance by appropriately switching off regularization. We call this a slingshot effect.
Interestingly, we find the best time for turning off augmentation is not always close to the end of training, contrary to what is shown in He et al. (2019). For example, without switchnng. FlipUD (100%) (vertically flipping every image) decreases test accuracy by almost 50% compared to clean baseline. When the augmentation is used for only the first third training, final test accuracy is above the baseline.
7

---

Published as a conference paper at ICLR 2021
He et al. (2019) hypothesized that the gain from turning augmentation off is due to recovery from a distribution shift. Indeed, for many detrimental transformations, the test accuracy gained by turning off the augmentation merely recovers the clean baseline performance. A few of the tested augmentations, such as #1 ipll (1008), , are fully deterministic. Thus, each time an image drawn in training, it is augmented the same way. When such an augmentation is turned off partway through training, the model then sees images- the clean ones-that that are now new. Indeed, when FlipLR (100%) is switched off the right time, its final test accuracy exceeds that of F İpLR (50%) without switching. In this way, switching augmentation off may adjust for not only low Affinity but also low Diversity.
4.3 INCREASED EFFECTIVE TRAINING SET SIZE IS CRUCIAL FOR DATA AUGMENTATION
Most augmentations tested and those used in practice have inherent stochasticity and thus may alter given training image differently each time the image is drawn. In the typical dynamic training mode, these augmentations increase the number of unique inputs seen across training epochs.
To further study how augmentations act as regularizers, we seek to discriminate this increase in effective dataset size from other effects. We train models with static augmentation, described in Sec. 3 This altered training set is used without further modification during training so that the number unique training inputs the same between the augmented and the clean training settings.
For almost all tested augmentations, using static augmentation yields lower test accuracy than the clean baseline. Where static augmentation shows a gain (versions of crop), the difference less than the standard error the mean. As in the dynamic case, poorer performance in the static case is transforms that have lower Affinity and lower Diversity.
Static augmentations also always perform worse than their non-deterministic, dynamic counterparts, as shown in Fig. [ This may be because the Diversity of static augmentation always less than the dynamic case (see also Sec. E). The decrease in Diversity in the static case suggests connection between Diversity and the number of training examples.
Figure 6: Static augmentations decrease diversity and performance. CIFAR-I -10, static augmentation performance is less than the clean baseline, (0, 0). and less than the dynamic augmentation case (left). Diversity in the static case is less than in the dynamic case (right). Augmentations with no stochas- ticity excluded because they trivially equal on the two axes. Diagonal line indi- cates where static and dynamic cases would be equal.
Dymamic Augmentaiion Augmeett Diversity
Dynamic Argmentation
2.5
0.0
-2.5
-3.0
-7.
2
-10.0
Static Augmentation Accurscy
Static Augmentation Diversity
Together, these results point to the following conclusion: Increased effective training set size is crucial to the performance benefit of data augmentation. An augmentation's 's Affinity and Diversity inform how useful the additional training examples are.
DISCUSSION
In this work, we conducted an empirical study in attempt to characterize the essential factors leading successful augmentation policies. This builds foundation for using these metries to quantify and design more complex and powerful combinations of augmentations. We further hope that Affinity and Diversity may serve as useful metrics of distribution shift more generally.
We believe that our analysis sheds light on the mechanisms of vision augmentations, such mixup. Cutout, and AutoAugmeet. Though earlier work has often explicitly focused on just one these metrics, chosen priors have implicitly ensured reasonable values for both. One way to achieve Diversity is to use combinations of many single augmentations, as in AutoAugment (Cubuk et al. 2018). Because transforms and hyperparameters in Cubuk et al. (2018) were chosen by optimizing performance on proxy tasks, the optimal policies include high and low Affinity transforms. Fast Autonut (Lim al. 2019). CTAugment (Berthelot al. 2019 Sohn et al. 2020), and
8

---

Published as a conference paper at ICLR 2021
differentiable RandAugment (Cubuk et al. 2019) all aim to increase Affinity by what Lim al. (2019) called "density-man matching" However these methods use the search space of AutoAugment and thus inherit its Diversity.
On the other hand, Adversarial AutoAugment (Zhag et al. 2019) focused on increasing Diversity by optimizing policies to increase the training loss. While this method did not explicitly aim to increase Affinity, it also used transforms and hyperparameters from the AutoAugment search space which implicitly have higher Affinity than random transformations one can apply to an image. For example, rotations are restricted to -30 to 30 degrees in Augonn,, which have higher Affinity than a random rotation. Without such a prior, the goal of maximizing training loss with no other constraints would lead to data augmentation policies that erase all the information from the images.
Our work the first one to propose and study these two metrics generally, which helps put previous work in context. Furthermore, our work suggests that further improvements can be expected if one tries to increase both of the metrics, instead of just one of them as was the case in previous work. Given work, an obvious strategy to find better policies would be to combine the losses of Adversarial Augment and Fast AutoAugment, which we plan to do future work. Finally, and knowing our insights from the vision domain, we speculate that a similar mechanism might be at play in other domains, such as speech recognition, and underlie why strategies such as SpecAugment, which appear to be relatively out of distribution, nonetheless work predict that they drastically increase diversity.
6 CONCLUSION
We attempted to quantify common intuition that more in-distribution and more diverse augmentation policies perform well. To this end, we introduced two casy-to-compute metrics, Affinity and Diversity, intended to measure to what extent a given augmentation is in-distribution and how complex the augmentation is to learn. Because they are model-dependent, these metrics capture the data shifts that affect model performance. We hope our findings provide a foundation for continued scientific study of data augmentation and more general distribution shifts.
ACKNOWLEDGEMENTS
The authors would like thank Alex Alemi, Justin Gilmer, Guy Gur-Ari, Albin Jones, Behnam Neyshabur, Zan Armstrong, and Ben Poole for thoughtful discussions on this work.
REFERENCES
Anonymous. Is robustness robust? on the interaction between augmentations and corruptions. In Submitted to International Conference on Learning Representations, 2021. URL https under review.
/ooeeneview net /
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint 1rXiv::805.12177, 2018.
J. R. Bellegarda, P. V. de Souza, A. J. Nadas, D. Nahamoo, M. A. Picheny, and L. R. Bahl. Ro- bust speaker adaptation using a piecewise linear acoustic mapping. In ICASSP-92: 1992 IEEE International Conference Acoustics, Speech, and Signal Processing, 1992.
Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boulanger- Boulanger-Lewaddone Lewandowski, Thomas Breuel, Youssouf Chherawala, Moustapha Cisse, Myriam Côté, Dumitru Erhan, Jeremy Eustache, et al. Deep learners benefit more from duttt--distributiin examples. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 164-172, 2011.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: /9/ 1,09785, 2019.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 11((518):859-87) Feb 2017. ISSN 1537-274X.

---

Published as a conference paper at ICLR 2021
doi: 10.108001661145922117.222288 2017.1285773
Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image classification. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 3642-3649. IEEE, 2012.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data, arXiv preprint arXiv: /805.0950, 2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint 1rXiv:190..137700 2019.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Russ R Salakhutdinov. Good semi- supervised learning that requires a bad gan. In Advances in neural information processing systems, pp. 6510-6520 2017.
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Christopher De Sa, and Christopher Ré. A kernel theory of modern data augmentation. Proceedings of machine learning research, 97:1528, 2019.
Stéphane d' Ascoli, Levent Sagun, Joan Bruna, and Giulio Biroli. Finding the needle in the haystack with convolutions: on the benefits of architectural bias, 2019.
Terrance Dries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint /70.v:::770..4555 2017.
Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recogni- tion performance under visual distortions. In 2017 26th international conference on computer communication and networks (ICCCN), pp. 1-7. IEEE, 2017.
Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1301-1310, 2017.
Hao-Shu Fang, Jianhua Sun, Runzhong Wang. Minghao Gou, Yong-Lu Li, and Cewu Lu. Instaboost: Boosting instance segmentation via probability map guided copy-pasting. In Proceedings of the IEEE International Conference on Computer Vision, pp. 682-691, 2019.
Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cubuk. Adversarial examples are natural consequence of test error in noise. arXiv preprint arXiv:1901.10511 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence, 2019.
Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International conference on machine learning. pp. 1319-1327. PMLR, 2013.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Process- ing Systems 27, pp. 2672 2680. Curran Associates, Inc., 2014. URL lattp://papers. nips cc het/ppprr/55 -eeee
ive ieeeraattiievvvveeeeeee www.eeveevaaeaaaaaeaeeaaaaaaaaaa wettsarrrrlleeeeeeeeeeemeea pdf
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: w/ww.ddeplearrinpoot..or
Will Grathwohl, Kuan-Chieh Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one, 2019.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. In European conference on computer vision. Springer, 2020.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Zhuoxun He. Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
URL httpttts:/pppeeeiiiiee net hett:/oromm.dddll2tttie..
Alex HemándezzGareíi and Peter König. Data augmentation instead of explicit regularization. arXiv preprint arXiv:1806.03852. 2018a.
Alex Hernández-Garría and Peter König. Further advantages of data augmentation on convolutional neural networks. In International Conference on Artificial Neural Networks, pp. 95-103. Springer, 2018b.
Daniel Ho, Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. arXiv preprint arXiv:7905.05380 095399 2019.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. arXiv preprint arXiv:1801.02229, 2018.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In Yoshua Bengio and cnclip.uiittreee .eerrierrdeeeeeeeee httddd//ooffee/c /ccr//
Yann LeCun (eds.), ICLR, 2014.
1c1r2014 html #KingmaW13
Alex Krizhevsky, Nya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, 2012.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, pp. 6665-6675. 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yare Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pp. 21-37. Springer, 2016.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv: 2019.
Jiquan Ngiam, Benjamin Caine, Wei Han, Brandon Yang, Yuning Chai, Pei Sun, Yin Zhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, et al. Starnet: Targeted computation for object detection in point clouds. arXiv preprint arXiv: 1069, 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization, 2016.
Daniel S Park, William Chan, Yu Zhang, Chung-Chent Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 1904::/900007779 2019.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose domain-specific transformations for data augmentation. In Advances in Neural Information Processing Systems, pp. 3239-3249, 2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar- 10 classifiers generalize to 10far-10?, 2018
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize ImageNet? volume 97 of Proceedings of Machine Learning Research, pp. Long Beach, California, USA, 09-15 15 Jun 2019. PMLR. URL http:ttp//pocceednngggoooo/////e mlr totss//97/reccht. 19..tt999... 9a. html
11

---

Published as a conference paper at ICLR 2021
Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv:1808.03300 03305, 2018.
Ikuro Sato, Hiroki Nishimura, and Kensuke Yokoi. Apac: Augmented pattern classification with neural networks. arXiv preprint arXiv:/505.0322 2015.
Patrice Y Simard, David Steinkraus, John Platt, et al. Best practices for convolutional neural networks applied visual document analysis. In Proceedings of International Conference Document Analysis and Recognition, 2003.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying aaii-suprvvied learning with consistency and confidence. arXiv preprint arXiv:2001.07685, 2020.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint orXiv: 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199, 2013.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arxiv preprint 19XX::1905.11945 2019.
Mingxing Tan, Ruoming Pang, and Quoc Le. Efficientdet: Scalable and efficient object detection.
arXiv preprint arXiv:191..09070 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy, 2019.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and lan Reid. A bayesian data augmentation approach for learning deep models. Advances in Neural Information Processing Systems, pp. 2794-2803, 2017.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International Conference on Machine Learning, pp. 1058-1066, 2013.
Sumio Watanabe, Asymptotic equivalence bayes cross validation and widely applicable information criterion in singular learning theory. J. Mach. Learn. Res., 11:3571-3594, December 2010. ISSN
1532-4435.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. arXiv preprint arXiv: 196666, 2019a.
Qizhe Xie, Zihang Dai, Eduard Hovy. Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation. arXiv preprint arXiv:1904.12845 2019b.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin Cubuk, and Justin Gilmer. fourier perspective on model robustness in computer vision. arXiv preprint 1rXiv:196.008988 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rX0v:1710.094122, 2017.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment, 2019.
12

---

Published as a conference paper at ICLR 2021
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. arXiv preprint arXiv: 1708: 708964996 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017.
13

---

Published as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIAL
A TRAINING METHODS
Ci models were trained using code based on AutoAugment code using the following choices:
1. Learning rate was decayed following a cosine decay schedule, starting with value of 0.1
2. 78050 training steps were used, with data shuffled after every epoch.
3. As implemented in the AutoAugment code, the WRN-28-2 model was used with stochastic gradient descent and momentum. The optimizer used cross entropy loss with l weight decay of 0.0005.
4. Before selecting the validation set, the full training set was shuffled and balanced such that the subset selected for training was balanced across classes.
5. Validation set was the last 5000 samples of the shuffled CIFAR-10 training data.
6. Models were trained using Python 2.7 and TensorFlow 1.13.
training time of 78k steps was chosen because showed reasonable convergence with the standard data augmentation of F FILPL,, Crop, and Cut out In the clean baseline case, test accuracy actually reached its peak much earlier than 78k steps.
With CIFAR- 10, experiments were also performed for training dataset sizes of 1024, 4096, and 16384. At smaller dataset sizes, the impact of augmentation and the Switch-off Lift tended be larger. These results are not shown in this paper.
ImageNet models were ResNet-50 trained using the Cloud TPU codebase Models were trained for 1 112.6k steps with a weight decay rate of le-4, and learning rate of 0.2, which was decayed by 10 at epochs 30, 60, and 80. Batch size was set to 1024.
B DETAILS OF AUGMENTATION
B.
CIFAR-10
On CIFAR-10, both color and affine transforms were tested, as given in the full results (see Sec. G Most augmentations were as defined in Cubuk er al. (2018) and additional conventions for aug- mentations as labeled in Fig. 7 are defined here. For Rotate, means each augmented image was rotated by exactly the stated amount, with a randomly-chosen direction. Variable means an augmented image was rotated random amount up to the given value in randomly-chosen direction. Shear r is defined similarly. Rotate (square) means that an image was rotated by an amount chosen randomly from [0", 90% 180" 270%].
Crop included padding before the random-location crop so that the final image remained 32 x 32 in size. The magnitude given for Crop is the number of pixels that were added each dimension. The magnitude given in the label for Cutout is the size, in pixels, of each dimension of the square cutout.
Pat catcuss an was defined as in Lopes et al. (201)). with the patch specified be contained entirely within the image domain. In Fig. 7 it is labeled by two hyperparameters: the size of the square patch (in pixels) that was applied and o mazz which the maximum standard deviation of the noise that could be selected for any given patch. Here, "fixed" means the patch size was always the same.
Since F FlipIR, Crop, and Cutcut are part of standard pipelines for CIFAR-10 (Hemánnez--Game & König 2018a; Goodfellow et al., 2013 Springenberg et al., 2014),, we tested combinations of the three augmentations (varying probabilities of each) well as these three augmentations plus single additional augmentation. As in standard processing of CIFAR-I0 10 images, the first augmentation applied was anything that is not one of FLipLR, Crop, Cutout. After that, augmentations were applied in the order Crop, then F FlipLR, then Cutout.
"available at htt.a availlabl thub. http://tcooooaaeeeooo/////ttttte creeeeeettttrreeetttttteeeee resnet
14

---

Published as a conference paper at ICLR 2021
Finally, we tested the CIFAR-10 Auto policy (Cubuk et al. 2018). RandAugment (Cubuk et al. 2019), and mixup (Zhagg et al. 2017). The hyperparameters for these augmentations followed
the guidelines described in the respective papers.
These augmentations are labeled Fig. 7
10
Diversity
0.2
0.6
1.0
Affinity
Figure 7: CIFAR-10: Labeled map of tested augmentations on the plane of Affinity and Diversity. Color distinguishes different hyperparameters for given transform. Legend below.
15

---

Published as a conference paper at ICLR 2021
Clean
Equarize 200%)
Autoougment
Houp
Stea Aborittt. o.1, 100%)
Coop4_1073)
Randuug
Stear Xb.riti. a.1. 50%)
Crost6
PacthCaastentteeee.. 5.1, 100%
Xtematee 31, 75%)
71 T1p0922000)
PasdrCowwinntteeeee2 5.2, 100%%
StearX 33, 100%)
71 T120.02000))
-ross4.132%)
Cu0:0015.2000
NasdhCowsierrbeeeeee 2.5, 120%)
5 SteurX 23, 50%0
F1 F1
Croos4..22%)
Nosdtoonseereeeeel2. 2.5, 100%)
StearXtummake 23. 75%)
(Coos44.253)
Cutouit1 1005)
Roshsannserreeeee. 2.5. 100%)
Stear 0.1. 1009)
H HILLX0000)
(coo)442552)
1 Cuuuiits 25%)
PachhGeeeerrtttee.2.. 2.5, 130%)
SteweXttted 50%)
TTp.922005))
-Cos/A.23%)
Culmutt165500
Pashheesstrttteet2... 2.5, 150%
StearXttte 75%6
T1gL922005)
(Cest4.23%)
Citnutt35 75%)
NasahGenaiarrteee11222 120%
StearXiited 200%)
-coss44.7%)
CCtoutt18 100%)
NashCeraiarrteeal..... 120%
eild 50%)
Cutoutt1a 23%)
Nushheenatarrieee..... 100%)
StearX 158)
(Cosi4_578))
Cutuuttle
20seg,
20seg.
PerhhGaostertteddd.... 13074
Rebamtvwatneee 5deo, 50%)
Flyt R2250
Catrot116. 150%)
PachhGanederttteee22 13.05
Retane/oaraaw Ideg 75%)
Cutnutt16 25%)
PacchGeasttrrtteed:2 13200%
Retam)/orraee c0ceg. 102%)
Cuteutt34.28 50%)
Pachheeustertteet22 2.1, 100%)
Retanatiees 15deg 50%
Cuteutt16. 75%)
Nasaheessiir/ttrrt22 2., 100%)
Retessikees 20de: 102%)
Figunga 90
Crop/4.25%1
2005
Nashheezauum/brrr222333 255 100%)
Retusifice. 20des 50%)
Crpp/.25%%
PenhGaicaarttedd22 1.5, 100%
Rebae//tcee, 20des 15%6
Cropp125%
PacthGeuceerrtttee3 1.5, 100%
Retans/Btes rodes 10%)
Crep/12551
6.755
PacchGewattrttteee3 2.5, 100%
Retansiikes, tdeg. 12%)
CrepAt,505
PacdhGeminntteee22.. 2.1, 100%)
Retassiikes, Ideg. 100%)
Crepp4.5051
hachheeminnneeeeee2.2 2.2, 100%)
Retessiikes, Ideg. 20%)
Crop:4.3051
RechGaacssmdaadd22.... 100%)
RebeelBees, 1deg. 10%0
Flyt R12535
Crepl4.50%)
PwchhGeooseerttttt..... 3.5 100%)
Robeellkes, 5deg. 40%0
PachhGewederrttttt22.... 150%)
Robealikees 5deg 50%)
F1 CCrp/4,75%
PasthGeasttrrttee224.4.4 130%)
Rstanalikes, 5deg. 50%)
T1L.22255 -Crpp.775%
6.
NasahGensinmteeee2 2.5, 120%)
Rstamilikes 5deg. 70%)
CCrpp4.75%
Rotaelffees,
-Cmp:s_1078)
-Cmp/h.107%)
-Cmp//.00%)
-Cmp/A
ParhhGwnssnnfeedd22 190%)
Retyelffies, K0deg.
-Crpp8,25k1
ParhhGanedertteedd... 155%1
Retwelftees, rodeg.
Flyt Ry..0
-cop Cuesue (5k)
PachhGanedentteel..... 1501
Retwraltess, rodeg. 50%)
fTg.2550% Crep//.5001
PacthGenetemtteeed... 130%
Retanalties, codeç.
TTp.550%% Crep1
PachhGeusttrtteeee3.... 32. 2.2.2221100 150%
Retanalikes, codeç 70%)
TTpL2/0%) CrepAA.0%)
RochGwusänrrheedd 32, 2.3, 100%)
Ritomeffees, codes, 10%)
RBLM50%) Crops4.5001
Pechisaosanrrteee 32, 5.5, 100%)
Retae/Biees, codes 5010
RLLM55 Cropss.756
PechiGsiccerrtteet.. 12, 5.4, 100%
Retaneeeuuar 10268
Fllt 9555) - Crepp1790 6.755)
PechiGeaceeerttted1.. 37, 1.5, 100%)
Retars/ecuars 50%)
Tyl-205%% - Crep/17551 (.558
PechhGeceeenttteed. 32, 2.5, 100%)
Bur2275%
Tyg.8055 CreppA.75% (.5268667755
Rodh6aansorrreedde... 32, 2.6. 100%)
Bur1o0ii
HILM7550 Cropp4.1000) Ci 10030
Roch@ansssrrreedd..... 0.1. 100%)
FIp 00%)
RI Crep04_1003) + +Cinorr16 2593
RechCansssmtttddd 100%)
Fipi A22550
Crepos_1020s +cumor116500
Pech@aaosen/ttettl... 103%)
Fipi A/505)
Crep9
CCtmon116. 71%)
Pech@Geeesnttteeel...... 101%)
bechGGestammtteetl... 103%)
CCrpp4.25%)
#oshhGussarrteeedd.... 0..10)
R1pu022561
-Crg0422501
RppD005041
Rpu0277961
Fl 90500
-
Rodhhaazzinneeedd.... 102%)
Cop14. 75%)
Coop94.00%
Co29441029)
RyLA775% -Cops47554 -G6004216.0000)
Poch@swsstnnteedd (.210
6 10081
-ropos.7561 (G
PerhhGaosdendteedd a. 10210
Cumund 25%0
Flyt 97790
Crepp5.75611
PerhhGanedenttedda a. 102%)
5 50%)
Flyt 9775%
Crep (6
NochGansiarrbeell.... 2. 2.0. 101%)
CLouui115 75%)
HILLN50%)
Cropsk, 100% cutcutlls. 1009) kqualoux)00))
Cropik, 190% cutcutlle. 2009) Rutonelfiedd 15dog, 50%
B.2 IMAGENET
On ImageNet, we experimented with FatchGaussian. Cutout, operations from the PIL imaging library" and techniques from the AutoAugment code, as described above for CIFAR-IO. In addition to Pat Catchhaussian (fixed), we also tested Pat PatchGassiian (variable), where the patch size was uniformly sampled up to a maximum size. The implementation here did not constrain the patch to be entirely contained within the image. Additionally, we experimented with Sol colaria zeadd. Solari zeAdd similar to Solari ze from the PII. library, but has an additional hyperparameter which determines how much value was added to each pixel that is below the threshold. Finally, we also experimented with Full Caussian and Random Erasing on ImageNet. Full Gaussian adds Gaussian noise to the whole image. Random Erasing is similar to Cutout, but randomly
https::/ppi www.rredddcc...oeo wctpe//oo.o.....//........
16

---

Published as a conference paper at ICLR 2021
samples the values of the pixels in the patch (Zhong et al. 2017) (whereas Cutout sets them to constant, gray pixel).
These augmentations are labeled in Fig. 8
We also experimented with combinations of transformations, using augmentations used in the AutoAugment policy. These are labeled RandomMul iAug (x, Yr z), where is the number of augmentations per subpolicy, y is the number of subpolicies, and z is 0, or 1, indicating two randomized policies ran.
2.50
2.25
2.00
Uiversity
1.75
1.50
X
Affinity
Figure 8: ImageNet: Labeled map of tested augmentations the plane of Affinity and Diversity. Color distinguishes different hyperparameters for given transform. Legend is below.
17

---

Published as a conference paper at ICLR 2021
Patch Gaundenctaes 200 1.4, 100%
Solaton A651-227 302 105%)
Digttremi... 100%
Patch Gaunsenctess 200 2.0. 100%9
Sclarte 282, 100%)
#njttnessm 2. 100%
Goecarntten 250 6.2 1:0%)
Stlatze 205,
Bightnwwut.
10099
Perth Gaundanctinn, 250 1.6, 100%)
Sclwton A6:10001
100%
Bighttweit.c 100%)
Patch Crundanctine, 250 2.6, 100%)
Sclaton A6510003
100%
Brittressn... 100%)
Patc Gasssare 200 6.2. 100%)
Sala Adu00025
100%)
Clean
Patch faed 200 6.5 100%)
Sata Ad10022 250. 100%
Cobrto. 200%
Patch tasd 200 8.8. 100%0
Solarss Adcicoss 250. 100%)
Caaad02. 100%
Parth Ganndnnciand 10 1.8, 150%)
Solvize Aositoss
100%)
Catad6.1 100%
Parm Gosndancians 100 2-0, 150%)
Solwtte Apcitoss 100%
Couri84 200%
Pash Gusosane 250. 6.2 100%
Salanas Adc1000s 200%)
coludes
Pash Gasosare 259 8.. 100%)
Salaras Aduicoss
Calrras
100%)
Conrast0.1 100%
Patch Gassdannians, 350 100%)
Satariee 46130073,
comtax00.2 100%
Patch Gauasiant facd. 400. 0.2. 100%)
Solaraz A6si0e7s, 250. 100%)
cone 100%
Patch Gasosart lacd. 403. 0.5 100%)
Solaraz Adsi000s 200. 200%)
Conmecs)4 10054
Parth Gaundanciaes, 6o3. 6.8, 100%)
Soletse AociCons 150. 100%%)
comusaa. 1009a
Patch Spsssent lacd. 400 1.8, 100%)
Salatiz A0 200. 100%))
comasaat 1908a
Patsh Groddan lacd. 400. 2.8, 100%)
Solatez A0 250. 100%))
Conmars9.7. 300%
Portartaw0 200%1
Solatte A6c15323
100%)
Cutastranttee 446, 130%)
Postertzal1 200%
Solaras Adc1C123, 100%)
cubarified 10254
Poctrtoves 10044
Tarsaww020 130%)
Pastetaaid 200%4
Tarsawako0 130%)
Cutarit 10191
Pustrizdd 7 10781
Forsinetion, 100%)
Eq am13205)
Random Eraingivarteb x 446, 100%
Tara 130%)
Random Erwinptloes. 125. 200%
Taraiooak00) 120%)
P 100%)
Rawdom Eneispthies, 155, 100%1
TarsawAR 1368)
D 102%
Random Cmeisptises, 185, 100%
TansaawsI0, 1388)
3 19583
Rondon Eroiipplies 30: 100%
Torskicc000
10780
Crathprtise
Craingrtses
KAugh, Rantonity
PuRGoussancinn
Rooitaaarate. 130%)
PukGousdene 1.5 100%)
Rebertaarirtee 10tre 1929)
Rxntonit,t//aatt, 3.31
Pu1Eausilen220 102%)
nolatiiaraalle 15dap 100%)
Ranzonifu BAugt. 7.01
Rcbare/quamm 100%)
Rebescarittee 20000 10010
Rawsomih, 2.31
0010
Resentartttlee 25deg 100%
Pastomi.E/ugh 15, 3)
Putsh Gnskriiiiii 100 S.,, 120%)
Routdoonttl 20lcy 19083
Ranionih 15,
Passh Sasii 100. 5.5 100%))
Recostorratte 5dey. 190%)
Raniomih (orionntt/4ee)0 15.
Passh Gesaitmmenttes 200 5.8, 130%)
Sharpnest33 3. 102%)
Ranssmit BAug
Goussmnaaratiee 1.6,
Shappressth
Groppreet
tacie 150. 5.8, 120%)
Grampneett 102%)
Randomifu B/
PutS Gussarr rasic 150 1.0, 100%))
Sharpnessth... 100%)
Parth Gxussore rasle 150. 2.6, 100%))
Shearsyver abre. 0.1. 1039)
Randomih Rluu52, 7. 01
Patsh tacit 200 5.1, 120%)
ShaurAtver abro 0.2 103%)
Ransomhu BAugh, 7.11
Passh tasai 200 2.5, 120%)
Shour abro. 0.3. 100%)
Runsomhu BAugt) 25. 21
Parm Gasadamroantttt 200 5.8, 130%))
Siversteer abro 100/9)
Rastomiti B/upp? 3)
Pussh Gusssark lasic 2.0, 120%)
Shoar
1075.)
Runssnttu LAupti
@xossant kasic 2.6, 120%)
com
100%
200000023 100%
1.01
aaaaaa 250 110%)
So 20222 100%
Rantomifu LAug 1.11
Putsh Sosil 250 0.5, 120%)
Solur cu50 10191
nunzonifu LAugt5, 3.01
Parth Geu 250 3.0, 130%))
So be Ad11-002 001, 1005
Panzommh Rlugh, 3.11
Patch Graddemoattteee 250 3.5, 130%)
So A051-002 053 100%
Raosomhh, BAugh3 7. D1
Path 0us salle 250 2.0, 100%))
Solor 0031-00 190. 100%
ConiuntheAAuut 7.11
Patch Gasallarreettee 200 5.2 150%)
Solarze Add1-002
152: 100%
Panssmit BAug?, 3)
Patch Gasailamneettt 200 C.S, 120%)
Solarie cs A621-202 200: 100%
Ransemit B'Ag
100%
3.6,
Goundrrraatttee 350 5.2, 120%)
Searce Ad51-227
100%
Rantomihi kAug0,
Goumdarraatta 350 c.5, 120%)
Seaere Ad51-127 153. 200%
Rantomifu BAuge4, 1.01
Pats G ralic 350. 5.6 130%))
Solrr 4211-2)
203 100%
Ravtinnh R.uuj44 1.11
Parth tadie 350 1.6, 150%))
So Ad11-27
253. 100%
Rawioohh 01
Patch taci 250 2.0. 120%)
Solar se Add1-52
093 100%
Rensomhh BAugi4 3.11
Parm Gwwedarra 100 5.2, 150%))
Sear X41)-50)
100%)
Rastsmih B/aggt,, D1
Parh Ga 100 5.5, 1301))
Sebrze X411-55)
102. 100%)
B/ugi, 7.31
Putsh Gussiant lasic 0.6 120%)
Ad11-13 157. 100%
Ranismih EAugt4,
Putch 0ossiinneeea 1.6,
203, 100%)
Ranisnthy Kitugt,
Gaundardiinn, 6.2, 150%)
A411-77
Putch Gasalurritea 100. 0.5, 100%)
Salur Ad21 103, 100%
Runsontfu LAugi4,
Puth leas, 100 D.8. 100%)
Solur Ad11-7) 150. 100%)
Runzonfh LAugt5, 1.01
Parch Gaundancfoes, 100 3.0, 10050
Se Ad11-77 201, 100%
Panzomhh KAugf, 1:11
Patch Casadanctssc, 100 2.0, 10050
So Ad11377 253, 100%
Pansomh EAught, 3. D1
Patsh fess 150 6.2 100%)
Soar ce 103) 00), 100%)
Raniimih EiwejtS,
Patch fess 0.S, 150%)
Solaris Ad51-02 053 100%
Ransemitu BAug5, 01
Patch fasaalane 5.6, 150%)
Solur Ad21-02, 102. 100%)
Runssmihu B'ugs, 7.31
1.0,
100%)
Rassomith
100%)
Brug,,
Grundercttes, 6.5, 150%)
Salaree Add1-127, 063, 100%
Rantomiti Blugif,
Patch Grumilenrieess, 200 0.8, 110%)
Salarie Ad51-127.2200 053, 100%
Ransomthu BAugi,,
Each augmentation was applied with a certain probability (given as percentage in the label). Each time an image was pulled for training, the given image was augmented with that probability.
18

---

Published as a conference paper at ICLR 2021
C ERROR ANALYSIS
All of the CIFAR-10 experiments were repeated with 10 different initialization. most cases, the resulting standard error on the mean (SEM) is small to show as error bars on plots. The error on each measurement is given in the full results (see Sec. G
Affinity and Switch-off Lift both initialization. For Affinity, the same trained model was used for inference on clean validation data and on augmented validation data. Thus, the variance of Affinity for the clean baseline is not independent of the variance of Affinity for a given augmentation. The difference between the augmentation case and the clean baseline case was taken on per-experiment basis (for each initialization of the clean baseline model) before the enror was computed.
computed from differences between runs that share the same
In the switching experiments, the final training without augmentation was completed starting from given checkpoint in the model that was trained with augmentation. Thus, each switching experiment shared an initialization with an experiment that had no switching. Again, this case difference was taken per-experiment basis before the error (based on the standard deviation) computed.
All ImageNet experiments shown are with one initialization. Thus, there are not statistics from which to analyze the error.
D SWITCHING OFF AUGMENTATIONS
For CIFAR-10, switching times were tested in increments of approximately 5k steps between 25k and ~ 75k steps. The best point for switching was determined by the final validation accuracy.
On ImageNet, tested turming augmentation off at 50, 60, 70, and 80 epochs. Total training took 90 epochs. The best point for switching was determined by the final test accuracy.
The Switch-off Lift was derived from the experiment at the best switch-off point for each augmenta- tion.
augmentations where the validation accuracy best at 25k, which means that further testing needed to find if the actual optimum switch off point is lower or the best case is not train at all with the given augmentation. Some of the best augmentations have a small negative Switch-off Lift, indicating that is better to train the entire time with the given augmentations. For completeness for all augmentations studied in the table below. The error is the standard error on the mean of the difference between the two over runs.
For CIFAR- 10, there
have listed the constant and optimal switching performance
Aug name long
Constant Aug
Turn Aug Off
Switching Lift Er
Clean Blur(100%)
0.90
0.54
0.89
1.1e-02
Blur(50%%
0.89
0.90
1.6e-03
Crop(4, 25%)
0.93
0.93
2.7c-04
Crop(4, 50%)
0.93
0.93
2.5c-04
Crop(4, 75%)
0.93
0.93
3.1e-04
Crop(4.10056)
0.93
0.93
3.3e-04
Cutout(16 16, 100%)
0.92
0.92
5.1c-04
Cutout( 16, 25%)
0.92
0.92
2.5c-04
Cutout( 16, 50%)
0.92
0.92
6.8c-04
Cutout( 16, 75%)
0.92
0.92
6.5e-04
(g00l
0.81
0.90
0.8e-03
(Q0al
0.89
0.90
1.5e-03
FlipLR(100%)
0.89
0.92
1.4e-03
F2)5LR(25%)
0,93
0.93
3.0e-04
(lipLR(50%)
0.93
0.93
1.9c-04
FlipLR(50%) Crop(4, 100%) + Cutout(16, 100%)...
0.95
0.95
4.0c-04
FlipL.(50%% Crop(4, 100%)
Cutout(16, 100%)...
0.95
0.95
4.4c-04
FlipLR(50%% Crop(4,100%)
0.95
0.95
4.8c-04
FlipL.R(755))
0.93
0.93
2.0e-04
FlipUD(100%)
0.40
0.90
2.
19

---

Published as a conference paper at ICLR 2021
FlipUD(25%)
0.90
0.91
6.5c-04
FlipUD(50%)
0,90
0.91
7.3c-04
Fl)pUD(75%)
0.89
0.91
1.0c-03
Inver(100%%)
0.58
0.90
4.3c-03
Invert(50%)
0.88
0.90
2.0c-03
Rotate(fixed, 15deg, 50%)
0.93
0.93
2.9c-04
Rotate(fixed, 20deg, 100%)
0.85
0.92
1c
Rotate(fixed, 50%)
0.93
0.93
3.0e-04
Rotate(fixed, 20deg, 75%)
0.92
0.93
3. le-40
Rotate(fixed, 45deg. 50%)
0.92
0.92
3.7e-04
Rotatc(fixed, 5deg. 75%)
0.92
0.92
1.6c-04
Rotatc(square, 100%)
0.90
0.92
9.5c-04
Rotatc(square, 50%)
0.91
0.92
7.5c-04
(oriaaeva 20deg, 100%)
0.93
0.93
3. 3.2e-04
Rotate(variable, 20degg, 50%)
0.93
0.93
0.8
Rotate(variable, 20deg, 75%)
0,93
0,93
2.5e-04
Rotatc(variable, 45, 100%)
0,92
0.93
1.7c-03
Rotatc(variable 5deg, 100%)
0.92
0.92
2.0c-04
Rotatc(variablee 5deg. 50%)
0.92
0.92
2.7c-04
Rotatc(variable 5deg, 75%)
0.92
0.92
1.6c-04
Rotate(variable 60deg, 100%)
0.92
0.93
5.3c-04
ShearX(fixed 10.1, 100%)
0.91
0.92
8.0e-04
ShearX(fixed, 50%)
0.92
0.92
3.2e-04
ShearX(fixed, 0.1, 75%)
0.92
0.92
0.4
ShearX(fixed, 0.3, 100%)
0,90
0,91
Ie-00
ShearX(fixed, 0.3, 50%)
0,92
0.92
4.2c-04
ShearX(fixed, 0.3, 75%)
0.92
0.92
2.6c-04
ShearX(variable, 0.1, 100%)
0.92
0.92
3.2c-04
ShearX(variable, 50%)
0.92
0.92
3.0e-04
SvearX(variable, 0.1, 75%)
0,92
0.92
2.9e-04
ShearX(variable, 0.3, 100%)
0.92
0.92
3.6e-04
She ShearX(variable, 0.3, 50%)
0.92
0.92
2.4c-04
ShearX(variable,, 75%)
0.92
0.92
3.4c-04
FlipLR(25%) Crop(4.25%) + Cutout(16.255%)
0.95
0.95
2.3e-04
Fl.5L.R(25%) Crop(4,25%) + Cutout((6650055)
0.95
0.95
3.7e-04
Fl.5l.R(255) Crop(4,25%) + Cutout(16,7555
0.95
0.95
3.5c-04
((25%,)(25%)
Crop(4,25%) Cuttou(16, 100%%)
0.95
0.95
0.6c-04
F225L.R(25%)
Crop(4,50%) (1tour((1,255%)
0.95
0.95
3.7c-04
FlipLR(25%)
Crop(4.50%% + Cutout((6.505)
0.95
0.95
4.7e-04
Fli5LR(25%)
Crop(4.50%) + Cutout( (utout(16,7556)
0.95
0.95
3.6e-04
Crop(4.50%) )+ + Cutout(1 6, 100%) FlipLR(25%)
0.95
0.95
3.0e-04
((25L Crop(4,75%)
1utour(16,2555))
0.95
0.95
0.4c-04
Crop(4,75%) + Cutour(16.50%) (lipLR(25%)
0.95
0.95
0.6c-04
Fl25LR(25%)
+ Cutout(1 Crop(4,75%)
0.95
0.95
3.3e-04
Fl.5L.((55%) + Crop(4,75%) + Cutout(16, 100%)
0.95
0.95
5.4c-04
Crop(4.100%) Cutout( 6.25%) F(25LR(25%%
0.95
0.95
3.2e-04
Crop(4,100%) Cutout( 66.00% FlipL.((25%)
0.95
0.95
3.3c-04
Crop(4.100%) Cutout( (moout(66.555)) 6.75%) FlipLR(25%)
0.95
0.95
-45e-04
Crop(4.100%) + Cutout( (roout(16,10)) 10.00 FlipLR(25%)
0.95
0.95
0.4e-04
(500L
Crop(4.25%) + Cutout(16, 16.55%)
0.95
0.95
2.3e-04
Crop(4,25%) (50%)
(1,0ut(165506))
0.95
0.95
2.2c-04
(50%)))50%))
Crop(4,25% + Cutout(1 (00uu(11,,75500 6,75%)
0.95
0.95
4.7c-04
Crop(4,25%% + Cutout(1 100%) F(50l.(550%)
0.95
0.95
3.5c-04
FlipLR(50%)
Crop(4.50%% + (0oout()62255))
0.95
0.95
le-4-
Crop(4.50%) )+ + Cutout(1 6,50%) F(ipLR(50%)
0.95
0.95
3.8c-04
Crop(4.50%) F50LLR(50%)
Cutout((1,,755)
0.95
0.95
0.7e-04
Crop(4,50%) + Cutout(16,1000) (l0pLR(50%)
0.95
0.95
5.6c-04
Crop(4,75%) + Cutout(16.255% FlipLR(50%)
0.95
0.95
3.2e-04
Crop(4.75%) + Cutout(16,500) (500L)(50%)
0.95
0.95
4.3c-04
(50%L
Crop(4,75%) Cutout(16 6,75%)
0.96
0.95
6c40-0
((50L,)(50%)
Crop(4,75%) Cutout(16, 100%)
0.95
0.95
4.2c-04
F(i0L.R(50%)
Crop(4, 100%% Cutout( 16.25%)
0.95
0.95
3.7e-04
FlipLR(50%)
Crop(4.100%) + Cutout( 16,5%%
0.95
0.96
3.6e-04
F500LR(50%)
(1op(4.100%) Cutout( (6.55%
0.95
0.95
2.8e-04
FlipLR(50%)
Crop(4.100%)
Cutout(16.
0.96
0.96
0.
20

---

Published as a conference paper at ICLR 2021
FlipLR(75%) Crop(4.25%) + Cutour(16,255%
0.95
0.95
2.7e-04
Crop(4.25%) + Cutour(16,505%) FlipLR(75%)
0.95
0.95
5.0c-04
FlipLR(75%)
Crop(4.25%) + Cutout(16,755)
0.95
0.95
3.9c-04
Crop(4,25%) + Cutour(16,1000))) 100%) FlipLR(75%%
0.95
0.95
3.3c-04
Crop(4.50%) + Cutour(16,255)
0.95
0.95
2.6c-04
Crop(4.50%) Fl)5L.R(75%)
Cutour(16,505%)
0.95
0.95
3.7c-04
Fl)pI.((55%)
Crop(4,50%) + Cutout(16,75535
0.95
0.95
3.4e-04
FlipLR(75%)
Crop(4.50%) + Cutout(16. 1utout(16,10000)
0.95
0.95
3.5e-04
FlipLR(75%)
Crop(4.75%) )+ + Cutour(16,2555))
0.95
0.95
0,6e-04
FlipLR(75%)
Crop(4.75%) Cutour(16_500%)
0.95
0.95
0.6e-04
F(75l.R(75%)
Crop(4,75%) + Cutout(1 (, 6,75%)
0.95
0.95
3.4c-04
Crop(4,75%) + Cutout(1 6, 100%) Fl.5L.((75%)
0.95
0.95
3.0c-04
Crop(4,100S, FlipL.((55%)
+ Cutout( 6,25%)
0.95
0.95
4. 4.3c-04
FlipLR(75%)
Crop(4,100%) Cutout( 66.50%
0.95
0.95
2.7e-04
FlipLR(75%)
Crop(4.100%% Cutout( 6,75%)
0.95
0.95
4.5c-04
FlipLR(75%) Crop(4.100%) Cutout( 6, 100%)
0.95
0.95
3.3e-04
FlipLR(100%) + Crop(4,25%) + Cutout( 6,25%)
0.93
0,94
7. 7,9c-04
FlipLR(100%) + Crop(4.25%) Cutout( 6,50%)
0.94
0.94
0.7c-04
FlipLR(100%) + Crop(4,25%) Cutout( 6, 6.75%)
0.94
0.94
7.
Fli00_ + Crop(4,25%) Cutout( 6010%)
0.94
0.94
5.3c-04
+ Crop(4.50%% FlipL.((100%)
+ Cutout( 1 6,25%)
0.94
0.94
5.8c-04
+ Crop(4,50%) FlipI_R(100%)
+ Cutout( 6,50%)
0.94
0.94
5.2c-04
+ Crop(4.50%) FlipLR(100%)
Cutout( 16 6.75%)
0.94
0.94
5,0c-04
FlipLR(100%% + Crop(4.50%) Cutout( 6.100%
0.94
0.94
0.8--04
FlipLR(100%) + Crop(4.75%) + Cutout( 6,25%
0.94
0.94
5.0e-04
FlipLR((10%%) )+ + Crop(4,75%) + Cutout( 6,50%)
0.94
0.94
7.5c-04
+ Crop(4,75%) FlipLR(100%)
+ Cutout( 6,75%)
0.94
0.95
4 4,4c-04
FlipLR((10%%) + Crop(4,75%) Cutout( 6, 100%)
0.94
0.95
6.6c-04
FlipLR(100%) Crop(4,100%) Cutout( (6,5%)
0.94
0.94
5.3e-04
FlipLR(110%%) + Crop(4,100%) + Cutout( 6650%)
0.94
0.94
4e70-04
FlipLR((00%) + Crop(4,100%) + Cutout( 6,75%%)
0.94
0.95
6.5e-04
FlipLR(100%) + Crop(4.100%) + Cutout( 16, 100%)
0.94
0.95
0.9c-04
Rotate(fixed, 5deg, 10%)
0.92
0.92
0.9c-04
Rotate(fixed, 5deg. 20%)
0.92
0,92
3.6e-04
Rotate(fixed, Sdeg, 30%)
0.92
0.92
3. lc-04
Rotate(fixed, 5deg. 40%)
0.92
0.92
1 1.9c-04
Rotate fixed, 5deg, 50%)
0.92
0.92
3.0c-04
Rotate(fieed, 5deg. 60%)
0.92
0.92
2.6c-04
Rotate(fixed, 5deg. 70%)
0.92
0.92
2.9c-04
Rotate(fixed 5deg, 80%)
0.92
0.92
2.3e-04
Rotate(fixe) 5deg, 90%)
0.92
0.92
3.4e-04
Rotate(fixed. 5deg, 100%)
0.92
0.92
0.5c-04
Rotatc(fixed, 60deg, 10%)
0.91
0.91
0.3c-04
Rotate(fixed, 60deg. 20%)
0.91
0.91
6.0c-04
Rotate fixed, 30%)
0.91
0.92
4.7c-04
Rotate(fixce) 60deg, 40%)
0.91
0.92
5.0e-04
Rotate(fixed, 00deg,50%%
0.92
0.92
1.9c-03
Rotate(fixed, 60deg, 60%)
0.91
0.91
5.3e-04
Rotate(fixed, 60deg. 70%)
0.91
0.92
1.4e-03
Rotate(fixed 60deg, 80%)
0.91
0.92
1.9e-03
Rotate(fixed, 60deg, 90%)
0.90
0.91
1.5c-03
Rotatc(fixed, 60deg, 100%)
0.41
0.91
3.9c-03
Patch(raussian(ffccd.44. 0.1, 100%)
0.90
0.90
4.3c-04
(aa.aausiinnnixee.4.... 0.2, 100%)
0.90
0.90
3.5e-04
PatchGaussianniiieee.e 0.3. 100%)
0.90
0.90
3.0e-04
PatchGaassiaan(ixxed... 0.5, 100%)
0.91
0.91
2.6c-04
Patch(Gasssian(iixcdd.44 0.8, 100%)
0.91
0.91
3.6e-04
Patch 1.0, 100%)
0.91
0.91
2.8c 2.8c-04
PacchGaassiannffice... 1.5, 100%)
0.91
0.91
4.2c-04
PatchGGaussian(fixed..4 2.0, 100%)
0.91
0.91
4.4c-04
PatchhGasssinnffxcd... 0.1, 100%)
0.90
0.90
0.6c-04
PatchGaasssann(fiee... 0.2, 100%)
0.91
0.91
2.8--44
PatchGaussian(tixed.. 0.3, 100%)
0.91
0.91
-.5e-04
PatchGaussian(fixer 0.5, 100%)
0.91
0.91
3.0e-04
PatchGaussian(fin 8i 0.8, 100%)
0.92
0.91
0.
21

---

Published as a conference paper at ICLR 2021
PatchGaussian(ixed..8 1.0, 100%)
0.92
0.91
0.6
Patch(Gassianmfxccd.8 1.5, 100%)
0,92
0.92
4.6c-04
PatchGaassian(fxeee.. 2.0, 100%)
0.92
0.92
0.8c-04
PatchGausiannfixed.ii 12, 0.1, 100%)
0.90
0.90
2.3c-04
PatchGaussian(ixedd.2 12, 0.2, 100%)
0.91
0.91
4.3c-04
PatchGGassiannfxxet.22 12, 0.3, 100%)
0.91
0.91
5.2c-04
PatchGGassiannfxxedd.2 12, 0.5, 100%)
0.91
0.91
4.4c-04
PatchGaassian(ixxdd..i 0.8, 100%)
0.91
0.91
6.7e-04
PathhGaussian(ixxed..2 12, 1.0, 100%)
0.92
0.92
0.9
PatchGaassiannfixee..11 1.5, 100%)
0.92
0.91
0.0c-04
PatchGGaussiannfixxc.122 12, 2.0, 100%)
0.92
0.92
7.3c-04
PatchGaaussianffxxed..6 16, 0.1, 100%)
0.90
0.90
5. 1c
aathhaaussinnffieee.tt 16, 0.2, 100%)
0.91
0.90
lc
PaichGaasssaan(iieed... 16, 0.3, 100%)
0.91
0.90
7 7.4e-04
PacchGaussiannfiixd...6 16. 0.5, 100%)
0.91
0.90
7.7e-04
Bachhaussianniixee.... 16, 0.8, 100%)
0,90
0.91
1.4e-03
PatchGaassiannticxcd.... 16, 1.0, 100%)
0,90
0.91
1.3c-03
PatchGaassianntixee... 16, 1.5, 100%)
0.90
0.90
7.5e-04
PatchGaasssiam(fxccd...6 2.0, 100%)
0.90
0.91
1.6c-03
Patch(Gasssiann(iccdd..222 0.1,, 100%)
0.90
0.90
1.3c-03
PatchGauussian(ixxed..00 0.2, 100%)
0.89
0.90
1.
Caichhaaussiann((xxd...2.......1 0.3, 100%)
0.89
0.90
1.6c-03
PacchGaussian(tixxe..2, 0.5, 100%)
0.89
0.90
1.6e-03
PatchGaussiann(ixxed.22 0.8, 100%)
0.88
0.90
1.8e-03
PacchGGaussianniixeed...., 1.0, 100%)
0.88
0.91
1e
PatchGaasssaan((ixccd.0000 1.5, 100%)
0.87
0.90
1.8c-03
PathhGaussian(ficee.220 2.0, 100%)
0.88
0.90
1,4c-03
Patch(Gaussian(ficcd.244 0.1, 100%)
0.89
0.90
9,7c-04
Patch(Gasssianniixed..24 0.2, 100%)
0.88
0.90
1.4e-03
PaichGaassiann(ixxed.444 0.3, 100%)
0.87
0.90
1.3e-03
(ai.ssaiaiaaimxxxe22444444 0.5, 100%)
0.86
0.90
2.0e-03
Paihh 0.8, 100%)
0.84
0.90
1 1.6e-03
PatchhGaussian(tiieed.244 1.0, 100%)
0.84
0.90
1.7e-03
PatchhGaussian(iixee...44 1.5, 100%)
0.83
0.90
2.4c-03
PatchGaussianm(xxxc..4, 2.0, 100%)
0.82
0.90
9.6c-04
PatchGaaussiam(fixed.22 0. 100%)
0.88
0.90
1 1.0c-03
PatchGaaussiamffxxed..2 0.2, 100%)
0.85
0.90
1 1,4c-03
PatchGaasssian(ixxdd.22 0.3, 100%)
0.83
0.90
1.8c-03
PalchGGaussianniiieed.., 0.5, 100%)
0.81
0.90
1.7e-03
PatchGaasssian(iiiee..2.. 0.8, 100%)
0.79
0.90
2.4e-03
PatchGGaussann(ixxed.2,, 1.0. 100%)
0.77
0.89
1.7e-03
PatchhGaussiannhiiccd..... 1.5, 100%)
0.74
0.89
2.4c-03
Patch(Gaussian(iiicdd.2,. 2.0, 100%)
0.72
0.90
3.3c-03
PatchGGasssiamtiixed..22. 0.1, 100%)
0.88
0.90
1.8c-03
Machhauussian((iied..22. 0.2, 100%)
0.85
0.90
1.6e-03
PatchGaussianfixee..22 0.3, 100%)
0.82
0.90
1.8c-03
PatchGaaussian(xxxxd.2, 0.5, 100%)
0.79
0.90
1.8c-03
Patch[Gassianntixeed.332 0.8, 100%)
0.79
0.89
2.5e-03
PatchGaassianntixed.33 1.0, 100%)
0.79
0.89
2.0e-03
PatchGaussian(ixed.33 32, 1.5, 100%)
0.81
0.89
1.5e-03
PatchGaussianffxeee..3 2.0, 100%)
0.81
0.89
1.7c-03
E DIVERSITY METRICS
We computed three possible diversity metrics, shown in Fig. 9 Entropy, Final Training Loss, Training Steps to Accuracy Threshold. The entropy was calculated only for augmentations that have a discrete stochasticity (such as Rotate (fixed) and not for augmentations that have continuous variation (such as Rotate avariabler or Pat chGauss ian). Final Training Loss is the batch statistic at the last step of training. For CIFAR- 10 experiments, this was averaged across the 10 initializations. For ImageNet, it was averaged over the last 10 steps of training. Training Steps to Accuracy Threshold is the number of training steps at which the training accuracy first hits a threshold of 97%. A few of the tested augmentation (extreme versions of Pat chGuusss an) did not reach this threshold in the given time and that column is left blank in the full results.
22

---

Published as a conference paper at ICLR 2021
Steps to Training 97% Accuracy
10
10
60k
Entropy
Entropy
4
40k
0.2
0.4
0.1
0.2
40k
60k
Final Training Loss
Final Training Loss
Steps to Training 97% Accuracy
Figure 9: CIFAR-10: Three different diversity metrics are strongly correlated for high entropy augmentations. Here, the entropy is calculated only for discrete augmentations.
Entropy unique in that it is independent of the model or dataset and is counting of states. However, it is difficult compare between discrete and continuously-van varyynn transforms and is clear how proper it compare even across different types of transforms.
Final Training Loss and Training Steps to Accuracy Threshold correlate well across the tested transforms. Entropy is highly correlated to these measures for Fat chtcasuuusi sin and versions of F FlipLR, Crop, and Cutout where only probabilities varying. For Rctate and Shear where magnitudes are varying well, clear.
the correlation between Entropy the other two measures
or training step based Diversity measures that they require fully training a new model for each augmentation policy. To ameliorate this computational cost, we also consider using the training loss, but for models trained for significantly shorter time. In Figure ao]and Figure [T] We consider CIFAR-10 models trained for 4.5% and ImageNet models trained for 12% of the full training time. We find good correlation between this short time measure and training loss computed alternate computationally less-cxpensive measure we find that 98.9% of pairs of CIFAR- 10 models and 9.66% of ImageNet models satisfy Incquality B
One shortcoming
end of training. With
30
1.4
Relative Test Accuracy
25
Early Diversity
1.3
1.2
1.1
1.00
1.25
1.30
2.00
Affinity
Eaity Divesit
Figure 10: CIFAR-10: (Left) Diversity as the average loss over the first 10 epochs (out of 222) epochs of training (10 measurements, one per epoch). This represents an 96.5% reduction in computational cost. 98.99 98.9% of pairs satisfy Inequality [ (Right) Farly Diversity compared to Diversity computed for full training, this represents a Spearman correlation of 0.87
23

---

Published as a conference paper at ICLR 2021
-06
1.3
Relative Test Accuracy
13
Early Diversity
1.2
Dlirsiicy Eany
1.2
13
1.1
1.0
-70
Affinity
Diversity
Figure 11: ImageNet: (Left) Diversity as the average loss over the first 13000 (out of 112000) steps of training (5 measurements, per epoch), this represents an 88% reduction in computational cost. 96.6% of pairs satisfy Inequality (Right) Early Diversity compared to Diversity computed for full training. This represents Spearman correlation of 0.82.
COMPARING AFFINITY TO OTHER RELATED MEASURES
We gain confidence in the Affinity mcasure by comparing it to other potential model-dependant measures of distribution shift. In Fig T2 we show the correlation between Affinity and these two measures: the mean log likelihood of augmented test images(Grahhwoh al. 2019) (labeled "logsumexp(logits)) and the Watanabe-Akaike information criterion (labeled as "WAIC" [Watanahe] 2010).
Like Affinity, these other two measures indicate how well a model trained on clean data comprehends augmented data.
Ogssumeppllogits
18
Digsumeppllogitt
16
-1
WAIC
144
12
10
0.50
0.75 Affinity
1.00
0.50
0.75 Affinity
1.00
Affinity
(a) CIFAR-10
(b) ImageNet
Figure 12: Affinity correlates with two other measures of how augmented images are related to trained model's distribution: logsumexp of the logits (left, for CIFAR-10, and right, for ImageNet) is the mean log likelihood for the image. WAIC (middle, for CIFAR-10) corrects for possible bias that estimate. In all three plots, numbers are referenced to the clean baseline, which is assigned a value of O.
G FULL RESULTS
The plotted data CIFAR-10 and ImageNet are given in .csv files uploaded https: http://storrgggg
googleaple 3, cwm/ wwtpccwwccpccpccccctccctc.ccccpc libllic__e cerlllccreeerrree research_daaeeeeeeeeeeeeeeeeeese dathhddta//a/ augmentation /ooon/aaaa... zipp In these ,csv files, blank cells generally indicate that given experiment (such as switching) was not done for the specified augmentation. In the of the training accuracy threshold proxy for diversity, blank cell indicates that for the given augmentation, training accuracy not reach the specified threshold during training.
H MODEL ROBUSTNESS ON THE AFFINITY -DINITY-DIVESITYY PLANE
To shed light on whether Affinity and Diversity inform model robustness we evaluated our ImageNet and CIFAR- 10 models on ImageNet-Ve V2 and CIFAR- 10.1 (Recht et al. 2019 2018) as well as on CIFAR-I0- 10-C (Hendrycks & Dietterich] 2019). The former two datasets attempt recreate the CIFAR-10 10 and ImageNet test closely possible, however model performance typically suffers these sets indicating some degree of
24

---

Published as a conference paper at ICLR 2021
distribution shift. CIFAR-IO-C is dataset designed model robustness variety of image corruptions. We display accuracy for cach individual corruption Tigure[66Jam the overall CIFAR-IO-C score Figure We find that for both ImageNet- V2 and CIFAR-10.1 and the robustness gap (difference between clean and robust accuracy) show consistent trends with higher diversity and higher affinity augmentations leading to more robust performance. For some of the CIFAR-10- corruptions
see similar trend. For other augmentations, (e-g- Gaussian noise, impulse noise, speckle and shot noise) we that models trained with patch Gaussian augmentations (red-circles) outperform this general trend. This perhaps consistent with picture models trained with augmentations well paired with specific corruption perform well (Yin et al. 2019) Anonymous 2021
CIFAR-10.1
CIFAR 20
Diversity
CIFAR-10 Accuracy
100 Exaaa
Accuracy
-50
60
70
Affinity
Affinity
CPAR-10 Accuraoy
Figure 13: (Left) Augmented model performance on CIFAR-10 10.1 relative to a clean baseline (acc 80.1). We see that robust accuracy higher for models trained with more diverse higher affinity augmentation policies. (Middle) Gap in performance between CIFAR-10. 10.1 and CIFAR-10 accuracy. (Right) CIFAR-10 10.1 accuracy versus CIFAR- 10 accuracy.
Accnracy ZA-aNaBeew
1.6
1.6
gd db eistiibe
0
60
Diversity
1.4
Diversity
1.4
ImageNet->e
-10
20
1.0
1.0
-60
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
Affinity
Affinity
ImageNet Accuracy
Figure 14: (Left) Augmented model performance on ImageNet-V V2 relative to clean bascline (acc 63.7). We see that robust accuracy higher for models trained with more diverse higher affinity augmentation policies. (Middle) Gap in performance between ImageNet- V2 and ImageNet accuracy. The gap diminishes for models trained with high diversity, high affinity augmentations. (Right) ImageNet- V2 accuracy versus ImageNet accuracy.
25

---

Published as a conference paper at ICLR 2021
CIFAR-10-C MCE
Diversity
3.0
2.0
-097
Affinity
Figure 15: Mean corruption eπrоr on CIFAR-10-C (lower is better): Augmentations with high Affinity and high Diversity generally lead to lower mean corruption scores, and thus better robustness performance. Patch Gaussian augmentations (red circles), however, outperform this trend, potentially due to the similarity between some the corruption transformations and Gaussian noise.
26

---

Published as a conference paper at ICLR 2021
Esanubb 3 bweid
Diversity
Dovrrsig
Oxvers
0.6
0.8
1.0
0.6
0.5
1.0
0.6
0.8
1.0
Affinity
Affinity
Affinity
Ardden 5 ot CVI
Diversi
0.6
0.5
Affinity
Affinity
Affinity
5.00
60) G
Diversity
Diversity
Affinity
Affinity
Affinity
uogou Soo--
Badd B oat dval
Diversity
Diversiu
Dnverss
phooltte
6.8
6.8
1.0
0.6
0.8
1.0
0.6
0.8
1.0
Affinity
Affinity
Affinity
Egedmass 5 WY3
5
Diversity
Diversin
Diversty
Aftinity
Aftinity
Atfinity
3-D 5
Diversity
Diversity
2i impulse
3.0
1.0
Affinity
Affinity
Affinity
Figure 16: Robustness performance on CIFAR-10-C: Figures show corruption error for 18 different corruption types averaged over five corruption strengths and compared the baseline model, AlexNet. used by Hendrycks & Dietterich 2019). Robust performance for some corruptions, such as brightness, maximal for models trained with high Affinity high Diversity augmentations. For other corruptions, such Gaussian noise or speckle, we do not see this. Red circles indicate models trained with patch Gaussian augmentation which are particularly robust to paired families of augmentations.
27


Reviews:
[{"review_id": "u5VDDAi_qVt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Edit after authors' responses\n\nI have upgraded my score (from 4 to 5) based on the clarifications provided by the authors and the updated manuscript. Please see the details in my extended comments: https://openreview.net/forum?id=ZcKPWuhG6wy&noteId=V7Wy0Mpsz7Q\n\n## Summary of the paper\n\nThis paper proposes two metrics, affinity and diversity, for assessing the value and contribution of data augmentation strategies (single transformation or combinations of them). Given a model trained on a clean data set (without data augmentation), the affinity of an augmentation strategy is defined as the difference between the accuracy of the model on augmented validation data, minus the accuracy on clean validation data. The diversity of an augmentation strategy is defined as the final training loss of a model trained with data augmented according to that strategy. The paper presents an empirical analysis of the affinity and diversity of a set of image augmentation strategies evaluated on a network architecture trained on CIFAR-10 and one trained on ImageNet. The main conclusion is that the contribution to a model's performance of an augmentation strategy is predicted by its joint affinity and diversity, but not separately.\n\n## Summary of merits and concerns\n\n### Merits\n\n+ The paper's overarching motivation of quantifying the usefulness of data augmentation strategies is interesting and definitely important, given the renewed interest in data augmentation by the machine learning community.\n+ The proposal of quantifying the data distribution shift and complexity (affinity and diversity, respectively) introduced by an augmentation strategy is reasonable, interesting and well motivated.\n+ The introduction of the problem and motivation for the proposal (Introduction) is comprehensible and interesting and the review of related work is exhaustive and relevant. This part of the paper is very well written and I quite enjoyed reading it.\n+ In the rest of the paper, the concepts and definitions introduced are easy to understand and the methods employed for empirically assessing the affinity and diversity are generally clear.\n\n### Concerns\n\n- I see some issues in the specific definition of affinity and diversity. Summarised (extended below), first, while the affinity can be easily computed for any augmentation strategy and pre-trained model, the diversity is computationally costly as it requires re-training the model;  second, in their current definition, the dependence on the specific model and data set makes the metrics hard to compare across models and data sets; third, affinity and diversity are defined in very different ways, one in terms of the accuracy of a pre-trained model, the other in terms of the training loss.\n- Although generally clear, the methodology employed falls short at demonstrating the contributions stated in the introduction and portraying a complete picture of how affinity and diversity can be used to assess the value of data augmentation strategies. We do gain some insights, but I have some concerns about the methodology and some important questions remain open.\n- One motivation for the introduction of metrics to quantify the merit and mechanisms of data augmentation strategies is that the reasons why cutout, SpecAugment and mixup work so well are not well understood. Another one is that (so-called) automatic data augmentation strategies, such as AutoAugment, are hugely computationally expensive. However, the paper does not really discuss how affinity and diversity explain the mechanisms of these augmentation strategies and how they can be used to discover new strategies more efficiently.\n- The presentation of the results, especially in the figures, can be improved, in my opinion, and hinders the clarity of the paper.\n\n## Evaluation and justification\n\nWhile acknowledging the merits and contributions of the paper, my concerns outweigh the positive aspects of the paper and hence my recommendation of rejection. I will discuss next in more depth these concerns in order to better justify my recommendation and with the intention to provide constructive feedback for potential subsequent work on the paper.\n\n### Definitions of affinity and diversity\n\nThe definitions of affinity and diversity proposed in this paper are intuitive, easy to understand and reasonable. Furthermore, as stated above, I agree that quantifying the concepts and intuitions behind these metrics is an important contribution. In fact, the main result that the value of an augmentation strategy depends on both its affinity and diversity matches my (and the author's) intuitions and expectations. However, I have several concerns about the specific way these quantities are defined and computed in the paper.\n\nFirst of all, since one of the motivations for proposing such metrics is to more efficiently discover new augmentation strategies and assess the existing ones, I think that an important feature of the metrics should be the efficiency and ease of computation. However, computing the diversity of an augmentation strategy for a model requires training the model end-to-end. This is not the case of the affinity, which can be computed for any pre-trained model, and I think a useful definition of the diversity should achieve the same goal.\n\nSecond, in the way affinity and diversity are defined in the paper, it is hard to compare augmentation strategies across models and data sets, even though they are identical, that is an image rotation is applied in the same way on CIFAR-10 and on ImageNet; on ResNet and on DenseNet. Thus, it would be desirable if comparisons across models and data sets would be possible. This mismatch is in fact reflected in the presentation of the results, hindering the clarity. For example, in Figures 3b and 3c, the colour codes represent very different things as the range varies in one case from -50 to +7 and in the other case from -70 to +0.6. Green dots in Figure 3b (CIFAR-10) represent augmentation strategies that improve the accuracy with respect to the baseline, while green dots in Figure 3c (ImageNet) correspond to strategies that hinder the performance. Furthermore, the range of affinity and diversity also differs greatly between the two plots. This has to do in part with a likely suboptimal way of presenting the results (more about this below), but also with the fact that the affinity and diversity in ways that are not comparable across models and data sets.\n\nOne reason why the metrics are not comparable is that they are defined in absolute terms, without any normalisation that cancels the dependence on specific aspects of the model and data set, such as the loss, the number of classes (which determines the loss and accuracy), etc. Moreover, the authors chose to define the affinity and diversity in terms of the (top-1) accuracy and the (cross-entropy) loss, respectively. However, other researchers that may wish to use these metrics in the future might prefer to assess their models using an alternative metric, such as the top-5 accuracy, commonly used for ImageNet, or trained their models with a different loss. Such choices would also affect the interpretation of affinity and diversity and complicate even further the comparisons. \n\nA suggestion would be to define the metrics in relative terms instead. Without claiming that these suggestions would be optimal, the affinity could be computed, for instance, as the accuracy on the augmented data divided by the baseline accuracy on the clean data (and optionally multiplied by 100 to turn it into a percentage): $\\mathcal{T} = A(m, D') / A(m, D) * 100$. This would represent the fraction of the percentage of the accuracy obtained by testing on augmented images. These would reduce the dependence on the specific metric (accuracy) and on the characteristics of the model and data set. This has been used for instance in [[1]](#references) to also compared the contribution to performance of models trained with different data augmentation strategies.\n\nThird, affinity and diversity are defined in very different ways. Intuitively, the concepts that these quantities aim to represent are both related to the data distribution: affinity is related to the shift in the data distribution introduced by an augmentation strategy; diversity is related to the complexity in the distribution introduced by the augmentation. However, the former is defined in terms of the accuracy of a pre-trained model with respect to a baseline, the other in terms of the absolute training loss achieved by training with the augmentation. These are very different, unrelated quantities. Again, without claiming that the following suggestion is optimal, one idea would be to define the diversity as measure of spread (standard deviation, variance). Have the authors considered defining the diversity along the lines of the variance of the affinity or of a related quantity? This would quantify the diversity of an augmentation strategy and at the same time remove the need to train a model end-to-end, as discussed in the first point.\n\nThe authors briefly discuss entropy as an alternative measure of diversity, despite the problem of computing it for continuously-varying transformations. This is also an interesting direction which could be worth exploring.\n\n### Results do not demonstrate all contributions\n\nThe authors list four main contributions of their paper in the introduction. The first one is the introduction of affinity and diversity as \"interpretable, easy-to-compute metrics for parametrizing augmentation performance\". This is satisfied, although I have discussed above some concerns about the definitions. \n\nThe second contribution is that \"performance is dependent on both metrics\". This is indeed reflected by the results in Figures 3b and 3c. However, I should also note in this regard that despite the large number of augmentation strategies analysed, only two network architectures, each trained on one data set, are included in the experimental setup, and some differences between the two plots could already be discussed. The claim that the performance gain introduced by an augmentation strategy increases when both the affinity and the diversity increase is well supported by the results in Figure 3b (WRN on CIFAR-10), but this is not so clear in Figure 3c (ResNet on ImageNet). As a matter of fact, the relative test accuracy in Figure 3c seems to be higher (more yellow) as diversity decreases, rather than increases, and affinity increases. It would be desirable to obtain similar results on other architectures and data sets in order to gain more evidence to support this claim. \n\nRelated to this point, I would like to note that having two sets of results (WRN on CIFAR-10 and ResNet on ImageNet), presented in Figure 3, the authors select one of them (WRN on CIFAR-10, Figure 3b), the one, out of two, that best supports the claim, for Figure 1 on the first page of the paper. This clearly introduces a selection bias that distorts the actual data (i.e. why not showing in Figure 1 the results on ImageNet?). Moreover, the range of the axes differs between Figure 1 and Figure 3b, and there is even a third version in Figure 7, at the supplementary material. I would appreciate it if the authors can comment on/clarify this.\n\nThe third contribution claims to \"connect augmentation to other familiar forms of regularization\". This point is addressed in Section 4.2, where the authors evaluate the performance of the models trained with data augmentation after turning off the augmentation partway through training. Although the phenomenon that performance sharply improves after turning off regularisers partway is interesting, this has been observed in previous works (reviewed by the authors) and the analysis in this paper, through the lens of affinity and diversity, does not provide new, significant insights, in my opinion. Further, taking into account the claim in the list of contributions, the analysis offers little insight about the connection of data augmentation with other forms of regularisation, beyond noting that the \"slingshot effect\" occurs also with data augmentation, which had been observed before.\n\nI would like to draw the attention to certain aspects of the methodology in this section that might distort the conclusions. For example, the authors conclude from Figure 5b that \"For some poor-performing augmentations, [switching off the augmentation] can actually bring the test accuracy above the baseline\". However, I would like to note that the authors report that in order to obtain these results, they tested multiple switch-off points and select the one that yields the best accuracy. This introduces a clear bias in Figure 5b towards best-case scenarios, which might give the impression that the switch-off lift is a general effect or, in the best case, the magnitude of the effect will be magnified. In order to gain a more accurate picture of the phenomenon, all the available data should be considered and ideally a statistical analysis should be carried out. Another source of bias in the visualisation of the results is that in Figure 5c, \"Where Switch-off Lift is negative, it is mapped to 0 on the color scale\".\n\nOn the other hand, the switch-off lift effect could be explained in simpler terms, at least partially. For example, the authors write \"Bad augmentations can become helpful if switched off\". First of all, this could occur by pure chance and be reflected in the reported results as an artifact of the selection bias pointed out in the previous paragraph. Second, we should simply think that it is expected that turning off a bad augmentation should improve the accuracy. As a matter of fact, the augmentation strategies used to illustrate this effect are `FlipUD(100%)` (I will assume this means vertical flip) and `Rotate(fixed, 20deg,100%)`. In both cases, with the augmentation on, the model does not see the original images, so an improvement is expected if suddenly the model does see them. If the final accuracy is actually above the baseline should be analysed through a statistical analysis, rather than focusing on the best case, which might be due to pure chance. Moreover, it is also worth questioning the accuracy on clean images is actually a good baseline, since this model sees fewer different images.\n\nFinally, the authors claim that \"Switch-off Lift varies with Affinity and Diversity\", from the results in Figure 5c. However, from this figure we observe that mainly, it varies with affinity. This makes sense: very low affinity is indicative of unrealistic or at least odd augmentations, which are detrimental if performed during the whole training procedure. If turned off, the model has time to fine tune on the actual images. \n\nThe temporal dynamics of training neural networks and its relation to regularisation is an interesting, active topic of research, but due to its complexity it should be analysed very rigorously in order to minimise the risk of leading ourselves astray.\n\nThe fourth contribution claims that affinity and diversity informs that \"performance is only improved when a transform increases the total number of unique training examples\". This aspect is addressed in Section 4.3. The authors \"seek to discriminate this increase in effective dataset size from other effects\". Again, this is an important and interesting topic, as well as hard to analyse, but in my opinion the methods fall short at justifying the conclusions. Here, the authors simply trained models with static augmentation and found that \"For almost all tested augmentations, using static augmentation yields lower test accuracy than the clean baseline\", which is not surprising because the augmented images differ from the original validation/test distribution on which the model is evaluated. However, this observation does not prove that the gain provided by data augmentation, when it does improve the performance, is due to the increase in the effective training set size. To be clear, this is likely to be the case, intuitively, but should be proven differently.\n\n### Cutout, SpecAugment, mixup are not discussed in terms of affinity and diversity\n\nGaining insights about the mechanisms that make some data augmentation techniques (Cutout, SpecAugment, mixup) work better than others would be an interesting contribution. In the introduction, the authors mention this as a motivation for proposing the affinity and diversity metrics. However, although (some of) these augmentation strategies are included in the experiments, there is no specific discussion about them anywhere in the paper. Having mentioned this in the introduction as a motivation for the paper, I did miss a discussion that provided new insights about how these methods work and when.\n\nSimilarly, the authors motivate the proposal of affinity and diversity as a way to better understand the mechanisms of data augmentation and discovering new techniques more efficiently. However, beyond presenting the empirical results in Section 4, the paper does not further discuss use cases of affinity and diversity to efficiently assess the value of new techniques, or analyse commonly used strategies in terms of these metrics. For example, a widely used data augmentation strategy in computer vision is the combination of horizontal flips and vertical and horizontal translations of about 10 % of the height and width. This has been found to provide large performance gains [[1, 2, 3]](#references), while additional transformations only marginally improved the accuracy. Knowing the effectiveness of this simple augmentation strategy, it would also be interesting to analyse its affinity and diversity.\n\n### Visualisation of results\n\nAlthough this aspect has not been decisive in my evaluation of the paper, I think there is room for improvement regarding the visual presentation of the results and hopefully the following feedback, from a careful read of the article, may help make the paper stronger.\n\n- Figure 3a: it would help to more clearly specify, perhaps directly in the plots, that the top row corresponds to CIFAR-10 and the bottom row to ImageNet.\n- Figures 3b and 3c: I have commented above on this figure specifically, about the possibility of changing the definitions of affinity and diversity that would improve the comparison across models and data sets and hence the interpretability of these figures. In any case, a confusing aspect of these figures is that the colour codes define different ranges of values in each plot, which have semantically very different interpretations. For instance, light green values on 3b correspond to augmentations that improve the accuracy with respect to the baseline, while the same colour on 3c correspond to augmentations that perform worse than the baseline. Given that there is a clear central point in the colour code, zero, where the semantic interpretation changes (positive vs. negative), I would strongly suggest to use a [perceptually uniform diverging colour palette](https://seaborn.pydata.org/tutorial/color_palettes.html#perceptually-uniform-divering-palettes).\n- Figure 4: I would suggest to colour-code the dots to reflect the probability of rotation. It would reduce the cognitive load to interpret the figure.\n- Figure 5a: the legends could be placed inside the axes to make space for larger figures\n- Figure 5b: indicate what the dash line represents\n\n## Questions\n\nThe questions I list below are mainly intended to raise awareness about relatively minor aspects of the paper that remained unclear to me while reading it, with the aim of providing constructive feedback to potentially improve the manuscript. The aspects that have been more decisive in my decision have been already commented above.\n\n- \"Random crop was also applied on all ImageNet models.\": Why is not random crop considered data augmentation in this paper?\n- How would the authors explain the strange variation of diversity with respect to the probability of rotation in Figure 4 centre? Is this a general behaviour in other augmentation strategies? Is the affinity as clear in other cases?\n- Figure 5c: how is it possible to get an improvement (Switch-Off Lift) of 50 %, while in Figure 5b all cases show smaller variation?\n- The specific version of the wide residual network reported in the paper is \"WRN 28-2\" However, 28-2 is not described in the original WRN paper, and is also not described in the github repository of AutoAugment. I suppose that it should instead read WRN 28-10. Assuming this is the case, I have an additional question: WRN 28-10 achieves, according to my own implementation, around 91.5 % accuracy on CIFAR-10 without any data augmentation, using the hyperparameters and regularisation of the original paper. If I interpret correctly Figure 5b, the baseline accuracy achieved by the authors is 89.7, which is significantly lower. I would appreciate it if the authors could clarify what I may have misunderstood.\n- In Section 4.3, what is the goal of comparing models trained with static vs dynamic data augmentation? How would models trained with static augmentation be better or have larger diversity?\n- \"transforms and hyperparameters from the AutoAugment search space [...] implicitly have high Affinity\": this is not what we see in the Figure 3b and Figure 7. Could the authors clarify this statement?\n\n## Minor comments and potential typos identified\n\n- \"some have proposed that augmentation strategies are effective because they increase the diversity of images seen by the model\": any reference where this claim is made?\n- Make sure that \"dataset\"/\"data set\" are spelled consistency throughout the article.\n- Would the authors venture any guess about the affinity and diversity of strategies in other data domains?\n+ I appreciate that the authors report (Section 3) details about how the test results were computed, including the standard errors\n- \"static training\" (Section 3): Do the authors mean \"static augmentation\"? Also, consider giving an example for better illustration.\n- It is slightly confusing that the augmentation function is denoted by $a$ in Definition 1, which is an unusual choice for a function. Would a capital letter be a better choice, since augmentations are generally stochastic functions?\n- \"KL divergence of the shifted data with respect to the original data\": consider specifying this mathematically too\n- What is the reason for the capitalisation of _Affinity_ and _Diversity_?\n- Typo: \"model-dependant\" (Section 3.1)\n- The gap between Figure 5's caption and the paragraph seems to have been manually reduced. If this is the case, it may be against the formatting guidelines and, especially, it hinders the readability.\n- Some data augmentation strategies are mentioned without previously introducing them. For example, `FlipUD` in Section 4.2\n\n## References\n\n[1] Hernández-García, Alex, and Peter König. \"Data augmentation instead of explicit regularization.\" arXiv preprint arXiv:1806.03852, 2018.\n\n[2] Goodfellow, Ian, et al. \"Maxout networks.\" International conference on machine learning. PMLR, 2013.\n\n[3] Springenberg, Jost Tobias, et al. \"Striving for simplicity: The all convolutional net.\" arXiv preprint arXiv:1412.6806, 2014.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tHwNQrJwwNe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically investigates two crucial factors: affinity and diversity in useful data augmentation strategies. Through extensive experiments on existing image augmentation methods, it demonstrates that a good augmentation practice should bring high affinity and diversity for validation and training data. Specifically, it uses the accuracy gap between augmented and clean validation data to measure affinity. The diversity is measured by final training loss with the augmentations used in training.\n\nPros\n1. The paper is well-written and easy-to-follow.\n2. The proposed two measurements for affinity and diversity are easy to compute and observe. They are both model-based, making the measures more adaptive to model biases.\n3. The experiments test extensive augmentation methods to support the affinity and diversity claims.\n\nCons:\n1. Adversarial examples are perceptually similar to the original data but can result in low validation accuracy. The proposed affinity guide may not help guide robust model training.\n2. In Figure 3, the highlights of the three augmentation methods, i.e., Mixup, AutoAugment, and RandAugment, are not difficult to visualize.\n3. In Figure 4, one diversity value (x-axis) seems to correspond to multiple test accuracy measurements, which looks confusing.\n4. Although both affinity and diversity are essential to measuring augmentation quality, it is unclear how to use them to guide training. It seems that both positive and negative affinity values may help, according to Figure 3. Moreover, higher training loss indicates higher diversity. But high training loss may also mean a model still does not converge. It may be difficult for a new task with little experience to use the two metrics to select useful augmentation strategies.\n\nSummary\n\nThe paper investigates two important factors (affinity and diversity) in measuring the effectiveness of data augmentation. It also provides two simple metrics to measure them. My main concern is how to use them in practice to guide training. Moreover, the affinity measure seems not helpful for robust training since adversarial examples can easily fail a model, resulting in low validation accuracy. But using them in training can help advance model robustness. The paper would be better if it can address the two concerns. Overall, I recommend it for acceptance if it can include discussions on these two concerns.\n\nPost-rebuttal updates\n\nThank the authors for the efforts in answering the questions. The responses have addressed my concerns about the robust training and using training loss to measure diversity. Exploring the data augmentation for robust training will be interesting. Besides,  training loss is usually sensitive to some other hyper-parameters such as optimizer and learning rate. So, investigating the robustness of this metric is also meaningful.\n\nMoreover, there is still a gap from applying the proposed metrics to training guidance. The current research mainly shows there are relations between the two metrics and data augmentation effectiveness.  How to merge them into one easily observable is still missing. Therefore, I keep my original score.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tCS3MXCOhMq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper analyzes data augmentation for image classification using two measures: Affinity and Diversity. These measures depend on both training data and model and are easy to compute. The authors show that the performance of image classifiers depends on both of these measures through extensive experiments.\n\n# Strengths\n\n* The introduced measures consider both data and models, which are inseparable in modern deep learning.\n* The measures can explain why some data augmentation methods work and others are not, intuitively.\n* Affinity is easy to compute. To obtain Affinity, one needs a model trained on clean data and uses validation sets with a given augmentation. One can reuse the trained model to measure other augmentations.\n* The experiments are extensive.\n\n# Weaknesses\n\n* Diversity requires to train a CNN model with each augmentation configuration. This requirement restricts the crucial application of the paired measures to find better augmentation configurations given a dataset and a model under limited computational cost.\n* The slingshot effect in section 4.2 is interesting. But Fig 5 (c) shows a weak connection between the slingshot effect and the proposed measures. Instead, I think the results suggest the existence of other factors. Explaining this effect by the proposed measures only would be difficult, and I recommend removing this subsection.\n* Section 4.3 shows dynamic augmentations increase the effective training data size (compared to static augmentations), and thus the performance improves. I think this is well-known, and that's why we call data augmentation \"data augmentation.\"\n* Experiments are conducted only on CIFAR-10 and ImageNet. I know these datasets are the de-facto standard, but datasets from other domains are preferable to be included to support the claims.\n\n# Rating\n\n5. This paper includes interesting and useful insights, but its novelty is limited. Besides, its applicability is also restricted because of the computational intensity of the Diversity measure.\n\n# Feedbacks\n\n* Titles for the color bars in Fig 2 are missing.\n* It took moments to understand T and B in Figure 3 (a) are top and bottom.\n* The margin between the caption and the main text on page 7 is too small, which confused me during reviewing.\n* Reference information is old. Lim et al. 2019 is accepted at NeurIPS, and Hataya et al. 2020 is accepted at ECCV.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aVBrhnK9bM4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper studies the problem of data augmentation that obtains new training examples by modifying existing ones. Data augmentation is popular in machine learning and artificial intelligence since it enhances the number of training examples. However, its effect on model performance remains unknown in practice. An augmentation operator (e.g. image rotation) can be either helpful or harmful. This paper introduces two novel metrics, named affinity and diversity, to quantify the effect of any given augmentation operator. The authors find that an operator with high affinity score and high diversity score leads to the best performance improvement. \n\n\n##########################################################################\n\nReasons for score: \n\nOverall, I like the idea of this paper about making the effects of augmentation operators tractable. The proposed affinity and diversity scores are great indicators to evaluate the usefulness of an arbitrary operator. The finding that higher scores are better is insightful. With that, practitioners are able to inspect a large number of operators and select the most effective ones. \n\nHowever, the computation of affinity and diversity scores may be very expensive. The runtime could be more than that runs an operator directly to get the performance improvement. It will be good if the authors can compare the efficiency and/or share some comments in the rebuttal.\n\n##########################################################################\n\nPros:\n\n1. The paper proposes a novel idea of quantifying the effect of data augmentation. Specifically, the idea introduces two metrics, affinity and diversity scores, to evaluate any given augmentation operator in its effect on model performance. The experiments showed that higher the scores, better the performance improvement. The finding is novel that should be the first time in the field. \n\n2. The paper justifies that either affinity or diversity alone does not predict model performance. They together can make the prediction deterministic. The finding suggests that it is nontrivial to quantify the effect of augmentation operators. The paper nicely leverages heat maps as shown in Figure 3(b) and 3(c) to reveal the finding. \n\n\n##########################################################################\n\nCons: \n\nI have a concern about computation efficiency. It seems it takes significant time to compute the proposed affinity and diversity scores. For a given operator, Definition 1 regarding affinity requires running the model once. Similarly, Definition 2 regarding diversity requires running the model another time. So it needs to run the model twice to get affinity and diversity scores. Empirically, we can just run the model once to calculate the actual model improvement of the operator. If so, it is inefficient to use affinity and diversity scores for the purpose of predicting model performance. \n\n\n##########################################################################\n\nQuestions during rebuttal: \n\nIt will be good if the authors can add experiments or discussions related to efficiency in runtime. Or the authors can extend the use cases beyond model performance prediction.  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews were largely split in the beginning. Although all reviewers find that the idea of diversity and affinity measures for data augmentation is intriguing and potentially useful, they also raised many concerns such as computationally expensive nature of the diversity metric, lack of clear methodology of how to utilize those metrics to design augmentation strategies in practice, and weak organization and presentation of some experiments which are seemingly less related to the main point of the paper. During the discussion phase authors made significant efforts to improve the paper, and some of the concerns are favorably addressed. As a result, two reviewers raised their initial scores, yet we still think the paper is on the borderline. 

Overall, this paper presents an interesting and unique idea that potentially stimulate the community, while it also has some key weaknesses and has much room for improvements. Considering both pros and cons, we decided to accept the paper.
Decision: Accept (Poster)