How Much Data Are Augmentations Worth?  An Investigation into Scaling Laws, Invariance, and Implicit Regularization
Authors: Jonas Geiping
Micah Goldblum
Gowthami Somepalli
Ravid Shwartz-Ziv
Tom Goldstein
Andrew Gordon Wilson
Venue: ICLR 2023 poster (2023)
Keywords: Data Augmentations
Stochasticity
Flatness
Neural Networks
Invariance

Abstract: Despite the clear performance benefits of data augmentations, little is known about why they are so effective. In this paper, we disentangle several key mechanisms through which data augmentations operate. Establishing an exchange rate between augmented and additional real data, we find that in out-of-distribution testing scenarios, augmentations which yield samples that are diverse, but inconsistent with the data distribution can be even more valuable than additional training data. Moreover, we find that data augmentations which encourage invariances can be more valuable than invariance alone, especially on small and medium sized training sets. Following this observation, we show that augmentations induce additional stochasticity during training, effectively flattening the loss landscape.

Published as a conference paper at ICLR 2023
How MUCH DATA ARE AUGMENTATIONS WORTH? AN INVESTIGATION INTO SCALING LAWS, INVARIANCE, AND IMPLICIT REGULARIZATION
Jonas Geiping
Micah Goldblum New York University
University of Maryland, College Park
Gowthami Somepalli
Ravid Shwartz-Ziv New York University
University of Maryland, College Park
Tom Goldstein
Andrew Gordon- Wilson New York University
University of Maryland, College Park
ABSTRACT
Despite the clear performance benefits of data augmentations, little known about why they are so effective. In this paper, we disentangle several key mechanisms through which data augmentations operate. Establishing an exchange rate between augmented and additional real data, we find that in -fitt--distribution testing scenarios, augmentations which yield samples that are diverse, but inconsistent with the data distribution can be even more valuable than additional training data. Moreover, we find that data augmentations which encourage invariances can be more valuable than invariance alone, especially small and medium sized training sets. Following this observation, we show that augmentations induce additional stochasticity during training, effectively flattening the loss landscape.
INTRODUCTION
Even with the proliferation of large-scale image datasets, deep neural networks for computer vision represent highly flexible model families and often contain orders of magnitude more parameters than the size of their training sets. As a result, large models trained limited datasets still have the capacity for improvement. To make up for this data shortage, standard operating procedure involves diversifying training data by augmenting samples with randomly applied transformations that preserve semantic content. These augmented samples expand the volume of data available for training, resulting downstream performance benefits that one might expect from a larger dataset. However, the now profound significance of data augmentation (DA) for boosting performance suggests that its benefits may be more nuanced than previously believed.
In addition to adding extra samples, augmentation promotes invariance by encouraging models to make consistent predictions across augmented views of each sample. The need to incorporate invariances in neural networks has motivated the development of architectures that are explicitly constrained to be equivariant to transformations (Weiler & Cesa, 2019; Finzi et al., 2020). the downstream effects of data augmentations were attributable solely to invariance, then we could replace DA with explicit model constraints. However, if explicit constraints cannot replicate the benefits of augmentation, then augmentations may affect training dynamics beyond imposing constraints.
Finally, augmentation may improve training by serving as an extra source stochasticity. Under DA, randomization during training comes not only from randomly selecting samples from the dataset to form batches but also from sampling transformations with which to augment data (Fort et al., 2022). Stochastic optimization is associated with benefits in non-convex problems wherein the optimizer can bias parameters towards flatter minima (Jastrzębski et al., 2018; Geiping et al., 2021; Liu et al., 2021a).
In this paper, we re-examine the role of data augmentation. In particular, quantify the effects of data augmentation in expanding available training data, promoting invariance, and acting as a source of stochasticity during training. In summary:

---

Published as a conference paper at ICLR 2023
We quantify the relationship between augmented views of training samples and extra data, evaluating the benefits of augmentations as the number of samples rises. We find that augmentations
confer comparable benefits to independently drawn samples on in-domain test sets and even stronger benefits on out-of-distribution testing.
We observe that models that learn invariances via data augmentation provide additional regularization compared to invariant architectures and we show that invariances that are uncharacteristic of the data distribution still benefit performance.
We then clarify the regularization benefits gained from augmentations through measurements of flatness and gradient noise showing how DA exhibits flatness-secking behavior.
RELATED WORK
Data Augmentations in Computer Vision. Data augmentations have been a staple of deep learning, used to deform handwritten digits as early as Yaeger et al. (1996) and LeCun et al. (1998), or to improve oversampling on class-imbalanced datasets (Chawla et al., 2002). These early works hypothesize that data augmentations are necessary to prevent overfitting when training neural networks since they typically contain many more parameters than training data points (LeCun al., 1998).
We restrict our study to augmentations which act on single sample and do not modify labels. Namely, we study augmentations which can be written as (T(x), y). where (x, y) denotes an input-label pair, and T T is a random transformation sampled from distribution of such transformations. For a broad and thorough discussion on image augmentations, their categorization, and applications to com- puter vision, see Shorten & Khoshgoftaar (2019) and Xu et al. (2022). We consider basic geometric (random crops, flips. perspective) and photometric (jitter, blur, contrast) transformations, and common augmentation policies, such AutoAug (Cubuk et al., 2019), RandAug (Cubuk al., 2020), AugMix (Hendrycks et al., 2020) and TrivialAug (Müller& & Hutter, 2021) which combine basic augmentations.
Understanding the Role of Augmentation and Invariance. Works such König (2018) 88 propose that data augmentations (DA) induce implicit regularization. Empirical eval- uations describe useful augmentations as "label preserving", namely they do not significantly change the conditional probability over labels (Taylor & Nitschke, 2018). Gontijo-Lopes et al. (2020b;a) in- vestigate empirical notions of consistency and diversity, and variations in dataset scales (Steiner et al., 2022). They measure consistency (referred to as affinity) as the performance of models trained without augmentation on augmented validation sets. They also measure diversity as the ratio of training loss of model trained with and without augmentations and conclude that strong data augmentations should be both consistent and diverse, an effect also seen in Kim et al. (2021). contrast to Gontijo-LLopes et al. (2020b), Marcu & Prugel-Bennete (2022) find that the value of data augmentations cannot be measured by how much they deform the data distribution. Other work proposes to learn invariances parameterized as augmentations from the data (Benton et al., 2020), investigates the number of sam- ples required to learn an invariance (Balestriero et al., 2022b), uncovers the tendency of augmentations to sacrifice performance on some classes in exchange for gains on others (Balestriero et al., 2022a), or argues that data augmentations cause models to misrepresent uncertainty (Kapoor et al., 2022).
Grrnandez-Garcch &
Theoretical investigations in Chen et al. (2020a) formalize data augmentations as label-preserving group actions and discuss an inherent invariance-variance trade-off. Variance regularization also arises when modeling augmentations for kernel classifiers (Dao al., 2019). For binary classifier with finite VC dimension, the bound on expected risk can be reduced through additional data generated via augmentations until inconsistency between augmented and real data distributions overwhelms would-be be gains (He et al., 2019b). The regularizing effect of data augmentations is investigated in LeJeune al. (2019) who propose a model under which continuous augmentations increase the smoothness of neural network decision boundaries. Rajput et al. (2019) similarly find that linear classifiers trained with sufficient augmentations can approximate the maximum margin solution. Hanin & Sun (2021) relate data augmentations to stochastic optimization. A different angle towards understanding invariances through data augmentations is presented in Zhu et al. (2021), where the effect of DA in increasing the theoretical sample cover of the distribution is investigated, and augmentations can reduce the amount of data required, if they "cover" the real distribution.
Stochastic Optimization and Neural Network Training. The implicit regularization of SGD is regarded as an essential component for neural network generalization (An, 1996; Neyshabur et al., 2017). Stochastic training which randomizes gradients can drive parameters into flatter minima,
2

---

Published as a conference paper at ICLR 2023
0.9
Effeerive Extm Samples from Aug.
0.8
CINIC-10 Val.. Accuracy
0.7
0.8
0.5
Augmentation Strateyy
Baselise (No Aug)
0.4
Poweline (No Aug?
TippCrpp
0.3
FlppdCrop Hona, Flips
-50k
Elips
0.2
Perpective
Perqpection
-100k
500
1000
2000
5000
10000
20000
50000
100000 200020
50L
L00L
150k
200k
2506
3006
350k
Number Base Samples
Number of Base Samplees
gain terms of effective extra samples from Equation (1). Fitted curves marked in solid colors, with extrapolated regions dashed. Left: Numher of hase samples (from CINIC-10) on the logarithmic horizontal axis compared to validation accu- racy. The scaling bchavior of each augmentation is closely matched by these power laws. Right: Number of base samples compared effective extra data, showing how the henefits each data augmentation scale the model trained on more and more data. Policies that are strong peak benefit (at 50 000 base samples TrivialAug generates effectively 100 000 extra samples), but also fall off faster than consistent augmentations, such as horizontal flips, which provide benefits up to 350 000 base samples.
Figure Power laws f(x) = ax ax + for select augmentations applied randomly
inconsistent, such as TrivialAug, reach highest
associated with superior generalization (Jastrzebski et al., 2018; Huang et al., 2020; Liu et al., 202 la). In fact, Geiping et al. (2021) ) find that neural networks trained with non-stochastic full batch gradient descent require explicit flatness-seeking regularizers in order to achieve comparable test accuracy. Data augmentations provide an additional source of stochasticity during training on top of batch sampling, which we will investigate in this work.
We fuse together the above three topics and explore the role data augmentations holistically at scale. In doing so, we fill several gaps in the literature discussed in this section. Unlike work which quantifies data augmentations in terms of accuracy boosts for fixed sample sizes, compare the benefits augmentations to those achieved by instead collecting more data. While other works have studied the role of data augmentations in learning invariance, find that even invariances which have no relationship to invariances in the training data distribution are still effective. Finally, we develop an understanding of batch augmentation by showing that stochastically applied augmentations increase gradient noise during training, leading to qualitatively distinct minima.
AUGMENTATIONS AS ADDITIONAL DATA
central role of data augmentation is to serve as extra data and expand limited datasets used for training large models. In this section, quantify this property, conducting series of experiments culminating in measurements exchange rates, which indicate exactly how much data an augmentation worth the number of additional data samples which would yield the same performance gain the augmentation. Such exchange rates constitute a novel angle for quantifying the practical benefits of data augmentations and conceptualize qualitative notions consistency, diversity, and robustness to distribution shifts, and allow us to observe the properties of data augmentations over wider range of dataset sizes. We conduct these experiments on subsets of the CINIC-10 dataset (Darlow et al., 2018), a drop-in replacement for CIFAR-10 10 (Krizhevsky, 2009), which contains 200 000 samples. This allows us to train models with augmented data on subsets similar to CIFAR- 10, but compare to refer- ence models trained without augmentations larger datasets. We start with ResNet-18 18 architectures, evaluating their exchange rate behavior, but consider other architectures Section 3.1.2. Our dataset setup is quite specific require running a pairing such as ImageNet/JFT-300 (Sun et al., 2017), which would be prohibitive for all experiments in this work. We include a validation of results on ImageNet Appendix F and other datasets in Appendix B, to verify that the behaviors observed are not specific to CIFAR-10.
CEPAR-1//CINIC-II revisiting this experiment e.g. for ImageNet would
3.1 EXCHANGE RATES: How MANY SAMPLES ARE AUGMENTATIONS WORTH?
We visualize the validation accuracies for a range of models trained with select augmentations in Figure 1 (left). The validation behavior of these models can be well described by power laws of the form f(x a + b, describing the relationship between number of samples and validation
3

---

Published as a conference paper at ICLR 2023
accuracy. From these power laws, derive the exchange rates of various augmentations compared to the reference curve of un-augmented models fiet a'x + b'. For an augmentation policy
described by Fayg = oz + b, we define its exchange rate via
U'Ettcctive Extra Samples from Augmentations (z) = fret (faug
(I)
for base dataset size z. We visualize this quantity in Figure (right). This metric measures exactly the gain in accuracy between the un-augmented and augmented power laws shown the left-hand side and converts this quantity into additional samples by evaluating on how much more data the reference models would need to realize the same accuracy the augmented model.
core property of augmentations that becomes evident this analysis the relationship between consistency of augmented samples with the underlying data distribution (Taylor & Nitschke, 2018; Gontijo-Lopes et al., 2020b; He et al., 2019b) and diversity of extra data (GontijoLopes et al., 2020b). In Figure 1, we find that augmentation strategy, such TrivialAug (Müller & Hutter, 2021) (a policy consisting of a random draw from a table of 14 common photo- and geometric transforms, applied in tandem with horizontal flips and random crops), shown in orange, is highly diverse, generating a large amount of effective extra samples when the number of base samples is small. However, this policy also falls off quickest as more base samples are added: The policy is ultimately inconsistent with the underlying data distributions and when enough real samples are present, gains through augmentation deteriorate. On the flip side, augmenting with only horizontal flips, shown in green, is less diverse, and hence yields limited impact for smaller dataset sizes. However, the augmentation is much more consistent with the data distribution, and as such horizontal flips are beneficial even for large dataset sizes.
The impact of augmentations is linked dataset size; diverse but inconsistent aug- mentations provide large gains for smaller sizes but are hindrances at scale. Estimation of the value of future data collection should take this effect of augmentations on scaling into account.
Takeaway:
3.1.1 MEASURING DIVERSITY AND CONSISTENCY DIRECTLY
We can disambiguate the effects of consistency and diversity further by analyzing these augmentations for a moment not as randomly applied augmentations, but as fixed enlargements of the dataset, replacing each base sample with fixed number of augmented views. In Figure 2 (right), we see that a replacement of each sample with a single augmented view generated by TrivialAug is detrimental to performance (see single repetition plotted in red). As such, TrivialAug is inconsistent with the data distribution. On the other hand, evaluating the gains realized by replicating each sample in the dataset fixed number of times via the augmentation policy, we also see that TrivialAug leads to significantly more diversity in this controlled experiment, compared to random flips (Figure 2, left).
Another way to look Figure is again as a measurement of extra data. Simply replacing each base sample by single augmented sample is not a benefit (see single repetition plotted in red), yet we quickly find that large gains can be attributed simply to the duplication (green) and quadruplication (purple) of existing data and fixed multiplication can be worth substantial extra data.
Repeticions
Randon
Effective Samples Ang.
25k
Effective Extn Samples from Aug.
a0c
25k
20k
17kc
10k.
-100 Repeiiiions -- ()
15
Bardiam
93k
1500
50k
300k
Number of Base Samples
Number of Base Samples
Figure 2: Exchange Rates via Equation (1) when creating larger datasets from a fixed number repetitions of base samples via random data augmentations. Zero repetitions denote the unaugmented data. Left: Exchange rate for random horizontal flips as the augmentation policy is repeated. Right: Exchange Rates for TrivialAug. Even few repetitions of existing samples generate most of the effective additional data observed in Figure 1.
4

---

Published as a conference paper at ICLR 2023
1004
Effeerive Extrn Samples from Aug.
Effective Extn Samples
0 Width
-00k
-008
-100
-100k
16
-150
ComMiter
PytamidNNt
BowNes
-200
150
VGG13
50%
1003
150k
2006
2506
300%
50L
100k
150k
2008
2508
300k
Number of Base Samples
Number of Base Samples
Figure Exchange Rates via Equation (1) for TrivialAug when modifying the model architecture. Rates for various widths ResNet-18 (left). the default other parts of this work. Exchange Rates for select vision architectures (right) with similar parameter counts. Exchange rates behave similarly for a range of models architcctures, but quantitative benefits increase as models widen and capacity increases.
Buseline (Na Aug.)
Effective Exrra Samples from Aug.
Crpp&Crpp
Exfective Extr Samples from
2004
2004
Hariz, Flips
Pasqueco
Busclin Aus,)
-100k
-100k
Elips&Copa
Hort,,
-200k
-000k
Perspective
-000
-400A
-400
50%
100%
150k
200k
2508
3001
1000
10k
100k
Number Rase Samples
Number
Figure 4: Exchange Rates as Figure 1, but evaluated on CIFAR-10 validation (left) and CIFAR-10- (right). Note that for CIFAR- 10-C, inconsistent augmentations worth more than any number of additional in-domain samples, Augmentations really are worth much more, under slight (left) and large (right) distribution shifts.
3.1.2 EXCHANGE RATES FOR MODEL SCALING AND MODEL VARIATIONS
Figure 3 evaluates the effective samples gained from augmentation as model width increases (left) and model architecture changes (right). For both plots, the references fiet are based on the unaugmented models with that same configuration of width or architecture. We find that model width of the evaluated ResNet-18 reliably correlates with the gains from augmentations to larger dataset sizes. Evaluating different architectures, we find that while global behavior similar for all models, the exact exchange rates are similar for the convolutional architectures of ResNet, PyramidNet (Han et al., 2017) and VGG (Simonyan Zisserman, 2014), mirroring their closely related inductive biases. On the other hand, from the two transformer architectures, we find that a notably limited benefit from augmentations versus real data for the Swin- Transformer (Liu et al., 2021b; 2022), but a large benefit for the ConvMixer (Trockman & Kolter, 2022). In direct comparison of both architectures, the Swin- Transformer contains a large number of features designed specifically for vision tasks, whereas Con- vMixer is much more general, that their differing inductive biases again reflected in Figure 3.
Takeaway: Relative gains through augmentations sample sizes scale are broadly consistent across model widths and architectures, and absolute gains increase with model capacity.
3.2 EXCHANGE RATES IN OUT-OF-DISTRIBUTION SETTINGS
A benefit of extra data generated from augmentations that is often underappreciated is illustrated in Figure 4, showing exchange rates of the same models trained on CINIC- 10 as analyzed in Figure 1, but evaluated on the CIFAR-10 validation set (left). Comparing CINIC-10 10 and CIFAR- 10, both datasets are nearly indistinguishable using simple summary statistics (Darlow et al., 2018), yet there is minor distribution shift caused by different image processing protocols during dataset curation. In this setting,
5

---

Published as a conference paper at ICLR 2023
Repetitions
Repeiiiion
Rostom
251K
Effective Extm Samples from Aug.
2006
Effecivv Extra Samples
100k
600k)
1004
500
-200
-300
-200
50k
100k
150k
200k
250k
3001
1030
10%
100k
Number Base Samples
Number Base Samplees
Figure 5: Exchange Rates from fixed number of repetitions of samples via TrivialAug. "0" repetitions denote the unaugmented data. As Figure 2, evaluated on C-FAR-10 val. (Ieft) and CIFAR-10-4 10-C (right). find that even for this mild OOD shift, replacing each sample via augmentation (red) wins above 75 000 samples. For CIFAR-1O-C, even single repetition is worth more than any almost number of additional in-domain samples.
diverse augmentations are now quickly on-par with models trained on many more samples. Figure shows that this effect is apparent, even if base samples are replaced by fixed number of augmented samples. Four augmented samples are quickly worth more than additional real samples, and with enough base samples, even replacing each sample by fixed augmented version is beneficial. These effects can be exaggerated by evaluating on the CIFAR-IO-C dataset of common corruptions applied to CIFAR-10 10 (Hendrycks & Dietterich, 2018). There, quickly find that evaluating exchange rates for this large distribution shift, that the stronger augmentations evaluated, quickly produces benefits beyond what collection containing
substantial amounts of in-domain data would achieve.
The increased diversity in these augmentations broadens the support of the data distribution, and the support of the CIFAR- 10 dataset appears to be well contained within the transformed data. In practical applications domain adaptation (Zhang et al., 2019; Ahuja et al., 2021) and unsupervised learning (Chen et al., 2020b), actively broadening the support of the data distribution in the face of uncertainty is advantageous as suddenly each augmentation is effectively worth much more data.
Takeaway: Data augmentations broaden the support training data, which significantly extends their usefulness even larger dataset scales in OOD testing scenarios.
AUGMENTATIONS AND INVARIANCE
1204
RaNes-18
Reltaa wit Bacz Flg Augs
Eltective Eata Sungles.
The success of augmentations is often attributed the invariances encoded into the model by en- forcing the assignment of identical labels across transformations of each training sample. the success of data augmentations can be attributed solely to invariance, then we can build exactly invariant models that achieve comparable ac- curacy when trained without data augmentation. Several works propose such mechanisms for constraining neural network layers to be invari- ant, and will leverage these in our study.
EXCNN
56k
2008
2506
Number Sumples
Figure 6: Exchange Rates for horiz. flips of ResNet and various invariant architectures. All models have an equal number of base parameters and are based on a Resnet-18 18 template. Invariant architectures show opposite scaling behavior to augmentations.
4.1 INVARIANT NEURAL NETWORKS WITHOUT AUGMENTATION
In order to probe the benefits of invariance without augmentations, we evaluate the following three methods for constructing invariant networks, for the case of invariance to horizontal flips: Prediction averaging: Insert augmented views of a sample into the model and average the corre- sponding predictions (Simonyan & Zisserman, 2014). We use this procedure during both training and inference. and refer to the approach applied to a ResNet as flip-invarian ResNet-18.
Orbit Selection: An invariance also be enforced via orbit selection (Gandikota al., 2022).

---

Published as a conference paper at ICLR 2023
Kone
Bandam Copp
Notibnnal Ph;
46
Test Enrr
Tost Accuracy
34
Training Samples Per Class
Augmentation
Figure 7: Out-of-distribution augmentations still boost performance. Left: Test error (log-scale) as a
function of training samples when test images are rotations of training images. Right: Test accuracy on rotated samples from CIFAR-10 that were not used for training. All experiments performed rotated CIFAR-10 samples with the ResNet- 18 architecture. Bars mark standard error over 5 trials.
which corresponds to a normalization of the invariance. Here, orbit mapping uniquely selects single element from the group transformation before the data is passed into the model.
E2CNN: General E(2)- E22uivarrr Steerable CNN (E2CNN) (Weiler & Cesa, 2019) constrains convolutional kernels to reflect a group equivariance. We instantiate an E2CNN with the same architecture the ResNet- 18 studied so far.
In Section 3, we saw that horizontal flips are consistent with the CIFAR-10 distribution, so may not be surprising that horizontal flip invariant networks perform better than those trained with neither augmentations nor invariance constraints. Moreover, networks trained with data augmentations outperform invariant models for small and medium sample sizes. However, we observe in Figure that all investigated invariant architectures (red, green, blue) catch up to models trained with random augmentations (in orange) as we increase the number of base samples. We will see in Section that data augmentations serve as flatness-seeking regularizers, and the benefits of this additional regularization wane as the number of samples increases.
4.2 DITT----DISTIUUUIOONN AUGMENTATIONS STILL IMPROVE PERFORMANCE
Previously, we observed the performance benefits of data augmentations which promote invariances consistent with the data distribution, approximately But can it still be useful to augment data with transformation that generates samples completely outside the support of the data distribution and which inconsistent with any label? To answer this question, construct synthetic dataset in which the exact invariances are known.
We begin by randomly sampling single base image from each CIFAR-10 class. We then construct 10 classes by continuously rotating each of the base images, so that all samples in a class correspond rotations of a single image. Thus, the classification task at hand is to determine which base image was rotated to form the test sample. We randomly sample rotations from each of these classes to serve as training data and another disjoint of rotations serve as test data. We then use horizontal flip and random crop data augmentations to generate out-of-distributin samples, since horizontally flipped or cropped image views cannot be formed merely via rotation. Note that this experiment is distinct from typical co- variariat shift setups where the distribution of data domains differs, but the support is far from disjoint and may even identical.
In Figure 7, we see that these out-of-distribution augmentations are beneficial nonetheless. Notably, random erops, which can generate significantly more unique views than horizontal flips, yield massive performance boosts for identifying rotated images, even though know that the cropped samples are distr--dittiittion. We also see in this figure that random crops are especially useful if we instead as test set rotations of samples from CIFAR-10 100 that were not used for training. Specifically, we assign a base test image and its rotations the same label as the base image from the training with the same CIFAR-I0 label. This experiment supports the observations from Section 3 that augmentations can be particularly beneficial for OOD generalization.
Takeaway: Comparing invariant architectures to augmentations, we find that augmentations dominate on smaller scales, but invariant architectures catch up in the large-sample regime. Augmentations can provide benefits even on apparently unrelated invariances, which is particularly helpful for OOD generalization.

---

Published as a conference paper at ICLR 2023
No Ang.
Nont, Flip Band, Crop
Tri, Ang. Hortr, Flip & Cop Rend Aug, & Horis Plip Rand Drp
Gradien Stundard Dev.
35
30
10
30
100
150
50
100
150
50
100
1.5 50 100
150
Fixed Views, Same Batch
No, No Yes, No Yes, Yes No, Yes
Figure 8: Randomly applied augmentations significantly increase stochasticity late in training but decrease stochasticity early. Standard deviation of gradient across epochs for different augmentations and different mini-batch sampling strategics. Shown is the mean over 10 runs, and shaded regions represent one standard error.
5 AUGMENTATIONS AS A SOURCE OF STOCHASTICITY DURING TRAINING
Typical loss functions are summed over training samples. During optimization, gradients are computed using small mini-batches of random samples, resulting in stochasticity. Augmentations increase the number of available data, often so much that we never sample the same data twice.
Since data augmentations expand and diversify the training set, they may serve as additional sources of stochasticity during optimization. If data augmentations do increase the variety of gradients, they could as result cause us find qualitatively different minima. Stochastic optimization is thought to be associated with flat minima of the loss landscape which are in turn associated with superior generalization (Jastrzębski et al., 2018; Huang al., 2020; Liu et al., 2021a). This flatness-seeking behavior may be the effect of both the augmented loss function and also how we sample it.
To put this hypothesis to the test, measure the standard deviation of gradients during optimization for models trained with and without data augmentations, and we quantify the flatness of the corresponding minima. We construct experiments that disentangle the augmented loss function from the additional stochasticity produced by sampling augmentations. We consider a "same batch" strategy in which gradient updates are averaged over multiple views of single image, resulting in lower stochasticity. We also consider "fixed views" where we repeat a frozen set of augmented views per element of the training set, as in Figure 2.
5.1 MEASURING STOCHASTICITY
To measure stochasticity, we train a model on a given training set and augmentation strategy, and we freeze the model every 10 epochs to estimate the standard deviation (formally the norm of parameter- wise standard deviations) of its gradients over randomly sampled batches comprising 128 base images, the same batch size used during training. That is, we measure the square root of the average squared distance between randomly sampled batch gradient and the mean gradient. We adopt filter-normaizeed distance function (L.i et al., 2018; Huang et al., 2020) to account for invariances in neural networks whereby shrinking the parameters in convolutional filters may not effect the network's 's output but may make the model more sensitive to parameter perturbations of a fixed size.
In Figure 8, we see that non-augmented datasets actually yield noisier gradients early in training, but this noise vanishes rapidly as over-fitting occurs. In contrast, randomly applied data augmentations result in flatter curves, indicating that the added diversity of views available for sampling preserves stochasticity later in training. We also see that for each augmentation policy, applying augmentations randomly results in the most stochasticity late in training, while including multiple random views in the same batch (Hoffer et al., 2020; Fort et al., 2022) results in less. Sampling augmentations from a fixed set of four views per sample (denoted "fixed views") results in even less stochasticity, and including each of the four views in every batch results in the least stochasticity (denoted "fixed vews". "same batch"). This ordering, which holds across all data augmentations we try, is consistent with the intuition that more randomness in augmentation leads more stochasticity in training, notably only manifesting during later epochs. We will now see that the late- training stochasticity measure correlates strongly with the flatness of the minima these optimization procedures find.
8

---

Published as a conference paper at ICLR 2023
5.2 MEASURING FLATNESS
We adopt the flatness measurements from Huang et al. (2020) as these measurements are non-local, do require Hessian computations which are dubious for non-smooth ReLU networks, and they are consistent with our filter-normalized gradient standard deviation measurements. Specifically, we measure the average filter-normalized distance in random directions from the trained model parameters before we reach loss function value of 1.0, where loss is evaluated on the non-augmented dataset. Under this metric, larger values correspond to flatter minima as parameters can be perturbed further without increasing loss. We use the same ResNet-18 18 models trained in the stochasticity experiments above with the same exact augmentation setups.
Investigating Figure 8 and Figure 9, (see also Table 4), we observe that flatness correlates strongly with late-training stochasticity. Models trained without augmentation or with non-random augmentation, where all views seen in each batch, are less stochastic at the end of training and find sharper minima. While previous works have associated SGD with flatness-seetr scekkng behavior (Jastrzebski et al., 2018; Geiping al., 2021), data augmentations appear to contribute to this phenomenon. Simply put, training with randomized data augmentations finds flatter minima, and models trained with strong data augmentations lie especially flat minima.
Takeaway: Randomly applied augmentations provide benefits beyond invariance by flattening the loss landscape, which is reflected in both measurements of flatness after training and measurements of gradient noise late in the training.
5.3 DATASET SCALING AND FLATNESS
Emppirical Flateess (Lighter is flattee)
Figure 9 directly measures flatness for several dataset scales, We first notice that base mod- els become flatter (with respect to their base samples) as the number of samples increases. Surprisingly, stronger augmentations can pro- duce this effect quicker and raise flatness values for lower sample sizes. As notable ample, TrivialAug produces models that remain relatively flat for all sample sizes considered. Furthermore, all augmentations converge to sim- ilar flatness in the sample size limit, as regular- ization becomes less relevant in the large data regime. Over all plots we can even correlate the number of samples gained through augmentations and flatness and find that for all weaker augmenta- tions, flatness of the solution is strongly correlated with the number of extra samples that are gained from the augmentation. We include more details in Appendix H.
20
-5
Augmentation Strategy
10
ElipskCrops
Ang)
Parspective
1000
Took.
Number of Base Samples
Figure Left: Flatness augmented models trained on several dataset sizes from Figure 1. All strategics converge similar levels of flatness when scaling.
Strong data augmentations flatten the loss landscape to levels otherwise only reached with significantly larger datasets.
Takeaway:
CONCLUSION
Data augmentations have a profound impact on the performance of neural networks, but their precise role has been well understood; for example, augmentations are simply a heuristic for learning certain symmetries, should we not prefer to directly encode these symmetries through advances in group equivariant networks? Through the lens of exchange rates and power laws, we observe the gains through augmentations datasets scale and domains change. We find that augmentations dominating invariant architectures on smaller scales, but, scale in opposite ways. Augmentations are further distinguished from invariances in the way they can improve performance even out-of-distribution Ultimately we find that we can connect these findings to the regularization effect induced by data augmentations, which also measure, showing how augmentations flatten the loss landscape. This work promotes an all-encompassing understanding of neural network training, shedding light on the nuanced but significant role of data augmentation in the success of deep learning.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
We foresee no direct negative societal consequences from this work. We do think that data augmen- tations are beneficial, especially in applications with only limited data, or where data curation is expensive. We argue that knowing how exchange a smaller (but verified and curated) dataset for larger dataset that is not augmented, but also due to its size less curated is hopefully helpful to the community.
REPRODUCIBILITY STATEMENT
We use an academic cluster with NVIDIA RTXA4000 cards and also NVIDIA GTX2080t cards. Each job scheduled on single GPU and the default setting of 60000 gradient steps takes roughly an hour to train and evaluate. Including all preliminary experiments we estimate total usage of about 400 GPU days for this project. replicate all experiments the main body without repeated trials, we estimate a requirement of about 10 GPU days. We provide code at https:/ /tte:///ittte github..hb.... Jonas//naeeeeaaoooeee /ing/ dat aaugs and with the supplementary material.
ACKNOWLEDGEMENTS
This research is supported by NSF CAREER IIS-2145492. NSF I-DISRE 193471, NIH ROIDA048764-01A NSF IIS- 1910266, NSF 1922658 NRT-HDR, Meta Core Data Science, Google AI Research, BigHat Biosciences, Capital One, and an Amazon Research Award.
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- -han-CChristophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance Principle Meets Information Bottleneck for Out-of- Distribution Generalization. In Advances in Neural Information Processing Systems, volume 34, pp. 3438-345 Curran Associates, Inc., 2021. URL https: / wwwppooeoiiinggeeo www.......riiiir wesarrrpp.. cc/ pappt//0///haahh113333333 6)
Abstract Atstrrt .tmml. (p.
Guozhong An. The Effects of Adding Noise During Backpropagation Training on a Generalization Performance. Neural Computation, 8(3):643-67) April 1996. ISSN 0899-7667. doi: 10.1162/ 1996..999.833664 (p. 2)
Randall Balestriero, Leon Bottou, and Yann LeCun. The Effects of Regularization and Data Augmentation are Class Dependent. arXiv:2204.03633 [cs, stat], April 2022a. URL http: 03632. (p.
//arxivvorg/abb/
Randall Balestriero, Ishan Misra, and Yann LeCun. A Duta-Augmenttttion Is Worth A Thousand Samples: Analytical Moments And Sampling Free Training. In Advances in Neural Information Processing Systems, October 2022b. URL https::t////oo http:::://p/peeveeeeeeee wtt.rriwwweeeeeeeeeeeeeeeeeeeeee id=ekO. xrVWwQp. (p. 2)
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew G Wilson. Learning In- variances in Neural Networks from Training Data. In Advances in Neural In- formation Processing Systems, volume 33, pp. 17605-17616. Curran Associates, Inc., 2020. URL https://proceetingss neurips httpppper//22///aee 04400044299900002222222200-0-0-0 -tt.aaaee2..e...--ttttttttccct. html. (p.
N. V. Chawla, K. W. Bowyer, O. Hall, and W. Kegelmeyer. SMOTE: Synthetic Minority Over- sampling Technique. Journal of Artificial Intelligence Research, 16:321-357, June 2002. ISSN 1076-9757. doi: 10.1613//air.95. URL https:/ /twp///www/..////////////.../ jair, wwwg/nngg/....o......eo..oe.eeee /atpa/pppppppp.i///////pp/p/pa/a jair/ (p.
Shuxiao Chen, Edgar Dobriban, and Jane Lee. A Group-Theorete Framework for Data Augmentation. In Advances in Neural Information Processing Systems, volume 33, pp. 21321-21333. Curran As- sociates, Inc., 2020a. URL ht http:tppppp//ppppeeepeeeeoeeeeee http://///oooednnnnn....r.reeeee neurips cc/ cc. htcpppaper//// 1aaper/2022 /2020/ /2020/222aaaaa /hash/
Abst:099400000000bbtotebbe html. (p.2)
10

---

Published as a conference paper at ICLR 2023
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference Machine Learning, pp. 1597-1607. PMLR, November 2020b. http::
//proceedinge ml press /v1 9/ chen20j html. (p. 6)
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pp.
2921-2926, May 2017. doi: 10.1 N0.11090CNN.22217996666
(p. 16)
Miles ber 2022.
Cranmer.
PySR:
High-Performance Symbolic Regression in Python, Novem- URL https: www./i/le//aaamimmaia.aaa httpo/eee//////eeeee /BySR/blo//
/tt.://222921111166eeeooo/oo//// mdd. (p. 26)
Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 20, pp. 17429-17442, Red Hook, NY, USA, December 2020. Curran Associates Inc. ISBN 978-1-71382-954.. (p. 26)
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. AutoAugment: Learning Augmentation Strategies From Data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019. URL httpps://opeaaccesss thecvf. com/ wwwtentnttttcooccoo CVPR 2019 httpr/////////.ooooooooooa/aaaaa htt.rtttttoontttaateess__esroooo
2ttp:P__221__pppp_____e_ 1. (p. 2, 16)
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. RandAugment: Practi- Advances in
cal Automated Data Augmentation with Reduced Search Space. Neural Information Processing Systems, volume 33, pp. 18613-18624. Curran Asso- ciates, Inc., 2020. URL
www.iinngnnnuuipeeem cc/ wtt.aacttttm.....mmmmmm (p. 16)
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Re. A Kernel Theory of Modern Data Augmentation. Proceedings of the 36th International Conference on Machine Learning, pp. 1528-1537, PMLR, May 2019. URL https::/ppat /tttp:://proceedinggs mlr.
press/v97/dee daol 9b. html. (p. 2)
Luke N. Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not ImageNet or CIFAR-10. arXiv:1810.0350 [cs, stat], October 2018. URL http://arxiv. brg/abs/// 1810.111111111 03505. (p. 3, 5, 16)
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. In Proceedings of the 37th International Conference on Machine Learning, pp. 3165-3176. PMLR, November 2020.
URL https: httpss///cccceeeeeeeeeee mll www.:mees........mmmmm.mmmmmm..e v11vv111111 ffn//iini222aam html. (p.
Stanislav Fort, Andrew Brock, Razvan Pascanu, Soham De, and Samuel L. Smith. Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error. 1eXiv:2105.13343 [cs], February 2022. URL http://tpp// /tttp:///rxvv.../// httaaaaww..gg//aaaaaaaa/aaeooeee www//rrgss.....o..o...eo...eee.e 2200//2105.0 221.//155...333..... (p. 1,
Kanchana Vaishnavi Gandikota, Jonas Geiping, Zorah Lähner, Adam Czapliński, and Michael Moeller. A Simple Strategy to Provable Invariance via Orbit Mapping. In Asian Conference on Computer Vision (ACCV), Macau, December 2022. arXiv. doi: 10.48550/arxiv.2209.111 http::/ org/ abs/ 220992222.. 11916. (p. 6)
URL
Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic Training is Not Necessary for Generalization. In International Conference Learning Representations, September 2021. URL atttps://openrevieww net forum? 2BESe (p. 1, 3, 9)
Raphael Gontijo-Lpees, Sylvia Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Tradeoff's in Data Augmentation: An Empirical Study. In International Conference on Learning Representations. September 2020a. URLhttta htt.httss///pperevvieeee net tttp:/oorwmmm...kk/k..ccc.cc (p. 2)
11

---

Published as a conference paper at ICLR 2023
Raphael Gontijo-Lppss, Sylvia J. Smullin, Ekin D. Cubuk, and Ethan Dyer. Affinity and Diversity: Quantifying Mechanisms Data Augmentation. arXiv:2002.08973 [cs. stat], June 2020b. URL httt:://arkaa org/ abs /tss/22000 20000002....0 08973. (p. 2p
Dongyoon Han, Jiwhan Kim, and Jummo Kim. Deep Pyramidal Residual Networks. In 2017 JEEE Conference Computer Vision and Patlern Recognition (CVPR), pp. 6307-6315, July 2017. doi: 10.1 N0.11099CVP.22017.6666 (p. 5)
Boris Hanin and Yi Sun. How Data Augmentation affects Optimization for Linear Regression. In Advances in Neural Information Processing Systems, volume 34, pp. 8095-8105, Curran As- sociates, Inc., 2021. URL httpsttt (ww.:/8888666644444 wtw.://ee....eeeeeeeeeeeeeeaeaa. Abstarctttttr html. (p.
newrips CC/ wwwpppppep//////////e/
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1 (0.10099VPR.2001..99 (p. 15)
Tong He, Zhi Zhang, Hang Zhang. Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of
Tricks for Image Classification with Convolutional Neural Networks. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 558-567, 2019a. URL https: http:///ppaaccccceeeeeeesssessss theove com/ content wVw___VVV/R_____E__2 /He_ Bag_ on_ witwithh wtt__ittt_ooooolutiiooomm Laettttiiiaaaaaua________
of_Tricks_ for Image Saaeaageelllssee wwwssilissaaaaaacaa on
Networks_CVPE. 201 paper. html. (p. 30)
Zhuoxun He, Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data. arXiv:/909.099 [cs, stat], November 2019b. URL http: arxivv. wrt.abs////ooo.oooo........... haa///bss19999 1at.bb//99999999188.. 09148. (p. 4)
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness Common Cor- ruptions and Perturbations. In International Conference on Learning Representations, September 2018. URL https: http//ppeeeeeeeeeeeeeeeeeeeeeeee net/ forum? d== Gt (p. 6, 16, 31)
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lak- shminarayanan. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. In International Conference on Learning Representations, March 2020. URL https://openrevieet wttrrriwwwweeeeeeeee
net
Sig.dddddgggxx......... (p. 2, 16)
Alex Hernández- Garefa and Peter König. Further Advantages Data Augmentation on Convolutional Neural Networks. In Véra Kúrková, Yannis Manolopoulos, Barbara Hammer, Lazaros Iliadis, and Ilias Maglogiannis (eds.), Artificial Neural Networks and Machine Learning ICANN 2018, Lecture Notes in Computer Science, pp. 95-103, Cham, 2018. Springer International Publishing. ISBN 778---000-01418-6 doi: 10.1007/978- 10.1007/978-3-030-00111 301----000000006-11111 10. (p.2)
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment Your Batch: Improving Generalization Through Instance Repetition. In 2020 IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 8126-8135, June 2020. doi: (p. 8)
10. N0.1109/CVPR4220..220000000 15.
W. Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K. Terry, Furong Huang, and Tom Goldstein. Understanding Generalization Through Visualizations. pp. 87-97. PMLR, February 2020. URL 3, 3.8.9) 9)
maw.iiinnssmmii.......mmi...mmii prressvv331 http///////hhahn... html. (p.
StanisMaaw Jastrzæbski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. In International Conference on Artificial Neural Networks 2018; International Conference on Learning Representations 2018 (Workshop Track), Rhodes, Greece, 2018. (p. 3.3.8.9
Sanyam Kapoor, Wesley J. Maddox, Pavel Izmailov, and Andrew Gordon Wilson. On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification. wxth::2233.16881|cs stat], March 2022. doi: 10.48550/arXiv.2203.. URL http: arxiv. org/ abs///b////// 2203 16481. (p. 2)
12

---

Published as a conference paper at ICLR 2023
Jachyung Kim, Dongyeop Kang, Sungsoo Ahn, and Jinwoo Shin. What Makes Better Augmen- tation Strategies? Augment Difficult but Not too Different. In International Conference on Learning Representations, September 2021. URL https:/ httpp:////ppeeeiewww wwt.:wwwww.eeeeeeeeeeee.eee.eeee forum?
d.d-icx3DQbCGGR (p.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009. URL https: //www, cs toronto. wew.edduu/eeee http///eannnngeeeeeeeee featfffatut---aaeeee 2009-TR pdf. (p. 3, 16)
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 36(11):2278-222) 1998. (p. 16)
Daniel LeJeune, Randall Balestriero, Hamid Javadi, and Richard G. Baraniuk. Implicit Rugosity
Regularization via Data Augmentation. arXiv:190S//1639 [cs, stat], October 2019. URL http:
//arxiv. wwt./ooor.ooooo.... abs / /1005 11639. (p. 2)
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Land- scape of Neural Nets. In Advances Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https: /ttps:////o/ee
neurips cc wtcppappr/2011 wttpo//eeeee....0..7.76666666660 wtt.r/666eb.911---ttatttt.caaae html. (p. 8)
Tianyi Liu, Yan Li, Song Wei, Enlu Zhou, and Tuo Zhao. Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization. Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1891-1899 PMLR, March 2021a. URL mlr press / v1vvv130/11/ 30//3////lleee html. (p. 3,
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In 202/ IEEF/CVF International Conference on Computer Vision (ICCV), pp. 9992-10002, Montreal, QC, Canada, October 2021b. IEEE. ISBN 978-1-66542-812-5, doi: 10. Dw.110//CCV48992..000 0D00//CCCV889222222.008888 2021.09988 URL https: feeexplore ieee org/ dowument / 991109900////// ().
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer V2: Scaling Up Capacity and Resolution. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recog- nition, pp. 12009-12019, 2022. URL https: / openaccess thecvf. com/ content/ CVPR2022 http1//tmmm////mmm Liu Swwn_ Trans__orroofomme..... V2 V2 pappep
paper. html. (p. 5)
Antonia Marcu and Adam Prugel-Bennett. On the Effects of Artificial Data Modification. In
Proceedings of the 39th International Conference on Machine Learning, pp. 15050-15069, PMLR,
June 2022. URL https: / http:ss:/ppcccedeeeeeeee ptwpeedinceggsssee ml1 press siseessvv62..
html. (p. 2)
Samuel G. Müller and Frank Hutter. TrivialAugment: Tuning-Free Yet State-of-the-Art Data Augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 774-782, 2021. URL https: / http//ccppccccccccocpoopcc.. thecvf. com/ content IICCC20I1C html Haml/MMulers www.lieee_rriiiilllaeeeeeetettte Tunnentttttttinnnneeeeeeeee Free Data_ onta__uummentaaiioo_cccooLoO
Staae-of-thte html. (p. 4, 16)
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of Optimization and Implicit Regularization in Deep Learning. arXiv:1705.03077 [es], May 2017.
URL http://///// / /arxiv..vvv org/ abs 03071. (p. 2)
Shashank Rajput, Zhili Feng. Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does Data Augmentation Lead to Positive Margin? In Proceedings of the 36th International Conference on Machine Learning, pp. 5321-5330. PMLR, May 2019. URL https://proceed:: mml.. 9a html. (p. 2)
peess/v97/rajeep
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-25) December 2015. ISSN 1775-1405, doi: 10.1007/s1 12000//01226--1-18166666 URL https: / //dof worg/10 11000//11163-1-11 5- 6-....8811.... 30)
13

---

Published as a conference paper at ICLR 2023
Connor Shorten and Taghi M. Khoshgoftaar. A survey Image Data Augmentation for Deep Learn- ing. Journal of Big Data, 6(1):60, July 2019. ISSN 2196-1115, doi: 10. 10.116///00077700900999--0 URL https: /ttp://........1111//0/000000010
(p.2)
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv: [cs]. September 2014, URL stttt:://arxis www.iiioo.roo.ooooooooooooo abs /1409. 1556. (p. 5, 6, 15)
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your VIT? Data, Augmentation, and Regularization in Vi- sion Transformers. Transactions on Machine Learning Research, June 2022. URL https: (p.2)
//openreview.oo www.reeiewweeee/ee...ee http://ww.ee//oummm22mm
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting Unrea- sonable Effectiveness of Data in Deep Learning Era. In Proceedings of the IEEE International Conference on Computer Vision, pp. 843-852, 2017. URL https: //openaccess aerevf.com. wwt.c/////cocntteeeeooeccceooeet _ccv_2017/htmll /utnl//uuunnn// Revisiting Unreasonable Effectiveness Seapneess_ccV_____ paper html. (p. 3, 30)
Luke Taylor and Geoff Nitschke. Improving Deep Learning with Generic Data Augmentation. In 2018 IEEE Symposium Series Compulational Intelligence (SSCI), pp. 1542-1547, November 2018. doi: 10. 109/S////11. 8628772.
(p. 2, 4)
Asher Trockman and J. Zico Kolter. Patches Are All You Need? arxiv:2201.09792//cs, January 2022. doi: 10.488550/2Xi..2201.09977 URL http: /ttp/aaxvvv///./////v////////aaa org abs /tt./////220
(p. 15)
Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. 01106077116000222200000000000000
URL
neurips CC/ wwccc//pppp. htcppappr/20199 9/hashh html. (p. 7)
Ross Wightman, Hugo Touvron, and Hervé Jégou. ResNet strikes back: An improved training procedure in timm. arXiv:2110.00476 [cs], October 2021. URL http: aaari.iiiiicoc org/ abs/ 2110. 00476.
30)
Mingle Xu, Sook Yoon, Alvaro Fuentes, and Dong Sun Park. A Comprehensive Survey of Image Augmentation Techniques for Deep Learning arxiv:2205, 01190//s,,.411sss May 2022. doi: 10.48550/ arXiv.2205.01491. URL http:htt//// http::ttttiii..ottt/ www.w.www.aaaaaaa.aa.oa.ooeaeeaa www.abb//222 2205 01491. (p.
Larry Yaeger, Richard Lyon, and Brandyn Webb. Effective training of neural network character classifier for word recognition. In Proceedings of the 9th International Conference on Neural Information Processing Systems, NIPS*96, 96, pp. 807-813, Cambridge, MA, USA, December 1996.
MIT Press. (p. 2)
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging Theory and Algorithm for Domain Adaptation. In Proceedings of the 36th International Conference on Machine Learn- ing, pp. 7404-7413, PMLR, May 2019. URL https://////...... /ttp://proceeeeeeee ingeeeidddnggg.neeeeeee ngs mll. r. w...press....................... 9 zhang195 html. (p.
Sicheng Zhu, Bang An, and Furong Huang. Understanding the Generalization Ben- efit of Model Invariance from Data Perspective. mation Processing Systems, volume 34, pp. 4328-4341. Curran Associates, Inc., 2021. URL https: settps://proceeding .twpeeddinnn..uuiii....ee. newrips cc/pappr//02//haaaee 2257ccbbeedddee2211000000ee3 Abt.trrraass--baacaatcccc
Advances Neural Infor-
Abstract tetstrat..hml... (p.
A EXPERIMENTAL SETUP
For all sections if not otherwise mentioned, we run the following protocol. We train the model (in the main body ResNet-18), with stochastic gradient descent for 60000 steps with batch size of 128. This corresponds to 160 epochs for a dataset of size 48000. We keep this number gradient steps fixed when increasing decreasing the dataset size. We linearly warm up the learning rate for
14

---

Published as a conference paper at ICLR 2023
the first 2000 steps (about 5 epochs) up peak rate of 0.1 and then decay to zero by half-cycle of cosine annealing. For all experiments, we include a standard weight decay of 5e - 4 and train with Nesterov momentum of 0.9. The data is shuffled randomly after every epoch and we record validation accuracy every 10000 steps. All training runs are non-deterministic based on stochasticity due to random shuffling and cudnn non-determinism. We run at least five trials for each experiment in the main body and three trials for each in the supplementary material. In each plot, the standard deviation is shaded. For five trials this corresponds close a 97.5% confidence interval.
We use CIFAR-10 in its default configuration. For CINIC-10, 10, we clean and resample the train and test sets. We first remove all CIFAR-10 10 train and test images from the dataset, we then further remove all exactly duplicated images and missing images, merge all remaining images and sample a new validation set of 10000 images. We provide code to replicate the creation of this cleaned dataset with the supplementary material. Overall we recover a new training set of size 193523. For CIFAR-10-C experiments the supp. material, we report average accuracy over all transformations in CIFAR- 10-C with severity of 3. For all experiments where we consider only a subset of the existing data (e.g. each experiment with less than 193523 samples for CINIC-10), 10), we sample a new subset of the training set for each experiment separately, to rule out confounding effects of good or bad splits of the training data, especially for smaller subset sizes. This new test set for CINIC-10 10 turn be harder than the CIFAR-10 test set, but we verify that the hyperparameters discussed above would result in more than 95% accuracy when training with CIFAR training data.
For experiments in the main body where data augmentations randomly sampled finite number of times, we store all augmentations in a database (1mib) that is recreated in each run. As a result, each experiment contains fixed set of finite views of each original datapoint, but these views are randomized across experiments. Due to random shuffling, samples from this enlarged dataset are drawn randomly and multiple views of the image are only guaranteed occur in the same batch in the batch augmentation experiments in Section 5.
To compute measurements of exchange rates, we first compute the mean validation accuracy CINIC-10 for each experiment. We then train the reference models for CINIC-10 subset sizes of 1000, 2000, 3000, 6000, 12000, 24000, 48000, 96000, 128000, 144000, 168000, 180000, 192000.
To estimate parameters a, b, for f2(r) az + b in the exchange rate plots in all sections we use a non-linear least-squares algorithm, initialized from starting parameters that describe the curve for no augmentations. For this we use the Gevenberg-Marquane implementation of MINP ACK, as wrapped in scipy.
To cross-reference the average validation accuracy of these reference models with our data augmen- tation experiments in Table I, we assert that validation accuracies are monotonically increasing as subset sizes increase and fit linear spline fie for interpolation. We then compute the exchange ratios of Table 1 via fre (z)/b, for the base dataset size b which is 48000 in Table 1 and input mean validation accuracy x for each experiment. For values outside the interval spanned by the minimal and maximal validation accuracies of the reference data, we reuse the power laws of the form [a) ax - + b described in Sec. 3.3 and again compute f (x)))//b We mark these extrapolated values by a in the table.
The ResNet- 18 model employed in the model is modern variant (He et al., 2016; 2019a) and contains the usual CIFAR-IU 10 stem consisting of a single 3 x 3 convolutional layer without pooling, instead of the ImageNet stem (of two convolutional layers with stride and max-pooling). For experiments in the supplementary material, we further consider a ResNet-8 8 (i.e. three stages and single block per stage) (He et al., 2016), a VGG-11 (Simonyan & Zisserman, 2014) with batch normalization, and Moxe architecture (Trockman & Kolter, 2022) of depth 8 with hidden dimension 128 and spatial kernel size 7.
We implement and run all experiments in PyTorch and make use of torchvision implementa- tions for a range of data augmentations investigated in this work. We provide code to replicate all experiments at hts githhu...... com www//oosseeepiingeeoooooo dat and with the supplemen- tary material.
15

---

Published as a conference paper at ICLR 2023
A.1
HYPERPARAMETERS FOR AUGMENTATIONS
For each augmentation we broadly follow established defaults. For completion we record these, and additional details here.
Horiz. Flips: A data point is flipped horizontally with probability 0.5.
Det. Horiz. Flips: Deterministic horizontal flips. For 1x, this corresponds to flipping every data point. 2x corresponds to both flips being contained in the dataset.
Vert. Flips: A data point is flipped vertically with probability 0.5.
Det. Vert. Flips: Deterministic vertical flips. For 1x, this corresponds to flipping every data point. 2x corresponds to both flips being contained in the dataset.
Random Crops: The image is padded by with hero-padding by 4 pixels in each direction and then image size 32 x 32 is cropped (This classical random cropping for CIFAR-10).
Tips&CCoops: Both random crops and horizontal flips are employed, as described above.
Perspectives: Performs a random perspective transform with probability 0.5 with bilinear resampling.
Jitter: Color jitter, randomly transforming contrast, hue and brightness of the image. For each distortion, sample a new scale uniformly from [0.5, 1.5].
Blur: Blurs the image with Gaussian blur with 3.
AutoAug: Employ the augmentation policy of Cubuk al. (2019), with the CIFAR-10 policy.
AugMix: The augmentation policy of Hendrycks et al. (2020).
RandAug: The augmentation policy of Cubuk al. (2020), again with the CIFAR- 10 policy.
TrivialAug: The augmentation policy of Müller & Hutter (2021) in its "wide" configuration.
AutoAAu&FFliss&&CC The AutoAug policy followed by random horizontal flips and random crops as described above.
Mix Flips&&&&& Crops: The Augmix policy followed by random horizontal flips and random crops as described above.
Fliid&&&&&ll&& The RandAug policy followed by random horizontal flips and random crops as described above.
TrivialAug&Fllipsss Crops: The TrivialAug policy followed by random horizontal flips and random crops as described above.
A.2 DATA LICENSING
We investigate, MNIST (LeCun et al., 1998), CI and CIFAR-100 (Krizhevsky, 2009), EM- NIST (Cohen al., 2017), CIFARIO-C (Hendrycks & Dietterich, 2018) and CINIC-10 (Darlow et al., 2018) and refer to these publications for additional details. We remove duplicates and missing data from CINIC-10 described in the experimental setup.
CONI YOL A00u93
HATE New freds
CONIC-BI 80 VOl 83
0.50
04
6:0
Coc
072
Size Generred Desset
Gonerned Daseet
Sba Generred Darates
Conmrnee
Figure 10: Repeated Data CINIC-10. These are vertical slices through Figure 2, For (left-to-right): 24000, 48000, 96000, 192000 base samples.
16

---

Published as a conference paper at ICLR 2023
Megrentaiis wpg.eei
1.0
Accuse
11.08 her Seqpi
CENR Va Acoa
0.2
LONK krocle
0.0
630
0.0
Sot fae
106
Siee Generned Dipsort
Geoecred Decoor
Sir Genecred Dasser
Groeared Diiser
Figure 11: Repeated Data for CINIC-10, evaluated CIFAR- 10. These are vertical slices through Figure 5 (left). For (left-t-riight: 24000, 48000, 96000, 192000 base samples.
CITARICC Accuno
CIPARI va Accurgy
CIPANI VA. ACOURO
0.5
36
MG
0.50
T00k
120c
300
1M
S0a Grearred Dapset
Sbe Ginerndd Dapsor
Sbs Geneord Dapaen
Sba of Grevered Dopoet
Figure 12: Repeated Data for CINIC-10, evaluated on CIFAR-IO-C. These are vertical slices through Figure (right). For (left-to-right): 24000, 48000, 96000, 192000 base samples.
B ADDITIONAL EXPERIMENTAL RESULTS
We include additional material for the experimental section in a series of figures and tables. Table is an extended table of the exchange rates for CINIC- 10 for a sample size of 48000, including repetitions up to 32x and ablating the number of steps (This is slice of Figure 1). Behavior is consistent over additional repetitions, so we chose not to include these additional rows the main body. This table further includes additional data augmentations not featured in the main body, over which behavior is consistent. Table and Table are then variants of this table where validation accuracy evaluated on CIFAR-10 and CIFAR-1O-C, respectively, CIFAR-IO-C is significant distribution shift that cannot be mitigated by additional CINIC- 10 data, only training on, e.g. blurred samples, provides robustness to this distortion. We further find that training with horizontal flips in our experimental setup is quickly disadvantageous.
To provide additional clarification for Figure 2, also provide slices through this plot at sample sizes of 24000, 48000, 96000 and 192000 in Figure 10 for CINIC validation data, Figure 11 for CIFAR-10 and Figure 12 for CIFAR-10-C data. Further, Figure 13 shows the data points underlying Figure for additional clarification.
An additional table comparing the end-of-training stochasticity in Figure and flatness measurements in Figure 9 for ease of references is Table 4.
C ADDITIONAL DATASETS AND MODELS
We further verify that the findings discussed in the main body are not limited to the choice dataset and model therein and provide additional reproduction aside from the investigation into model architectures in Figure 3. We repeat the tabular representation of exchange rates and a simplfied form of Figure II for tiny ResNet-8 in Figure 14 and Table 5, a VGG-I1 in Figure 15 and Table 6 and ConvMixer (as representative of modern
Trrnns///ranormmer variations) in Figure 16 and Table 7.
We then further repeat these experiments with models trained on the MNIST training set Figure 17 and Table 8, CIFAR-IOW in Figure 18 and Table 9, as well as EMNIST in Figure 19 and Table 10.
We further include repeated experiments for Sec. 5 on CIFAR- 100 in Figure 20 and Table 11.
17

---

Published as a conference paper at ICLR 2023
0.9
0.7
Val. Acourcy
0.6
Arch, Augmeniaiin Streteyy
RodNit188, Baselax
0.4
ResNict 18, TivialhughFllssCCer
ConsMiser, Baseline (No AngJ
0.3
ConvMixer, TnvalNighFllsssCCoo
VOGTE Boseline
PyramidNet, Baseline Ang)
PenmiilNet, Trivialkugh Hipee Crope
SwiiToandomee: Baaline (Ni Aug)
SwilTrandomes Titial/ug@FiieaCCpopp
50k
1006
200k
250
Samples
Figure 13: Power laws [xx - b for TrivialAug with flips and crops applied randomly. Equation (1). Fitted power law curves marked in solid colors, with extrapolated regions dashed. Standard deviation around measured samples (dotted) is shaded. Figure is based on this data, comparing exchanges from unaugmented to augmented separately for each model.
Table 1: Extended table of Exchange rates for augmentations applied to 48000 base samples from the CINIC- 10 training set, compared reference models trained without augmentations on up to 192000 samples. measure the exchange rate w.r.t. accuracy on the in-domain CINIC-10 val. set. Values marked with fall outside range of reference datasets and are extrapolated using power laws. For select augmentations we also include experiments with 240000 steps, i.c. 640 passes through the data to verify the utility of our chosen schedule of 60000 steps.
CINIC-10 (in-domain)
Augmentation
Ix
2x
4x
8x
16x
32x
rand (160)
rand (640)
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Horiz. Flips
0.99
1.55
1.79
1.84
1.85
1.79
1.88
Det. Horiz. Flips
0.96
1.89
Vert. Flips
0.68
0.94
1.09
1.23
1.17
1.14
1.25
1.20
Det. Vert. Flips
0.05
1.30
Random Crops
0.98
1.90
2.54
2.61
2.74
2.59
2.74
Clips&Crops
0.99
1.93
2.94
3.72
4.00*
4.00
3.79
Perspectives
0.88
1.53
1.89
2.29
2.54
2.68
2.50
Jitter
0.91
0.92
0.90
0.87
0.82
0.81
0.93
0.88
Blur
0.75
0.76
0.69
AutoAug
0.78
0.95
1.02
1.20
1.39
1.52
1.63
1.77
AugMix
0.87
0.95
0.98
1.00
1.02
1.00
1.14
1.13
RandAug
0.88
1.49
2.20
2.51
2.67
2.49
TrivialAug
0.72
1.87
2.12
AutoAuug&Fliss&Crops
0.75
1.43
2.22
3.21
4,00*
4.00*
4.00
4,08*
Augmix&&Flips&Cropp
0.86
1.62
2.50
3.09
3.82
3.84
3.74
3.74
RandAuugFFiis&&CCCo
0.84
1.71
2.65
3.78
4.00*
4.00
4.00
TrivialAug& &lips&&rropss
0.70
2.86
3.71
4.00
4.00*
18

---

Published as a conference paper at ICLR 2023
Table Extended table Exchange rates for augmentations applied to 48000 base samples from CINIC-10
training set, compared reference models trained without augmentations on up to 192000 samples. measure the exchange rate w.rt. accuracy on the CIFAR-10 val. set. Values marked with fall outside the range of reference datascts and extrapolated using power laws.
CIFAR-10 (slightly out-of of-domiin)
Augmentation
1x
2x
4x
8x
16x
32x
rand (160)
1.00
L.00
1.00
1.00
1.00
1.00
1.00
Horiz. Flips
0.95
1.34
1.58
1.37
1,42
1.35
1.66
Det. Horiz, Flips
0.95
1.46
Vert. Flips
0.47
0.56
0.62
0.64
0.64
0.66
0.68
Det. Vert. Flips
0,02*
0.71
Random Crops
0.94
1.82
1.93
1.91
1.75
1.92
1.91
Frips&Crops
1.78
2.15
2.58
2.26
3.05
1.94
Perspectives
0.95
2.06
3.29
4.02*
4,73*
4.96
4.34*
Jitter
0.97
1.04
1.09
0.95
0.89
0.86
1.08
Blur
1.52
1.44
1.35
1.20
1.03
0.97
1.40
AutoAug
0.99
1.60
2.00
2.39
3.14
3.46
4.00
AugMix
1.77
2.38
2.61
3.10
3.18
3.20
3.31
RandAug
1.15
2.29
3.42
4.00*
4.56*
5.19
4.02*
TrivialAug
0.89
1.81
2.30
3.17
4.00*
4.02
4.78*
AutpAuug&Flipp&Cooops
0.96
2.53
4.46*
6.30
6,93*
7.27
7.18*
AugMix&&Fiiss&CCC
1.74
4.41
6,86*
8.66*
8.92*
9.45
9.01
0.96
2.32
4.00*
5.10*
6.24*
6.80
5.12*
IrivialAng&FliissCCrooo
0.93
2.43
4.30*
6.10*
6.84*
7.34
7.60*
Table Extended table Exchange rates for augmentations applied to 48000 base samples from CINIC-10 training set, compared to reference models trained without augmentations on to 192000 samples. We measure the exchange rate w.r.t. accuracy the CIFAR-10-C val. set. Values marked with fall outside the range of reference datasets and extrapolated using power laws. Note that especially values 10 are extensive measured range. Values marked
extrapolation outside law, meaning that (at least according to the behavior predicted by it), no amount of additional real data with be sufficient to match the accuracy achieved with this augmentation there is no exchange rate.
are outside the range of estimated power
CIFAR-10-C (out-of-domain)
Augmentation
Ix
4x
8x
32x
rand (160)
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Horiz. Flips
0.84
0,67
0.66
0.55
0.53
0.52
0.63
Det. Horiz. Flips
0,93
0.55
Vert. Flips
0.15
0.11
0.11
0.11
0.12
0.11
Det. Vert. Flips
0.02*
0.12
Random Crops
0.82
0.86
0.70
0.66
0.60
0.69
0.67
Flips&Crops
0.86
0.66
0.63
0.78
0.64
0.71
0.25
Perspectives
34.16*
Jitter
16.81
190.76*
16.99*
7.26*
3.08
163.08
Blur
AutoAug
AugMix
RandAug
TrivialAug
AutoAug&FFipssCCrppp
AugMix&Flips&&ropp
RandAug&Flips&Crug
91.64*
SwvialAug&FlipsCCCoe
19

---

Published as a conference paper at ICLR 2023
Table 4: End-of-training stochasticity correlates strongly with flatness. Gradient standard deviation across batches the end of training and flatness measurements for ResNet-18 18 models trained CTFAR-10 with various augmentations and strategies sampling augmented views.
Augmentation
Fixed Views
Same Batch
Grad. Std.
Flatness
No Augmentation
13.201
11.4192
24.7397
16.3284
X
13.612
10.1113
Flips & Crops
15.676
11.914
7
10.263
8.2033
X
31.339
20.1336
X
22.689
16.3189
TrivialAug &Flips & Crops
25.53
16.1076
14.105
10.4287
X
29.912
18.6886
RandAug & Horiz. Flip Random Crop
X
16.585
11,5707
22.0127
13.3998
10.865
8.1097
Table Extended table Exchange rates for augmentations applied to 48000 base samples from CINIC-10 training set, compared to reference models trained without augmentations on up to 192000 samples for ResNet-8 models. We measure the exchange rate w.r.t. accuracy on the CINIC-10 val. set. Values marked with * fall outside the range of reference datasets and are extrapolated using power laws.
CINIC- (in-domain)
Augmentation
1x
2x
4x
8x
16x
32x
rand (160)
rand (640)
1.00
1,00
1.00
1.00
1.00
1.00
1.00
1.00
Horiz. Flips
1.02
1.50
1.74
1.78
1.69
1.67
1.91
1.69
Det. Horiz. Flips
1.05
2.03
Vert. Flips
0.68
0.92
1.10
1.03
1.02
1.00
1.22
1.09
Det. Vert. Flips
0.09
1.31
Random Crops
0.98
1.82
2.40
2.44
2.51
2.62
2.62
Flips&rrops
0.96
1.89
2.69
2.99
3.33
4.00*
1.07
3.83
Perspectives
0.90
1.57
1.95
2.13
2.29
2.18
2.39
2.35
Jitter
1.00
1.15
1.18
1.05
1.06
1.04
1.24
1.21
Blur
0.77
0.90
0.98
0.96
0.96
0.93
1.05
0.97
AutoAug
0.93
1.32
1.64
1.64
1.69
T.68
1.91
1.80
AugMix
1.03
1.31
1.52
1.54
1.54
1.48
1.75
1.67
RandAug
0.97
1.66
2.06
2.26
2.42
2.45
2.67
2.68
TrivialAug
0.82
1.39
1.84
1.96
2.07
1.99
2.19
2.24
Autisuug&lipss&Cropps
0.85
1.62
2.15
2.65
2.89
3.14
2.62
3.00
AugMix&Flips&Crrop
0.92
1.75
2.44
2.79
3.08
3.45
2.82
3.22
RandAAug&Flips&Crrp
0.93
1.78
2.47
2.84
3.28
4.00*
2.84
3.92
TrivialAug&FFiissCCCn
0.75
1.62
2.03
2.51
2.75
2.91
2.52
2.93
20

---

Published as a conference paper at ICLR 2023
0.74
CINIC-10 Val, Accuracy
CIBAR-10 Vol. Acturayy
BSE bore camplis
0.68
0.64
0.65
Emptes
Horts Flip
0.6
Tivialkag
66
Perqaerive
1002
134
I0M
7.7
100%
1M
10
Samples
from Samples
Figure 14: Validation accuracy dataset size larger datasets generated from fixed number of base samples and selected data augmentations. ResNet-8 models trained fixed datasets generated via augmentation from 48000 samples from CINIC-10 train set and evaluated on CINIC-10 val. set (left) and the CIFAR-10 val. set (right), std. error over runs shaded. The accuracy of reference models trained without augmentations is marked with horizontal lines.
Table Extended table Exchange rates for augmentations applied to 48000 base samples from CINIC-10
training set, compared to reference models trained without augmentations on up to 192000 samples for VGG-11
models. We measure the exchange rate w.r.t. accuracy on the CINIC-10 10 val. set. Values marked with fall outside the range of reference datasets and are extrapolated using power laws.
CINIC-10 (in-domain)
Augmentation
1x
2x
4x
8x
16x
32x
rand (160)
rand (640)
1.00
1.00
1.00
1.00
1.00
1,00
1.00
1.00
Horiz. Flips
1.00
1.54
1.77
1.77
1.78
1.89
Det. Horiz, Flips
0.98
1.97
Vert. Flips
0.54
0.83
0.93
0.95
0.93
0.92
1.00
0.97
Det. Vert. Flips
0.02
1.08
Random Crops
0.98
1.70
2.11
2.12
2.22
2.11
2.24
Flips&Crops
0.98
1.86
2.99
3.08
3.11
1.01
3.15
Perspectives
0.79
1.20
1.54
1.56
1.51
1.46
1.76
1.51
Jitter
0.89
0.86
0.86
0.84
0.79
0.81
0.86
0.82
Blur
0.60
0.59
0.58
0.54
0.56
0.51
0.62
0.55
AutoAug
0.72
0.91
1.05
1.10
1.26
1.24
1.59
1.51
AugMix
0.91
0.97
0.97
0.96
0.95
0.97
1.02
1.00
RandAug
0.83
1.37
1.79
1.92
2.00
1.98
2.14
2.19
TrivialAug
0.64
0.94
1.14
1.37
1.55
1.65
1.94
2.00
AutoAuugFFlissCrooss
0.67
1.28
2.09
2.68
3.13
3.21
4,00*
4,00*
AugMix&&Flips&Cropss
0.83
1.55
2.31
2.79
3.06
2.81
3.22
3.12
RandAug&Flipx&CCnn
0.79
1.57
2.27
3.09
3.32
3.33
3.97
3.92
ThvialAug&FliipsCCopps
0.54
1.13
1.83
2.46
2.93
3.35
4.00*
4.00*
21

---

Published as a conference paper at ICLR 2023
015
CINIC-10 Val. Accuracy
CIR/RR-1 Vol. Acturayy
Hik hone cumples
6.7
0.75
dik Junr curlee
0.03
Horts Flip
0.8
Tivialkag
Perqaetive
1002
134
10M
3.7.4
100k
6.7
1M
10
Samples
from Samples
Figure 15: Validation accuracy dataset size as larger datasets generated from fixed number of base samples and selected data augmentations. VGG-11 models trained fixed datasets generated via augmentation from 48000 samples from CINIC-10 train set and evaluated on CINIC-10 val. set (left) and the CIFAR-10 val. set (right), std. error over runs shaded. The accuracy of reference models trained without augmentations is marked with horizontal lines.
Table 7: Extended table Exchange rates augmentations applied to 48000 base samples from CINIC- 10 training set, compared to reference models trained without augmentations on up to 192000 samples for ConvMixer models. We measure the exchange rate w.r.t. accuracy on the CINIC-10 val. set. Values marked with fall outside the range of reference datasets and extrapolated using power laws.
CINIC-10 (in-domain)
Augmentation
1x
2x
4x
8x
16x
32x
rand (160)
rand (640)
1.00
1.00
1.00
1.00
1.00
1,00
1.00
1.00
Horiz. Flips
1.37
1.39
1.41
Det. Horiz. Flips
1.04
1.50
Vert. Flips
0.37
0.45
0.45
0.48
0.55
0.50
0.48
0.49
Det. Vert. Flips
0.04
0.43
Random Crops
0.78
0.93
0.97
1.18
1.36
1.30
1.00
1.26
Flips&Crops
1.40
1.63
1.72
Perspectives
0.71
0.97
1.26
1.85
2.24
2.03
2.04
2.33
Jitter
0.78
0.91
0.90
1.04
1.11
1.34
0.87
1.26
Blur
0.34
0.46
0.51
0.49
0.56
0.61
0.54
0.65
AutoAug
0.59
0.89
1.42
1.77
1.96
2,30
1.57
2.07
AugMix
0.77
0.91
0.89
1.31
1.60
1.77
1.40
1.70
RandAug
0.80
1.28
1.71
1.76
2.05
2.59
2.21
2.28
TrivialAug
0.48
1.11
1.72
2.31
2.84
2.91
2.28
2.66
AutoAuugFFlissCrooss
0.49
1.04
1.85
2.19
2.67
2,80
2.49
2.91
AugMix&&Flips&Crops
0.65
1.22
1.58
1.91
2.10
2.12
1.92
2.20
RandAuggFlips&Crops
0.57
1.19
1.82
2.40
2.65
2.69
2.89
2.89
ThvialAug&FliipsCCopps
0.41
1.07
1.83
2.21
2.92
3.55
2.67
3.08
22

---

Published as a conference paper at ICLR 2023
0.8
L52% base sattplis
CINIC-D0 30 Val. Accurcy
CIFAR-10 Vol. Accuncy
1o bwweemmtt cimples
0.75
0.24
0.72
Aug. Strategy
Horis, Flip
0.63
TricilAng
0.68
100k
Size of Datases Generated from 48k: Base Samples
Size of Dataser Generated from 48k Base Samples
Figure 16: Validation accuracy versus dataset size as larger datasets are generated from a fixed number of base samples and selected data augmentations. ConvMixer models are trained on fixed datasets generated via augmentation from 48000 base samples from the CINIC-10 train set and evaluated the CINIC-10 val. set (left) and the CIFAR-10 10 val. (right), std. over shaded. The accuracy of reference models trained without augmentations marked with horizontal lines.
Table 8: Extended table of Exchange rates for augmentations applied 48000 base samples from the MNIST training set, compared reference models trained without augmentations on up to 60000 samples ResNet-18 models. We measure the exchange rate w.r.t. accuracy on the MNIST val. set. Values marked with fall outside the range of reference datasets and are extrapolated using power laws. Values marked with / are outside the range of the estimated power law, meaning that (at least according the behavior predicted by it), amount of additional real data with be sufficient match the accuracy achieved with this augmentation there exchange rate.
MNIST (in-domain)
Augmentation
1x
2x
4x
8x
16x
32x
rand (160)
rand (640)
1.00
1.00
1,00
1.00
1.00
1.00
1.00
1.00
Horiz. Flips
0.24
0.24
0.34
0.27
0.26
0.29
0.40
0.25
Det. Horiz. Flips
0.02*
Vert. Flips
0.20
0.22
0.22
0.22
0.22
0.24
0.23
0.22
Det. Vert. Flips
0.02*
Random Crops
1.11
1.16
1.17
/
/
37.32*
Flips&Crops
0.17
0.23
0.21
0.22
0.24
0.25
0.88
0.23
Perspectives
1.16
1.14
7.84*
1.17
1.17
/
1.15
37.32*
Jitler
1.08
0.81
1.17
1.08
1.11
0.81
1.10
Blur
1.08
1.10
1.12
0.77
1.15
7.84
1.17
1.14
AutoAug
0.72
1.16
1.08
1.16
4.38*
7.84*
7.84
AugMix
1.11
1.14
1.16
1.16
1.17
1.17
1.12
1.15
RandAug
1.10
4.38*
1.16
TrivialAug
0.54
4.38*
1.12
1.12
4.38*
/
1.16
0.14
0.17
0.20
0.24
0.26
0.24
0.24
0.26
AugMix&&Flips&&ropss
0.15
0.21
0.21
0.24
0.23
0.24
0.24
0.24
RandAug&&Fipss&Copp
0.17
0.21
0.21
0.23
0.24
0.23
0,23
SwvialAng&Flis&&CCCe
0.12
0.15
0.16
0.21
0.22
0.23
0.21
0.24
23

---

Published as a conference paper at ICLR 2023
0.036
400 5320 grnnits
MNNST Accuracy
MNIST Val. Accuracy
0.98
0.96
0.992
0.99
Auginentation Strategy Baseline
Aug. Strateyy
Tassalku
0.988
0.0
Horio Flip
0.880
Peequeciie
100k.
1M
10M
5
1000
10k
100%
Size Dataset Generated from 40k Base Samples
Number Base Samples
Figure 17: Left: Validation accuracy base samples and selected data augmentations. ResNet-18 models are trained on fixed datasets generated via augmentation from 48000 base samples from the MNIST train set and evaluated on the MNIST val. set. Right: Extrapolated scaling behavior of reference models for MNIST.
dataset size larger datasets generated from fixed number
Table 9: Extended table of Exchange rates for augmentations applied to 48000 base samples from the CTFAR- 100 training set, compared reference models trained without augmentations up to 50000 samples for ResNet-18 models. We measure the exchange rate accuracy the CIFAR-100 val. set. Values marked with fall outside the range of reference datasets and are extrapolated using power laws.
CIFAR-100 (in-domain)
Augmentation
1x
2x
4x
8x
16x
32x
rand (160)
rand (640)
1.00
1.00
1.00
1.00
1.00
1.00
Horiz. Flips
0.95
1.39
1.57*
1,59
1.56*
1,52*
1.61*
1.55*
Det. Horiz. Flips
0.89
1.64
Vert. Flips
0.62
0.94
1.13*
1.14
1.11*
1.02
1.18*
1.14*
Det. Vert, Flips
0.18
1.19
Random Crops
0.90
1.58
1.88*
2.00
1.99*
1.99*
2.04
Flips&Crops
0.87
1.60*
2.08*
2.30*
2,35*
2.28*
0.99
2.35*
Perspectives
0.78
1.33*
1.66*
1.90
1.97
1.94*
1.87*
1.96*
Jitter
0.88
0.90
0.94
0.88
0.82
0.80
0.90
0.82
Blur
0.77
0.75
0.74
0.70
0.67
0.67
0.73
0.69
AutoAug
0.71
0.92
0.99
1.01
1.10
1.14
1.40*
1.37
AugMix
0.86
0.97
1.00
1.02
1,03
1.02
1.15*
1.14*
RandAug
0.79
1.30*
1.58*
1.80
1,86
1.88*
1.81
1,87
TrivialAug
0.59
0.91
1.12*
1.21
1.32*
1.48*
1.78*
1,86*
& Flips&CCrrrss
0.60
1,22
1.73*
2.04
2.20
2.23*
2.29*
2.37*
AugMix&&Fips&&Copss
0.75
1.47
1.94*
2.23
2.26*
2.30*
2.15*
2.20*
RandAuugFliiss&CCmm
0.67
1.39*
1,94*
2.23*
2.26*
2.31
2.24*
2,24*
TivialAAngFFiis&CCCCC
0.51
1.03
1,58*
1.92
2.10*
2.22
2.37*
2,55*
CIFAR100 Val. Accurooy
0.75
CIPARI00 Vol. Actunry
0.7
0.6
d7k
0.63
Augmentation Strategy Roelir (No Arg)
Aug. Strategy
Horis, Flip
Perppecivee
Size of Datasee Generated from 48kc Base Samples
Number of Base Samples
Figure 18: Left: Validation accuracy versus dataset size as larger datasets are generated from a fixed number of base samples and selected data augmentations. ResNet-18 models are trained on fixed datascts generated via augmentation from 48000 hase samples from the CIFAR-100 100 train set and evaluated on the CIFAR- 100 val. set. Right: Extrapolated scaling bchavior of reference models for CIFAR-100.
24

---

Published as a conference paper at ICLR 2023
Table 10: Extended table of Exchange rates for augmentations applied to 48000 base samples from the EMNIST training set, compared reference models trained without augmentations for ResNet-18 models. We measure the exchange rate w.r.t. accuracy the EMNIST val. set. Values marked with fall outside the range reference datasets and are extrapolated using power laws. Values marked with / are outside the range the estimated power law, meaning that (at least according to the behavior predicted by it), no amount of additional real data with be sufficient to match the accuracy achieved with this augmentation there is no exchange rate.
to 124800 samples
EMNIST (in-domain)
Augmentation
Ix
2x
4x
8x
16x
32x
rand (160)
rand (640)
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Horiz. Flips
0.12
0.15
0.15
0.16
0.15
0.15
0.16
0.16
Det. Horiz. Flips
0.02
Vert. Flips
0.12
0.16
0.17
0.18
0.16
0.17
0.17
0.17
Det. Vert. Flips
0.02
Random Crops
0.61
0.93
1.90
2.10
5.90*
V
Clips&Crops
0.10
0.15
0.18
0.19
0.23
1.12
0,20
Perspectives
0.95
2.04
1.99
2.11
2.18
Jitter
0.72
1,45
0.90
0.91
1.36
0.98
1.03
1.24
Blur
0.82
0.81
0.75
0.88
0.94
0.93
0.92
1.16
AutoAug
1.24
1.26
0.99
0.92
2.07
13.79
2.03
AugMix
1.60
0.90
1.04
2.10
2.04
2.02
1.56
1.99
RandAug
1.24
2.14
2.05
2.07
V
V
V
TrivialAug
0.86
0.79
0.96
1.91
1.57
1.84
V
V
0.10
0.12
0.12
0.14
0.21
0.21
0.30
0.25
AugMixx&&Fliss&Croos
0.10
0.12
0.12
0.17
0.17
0.19
0.21
0.16
Rand/Aug&Flips&CCamm
0.10
0.13
0.15
0.18
0.20
0.20
0.24
0.21
TrivalAAug&FiissCCoos
0.06
0.10
0.11
0.15
0.20
0.22
0.28
0.23
0.55
Tun
0.9
EMNIST Val. Accuary
0.34
EMNIST Val. Acturcy
0.85
0.03
0.8
0.32
Augmentation Strategy
0.9
Strategy Peopperive
0.7
Bastine (No Aug,
0.9
TriialAgg
Flip
0.89
7.7.5
T0oR
1M
I0M
1000
10k
100k
Samples
Samples
Figure 19: Left: Validation accuracy of base samples and selected data augmentations. ResNet-18 models trained on fixed datasets generated via augmentation from 48000 base samples from the EMNIST train and evaluated on the EMNIST val. set. Right: Extrapolated scaling behavior of reference models for EMNIST.
dataset size larger datasets generated from fixed number
Table Gradient standard deviation across batches at of training and flatness measurements Mobilenet V2 models trained on CIFAR-100 with various augmentations and strategies for sampling augmented views. Averaged over 3 runs
Augmentation
Fixed Views
Same Batch
Grad. Std.
Flatness
No Augmentation
46.39
7.78
No
No
46.37
7.79
Horiz. Flip & Rand. Crop
Yes
No
42.62
3.41
25

---

Published as a conference paper at ICLR 2023
Godlien Sundadd Dev. Dev.
Augmentation, Floed Views, Same Batch
Aug.
Hortr. Fig Rand. Grop, No, No
Horiz. Rand.
Figure 20: Standard deviation of gradient across epochs for different augmentations and different mini-batch sampling strategies. Each indicates mean over rums, shaded regions represent confidence intervals of width one standard error.
D ALTERNATIVE POWER LAWS
In the main body, we propose to investigate power laws of the form f(x) ax and fit these to measured data to estimate scaling rates for accuracy with respect to number of samples. We believe this choice to sound in principle, as over all variants, e.g. in Figure 1 (left), these curves fit experimental results well. In this section, though, we want to validate this choice and discuss possible alternatives.
D.1 EQUATIONS SUGGESTED VIA SYMBOLIC REGRESSION
To potentially discover alternative functional forms we turn to symbolic regression using machine learning scoring rules (Cranmer et al., 2020) using the implementation Cranmer (2022). We search for symbolic expressions containing elementary operations and exponential functions with symbolic complexity less or equal than 10. For exponentiation, we limit complexity in the exponent to 1. We search for 24000 iterations and then extract the functional form of the equation discarding the constants obtained during discovery, and use standard nonlinear regression, as described previously fit constants for each curve f and faug. We symbolically invert the found equation f and compute described in Section 3.
f (ffmg) (x)
To run a representative example with higher complexity, we re-evaluate Figure and discover a new functional form described above using data from baseline experiments without augmentation. We first note that we refind a result close to our original hypothesis with fsym(x) = 0.061227907 1.2693417, at complexity 5. Yet, we find the following function with complexity 10 using symbolic regression:
fsym (I) 0.8976371444333596
i.c. the functional form
f(x) = exp (ox
We plot results for re-analysis of Figure with this form in Figure 21. Unsurprisingly, this is better fit for the baseline without augmentations, data on which this function was found. Interestingly though, this does not lead significant qualitative changes. The perspective transform valued differently based on this new fit of the baseline, but trends are broadly consistent.
we take this to the extreme, and search for a functional form specifically without atoms used previously (containing now only additions, subtractions, divisions and multiplications), we find
10798. 98356334254
fsym xx) = 0. 0..86576673398557
17236. 4788563776533222
from which we fit
b
f(x)
T +
26

---

Published as a conference paper at ICLR 2023
Readire (No Aug?
BlpokCopps
Horx Hlpe
CINIC-10 10 Val.. Accuracy
Prospective
Effeettive Extra Samples from Aug.
0.
1506
0.9
Augmentation Strateyy
Poweline (No Aug?
04
Flpp&Crpp Hona, Flips
0.3
Perpective
500
1000
2000
5000
10000
20000
50000 100000 200000
30k
100k
1500
2006
2508
30Ck
350k
Number Base Samples
Number of Base Samples
Figure 21: Func. Form f(r = exx tea+tyy found via symholic regression for select augmentations applied randomly and the gain in terms of effective extra samples from Equation (I). Fitted curves marked in solid colors, with extrapolated regions dashed. Left: Number of base samples (from CINIC-10) horizontal axis compared to validation accuracy. Right: Number base samples compared to effective extra data, showing how the benefits of each data augmentation scale as the model is trained on more and more data.
the logarithmic
Aug?
Plpp
0.0
Harx Flipe
CINIC-10 11 Val. Actirxy
0.7
Perspective
Effectivee Extm Samples from Aug.
TrecllArg&FlppACCng
3006
0.5
Augmentation Strategy
2006
Poeeline (No Aug)
0.4
Fipp&Cnps
Plip
1006.
0.3
Prequeitee
500
1000
2000
5000
10000
20000
50000
100000 200000
300
100k
1500
2006
250h
200k
350
Number Base Samples
Number of Base Samples
Figure 22: Func. Form f(x) randomly and the gain colors, with extrapolated regions dashed. Left: Number of base samples (from CINIC-10) on logarithmic horizontal axis compared to validation accuracy. Right: Number of base samples compared to effective extra data, showing how the benefits of each data augmentation scale the model is trained on more and more data.
found via symbolic regression for select augmentations applied of effective extra samples from Equation (1). Fitted curves marked in solid
which we also use for an exemplary reanalysis and show in Figure 22. This agains fits the baseline very well, but for example fits the result with flips and crops less well than Figure
We could ultimately also search for functions with higher complexity. For example, this is symbolic regression result with larger complexity:
Ssym 0. 70768861 592998 to 4000000000000000000000000 7 ARORTOESES
0. 0.1037784774689999,
from which we extract
Saym a 1yo
and which we visualize in Figure 23. Here, we note the close relationship to the previously found result at complexity 10, and similar conclusions hold.
D.2 ALTERNATIVE SUGGESTIONS
Alternatively, we know that the functional form f(x) = ax ax + can only locally describe the relationship between accuracy and data samples, the resulting function is potentially unbounded above for some choices of (a, b, c), yet accuracy is strictly bounded by 1.0. As, such, we might wonder about fitting globally accurate form, such as f(x) tanh (æ -) + b, which is bounded. We include a re- re-interpretation with this form Figure 24, but again find no broad qualitative changes.
27

---

Published as a conference paper at ICLR 2023
Reedire (No Aug?
Horx Flip
CINIC- 10 Vall. Accuracy
Prespective
Effeettve Extr Samples from Aug.
TeccaAArgFFippACCo
0.
2006
0.5
Augmentation Strateyy
2506
Poweline (No Aug?
2006
04
Flpp&Crop
1506
Hona, Flips
100k
0.3
Perpective
500
1000
2000
5000
10000
20000
50000
100000 200000
300
100k
1500
2006
2508
30Ck
350k
Number Base Samples
Number of Base Samples
Figure 23: Func. Form f(x) = a (ox+1y² found via symholic regression for select augmentations applied randomly and the gain in terms of effective extra samples from Equation (I). Fitted curves marked in solid colors, with extrapolated regions dashed. Left: Number of base samples (from CINIC-10) horizontal axis compared to validation accuracy. Right: Number base samples compared to effective extra data, showing how the benefits of each data augmentation scale as the model is trained on more and more data.
the logarithmic
0.9
Baselinz (Na
Cripu&Cropp
Haria, Flips
CINIC-10 Val. Accuray
0.7
Pecipactive
Effective Extra Samples from Ang
0.6
100k
0.5
Augmentation Stratey
50k
Basclin (No Aug.)
0.0
FripsdCCpps
Plip
0.2
Perppcctve
-50k
-100k
1000
2000
5000
10000
20000)
500000
100000
200000
100k 150k
2000
250k
2006
350h
Number of Base Samples
Number Base Samplees
Figure 24: Func. bounded Form f(x) tanh (x-*) + found via symbolic regression for select augmenta- tions applied randomly and the gain in terms of effective samples from Equation (1). Fitted curves marked solid colors, with extrapolated regions dashed. Left: Number base samples (from CINIC-IO) on the logarith- mic horizontal axis compared to validation accuracy. Right: Number base samples compared to effective extra data, showing how the benefits of each data augmentation scale as the model is trained on more and more data.
E EXPERIMENTAL ABLATIONS
To validate the experimental setup executed in the main body of this work, run series of comparative studies. First, we construct a CINIC-10 setup in this work with a novel, sanitized test set that makes it hard to compare validation accuracies to baseline CIFAR-10 and 10. As such, we verify our implementations of all architectures and chosen budget and training setup in Table 12, where we find these implementations to perform as expected, and reach high performance on CIFAR-10. 10. For reference, we also record the number of parameters for each investigated model here.
Secondly, we vary the budget used in our study. In all other experiments in this work we consider a budget of 60000 steps (which 160 epochs batch size 128 and dataset size of 48000). In Figure 25, we check, for the case of ArivialAug, how the results of Figure 1 vary, as the budget is varied. There, we find that qualitative behavior stable across all considered budget variations, but quantitative results depend on the chosen budget.
Third, we switch from validation accuracy after training to peak validation accuracy observed during training in Figure 26. Peak validation accuracy during training stopping results, if the investigated models would overfit to their training data. However, we find in Figure 26 that behavior and analysis are almost indistinguishable under this metric, validating the choice of final validation accuracy without early stopping and the choice fixed budgets for all dataset sizes.
oracle would return early-
28

---

Published as a conference paper at ICLR 2023
Model Architecture
Number of Parameters
CIFAR-I0 Validation Accuracy
ResNet-188 (width 64)
11 173 962
94.75
ResNet-18 (width 4)
44 622
78.25
ResNet-18 (width 16)
7014 662
90.76
ResNer-18 (width 128)
44 662 922
95.08
ResNet-11 (width 256)
178 585 866
95.38
ResNet-8
78 044
81.06
ResNet-20
272 474
88.82
ResNet-110
1730 714
93.51
ResNet-152
156 618
95.39
PyramidNet-110
3904 446
94.95
VGG13
4416 010
93.07
Swinfrranfrrmer (v2)
090 794
77.86
ConvMixer
277 962
94.57
Baseline validation of implementation. For each of the model architectures investigated in this work, we report number of parameters and validation accuracy of the model when trained on CIFAR-10 with the same experimental setup as described in the remainder of this work and augmentation policy TrivialAug & Random Crops & Flips.
Table 12:
130000
Edfective Extra Samples from Aug.
30000
100
600000
900000
50%
-50k
50k
1006
150k
200k
250k
200k
Number Samples
Figure 25: Power laws, Variant of Figure 1. Number of base samples compared to effective extra data, showing how the benefits of each TrivialAug scale the model trained data and vary the model is trained with various step budgets.
We further include additional model architectures in Figure 27, validating that model family is much stronger predictor of qualitative behavior than model size, by including additional results for the larger ResNet-1 10, ResNet- 152, and the smaller ResNet-8 and ResNet-20.
0.9
1000
0.0
E-fecttve Extro Samples from Aug.
0.7
Peak CINIC-30 Val.
0.5
Augmentation Strategy
Bradine Aug.)
(No Aug,)
Flips&
0.3
Flips&CCpp
Flips
-00
Huric. Flips
Peoquerive
Projxctivv
TovaiAAu&flpsssCCg
200L Number of Base Samples
Number of Base Samples
Figure 26: Power laws based on Peak Val. Accuracy, Variant of Figure 1. Evaluating f(z) = ox + for select augmentations applied randomly and the gain in terms of effective extra samples from Equation (1). Fitted curves marked in solid colors, with extrapolated regions dashed. Left: Number of base samples (from CINIC-10) on logarithmic horizontal axis compared to peak validation accuracy. Right: Number base samples compared to effective extra data, showing how the benefits each data augmentation scale as the model is trained on more and more data.
29

---

Published as a conference paper at ICLR 2023
Aoch, Argrassation
100k
Retiin 1t, Teer moeterchee kgg
Effective Extra Samples from Akg.
CNNCC 50 Aooaic
Rashie-l1 Busilie
Rablin-ikk Toraktaght look
Redie
Rattie-122 Crapee
Redivat Sontinn the
-100k
Restien, Thie pephi
VeGrE Aug)
Commitser
PyramicNei 110
ResNes 110
ResNer 152
Resher 18
BesNeeZD
ResNedi
SwinTrandiomer
Pynoratiio Trivsskug pulli
2004
Sobiluaatoma Budiae Aea)
VGG13
hukihowlssn to Augolt po
Narber of Bese Samples
Number of Base Samples
Figure 27: Extension of Figure Power Laws and Exchange Rates via Equation (1) for TrivialAug when modifying more model architectures. Left: Accuracy power laws for various vision architectures. Right: Exchange Rates for select vision architectures. Exchange rates behave similarly for large classes of architectures.
Effeerive Extr Samples from Aag.
400M
0.8
ImageNer Val. Accuary
200M
0.6
-20M
0.4
-400M
0.2
Augmentation: Strategy
Buseline (No Aug]
(No Ang)
Elip
Rand. Resized Crops Flips
Trova RBCorps
10K
100k
1M
10M
100M
0.28
044
0.88
0.88
18
Rase Samples
Nomber Base Samples
Figure Power laws for a ResNet-50 on ImageNet (xx az + for select augmentations applied randomly and the gain in terms of effective extra samples from Equation (1). Fitted curves marked solid colors, with extrapolated regions dashed. Left: Number of base samples (from ImageNet) on the logarithmic horizontal axis compared to validation accuracy. The scaling behavior of each augmentation is closcly matched by these power laws. Right: Number of base samples compared to effective extra data, showing how the benefits of each data augmentation scale as the model trained on more and more data.
F IMAGENET RESULTS
We also investigate how well these power laws would fit ImageNet training runs (Russakovsky et al., 2015). ImageNet is an ideal setting for our analysis, given that the dataset is relatively small compared to its complexity. Given unlimited compute one would rather train reference models on larger splits, e.g. ImageNet-21k or JFT-300m (Sun et al., 2017) and compare the validation accuracy of those reference models to ImageNet models trained with augmentations, as discussed in Section 3.
To train these models we follow the recently proposed training regime of Wightman et al. (2021). We train for 65536 steps (about 100 epochs) using the LAMB optimizer with peak learning rate of Se - 3 and cosine decay after a warmup of 3125 steps. We additionally apply label smoothing with 0.1. We train a ResNet-50 model as described in He et al. (2019a). As unaugmented baseline, consider a pre- pre-processing of centered crops size 224. For the augmented variant, we augment with random resized crops (with ratios between 3/ 3/4 and 4/3 usual) of size 224 and random horizontal flips. For TrivialAug, we use TrivialAug as described on top of these random resized crops and flips. In all cases, validate by resizing to 256 x 256 and center cropping to 224 x 224.
30

---

Published as a conference paper at ICLR 2023
Asgrantation StaKEE
Auginestation Stralegyy
Bunka Aaginiis
Toual www.tegiihbllrrreeeee
Empirical Flannsss
Huii Flys
Flips&Cropp
Baseline (No Ang
Propective
Trivid Aughi
46
Erejica Tlaress
Validation Accunccy
Figure 29: Dataset scaling and flatness on CINIC-10 Direct measurements flatness plotted measures of model performance on CINIC-10. Left: Flatness compared to effective exchange rates for a number of augmentation strategies and dataset sizes. Trend lines are ordinary linear regression, r values are included in the legend. Right: Same data, but plotted against raw CINIC-10 validation accuracy. Dataset size is inverse-square proportional marker size.
G REMARK: WELL-DEFINEDNESS OF EXCHANGE RATES
The exchange rate defined Equation (1) relies on the functional form of both feer and fmug. To be more precise, in this work would ideally assume that both functions strietly monotonically increasing maps f [0, oo) > [0, 1) and surjective. However, both due to the simplicity of the chosen power laws (which are potentially unbounded), and actual observed data, both bounded range and surjectiveness can be violated practice. As such, as f is only defined on Im(f) C [0, 1), this can result in the exchange rate being only defined see this happening in Figure (right), where we analyze an exchange on the out-of-domain dataset CIFAR- 10-C (Hendrycks & Dietterich, 2018). Here, the exchange rate for e.g. TrivialAug with flips and crops is undefined for samples sizes greater than about 3000, with the exchange rate approaching infinity as samples sizes approach this limit.
subset of sample sizes S [0, 00). We can
While this may be inconvenient to visualize, we believe it to be an interesting feature of the concept of exchange rates. the undefined range, there really is no exchange possible: Based on existing experimental data, even extrapolation to infinite additional in-domain CINIC data is not enough to increase accuracy on out-of-domain CIFAR-I0-C to the same value that is reached with the augmentation broadening the data distribution.
H DATASET SCALING AND FLATNESS
In extension of Section 5.3, we provide additional data in Figure 29, As discussed in the main body, we find strong correlation in the number of samples gained through augmentation and flatness for all augmentation strategies except TrivialAug. We hone in this trend a function of dataset size on the right hand side of Figure 29. TrivialAug is possibly an outlier as it produces models that remain relatively flat for all sample sizes considered, as discussed in Section 5.3, and may lead to models that are "artificially" flat, but without correlation with accuracy or improved exchange rates.
31


Reviews:
[{"review_id": "vu2BtwP6GMn", "reviewer": "Reviewer_G4Yp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes data augmentation in detail for (small) computer vision tasks with (small) networks. In particular, the paper analyzes the following aspects of data augmentation:\n* What is the *exchange rate* between augmentations and more data?\n* How does augmentation help OOD generalization in comparison to more data?\n* How does augmentation relate to invariance in the models?\n* Augmentations add a source of randomness to SGD that can help to find \"flatter minima\".\n* A large number of experiments is done, mostly on CIFAR-related datasets, with a variety of architectures and dataset sizes. The exchange rate is based on fitting of power-law curves to the results.\n", "summary_of_the_review": "The paper contains several interesting novel ideas and results. However, there are also several issues around extrapolation and interpretation of the results. I therefore think that the paper in its current form is marginally below the acceptance threshold, but I would be happy to increase my score during the discussion period if some of the concerns can be addressed by the authors.", "strength_and_weaknesses": "**Strengths**\n* The topic of the paper is interesting and relevant to ICLR. The concept of an \"exchange rate\" between augmentations and more data is intuitive and the paper motivates it well.\n* The experiments are extensive and cover interesting questions.\n* The paper is generally well-written and easy to follow. \n* All important design decisions are explained and the used code is included in the supplementary material.\n* The detailed analysis of gradient standard deviation during training with and without augmentations is specifically interesting in my opinion.\n\n**Weaknesses**\n\nThe extrapolation into regimes where the use of augmentation corresponds to a \"negative amount of additional training data\" seems a bit strange to me, at least under \"normal conditions\" like in Fig.1. I think most practitioners would argue that this is not plausible. This seems to correspond to interpolations into an accuracy that approaches the best achievable rates. Also, f_{ref}^{-1} may not be well-defined any more for accuracies that are significantly above the largest observed \"reference\" accuracy. It looks to me like the extrapolations beyond the max. possible \"number of base samples\" are more confusing than they help. Further, this may point to potential problems in the functional form of the power law that was used, maybe the used functional form is too simple to yield good results in extrapolation?\n\nThis extrapolation effect then seems to also lead to some non-obvious conclusions, e.g. \"diverse but inconsistent augmentations [...] are hindrances at scale\" where it is not entirely clear to me that the qualification \"inconsistent\" is obvious from the experiments. \n\nStill regarding extrapolation:\n* In Fig3 (left) it seems that for width 4 (and partially for width 8) augmentations *hurt* performance when using more than 100k training samples. This seems quite surprising and would probably warrant an explicit discussion. Or maybe TrivialAug is just not useful in this setting? Similar for Fig.3 (right) and the SwinTransformer.\n* In Fig4 (right) it also seems somewhat implausible that for a number of base samples in the thousands, augmentations would be worth a factor of 100 more in training data (even though this is for generalization under distribution shift). Similarly, the graphs seem to indicate that for \"horizontal flips\", the augmentation is worth \"minus half the training data\" for ~100k base samples. This is also somewhat implausible to me.\n\nAt the start of Sec.3, I had a hard time understanding what exactly \"0\", \"1\", etc. mean in Fig.2 and Fig.5. I read Section 3.1.1 several times and I am still not sure how to exactly interpret the experiments. I'd appreciate it if the authors could clarify this.\n\nSimilarly, in the second paragraph of Sec 4.2 it was not clear to me how exactly the train and test data are constructed. E.g. often only rotations by multiples of 90 degrees are used for digital images, and the x-range of Fig.7 (left) also only goes up to 4. Does this mean that up to 4 samples were used? But then the test data would be included in the train data for 4 samples per class, which seems not to be the case. I'd appreciate it if the authors could clarify this a bit more.\n\nI had the impression that the conclusions drawn in Sec 4.2 are a bit too strong. The experiment is interesting but it seems ultimately a bit simple and some further analyses (e.g. breakdown of improvements by classes, example successes and failures) are needed. It seems not impossible that for large augmentation ranges, the network could just learn to recognize dominant color distributions in this particular scenario. Can this be ruled out?\n\n**Additional Comments**\n\nSec. 3.1.2 / Fig. 3 (left) - this seems to point to the fact that augmentation/regularization helps more with increased model capacity. Maybe this is obvious, but I don't think I saw this being discussed in the paper.\n\nI am slightly confused that the authors use a ResNet *18* for CIFAR-like images. In the original paper by He et al. a ResNet *20* was specifically designed for CIFAR-like data, while the *18* version is used for larger ImageNet images. On the other hand, the authors discuss in the Appendix that they used a \"usual CIFAR-10 stem\" and refer to the \"Bag of Tricks\" paper. In that paper I did not find any mention of CIFAR, though. I also briefly looked at the code and got the impression that the \"18\" version seems to use a [2, 2, 2, 2] configuration (which is the classical ImageNet variant as far as I understand). But I may have been confused here or it is considered irrelevant for the experiments here. Could the authors clarify this? \n\nIt would also be good if the best achieved accuracies of the networks could be compared to results from the literature. Here, the goal would not be to claim state-of-the-art performance, but it would be good for the reader to see that the experiments are run in a regime that is sufficiently close to other published numbers, such that it remains plausible that the drawn conclusions remain valid in that regime and are not influenced too much by operating far from that point.\n\nA few additional related works that could be relevant (I'm not suggesting all of these need to be included or cited):\n* [Bad Global Minima Exist and SGD Can Reach Them](https://arxiv.org/pdf/1906.02613.pdf) has some results on how augmentation is related to \"better minima\".\n* [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://arxiv.org/pdf/2106.10270.pdf) also discusses the relation between augmentation, regularization, and training data size.\n* [A Group-Theoretic Framework for Data Augmentation](https://arxiv.org/pdf/1907.10905.pdf) also seems relevant in this context.\n\nMinor comments/typos (not affecting evaluation):\n* p.1 \"benefits of augmentations and the number of samples rises\": and->as ?\n* Fig. 2 (and potentially others): The colors for \"1\" and \"random\" are hard to distinguish.\n* p.4 \"can be attributes simply\" -> attributed\n* In Fig.6, does \"Aug\" here mean \"only flipping\"?\n* Sec.4.1: I think another common name for \"prediction averaging\" is \"test-time augmentation\", and \"orbit selection\" is also called \"normalization\". Maybe it would be useful to mention these, not sure.\n* p.7 \"We instantiate and E2CNN\" -> an\n* References: in some reference titles, lowercase is used where it shouldn't, e.g. \"lie group\", \"sgd\", \"bayesian\", \"cnns\".\n* References: in some references, only the ArXiv version is references, while the paper has appeared in a conference or journal, too.\n\n---------------\n### Update after authors' response\n\nI would like to thank the authors for their clarifications and the changes made to the paper. I think these have addressed my concerns and I think the quality of the paper has improved overall. I am still a bit worried that the nuances of the results in the extrapolation regime are easy to miss for the reader. (De-emphasizing these in the graphs would be one way to address that.) But overall I think the paper is now above the acceptance threshold. Therefore I am increasing my score.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CqK8pOlCbjN", "reviewer": "Reviewer_iXJA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tries to quantify the value of augmentation by measuring its effectiveness compared to adding more data. It tried to also examine the effectiveness under different settings such as model size, variants, and out-of-distribution settings. Lastly, it tried to explain why augmentation is effective by measuring gradient standard deviations and noted that it has a regularizing effect because it flattens the gradient.", "summary_of_the_review": "Augmentation is a technique often employed by ML practitioners for its practical values. The paper tried to quantify trade-off between augmentation vs. one additional labeled data and this framework of trade-off can benefit the practitioner to make the right decision at the right time.\n\nHowever, despite its very empirical nature, it lacks experiments on different tasks, on datasets of varying size and complexity, and on state-of-the-art models since, at their scales, the augmentation may exhibit very different characteristics. Without them, this paper can only be served as intuition reaffirming data but cannot be used as a rule or a way to generally compute the exchange rate.\n", "strength_and_weaknesses": "While benefits of augmentation are quite well proven empirically, there wasn't too much valuation of one additional data collected. This paper delves into when, how much, and what augmentations can help which is quite practical. \n\nWhile it lays a good starting point, it mostly deals with empirical finding on a relatively small and simple dataset. Without a theoretical finding, one would need to conduct similar experiments outs to evaluate usefulness of augmentations on a specific dataset, model, and/or task. Given that the shape of the curve can differ by any of those factors, the paper can be seen as only reinforcing intuitions built by others but the conclusion is mostly limited to CIFAR and CINIC dataset. \n\nAlso, given that experiments dealt with models with relatively low capacities and very small number of classes, it certainly would have been more interesting to see evaluation done on more complex task where there are more uncertainty in data and where it is difficult to over train (in the paper, model was trained sometimes up to 160 epochs). For example, the benefit of augmentation is less noticeable, at least compared to other charts in the main body of the paper, in the figure 17 and 19 and it certainly is less interesting for SwinTransformer, a model with more capacity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BQrp6wGhZ_T", "reviewer": "Reviewer_VSLP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors empirically investigate the effects of data augmentation on model performance. They first define a performance metric called \"exchange rates\", measuring the effects of data augmentation in terms of the size of extra real data required for the same performance. From various perspectives such as consistency and diversity of the augmentation method, base sample size, model capacity and architecture, in-distribution vs. out-of-distribution, they find interesting empirical performance patterns. They further study whether the performance boost can be solely attributed to utilizing invariance in the data distribution. Finally, they link the empirical findings with the implicit regularization and flat minimum literature by measuring the gradient norm and flatness metric of the trained models. ", "summary_of_the_review": "The paper presents a systematic empirical investigation of an important topic in modern machine learning -- data augmentation. The idea of measuring data augmentation performance in \"units of data\" can be applied and compared across datasets and model architectures, and brings in a new perspective to interpret the performance improvement. The paper is well-organized, with well-controlled experiments and clear presentation of results to support the conclusions of each section. All in all, I think the paper would be a valuable addition to the literature of data augmentation research.", "strength_and_weaknesses": "Strength\n- Data augmentation is an area that has huge practical impact but is short of systematic studies. This paper is an important addition.\n- The \"exchange rate\" concept provides a new tool to measure the effects of data augmentation. I think it has the benefit of being more universal / \"normalized\" compared with other metrics such as \"accuracy improvement\", when applied across datasets and model architectures.\n- Many aspects / hypotheses are considered; conclusions are drawn using carefully designed experiments.\n\nWeakness\n- While I understand that the power-law model is a visually plausible and convenient choice, I don't find it overwhelmingly convincing. For example, in Figure 1 (left) the performance of the baseline seems to also saturate close to 200k base samples, yet the power-law model assumed a linear extrapolation.\n- I find the words \"repetition\"/\"repeat\" in Sec. 3.1.1 and Figure 2 to be confusing. Literally it seems to mean iterating through the same dataset (with the same order) without generating new data. From the context, however, it seems to represent how many augmented data are generated (measured as multiples of base samples). Could you confirm if the latter understanding is correct? Also, what does \"random\" mean in the legend of Figure 2?\n\nOther feedback: \n- The paper only studies label-preserving data augmentations; conclusions about other data augmentation methods, e.g. mixup, require further research.\n- On the benefit of data augmentation for out-of-distribution dataset, it seems to me that links could be drawn from the area of domain adaptation and / or (un)supervised pretraining, etc. Given previous research in those areas, it is not surprising that diverse augmented data may work better than real data from the source domain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "AD00FGGa-i0", "reviewer": "Reviewer_WExn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the reason why data augmentation is effective in learning models. They look into different hypotheses: how augmentation encourages invariance, variance regularization, providing an additional source of stochasticity. They also look into the scaling laws of augmentation. The findings of this paper shed light on where and when augmentation is most effective in computer vision.", "summary_of_the_review": "This work is well-motivated, clearly explained, and sufficiently supported by analysis. There are some questions for the authors (see section Strength And Weaknesses), however overall it's a valuable contribution to the field. ", "strength_and_weaknesses": "- The idea is nice, focusing on getting a better understanding of the impact of data augmentation is an important research direction. The findings can improve how data augmentation is done in both computer vision and NLP tasks.\n\n- The presentation of \"takeaway\"s from the analysis of each section is helpful for summarizing the findings.\n\n- The authors mainly focus on computer vision where the image manipulation functions are very straightforward (e.g., flipping an image). I'm wondering how the findings and takeaways of this work generalizes to text augmentation as well. In NLP, by manipulating texts to augment the datasets, we inadvertently introduce noise. That is because [automatic] context manipulation can lead to grammatically or semantically incorrect sentences. What is the authors' insight about their findings in this scenario?\n\n- In figure 3 the authors show that SwinTransformer is less positively impacted by the augmentations in comparison with other architectures. Is that interpreted as how transformers are in general less impacted or is there something about the SwinTransformer architecture in particular?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster