Understanding the detrimental class-level effects of data augmentation
Authors: Polina Kirichenko
Mark Ibrahim
Randall Balestriero
Diane Bouchacourt
Shanmukha Ramakrishna Vedantam
Hamed Firooz
Andrew Gordon Wilson
Venue: NeurIPS 2023 poster (2023)
Keywords: data augmentation
class-dependent bias

Abstract: Data augmentation (DA) encodes invariance and provides implicit regularization critical to a model's performance in image classification tasks. However, while DA improves average accuracy, recent studies have shown that its impact can be highly class dependent: achieving optimal average accuracy comes at the cost of significantly hurting individual class accuracy by as much as 20% on ImageNet. There has been little progress in resolving class-level accuracy drops due to a limited understanding of these effects. In this work, we present a framework for understanding how DA interacts with class-level learning dynamics. Using higher-quality multi-label annotations on ImageNet, we systematically categorize the affected classes and find that the majority are inherently ambiguous, co-occur, or involve fine-grained distinctions, while DA controls the model's bias towards one of the closely related classes. While many of the previously reported performance drops are explained by multi-label annotations, we identify other sources of accuracy degradations by analyzing class confusions. We show that simple class-conditional augmentation strategies informed by our framework improve performance on the negatively affected classes.

Understanding the Detrimental Class-level Effects of Data Augmentation
Polina Kirichenko
Mark Ibrahim²
Randall Balestriero"
Diane Bouchacourt Andrew Gordon Wilson
Ramakrishna Vedantam"
Hamed Firooz²
Neww York University
Meta AI
Abstract
Data augmentation (DA) encodes invariance and provides implicit regularization critical to a model's performance in image classification tasks. However, while DA improves average accuracy, recent studies have shown that its impact be highly class dependent: achieving optimal average accuracy comes at the cost of significantly hurting individual class accuracy by as much 20% on ImageNet. There has been little progress in resolving class-level accuracy drops due to a limited understanding of these effects. this work, we present framework for understanding how DA interacts with class-level learning dynamics. Using higher- quality multi-label annotations on ImageNet, we systematically categorize the affected classes and find that the majority are inherently ambiguous, co-occur, involve fine-grained distinctions, while DA controls the model's bias towards one of the closely related classes. While many of the previously reported performance drops are explained by multi-label annotations, our analysis of class confusions re- veals other sources of accuracy degradation. We show that simple augmentation strategies informed by our framework improve performance on the negatively affected classes.
Introduction
Data augmentation (DA) provides numerous benefits for training of deep neural networks including promoting invariance and providing regularization. particular, DA significantly improves the generalization performance in image classification problems when measured by average accuracy 24 19 2 18 However, Balestricro et al. [ and Bouchacourt et al. [9] showed that strong DA, in particular, Random Resized Crop (RRC) used in training of most modern computer vision models, may disproportionately hurt accuracies on some classes, c.g. with up to 20% class-level degradation ImageNet compared milder augmentation settings (see Figure [ left). Performance degradation even on a small set of classes might result in poor generalization on downstream tasks related to the affected classes (55), while in other applications it would be unethical to sacrifice accuracy some classes for improvements in average accuracy 28
Balestriero et al. 1 attempted to address class-level performance degradation by applying DA selectively to the classes where the accuracy improves with DA strength. Surprisingly, they found that this augmentation policy did not address the issue and the performance still degraded with augmentation strength. this work we perform detailed analysis and explore the mechanisms causing the class-level performance degradation. In particular, we identify the interactions between clssscondiitonal data distributions as the cause of the class-level performance degradation with augmentation: DA creates overlap between the data distributions associated with different classes. As a simple illustrative example, in Figure (right) we show that the standard RRC operation creates overlap between the "car" and "wheel" classes. As result, the model learns to predict label "car" on "wheel" images, and the performance on the "wheel" class drops. Importantly,
con-augmented classes
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Stronger augmentation degradation accuracy some classes
Analyzing class confusion types
Accuracy degradation
confusions induced by augmentation
pytorch Randem Resized Crop to model)
Imageleet
fine-grained and non-trivial
Iupeit came
36 dooss
rng 50 nnn dron
The dhad cheataal
Ambigupus
sunglass
sungltsocs
stronger sugnentatin
2003
Removing augmentation affected classes does not recover performance
Augmentation strategy which accounts class interactions leeds +2 +2.5% improvvement affected ImageNet clesses
Figure I: We show that the classes negatively affected by data augmentation are often ambigu- ous, co-occurring fine-grained categories and analyze how data augmentation exacerbates class confusions. Left: Average accuracy of ResNet-50 on ImageNet against Random Resized Crop (RRC) data augmentation strength: average of all classes (blue), average of the 50 classes which stronger RRC hurts accuracy the most (red), and the average of the remaining 950 classes (green). Yellow line indicates the default RRC setting used in training of most computer vision models. Middle: We systematically categorize the types of class confusions exacerbated by strong data augmentation: while some of them include ambiguous or correlated classes, there is a number of fine-gra ine-grained and non-trivial confusions. Right: Often the class-level accuracy drops due to overlap with other classes after applying augmentation: e.g. heavily augmented samples from "car" class can look like typical images from "wheel" class. As result, the model learns to predict "car" on "wheel" images, and the accuracy on the "wheel" class drops. To resolve the negative effect strong augmentation on classes like "wheel", we should modify augmentation strength of classes like "car".
if we want to improve the performance the "wheel" class, need to modify the augmentation policy on the class "car" and not "wheel" as was done in prior work [T] We summarize our findings in Figure I In particular, our contributions the following:
We refine the analysis of class-level effects of data augmentations by correcting for label ambiguity. Specifically, we use multi-label annotations on ImageNet [4] and measure the effects of data augmentation for each class in terms of the original and multi-label accuracy. Through this analysis, we find that class-level performance degradation reported in Balestriero al. (2) and Bouchacourt et al. 9 is overestimated (Section 4
We systematically categorize the class confusions exacerbated by strong augmentation and find that many affected classes are ambiguous co-occurring and often affected by label noise (Figure middle and Section 5). We focus on addressing the remaining fine-grained and non-trivial class confusions.
We show that for addressing DA biases is important to consider the classes with an increasing number of false positive mistakes, and only the classes negatively affected in accuracy. By taking into account our observations on DA affecting class interactions, we propose a simple
conditionl data augmentation strategy that leads to improvement on the affected group of classes by 2.5% on ImageNet (Section 6). This improvement is in contrast to the previously explored cass-coonitional DA in Balestriero et which failed to improve class-level accuracy.
We confirm our findings across multiple computer vision architectures including ResNet-50 (22), EfficientNet [61 and ViT [15] (Section F.. multiple data augmentation
transformations including Random Resized Crop, mixup 73], RandAugment [13 and colojitter (Section[G), well as two additional datasets besides ImageNet (Section

---

2 Related work
Understanding data augmentation, invariance and regularization.
Germández-Gardde and
König (24) analyzed the DA from the perspective of implieit regularization. Botev et al. [8] propose an explicit regularizer that encourages invariance and show that it leads to improved generalization. Balestriero al. [2] derive an explicit regularizer to simulate DA to quantify its benefits and limitations and estimate the number of samples for learning invariance. Gontijo-Lopes et al. [19]
and Geiping et al. [18] study the mechanisms behind the effectiveness of DA, which include data diversity, exchange rates between real and augmented data, additional stochasticity and distribution shift. Bouchacourt et al. [ measure the learned invariances using DA. Lin al. (37] studied how data augmentation induces implicit spectral regularization which improves generalization.
Biases of data augmentations. While DA is commonly applied to improve generalization and robustness, a number of prior works identified its potential negative effects. Hermann et al. [3 showed that decreasing minimum crop size in Random Resized Crops leads to increased texture bias. Shah et al. [56] showed that using standard DA amplifies model's reliance on spurious features compared to models trained without augmentations. Idrissi et al. [30] provided a thorough analysis on how the strength of DA for different transformations has a disparate effect on subgroups of data corresponding different factors of variation. Kapoor et al. [X suggested that DA can cause models to misinterpret uncertainty. Izmailov et al. [31 showed that DA can hurt the quality of learned features on some classification tasks with spurious correlations. Balestriero et al. [ and Bouchacourt et al. [9] showed that strong DA may disproportionately hurt accuracies on some classes on ImageNet, and in this work we focus on understanding this class-level performance degradation through the lens of interactions between classes.
Multi-label annotations on ImageNet. A number of prior works identified that ImageNet dataset contains label noise such as ambiguous classes, multi-object images and mislabeled examples (55. Tsipras et al. [67] found that nearly 20% of ImageNet validation set images contain objects from multiple classes. Hooker et al. (27) ran a human study and showed that examples most affected by pruning neural network are often mislabeled, multi-object or fine-grained. Yun et al. (72] generate pixel-level multi-label annotations for ImageNet train set using large-scale computer vision model. Beyer et al. 4 provide re-assessed (ReaL) multi-label annotations for ImageNet validation set which aim to resolve label noise issues, and we use ReaL labels in our analysis to refine the understanding of per-class effects of DA.
4 69 46 55
Adaptive and learnable data augmentation. may exacerbate data bias which may lead to model' suboptimal performance the original data distribution. They propose to train the model on mix of augmented and unaugmented samples and then fine-tune it on unaugmented data after training which showed improved performance on CIFAR dataset. Raghunathan et al. [49 showed standard error in linear regression could increase when training with original data and data augmentation, even when data augmentation is label-preserving Rey-Area et al. [5I) and Ratner et al. [50) learn DA transformation using GAN framework, while Hu and Li [29] study the bias of GAN-lerrned data augmentation. Fujii al. [17 take into account the distances between classes to adapt siixed-sample DA. Hauberg et al. 21 learn class-specific
Xu et al. [70] showed that data augmentation
DA on MNIST. Numerous works, e.g. Cubuk et al. 112, Lim et al. [36], Ho et al. 225] Hataya et al.
20. Li et al. [35. Cubuk et al. 133, Tang et al. [62, Müller and Hutter 43) and Zheng et al. [74] find dataset- deppnett augmentation strategies. Benton et al. ( proposed Augerino framework to learn augmentation form training data. Zhou et al. (75) Cheung and Yeung [[I), Mahan et al. 40) and Miao al. [41 learn class- or input-dependent augmentation policies. Yao et al. propose to modify mixed-sample augmentation to improve out-of-domain generalization.
Robustness and model evaluation beyond average accuracy. While Miller et al. 442 showed that model's ' average accuracy is strongly correlated with its out-of-distribution performance, there is a number of works that showed that only evaluating average performance can be deceptive. Teney et al. [64] showed counter-example for "accuracy- -o-tte--tttttteee tet-the-liee phenomenon. Kaplun et al. 33 show that while model's ' average accuracy improves during training, it may decrease on a subset of examples. Sagawa et al. [54] show that training with Empirical Risk Minimization may lead to suboptimal performance in the worst case. Bitterwolf et al. (6] evaluated ImageNet models' performance in terms of number of metrics beyond average accuracy, including worst-class accuracy and precision. Richards et al. [52] demonstrate that improved average accuracy on ImageNet and standard out-of- distribution ImageNet variants may lead to exacerbated geographical disparities.
3

---

3 Evaluation setup and notation
Since we aim to understand class-level accuracy degradation emerging with strong data augmentation reported in Balestriero et al. D. we closely follow their experimental setup. We focus on ResNet-50 models [22] trained on ImageNet [53] and study how average and class-level performance changes depending on the Random Resized Crop augmentation strength: it is by far the most widely adopted augmentation which leads to significant average accuracy improvements and is used for training the state-of-the-art computer vision models [58 65 38 We train ResNet-50 for 88 epochs using
label smoothing with a = 0.1 [60]. We use image resolution Rerain 176 during training and evaluate images with resolution Reest 224 following Balestriero et al. (. Touvron et al. [66]
and torchvision training recipe] More implementation details can be found in Appendix A
Data augmentation. We apply random horizontal flips and Random Resized Crop (RRC) DA when training our models. In particular, for an input image of size h x w the RRC transformation samples the crop scale ~ (stow, Supl and the aspect ratio r ~ Ur Tupl, where U(a, bj denotes uniform distribution between and b. RRC then takes random crop of size v shver's v shw/rr and resizes it to a chosen resolution R x R. We use the standard values for Sup = 100% and aspect ratios 4/3, and vary the lower bound of the crop scale Stow (for simplicity, we will further use s) between 8% and 100% which controls the strength of augmentation: = 8% corresponds to the strongest augmentation (note this is the default value in pytorch (88) RRC implementation) and 8 100% corresponds no cropping hence no augmentation. For each augmentation strength, we train 10 models with different random seeds.
Nlow 3/4,
ReaL labels. Beyer et al. [4] used large-scale vision models to generate new label proposals for ImageNet validation set which were then evaluated by human annotators. These Reassessed Labels (Real.) correct the label noise present in the original labels including mislabeled examples, multi-object images and ambiguous classes. Since there are possibly multiple Real. labels for each image, model's s prediction is considered correct if it matches one of the plausible labels. Further we will use l Reat (2) to denote the set of ReaL labels of example x.
Evaluation metries. We aim to measure performance of model fx( as a function of DA strength, i.e. RRC lower bound of the crop scale 8. We measure average accuracy a(s), and per-class accuracy (s) with respect to both original ImageNet labels and ReaL multi-label annotations given by:
of (s) = 1/|xa|
Σ (ff(z)= x
k] and ak
ReaL (s) 1/|Xal X
Σ [[fa(z)= E (x)),
where Xx are images from class k in validation set. We will refer a and ReaL original accuracy and Real. accuracy, respectively. To quantify the class accuracy drops, Balestriero et al. M compare the per-class accuracy of models trained with the strongest DA (s 8%) and models trained without augmentation (s 100% which effectively just resizes input images without cropping). while Bouchacourt et al. g compared class-level accuracy of models trained with RRC with s 8% and models trained with fixed size Center Crop. In our analysis, we evaluate per-class accuracy drops comparing the maximum accuracy attained on particular class k across all augmentation levels max, a& (s) and accuracy on that class when training with strongest DA (g(s = 8%). We will refer to the classes with the highest accuracy degradation computed by:
Aak max a (8) ax(s 8%)
as the classes most negatively affected by DA (with respect to either original labels or multi-label ReaL annotations). To summarize performance on the affected classes, we will evaluate average accuracy of classes with the highest Aa; (in many cases focusing on 5% of ImageNet classes with the highest accuracy drop following Balestriero et al. (.). In the following sections, we also highlight the importance of measuring other metrics beyond average and per-class accuracy which comprise a more thorough evaluation of DA biases.
While we focus the analysis of ResNet-50 ImageNet with RRC augmentation as the main setup following prior work 19. we additionally confirm our observations on other architectures (Efficient- Net [61) and ViT [14) in Appendix F. other data augmentation transformations (RandAugment [3]
colorjitter and mixup (73) ), and other datasets (CIFAR-100 and Flowers 102 (4)) in Appendix G
toroooo-ttrii--eeeeeeeeeeeeeeeee state-of- art-aodeeeeeeeeeeeeeeeee--eeeeee wawdddee-eiing-eeee wwweing-torchiiiioooo catriiiiiio-attt-t-ee -atest-prinitivee

---

Aug all IN classes Aut
Avg 950 dlsses
Avg 50 dassess Dud" Soy
On" iI, daw do
Uieribution of per-cliss der Aufiii
validation
05-
oigu BeLL
20
002
born spider Wosi
overskirt 15, Soad
academic fook
50-
valdstion acc
60-
-0-
100
75
30
100
75
35
100
15
50
55
Aconacy 600 (5)
lower
RRC scale lower bound
lower
stronger sugmentation
original
ReaL
We find that for many classes the negative effects of strong data augmentation are muted if we use high-quality multi-label annotations. Left: Average and per-class accuracy of ResNet-50 trained on ImageNet evaluated with original and ReaL. labels as function of Random Resized Crop augmentation strength (s = 8% corresponds to the strongest and default augmentation). The top row shows the average accuracy of all ImageNet classes, the 50 classes with the highest original accuracy degradation and the remaining 950 classes. The bottom row shows the accuracy of 3 individual classes most significantly affected in original accuracy when using strong augmentation. Right: Distribution of per-class accuracy drops Aag for original and Real. labels. The distribution of Aag has a heavier tail compared to the one computed with Real. labels.
Figure 2:
4 Per-class accuracy degradation with strong data augmentation is overestimated due to label ambiguity
Previous studies reported that the performance of ImageNet models is effectively better when evaluated using re-assessed multi-label annotations which address label noise issues in ImageNet
These works showed that recent performance improvements on ImageNet might be saturating, but the effects of this label noise on granular per-class performance has not been previously studied. In particular, it is unclear how correcting for label ambiguity would affect the results of Balestriero et al. [ and Bouchacourt et al. 9 on the effects of DA on class-level performance.
57 69
We observe that for many classes with severe drops in accuracy on original labels, the class-level ReaL multi-label accuracy is considerably less affected. The right panel of Figure 2 shows the distributions of per-class accuracy drops Aag" and Aa AaReat and Aag" has a much heavier tail. Using multi-label accuracy in evaluation reveals there are much fewer classes which have severe effective performance drop: e.g. only 37 classes with A AaRaLL opposed to 83 classes with Dag > 4%, moreover, there are no classes with Aa Real > 11%
note that the distribution of
ReaL > 4% as
On the left panel of Figure 2 we show how multi-label accuracy evaluation impacts the average and individual class performance across different augmentation strengths s. particular, in the top row plots we see that while the average accuracy of all classes follows a similar trend when evaluated with either original or Real labels, the average accuracy of 50 classes most negatively affected in original accuracy only decreases by 1 % with Real. labels as opposed to more significant 5% drop with original labels. The bottom row shows the accuracy for "barn spider", "overskirt" and "academic gown" classes which have the highest Aagr, and accuracy trends for all 50 most negatively affected classes are shown in Appendix C For many of these classes which are hurt in original accuracy by using stronger DA, the ReaL accuracy is much less affected. For example, for the class "barn spider" the original accuracy is decreased from 63% to 47% if we use the model trained with RRC s = 8% compared to 70%, while the highest ReaL accuracy is achieved by the strongest augmentation setting on this class.
However, there are still classes for which the Real. accuracy degrades with stronger augmentation, and in Appendix C we show Real accuracy trends against augmentation strength s for 50 classes with the highest Aa ReaL, While some of them (especially classes from the "animal" categories)
5

---

may still be affected by the remaining label noise 669 68 57 39 for other classes the strongest DA leads to suboptimal accuracy. In the next section, we aim to understand why strong DA hurts the performance on these classes. We analyze model's predictions and consistent confusions on these classes and find that the degradation in performance is caused by the interactions between class-conditional distributions induced by DA.
5 Data augmentation most significantly affects classification of ambiguous, co-oceurring and fine-grained categories
In this section, we aim understand the reasons behind per-class accuracy degradation when using stronger data augmentation by analyzing the most common mistakes the models make on the affected classes and how they evolve as we vary the data augmentation strength. We consider the classes most affected by strong DA (see Figures in Appendix ( which do not belong to the "animal" subtree category in the WordNet hierarchy [16] since fine-grained animal classes were reported to have We focus on the 50 classes with the highest Aag (corresponding to Aag > 5%), and 50 classes with the highest RaaL (corresponding to Ao Real. 4%). For pair of classes & and we define the confusion rate (CR) as:
higher label noise in previous studies [68 57 39 4.
(R) =
[ff(()= - TEXx
Σ
i.e. the ratio of examples from class l misclassified as 1. For each affected class, we identify most common confusions and track the CR against the RRC crop scale lower bound s. We also analyze the reverse confusion rate C (R)))
We observe that in many cases DA strength controls the model's preference in predicting one or another plausible ReaL label, or preference among semantically similar classes. We roughly outline the most common types of confusions on the classes which significantly affected by DA. The different types of confusion differ in the extent to which the accuracy degradation can be attributed to label noise versus the presence of DA. We also characterize how DA effectively changes the data distribution of these classes leading to changes in performance. These categories are closely related common mistake types on ImageNet identified by Beyer al. [4] and Vasudevan al. [69] but focus on class-level interactions as opposed to instance-level mistakes and particularly connect them to the impact of DA. We use semantic similarity and Real. labels co-occurence a confusion category for a pair of classes. We can measure semantic similarity by (a) WordNet class similarity, given by the Wu-Palmer score which relies on the most specific common ancestor of the class pair in the WordNet tree, and (b) similarity of the class name embeddingg To estimate the intrinsic distribution overlap of the classes, we compute ReaL labels co-occurrence between classes and as:
criteria to identify
Ok Σ [[k E (ReaL(x)) I[l E |ReaL (x)1/ Σ I[k E RReaL (x)), rex
FEX
which is the ratio of examples that have both labels l and I among the examples with the label k. Using these metrics, depending on a higher or lower semantic similarity and higher or lower ReaL labels overlap, we categorize confused class pairs as ambiguous, co-occurring, fine-grained or semantically unrelated. Below we discuss each category in detail, and the examples are shown in Figure and Appendix Figure We provide more details on computing the metries for identifying the confusion type and categorize the confusions of all affected classes in Appendix D
Class-conditions distributions induced by DA. To aid our understanding of the class-specific effects of DA. it is helpful to reason about how parametrized class of DA transformations T.( changes the distributions of each class in the training data (z). We denote the augmented class distributions by T. (pk). In particular, if supports of the distributions Te (p()) and (p:)) for two classes and overlap, the model is trained to predict different labels i and on similar inputs corresponding to features from both classes l and which will lead to performance degradation. Some class distributions P& and pt are intrinsically almost coinciding or highly overlapping in the ImageNet dataset, while others have distinct supports, but in all cases the parameters of DA will control the overlap of the induced class distributions (px) and To (pt). and thus the biases of the model when making predictions on such classes
We use NLTX library 5 for WordNet and spaCy library 26 for embeddings similarity.

---

Higher semantic similarity
Lower semantic similarity
Ambiguous
Co-occurring
prodiction 'urples
prodction aagglooo"
andictors baubart
predction "surche"
Legles
Langlas
Higher Real overlap
H427 mnstionsiigh
grentation
Fine-grained
Semantically unrelated
prddriiec Toardee
prlktiies on "wasde"
Lower ReaL
overlap
Figure 3: Types of class confusions affected by data augmentation with varied semantic similarity and data distribution overlap. Each panel shows pair of confused classes which categorize into: ambiguous, co-occurring, fime-rrrnniime-griinne fine-grained and semantically unrelated, depending on the inherent class overlap and semantic similarity. For each confused class pair, the left subplot corresponds to the class & whose accuracy decreases with strong data augmentation (DA), e.g. "sunglass" on top left panel: the ratio of validation samples from that class which are classified correctly decreases with stronger DA, while the confusion rate with another class (c.g. class "sunglasses" on top left panel) increases. The right subplot shows the percent examples from class I that get classified as l: or against DA strength.
Intrinsically ambiguous or semantically (almost) identical classes. Prior works [e.g.
67 identified that some pairs of ImageNet classes are practically indistinguishable, e.g. "sunglasses" and "sunglass", "monitor" and "screen", "maillot" and "maillot, tank suit". These pairs of classes generally have higher semantic similarity and higher Real. labels overlap Cka- We observe that in many cases the accuracy on one class within the ambiguous pair degrades with stronger augmentations, while the accuracy on another class improves. The supports distributions of these class pairs pk and pr highly overlap or even coincide, but with varying 8 depending how the supports of (p(ps) and T (pt) overlap the model would be biased towards predicting one of the classes. Figure top left panel, show how the frequencies of most commonly predicted labels change pair classes "sunglass" and "sunglasses" as we vary the crop scale parameter. The ReaL labels for these classes overlap with Cuu 87%. and the majority of confusions are resolved after accounting for multi-label annotations. We note that for images from both classes the frequency "sunglasses" label increases with stronger DA while "sunglass" predictions have the opposite trend. Models trained on ImageNet often achieve a better-than, thhanranddo---e random-guesss gugsss accuracy when classifying between these classes due to overfitting to marginal statistical differences and idiosyncrasies of the data labeling pipelines. While DA strength controls model's bias towards predicting one or another plausible label, the models not effectively making mistakes when confusing such classes.
ambiguous
For the remaining categories described below, the class distributions become more overlapping when strong DA is applied during training, and data augmentation amplifies causes problematic misclassification.
Co-occurring or overlapping classes. There is a number of classes in ImageNet which correspond to semantically different objects which often appear together, e.g. "academic gown" and "mortar- board", "Windsor tie" and "suit", "assault rifle" and "military uniform", "seashore" and "sandbar". These pairs of classes have rather high overlap in Real labels depending on the spurious correlation strength, and their semantic similarity can vary, but generally it would be lower than for ambiguous classes. The class distributions of co-occurring classes inherently overlap, however, stronger DA may increase the overlap in class distribution supports. For example, with RRC we may augment the
7

---

sample such that only the spuriously co- co-occurring object, but not the main object, is left in the image, and the model would still be trained to predict the original label: e.g. we can crop just the mortarboard in an image labeled as "academic gown". It was previously shown that RRC can increase model's s reliance on spurious correlations 23 56 which can lead to meaningful mistakes, not explained by label ambiguity. In Tigure[3Jtop right panel, we show how DA strength impacts model's s bias towards predicting "sandbar" or "seashore" class (these classes co-occur with Cw = 64% and the majority of confusions are resolved by accounting for ReaL labels). We emphasize that unlike the ambiguous classes, the co-occuring classes cause meaningful mistakes on the test data, which are not always resolved by multi-label annotations. For example, the model will be biased to predict "academic gown" even when shown an image of just the mortarboard.
Fine-grained categories. There is a number of semantically related class pairs like "tobacco shop" and "barbershop". "frying pan" and "wok", "violin" and "cello", where objects appear in related contexts, share some visually similar features and generally represent fine-grained categories of similar object type. These classes have high semantic similarity and do not have a significant Real. label overlap (sometimes such examples are affected by mislabeling but such classes generally do not co-occur). The class distributions for such categories are close to each other or slightly overlapping, but strong DA pulls them closer, and T(p:) and T(ps) would be more overlapping due to e.g. RRC resulting in the crops of the visually similar features or shared contexts in the augmented images from different categories. In Figure Bl bottom left panel, we show how model's confusion rates change depending on RRC crop scale for fine-grained classes "frying pan" and "wok" (these classes rarely overlap with only Cut = 9%).
Semantically unrelated. In the rare but most problematic cases, the stronger DA will result in confusion of semantically unrelated classes. While they share similar low-level visual features, they are semantically different, their distributions pk and pt and ReaL labels do not overlap, and they get confused with one another specifically because of strong DA), for example, categories like "muzzle" and "sandal", "bath towel" and "pillow". Figure 3 bottom right panel shows how confusions between unrelated classes "bath towel" and "pillow" emerge with stronger DA.
In Appendix D we show a larger selection of example pairs from each confusion category. Among the confusions on the most significantly affected classes, approximately 53% are fine-grained, 21% are co-occurring, 16% are ambiguous and the remaining 10% are semantically unrelated. While the confusion of semantically unrelated categories is the most rare, it is potentially most concerning since it corresponds to more severe mistakes.
While in some cases we can intuitively explain why the model becomes biased to predict one of the two closely related classes with stronger DA (such as with "car" and "wheel" classes in Figure D. in other cases the bias arises due to statistical differences in the training data such as data imbalance. ImageNet train set is not perfectly balanced with 104 classes out of 1000 containing less training examples than the remaining classes. We observed that 21% of these less represented classes were among the ones most significantly affected in original or ReaL per-class accuracy. Since DA can push condiicooti distributions of related classes closer together, if one of such classes is less represented in training data it may be more likely to be impacted by stronger augmentation. In Appendix
C we show how average accuracy on the underrepresented classes changes depending on the data augmentation strength.
We observe that using other data augmentation transformations such as RandAugment 13 or colorjitter on ImageNet, mixup [73 on CIFAR-100 results in similar effects of exacerbated confusions of related categories (see Appendix G).. In particular, RandAugment and colorjitter often affect class pairs which are distinct in their color or texture, while mixup on CIFAR- may amplify confusions of classes within the same superclass.
6 Class-conditional data augmentation policy interventions
we showed that class-level performance degradation occurs with stronger augmentation because of the interactions between different classes. The models tend to consistently misclassify images of one class to another related, e.g. co-occurring, class. In this section, we use these insights to develop a class-conditional augmentation policy and improve the performance on the classes negatively affected by DA.
In Section 5]
8

---

Table 1: Class-conditional data augmentation policy informed by our insights improves per- formance on the negatively affected classes while maintaining high overall accuracy. Average accuracy of different augmentation policies on all ImageNet classes, negatively affected classes and remaining majority of classes.
Avg acc of 50 classes
Avg acc of 950 classes
Augmentation strategy
Avg acc
8%
(6.7920.03
53.93 10.20
77 77.9910.02
Standard DA
8 60%
74.65
59.11 +0.20
75.47
Class-cond. Balestriero et al.
[
76. 10.05
43.02 20.28
77.8510.04
m 10
76.70
54.99
77.84+0.03
Our class-cond. DA
m 30
76.70.o.0
55.48 10.23
77.82
m 50
76.68
56.34
77.7510.04
Balestriero et al. M identified that DA leads to degradation in accuracy for some classes and also showed that a naive class-conditional augmentation approach is not sufficient for removing these negative effects. In particular, using oracle knowledge of validation accuracies of models pretrained with different augmentation levels, they evaluated a DA strategy where augmentation is applied to all classes except the ones with degraded accuracy (i.e. classes with Aag 0) which are instead processed with Center Crop. Since this approach didn't 't recover the accuracy on the affected classes, they hypothesized that DA induces a general invariance or an implicit bias that still negatively affects classes that are not augmented when DA is applied to the majority of the training data.
We explore simple class-conditional augmentation strategy based on our insights regarding class confusions. By changing the augmentation strength for as few as 1% to 5% of classes, we observe substantial improvements on the classes negatively affected by the standard DA. We also provide al- ternative explanation for why the class-conditional augmentation approach from Balestriero et al. M was not effective. In particular, we found in many cases that as we train models with stronger augmentation, a class k negatively affected by DA consistently gets misclassified as related class / (see Section 55.. We can precisely describe these confusions terms of False Negative (FN) mistakes for class k (not recognizing an instance from class k) and False Positive (FP) mistakes for class ! (misclassifying an instance from another class as class l):
(N) Σ [[f(()) and FP) ZEX,
[[f(()) 4j. (#EX)N(*XX))
Σ
We argue that to address the degraded accuracy of class & also important to consider DA effect on class I. In Appendix E we show the number of class-level False Positive mistakes as a function of DA strength for the set of classes with the highest AFP- = (P((s = 8%) min, (s) (i.e. the classes for which the number of FP mistakes increased the most with standard augmentation). Note that these classes are often semantically related to and confused with the ones affected in accuracy, for example "stage" is confused with "guitar", "barbershop" with "tobacco shop".
We explore simple adaptation of DA policy informed by the following observations: (1) generally stronger DA is helpful for the majority of classes and leads to learning more diverse features, (2) substantially increased number of False Positive mistakes for a particular class likely indicates that its augmented data distribution overlaps with other classes and it might negatively affect accuracy of those related classes. Further, we discuss training the model from scratch using classccon augmentation policy, and in Appendix E we consider fine-tuning the model using classcconiiiional augmentation starting from a checkpoint pre-trained with the strongest augmentation strength.
Class-conditional augmentation policy. In general, for class that is not closely related to other classes, strong RRC augmentation should lead to learning diverse features correlated with the class label and improving accuracy. Thus, by default we set the strongest data augmentation value s 8% for the majority of classes, and change augmentation level for a small subset of classes. We change the augmentation strength for m classes for which False Positive mistakes grew the most with stronger DA (i.e. the classes with the highest A FP). However, completely removing augmentation from these classes would hurt their accuracy (or equivalently increase the number

---

of False Negative mistakes) so we need to balance the tradeoff between learning diverse features and avoiding class confusions. As a heuristic, we set augmentation strength for each class / to s" = arg min, (s)) + (s) corresponding to the minimum of the total number of class-related mistakes across augmentation levels.
We vary the number of classes for which we change augmentations in the range m E {100,0,, 30, 50}. In Table [ we show the results where the parameters of the augmentation policy are defined using original ImageNet labels and in Appendix E we use ReaL multi-label annotations. We compare this intervention the baseline model trained with the strongest augmentation s 8%, mild augmentation level s = 60% optimal for average accuracy on the affected set of classes, and the class-com conditional augmentation approach studied in Balestriero et al. []] where we remove augmentation from the negatively affected classes, We find existing approaches sacrifice accuracy on the subset of negatively affected classes for overall average accuracy or vice versa. For example, as we previously observed the default model trained with 8 = 8% achieves high average accuracy on the majority of classes but suboptimal accuracy on the 50 classes affected by strong augmentation. Optimal performance on these 50 classes is attained by the model trained with 60%, but the overall average accuracy significantly degrades. Removing augmentation from the negatively affected classes only exacerbates the effect and decreases the accuracy both on the affected set and on average. At the same time, tuning down augmentation level on 1 to 5% of classes with the highest FP mistake increase improves the accuracy the affected classes by 2.5% for m = 50, and taking into account the tradeoff between False Positive and False Negative mistakes helps to maintain high average accuracy overall and on majority of classes (the average accuracy decreased by 0.1% for = 50). These results support hypothesis and demonstrate how a simple intervention on a small number of classes informed by the appropriate metrics can substantially improve performance the negatively affected data.
In Appendix F we study the class-level accuracy degradation of the ViT-S [5] model trained on ImageNet with varied RRC augmentation strength. We observe that a similar set of classes is negatively affected accuracy with stronger augmentation and multi-label annotations resolve some cases of accuracy degradation. Similarly, we identify the same categories of class confusions. By conducting caas-cooniiional augmentation intervention and adapting augmentation strength for 10 classes with the highest increase in False Positive mistakes, improve the average accuracy on the degraded classes by over 3%: from 52.28% + 0.18% to 55.49% + 0.07%.
7 Discussion
In this work, we provide new insights into the class-level accuracy degradation on ImageNet when using standard data augmentation. We show that it is important to consider the interactions among class-conditional data distributions, and how data augmentation affects these interactions. We sys- tematically categorize the most significantly affected classes as inherently ambiguous, co-occurring, or involving fine-graine grained distinctions. These categories often suffer from label noise and thus the overall negative effect is significantly muted when evaluating performance with cleaner multi-labed annotations. However, we also identify non-trivial cases, where the negative effects of data augmenta- tion cannot be explained by label noise such as fine-grained categories. Finally, in contrast to prior attempts, we show that a simple class-conditional data augmentation policy based on our insights can significantly improve performance on the classes negatively affected by standard data augmentation.
Practical recommendations. When evaluating model performance, one should not only check average accuracy, which may conceal class-level learning dynamics. Instead, we recommend re- searchers also consider other metrics such as False Negative and False Positive mistakes to better detect which confusions data augmentation introduces or exacerbates. In particular, when training a model with strong augmentations, one should train another model with milder augmentation to check whether these finer-grained metrics degraded as an indicator that augmentation is biasing the model's 's predictions. We can then design targeted augmentation policies to improve performance on the groups negatively affected by standard augmentation.
10

---

Acknowledgements
We thank Yucen Lily Li, Sanae Lotfi, Pavel Izmailov, David Lopez-Paz and Badr Idrissi for useful discussions. Polina Kirichenko and Andrew Gordon Wilson were partially supported by NSF CAREER NS-2145492, NSF I-DISRE 193471, NIH R01DA048764-01A, NSF IIS- 1910266, NSF 1922658 NRT-HDR, Meta Core Data Science, Google AI Research, BigHat Biosciences, Capital One, and an Amazon Research Award.
References
[1] Balestriero, R., Bottou, L., and LeCun, Y. (2022a). The effects of regularization and data augmentation are class dependent. arXiv preprint arXiv:2204.0366
[2] Balestriero, R., Misra, I,, and LeCun, Y. (2022b). A data-augmentation. is worth thousand samples: Exact quantification from analytical augmented sample moments. arXiv preprint 2XXx:2202.08325
[3] Benton, G., Finzi, M., Izmailov, P., and Wilson, A. G. (2020). Learning invariances in neural networks from training data, Advances in neural information processing systems, 33:17605-
[4] Beyer, L., Hénaff, O. J. Kolesnikov, A., Zhai, X., and Oord, A. v. d. (2020). Are we done with imagenet? arXiv preprint ar arXiv:2006.00 07.59.
[5] Bird. S., Klein, E., and Loper. E. (2009), Natural language processing with Python: analyzing text with the natural language toolkit. O'Reelly Media, Inc.".
[6] Bitterwolf, J., Meinke, A., Boreiko, V., and Hein, M. (2022). Classifiers should do well even on their worst classes. In ICMI. 2022 Shift Happens Workshop.
[7] Blodgett, S. L., Green, L., and O' C'Cnnor, B. (2016). Demographic dialectal variation in social media: A case study of aatercan english. arXiv preprint arXiv:1608.066 1608.0868..
[8] Botev, A., Bauer, M., and De, (2022). Regularising for invariance data augmentation improves supervised learning. arXiv preprint arXiv.200303300.
[9] Bouchacourt, D., Ibrahim, M., and Morcos, (2021). Grounding inductive biases in natural images: invariance stems from variations in data. Advances in Neural Information Processing Systems, 34:19566- 19579.
[10] Buolamwini, and Gebru, (2018). Gender shades: Intersectional accuracy disparities commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91. PMLR.
[11] Cheung, T.-H. and Yeung, D.-Y. (2022). Adaaug: Learning class-and instance adaapivv data augmentation policies. In International Conference on Learning Representations.
[12] Cubuk, E. D.. Zoph, B., Mane, D., Vasudevan, V., and Le, Q V. (2018). Autoaugment: Learning augmentation policies from data. arXiv preprint 18Xx:1805,00500
[13] Cubuk, E. D.. Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition workshops, pages 702-703.
[14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020), image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11992
[15] Dosovitskiy, A., Beyer. L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR.
[16] Fellbaum, C. (1998). WordNet: An Electronic Lexical Database. The MIT Press.
[17] Fujii, S., Ishii, Y., Kozuka, K., Hirakawa, T., Yamashita, T., and Fujiyoshi, H. (2022). Data augmentation by selecting mixed classes considering distance between classes. arXiv preprint 220.00200
[18] Geiping. J., Goldblum, M., Somepalli, G., Shwartz-ZZi Ziv. R., Goldstein, T., and Wilson, A. G. (2022). How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. arXiv preprint Xxxv:2210.06444/

---

[19] Lopt,,oLopes, R., Smullin, S. J., Cubuk, E. D., and Dyer, E. (2020). Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.08973.
[20] Hataya, R., Zdenek, J., Yoshizoe, K., and Nakayama, H. (2020). Faster autoaugment: Learning augmenta- tion strategies using backpropagation. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pages 1-16. Springer.
[21] Hauberg, S., Freifeld, O., Larsen, A. B. L., Fisher, J., and Hansen, L. (2016). Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation. In Artificial intelligence and statistics, pages 342-350. PMLR.
[22] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In CVPR.
[23] Hermann, K.. Chen, T., and Kornblith, S. (2020). The origins and prevalence of texture bias in convolutional neural networks, Advances in Neural Information Processing Systems, 33:19000-1199 19015.
[24] Hernández-Garrfa, A. and König, P. (2018). Further advantages of data augmentation on convolutional neural networks. In Artificial Neural Networks and Machine Learning- ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part 27, pages 95-103. Springer.
[25] Ho, D., Liang, E., Chen, X., Stoica, I., and Abbeel, P. (2019). Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pages 12411. PMLR.
[26] Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. (2020). spacy: Industrial-strennth natural language processing in python.
[27] Hooker, S., Courville, A., Clark, G., Dauphin, Y., and Frome, A. (2019). What do compressed deep neural networks forget? arXiv preprint 1rXiv:19/1.05248.
[28] Hovy, D. and Søgaard, A. (2015). Tagging performance correlates with author age. Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference natural language processing (volume 2: Short pupers), pages 483-488.
[29] Hu, M. and Li, J. (2019). Exploring bias in gan based data augmentation for small samples. arXiv preprint WXXi:1905.08495
[30] Idrissi, B. Y., Bouchacourt, D., Balestricro, R., Evtimov, I., Hazirbas, C., Ballas, N., Vincent, P., Drozdzal, M., Lopez-Paz, D., and Ibrahim, M. (2022). Imagenet-x: Understanding model mistakes with factor of variation annotations. arXiv preprint arXiv:2211.01866.
[31] Izmailov, P., Kirichenko, P., Gruver, N., Wilson, G. (2022). On feature learning in the presence of spurious correlations. arXiv preprint arXiv:2210.1 11369.
[32] Kaplun, G., Ghosh, N., Garg, S., Barak, B., and Nakkiran, P. (2022). Deconstructing distributions: pointwise framework of learning. arXiv preprint 2XXv:2202.099311
[33] Kapoor, S., Maddox, W. J., Izmailov, P., and Wilson, A. G. (2022). On uncertainty, tempering, and data augmentation in bayesian classification. arXiv preprint 2203 16481.
[34] Leclere, G., Ilyas, A., Engstrom, L., Park. S. M., Salman, II, and Madry, A. (2022). FFCV: Accelerating training by removing data bottlenecks, https:///gthub. wwcttauub.oomm///aaa/aa/ http//fffffffffc/////ooocc commit XXXXXXX.
[35] Li, Y.. Hu, G., Wang. Y., Hospedales. T., Robertson, N. M., and Yang. Y. (2020). Dada: Differentiable automatic data augmentation. arXiv preprint arXiv:2003.03880.
[36] Lim, S., Kim, I., Kim, T., Kim, C., and Kim, S. (2019). Fast autoaugment. Advances Neural Information Processing Systems, 32.
[37] Lin, C.-H., Kaushik, C., Dyer, L., and Muthukumar, V. (2022). The good, the had and the ugly sides of data augmentation: An implicit spectral regularization perspective. arXiv preprint arXiv:2210.05002
[38] Liu, Z. Mao, H. Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A convnet for the 2020s. arXiv preprini 22X1v:2001.03555
[39] Luccioni, A. S. and Rolnick, D. (2022). Bugs in the data: How imagenet misrepresents biodiversity, arXiv preprint arXiv:2208.1166
12

---

[40] Mahan, S., Kvinge, H.. and Doster, T. (2021). Rotating spiders and reflecting dogs: class conditional approach learning data augmentation distributions. arXiv preprint urXiv:2106.0400,
[41] Miao, N., Mathicu, E., Dubois, Y., Rainforth, T., Tch, Y. W., Foster, A., and Kim, H. (2022). Learning instance-specific data augmentations. arXiv preprint arXi::2266.0000 000500.
[42] Miller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. (2021). Accuracy on the line: on the strong correlation between out-of out-of-distriben distribution and in-distribution generalization. In International Conference Machine Learning, pages 7721-7 7721-7735. PMLR.
[43] Müller, S. G. and Hutter, F. (2021). Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 774-782.
[44] Nilsback, M.-E. and Zisserman, A. (2008). Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pages 722-729.
[45] Northeutt, C., Jiang. L.., and Chuang, (2021a). Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373 10o1333-1411
[46] Northcutt, G., Athalye, A., and Mueller, (2021b), Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv: 14749.
[47] Paszke, A.. Gross. S., Chintala, S., Chanan, G., Yang, E., Vito, Z., Lin, Z., Desmaison, A., Antiga. L.. and Lerer. A. (2017). Automatic differentiation in pytorch.
[48] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, 7.,, Gimelshein, N., Antiga, I.., Desmaison, A., Kopf, A., Yang, E.,, DeVito, 7., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high- deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc.
[49] Raghunathan, A., Xie, S. M., Yang, F., Duchi, J., and Liang, P. (2020). Understanding and mitigating the tradcoff between robustness and accuracy. arXiv preprint 2002 10716.
[50] Ratner, A. J., Ehrenberg, H., Hussain, Z., Dunnmon, J., and Ré, (2017). Learning to compose domain- specific transformations for data augmentation. Advances in neural information processing systems, 30.
[51] Rey-Area, M., Guirado, E., Tabik, S., and Ruiz-Hidalgo, J. (2020). Fucitnet: Improving the generalization of deep learning networks by the fusion learned class-inherent transformations. Information Fusion, 63:188-195.
[52] Richards, M., Kirichenko. P.. Bouchacourt, D., and Ibrahim, M. (2023). Does progress on object recognition benchmarks improve real real-world generalization? arXiv preprint arXiv:2307. 13136.
[53] Russakovsky, O.,, Deng, J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision, 11:2211-222.
[54] Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. (2019). Distributionally robust neural net- works for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.087.
[55] Salman, H., Jain, S., Ilyas, A., Engstrom, L., Wong, E., and Madry, A. (2022). When does bias transfer in transfer learning? arXiv preprint arXiv:2207.028-2
[56] Shah, H., Park, S. M., Ilyas, A., and Madry, A. (2022). Modeldiff: A framework for comparing learning algorithms. arXiv preprint arXiv:2211.1249i.
[57] Shankar, V., Roclofs, R., Mania, H., Fang. A., Recht, B., and Schmidt, L. (2020). Evaluating machine accuracy imagenet. In International Conference Machine Learning, pages 8634-8644. PMLR.
[58] Steiner, A.. Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., and Beyer, L. (2021). How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2166.1 10..27..00....00..
[59] Stock, P. and Cisse, M. (2018). Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In Proceedings of the Europeon Conference Computer Vision (ECCV), pages 498-512.
[60] Szegedy, C., Vanhoucke, loffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages
13

---

[61] Tan, M. and Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 3105-6114 14. PMLR.
[62] Tang, Z., Peng, X., Li, T., Zhu, Y., and Metaxas, D. N. (2019). Adatransform: Adaptive data transformation.
In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 1998-3006.
[63] Tatman, R. (2017). Gender and dialect bias in youtube's automatic captions. In Proceedings of the first
ACL workshop on ethics in natural language processing, pages 53-59.
[64] Teney, D., Lin, Y., Oh, S. J., and Abbasnejad, E. (2022). Id and ood performance sometimes inversely correlated on real-world datasets. arXiv preprint arXiv:2209.0040 00613.
[65] Touvron, H., Cord, M., Douze, M., Massa, F, Sablayrolles, A., and Jégou, H. (2021). Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 11347-10357, PMLR.
[66] Touvron, H., Vedaldi, A., Douze, M., and Jégou, H. (2019). Fixing the train-test resolution discrepancy. Advances in neural information processing systems,
[67] Tsipras, D., Santurkar, S., Engstrom, I.., Tlyas, A., and Madry, (2020). From imagenet image classification: Contextualizing progress on benchmarks. In International Conference on Machine Learning. pages 9625-9635. PMLR.
[68] Van Hom, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., and Belongie, (2015). Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595-605 6044
[69] Vasudevan, V., Caine, B., Lontijo-Lpes, R., Pridovich-Keil, S., and Roelofs, R. (2022). When does dough become bagel? analyzing the remaining mistakes imagenet. arXiv preprint arXiv:2005.04556
[70] Xu. Y. Noy, A.. Lin, M. Qian, Q. Li. H., and Jin, R. (2020). Wemix: How to better utilize data augmentation. arXiv preprint 2rXiv:2010.01267.
[71] Yao, H., Wang, Y., Li, S., Zhang, L., Liang, W., Zou, J., and Finn, C. (2022). Improving robustness via selective augmentation. In International Conference Machine Learning, pages 25407-25437. PMLR.
[72] Yun, S., Oh, S. J., Heo. B., Han, D., Choe, J., and Chun, S. (2021). Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2340-22 2340-2350.
[73] Zhang, H., Cisse, M., Dauphin, N., and Lopez-Paz, D. (2017), mixup: Beyond empirical risk minimiza- tion. arXiv preprint arXiv:
[74] Zheng, Y., Zhang. Z., Yan, S., and Zhang, M. (2022). Deep autoaugment. arXiv preprint arXiv:2203.06172
[75] Zhou, F., Li. J.. Xie, C., Chen, F., Hong, L., Sun, R., and Li, Z. (2021). Metaaugment: Sample-aware data augmentation policy learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11097-11105.
14

---

Appendix Outline
This appendix organized as follows:
In Section A
we provide additional details on the training procedure and hyper-parameters used in our experiments.
Section B
we provide precise definitions of our evaluation metries.
Section we visualize the accuracy on the classes most negatively affected by data
augmentation.
We discuss the different types of class confusions caused by data augmentation Section D
we provide additional experiments for the proposed class-conditional data augmentation policy.
In Section
In Section we provide additional results with EfficientNet and Vision Transformer archi- tectures.
In Section G we confirm our results on additional data augmentation transformations and two addditional datasets.
We discuss broader impacts and limitations in Section
A Training details
Following [ we train ResNet-50 models for 88 epochs with SGD with momentum 0.9, using batch size 1024, weight decay 10-4, and label smoothing 0.1. We use cyclic learning rate schedule starting from the initial learning rate 10-4 with the peak value 1 after 2 epochs and linearly decaying to 0 until the end training. We use PyTorch [47), automatic mixed precision training with torch. amp package ffcv package [34] for fast data loading. We use image resolution 176 during training, and resolution 224 during evaluation, following Touvron et al. [66] and torchvision training recipe"] Balestriero et al. [ also use different image resolution at training and test time: ramping up resolution from 160 192 during training and evaluating models on images with resolution 256. We train 10 independent models with different random seeds for each augmentation strength E {8, 20, 30, 40, 50, 60, 70, 80, 90, 99%} where s 8% corresponds to the strongest and default. augmentation.
B Evaluation metrics
To understand the biases introduced or exacerbated by data augmentation, fine-grained metrics and evaluate them for models trained with different augmentation levels. We compute these metrics using original ImageNet validation labels and ReaL multi-label annotations []. We use f.(-) to denote neural network trained with augmentation parameter s, (ReaL (z) ReaL labels for validation example x, X set of all validation images, X₁ the validation examples with the original label k.
number of
Accuracy. The average accuracy for original and Real. labels is defined
or (s) 1/|X|
Σ [xf = k] and a ReaL 1/|X| x
Σ [[fa(x) (eeaL())), #EX
while for per-class accuracies ag (s) and a Reat(s) (s) the summation is over the set X₁ instead of all validation examples X. The accuracy on class with original labels ag (s) also correspond to recall of the model on that class.
Confusion. In Section 5 we looked class confusions, in particular, for pair of classes and / the confusion rate (CR) is defined as:
15

---

CRR-n(S) 1/|Xxa
[lf(()) 1]. 2EXk
Σ
ic. the ratio of examples from class & misclassified as We are only discussing confusions CR in the context of original labels.
False Positive and False Negative mistakes. In Section 6 emphasized the importance of looking at how data augmentation impacts not only per-class accuracy but also the number of False Positive (FP) mistakes for a particular class:
FP (=EX))(aEXx))
Σ
([(()))= =k) and P.R (8)=
Σ
T[,(2)) k)
Reat
for original and Real labels respectively. The number of False Negative mistakes on class k: in terms of the original labels are directly related to the accuracy, or recall, on that class:
FN (s)
E [lf(()) + |x (s)),, zEX;
while for multi-label annotations we define as:
FN (s)
E (arE (
[f)(()) & [ReaL())),
iee the number of examples x which were misclassidied by the model where was in the Real. label set Reat (zr). In Section 6 we explored s arg min (N+(s) + FN&(s) proxy for optimal class-conditional augmentation level which emphasizes the inherent tradeoff between class-level accuracy and the number of False Positive mistakes.
Affected classes. We are focusing on analyzing model's behavior on the classes which were negatively affected by strong (default) augmentation in terms of original or ReaL accuracy, i.e. classes where the accuracy drop Aa;- = (x((s{ ag (s = 8%) from ag (sf) )= = max, (s(s) to ax = 8%) is the highest. We focus 5% of classes (50 classes) with the highest Aa; following Balestriero et al. [I] and after interventions in Section [ In Section we also look classes where the number of FP mistakes increased the most with strong DA, i.e. with the highest AFP; = F& = 8%) - ming Ps()).
and measure the average accuracy on this set of classes as a function
C Accuracy of the classes most negatively affected by data augmentation
We show the per-class accuracies as function of data augmentation strength for (1) the 50 classes most negatively affected in original accuracy, i.c. with the highest Aag in Figure 4 and (2) 50 classes most negatively affected ReaL accuracy in Figure 5
16

---

2n 16 dul
18 day
2 12 3e m0
2, 11.33
Sterion
Windor Aar
tenter Aurt
Au u 11, Aarhen
Aut
Aert i 5, Jeltet
1
(8) Soa
tobacso shop
monastery
taled fox
Urrasee Apr Aufint
paralid bers Aud 5. Au
Av K
Auf"
Aut S.
(8)
malbay Sel - 7.36 fast
21ain
bulerpoot way
mallin du
(5) Lae
sombeen
tage plaer
redscee
lifhom
06) 1oc.eaaa
dasuclly
RPI
zdask
pul:
muzde fist
frying pxn
ray
09
hare Soca
backkand-tan thi
Aul Rost
Englsh Pest
safeny De the
der Nest
Walker
Ard Au
Solt
De Plok
BSC
SRC
JIVERAT ALETTODLAIO
original label
labels
Figure 4: Per-class class validation accuracies of ResNet-50 trained on ImageNet computed with original and ReaL labels as a function of Random Resized Crop data augmentation scale lower bound s. We show the accuracy trends for the classes with the highest difference between the maximum accuracy on that class across augmentation levels max, ag (s) and the accuracy of the model trained with 8%. On each subplot below the name of the class we show the accuracy drops with respect to original and Real labels: Angr and ReaL, We report the mean and standard emorr over 10 independent runs of the network.
17

---

Au in & day Sa 11
Sa Sa
Surt Ty delied
Sur Ruf a
vd (h)
Windor Ay 11 Aatlin
handhaed
beteppoo and Aathee
Sutt
Sallis
Aurth
Aur Authos on?
Aue u 1, Cafifin a
43 sec (N)
ub
darrudlly
mireature goodle Aut" Au
hair sids
Autl Au d
6, Na
Audf 5. An
Auel
SSE
122
rnd wolk
Rork-and-tnn roothourd C, de =
beance Sul Joffer
Dr Aur" &
Insuanee
Aul" =
Dur -
poreggneate
mailbag Pat
nipple
pars bors
Aur
San
book
hone theate
arcce leard
stepwachh
gronsesiaal
Yoridim
plange Pot
powe Sart
shopping Surtet
dighal foa
autcas Ju
Wiittoo 15)
Ser" Antre
Oet" Miat
toy de thet
toy Aar
Do Prss
est
Stafforhhice
Frying
Au Rost
3BC
original
Real
Figure 5: Per-class class validation accuracies of ResNet-50 trained on ImageNet computed with original and ReaL labels as a function of Random Resized Crop data augmentation scale lower bound s. We show the accuracy trends for the classes with the highest difference between the maximum ReaL accuracy on that class across augmentation levels max, a Re) and the ReaL accuracy of the model trained with 8%. On each subplot below the name of the class we show accuracy drops with respect original and Real labels: Dag and ReaL, We report the mean and standard error over 10 independent runs of the network.
18

---

D Class confusion types
In Table 2 show the classes most negatively affected in accuracy by strong data augmentation (column "A "Affected class k") and the confusions the model starts making more frequently with stronger augmentation ("Confused class (*)) particular, we study the union of 50 classes most affected in original accuracy and 50 classes most affected in ReaL accuracy (see Section C which do not belong the animal subtree in WordNet tree. We focus on the confusions / where confusion rate difference
ACR CRR-- 8%) min CR
the highest for class and above 2.5% (see Section B for definition of confusion rate (s)) Additionally for each pair of confused classes & and we also look at
A max CR (s) CR 8%)
which characterizes to what extent the model trained with weaker augmentation starts making the reverse confusion more often compared to the strong DA model.
To quantitatively estimate the confusion type for each pair of classes, we measure the intrinsic distribution overlap of the classes and their semantic similarity. We compute the overlap in Real labels for classes & and 1, which is the ratio of examples that have both labels k and I among the examples with the label A::
Cl
Σ Thk E Tnett (1)) T (neat(()))) ZEX
Σ zEX
[h: [x))
and intersection-over-uiion of the two classes:
IoUNi
-r'k xEX
rik RReaL (z)] Il e RRea Σ aEX
Ilk E LReaL (x) or E (ReaL
We use WordNet class similarity and similarity of word embeddings from spaCy [26] to measure semantic similarity. Note that these metrics only serve approximate measures of distribution overlap and semantic distance since (1) the ReaL labels still contain some amount of label noise and may contain mislabelled examples examples that are missing some of the plausible labels, (2) the WordNet distance is sometimes low for classes that are semantically very similar, and (3) spaCy doesn't 't have representation for all words and is underestimating the similarity of closely related concepts. However, all together these metrics can point towards one of the appropriate confusion type categories.
In Figure we show more examples the confusion rates for different pairs of classes and as function of data augmentation strength where k is among the ones most negatively affected in accuracy and is the class the model misclassified examples from the class k to. We show example pairs from different confusion types defined in Section
In Figure 7 we show average original and ReaL accuracy on the 104 classes which have less than 1300 examples in the ImageNet train split (while the remaining majority of classes have exactly 1300 examples train data). We hypothesize that if one of the confused classes has less examples in the train data and DA leads to higher class distribution overlap, the underrepresented class will likely be affected in accuracy by stronger augmentation.
E Additional details for the class-conditional augmentation intervention experiments
In Figures 3 and we show how the number of False Positive (FP) mistakes changes with data augmentation strength for the set of classes where FP number increased the most with strong DA (see Figure [8) for the set of classes where original FP mistakes increased the most and Figure 2 for ReaL FP mistakes). In Section 6 we conducted
condiionniii data augmentation interventions changing the DA strength for these sets of classes and showed that improved the accuracy on the classes negatively affected in accuracy. While in Section 6 we show results for adapting augmentation level for classes using original labels to evaluate False Positive and False Negative mistakes, Table 3 we show analogous results when using Real. labels which also shows that this targeted augmentation policy intervention for small number of classes leads to improvement in Real average accuracy on the affected classes (we specifically consider the set of classes affected in ReaL accuracy).
19

---

Ambiguous
Ambiguous
oned ciinnt on "roilet, tark uaie
predictions on "maillt"
predictions on "oower, CRT scree
predictoon on 'noonirr'
Agmoriatin stregh
seprotation strrgt
strugh
Co-occurring
Co-occurrig
predeciion scaderis gone
proiction "mareeberd"
profecions 'niltay unifsm"
stieen gin ertalbard
med
rilry calon
compa
unst ok
analeme mem gown nortubbed
aigrortatonn stnggh
segroraatinn
Fine-grained
Fine-grained
"sobecoo shop"
prdictira on "urbestop"
pesiett 'vennin"
Mesidiors "hoosker""
sugaetation aregit
supeartztion umeeth
saproctation ureegh
sagrentati aoughh
Fine-grained
Semantically unrelated
predictors Vraaee"
prliction "baraate!
prodiction 'bath toed
pod ciim 'sllw"
thakk Lavise
troba Lzwaw
beit tove
bei love
pilow
pll
zagrosttatio snarght
Figure 6: Confusion rate for classes most negatively affected by strong data augmentation and corresponding classes they get confused with. We categorize confusions into ambiguous, co- occurring, fine-graie fine-grained and unrelated.
Avg underrepresented ImageNet classes
original
ReaL
validation
RRC lower bound
RRC scale lower bound
Figure 7: Average accuracy of the 104 underrepresented ImageNet classes depending the Random Resized Crop augmentation strength. Average original accuracy on the underrepresented subset of ImageNet decreases by 0.6% with stronger augmentation.
20

---

Table 2: Confusions on the classes most affected by data augmentation.
Affected class
Confused class
A conf. rate (%)
Label co-occur.
Semantic sim.
Confusion
ACIIs
AC
Cu.
loU
WN
spacy
type
overskirt
hoopskirt
5.80
3.60
0.31
0.17
0.91
fine-gr. (ambig.)
honnet
4.20
0.00
0.03
0.02
0.73
0.32
fine-gr
gown
4.00
2.40
0.50
0.21
0.73
0.37
fine-gr. (umbig.)
trench coal
3.60
0.40
0.00
0.00
0.75
0.42
line-gr. gr.
acudemic gown
mottarbourd
18.40
7,00
0.72
0.50
0.73
0.10
c0-occur.
sunglass
sunglasses
13.00
22.40
0.87
0.81
0.64
0.84
ambig.
maillot
maillot
15.00
7.20
0.73
0.63
0.70
1.00
ambig.
Windsor tie
suit
7.20
4.00
0.61
0.32
0.82
0.24
occur.
screen
desktop computer
7.80
7,00
0.59
0.29
0.64
0.62
ambig.
monitor
3.20
6.40
0.87
0.37
0.63
0.44
ambig.
tobacco shop
barbershop
5.20
2.80
0.00
0.00
0.91
0.56
fine-gr.
bookshop
6.80
6.40
0.00
0.00
0.91
0.53
fine-gr gr.
monastery
church
2.80
6.80
0.11
0.03
0.70
0.71
fine-gr.
castle
2.80
11.20
0.00
0.00
0.60
0.69
fine-gr.
thresher
harvester
6.60
16.40
0.01
0.01
0.90
0.49
fine-gr.
parallel bars
horizontal bar
3.20
2,80
0.00
0.00
0.90
0.75
fine-gr.
bulance beam
3.00
1.00
0.02
0.01
0.90
0.15
fine-gr.
mailbag
purse
12.80
2.00
0.10
0.06
0.89
0.19
fine-gr.
backpack
1.00
3.60
0.00
0.00
0.89
0.16
fine-gr.
chain
necklace
9.40
4.40
0.15
0.09
0.53
0.31
umbig.
bulletproof vest
military uniform
5.60
3.40
0.31
0.13
0.76
0.38
co-occur. (ambig.) co-occur.
assault rifle
3.20
0.40
0.32
0.17
0.40
0.35
sombrero
cowbay hat
7.40
4.80
0.15
0.05
0.91
0.51
fine-gr.
velvet
purse
3.60
2.60
0.00
0.00
0.62
0.29
unrelated unrelated
necklace
3.00
0.00
0.00
0.00
0.62
0.51
tape player
radio
3.20
4.60
0.00
0.00
0.67
0.27
fine-gr.
cassette player
3.00
0.20
0.08
0.01
0.89
0.85
fine-gr.
assault rifle
military uniform
8.40
0.40
0.47
0.24
0.42
0.42
co-occur.
cornet
trombone
4.80
2.40
0.23
0.14
0.91
0.41
fine-gr.
pole
truffic light
4.00
0.40
0.05
0.03
0.12
0.21
unrelated
muzzle
sandal
3.20
0.00
0.00
0.00
0.56
0.23
unrelated
5.40
4.40
0.81
0.52
0.78
0.23
ambig.
vault
allar
6.40
4.40
0.21
0.12
0.62
0.41
fine-ge gr. (ambig.)
frying pun
Dutch oven
6.00
3,00
0.00
0.00
0.40
0.59
fine-gr.
wok
3.40
2.60
0.09
0.05
0.92
0.72
fine-gr.
French loaf
bakery
4.40
1.80
0.10
0.06
0.24
0.42
co-occur.
barrel
rain barrel
7.60
2.20
0.16
0.07
0.76
0.70
fine-gr. (ambig.)
spatula
wooden spoon
1410
2.80
0.21
0.12
0.57
0.62
fine-gr.
sax
flute
3.20
0.40
0.00
0.00
0.83
0.65
fine-gr.
seashore
sandhar
3.80
2.80
0.64
0.47
0.57
0.69
co-occur.
coffee mug
cup
7.80
0.80
0.61
0.34
0.19
0.63
ambig.
espresso
3.00
2.60
0.18
0.13
0.21
0.72
co-occur.
hreastplate
cuirass
6.00
6.40
0.71
0.50
0.67
0.48
ambig.
shield
3.20
1.20
0.07
0.05
0.70
0.59
beacon
breakwater
7.80
0.60
0.07
0.04
0.71
0.33
co-occur.
suit
miniskirt
3.20
1.60
0.02
0.01
0.86
0.32
fine-gr.
hand-held computer
collular telephone
8.80
3.60
0.22
0.06
0.50
0.42
ambig.
notebook
1.60
0.40
0.03
0.01
0.92
0.32
fine-gr.
stopwatch
digital watch
4.80
0.60
0.00
0.00
0.83
0.62
fine-gr.
strawberry
trifle
4.40
1.40
0.06
0.03
0.32
0.40
co-oocur.
trimaran
catamaran
4.80
1.40
0.18
0.09
0.92
0.60
fine gr.
digital clock
digital watch
3.00
7.00
0.02
0.01
0.83
0.71
fine-gr.
hair slide
necklace
5.60
0.60
0.00
0.00
0.50
0.42
fine-gr.
hook
nocklace
3.60
0.00
0.00
0.00
0.53
0.33
unrelated
backpack
purse
3.00
0.00
0.02
0.01
0.89
0.56
fine gr.
home theater
monitor
2.80
0.00
0.03
0.00
0.56
0.18
co-oocur.
hath towel
pillow
4.40
0.60
0.00
0.00
0.59
0.56
unrelated
21

---

SFP" S.APPpe 11
3FP 25, 21,FPphe 16
AFp = La AFpRid
AF pr 14 A.FFppal a.
AFpe APpiltt
reddare
larteohop AF ILAFFoo m
SF pr 3. 3
3F 11. phpe phut
3Fp 11 AFpitd
chanch A
rade: Arpe I1. A.poaa
makkit 3,52e
11.85 Wull
S. 11.
ArPp
10, A.PPoss
Irokondo 3PP R.
rudoar AFP I0 A.pitot
cails S. par E. plint
plte 3 Septes
5F
3. piant
Woakes
retebook APp J.A
Dunch ove A. 3.3P prplo
pllow SPP 5,38 prist parnt
Tautaa 3 5
APP A.p
outh
liptop
Day A. -.
Lookdhor
APPs
A votakes
entertabeeett cona S.Pppii
grocryy stoe
Piol
trkade 5.8.prp
pobo
zarden pallind
rm pep
APPs 7.3
3. pholt
phun
1E3
greas APp APpllis
degsack Agyl Aaplli
APP APplia
Appe Nul poos
streggr angrentation
original IN label
ReaL labels
Figure 8: The number of per-class False Positive (FP) mistakes for the set of classes where FP computed with original labels increases the most when using strong data augmentation. We show the trends using both original and ReaL labels.
22

---

LFP 22,5Ppte
SFP 25, Arpphei -1
SF Srplie 11
3, R.SF
AFP a. S.A.phi
SF 11,55 3. ]
AFpe - 1n, 3. phut
AFP = i1 aFpitd
3FP T.3Fptal
AF 14 Appllt =
home thater
Dusch
conboy Arpe 12. AFjila
nok
iPdd K.
APpl
Appl 9, S.Pplle
APPe
in. Arpostt
modicin deay APP 6, 6,5P
plloe Appe
enkads SPP 3.38 3.38
saath SPP - Apptest
APP A.ppplo
100kks
Ice bily APP 5.A 6
foueus 3P
Pida tpder 3F poit
botteb APp A.ppoo
APP A.p
AF
nbla trecc
Chucc
CECTM
pakap
pron Read
Caalle
coffent
slue shop plot
intsstuee pencher
careaa
ahoppna pappii
bookshop
APps Appee
phunt
3.
15-
giorey APp APpelis
swing -ppee poons
APP APples
Agyer plono
streggr angreotation
original IN label
ReaL labels
Figure 9: The number of per-class False Positive (FP) mistakes for the set of classes where FP computed with ReaL labels increases the when using strong data augmentation. We show the trends using both original and Real labels.
23

---

Table 3: Class-conditional augmentation intervention using ReaL labels.
# classes with adapted aug.
ReaL avg acc
ReaL avg acc of 50 aft. classes
ReaL avg acc of remaining 950 classes
m
83. 00.00
70.66+0.08
84.00=0.01
10
83.63
72.01
83.8640.01
772 30
83.64
72.28
83.86+0.01
50
83.57
72.20
83.7840.01
We also experimented with fine-tuning the model from the checkpoint trained with the strongest augmentation s 8% using either regular augmentation policy which was used during training or class-conditional policy with augmentation strength changed for k 10 classes as Section 6 we fine-tuned the model for 5 epochs with linearly decaying learning rate starting from the value 10 However, both regular and class-conditional DA lead slight drop in average accuracy on all classes (from 76.79% to 76. 73% for either DA) and in particular the accuracy dropped more significantly for negatively affected classes: from 53.93% to 53.4%. We hypothesize that this is due to model memorizing train examples even class conditiooa augmentation policy is not able to recover performance on the affected classes if we re-use the same data for fine-tuning. the future analysis, we will explore whether it is possible to alleviate DA bias if we fine-tune the model from earlier checkpoint as opposed to fully trained model or if we use additional held-out data for fine-tuning.
Additional architecture results: EfficientNet and VIT
In Figures I0 and we show the per-class accuracy trends for classes most affected in original and ReaL accuracy of EfficientNet-B0 [61] model, trained using a similar setup to the main ResNet-50 model (see Section A).. We can see that many affected classes the same for ResNet-50 and EfficientNet models.
We also train Vision Transformer model VIT-S [58 65 varying the RRC scale lower bound in the range E {10%, 20%.... 90%} and report the results Figure 12, Generally, we confirm that observations hold for VIT. While the optimal average accuracy obtained with the strongest augmentation, for several classes accuracy significantly degrades. Evaluation with multi-label annotations reveals that some of the confusions are due inherent label ambiguity class overlap. We also identify the same high-level class confusion categorized as ambiguous, co-occurring, fine- grained and semantically unrelated (see Fig 13). By conducting data augmentation intervention from Section of the paper and changing the RRC augmentation strength for just 10 classes, we improve the accuracy on the degraded classes by over 3% (from 52.28% + 0.18% 55.49% 0.07%).
24

---

greee nonla
bon spider
bock and tnd coothousd Sur
bakerpood west
Peper cowel
Baxc Ad
horred viger
sunglam
dumbbell
Winder 28
villtto acc (N)
ponarger or
Consett sbyer
oole
highorn Pot
sporib Autiic
aettons Ngtiia
imbela
vallaie oe (h)
Siocian
ErdeBacter
Sur Sa fas
handhhald Aa
2x"" 12 Ay
de Prat
Out Port
3u" 12, Au
S 12 So
Sul" Autla
Ar Il
Aut i1, Au
vilaiibo (h)
chocolate Su 14
wolhoun! 3x - 3e =
2x 2a
Sur" 11, de
E.
Jut 11
faom poweder Sxlf 10, de
acousic
bagnoes srake
Se El de 12
12
Sul" 33, de E.
Oofft 11, De
An 11, De 14
wddecton ax PS
bhoccaaas Sult 10, der
mude
whislle Ju
book 10, Dut
Lt
Burt 20,
10, Mi.
Aur"
BRC waak bound
3RC surk: love board
RRC male kere bound
BRC saak kwez bound
RPC scal lover loonal
original
Real.
Figure 10: Per-class class validation accuracies of EfficientNet-BB trained on ImageNet computed with original and ReaL labels function of Random Resized Crop data augmentation scale lower bound s. We show the accuracy trends for the classes with the highest difference between the maximum accuracy
that class across augmentation levels max, ag (s) and the accuracy of the model trained with s 8%% On each subplot below the name of the class we show the accuracy drops with respect to original and ReaL labels: Aagr and Aa Real.
25

---

greee nonla
bon spider
bock and ton coothousd Sur
bakerpood west
Peper cowel
Baxc Ad
horred viper
sunglam
dumbbell
Winder 28
villtto acc (N)
ponarger or
Conseta sbyer
oole
highorn Pot
sporib Autiic
aettons Ngtiia
imbela
vallaie oe (h)
Siocian Ral
ErdeBacter
Sur Sa fas
handhhald Aa
2x"" 12 Ay
de Prat
Out Port
3u" 12, Au
S 12 So
Sul" Autla
Ar Il
Aut i1, Au
villiibo (h)
chocolate Su 14
wolhoun! 3x - 3e w
2x 2a
Sur" 11, de
E.
Jet 11
v (M)
faom poweder Sxlf 10, de
acousic
bagnoes srake
Se El de 12
12
Sul" 23, de E.
Oofft 11, De
An 11, De 14
wddectoo ax. PS
bhoccaaas Sult 10, der
mude
whislle Ju
book 10, Dut
Lt
Burt 20,
10, Mi.
Aur"
BRC waak bound
3RC surk: love board
2RC male kere bound
BRC saak kwez bound
RPC scal lover loonal
original
Real.
Figure 1I: Per-class class validation accuracies of EfficientNet-BB trained on ImageNet computed with original and ReaL labels as function of Random Resized Crop data augmentation scale lower bound s. We show the accuracy trends for the classes with the highest difference between the maximum ReaL accuracy that class across augmentation levels max, ag ag (s) and the ReaL accuracy of the model trained with s = 8%. On each subplot below the name of the class we show the accuracy drops with respect to original and ReaL labels: Aagr and Reat.
26

---

Avg. all IN classes An
Avg 950
Avg 50 IN classes
An Rost
Ar
Ao Nook
Anc Antiest
validation aec (5)
70-
60
70-
50
66
Windsor tie
honeycamb
collie
Aa
10, Aaoo Raot
Aat" 11, An Roa.
10
An
11, An Rool
valiéation acc (%)
90-
40
10
000
CXI
20
63
30
501
40
RRC scale lower bound
RRC scale lower bound
RRC scale lower bound
stronger augmentation
original
Real
Figure 12: VIT-S trained on ImageNet with varied Random Resized Crop (RRC) augmentation strength. Average and per-class accuracy of ViT-S evaluated with original and ReaL labels as function of RRC augmentation strength. The top row shows the average accuracy all classes, the 50 classes with the highest accuracy degradation and the remaining 950 classes. The bottom row shows the accuracy of classes most significantly affected in accuracy when using strong augmentation.
Ambiguous
Co-occurring
pedirticse "urghow"
prddetions on 'ungbeer
pedetiin Wirder
pedrtioss
Setv aprppe
outur Miph
edor
Kidee
sengeen gles
set
at
D0 2013
Fine-grained
Semantically unrelated
"aae"
Face
saper
Figure 13: Class confusion types of VIT-S model trained on ImageNet with varied Random Resized Crop (RRC) augmentation strength. Each panel shows pair of confused classes which we categorize into: ambiguous, co-occurring, fine--raiin teni-gr and semantically unrelated. For each confused class pair, the left subplot corresponds to the class l affected in accuracy by strong data augmentation (DA), e.g. "sunglass" on top left panel: the ratio of validation samples from that class that get classified as i decreases with stronger DA, while the confusion rate with another class I (e.g. class "sunglasses" on top left panel) increases. The right subplot shows the percent examples from class / that get classified as & or / against DA strength. Generally, we observe that for VIT-S the class confusions which are exacerbated with stronger DA are similar to the ones ResNet-50.
27

---

G Additional augmentation transformations and datasets
G.1 Additional augmentation transformations
Ragmentt RandAugment [13] randomly applies color perturbations, translations and affine transformations. We train ViT-S with the RRC $ 10% and vary the RandAugnment magnitude m values above m 9 lead to significant
in the range (1,3....) .9] (m = 9 is standard for VIT [88 65 degradation in average accuracy). We report results Fig 14 While RandAngment strength has smaller effect on accuracy than RRC, we still observe an increase of around 0.5% in average performance with m 9. However, that comes at the cost of about 4% accuracy drop for a minority of classes. In Figure 14 we show examples of class confusions exacerbated by RandAugment, which we can categorize analogously to Section
Colorjitter We train a ResNet-50 using the strongest RRC augmentation with colorjitter applied with probability 0.5 and intensity c = 0.1 for all parameters (brightness, contrast, saturation and hue). Applying colorjitter with higher probability or intensity leads to degraded average accuracy while colorjitter leads slight improvement of 0.1% in average aceuracy. In Figure [5] we show the distribution of accuracy improvements and degradations due to colorjitter, well as the examples of class confusions that were exacerbated.
G.2 Additional augmentation transformations and datasets
CIFAR-100 + mixup We study mixup 73 augmentation for ResNet18 on CIFAR-100. We train for 100 epochs using Random Crop, and mixup with a = 0.5 improves the average accuracy from 78.11 0.15%% to 78.53 + 0.35% However, we observe degradations for some per-class accuracies (see Figure T6). The exacerbated confusions are mainly within the same superclass categories of CIFAR-100 which is aligned with prior results ImageNet where observed that fineggriined confusions are more significantly affected by augmentation.
Flowers102 We study the effect of applying standard RRC in Flowers102 classification task, and while using augmentation improves the average accuracy by 2%, we observe that some classes are negatively affected (see Figure 16).
H Broader impact and limitations
In our analysis we mainly focused on the setup of analyzing class-level accuracy drops of ResNet-50 model trained on ImageNet with varied Random Resized Crop (RRC) data augmentation. Sections and G we confirm our observations on additional architectures (EfficientNet-BO and ViT-S), data
VIT-S ImageNet
ped cas Tve
72.0
sug
2vg 950 dasses
71.
avg
validation sccuracy (%6
04
RandAugment magnitude
RandAugment. VIT-S model trained on ImageNet with varied RandAugment magnitude m (larger values of m correspond to stronger augmentation). The left panel shows the average accuracy of all ImageNet classes, the 50 classes with the highest accuracy degradation and the remaining 950 classes. The right panels show the examples of class confusions which are exacerbated by stronger Rand/Augment augmentation.
Figure 14: Different augmentation types:
28

---

Yountain pes val imps
140
Distrisution of per dass accuracy diff. from coleritter
- pel Saaap
and bibcer
100
eleette
"spider monkey" val imgs
48
% val exumlles
12
and wide vallay
and squiai waley
Accuracy xap
Different augmentation type: colorjitter. Comparison of ResNet-50 trained on ImageNet with Random Resized Crop s 8% when using and not using colorjitter augmentation (applied with probability changes when applying versus not applying colorjitrr: most classes benefit from augmentation, while significant number of classes is negatively affected. The right panels show the examples of class confusions exacerbated by applying colorjitter.
Figure 15:
0.5 and intensity 0.1). The histogram shows per-class accuracy
Datribut placei dil. ban moap
"oak tred"
inap
S. esrple
Dustiutes pe-dor scuro Fovmoo
178
19.
100
72
So gande
Figure 16:
Different datasets: CIFAR-100 and Flowers102. Left: The histogram shows per-
class accuracy changes when applying versus not applying mixup with a = 0.5 when training on CIFAR-100. Middle: The examples of class confusions exacerbated by applying mixup on CIFAR-100, the exacerbated confusions are mostly fine-grained and lie within CIFAR- 100 super- classes. Right: The histogram shows per-class accuracy changes when applying versus not applying standard Random Resized Crop 8% when training ResNet-32 on Flowers102 dataset.
augmentation transformations (RandAugment,, mixup and colorjitter) and datasets (CIFAR-100 and Flowers102). The same proposed framework can be extended to better understand the biases of other augmentations, architectures and satasets the future work. While we provide quantitative metrics to describe each confusion type affected by data augmentation, the categorization is not strict due to the remaining noise in Real. labels and imprecise word similarity metrics.
A potential negative outcome that can result from misinterpretation of our analysis in Section 4 if the practitioners assume that data augmentation does not have any negative effects since we discover that previously reported performance drops were overestimated due to label noise. We emphasize that while some of the class-level accuracy drops were indeed due label ambiguity or co- co-occurring objects, data augmentation does exacerbate model's bias and introduces class confusions (often between fine-graine -raiine categories but sometimes even for semantically unrelated classes that share visually similar features). We encourage researchers to carefully study the negative impact of DA using fine-grained metrics beyond average accuracy (such as per-class accuracy, False Positive mistakes and class confusions) to better understand its biases.
We estimate the total compute used in the process of working on this paper roughly 5000 GPU hours. The compute usage is dominated by training models for different augmentation strengths (Section 4. The experiments were run on GPU clusters on Nvidia Tesla V100, Titan RTX, RTX8000, 3080 and 1080Ti GPUs.
Compute.
29


Reviews:
[{"review_id": "AjVpvyXWHT", "reviewer": "Reviewer_2Ged", "summary": "The authors study the effect of data augmentations on the classwise performance under data imbalance. They focus on the rezised cropping operation and  distinguish between 4 different failure cases by using multi-label annotations. They also show that it is possible to recover some of them but using an informed class-conditional augmentation. ", "questions": "Improving the qualit of this paper would definitely require adressing the major weakness cited above", "limitations": "none", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The tackled problem is very relevent since it aims to understand the failure cases introduced but the standard data augmentation pipeline. \nThe authors provide decent empirical support to their claims and the paper is overall well written. ", "weaknesses": "* Although the introduced class conditional augmentation seems to help recovering some failures cases,  the overall performance of the model is not improving!  This unexpected behavior is worth more investigation\n\n* The last category of failures denoted 'semantically unrelated' remains unexplained \n\n* Minor: in line 267, 268: the authors claim that figure 3 shows the confusion between bath towel and pillow while it acctually shows the confusion between muzzle and sandal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback! We respond to your questions below, and we will remain available for any further discussion and clarifications throughout the discussion period.\n\n### Average performance improvement\n\nThe reviewer raises an interesting point regarding the average performance not improving upon adjusting the per-class DA strength. We emphasize that the classes on which our intervention changes the DA only account for a very small portion of the dataset (1-5%). As such, even in the ideal scenario in which we  significantly improve the accuracy on these classes, the impact on the average performance will be quite minimal. For example, a 10% accuracy improvement on 1% of the classes would only result in 0.1% improvement in average accuracy. Consequently, the goal of our intervention is to improve the performance specifically on the classes negatively affected by the default augmentation policy, and not the average performance. In the paper, we show that our intervention improves the performance on the target classes, while not significantly affecting the performance on the remaining classes, as expected.\n\nWe also note that interventions targeted at improving performance disparities often significantly *hurt* the average performance, which is not the case with our intervention. Moreover, we believe that our observations could inspire future work, which would target improving the average performance, by designing more elaborate augmentation policies.\n\n\n### Categories of class confusions\n\nOne of our key conclusion is that DA (e.g. RRC) reduces the performance of the model on a minority of the classes by introducing confusing (image, label) pairs which could be attributed to different classes. The prototypical example we employed was on how the RRC of a car picture might produce a wheel picture. As such car and wheel can be considered as ambiguous, or related, under the RRC DA. Therefore, by monitoring the True Positive, False Positive, False Negative and True Negative rates of those two classes, a relation can be drawn (this is our main observation also enabling our per-class DA intervention). \n\nFor the **semantically unrelated** confusion category, the classes often share similar visual features. For example, the classes muzzle and both show a net-like structure. Similarly, the classes flute and rifle share a long tube shape. Depending on the augmentation strength the model assigns objects with these features to one of the two conflicting classes.\n\n**Line 267, 268**: We thank the reviewer for their careful reading and for noticing that mistake, this will be corrected in our final submission.\n\nWe hope the above answers have alleviated your concerns, but if you have any further questions we are happy to discuss at any point throughout the discussion period."}, {"review_id": "v03TjJ4C2d", "reviewer": "Reviewer_nQVM", "summary": "The paper presents a meta study on the effects of data augmentation over classes. In particular, authors works on ResNet50 architectures trained on ImageNet, and show how for some classes, strong data augmentation drastically decreases the per-class accuracy. The paper focuses on random resized crop augmentations and specifically by varying the lowest possible crop size. Authors discuss these results in light of different types of overlap between classes and show how considering a different ground truth shows that the models are not always learning something wrong, but maybe learn to focus on object parts rather than whole objects. ", "questions": "- L33: as pointed out above, it would be interesting to see the effects on other datasets where classes are less and ideally only a subset of classes interact. Is there any literature on this that can be used here? \n- I would try to motivate why only the RRC is used. Be it simplicity, be it as an example of widely used augmentation, but I could not stop thinking in the paper how other DA strategies would behave. \n- I wonder if some dataset annotations where object parts annotations are available (eg https://github.com/TACJu/PartImageNet) could not be used to study in more detail the relationship between local crop and object parts\n- I missed at the end of Section 2 a summary of main limitations of current SOTA and to what this paper is answering specifically. \n- I would try to motivate (section 3 beginning) why only focusing on RRC, ImageNet and ResNet50. I agree on the choices, but is there something more generic we can draw out of this? What if I replace the imageNet with a DenseNet or use ResNet to classify COCO in multilabel settings? I missed some more general guidelines and motivation on the study. \n- L215-128 / L248-251: Would it be relevant to check if the model, in its top-K predictions, contains the classes that ReaL would show as most ambiguous? Independently of the object part relationship / co-occurrences / semantic similarity, a biased model should tend to predict those classes with high probabilities, maybe the predictions themselves and the co-occurrence of predictions could somehow show some of these effects. ", "limitations": "The paper does not discuss limitations of the study, but I think some points mentioned above could be discussed, at least, to strengthen motivations and underline that some of the effects observed could be observed in general. ", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Interesting study overall, which extends prior work on the same topic. The paper exposes some effects that might be obvious, but does so in a systematic and well constructed manner. \n- The paper is very well written, and well presented. It was easy to follow and to understand main points. \n- Supplementary materials contain interesting plots and additional results nicely complementing the paper. ", "weaknesses": "- Limited to one data augmentation strategy, it would be interesting to see how other potentially aggressive class-confusing augmentations effect the classification scores (eg. colour changes, contrast enhancements, affine transformations, etc.) interact with RRC and among them. I understand this would make such analysis considerably more complex, but in the end, those are commonly used and it rarely happens that one relies exclusively on RRC.  \n- The study is strongly limited by the use of ImageNet only, other datasets have multilabel annotations. It would be really beneficial to understand the effects on other datasets, where the total number of classes is lower and potentially the relatedness across classes also reduced. \n- Similarly, it would be interesting to understand if other CNN models or ViT would behave the same. I think it would nicely close the circle. I think such study on different models and different datasets would be quite interesting overall.\n- In conclusion, although well done and well presented, I think this paper does not make a substantial contribution in this current form, but could potentially do so if the breadth and scope are extended.  \n- I kept wondering what would happen if I train / fine tune the classifier in a multi-label setting, instead of just using ReaL as enhanced ground truth. It could be possible to do so using eg. coco? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! Please also see our separate general post, which contains new experiments inspired by your comments and clarification on the setup. Inspired by your feedback we have significantly increased the scope of our paper, and included results for new architectures, datasets and data augmentation types. We hope you can consider our responses in your final evaluation, and please let us know if we can answer any additional questions during the discussion period.\n\n### Limitations of previous studies and motivation and contributions of our work\n\nAlthough recent studies observed that data augmentation may lead to severe class-level accuracy drops [1, 2], no method has yet addressed this issue due to limited understanding on the origins of that performance degradation. To our knowledge, our work is the first systematic investigation narrowing down the root cause of class-level accuracy degradation when using DA. As a direct result of this finding, we propose a class-dependent DA intervention that takes into account the trade-off between class-level false negative and false positive mistakes. Unlike previous attempts to mitigate this issue [1], our data augmentation intervention solely changes the DA strength for a few classes, significantly improving the accuracy of the negatively affected classes, while retaining strong accuracy on average. \n\n### The choice of the dataset, models and augmentation type, and additional experiments\n\nThe focus on ImageNet stemmed from previous studies which reported class level accuracy drops primarily on that specific dataset[1, 2]. Inspired by your suggestion, we consider additional datasets to extend our insights beyond ImageNet, and include analysis on CIFAR-100 and Flowers102 in our general response. \n\nSimilarly, our focus on RRC stems from it being the most impactful DA in terms of final average accuracy increase. Most computer vision models use RRC with [8%, 100%] bounds, sometimes combined with additional DAs. Yet, it is also natural to ask if our conclusions extend beyond RRC which is why we have considered RandAugment,  colorjitter, and mixup in our general answer.\n\nBeyond the dataset and DA, the choice of architecture is also crucial. While we explored ResNet and EfficientNets, we acknowledge that those belong to the same convolutional family, and thus, as per your suggestion, we have added ViT-S in our general answer. \n\nIn all of these new experiments we found results to be consistent with our observations in the main paper. We hope that the addition of two datasets, one non-convolutional architecture, and three additional DAs help reinforce our methodology and conclusions.\n\n[1] Balestriero et al, The Effects of Regularization and Data Augmentation are Class Dependent, 2022\n\n[2] Bouchacourt et al, Grounding inductive biases in natural images:invariance stems from variations in data, 2021"}, {"review_id": "aoOFPOHCTV", "reviewer": "Reviewer_EjrR", "summary": "The authors explore the role of random resize crop in ImageNet performance. First, they improve on analysis in prior work and show that class-level performance degradation has been over-stated, and that when multi-label annotations are used one of the labels is often still predicted. Next, by inspection, the authors find that many  classes whose accuracies are affected random-resize crop are either completely ambiguous (sunglass vs sunglasses) or co-occurring (suit, tie; car, wheel). Finally, they suggest an intervention which takes false positives into account in addition to false negatives, dramatically improving accuracy on most-affected classes while preserving overall accuracy.", "questions": "See weaknesses. \n\n* Does the intervention have more impact on learned features, or on the final classification layer?  In other words, if the backbone was frozen after standard training and the final layer was retrained using the proposed intervention, what would the effect be? ", "limitations": "Authors have adequately addressed limitations. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: The idea of using multi-label annotations to judge the severity of the mistakes is sensible.\nThe idea of using False Positives for conditioning augmentation is an interesting one. \nIn general, the idea of using labels in the training procedure as a conditioning mechanism outside of the loss is interesting. \n\nQuality: The methodology seems sound and the claims are supported by the evidence. \n\nClarity :The paper is clearly written and the figures are well understood. \n\nSignificance: The paper is significant in that it sheds light on a process which is often used as a black box (data augmentation), and moves us towards more bespoke models. It is the case that we often report best top-1 accuracies, averaged over a validation set, and that this obscures the variation in different kinds of mistakes. However, certain kinds of mistakes are much more expensive than other kinds. Then, improving performance for certain classes while retaining performance of others is an important intervention for developing deployable models.  This work takes a step towards the understanding needed for making these kinds of trade offs. ", "weaknesses": "* The analysis is done only on a single model (supervised ResNet-50), on a single dataset (ImageNet). While I expect conclusions to be similar across other datasets/models, it would be interesting and important to confirm this. \n\n* Similarly, only consider RRC. What about other common augmentations? \n\n* ImageNet pretraining is frequently used for transfer learning. Does the class-conditional augmentation intervention impact transfer performance? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review! We appreciate that you find our work interesting, significant and clearly written.\n\n**Model, dataset, data augmentation**: We agree that confirming the applicability of our insights and methods to other models and datasets is crucial. To that end, we have provided supporting experiments which include a non-convolutional architecture (ViT-S), two datasets (CIFAR100, Flowers102), and three DAs (colorjitter, mixup and RandAugment). While we describe in detail the experiments and findings in the general answer, we mention here that our conclusions hold in those settings, and that the intervention we propose also transfers to those cases. We hope that those findings will provide further evidence for the validity of our findings in the broader context of computer vision.\n\n**Effects on learned features vs final layer**.  We have explored a similar experiment in the Appendix F of our submission: we experimented with fine-tuning the model from the checkpoint trained with the strongest augmentation using class-conditional policy with augmentation strength changed for 10 classes as in Section 6. However, we found that such fine-tuning leads to a drop in average accuracy and accuracy of the negatively affected classes. We hypothesize that this might be due to a feature extractor learning unwanted invariances or model memorizing the training data. In the future analysis, we will explore whether it is possible to alleviate DA bias if we fine-tune the model from an earlier checkpoint as opposed to fully trained model or if we use additional held-out data for fine-tuning.\n\n**Transfer learning**: This is a very interesting suggestion! We prioritized looking at different architectures, datasets and DAs for our rebuttal, but we agree that it is an important avenue that we hope to explore in the future.\n"}, {"review_id": "g4WQNzWRQz", "reviewer": "Reviewer_wB8d", "summary": "This paper studies Data augmentation (DA). Although DA improves overall accuracy, recent studies have pointed out that it can adversely affect individual class accuracy by up to 20% on ImageNet. This happens due to a lack of understanding of how DA impacts class-level learning dynamics. This research offers a framework to understand this interaction. By using high-quality multi-label annotations on ImageNet, it is found that most affected classes are inherently ambiguous, co-occurring, or involve fine-grained distinctions. Although multi-label annotations explain many previously reported performance drops, the analysis reveals other sources of accuracy degradation. It is demonstrated that class-conditional augmentation strategies, informed by this framework, can enhance the performance on classes negatively impacted by DA.\n\n\n--------Post-rebuttal------------\n\nThe rebuttal has fairly addressed my concerns. Accordingly, I have improved my score.", "questions": "It would be beneficial to extend the analysis to encompass a wider range of augmentation methods, such as the hyperparameters in RandAugment and Data MixUp. This could potentially ascertain whether the observations drawn from the study are sufficiently generalized across various data augmentation methods, thus enhancing the validity and universality of the findings.\n\nAre the observations consistent when applied to Vision Transformer (ViT)-based models? Although this study demonstrates the consistency of findings between ResNet and EfficientNet (both are Convolutional Neural Network (CNN) architectures), providing additional analysis with ViT-based models could significantly strengthen the overall solidity of the paper. This extended investigation would provide a more comprehensive understanding of how different model architectures interact with the described phenomenon.", "limitations": "Please refer to the weaknesses and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This study conducts an analysis of data augmentation bias, a process that bears significant practical implications for real-world applications. The meticulous examination of such bias provides insight into how it affects and shapes models in practical scenarios.\n\nThe research specifically investigates the impact of random crop scale on model training. It is revealed that the class-level performance drop experienced during ImageNet training can be mitigated through multi-label annotation. Intriguingly, the primary cause of this drop is found to be the co-occurring, ambiguous, and noisy conditions present within the class labels. This discovery is not only fascinating but potentially inspiring, opening up new avenues for further exploration in this domain.\n\nThe study introduces a novel class-conditional data augmentation strategy. This innovative approach shows promise in further alleviating data augmentation bias, thereby enhancing the robustness and performance of models in real-world deployments.", "weaknesses": "This work is limited by only one augmentation method (i.e., random crop ) and only CNN-based architectures (ResNet and EfficientNet).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their clear understanding of our submission and for noting the importance of our study! Please see our general response in which we detail our findings on new datasets, architectures and data augmentations (notably including the suggested ViT model). We observe that our insights, methodology, and DA intervention transfer to those different settings. As a brief summary, added experiments include one non-convolutional architecture (ViT, as per your suggestion), and three additional DAs (RandAugment, mixup, and colorjitter), and two new image datasets (CIFAR100 and Flowers102). We hope you can consider our responses in your final evaluation."}]

Author rebuttal:
We thank all reviewers for their feedback! We are happy that the reviewers found our paper is “systematic and well constructed” (nQVM), “bears significant practical implications” (wB8d) and “sheds light on a process which is often used as a black box” (EjrR). Data augmentation (DA) is essential in deep learning, yet it is poorly understood. Recent studies observed that DA may lead to severe class-level accuracy drops [1, 2]. To our knowledge, our submission is the first systematic investigation of these degradations, narrowing down their cause and partially alleviating them. In particular: 

(1) We correct prior work analysis on ImageNet using multi-label annotations: many previously reported cases of class-level performance degradation are explained by label ambiguity;

(2) We systematically categorize class confusions exacerbated by DA, using multi-label annotations and class similarities, revealing that the majority of these confusions concern ambiguous, spuriously correlated or fine-grained categories; 

(3) We propose a class-dependent DA intervention inspired by our findings. Unlike previous attempts [1], our intervention only changes the DA strength for a few classes, and significantly improves the accuracy on the affected classes. 

We believe that our work makes significant advances towards reliably deploying DA in the real world, where we must understand and remedy its detrimental effects to avoid costly mistakes. We have added multiple experiments inspired by reviewer feedback, and hope these results can be considered in the final assessment.

## Additional Experiments

The main feedback shared across reviewers was on the generality of our results beyond ImageNet, ResNet-50, and the Random Resized Crop (RRC). We have conducted several novel experiments during the rebuttal and significantly extended the scope of our work, as we describe below. We will add these results to our final submission.

### Vision Transformer

We train a Vision Transformer model ViT-S [3, 4] varying the RRC scale lower bound $s$ in the set {10%, 20%, …, 90%} and report the results in Figure 1 of the pdf. Generally, we confirm that our observations hold for ViT.

While the optimal average accuracy is obtained with the strongest augmentation, for several classes accuracy significantly degrades. Evaluation with multi-label annotations reveals that some of the confusions are due to inherent label ambiguity or class overlap (Figure 1 left). We also identify the same high-level class confusion categorized as ambiguous, co-occurring, fine-grained and semantically unrelated (Fig 1 right).

By conducting a data augmentation intervention from Section 6 of the paper and changing the RRC augmentation strength for just 10 classes, we improve the accuracy on the degraded classes by over 3% (from 52.28 $\pm$ 0.18% to 55.49 $\pm$ 0.07%).


### Additional data augmentation types

**RandAugment**: RandAugment [6] randomly applies color perturbations, translations and affine transformations. We train ViT-S with the RRC $s=10\%$ and vary the RandAugnment magnitude $m$ in the range {1, 3, … 9} ($m=9$ is standard for ViT [3, 4]; values above $m=9$ lead to significant degradation). We report results in Fig 2 left. While RandAugment strength has a smaller effect on accuracy than RRC, we still observe an increase of around 0.5% in average performance with $m=9$. However, that comes at the cost of about $4\%$ accuracy drop for a minority of classes. In Figure 2 we show examples of class confusions exacerbated by RandAugment, which we can categorize analogously to Section 5 of our paper.
 
**colorjitter**: We train a ResNet-50 using the strongest RRC augmentation with colorjitter applied with probability 0.5 and intensity $c=0.1$ for all parameters (brightness, contrast, saturation and hue). Applying colorjitter with higher probability or intensity leads to degraded average accuracy while colorjitter $c=0.1$ leads to a slight improvement of $0.1\%$ in average accuracy. In Figure 2 right we show the distribution of accuracy improvements and degradations due to colorjitter, as well as the examples of class confusions that were exacerbated.

### New datasets and augmentation

**CIFAR-100 + mixup**: We study mixup [7] augmentation for ResNet18 on CIFAR-100. We train for 100 epochs using Random Crop and mixup with $\alpha=0.5$ improves the average accuracy from $78.11 \pm 0.15\%$ to $78.53 \pm 0.35\%$. However, we observe degradations for some per-class accuracies (see Figure 3). The exacerbated confusions are mainly within the same superclass categories of CIFAR-100 which is aligned with our prior results on ImageNet where we observed that fine-grained confusions are more significantly affected by augmentation. 

**Flowers102**: We study the effect of applying standard RRC in Flowers102 classification task, and while using augmentation improves the average accuracy by $2\%$, we observe that some classes are negatively affected (see Figure 3).

## Motivation of the setup

While we significantly increased the scope of our analysis in the rebuttal, we would like to clarify the motivation of the setup in our submission. We focus on ImageNet as it is a widely used large-scale benchmark, and the class-level accuracy drops were previously noticed specifically on ImageNet [1, 2]. We chose to study Random Resized Crop as it is used in training most modern state-of-the-art computer vision models [e.g. 3, 4, 5], and it leads to the most significant improvements in terms of average accuracy, compared to more modest accuracy gains achieved with colorjitter, mixup, or RandAugment. 

### References 

[1] https://arxiv.org/abs/2204.03632

[2] https://arxiv.org/abs/2106.05121

[3] https://arxiv.org/abs/2106.10270

[4] https://arxiv.org/abs/2012.12877

[5] https://arxiv.org/abs/2201.03545

[6] https://arxiv.org/abs/1909.13719

[7] https://arxiv.org/abs/1710.09412

Meta-review: The submission studies the per-class impact of data augmentation on the ImageNet image classification benchmark. They focus on investigating CNN architectures and the random crop data augmentation. The authors identify some limitations of the prior studies on this topic (that the most affected classes are often inherently ambiguous, co-occurring, etc.), and demonstrate the effectiveness of class-conditional data augmentation strategies.

The submission received overall positive reviews, where the reviewers found the submission to study an interesting and significant problem,  the findings on the root causes of per-class performance degradation to be inspiring, and the paper to be well written. They also had concerns that the scope of experiments to be limited, studying only the CNN architecture with one type of data augmentation on one dataset. The authors provided a thorough rebuttal with additional studies on more model architecture (ViT), data augmentation types, and datasets. The rebuttal successfully addressed the concerns from three out of four reviewers, and they all recommended acceptance. Reviewer 2Ged did not respond to the rebuttal, and the AC has confirmed that the raised questions have been properly addressed by the rebuttal.

The AC thus recommends acceptance of the submission. The authors should make sure to incorporate the new results in the rebuttal into the final version.
Decision: Accept (poster)