What are effective labels for augmented data? Improving robustness with AutoLabel
Authors: Yao Qin
Xuezhi Wang
Balaji Lakshminarayanan
Ed Chi
Alex Beutel
Venue:  (2021)
Keywords: data augmentation
image classification
calibration
distributional shifts
adversarial robustness

Abstract: A wide breadth of research has devised data augmentation approaches that can improve both accuracy and generalization performance for neural networks. However, augmented data can end up being far from the clean data and what is the appropriate label is less clear. Despite this, most existing work simply reuses the original label from the clean data, and the choice of label accompanying the augmented data is relatively less explored. In this paper, we propose AutoLabel to automatically learn the labels for augmented data, based on the distance between the clean distribution and augmented distribution. AutoLabel is built on label smoothing and is guided by the calibration-performance over a hold-out validation set. We show that AutoLabel is a generic framework that can be easily applied to existing data augmentation methods, including AugMix, mixup, and adversarial training. Experiments on CIFAR-10, CIFAR-100 and ImageNet show that AutoLabel can improve models' accuracy and calibration performance, especially under distributional shift. Additionally, we demonstrate that AutoLabel can help adversarial training by bridging the gap between clean accuracy and adversarial robustness.

Under review as a conference paper at ICLR 2021
WHAT ARE EFFECTIVE LABELS FOR AUGMENTED DATA? IMPROVING ROBUSTNESS WITH AUTOLABEL
Anonymous authors Paper under double-blind review
ABSTRACT
A wide breadth of research has devised data augmentation approaches that can im- prove both accuracy and generalization performance for neural networks. However, augmented data can end up being far from the clean data and what is the appropriate label is less clear. Despite this, most existing work simply reuses the original label from the clean data, and the choice of label accompanying the augmented data is relatively less explored. In this paper, we propose Aut oLabel 1 to automatically learn the labels for augmented data, based on the distance between the clean distri- bution and augmented distribution. Aubel is built on label smoothing and is guided by the calibration performance over hold-out validation set. We show that Aut Laaeel is a generic framework that can be easily applied to existing data augmentation methods, including AugMix, mixup, and adversarial training. Experiments on CIFAR- 10, CIFAR-100 and ImageNet show that Aut AutoLabe, can improve models' accuracy and calibration performance, especially under distribu- tional shift. Additionally, we demonstrate that Aut oLabe 1 can help adversarial training by bridging the gap between clean accuracy and adversarial robustness.
celibration-performance
INTRODUCTION
Deep neural networks are increasingly being used in high-stakes applications such as healthcare and autonomous driving. For safe deployment, we only want models be accurate on expected test cases (independent and identically distributed samples), but we also want models to be robust to distribution shift (Amodei al., 2016) and to not be vulnerable to adversarial attacks (Goodfellow et al., 2014; Carlini & Wagner, 2017; Madry et al., 2017: Qin et al., 2020b). Recent work has shown that the accuracy of state-of-the-art models drops significantly when tested on corrupted data (Hendrycks & Dietterich, 2019). Furthermore, these models are not just wrong on these unexpected examples, but also overconfident Ovadia et al. (2019) showed that calibration of models degrades under shift. Calibration measures the gap between model's own estimate of correctness Ka.k.a. confidence) versus the empirical accuracy, which measures the actual probability of correctness. Building models that are accurate and robust, i.e. can be trusted under unexpeeted inputs from both distributional shift and adversarial attacks, is challenging but important research problem.
Improving both calibration under distribution shift and adversarial robustness has been the focus of numerous research directions. While there are many approaches to addressing these problems, one of the fundamental building blocks is data augmentation: generating synthetic examples, typically by modifying existing training examples, that provide additional training data outside the empirical training distribution. A wide breadth of literature has explored what are effective ways to modify training examples, such as making use of domain knowledge through babeerires transformations (Hendrycks al., 2020) or adding adversarially generated, imperceptible noise (Madry al., 2017; Zhang et al., 2019). Approaches like these have been shown to improve the robustness and calibration of overparametrized neural networks as they alleviate the issue of neural networks overfitting to spurious features that do not generalize beyond the i.i.d. test set.
In the broad amount of research on data augmentation, most of it attempts to apply transformations that do not change the true label such that the label of the original example can also be assumed to be the label of the transformed example, without expensive manual review. While there has been significant amount of work in how to construct such gpeud--xxappees in input space, there has been relatively little attention on whether this assumption of label -abee-rreservation holds in practice and what label should be assigned such augmented inputs. For instance, many popular methods assign

---

Under review as a conference paper at ICLR 2021
0.91
Clean image
0.03 0.03
0.03
2
Classifier
Classes
Originaal
(ii)
0.52
One hot Label 1
0.16 0.16
0.16
Classes
2 Classes
Augmented
Labels learned by AutoLabel
images
(iin) d 09 (kv)
(a) Pipeline
(b) Augmented images
varying distances
Figure 1: (a): An example showing Aubelabl assigning different labels augmented images (e.g., by Aug (Hendrycks et al., 2020)) based on their transformation distances to the clean image. The label for the true class is automatically learned based on the calibration performance on validation set. (b): Examples of images augmented by AugMix with different distances to the original image.
one-hot targets to both training data as well as augmented inputs that can be quite far away from the training data where even human raters may not be 100% sure of the label. This runs the risk of adding noise to the training process and degrading accuracy.
With this observation, in this paper we investigate the choice of target labels for augmented inputs and propose AutoLab abbe. 1, a method that automatically adapts the confidence assigned to augmented labels, assigning high confidence
inputs close the training data and lowering the confidence as we move farther away from the training data. Figure 1 (left) gives a high-level overview of our proposed Aut oLabel along with examples of augmented images of varying distances generating by AugMix (Hendrycks et al., 2020) on the right. Our key contributions are as follows:
We propose Autolabel different extents based on the distance between the augmented data and the clean data.
distance-based approach where the training labels smoothed to
We show that Autolabel is complementary to methods which focus on generating augmented inputs by combining it with popular methods for data augmentation, such as Aug (Hendrycks et al., 2020), mixup (Zhang et al., 2018) as well as adversarial training (Madry et al., 2017).
We show that AutoLabel significantly improves the calibration of models (and accuracy, al- though less dramatically) on both clean and corrupted data for CIFARIO. 10, CIFARI00 and ImageNet. In addition, Aut oLabel also helps bridge the gap between accuracy and adversarial robustness.
2 RELATED WORK
Data Augmentation. Recent work has shown that introducing additional training examples can further improve a model's accuracy and generalization (Devries & Taylor, 2017: Cubuk et al., 2018; Yun et al., 2019; Takahashi et al., 2019; Lopes et al., 2019; Zhong et al., 2020). For example, AugMix (Hendrycks et al., 2020) utilizes stochasticity and diverse augmentations, together with a consistency loss over the augmentations, achieve state-of-the-ard corruption robustness. Mixup (Zhang et al., 2018), on the other hand, trains a neural network over convex combinations of pairs of examples and shows improved generalization of neural networks. Furthermore, adversarial training (Goodfellow et al., 2014; Madry et al., 2017; Zhang et al., 2019) can also be thought as special data augmentation technique aiming for improving model's adversarial robustness. In this paper, we investigate the choice of the target labels for augmented inputs and show how to apply AutoLabel to these existing data augmentation techniques to further improve model's robustness.
Calibration and Uncertainty Estimates. A variety of methods have been developed for improving a model's calibration, e.g., post-hoc calibration by temperature scaling (Guo al., 2017) and multi- classs Dirichlet calibration (Kull et al., 2019). Model's predictive uncertainty can also be quantified using Bayesian neural networks and approximate Bayesian approaches, e.g., variational inference (Graves, 2011; Blundell al., 2015), MCMC sampling based stochastic gradients (Welling & Teh 2011), and dropout-basse based variational inference (Kingma et al., 2015; Gal & Ghahramani, 2016). In addition to calibration over in- dis-ribbutionn data, calibration can further degrade under unseen data shifts, where ensemble of deep neural networks (Takshminarryanaa et al., 2017) is shown to be most robust dataset shift.
recently, Ovadia et al. (2019) show that model
2

---

Under review as a conference paper at ICLR 2021
On the other hand, several data augmentation methods have also been shown to improve model's calibration under data shifts. For example, AugMix is shown to improve uncertainty measures on corrupted image classification benchmarks (Hendrycks et al., 2020). Thulasidasan et al. (2019) demonstrate that neural networks trained with mixup are significantly better calibrated under dataset shift, and are less prone to over-confident predictions on autrof--istribuutin data.
Label Smoothing. Label smoothing, initially proposed in Szegedy et al. (2016), is used to prevent a model from being too over-confident in its predictions, thus improving its generalization ability. It has been shown by Müller al. (2019); Thulasidasan et al. (2019) that label smoothing can also effectively improve the quality of a model's uncertainty estimates. Our work is most closely related to the adaptive label smoothing algorithm in (Qin et al., 2020c). Qin et al. (2020c) observe the connection between adversarial robustness and uncertainty, and propose an algorithm for adaptively updating the amount of label smoothing based on the adversarial vulnerability of clean data to improve model's 's calibration. In contrast, we propose to adaptively smooth the labels for augmented data based instead on the distance to the clean training data, and show can further improve a model's 's accuracy, calibration and adversarial robustness.
NOTATION AND CALIBRATION METRIC
Given clean dataset D = {(TT( Wi)] fim.... fim....... .m. we consider a classifier f(-) for a K- class classification problem, where y {l,..., ..., K.. The one-hot encoding of the label is denoted as ÿ E {0, 1 1 where the label for the true class Wk== = and D = 0 for others. Let (x) denote the predicted probability for the k-th class. We use fzx) == argmax, fie (x) represent the predicted class and c(x) == max f& (z) as model's 's confidence of the predicted class. In addition to the training data D, we also have clean validation set Dy drawn i. from the same distribution.
Notations:
Expected Calibration Error (ECE) We use Expected Calibration Error (ECE) to measure a model's calibration performance as (Guo et al., 2017: Ovadia et al., 2019). This metric measures how well aligned the average accuracy and the average predicted confidence are, after dividing the input data into R buckets: ECE Σx [Bel [B,) - conf (B,)l, where B, indexes the r-th confidence bucket and m denotes the data size, the accuracy and the confidence of B, are defined as acc (B)) [r.]] EieB, (x; = y&) and (B,) B. EiEB, c(x:).
m.
4
ALGORITHM
We now dive into our algorithm for more effectively setting labels for augmented data. We will first discuss the general Autolaabe algorithm and then discuss how it can be applied to three different data augmentation techniques.
4.1
AUTOLABEL
Given a data augmentation technique Aug that takes in example z and outputs a transformed version, Autolabe) is mainly composed of two components: (1) a measure of the transformation distance from the original input example, and (2) subroutine for updating labels of the augmented examples during training. Many data augmentation approaches have hyperparameters that reflect how large the transformation should be: we refer to this as the transformation distance s. As examples, which we will discuss in-depth below, this can take the form of the number of transformations in AugMix (Hendrycks et al., 2020) or the norm of the adversarial perturbation in Madry et al. (2017). Put together, we use Aug(I, s) to denote the generation of a transformed example.
Our key insight that the confidence in the labels associated with the augmented data likely depends on how significant the transformation is. To make use of this insight, we would like to assign different labels for the augmented training data based on their transformation distances. The labels can be updated according to the calibration of the model on the augmented validation data. To this end, must be able to split both the training data and validation data into buckets for measuring calibration. Here, we will discretize the transformation distance into N buckets {S,, ... SSw where each S, is a range, and we can generate augmented validation data for bucket S, by sampling a distance uniformly in that range ~ (S.)) to generate Aug(I, s).
Autelabel takes a label smoothing approach where the amount of smoothing depends on the calibration against a similarly augmented validation set Q(S.) {(Aug(a,, 8), ve) (xa, y.)
3

---

Under review as a conference paper at ICLR 2021
Table 1: Distance measure under various data augmentation methods used by Auto
Augmentation Method
AugM
mixup
Adv. Training
depth of augmentation chain d, mixing parameter
Distance determined by
mixing parameter 7
max &o norm
Distance buckets
[aN
( (min(x,1-33)
Y
Symaz
Dv.s ~ (.,)., This validation data @(S,) is used to learn a smoothed label (for the true class) (k=y(S,) that will be used for any training example transformed by distance E Sn. After each training epoch i, AutoLabel updates (Sx) according to:
= -S - a ECE' (Q(S.)) sign(Conf' (Q(S.)) - Acc
(I)
where (CS)))())) is the expected calibration error on the augmented validation set, and Acc()(S,)) and Conf (SS.) are the accuracy and confidence on the augmented validation set, respectively. The sign of (Conf( ()) - Acc( indicates if the model overall over-confident (> 0) or under-confident (< 0). Intuitively, if the model is overconfident on the validation set, we should reduce the label given to the true class J otherwise we should increase Øk=y. The expected cali- bration error on the augmented validation set ECE( Q) 2 0 suggests what extent we should adjust the labels as the optimal result ECH( Q) = O when the training converges. The hyperparameter a controls the step size of updating the labels. Since o stands for the probability of the true class, we clip the value be within [Acc' (Q), 1] after each update. Acc' () is used as the minimum clipping value to prevent * from being too small as (C) + k when the classifier is random guesser.
Given the updated label for the true class Ukty. the labels for other classes are uniformly distributed:
1
K-1
where K is the number of classes in the dataset and Σk k= us = 1. Finally, AutoLabel trains the model using jj(S,) as the target for the cross-entropy loss across the augmented data. For each example in the validation dataset Dy: we will sample a distance s in order to generate it's transformed version; we describe below this sampling distribution for each method.
AutoLabel can easily slot into existing data augmentation methods and further improve model's generalization and robustness. In the following sections, we demonstrate how to apply Aut LLabel to learn more appropriate labels over three representative data augmentation methods:
AugMix (Hendrycks et al., 2020): Mixing diverse simple augmentations in convex combinations.
mixup (Zhang et al., 2018): A linear interpolation of random clean images.
Adversarial Training (Madry et al., 2017): A special case of data augmentation to improve adversarial robustness via training on constructed adversarial examples.
Table shows an overview of how Autolabel differentiates the augmented data based the transformation distance under each data augmentation method. We present a complete pseudocode for AutoLabel in Algorithm in Appendix.
4.2 AUTOLABEL FOR AUGMIX
An overview of AugMix AugMix (Hendrycks et al., 2020) augments the input data via feeding the input x into augmentation chain' which consists of d E {1,2,3) augmentations randomly sampled from a set of diverse augmentation operations, e.g., color, sheer, translation. Then a convex combination is performed to mix the augmented image Taug with the original image x::
Augaugmir (z) = x + (1 (1-Xi A) Zang
(3)
where the mixing parameter À E [0, 1] is randomly sampled from uniform distribution. The model trained on the augmented data pairs (AAugaggmia (x), j)), where ý the one-hot encoding of the label associated with the original image.
'The original AugMix implementation (Hendrycks et al., 2020) uses 3 augmentation chains and then mixes the results from each of the augmentation chain via convex combinations. However, we consistently observe an accuracy increase when we use augmentation chain. Hence, we use AugMix with augmentation chain.
4

---

Under review as a conference paper at ICLR 2021
Transformation Distance In AugMix, the transformation distance is controlled by two parameters: (1) the depth of the augmentation chain d, which decides how many augmentation operations are applied to the original image; (2) the mixing parameter À in Equ (3), which controls the ratio of the augmented image Jaug and the original image x. In Figure (bb (ii) and (iv), we show the image that augmented by augmentation chain with depth d = 1 and d = 3 without mixing with the original image (A = 0). We can clearly see that a deeper augmentation chain causes the image to quickly degrade and drift off the data manifold. In addition, when comparing the augmented images with different mixing parameter X, shown in Figure 1(b) (iii) and (iv), we can see that as À - O. the augmented image further away from the clean image. As result, we can define the distance bucket Sa.n for the augmented data as: Sa,, = Sa.IANT: 2 Note here S is indexed by both d and À since the distance relies on both the depth of the augmentation chain d and the mixing parameter X, and in total we have N dmaa buckets, where dmaz denotes the maximum depth of the augmentation chain.
Update Labels To learn the labels for augmented training data that are within distance bucket Sd,n to the clean distribution, Aut oLabel constructs an augmented validation set @(Sa.) by feeding the validation images into augmentation chain with the depth d and then randomly sample a mixing parameter X' from a uniformly distribution: X' ~ u( 2+1 "N' to mix the original image
and the augmented image Eqn (3). Finally, outoLabll updates the labels ÿj ((),,.)) according to Eqn (1) & (2) and train the model using these updated labels.
4.3 AUTOLABEL FOR MIXUP
Overview of mixup Mixup, originally proposed by Zhang al. (2018), can consistently improve classification accuracy and further has been shown to be able to help with calibration in (Thulasidasan al., 2019). Specifically, the input data as well as their labels are augmented by
Au muguup (24, 2z) yr, + - -)
(4)
Augmirup (yt, Uj) = Tύs (1 7)66
where x, and vi are two randomly sampled input data and ÿ and by are their associated one-hot labels. The model is trained with the standard cross-entropy loss ((((mmi))u)) Mmmirup) and the mixing parameter y E [0, 1] that determines the mixing ratio is randomly sampled from a Beta distribution Beta(B, (3) at each training iteration. Rather than using the same mixing parameter y to combine the labels, we show how to apply Aut oLabel to automatically learn its labels Aug (Mss 8j) based on the validation calibration.
Transformation Distance The transformation distance in mixup is determined by the mixing parameter in Eqn (4). When 7 - 0.5, combining two images equally, the augmented image Augmizup (X', XT) is the most far away from the clean distribution. On the other hand, when 7 + O, the augmented image is close to the original image. Hence, the distance bucket S, for each augmented example Augmirupp22. xy)) can be defined as: S₂ = (12N-(min(7,,111)
Update Labels To Iearn the labels for the augmented training image within the distance bucket Sg. AutoLabel 1 constructs an augmented validation set @(S,) by randomly mixing two images from validation data with a mixing parameter y that is sampled from uniform distribution: y ~ u (22- 22
n+1) and 7 E [0, 0.5). Unlike AugMix, there are two classes y/s and y/ existing in the augmented image Augmizup (Ir, wj)) Due to > E 0.5). the class in the image Iy plays a dominant role in determining the main class in Augmirup (Zt, 2zy). Therefore, we follow Eqn (1) to update the label Ok=v, for the class k- = yy- Unlike Equ (2) that uniformly distributes the probability 1 yk==,
to all other classes, update the label for the class k y/s ÿk=y, min(1 ū 2 ūk=w,) and then distribute the probability Ōk=y. Üi-=y; to all other K 2 classes Finally, the model is trained by minimizing the cross-entropy loss with the new labels ÿ as the target.
4.4 AUTOLABEL FOR ADVERSARIAL TRAINING
Overview of Adversarial Training Adversarial training (Goodfellow al., 2014) can be formu- lated as solving the min-max problem: min [[ [max L(f(z + w), y)]. where denotes the adversarial perturbation, denotes the maximum to norm and one-hot encoding of the label y
In the special case where À 0, we merge it into bucket S to avoid creating an additional bucket which only contains data under this special case. Similar operation applied r in mixup and in adversarial training.
5

---

Under review as a conference paper at ICLR 2021
is used as the target for the cross-entropy loss L. In (Madry al., 2017), the inner maximization problem is approximately solved by generating projected gradient descent (PGD) attacks. Therefore, adversarial training can be considered as specific data augmentation that aims for improving model's adversarial robustness.
Transformation Distance First, AutoTabel differentiates the adversarial examples according to the distance between the adversarial examples and clean data. In adversarial training, the distance is approximately captured by the lo norm of the adversarial perturbation Unlike Madry et al. (2017) using a fixed construct PGD adversarial attacks during training, we randomly sample from uniform distribution ~ 2400, (maz) to construct PGD adversarial attacks with different distances to the original data. If the & norm of the adversarial perturbation is bounded by c, then for those constructed adversarial examples, we can determine their distance bucket Sn, as: S, Sie
Update Labels To learn the labels for adversarial examples within a distance bucket S, to the clean data, AutoLabel constructs adversarial examples for the validation set with the &o norm of the adversarial perturbation bounded by e', where e' is randomly sampled from a uniform distribution:
é ~ U(( P.Cmuxx (n+1))rmmms) ). Then can update the training labels (S,,) for adversarial examples
following Equ (1) and (2) and train the model using cross entropy loss on these new labels.
EXPERIMENTS
5.1 DATASETS AND EVALUATION METRICS
Datasets We perform experiments over three widely-used datasets: CIFARIO (Krizhevsky, 2009), CIFAR 100 (Krizhevsky, 2009), and ImageNet (Russakovsky et al., 2015). We use Wide ResNet-28- 10 v2 (Zagoruyko & Komodakis, 2016) for both CIFAR datasets, and ResNet-50 (He al., 2016) for ImageNet as our basic model architectures. In addition, we test our model's robustness over set of corrupted datasets (Hendrycks & Dietterich, 2019) with different types (17 types for CIFARIO and CIFAR 100 and 15 types for ImageNet) of corruptions that are frequently encountered in natural images. Each type of corruption has five levels of corruption severities.
Evaluation Metries We report the classification accuracy and expected calibration error on the clean datasets as well as the corrupted datasets, denoted as Acc (higher is better) and ECE (lower better) respectively. The accuracy and expected calibration error on the corrupted datasets are average
represented as cAec (higher is better) and cECE (lower is better), which are computed over all the corruption types across 5 corruption severities in the corrupted datasets. When evaluating adversarial robustness, et al., 2017). These attacks are generated by 50 iterations and random restarts bounded by e {0.02, 0.03, 0.04] with the image scale is [0, 1]. The step size is e/4.
report the accuracy of the model against white-box PGD attacks (Madry
5.2 AUTOLABEL IMPROVES THE CALIBRATION OF DATA AUGMENTATION TECHNIQUES
In this section, we apply Autolabel to three data augmentations: AugMix, mixup and adversarial training, and show that Autolabel 1 can help improve model's generalization and robustness.
Improve accuracy and calibration clean data We first investigate if AutoLabel can make data augmentation approaches more effective in improving accuracy and calibration on clean test data. For AugMix, we see in Table a clear picture: Aut Labell consistently helps AugMix improve both accuracy and calibration across CIFARIO, CIFARIO0 and ImageNet.
Table 2: Effects of Auto 1 for AugMix on CIFARIO, 10, CIFAR100 and ImageNet and the corresponding corrupted datasets. All numbers reported in the table is an average of 4 independent runs and in %. The arrow indicates better direction. Best results are highlighted in Bold.
Acc/cAce (t)
ECE/FECE (L)
Method
CIFAR10
CIFAR100
ImageNet
CIFARIO
CIFARI00
ImageNet
AugMix
96.9/87.7
80.6/63.9 9
76.3 /47.0
1.0/4.1
5.1/11.8
1.9/5.5
AutoLabel
96.9/88.2
81.6/65.0
76.7/47.6
0.9/2.7
1.8/4.3
1.4/4.3
To test how well AutoLabel improves mixup (Zhang et al., 2018) is more nuanced because mixup's bascline effectiveness is sensitive to its hyperparameters. In particular, the mixing parameter y in Eqn (4) is sampled from a beta distribution Beta (3, 3). When / 0, most sampled y are close to 0

---

Under review as a conference paper at ICLR 2021
CIFARIO Ca lloration
CIFARIOO Calibration
Image Vet Calibration
10
AugMix
AugMix
AugMix
AutoLabel
15
AutoLabe!
AutoLabel
ECE (%)
ECE (16)
20
4-
Corrupt on Severity
Corruption Saverity
Corruption Severity
Figure 2: Calibration across corruption severities three datasets. Severity 0 denotes clean data.
The improvment of Autolabel over AugMix is more significant as corruption severity increases.
Table 3: Effects of AutoLabel for mixup on CIFARIO and CIFARIO0. All numbers reported are averaged over 4 independent runs and in %. Best highlighted in bold.
100
CIFAR10 Adversaria Rebustness
CIFAR100 Adverso tal Rebust
Accuracy Pk)
Accurscy M9)
60
CCAT
8
Al AAstitubi
40
QCAT
CIFARIO
CIFARI00
Method
0.00
0.01
0.04
0.00
0.01
0.02
0.03
0.04
Acc
ECE
Acc
ECE
Vanilla
95.6
2.6
79.5
6.1
Figure 3: Effects of Autolabel for adversarial training on CIFARIO and CIFARIOO 100. Each model tested on the loo norm based white-box PGD attacks generated by 50 iterations and 3 random restarts. The axis is the maximum foo norm. O denotes the clean data.
mixup (3 = 0.2)
96.2
0.8
80.8
1.8
mixup (3 !)
96.5
5.3
80.9
5.5
AAutoLabel (3=1)
96.7
0.6
81.1
1.2
or I; when 8 = 1, 7 is randomly sampled from uniform distribution. We observe that mixup suffers from a trade-off between accuracy and calibration on the clean data on CIFAR 10 and CIFAR 100, shown in Table When the hyperparameter 3 is large, e.g., 3 diverse augmented data compared to smaller 3, e.g., B = 0.2. This results in a higher accuracy but leads the model to too under-confident and much larger calibration error on the clean data. This trade-off off between clean accuracy and calibration of mixup is also observed in other datasets and networks in Figure 2(j) in Thulasidasan et (2019). After applying AutoLabel to mixup to automatically adjust the labels for augmented images, we find that the trade-off between accuracy and calibration is well addressed: high accuracy and low calibration error are achieved on the clean data, shown in Table 3. However, this trade-off between accuracy and calibration does not exist in the large scale datasets, e.g., ImageNet, where = 0.2 consistently has the best accuracy and calibration and we did not observe significant improvement when applying AutoLabel 1 to mixup ImageNet.
the model trained on more
Improve robustness under distributional shifts While top-line improvements on accuracy are valuable, we'd e'd like to also understand how this method getting these improvements, and one key way is to see the benefits on corrupted data that the approach is more directly targeting. When applying Autolabel to AugMix, we again see a very clear picture in Table on three corrupted datasets, AutoLabel significantly improves both accuracy and calibration under distribution shifts. We similarly find benefits when applying AutoLabel to mixup: applying AutoLabel to mixup with 3 1, from 55.6% to 56.9% and reduce the mean calibration error from 11.2% to 10.0%. Similar results on CIFAR10-C are shown in Table 2 in Appendix.
further improve the mean accuracy CIFAR 100-C over mixup with / = 0.2
Digging into these results further, we find evidence of why Aut is working. In Figure 2 we analyze how calibration performance changes with the severity of the corruption being tested against, comparing AugMix and AugMix + Autolabel. We see that the baseline AugMix approach is increasingly badly calibrated the comuption increases, but Autolabel,, that trend significantly, and is thus increasingly important as the corruption size increases.
intended, dampens
Improve trade-off between adversarial robustness & accuracy Similar to human-designed data augmentation, we also believe adversarial training runs the risk of attacks not being truly label- preserving (Qin et al., 2020a) and thus should benefit from setting labels carefully. Therefore, we would like to see if Autolaber can improve the trade-off between adversarial robustness and accuracy. As a baseline, we perform adversarial training with PGD attacks as (Madry et al., 2017) on CIFAR10 10 and CIFAR100 datasets. Then we test the models against white-bon untargeted PGD attacks bounded with different &os norms. We compare adversarial training (AT) with and without AutoLabel 1 in Figure 3. There we that Autolabel effectively helps improve the trade-off
7

---

Under review as a conference paper at ICLR 2021
Table 4: Effects of AutoLabel for adversarial training under distributional shifts. All the numbers reported are average over independents runs and in %. The best result is highlighted in bold.
AccleAce(1)
ECE/&BCE(L)
Method
CIFARIO
CIFARI00
CIFAR10
CIFARI00
Vanilla AT
95.6/76.0
79.5/52.0
2.6/15.8
6.1/17.6
93.6/83,9
71,5/58.1
3.7/10.5
8,0/13.5
CCAT
93.2/68.9
74.8/49.8
2.4/9.9
7.9/16.1
AT' + Autolabel
94.6/83.6
75.3/60.2
2.0/6.5
4.2/6.9
between adversarial robustness against PGD adversarial attacks and accuracy, achieving higher accuracy on both clean and different levels of adversarial attacks.
We also compare our method with the recent work confidence-calibratte adversarial training (CCAT) (Stutz et al., 2020), which smooths the labels for adversarial examples according to ÿ = (() + (1 9(8) K where the balancing parameter g follows "power transition": g(0) == (1 min(1, [ )),, p is set to 10 as (Stutz et al., 2020) to ensure uniform distribution is used as the labels for adversarial examples when || 2 c. CCAT is shown to help improve adversarial aceuracy after confidence-
confidence-thresooliin thresholding. However, using the uniform distribution for adversarial examples results in a loss of class information and leads to a poor adversarial robustness detection mechanism, shown in Figure
without any coffieenc-thhreholddee
Improve adversarial training to be beneficial calibration We further validate if adversarial training can improve model's ' robustness under distribution shifts. Similar to recent work (Xie et al., 2020), we observe that training models with adversarial examples bounded with smaller lo norm, e.g.. ||0||> I 0.01, can benefit more to the corrupted accuracy with small accuracy sacrifice on the clean data. Therefore, we train all models with PGD attacks bounded by ||o]|co S 0.01 updated with 10 iterations. In Table 4, present the results of four models: a vanilla model without adversarial training, adversarial training (AT) (Madry et al., 2017), confidence-calibratee adversarial training (CCAT) (Stutz et al., 2020) and adversarial training the accuracy on the clean and corrupted datasets, Aut Aut Laabel consistently has a better balance of classification accuracy between clean and corrupted data. In addition, Aut oLabel 1 has significant improvement over model's calibration performance with a much smaller calibration error on both clean and corrupted data compared vanilla model, AT and CCAT. We explains this as result of that Aut oLabel updates training labels based on the validation calibration performance. In constrast, a pre-defined function to smooth the labels, e.g. the power transition used in CCAT, does not significantly help model's 's calibration.
When comparing
An extensive ablation study comparing AutoLabel with label smoothing (Szegedy et al., 2016; Müller et al., 2019) as well as temperature scaling (Guo et al., 2017) are included in Section B in Appendix, where we find that AutoLabel consistently has better improvement over model's calibration on both clean and corrupted datasets. Note that temperature scaling, as a post-hoc calibration method, does not help improve model's accuracy, where outela significantly helps.
Taken together, we believe this is strong evidence that oubel can improve the effectiveness of data augmentation techniques to improve models' robustness under distribution shifts, especially calibration performance.
6 CONCLUSION
In this paper, we propose Aut olabel to automatically adapt labels for augmented data, and show it is beneficial both model accuracy and calibration, compared reusing one-hot labels as in common in existing data augmentation works. We found that the distance between the augmented data and the clean data key to how we should adapt the labels and to what extent, and we propose calibration- -ouiiattiogguieedd approach to further adjust the labels automatically. We demonstrate the effectiveness of Aut Labbel under three representative data augmentation methods: AugMix, mixup, and adversarial training, We see that Auto greatly improves the calibration over clean data and corrupted data, and also helps adversarial training with a better trade-off between clean accuracy and adversarial robustness. More generally, believe that more nuanced approaches to setting labels for augmented data, beyond assuming label- pheseerien transformations, will lead to more effective data augmentation techniques, and look forward to further work in this space.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. arXiv preprint met.:66666.6655.. 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ICML, volume 37, pp. 1613-1622, Lille, France, 07-09 Jul 2015. PMLR.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEF, 2017.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc Le. AutoAugment: Learning augmentation policies from data. CVPR, 2018.
Terrance Devries and Graham Taylor. Improved regularization convolutional neural networks with CutOut. arXiv preprint arXiv: 04552, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model
uncertainty in deep learning. In International Conference Machine Learning, pp. 1050-1059, 2016.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2014.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems 24, pp. 2348 22366, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustmess and uncertainty. In International Conference on Learning Representations, 2020.
Durk P Kingma, Tim Salimans, and Max Welling, Variational dropout and the local reparameterization trick. In Advances Neural Information Processing Systems, pp. 2575-2583, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems 32, pp. 12316-12326. 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp- 6405 60166 2017.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Dogus Cubuk. Improv- ing robustness without sacrificing accuracy with patch Gaussian augmentation. arXiv preprint /rXiv:1966.026/1 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2017.

---

Under review as a conference paper at ICLR 2021
Rafael Müller, Simon Korblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems. pp. 4694 4703, 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 11991-14002, 2019.
Yao Qin, Nicholas Frosst, Colin Raffel, Garrison Cottrell and Geoffrey Hinton. Deflecting adversarial attacks, arXiv preprint 1xXiv:2002.074005 2020a.
Yao Qin, Nicholas Frosst, Sara Sabour, Colin Raffel, Garrison Cottrell, and Geoffrey Hinton. Detect-
ing and diagnosing adversarial images with clss--conditional capsule reconstructions. International Conference Learning Representations, 2020b.
Yao Qin, Xuezhi Wang, Alex Beutel, and Ed H. Chi. Improving uncertainty estimates through the relationship with adversarial robustness. arXiv preprint arXiv:2006.1633 2020c.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115: 211-252, 2015.
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-calirraaed adversarial training: General- izing unseen attacks. In Proc. of the International Conference Machine Learning (ICML), 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, 2016.
Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. Data augmentation using random image cropping and patching for deep cnns. IEEE Transactions on Circuits and Systems for Video Technology, PP:1-1, 08 2019. doi: 10.1 109/TCSVV:201929955
Sunil Thulasidasan, Gopinath Chennupati, Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Ellen Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems, 2019.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on International Conference Machine Learning, ICML'IL 11, 11, pp. 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 978 0781450306195
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoe V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 819-828, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, 112111100011100 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on International Conference on Machine Learning, 2019.
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In International Conference on Learning Representation, 2018.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In AAAI, 2020.
10


Reviews:
[{"review_id": "zzHjuRJquD8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to adaptively smooth the augmentation data labels based on the augmentation strength. The motivation is that the label-preserving assumption may not hold for data with substantial augmentations. The labeling smoothing uses the calibration performance on the validation data. If the model is overconfident on the validation data, the method will enforce more considerable smoothing. The paper applies it to three popular data augmentation methods: AugMix, Mixup, and adversarial training. Experiments on CIFAR-10, CIFAR-100, and ImageNet show the improvements for both accuracy and calibration.\n\nPros\n1. Automatically tuning the labels for augmented data is interesting, as most previous works mainly focus on new augmentation methods.\n2. The proposed method is general, applicable to different types of augmentations: AugMix, Mixup, and adversarial training.\n3. Experiments demonstrate its effectiveness on benchmark image classification datasets measured by accuracy and calibration.\n\nCons\n1. Although divided into N buckets, estimating the confidence and accuracy for a transformation bucket may still need several samples. The paper does not discuss this critical factor and provides an ablation study on it.\n2. It needs more justification to use validation data's calibration for label smoothing. What's its advantage compared with other tools?\n3. Why does it require 2N divisions for the Mixup design? I don't see a clear link between 2 samples in Mixup and using 2N here. \n4. More explanation on the equation to update the label y_{k=y_i} in the Mixup part is necessary. \n5. According to the experiments, the accuracy improvements are small. Calibration errors drop clearly because the validation calibration information is used in training. I don't see the description of the train-validation split. \n6. Ablation of some key hyper-parameters are missing, e.g., the bucket number N. The experiment section seems not to mention the chosen value of N.\n7. The label smoothing rule, i.e., Equation 1, is similar to previous work 1, which smoothes labels for adversarial examples. Technically, this paper mainly applies the idea to more augmentation settings.\n\nSummary\n\nChallenging the label-preserving assumption differentiates the proposed method from a majority of works in the data augmentation area. The proposed method is general to adjust labels for different data augmentation methods. My concerns lie in the technical novelty and lack of clarifications on the method design and experimental setup. Also, the empirical performance on accuracy does not show apparent improvements. More explanations on this may be better. Overall, I currently hold my score as marginally below the acceptance threshold.\n \n\nReference\n\n[1]. Improving uncertainty estimates through the relationship with adversarial robustness. Arxiv 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ff9LwmT5LQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes to use label smoothing to determine the labels of augmented samples. The distance between the original data distribution and the augmented data distribution is used to determine the amount of smoothing in the smoothed label. This technique can be used in combination with several data augmentation techniques e.g. AugMix, Mixup, and adversarial training. \n\nStrength:\n- This paper is structured clearly and easy to follow.\n- The idea is straightforward and the proposed distance measures for the three data augmentation methods are straightforward.\n\nWeakness:\n- The smoothed label in equation (1) and (2) of each augmented sample is based on an augmented validation set $Q(S_{n})$ which however directly uses the validations samples' original labels. Is this a contradiction to the main motivation of using a smoothed label for augmented data?\n\n- Using smoothed label for augmented data is not quite novel,  and has already being used in combination with adversarial training in [1]. Can the authors please clarify the connection and difference with this paper?\n\n- The accuracy results of corrupted data (adversarial samples) in Table 4 are not very convincing. The  $l_{\\infty}$ perturbations are upper bounded by only 0.01, while the commonly used perturbation bound is 8/255. Thus the attack in this paper may not strong enough, thus making the robustness conclusion not convincing. Can the authors show results using a larger commonly used perturbation bound?\n\n-  We can avoid accessing the label of adversarial samples for adversarial corrupted data. For example, TRADES[2] substitutes the original label with the model's output on clean data when training with corrupted data. Hence the proposed method may not be very useful for adversarial training. \n\nReference:\n- [1]. Cheng, Minhao, et al. \"Cat: Customized adversarial training for improved robustness.\" arXiv preprint arXiv:2002.06789, 2020.\n\n- [2]. Zhang, Hongyang, et al. \"Theoretically Principled Trade-off between Robustness and Accuracy.\" ICML. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z9AWnQmcgs5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a new method, called AutoLabel, to assign labels to synthetic samples. The label assignment is calculated by the distance between the clean data and the augmented data, and updated by performing a label smoothing based on the calibration performance over a hold-out-validation data set. Experiments show performance improvement over vanilla baseline model. \n\nThe paper is well motivated and the problem is important to the community. But the current form of the paper is weak due to the insufficient experimental studies and the minor performance improvement obtained with additional computational cost, compared to the baseline models. I have the following major comments.\n\n1.\tThe proposed label assignment is closely related to label smoothing as emphasized in the related work section and the main text (“AutoLabel takes a label smoothing approach where the amount of smoothing …”). I think it would be useful to use label smoothing as a main comparison baseline in the experiments. I saw some experiments with Cifar10 and Cifar100 in the Supplementary Material, but I think they should be moved to the main text.  Also, comparison with the label smoothing on Cifar10 show very similar results (but the label smoothing is much easier to be implemented), so it would useful to also provide the comparison result on ImageNet. Also, I think the idea is a bit similar to the Structural label smoothing technique (Reguralization via structural label smoothing, Li et al., 2020), where the data is first clustered and then the approach assigns different smoothing mass to different clusters formed. I think this may also be a valid comparison baseline as well. \n\n2.\tOne main disadvantage of the proposed method is that it seems to be sensitive to the distance function chosen for the label assignment and this function has to be adjusted for different data augmentation schema. The current justification for the three data augmentation schemas namely AugMix, MixUp, and Adversarial Training is weak to me. For example, in Mixup, the random mixing ratio may not reflect the salient features of the mixed image and leverage this ratio for the distance function for label assignment could be misleading sometimes (see “Mixup as locally linear out-of-manifold regularization”, AAAI2019, for more detail). In this sense, in the experiments, it would be very useful to provide further analysis and discussion of these distance functions in terms of how they impact the model’s performance. Also, I wonder if it makes sense to leverage the saliency in the mixed samples to compute the distance function. Leveraging salient features have been shown to be promising by recent research such as in the Puzzle Mix method (ICML 2020).\n\n3.\tThe way the paper generates the mixed modeling target for the mixed input in Mixup is related to that proposed in this paper: nonlinear mixup: out-of-manifold data augmentation for text classification,  AAAI2020. In the nonlinear Mixup method, the mixed label for a mixed input in Mixup is assigned by computing a transformation based on the inputs. It would be beneficial to briefly discuss the difference of the two methods for label assignment in Mixup. \n\n4.\tThe experiments are a bit weak to me. I think experiments with some additional network architectures, such as ResNet-18 in Table 2, would improve the paper.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OD3TfSFUgA3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review:\n\n[Summary] In this work, authors address the problem of how to achieve a more feasible data augmentation not only from the samples but also considering the accompanying labels. Authors solution is framed on a framework called ‘AutoLabel’, in which they leverage on already existing concept including: mainly on expected calibration error, and AugMix (in general mixup principles).\n\n[Cons]\n-- The problem of adjusting labels for augmented data is indeed of interest for the community. \n\n-- The motivation of the work is good. \n\n\n[Pros]\n\n-- Several strong statements are not supported, and therefore, the level of novelty is hard to appreciate. \n\n-- There is a lack of formalism to transmit the idea whilst there is empirical evidence of a few statements- there is not enough theoretical background to support the authors findings. \n\n\n\nDetailed comments for authors:\n\n-- [Novelty of the work] The proposed method strongly leverages the widely used  expected calibration error along with principles of mixup. Therefore, it is hard to appreciate the significance of the proposed method. Moreover, central observations are unsupported:\n > A major observation of the authors is: “Our key insight is that the confidence in the labels associated with the augmented data likely depends on how significant the transformation is.”  This is the definitional concept behind when generating pseudo-labels (-examples). That is, one seeks to get a high confidence (high probability to belong to the k class) to produce usable augmentation -- even from a set of feasible augmentation one can further average on the confidence produced on the label.  Authors should further elaborate on this idea as this is reported as highlighted in the paper.\n> The author's statement “assumption of label-preservation holds in practice” this also echoes the previous point, a major consideration is a ‘feasible augmentation’. This is a major area not only in the supervised setting  but also in semi-supervised learning -- how to define these $\\delta$ augmentations (perturbations) is far from being trivial not only in the sense of the augmented labels but also samples. Authors should formalise the assumption in the work including a ‘feasible augmentation’ of the samples (not the labels) as it is highly dependent on the augmented label; strong and weak augmentations how these concepts hold in the paper? \n> Author central contribution is given in (1) and the following sections revise concepts and connections with the principles of mixup. Authors should focus on giving a further interpretation of (1) along with the  assumptions that hold and how a certain modelling hypothesis fails this expression.\n\n-- What are the connections with SOTA augmenters? For example RandAugment, and CTAugment. \n\n[*] Cubuk, Ekin D., et al. \"Randaugment: Practical automated data augmentation with a reduced search space.\" CVPRW. 2020.\n\n-- From [**] the randomisation test where the true labels were replaced by random labels yet deep nets easily fits them raised the question of -- Are augmented labels really  better than search for feasible augmented samples (instead of labels)?  If one defines a feasible augmented sample then the label preserving will hold. \n\n[**] Zhang, Chiyuan, et al. \"Understanding deep learning requires rethinking generalization.\" ICLR 2017.\n\n[Experimental Results]  There is a lack of strong discussion and findings that somehow weakens the paper. One would like to see connections with other augmenters such as RandAugment, CTAugment, Cutout etc. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a simple and interesting method that adaptively smoothes the labels of augmented data based on a distance to the “clean” training data. The reviewers have raised concerns about limited novelty, minor improvement over baselines, and insufficient experiments. The author’s response was not sufficient to eliminate these concerns. The AC agrees with the reviewers that the paper does not pass the acceptance bar of ICLR.
Decision: Reject