ESP: Exponential Smoothing on Perturbations for Increasing Robustness to Data Corruptions
Authors: Weebum Yoo
Hwanjo Yu
Sung Whan Yoon
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Learning
Model Robustness
Domain Generalization
Common Corruption

Abstract: Despite the great advances in the machine learning field over the past decade, deep learning algorithms are often vulnerable to data corruption in real-world environments. We propose a simple yet efficient data augmentation method named Exponential Smoothing on Perturbations (ESP) that imposes perturbations on training data to enhance a model’s robustness to unforeseen data corruptions. With the perturbation on the input side, the target label of a sample is smoothed with an exponentially decaying confidence level with respect to the size of the perturbation. ESP enforces a contour-like decision boundary that smoothly encompasses the region around inter-class samples. We theoretically show that perturbations in input space can encourage a model to find a flat minimum on the parameter space, which makes a model robust to domain shifts. In the extensive evaluation on common corruption benchmarks including MNIST-C, CIFAR-10/100-C, and Tiny-ImageNet-C, our method improves the robustness of a model both as a standalone method and in conjunction with the previous state-of-the-art augmentation-based methods. ESP is a model-agnostic algorithm in the sense that it is neither model-specific nor data-specific.

Under review as conference paper ICLR 2023
ESP: EXPONENTIAL SMOOTHING ON PERTURBATIONS FOR INCREASING ROBUSTNESS TO DATA CORRUP- TIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the great advances the machine learning field over the past decade, deep learning algorithms are often vulnerable to data corruption in real-world environments environments. We propose simple yet efficient data augmentation method named Exponential Smoothing on Perturbations (ESP) that imposes perturbations on training data to enhance a model's robustness to unforeseen data corruptions. With the perturbation the input side, the target label of a sample is smoothed with exponentially decaying confidence level with respect to the of the perturbation. ESP enforces a contour-like decision boundary that smoothly encompasses the region around inter-class samples. We theoretically show that perturbations in inpul space can encourage a model find flat minimum on the parameter space, which makes a model robust to domain shifts. In the extensive evaluation on common corruption benchmarks including MNIST-C, CIFAR-IO/ 1FFAR-10/100--- and Tiny- Imageett--eet- C, our method improves the robustness of model both as a standalone method and in conjunction with the previous state-of-the- augmentation based methods. ESP is model-agnostic algorithm in the sense that it is neither model-specific nor data-specific.
INTRODUCTION
Over the past decade, deep learning models have rapidly evolved to update state-of-the-art perfor- mance on wide range of machine learning tasks, including computer vision, natural language processing, reinforcement learning, etc. Despite the remarkable advances in learning algorithms, deep models are often prone to data corruptions that hinder the successful training of networks. Albeit the importance of robust training, it very recent that the robustness of deep models real- world-driven data corruption has gained attention in the machine learning society. The vulnerability of the deep neural network (DNN) against adversarial perturbations was first raised way back in the early 2010s (Szegedy al., 2013; Goodfellow et al., 2015), and numerous methods have been proposed to enhance the model's 's robustness since then (Cui et al., 2021; Salman et al., 2020; Madry et al., 2018). On the other hand, the benchmarks for evaluating DNN's robustness to real-world driven common corruptions such as noise, blur, fog, etc., have been only recently established for the image classification tasks (Hendrycks improve the model robustness against the common corruptions are at their early stage of development (Hendrycks et al., 2021b;a; Rusak al., 2020; Wang et al., 2021a).
Dietterich, 2019; Mu Gilmer, 2019), and algorithms to
Recent approaches for improving the robustness to common corruptions in the image classification tasks either utilize image augmentation methods (Hendrycks et al., 2021b:a: Rusak et al., 2020; Calian et al., 2021), propose novel model architectures (Kim et al., 2021; Mao et al., 2021; He et al., 2021), or adopt the adaptation learning settings (Wang et al., 2021a; Rusak et al., 2021). While it is not yet revealed what the most dominant strategy against common corruption is, the group of augmentation-baan -based methods shares the desirable property that it can be easily combined with other promising methods to further enhance the model's 's robustness. In this aspect, ingmentation-based methods can be regarded as monetiag algorithms with a wide range of applicability. In addition. empirical evidence demonstrates that exploiting diverse data augmentations can effectively enhance the model's robustness against common corruptions in many real-world scenarios (Hendrycks al.,

---

Under review conference paper ICLR 2023
1g
(b)
Figure I: Visualization of the decision boundaries of the classifiers trained with the original dataset (denoted as 'Naive'), the augmented dataset with random noise with fixed Lg distance (denoted "I2"), and augmented dataset with our method (denoted ESP) The spiral-like distributed inputs from two classes are generated for the original dataset. The decision boundaries of the different classifiers illustrated. (a) ESP enforces contour-like decision boundary which generalizes better than L2 (b) ESP is less sensitive the maximum perturbation size compared to L2.
202 la; Calian et al., 2021). However, there has been weak theoretical understanding on how such data augmentations can enhance the model robustness.
We propose method named Exponential Smoothing Perturbations (ESP) that introduces the data perturbation in the form of Ly distance-based stochastic noises on the imput space. Also, ESP smoothes the confidence level of the target label for the perturbed input to be decaying with respect the size of the perturbation. In addition, we theoretically show that input perturbations that have bounded L2 norm can make a model find flatter minima in the parameter space. A model with flat minima has strong domain generalization capability (Cha et al., 2021a) and robustness to adversarial examples (Stutz al., 2021).
In the extensive simulations on the common corruption benchmarks, including MNIST-C (Mu Gilmer, 2019) and CIFAR-10/100-4 (Hendrycks & Dietterich, 2019), and Tiny-ImageNete a standalone ESP combined model in conjunction with prior data augmentation methods achieves stateoof-the-an accuracies with considerable margins. The main contribution of this paper is threefold:
We provide perspective label smoothing (Szegedy al., 2016) as tool embed the uncertainty of data perturbations in the input space. Furthermore, we show that the optimal decision boundary formed by the label smoothing function of ESP makes the classifier more affected by the topologies of manifolds and less affected by the number of datapoints manifolds.
We demonstrate that our method ESP, is least effective as L2 distance based perturbations both empirically and theonetically. With minimal assumptions and implications on the nature of datasets and models, ESP is shown to improve model robustness in common corruption benchmarks in all experiment cases further than L2 noise.
We analyze how perturbations input space can be related to perturbations parameter space. It been proved that finding flat minima in the parameter space makes classifier robust against distribution shifts in the test dataset (Cha el al., 2021b). We partially formalize the above idea by considering how the perturbation regions in the input space and parameter space can be related each other linear model.
2 RELATED WORK
Recently, various strategies for enhancing model robustness have been suggested. Here, we categorize prior methods into three types: data augmentation-basse model-specific, and adaptation based approaches.
Data Augmentation-basel Approaches The most popular approach to increase model robustness is augmenting training data to mimic the corruptions as form of data transformation. AugMix of Hendrycks al. (2021b) is image augmentation method that composes randomly-sampled basic image processing operations to produce a novel image that maintains the semantic information of the original image sample. In the training phase, AugMix utilizes the generated novel image samples located around the original sample in the input space. To be specific, the divergence between the

---

Under review as conference paper ICLR 2023
posterior distributions of the original and augmented samples is minimized for model embed the augmented samples close to the original one. When we compare Exponential Smoothing on Perturbations (ESP) with AugMix, our method adopts a simpler form of augmentation with Le- norm-bounded perturbations on the input space and directly trains the augmented sample with the exponentially smoothed soft label.
Another group of approaches leverages parameterized models for data augmentation. DeepAugment of Hendrycks et al. (2021a) distorts the image using pretrained mmaee-t--nmage transforming model to generate augmented images. Besides, DeepAugment adopts the perturbations on networks by employing predefined processes of hidden signals such as zeroing, negating, transposing, etc. The main differences between ESP and DeepAugment are the utilization of parameterized augmentation methods and the perturbations on networks. In the view of augmentation, ESP does not require additional deep networks for transforming original images. In our theoretical analysis, we provide an insight that relates the bounded perturbations on input via ESP to the perturbations in the parameter space, On the other hand, DeepAugment introduces model perturbations by processing the hidden signals, which are not restricted to the form of L2 norm bounded perturbation in the parameter space. Also, the robustness of DeepAugment is not theoretically guaranteed. Adversarial Noise Training (ANT) of Rusak et al. (2020) trains an additional noise generator that produces adversarial noise that maximally confuses the classifier. ANT is related to our method where they focus on imposing the noise-based perturbations respect to the size the perturbation and does not require additional training a noise generator. ESP is neither model-specine nor data-specific when compared to ANT which relies on model- and data-specifie noise generator.
the input space. However, ESP explicitly smooths the target label with
Mixup of Zhang et al. (2018) linearly interpolates between two data points from different classes and trains a classification model on the dataset that includes combined samples. The interpolated image is labeled by the interpolation between two one-hot labels of the original data samples. Mixup differs from our method, where the two samples are interpolated to construct novel training sample. From the perspective of the label smoothing by ESP, Mixup also smooths the target label of the combined sample by interpolating the original labels. However, Mixup suffers from the manifold intrusion problem due to the confict between the interpolated manifold and other original manifolds (Hendrycks et al., 2021b). ESP can alleviate the manifold intrusion problem by choosing a proper perturbation size and the degree of smoothing on the input space.
Model Architecture-based Approaches Another branch of approaches is developing model- specific training scheme. Based on the clean image samples, QualNet of Kim et al. (2021) pretrains a classifier with the invertible architecture and inverts it to obtain a decoder that is capable of reconstructing original images from the corresponding feature vectors. The prepared decoder is used as a reconstruction module that takes the features from new target classifier to be trained the second stage. Even from the corrupted input samples with low quality, the target classifier is then trained to construct cican-like features that be decoded into high-quality images. Vision transformer of (Dosovitskiy et al., 2021) has recently gained attention in building a robust vision classifier. Some works have changed the components of vision transformers to gain robustness (Mao al., 2021; Zhou et al., 2022; Mao et al., 2022), while others have designed self-supervised tasks for vision transformers (He et al., 2021). Despite the fact that the vision transformer- based approaches have been continuously updating their remarkable performance on common corruption benchmarks, they suffer from deficient generalizability. The group of model-specific methods relies carefully designed model architectures so that they have limitations be combined with other methods.
Approaches The other approach borrows the concept of domain adap- tation to improve model robustness. Test Entropy Minimization (TENT) of Wang et al. (2021a) is domain adaptation method that tunes the parameters of the batch normalization layers in the test time. The adaplation method indeed enhances the generalization capability to the common corruption that can be considered as input domain shifts. Robust Pscudo Labeling (RPL) of Rusak et al. (2021) assumes the unsupervised domain adaptation setting and exploits a self-learning method for training classifiers. The branch of adaptation- methods requires additional access to the target data either at the training stage (domain adaptation) or at the test time (test-time adaptation), which makes their usage restricted to specific circumstances.
Domain Adaptation-

---

Under review as conference paper ICLR 2023
3 ESP: EXPONENTIAL SMOOTHING ON PERTURBATIONS
3.1 BACKGROUND AND MOTIVATIONS OF ESP
Herein, we present the background and motivations for the algorithmic details of ESP.
Specifications of Lg-Norm-baset Noise When adding L2 distance-based random noise to an input, the perturbed inpul often lies outside the valid input domain, e-g., [0, 1] for samples in MNIST dataset. A simple clipping of the perturbed sample into the valid input domain probably results in a smaller effective noise than desired. To cope with the problem, we utilize the noise rescaling and clipping algorithm of (Rauber& & Bethge, 2020) that preserves the desired 12-norm of the noise while restricting the perturbed input the valid domain. For the sake of simplicity, we will denote the rescaled and clipped L2 distance-based random noise simply as Lg noise henceforth.
Desired Properties of L2 Noise Too large noise probably intrudes on other classes. When thinking of perturbed input sample that moves far away from the original data point, the noised sample can intrude on other class manifolds so that the model robustness eventually decreases. In Figure Ib, 12 noise with an excessive amount severely deteriorates the training of classifiers. However, reducing the size of La noise raises another issuc. The augmented samples should locate effectively far away from the original data point to guarantee sufficient margins of decision boundaries. To this end, we utilize truncated Gaussian distribution with non-zero mean >> 0 to sample the power of L2 noise.
Desired Properties of Label Smoothing Label smoothing is conventionally exploited for model static fashion (M uller et al., 2019).
calibration and penultimate layer's 's equidistant embedding Label smoothing assigns 1 - a for the true label and a/ (C'C11 1) for the other labels, where a E (0, 1) is a constant hyperparameter and C' is the number of classes. On the other hand, we re-purpose the label smoothing technique as tool for embedding the uncertainly of perturbations in the inpul space and defining a as a perturbation size-dependent hyperparamcter. By giving stronger label smoothing to larger perturbations, the decision boundaries of classifier are less affected by the number of perturbed data points but more by the distribution of data points. Nonetheless, shown in Figure Ia, the application of smoothing from the region nearby the original data point can sharply shape the decision boundaries without sufficient margins.
Our method, ESP processes probabilistic samplings of perturbation size and dynamic label smooth- ing functions that are carefully chosen. In the following sections, we formally describe our data augmentation strategy that guarantees the aforementioned properties of L2 noise and label smoothing.
3.2 ALGORITHMIC DETAILS OF ESP
ESP consists of three components: random orientation sampling, random size sampling, and a smoothing function. First, the orientation of the perturbation vector, i.e., the normalized directional vector of the perturbation, is randomly sampled. Since isotropic Gaussian has equal probability over the vector orientations, we have implemented the random orientation sampling by sampling Gaussian vector and normalizing it. Second, the size of perturbation vector is sampled with a pre-defined probability density function. In our experiments, truncated Gaussian distribution is used. Finally, the hard label of the a datapoint is smoothed with respect to the L2 norm, or the size of the perturbation vector. While any arbitrary nonincreasing function can be used, we use an exponentially decaying function to smooth the original label, after a certain threshold e of the perturbation size. One reason using exponential function ae *s is that for every and 72, ac -Max+ /ac -Az: = ar -Xx=++6) /ae ar -A6 holds. In other words, the original lahel is smoothed exponentially as the perturbation size grows, and the extent of smoothing is solely dependent on the relative sizes of perturbations.
There are total four hyperparameters consisting of ESP, three for the truncated Gaussian and one for the smoothing function. To be more specific, the truncated Gaussian Niome (s),, a, T, r) defines the probability density function on the size of perturbation vector. The smoothing function s(;,,,, C.) smooths the ground truth label of the perturbed datapoint, and & determines the extent of exponential label smoothing. For the right half, smoothing function reduces the confidence in the true label in exponential way that interpolates (e. 1) and (e 1/C + {). To reduce the hyperparameter search space, we have used a 0.57 and x e, of which the values are determined

---

Under review conference paper ICLR 2023
400-
4/+0
100 Now
Figure? 2: Illustration on the components and theoretical property of ESP. Left: Smoothing function s(2) and probability density function p(z) w.r.l perturbation size 2. Smoothing function smooths the true label exponentially after threshold e. Hyperparameter adjusts the scale of smoothing. Probability density function defines the size perturbation, whose maximum and minimum bound symmetric w.r.t = e. 0+ (=). (=).0-(3 (=) are the indicator functions l[[ 2 0] and I[= > 0j, respectively. Right: The smoothing function of ESP makes classifier less sensitive the number of datapoints consisting of manifolds, resulting more reasonable decision boundary (Theorem 3.3.)
100, 10, M = 10, || - 0.25.
Algorithm ESP psuedocode
Input: impul data (x, noise hyperparameter e, 0,0,7 smoothness hyperparameler Ş Output: augmented data (x', v')
k (irune(F; 7,6,7,,76-77
V 1)..
+ x + 6, where 8 = ku, u T if y. =
T,, C)
v/
s(k; T,S,
otherwise where
e -==(==
2 and A
(Z;T,E..7 =
T-6 ln
otherwise
+ EC
return z',
empirically. Two plots on the left side of Figure indicate the smoothing function and the truncated Gaussian. Algorithm is the pseudocode of ESP.
3.3 THEORFTICAL PROPERTIES OF ESP
1/C, ESPP can be viewed as a generalized form of L2 noise family. When T - and & + 1- I/C, ESP has the same effect Lg noise with the constant perturbation radius Hercin, provide the theoretical analysis ESP, i.e., the decision boundary of ESP is less affected by the imbalance between classes, and the relationship between the ESP's input perturbations and flat minima. When focusing the first property, the label smoothing function of ESP makes the optimal decision boundary less affected by the number of datapoints composing manifolds, but more by the position of manifolds and the distance between manifolds. For simplicity, we provide a binary classification task with the imbalance between two clusters.
Problem Formulation Suppose binary classification dataset D is given follows:
((x,))) D=((x,)) ((x + 61, 1,0, (x: + ca, 1,0), (x2 ca+ 1, 1), ... (X2 + 60, 0, 1)}, (1)
=
where x E RM y E {0, 1)2. Also, and are integer and b = a.M Larger M implics the imbalance between classes, i.e., the class
1) contains M times larger number of samples than the class y = (1,0). Xı and X2 are the centroids of two different classes, respectively. e; indicates the deviations around the centroids. The problem setting simulates binary classification task, where data samples are located around their class centroids and the number of samples of two classes is imbalanced by the factor of M. By letting 6; -> for all C [a + b), we can make each cluster to be concentrated, i.c., it stands for binary classification task with the clearly separated clusters.
Then let us denote perturbed data point without label smoothing as (Xs, Vs) == (x + d.y). To be specific, (x, y) is uniformly sampled data point from D, and ku, k Ni (e, e T,

---

Under review as conference paper ICLR 2023
r),u == e/||e|1. ~ NiO, I), i.e., ESP without label smoothing. Similarly, let us denote a perturbed data point via ESP as (Xg, == (x + 6, where y= =(l]61),, s(2) := 0,, (-= + e) + 0- (z <)e -A=-ss for some À > 0. We will use yoı and yaa to denote the the first and the second elements of yo, and similarly VE1, V82 for ye-
When M becomes larger than certain value, in other words, when the imbalance becomes severe, the perturbed sample from an impul x becomes more likely be classified into the dominant class (0, 1), Le. M6, 2 0.5 and VE, 2 0.5 for ESP without smoothing and ESP respectively. The following mathematical statement claims that ESP endures more severe class imbalance that makes any perturbed inputs be more likely to be classified into the dominant class.
Theorem 1. For any x, let n' RRt be the number such that for all M 2 ré,, Blyss Xg3= K x] 2 0.5
holds, and let n E R be the number such that for all M 2 n1 E y82 X = x] 2 0.5 holds. Then, 2 n'.
Theorem 1 claims that the smoothing function of ESP makes the optimal decision boundary, {2r ElyEr x] = BByss =] = 0.5), less affected by the severity of the data imbalance, M. Figure 2. we simulate a binary classification task with strong class imbalance and show that ESP is less sensitive the imbalance. The decision boundary of ESP locates around the middle of two manifolds, but ESP without smoothing. pushes the decision boundary to the inferior class, which makes almost all regions classified into the dominant class. Proof of Theorem 1 is on Appendix D.
In the perspective of domain generalization, researches insist that seeking flat minima in the parameter space increases model robustness against distribution shifts (Iımailov al., 2018; Cha et al., 2021b), perturbation in the input space can be related to the perturbation in the parameter space, we can deduce that the perturbation of ESP encourages a model to find flat minima in the parameter space. We partially formalize the property by considering a linear model with sigmoid activation function.
Problem Formulation Given a linear model f x ++ oW x + b). where x E R", W E Rm** (m < n C IR' we consider the relationship between input perturbation (8 E R²) and parameter perturbation (A E Rm**") that satisfies oW
5) b) = o(( + A)z + b). When we have input perturbation bounded by Ly-norm, i.e.. || S 7, what will be the possible perturbation region Rs for A so that for any || S y, there exists A E R& satisfying the equality or vice versa? Conversely,
what will be the perturbation region Rs for 8, given ||||| 7
Definition 1. (Definition of R&) Given W Rmxx {A E Rm ||||| S 7}, Rg E R. is a region that satisfies the following constraint:
R", and parameter perturbation region
A || S 7, 3,36 E Rş s.t. W8 At and V 8 E Rs. 3 || Y s.t. W6- Ax
Definition 2. (Defmitiinn of Raj Given W Rmxx 1) (#1, "...... .IN (r. E RD /(0} for E [N]), and input perturbation region {ō E Rn | ||ō|| < 7). RA E Rinxn is a region that satisfies the following constraint:
Vr E D2 V 7, 3A E RA s.t. WS= = Ar and Vr E D, E =|و||||| S s.t. W8 = Ar
With these definitions the regions of interest, we now present theorems converting perturbations in input space to parameter space (Theorem 2) and vice versa (Theorem 3, 4.) Colloquially, Theorem 2 states that perturbations bounded by L2 norm in the parameter space can be converted to the pertur- bations bounded by an rotated ellipsoid in the input space. Meanwhile, converting the perturbation in the input space to parameter space in closed form expression is infeasible. As an alternative, we provide the subset and the superset of the converted perturbation region in the hyperparameter space in Theorem 3 and 4. X, is (m x n2 square matrix which is defined by the input and weight W, The formal definition on X2 is stated Appendix C.
Theorem 2. Given W E Rmxx, x E R.,,, and parameter perturbation region {A E RMXn Y. a volume-zero m-dim rotated ellipsoid satisfies the definition of Rs.
||||| S
Theorem 3. Given W E mmxn D =x11---- ... ,IN E /R0) for E [N]), and input pertur-
bation region {ō E R [|6|| S {7). let Imaz == max, ||=:|| and Amin = min(Ar.".. ",mm.) ..m)-
(A E Rmxx ||||| S [[|=maxi) 1|2 [2/22, (/2 1-1) RA
Theorem Given W E Ruxe, D (2 .. [N; E RR/((]) for E [N]). and input perturbation region (5 E R" ||6]| S {)), let R; = (d E Ruxn d x d & 1) and I := (R.
[]]] RA C (arg min min Ryye R,, er max l1p

---

Under review conference paper ICLR 2023
ESP introduces a bounded noise into input samples, and means that the perturbed samples can deviate in the bounding sphere around the clean input sample. Also, ESP smoothly decays the confidence level of labels for the perturbed imputs in the bounding sphere. The aforementioned mathematical statements claim that the perturbed inputs in bounded sphere can be interpreted the perturhed parameter space in a bounded ellipsoid for the clean input samples, and vice versa. Consequently, label smoothing of ESP makes the perturbations of parameters in the bounded ellipsoid smooth labels the clean input samples. Then the loss landscape in the parameter space becomes more smooth by preventing the rapid change of Ioss values due to the perturbations of parameters. Along with the thcoretical analysis, we empirically confirm that ESP achieves flatter minima than the baselines. The proofs of Theorem 2, 3, and 4 are on Appendix B and C.
EXPERIMENTS
Table I: Model robustness over MNIST-C, CIFAR-10/100-C, and Tiny-ImageNet-C benchmarks in the measure of average corruption error (lower is better). The reported values are the average corruption of three individual runs for each method. The best results marked in bold.
Augmentation
MNIST-C
CIFAR-10-C
CIFAR-100-C
Tiny-IN-C
Naive Naive L2
8.01 0.10
25.57 + 0.45
52.21 + 0.47
75.49 + 0.24
7.07 0.43
18.55 = 0.26
45.64 + 0.11
75.09 + 0.15
Naive ESP
6.45 0.02
16.17 = 0.41
40.28 0.29
73.97 0.37
AugMix
14.36 + 0.30
10.67 = 0.09
35.50 0.10
67.78 + 0.48
AugMix + L2
12.02 + 0.29
10.36 = 0.07
35.12 0.20
67
0.01
AugMix + ESP
11.69 + 0.29
8.62 0.11
34.59 0.18
67.71 + 0.03
DeepAugment DeepAugment Lg
10.68 + 0.27
13.21 + 0.11
39.54 + 0.04
64.75 + 0.32
10.57 0.06
11.94 = 0.21
= 3
64.58 + 0.33
DeepAugment + ESP
10.35
11.16 = 0.07
36.46 + 0.21
61.43 0.07
AugMix + DeepAug
7.46
9.15
32.56 + 0.05 0.05
60.61 + 0.13
AugMix DeepAug Lg
7.22 0.05
32.44 = 0.06
61.07 + 0.19
AugMix DeepAug ESP
7.09 0.33
8.90 = 0.11
32.23 + 0.17
59.02 + 0.21
4.1 DATASET STATISTICS
MNIST-C (Mu & Gilmer, 2019) 15 corruptions (brightness, canny edges, dotted line, fog, glass blur, impulse noise, motion blur, rotate, scale, shear, shot noise, spatter, stripe, translate, zigzag). There are 10,000 images corresponding to each corruption, resulting in total 150,000 images.
CIFAR- 101RR-,0/000C, ImageNeCe (Hendrycks & Dietterich, 2019) 15 corruptions (brightness, contrast, defocus blur, clastic transform, fog, frost, Gaussian, glass, impulse noise, jpeg compres- sion, motion blur, pixelate, shot noise, snow, zoom blur), 5 severities. There are 10,000 images corresponding each severity, resulting in total 750,000 images.
4.2 EXPERIMENTAL SETUP
Model Architecture For MNIST-C benchmark, we have used convolutional neural network architec- ture proposed in (Rony et al.. 2019). For CIFAR- 10/100001ICCC- benchmarks, have used WRN-40-2 model (Zagoruyko Komodakis, 2016) as backhone network. For Tiny-ImageNet-C benchmark, ResNet18 (He et al., 2016) has been employed.
Optimizer In all our experiments, SGD momentum with initial learning rate of 0.1 and momentum value of 0.9 has been used. For both MNIST-C and 1011-- experiments, we have used cosine learning rate decay scheduling to train the model until convergence as in Hendrycks et al. (2021b). For ImageNee benchmark, we have utilized step learning rate decay scheduling at 100 and 150 epoch with the coefficient of 0.1 as in Wang et al. (2021b).

---

Under review conference paper ICLR 2023
Hyperparameter Tuning We have used grid search to find the optimal hyperparameters for L2 noise and ESP. As mentioned in Section 3.2, we use = 0.5T and T R e to reduce the hyperparameter scarch space. Despite the fact that there is general rule for deciding & values, we have chosen { such that the maximally smoothed true label (C-1 + {) y times higher than the other labels (1-C- G)/(C - 1)). Specifically, we have chosen = {10, 20) for MNIST-C/CIFAR- 10- 7. {20, 50, 100} for CIFAR- 100-C, and y {200} for Tiny-ImageN
Such choice of (0.426, 0. 0.590 for MINIST-CICIEAR-1OCCC & {0.158, 0.326, 0.493) for
gamma results CIFAR-100-C, and & {0.496) for Tiny-ImageNet-C.
Evaluations As in Hendrycks et al. (2021b), we have calculated the average corruption error across different corruption types and severities.
Further experiment details can found Appendix E.
4.3 RESULTS
We first examined the performance of ESP with respect to different data augmentations in common corruption benchmarks (referring Table 5.) On the MNIST-C benchmark, AugMix and DeepAug- ment impaired model robustness in contrast to L22 noise and ESP. The ensemble of AugMix and Degpengment slightly increased model robustness, but was still inferior to Lg noise and ESP. On the other hand, Ly noise and ESP increased model robustness either standalone method or combined with other methods. In the CIFAR- 10-100-C experiment, all methods improved model robustness both solely and in composition with other methods as well. On the CIFAR-10-C benchmark, AugMix enhanced model robustness the most sole method, and AugMix composed with ESP enhanced robustness the most as an ensemble method. CIFAR-IO0-- experiment, AugMix exhibited best performance among the sole methods, and combining AugMix, DeepAugment, and ESP together yielded the highest robustness among ensemble methods. In Sinyy-mmaeNee-CC experiment, ESP boosted model robustness in circumstances by large margin, in constrast to L2 noise which trivial or no improvement on the model robustness. In general, ESP showed consistent improvement on the model's robustness in all experiment cases, dominating L2 distance based noise.
Noks
Aughtt
Augil
Figure 3: Model robustness across different hyperparameter configurations consisting of ESP's search space in MNIST-C (left), CIFAR-IO-C (middle), and CIFAR- 100-C (right) dataset. X-axis represents the value of ESP, and y-axis represents the error according to varying e. The average corruption error of original augmentation method is represented gray line.
Next, we compared the model performance over different hyperparameters consisting the search space of ESP (Figure 3.) While ESP improved model robustness in most cases, the amount of robustness gain differed meaningfully with respect to the perturbation size (e) and its corresponding hyperparameters (r, σ)) in many cases, drawing a convex average corruption error loss graph with respect to the perturbation size. One interpretation of the convex-shaped loss graph is that when the perturbation too small, model learns decision boundary that not general enough; however when the perturbation is too large, the dccision boundary tumbles down due to fuzzy data augmentations intruding each other's manifolds severely.
Subsequently, we investigated the flatness of each augmentation method in the CIFAR-10-C bench- mark (Figure 4). With varying radius, we used Monte-Care simulation with 50 individual samples for each method. The input perturbation that has bounded Le norm of ESP encouraged the model to find a relatively flat minimum in the hyperparameter space compared to the naive Empirical Risk

---

Under review conference paper ICLR 2023
6.000
C.004
0.002
Maw Aayta
Right Drscha
Ngto Deping
Figure 4: Experiment results on the removal of smoothing function and on the flatness of local minima. Left: An ablation study on the smoothing function of ESP. A+D denotes the ensemble of AugMix and Dugmentment methods. Right: The flatness of Empirical Risk Minimization (denoted as naive), ESP, AugMix. and DeepAugment, respectively in CIFAR-10-4 benchmark.
Minimizationn (ERM). While AugMix found the flattest minimum the parameter space, the local minimum found by DeepAugment was escalating the most.
Finally, we analyzed how the removal of the smoothing function in ESP can affect performance. In MNIST-C and CIFAR-10-C benchmarks, there were statistically meaningful differences in the model robustness. Nevertheless, removing the smoothing function of ESP significantly harmed model robustness in SFFAR-100-CC and Tiny- Mmag--eeeeeet-CCC benchmarks. One possible explanation for this phenomenon is that the increase in the diversity of classes results in smaller and diverse data manifolds with different labels. This may induce larger overlapping areas of perturbations with different labels in the input space.
DISCUSSION
AugMix and DeepAugment damaged robustness in MNIST-C benchmark, but prominently enhanced model robustness in CIFAR-100100- and Tiny-ImageNet-C benchmark. On the other hand, ESP showed the tendency consistently improve model robustness mild way. We interpret this phenomenon as the difference between each method's inductive bias. Since ESP is a high-level data-agnostic algorithm, the robustness gain of data augmentation may not be drastic compared to the existing methods. Nonetheless, there is more room for exploiting ESP, regardless the semantics dataset.
However, ESP is sensitive to the choice of hyperparameter that determines the maximal perturbation size. The problem stems from the intrinsic nature of perturbation based augmentation methods. With varying data distributions on different tasks, we cannol estimate the sweet spot ESP before actually conducting model training with varying perturbation sizes. Insufficient perturbations will trivially improve robustness, while intense perturbations will demolish the decision boundary of the target model due to manifold intrusions overwhelming the end.
CONCLUSION
Deep neural networks (DNNs) being prone to real-world driven common data corruptions, various methods have been proposed to increase model robustness. Among several approaches, we have focused on developing augmentation-baan based method due its broad applicability. Inspired by the robustness gain achieved by simple Lg distance based random noise, we have proposed an efficient and general data augmentation method, ESP, that makes classifier robust to diverse image data corruptions without strong inductive bias on the nature of dataset. The data augmentation nature of ESP enforces a classifier to have a contour- like decision boundary, different from most of the existing DNN learning algorithms. Moreover, we have provided theoretical analysis and experiment result on how perturbations with bounded L2 norm can be related the perturbations in the parameter space. Despite the fact that we have only exploited corrupted image classification benchmarks on measuring the robustness gain, ESP can be exploited to different classification tasks other than image classification to enhance model's robustness to unexpected data corruptions.


Reviews:
[{"review_id": "zsNwGjfV8A", "reviewer": "Reviewer_rCik", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for improving robustness to common corruptions via L2-norm bounded data augmentations of varying sizes. The authors show theoretically that data augmentation of this form results in models with flatter minima in parameter space, which has been shown to result in better generalization. This paper also proposes a method for label smoothing that exponentially smooths the labels based on the size of the perturbation. ", "summary_of_the_review": "While this paper does report improvements across all datasets in terms of the common corruptions performance and includes some theoretical justifications, it's unclear to me whether the proposed method results in performance improvements for a variety of different unseen perturbations, or just noise-based perturbations. It is also unclear how the hyperparameters would be chosen in practice for unseen perturbations. Given these uncertainties, I'm leaning towards reject. ", "strength_and_weaknesses": "Strengths: \n- The paper is mostly well written, and the contributions/methods are clear. \n- The paper provides some theoretical justifications for their method. \n- The proposed method results in consistent improvements in generalization on common corruptions across different datasets.\n\nWeaknesses:\n- Since the proposed method is augmenting the data with noise, and evaluating on the common corruptions dataset, of which several of the corruptions are noise-based corruptions, I think it's important to state whether robustness to all of the corruption types improves, or just those noise type corruptions. \n- The authors mention the proposed method is sensitive to the choice of hyperparameter determining the max perturbation size. When doing the hyperparameter search, how do you determine the optimal hyperparameters? I.e. are you choosing the hyperparameters based on performance on the common corruptions test set? In practice, you'd need some other validation metric for choosing hyperparameters, as fundamentally you are trying to improve performance on unseen, out of distribution data. \n- The paper does not report performance on clean/unperturbed test images. How does the proposed data augmentation scheme affect standard test accuracy?\n\nOther suggestions:\n- I would maybe state earlier in the paper that the label smoothing technique only results in improvements for datasets with a larger number of classes. \n- Most of the figures are too small to properly read. It's a bit difficult to gather any insights from Figure 3. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tou4E3hGmMQ", "reviewer": "Reviewer_6X1k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new augmentation strategy ESP, which augment training with random perturbation with smoothed labels. The experiments showed that ESP outperforms vanilla L2 augmentation on common corruption robustness. The authors also gave theoretical analysis on the properties of ESP.", "summary_of_the_review": "In all, I think this paper proposed a simple yet effective method to boost the model robustness against common corruption. The theoretical analysis also provides some insights. However, as I have mentioned in the Weaknesses, I hope the authors can have more experimental comparison and also compare with some related existing theoretical results.", "strength_and_weaknesses": "Strength:\n* The theoretical analysis provides some insights on the benefits of label smoothing.\n* The proposed ESP augmentation has the potential to be combined with many existing works on common corruption robustness to further boost the model robustness.\n\nWeaknesses:\n* The theoretical analysis is not clear enough. The imbalance's severity N and the normal distribution use the same letter N, which makes me confused when reading this section. \n* Theorem 1 states that ESP can make the optimal boundary less sensitive to imbalance severity, but why is this a desired property for ESP? What is the benefit for a less-sensitive boundary?\n* Theorem 2 proves that perturbations in the input space can be converted to perturbations in parameter space. This result is available in [1] Lemma A.4.\n* In experiments, the paper only compares the result with vanilla L2. I suggest the authors to compare with more label smoothing methods and also mixup.\n\n[1] Convergence of Adversarial Training in Overparametrized Neural Networks\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZmCZ8zjWeql", "reviewer": "Reviewer_ST3E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a data-agnostic and model-agnostic data augmentation method ESP to enhance a model’s robustness to unforeseen data corruptions. Concretely, ESP consists of two parts, perturbation adding and exponential label smoothing. The added noise perturbation in ESP is sampled from a truncated Gaussian distribution, while Label smoothing here takes the magnitude of perturbation into consideration. Stronger perturbations are companioned by stronger label smoothing.\nThe author provides theoretical proof that perturbations in input space can encourage a model to find a flat minimum on the parameter space. \nExperiments demonstrate that ESP when combined with existing augmentation methods improves the robustness to different degrees.\n", "summary_of_the_review": "Given the weakness listed above, I tend to reject this paper before rebuttal.", "strength_and_weaknesses": "Strength:\n1. The proposed data augmentation is simple, effective, and also theoretically justified.\n2. Both sample-specific adversarial perturbation and universal adversarial perturbation are considered in this study.\n\nWeakness:\n1. As also pointed out by the author, the proposed method is sensitive to hyperparameters. The reported improvement in Table is often marginal. The two facts make the proposed method less practical.\n2. Given the nature of the proposed data augmentation, the author should report the mean and variance of each experiment since randomness is brought by sampling from a truncated Gaussian distribution. This is necessary since the improvements reported are not significant.\n3. In Figure 4, the authors aim to claim the effectiveness of exponential label smoothing. However, the standard label smoothing should be included as a baseline. The improvement of ESP over perturbation-only baseline can be brought by the effectiveness of standard label smoothing.\n4. In Figure 2, the theoretical properties of ESP are illustrated. Is it possible to draw the illustration with real experimental data?\n5. There are many appendix points in the main text. However, no appendix is found in the submission.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DXF56Q9fPx7", "reviewer": "Reviewer_eTKu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new data augmentation method, which jointly augments data and label, using an empirical hand-crafted rule: The input image is added a random Gaussian noise, and the corresponding label is smoothed with the strength proportional to the image noise magnitude. I think the goal of such data augmentation is to enforce model to have a certain level of smoothness, as regularized by the strength of the augmentation. And ideally improve model robustness. However, the empirical results shows marginal improvements. For example, on CIFAR0-C and CIFAR100-C, it only brings 0.39% and 0.21% accuracy gain over the second best method. I have some assumptions on why the proposed method not bringing good performance gains. Please see details below. ", "summary_of_the_review": "Please see above", "strength_and_weaknesses": "1. The isotropic random noise might not be the best choice for image augmentation. Imagine two random noises with the same magnitude, one is focused on the foreground object, while the other is focused on the background pixels. The former on should be accompanied with a larger perturbation on the label (i.e. a more smoothed label in your framework), while the later should be accompanied with a smaller label perturbation. This is because the background noise hardly affects the semantic meaning of the image, and thus the label shouldn't change much. However, in your current design, both noises would have a same level of label perturbation, which might not be the best design in my opinion. \n\n2. Besides the marginal performance gain, the proposed method also suffers from hyper-parameter tuning overhead. Specifically, it introduces four new hyper-parameters to tune. I wonder whether it is worthy to pursue the marginal performance at such cost.\n\n3. Since the performance gain on the small datasets are maringal, I wonder whether it brings benefits on larger datasets such as ImageNet(-C).\n\n4. Has the performance on clean test sets been reported in the main text? Both clean accuracy and robustness are important for a real-world model. Previous works such as AugMix and DeepAug usually report clean accuracy alongside robustness. \n\n5. It seems to me that MixUp/CutMix/etc. also share similar idea to jointly perturb label and input. Maybe the author should discuss why the proposed way is better than those previous methods and provide empirical comparison results?\n\nOverall, I suggest revision and resubmit considering the above limitations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject