Why adversarial training can hurt robust accuracy
Authors: Jacob Clarysse
Julia Hörrmann
Fanny Yang
Venue: ICLR 2023 poster (2023)
Keywords: Adversarial training
Learning Theory
Robust generalisation

Abstract: Machine learning classifiers with high test accuracy often perform poorly under adversarial attacks. It is commonly believed that adversarial training alleviates this issue. In this paper, we demonstrate that, surprisingly, the opposite can be true for a natural class of perceptible perturbations --- even though adversarial training helps when enough data is  available, it may in fact hurt robust generalization in the small sample size regime. We first prove this phenomenon for a high-dimensional linear classification setting with noiseless observations. Using intuitive insights from the proof, we could surprisingly find perturbations on standard image datasets for which this behavior persists. Specifically, it occurs for perceptible attacks that effectively reduce class information such as object occlusions or corruptions. 

Published as a conference paper at ICLR 2023
WHY ADVERSARIAL TRAINING CAN HURT ROBUST ACCURACY
Jacob Clarysse', Julia Hörrmann', Fanny Yang' Department of Computer Science, ETH Zürich
2. Department of Mathematics, ETH Zürich jacob. cl glaryssed fan. yang e www.fee.e...... ch; [julia. hoe noerrmann) ( estat.tttmm.......e.... math.h.. ethz.ch
ABSTRACT
Machine learning classifiers with high test accuracy often perform poorly under ad- versarial perturbations. It is commonly believed that adversarial training alleviates this issue. In this paper, we demonstrate that, surprisingly. the opposite can be true for natural class of perceptible perturbations even though adversarial training helps when enough data is available, it may in fact hurt robust generalization in the small sample size regime. We first prove this phenomenon for a high-dimensional linear classification setting with noiseless observations. Using intuitive insights from the proof, we could find perturbations on standard image datasets for which this behavior persists. Specifically, it occurs for perceptible perturbations that effectively reduce class information such as object occlusions or corruptions.
INTRODUCTION
Today's best-performing classifiers are vulnerable to ad- versarial attacks Goodfellow et al. (2015); Szegedy et al. (2014) and exhibit high robust error: for many inputs, their predictions change under adversarial perturbations, even though the true class stays the same. Such content- preserving (Gilmer et al. 2018), consistent (Raghunathan al.) (020) attacks can be either perceptible or imper- ceptible. For image datasets, most work to date studies imperceptible attacks that are based on perturbations with limited strength or attack budget. These include bounded f--norm perturbations (Goodfellow al.] [015 Madry et 2018 MMosavi-
30+
standard training adversarial training
robust error
20+
250
500
750
1000
number of samples
Figure 1: On the Waterbirds dataset attacked by the adversarial illumination attack, adver- sarial training (yellow) yields higher robust error than standard training (blue) when the sample size is small, even though it helps for large sample sizes and in setting where the standard error of standard training is small. (see App. for details).
et 2016)), small transfor- mations using image processing techniques (Ghiasi et al.,
2019
Zhao al. 2020 Laidlaw et al. [222]] Luo al.
2018
or nearby samples on the data manifold (Lin et al. Zhou al. 2020p. Even though they do not visibly
2020
change the image by definition, imperceptible attacks can often successfully fool a learned classifier.
On the other hand, perturbations that naturally occur and are physically realizable are commonly perceptible. Some perceptible perturbations specifically target the object to be recognized: these include occlusions (e.g. stickers placed on traffic signs (Eykholt et al. 2018) or masks of different sizes that cover important features of human faces (Wu et al. 2020h) or corruptions that are caused by the image capturing process (animals that move faster than the shutter speed or objects that are not well-lit, see Figure 2). Others transform the whole image and are not confined the object itself, such as rotations, translations or corruptions Engstrom et al.] (2019); Kang et al. (2019). In this paper, we refer to such perceptible attacks as directed attacks. In contrast to other attacks, they effectively reduce useful class information in the input for any model, without necessarily changing the true label
we say that they are directed and consistent, more formally defined in Section 2 For example, stop sign with small sticker could partially cover the text without losing its semantic meaning. Similarly, flying bird captured with a long exposure time can induce motion blur in the final image without becoming unrecognizable to the observer.

---

Published as a conference paper at ICLR 2023
directed
imperceptible
perceptible
(a) Masks
(b) Original
(c) Lighting
(d) Blur
(e) Classification perturbations
Figure 2: Examples of directed attacks on CIFARIO and the Waterbirds dataset. In Figure 2a
corrupt the
image with black mask of size and apply motion blur on the bird in the image respectively. All perturbations reduce the information about class in the images: they are the result of directed attacks. (e) Directed attacks are subset of perceptible attacks.
and in Figure and we change the lighting conditions (darkening)
In the literature so far, is widely acknowledged that adversarial training with the same perturbation type and budget as during test time often achieves significantly lower robust error than standard training (Madry et al. 2018 Zhang et al. 2019 Bai et al. 2021).
In contrast, we show that adversarial training not only increases standard error (Zhang al. 2019
Tsipras et al. 2019 Stutz et al. 2019 Raghunathan al. 2020), but surprisingly, in the low sample
regime,
adversarial training may even increase the robust error compared to standard training!
Figure D illustrates the main message of our paper on the Waterbirds dataset: Although adversarial training with directed attacks outperforms standard training when enough training samples are available, it is inferior when the sample size is small (but still large enough to obtain small standard test error).
Our contributions are as follows:
We prove that, almost surely, adversarially training a linear classifier on separable data yields a monotonically increasing robust error as the perturbation budget grows. We further establish high-probability non-asymptotic lower bounds on the robust error gap between adversarial and standard training.
Our proof provides intuition for why this lower bound on the gap is particularly large for directed attacks in the low sample regime.
We observe empirically for different directed attacks on real-world image datasets that this behavior persists: adversarial training for directed attacks hurts robust accuracy when the sample size is small.
2 ROBUST CLASSIFICATION
We first introduce our robust classification setting more formally by defining the notions of adversarial robustness, directed attacks and adversarial training used throughout the paper.
Adversarially robust classifiers For inputs x E Rd, we consider multi-class classifiers associated
with parameterized functions fo Rd R if K and fo Rd - K 2, where K the number of labels. For example, [z) could be linear model (as in Section 3 neural network
(as in Section The output label predictions are obtained by h((f(())
(fg((f(()) for K
and h (=)) arg max.e(1,...x? fo(x)) for K 2.
In order to convince practitioners to use machine learning models in the wild, it key to demonstrate that they exhibit robustness. One kind of robusiness is that they do not change prediction when the input is subject to consistent perturbations, which are small class-preserve Mathematically speaking, for the underlying joint data distribution P, the model should have small
perturbations.
e, robust error, defined as
Err(O; Eie) = E(Z,y)~ max
{(fo(z').))
where É is the class determined by (x)) is equal to and 1 otherwise. Further, T(x; Exe) indicates a perturbation set around of certain transformation type with size ctsst- Note that
2

---

Published as a conference paper at ICLR 2023
the (standard) error ((..))-o((foxx).) y) of classifier corresponds to Err(0: 0) the robust error evaluated at E = 0.
Directed attacks The inner maximization in Equation is often called the adversarial attack of the input for the model fo and the corresponding solution referred to as the adversarial example. In this paper, we consider directed attacks that effectively reduce the information about the true classes, with image-based examples depicted in Figure For linear classification, we analyze directed attacks in the form of additive perturbations that are constrained to the direction of the optimal decision boundary (see details in Section 3.I). In particular, note that the set of directed perturbations is restricted to directions attacking the Bayes optimal classifier.
Adversarial training A common approach to obtain classifiers with a good robust accuracy is to minimize the training objective Lr, with a surrogate robust classification loss L
(0) = T2 Σ
max I(fo(1))oi) (ET())))
also called adversarial training. In practice, we often use the cross entropy loss T.(z) log(1 (1 e)*) and minimize the robust objective by using first order optimization methods such as (stochastic) gradient descent. SGD is also the algorithm that we focus on in both the theoretical and experimental sections. When the desired type of robustness is known in advance, it is standard practice to use the same perturbation set for training for testing, i.e. T(2; fw) Tx;; fue). For example, Madry et al. (2018) show that the robust error sharply increases for 6u < ep- In this paper, we demonstrate that for directed attacks in the small sample size regime, in fact, the opposite is true.
3 THEORETICAL RESULTS
In this section, we prove for linear functions fa(x) = 0Tx x that in the case of directed attacks, robust generalization deteriorates with increasing fu- The proof, albeit in simple setting, provides explanations for why adversarial training fails in the high-dimensional regime for such attacks.
3.1 SETTING
We now introduce the precise linear setting used in our theoretical results.
Data model We assume that the ground truth and hypothesis class are given by linear functions fo(x) = 0 x and the sample size n is lower than the ambient dimension minus one. The generative distribution F,, is similar Tsipras al. 2019 Nagarajan & Kolter 2019: The label y {+1,- 11 is drawn with equal probability and the covariate vector is sampled as x [y Ø with the random vector 7 E Rd-1 drawn from standard normal distribution, i.e. T ~ N/(0, of (Tan). We would like to learn classifier that has low robust error by using dataset 1) (z,, y4 y0). with i.i.d. samples from P., Intuitively, the separation distance reflects the signal strength of the data distribution.
Notice that the distribution P, is noiseless: for a given input x, the label y = (ig) is determinis- tic. Further, the Bayes optimal linear classifier (also referred to as the ground fruth) is parameterized by the first standard coordinate vector, 8* e1 By definition, the ground truth is robust against all perturbations that do not change the sign in the first coordinate of the sample, i.c. consistent perturbations, and hence so is the optimal robust classifier.
Directed attacks In this paper, we focus on consistent directed attacks that by definition efficiently concentrate their attack budget to reduce the class information. For our linear setting this information lies in the first entry. Hence, we can model such attacks by additive perturbations in the first dimension
T(I;e) {a' r + 8 8 Beg and E S 3 S e).
(3)
Note that this attack is always in the direction of the signal dimension, i.e. the Bayes optimal classifier or equivalently the ground truth. Furthermore, when e < 5, it is a consistent directed attack. Observe how this is different from & attackk an lp attack, depending that only has a very small component in the signal direction.
the model, may add perturbation
Note that the result more generally holds for non-sparse models that are not axis aligned by way of simple rotation 2 Un. In that case the distribution is characterized by 0* = u1, where uy is the first column vector of U, and rotated Gaussian in the d dimensions orthogonal to 0*
3

---

Published as a conference paper at ICLR 2023
robust gap
experimental lover bound
robust error gap
robust error
20
20
10
adversaiial training
experimental
01
standard training
lower
150
adversarial budget
d/n
d/n
(a) Robust error increase with (b) Shandard-adversaae training (c) Effect of overparameterization
Figure 3: Experimental verification of Theorem 3.1 (a) set = 1000, r 12 and n = 50. The robust
error gap between standard and adversarial training as function of the adversarial budget 6 = 5 independent experiments (bluc) and the lower bound given in Thcorem 3.1 (gray). In (b) and (c), set d 10000 and vary the number samples gap the lower bound of Theorem 3.1
The robust error of standard and adversarial training with 6r, 4.5. (c) The error more experimental details see Appendix
We study classifier that is the solution of running gradient descent on the adversarial logistic loss. A long line of work (Soudry et al. 2018 Ji & Telgarsky 2019 Chizat & Bach 2020 Nacson et al. 2019) Liu al.) 2020) studies the implicit bias of (SJGD
Robust max- classifier
on the (standard) logistic loss and separable data. In particular, they show directional convergence the max-margin solution. For the adversarial logistic loss and linear models in particular, (S)GD
converges to the robust max-f2-marge solution (Li et al. 2020).
arg max 001 (T(
min
y:0 x/4
Even though our result is proven for the max-2y-me mmrginn classifier, it can easily be extended to other interpolators.
3.2 MAIN RESULTS
We are now ready to characterize the re-roust error a function of Gr. the separation r, the dimension and sample size n of the data. In the theorem statement we use the following quantities
d -
2 log (2/6)
SPmin
1
r/2 ti
n
n
d -
2 (2/5)
max
1
r/2
Ese
n2
that arise from concentration bounds for the singular values of the random data matrix. Further, let = Pros and denote by & the cumulative distribution function of standard normal.
v2
Theorem 3.1. Assume d - >>>> For test samples from Fr. perturbation set type T as in Equation and any 0 S tu < 5, the following holds for the robustt error of the classifier (Equation resulting from ur-udversarial training:
1. The 6 robustttt error of the robust max-margin estimator reads
-cr)
Err(Oer:
for random quantity @ > 0 depending O,r, Ge and is hence strictly increasing in the adversarial training budget Eir-
2. With probability at least 1 - ō, we further have Pmin S S Pmax and the following lower bound the robust error increase by adversarially training with size Eir
r/2 mmin
(r/2 min (Cn
Err(O E.r(@":ea)) (x)) 2
+b
(6)
Pminn
The proof can be found Appendix A and primarily relies on estimation of singular values of high-dimensional matrices. Note that the theorem holds for any eu < 3 and hence also directly
4

---

Published as a conference paper at ICLR 2023
Original insuts
-Stndaad training
ground Adversarial truch
Perturbed Inputs
robust error
10
d/n = 20.0
d/n 20.0
Mandard
robust
susceptibiity
non-iggnal dimension
d/n 1.25
error
20
20
adversarial budget drr (a) Robust error VS Car
adversarial budget
signal dimension
(b) Robust error decomposition
(c) Intuition in 2D
Figure 4: (a) We set d = 1000 and 12. The robust error as function of the adversarial training budget en for different d/n. (b) The robust error decomposition into susceptibility and standard error as a function of the adversarial budget cn. Full experimental details can be found in Section (c) 2D illustration providing intuition for the linear setting. The effect of adversarial training with directed attacks is captured in the yellow dotted lines: adversarially perturbed training points move closer the true boundary which in turn tilts the decision boundary more heavily in the wrong direction.
applies to the standard error by setting E 0. In Figure 3 empirically confirm the statements of Theorem 3.1 by performing multiple experiments on synthetic datasets as described in Subsection 3.1 with different choices of d/n and 6u. In the first statement, we prove that for small sample-size (n < d - 1) noiseless data, almost surely, the robust error increases monotonically with adversarial training budget ei > O. In Figure 33| we plot the robust error gap between standard and adversarial logistic regression as a function of the adversarial training budget fir for 5 runs.
The second statement establishes simplified lower bound the robust error increase for adversarial training (for a fixed fn Eie) compared to standard training. In Figures [3 and [c] we show how the lower bound closely predicts the robust error gap in our synthetic experiments. Furthermore, by the dependence of (miin on the overparameterization ratio d/n, the lower bound on the robust error gap is amplified for large d/n. Indeed, Figure shows how the error gap increases with d/n both theoretically and experimentally. However, when d/n n increases above a certain threshold, the gap decreases again, as standard training fails to learn the signal and yields a high error (see Figure 3b)
3.3 PROOF INTUITION
The reason that adversarial training hurts robust generalization is based on an extreme robust vs. standard error trade-off. We now provide intuition for the effect of directed attacks and the low sample regime on the ro-rousst max-ty-margha solution by decomposing the robust error Err(#; Ge). Notice that 6g robust error Err(8; Ge) can be written as the probability of the union of two events: the event that the classifier based on wrong and the event that the classifier is susceptible to attacks:
Err(O; (we = Ez.gr [(ufo(x) < 0} V max Iffo(=)fo(z') < <0] (TE
< Err(6; 0) + Susc(O) (ue)
(7)
where Susc(O; Gw) the expectation of the maximization term in Equation 7 Susc(O; Ge) represents the ---uttack-suxcppiiill
of classifier induced by 0 and (f; 0) its standard error. In our linear setting, we can lower bound Equation by Err(f; 0) + 1 Susse(f; G.). Hence, Equation 7 suggests that 2 the robust error can only be small if both the standard error and susceptibility are small. In Figure we plot the decomposition of the robust error in standard error and susceptibility for adversarial logistic regression with increasing fu. We observe that increasing & increases the standard error too drastically compared to the decrease in susceptibility, leading to a drop in robust accuracy. For completeness, in Appendix B we provide upper and lower bounds for the susceptibility score. We now explain why, in the small-sample size regime, adversarial training with directed attacks 3 may increase standard error to the extent that it dominates the decrease in susceptibility
A key observation is that the robust max- max-l2-margin solution of a dataset = {(x,, }: maximizes the minimum margin that reads min (n) y (a; - y& lez). where 01 refers to the i-th entry of vector 0. Therefore, it simply corresponds to the max- -ma-gin solution of the dataset shifted towards the decision boundary D., = ((x; yeen [0] le,, 91) M Using this fact, we obtain a closed-form expression of the (normalized) max-margin solution 4 as a function of fir that reads
1
gv (r (r-26sh 22 + 452
2E,, 27.0
(8)
5

---

Published as a conference paper at ICLR 2023
where ||0||2 and 0 is random quantity associated with the max-ey- solution of the 1 dimensional Gaussian imputs orthogonal to the signal direction (see Lemma A.I in Section A).
In high dimensions, with high probability any two Gaussian random vectors are far apart in our distributional setting, this corresponds to the vectors being far apart in the non-signal directions. In Figure 4c] we illustrate the phenomenon using 2D cartoon, where the few samples in the dataset all far apart in the non-signal direction. We see how shifting the dataset closer to the true decision boundary, may result in max-margin solution (yellow) that aligns much worse with the ground truth (gray), compared to the estimator learned from the original points (blue). Even though the new (robust max- margin)) classifier (yellow) is less susceptible to attacks in the signal dimension, also uses the signal dimension less. Mathematically. this is reflected in the expression of the max max-margin solution in Equation X
We see that the first (signal) dimension is used less as frr increases.
3.4 GENERALITY OF THE RESULTS
In this section we discuss how Theorem 3.1 might generalize to other perturbation sets and models.
Signal direction is known The type of additive perturbations used in Theorem 3.1 defined in Equation 3] is explicitly constrained to the direction of the true signal. This choice is reminiscent of corruptions where every possible perturbation in the set is directly targeted at the object to be recognized, such as motion blur of moving objects. Such corruptions are also studied in the context of domain generalization and adaptation (Schneider et al., 2020).
Directed attacks in general. however, may also consist of perturbation sets that are only strongly biased towards the true signal direction. They may find the true signal direction only when the inner maximization is exact. The following corollary extends Theorem 3.1 small l -pertuurbations
T( e) = {a' = x + 5 || S <c},
for 0 < < 5 that reflect such attacks. We state the corollary here and give the proof in Appendix A
Corollary 3.2. Theorem 3.7 also holds for f with perturbation sets defined in g
The proof uses the fact that the inner maximization effectively results in a sparse perturbation
equivalent to the attack resulting from the perturbation set defined in Equation] 3
Other models Motivated by the implicit bias results of (stochastic) gradient descent on the logistic loss, Theorem 3.1 is proven for the max-fg-margin solution. We would like to conjecture that for the data distribution in Section 3 adversarial training can hurt robust generalization also for other models with zero training error (interpolators in short). For example, Adaboost is a widely used algorithm that converges to the max-é1 margin classifier (Telgarsky| 2013). One might argue that for a sparse ground truth, the max-é1 -margin classifier should (at least in the noiseless case) have the right inductive bias to alleviate large bias in high dimensions. Hence, in many cases the (sparse) max-f; max-e--margin solution might align with the ground truth for given dataset. However, we conjecture that even in this case, the robust max- -margin solution would be misled to choose a wrong sparse solution. This can be seen with the help of the cartoon illustration in Figure 4c
4 REAL-WORLD EXPERIMENTS
In this section, we demonstrate that the proof intuition of the linear case may generalize to more complex models. Specifically, the insights from Section 3 helped to identify realistic directed attacks on standard image datasets for which adversarial training hurts robust accuracy in the low sample regime. In what follows, we present experimental results for corruption attacks the Waterbirds dataset. Due to space constraints, results on the mask attacks on CIFAR-10 can be found in Appendix [ The corresponding experimental details and more results on other additional image datasets (such as the hand gestures dataset) can be found in Appendices D and
4.1 DATASETS AND MODELS
We consider three datasets: the Waterbirds dataset, CIFAR-10 10 and a hand gesture datasets. Due to space constraints, we describe CIFAR-10 and the hand gesture dataset in Appendix [a Apart from CIFAR-10 and the hand gesture dataset, we build a new version of the Waterbirds dataset, consisting of images of water- and landbirds of size 256 x 256 and labels that distinguish the two

---

Published as a conference paper at ICLR 2023
30
robust
standard
X
suic
30-
sta ndard training
30-
robust error
adversarial training
255
roluust
error
20-
200
20
20
71
30
3.1
0.1
0.2
0.3
0.3
0.0
0.2
0.4
250
500
750
1000
advorsarial budget Ct
adversarial budget cir
number of samples
(a) Robust error with increasing tur
(b) Robust error decomposition
(c) Robust error vs. #samples
Figure 5: Experiments on the Waterbirds dataset considering the adversarial illumination attack with 6. 0.3. We plot the mean and standard deviation of the mean of several independent experiments. (a) The robust error increases with larger fr in the low sample size regime. (b) We
20 and plot the robust error decomposition as in Equation / with increasing fg- While the susceptibility decreases slightly, the increase in standard error is much more severe, resulting in an increase in robust error. (c) The robust error of standard training and adversarial training as a function of the number of samples, where the smallest sample size still yields small (< 10%) standard test error for standard training. While adversarial training hurts for small sample sizes, helps for larger sample sizes. For more experimental details see App.
types of birds. Using code provided by Sagawa et al. 2020). we construct the dataset as follows: First, we sample equally many water- and landbirds from the CUB-200 dataset (Welinder al.) 2010). Then, we segment the birds and paste them onto a background image that is randomly sampled (without replacement) from the Places-256 dataset (Zhou et al., 2017). Also, following the choice of Sagawa et al. (2020), we use as models a ResNet50 and a ResNet18 8 that were both pretrained on ImageNet and achieve near perfect standard accuracy. In Appendix D this section by reporting the results of similar experiments with different architectures.
we complement the results of
4.2 IMPLEMENTATION OF THE DIRECTED ATTACKS
In this section, we consider two attacks on the Waterbirds dataset: motion blur and adversarial illumination depicted in Figure 2 In Appendix D we also discuss the mask attack, which should mimic occlusions of objects in images that are physically realizable (Eykholt et al., 2018 Wu et al. 2020). On the other hand, motion blur may arise naturally when photographing fast moving objects with a slow shutter speed. Lastly, adversarial illumination may result from adversarial lighting conditions. Next, describe the motion blur and adversarial illumination attacks in more detail.
Motion blur For the Waterbirds dataset we can implement motion blur attacks the object (the bird) specifically, a natural corruption that could occur if birds move at speeds that are faster than the shutter speed. The aim is to be robust against all motion blur severity levels up to Mmaz 15. To simulate motion blur, we apply a motion blur filter with a kernel of size M on the segmented bird before we paste it onto the background image. We can change the severity level of the motion blur by increasing the kernel size of the filter. See Appendix for concrete expressions of the motion blur kernel. Intuitively the worst attack should be the most severe blur, rendering a search over range of severity superfluus. However, similar to rotations, this is not necessarily true in practice since the training loss on neural networks generally nonconvex. Therefore, for an exact evaluation of the robust error at test time, we perform a full grid search over all kernel sizes in [1, 2....... M We refer to Figure 2d and Section D for an illustration of our motion blur attack. During training time, we perform approximate search over kernels with sizes 2i for i I,..., Mmex/2.
Adversarial illumination As second attack on the Waterbirds dataset, we consider adversarial illumination. The adversary can darken or brighten the bird without corrupting the background of the image. The attack aims to model images where the object at interest is hidden in shadows or placed against bright light. To compute the adversarial illumination attack, we modify the brightness of the segmented bird by adding a constant a E [-Gee, Gie] to all pixel values, before pasting the bird onto the background image. With an analogous argument as for the adversarial search for motion blur, the exact evaluation requires an actual search over the interval Gee fie] We find the most adversarial lighting level, i.e. the value of a, by equidistantly partitioning the interval [-Gre, fie] in K steps and performing full list-search over all steps. See Figure 2c and Appendix ( illustration of the adversarial illumination attack. We choose K = 65, 33 during test and training time respectively.
Adversarial training For all datasets and attacks, we run SGD until convergence on the robust cross-entropy loss 2 In each iteration, we search for an adversarial example as described above and
7

---

Published as a conference paper at ICLR 2023
standard
robust
standard training adiverserial training
robust efror
45-
10
20
error
robust
40-
35
20
15
20
50
150
adversarial budget
adversarial budget Cir
number of samples
(a) Robust error with increasing Cu (b) Robust error decomposition
(c) Robust error vs. #samples
Figure 6: Experiments on the (subsampled) Waterbirds dataset using the motion blur attack. (a) Even though
adversarial training hurts robust generalization for low sample size (n = 20), it helps for n = 50. (b) For n = 20, the decomposition of the robust crror in standard error and susceptibility as function of adversarial budget ex- The increase in standard error is more severe than the drop in susceptibility, leading increase in robust error. (c) The robust error of standard and adversarial training settings where the test error after standard training is small as function of the number of samples. While adversarial training hurts for small sample sizes, it helps for larger sample sizes, For each experiment we plot the mean and standard deviation of the mean of independent experiments. For more experimental details see App. D
slight
update the weights using a gradient with respect to the resulting perturbed example (Goodfellow al. 2015 Madry al. 2018). For every experiment, we choose the learning rate and weight decay parameters that minimize the robust error on a hold-out dataset.
4.3 ADVERSARIAL TRAINING CAN HURT ROBUST GENERALIZATION
We now present our experimental results on the Waterbirds dataset for both motion blur and adversarial illumination attacks. First of all, Figure 5a and show that the phenomenon characterized in the linear setting by Theorem 3.1 also occurs for directed attacks the Waterbirds dataset: as increase the adversarial training budget 6r starting from zero (standard training), the robust error monotonically increases.
Furthermore, to gain intuition as described in Section 3.3 we also plot the robust error decomposition (Equation 7 consisting of the standard error and susceptibility in Figure 5bb and 6bb Recall that we measure susceptibility as the fraction of data points in the test set for which the classifier predicts a different class under an adversarial attack. As in our linear example, we observe an increase in robust eтror despite a slight drop in susceptibility, because of the more severe increase in standard error. Moreover, Figures| I and 6cd shos show that analogous to our linear example, this phenomenon specific to the low sample regime: for large sample size adversarial training outperforms standard training as expected. Note again that even the smallest sample size is large enough to yield a standard test error < 10% for standard training. Similar experiments for CIFAR-10 can be found in Appendix E Finally, we empirically confirm Appendix that our phenomenon specific to directed attacks: for undirected attacks such as bounded lo and lg ball perturbations, adversarial training helps robust generalization also in the low sample size regime.
4.4 DISCUSSION
We now discuss how different algorithmic choices, motivated by related work, might affect how adversarial training hurts robust generalization.
Catastrophic overfitting Often the worst-case perturbation during adversarial training is found using an approximate algorithm such as SGD. It is common belief that using the strongest attack (in the motion blur case, full grid search) during training also results in better robust generalization. In particular, the literature on catastrophic overfitting shows that weaker attacks during training lead to bad performance on stronger attacks during testing (Wong et al. 2020 Andriushhhenko & Flammarion] 2020; Li et al. 2021). Our results suggests the opposite in the low sample size regime for directed attacks: the weaker the attack during training, the better adversarial training performs.
Robust overfitting Recent work observes empirically (Rice et al., 2020) and theoretically (Sanyal et al. 2020 Donhauser et al., (022), that perfectly minimizing the adversarial loss during training might in fact be suboptimal for robust generalization; that is, classical regularization techniques might lead to higher robust accuracy. This phenomenon is often referred to as robust overfitting. May the phenomenon be mitigated using standard regularization techniques? In Appendix D we shed light
8

---

Published as a conference paper at ICLR 2023
on this question and show that adversarial training hurts robust generalization when standard regularization methods such as early stopping are used.
5 RELATED WORK
Robust and non-robust useful features In the words of Ilyas et al. (2019) and Springer et al. (202T) we can describe the intuition behind "our phenomenon" as follows: for directed attacks, all robust features become less useful. but adversarial training uses robust features more. In the small sample-size regime, n < 1 in particular, robust learning assigns too much weight on the robust (possibly non-useful) features that then dominate the non-robust (but useful)featuree Even though they define these concepts, they don't make our statement, but show that adversarial training reduces the reliance on non-robust but possibly useful features.
Small sample size and robustness A direct consequence of Theorem 3.1 is that in order to achieve the same robust errоr as standard training, adversarial training requires more samples. This statement might remind the reader of sample complexity results for robust generalization in Schhmidt et al. (2018); : Yin et al. (2019); Khim & Loh (2018). While those results compare sample complexity bounds for standard vs. robust error, our theorem statement compares two algorithms, standard vs. adversarial training, with respect to the robust error.
Trade-off between standard and robust error Many papers observed that though adversarial training decreases robust eπrоr compared to standard training, it may lead to an increase in standard test error Madry et al. (2018) 8 Zhang al. (2019). For example, Tsipras et (2019); Zhang et al.]
(2019): Javanmard et a (220)) Dobriban et al. (2020); Chen et al. (2020) study settings where the Bayes optimal robust classifier is not equal to the Bayes optimal (standard) classifier (i.e. the perturbations are inconsistent or the dataset is non-separable). [Raghunathan et al. (2020) study consistent perturbations, as our paper, and prove that for small sample size, fitting adversarial examples can increase standard error even in the absence of noise. Empirically. Dong et al. (202T); Mendonça et al. (2022) show that for ap-attacks low-quality data might be the main cause of the trade-off. While atorementioned works focus on the decrease in standard error, prove that for directed attacks, in the small sample regime adversarial training may in fact increase robust error.
Mitigation of the trade-off A long line of work has proposed procedures to mitigate the trade-off between robust and standard accuracy. For example Alayrac et al. (2019); Carmon et al. (2019) Zhai et al.] (2019); Raghunathan et al. (2020) study robust self training, which leverages a large set of unlabelled data, Lee al. (2020); Lamb et al. (2019); Xu et al. (2020) use data augmentation by interpolation. Ding et al. (2020); Balaji et al. (2019): Cheng et al. (2020) on the other hand propose to adaptive perturbation budgets Gr that vary across inputs. The intuition behind our theoretical analysis suggests that the standard mitigation procedures for imperceptible perturbations may not work for perceptible directed attacks, because all relevant features are non-robust, We leave a thorough empirical study as interesting future work.
6 SUMMARY AND FUTURE WORK
This paper aims to caution the practitioner against blindly following current widespread practices to increase the robust performance of machine learning models. Specifically, adversarial training is currently recognized to be one of the most effective defense mechanisms for e fp-perturbations, significantly outperforming robust performance of standard training. However, we prove that in the low sample size regime this common wisdom is not applicable for consistent directed attacks, which efficiently focus their attack budget to target the ground truth class information. In terms of follow-up work on directed attacks in the low sample regime, there are some concrete questions that would be interesting to explore. For example, as discussed in Section it would be useful to test whether some methods to mitigate the standard accuracy vs. robustness trade-off would also relieve the perils of adversarial training for directed attacks. Further, we hypothesize that when few samples are available, one should avoid training with attacks that may heavily reduce class information independently of the attacks at test time. If this hypothesis were confirmed, it would break with yet another general rule that the best defense perturbation type should always match the attack during evaluation.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
Supported by the Hasler Foundation grant number 21050.
REFERENCES
Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances in Newral Information Processing Systems, pp. 12214- 12242-, 2019.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 2020.
Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. In Zhi-Hua Zhou (ed.), The 30th International Joint Conference on Artificial Intelligence, pp. 4312 4321. International Joint Conferences on Artificial Intelligence Organization, 2021.
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXiv: 190.0800,, 2019.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John Duchi. Unlabeled data improves adversarial robustness. In The 33rd International Conference on Neural Information Processing Systems, pp. 11 11192-11200 1203, 2019.
Lin Chen, Yifci Min, Mingrui Zhang, and Amin Karbasi. More data can expand the generalization gap between adversarially robust and standard models. In The 36th International Conference Machine Learning, pp. 1670-1680, 2020.
Minhao Cheng, Qi Lei, Pin- Yu Chen, Inderjit Dhillon, and Cho-Jui Hsich. Cat Customized adversarial training for improved robustness. arXiv preprint arXiv:2002.067880 2020.
Lénaïc Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In The 7th International Conference on Learning Theory, pp-
1305-1338, 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fi attacks. The 37th International Conference on Machine Learning, pp. 2206-2216, 2020.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In The 8th International Conference on Learning Representations, 2020.
Edgar Dobriban, Hamed Hassani. David Hong, and Alexander Robey. Provable tradeoff's in adversar- ially robust classification. arXiv preprint #XXv:2006.0516/ 2020.
Chengyu Dong, Liyuan Liu, and Jingbo Shang. Data quality matters for adversarial training: An empirical study, 2021.
Konstantin Donhauser, Alexandru Tifrea, Michael Aerni, Reinhard Heckel, and Fanny Yang. Interpo- lation hurt robust generalization when there is noise. The 36th conference on Advances in Neural Information Processing Systems, 2021.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry, Exploring the landscape of spatial robustness. In The 36th International Conference Machine Learning, pp. 1802-1811, 2019.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical physical-world attacks deep learning visual classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1625-1634, 2018.
10

---

Published as a conference paper at ICLR 2023
Amin Ghiasi, Ali Shafahi, and Tom Goldstein. Breaking certified defenses: semantic adversarial examples with spoofed robustness certificates. In The 6th International Conference on Learning Representations, 2019.
Justin Gilmer, Ryan Adams, Ian Goodfellow, David Andersen, and George E Dahl. Motivating the 2018.
rules of the game for adversarial example research. arXiv preprint
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In The 3th International Conference on Learning Representations, pp. 1-10, 2015.
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In The 33rd conference on Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Adel Javanmard, Mahdi Soltanolkotabi, and Hamed Hassani. Precise tradeoffs in adversarial training for linear regression. In Conference on Learning Theory, pp. 2034-2078, 2020.
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In The 32nd Conference on Learning Theory, pp. 1772-1798, 2019.
Daniel Kang, Yi Sun, Tom Brown, Dan Hendryeks, and Jacob Steinhardt. Transfer of adversarial robustness between perturbation types. arXiv e-prints, pp. arXiv-1905, 2019.
Justin Khim and Po-Ling Loh. Adversarial risk bounds via function transformation. arXiv preprint arXiv:1810.09519, 2018
Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In The 9th International Conference on Learning Representation, 2021.
Alex Lamb, Vikas Verma, Juho Kannala, and Yoshua Bengio. Interpolated adversarial training: Achieving robust neural networks without sacrificing too much accuracy. In The 12th ACM Workshop on Artificial Intelligence and Security, pp. 95-103, 2019.
Saehyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adversari- ally robust generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Bai Li, Shiqi Wang, Suman Jana, and Lawrence Carin. Towards understanding fast adversarial training. arXiv preprint arXiv:2006.03083 03089, 2021.
Yan Li, Ethan X.Fang, Huan Xu, and Tuo Zhao. Implicit bias of gradient descent based adversarial training on separable data. In The 8th International Conference on Learning Representations, 2020.
Wei-An Lin, Chun Pong Lau, Alexander Levine, Rama Chellappa, and Soheil Feizi. Dual manifold adversarial robustness: Defense against Ip and non-lp adversarial attacks. In The 34th conference on Advances in Neural Information Processing Systems, pp. 3487-3498, 2020.
Chen Liu, Mathieu Salzmano, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. The 35th conference on Advances in Neural Information Processing Systems, pp.
2020.
Bo Luo, Yannan Liu, Lingxiao Wei, and Qiang Xu. Towards imperceptible and robust adversarial example attacks against neural networks. In The 32nd AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In The 6th International Conference on Learning Representations, 2018.
Tomás Mantecón, Carlos R. del Blanco, Fernando Jaureguizar, and Narciso García. A real-time gesture recognition system using near-infrared imagery. PLOS ONE, pp. 1-17, 2019.
11

---

Published as a conference paper at ICLR 2023
Marcele OK Mendonça, Javier Maroto, Pascal Frossard, and Paulo SR Diniz. Adversarial training with informed data selection. In The 30th European Signal Processing Conference (EUSIPCO). pp. 608-612, 2022.
Seyed-Mohsen Moosavi-Dezfoolle Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In The IEEE conference computer vision and paltern recognition (CVPR), pp. 2574-2582, 2016.
Abdullah Mujahid, Mazhar Javed Awan, Awais Yasin, Mazin Abed Mohammed, Robertas Damaševičius, Rytis Maskeliūnas, and Karrar Hameed Abdulkareem. Real-time hand gesture recognition based on deep learning yolov3 model. Applied Sciences, 2021.
Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry, Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3051-3059, 2019.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain general- ization in deep learning. In The 33d conference Advances in Neural Information Processing Systems, pp. 1161 1-111111222 2019.
Munir Oudah, Ali Al-Naji, and Javaan Chahl. Hand gesture recognition based computer vision: A review techniques. Journal of Imaging, 2020.
Huy Phan. hugymphanfpytorccccffaatu 1 2021.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In The 37th International Conference Machine Learning, pp. 7909-7911 2020.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In The 37th International Conference on Machine Learning, pp. 8093-8104, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In The 7th International Conference on Learning Representations, 2020.
Amartya Sanyal, Puneet K Dokania, Varun Kanade, and Philip Tor. How benign is benign overfitting? In The 8th International Conference on Learning Representations, 2020.
I Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. In The 32nd conference Advances Neural Information Processing Systems, 2018.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In The 34th conference on Advances in Neural Information Processing Systems, pp. 11539-112 1551, 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, pp. 1-57, 2018.
Jacob M Springer, Melanie Mitchell, and Garrett T Kenyon. Adversarial perturbations are not so weird: Entanglement of robust and non-robust features in neural network classifiers. arXiv preprint arXiv:2102.05110, 2021.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and general- ization. In The IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp- 6967-6987, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In The 2nd International Conference on Learning Representations, 2014.
Matus Telgarsky. Margins, shrinkage, and boosting. In The 30th International Conference on Machine Learning, pp. 307-315, 2013.
12

---

Published as a conference paper at ICLR 2023
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumer, and Aleksander Madry. Robustness may be at odds with accuracy. In The 7th International Conference on Learning Representations, 2019.
Roman Vershynin. Introduction the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.302) 2010.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical report, California Institute of Technology, 2010.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. In The 8th International Conference on Learning Representations, 2020.
Tong Wu, Liang Tong, and Yevgeniy Vorobeychik. Defending against physically realizable attacks on image classification. In The 8th International Conference on Learning Representations, 2020.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adver- sarial domain adaptation with domain mixup. In The AAAI Conference on Artificial Intelligence, pp. 6502-6509, 2020.
Shuai Yang, Prashan Premaratne, and Peter Vial. Hand gesture recognition: An overview. In The 5th IEEE International Conference on Broadband Network Multimedia Technology, pp. 63-69, 2013.
Dong Yin, Ramchandran Kannan, and Peter Bartlett. Rademacher complexity for adversarially robust generalization. In The 36th International conference on machine learning, pp. 7085-7094, 2019.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hoperoft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint arXiv: 1956.00555, 2019.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In The 36th International Conference Machine Learning, pp. 7472-7482, 2019.
Zhengyu Zhao, Zhuoran Liu, and Martha Larson. Towards large yet imperceptible adversarial image perturbations with perceptual color distance. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1039-1048, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2017.
Jianli Zhou, Chao Liang, and Jun Chen. Manifold projection for adversarial defense on face recognition, In The 16th European Conference Computer Vision, pp. 288 305,305,000 2020.
13

---

Published as a conference paper at ICLR 2023
A THEORETICAL STATEMENTS FOR THE LINEAR MODEL
Before we present the proof of the theorem, we introduce two lemmas are of separate interest that are used throughout the proof of Theorem 1. Recall that the definition of the (standard normal- ized) maximum-l2-man solution (max-margin solution in short) of a dataset D = ((I,,)))) corresponds to
arg max min y:0"
Tits
(10)
|10|a.51 ie(n)
by simply setting frr = 0 in Equation 4 for dataset D {(r,, y:) we refer the induced dataset D as the dataset with covariate vectors stripped of the first element, i.e.
The by-margin of then reads min 9:0 T₁. Furthermore
D= (() = {((x)) 8:)
(1I)
where (24) refers the last d elements of the vector z4. Furthermore, remember that for any vector [(}] refers to the j-th element of and 6j denotes the j-th canonical basis vector. Further, recall the distribution IP, as defined in Section 3.1 the label E {+1, -1) is drawn with equal probability and the covariate vector is sampled as x 2 ø] where
IR-1 is a random vector drawn from standard normal distribution, i.e. I NN (a-1). We generally allow r, used to sample the training data, to differ from Tlest. which is used during test time.
The following lemma derives a closed-form expression for the normalized max-man margin solution for any dataset with fixed separation r in the signal component, and that is linearly separable in the last
- 1 coordinates with margin 7.
Lemma A.1. Let D = ((rs, y:) be a dataset that consists of points (x, y) E Rd x {+1} and Tn 5. i.e. the covariates x; are deterministic in their first coordinate given ys with separation distance r. Furthermore, let the induced dataset D also be linearly separable by the normalized max-ly-margin solution with an ly-margin 7. Then, the normalized max-margin solution of the original dataset D given by
0
r. 298
(12)
45
Further, the standard accuracy of 0 for data drawn from rwe reads
Yen (Y X 0) = +
Tiest
(13)
40 7
The proof can be found in Section A.3 The next lemma provides high probability upper and lower bounds for the margin ff of D when ā; are drawn from the normal distribution.
Lemma A.2. Let D = {(T,, )= be random dataset where yi E {+1} are equally distributed and oq al- for all and &y is the maximum E2 margin that can be written
max min y: T.--
[10 (E(m)
Then, for any Z with probability greater than 2e- 24, we have Tmin (t) S 5 S Amax (1) where
Fmax (t) = (v
t
(V
d Vn
t
7min (t)
Vn
A. PROOF OF THEOREM
Given a dataset { (I, w)) drawn from Pr. easy to see that the (normalized) er-robust max-margin solution 4 of D with respect to hignal-atacking perturbations T's:; 2;) as defined in Equation can be written
min arg ((
v x
arg max min
y:0 (x: + Be1)
[8]|.S1 xe(m),
arg aarg max min Tx yicu sign (ign()))))1).
(z,
|1 se(n)
14

---

Published as a conference paper at ICLR 2023
Note that by definition, it is equivalent to the (standard normalized) max-margin solution of the shifted dataset c., = {zz fu sign(f e1, (4) ()h== Since De, satisfies the assumptions of Lemma A.I it then follows directly that the normalized e, rorossst max-margin solution reads
g
r
2c 27
(14)
V(r
2ea 47
by replacing r by r 26g in Equation
12 Similar to above, E Rd is the (standard normalized) max-mnrginn solution of { y.))==1 and f the corresponding margin.
Proof 1. We can now compute the eg-robust accuracy the en-robsst max margin estimator 0 for a given dataset D as a function 7. Note that in the expression of 0ev, all values are fixed for a fixed dataset, while 0 S Eir S r 2max can be chosen. First note that for distribution IP, the fo-rooust accuracy, defined as one minus the robust error (Equation I for classifier associated with vector 0, can be written
Acc(O: (w) ) Ex T{ min YoTx x' >0] r'E((X
(15)
ExYyP, [I(YO'X X 60 > 0}] = Ex,Y= [MAY (X YY-Yes sign(0 0}]
Now, recall that by Equation 14 and the assumption in the theorem, we have 2cp > 0, so that sign(0#") 1. Further, using the definition of the T(mr in Equation 3 and by definition of the distribution P,, we have X(J] = y Plugging into Equation 15 then yields
Acc(O :ew) ew)
= EX
1(y0 "(X - Y(we))
(X-Yewel) > >0}]
T( (X-1 + Y (5
Ge e2) > 0]]
= =Exyy~
2
PP (Y0 0)
where X-1 shorthand for the random vector X-1 (0; 12) X(a). The assumptions in Lemma A. (D., is linearly separable) are satisfied whenever the n < d samples are distinct, i.e.
with probability one. Hence applying Lemma A.I with Prest = 22 and r 26g yields
r((r 2(xe)
26
Acc(f (ue) Φ
Ca
(16)
40
207
Theorem statement a) then follows by noting that db is monotically decreasing function in The expression for the robust error then follows by noting that (--2) (22) for any and defining
oΩ
(17)
r/2
tsc
Proof of 2. First define Pmins Pmax using Fumin Fmax as in Equation 17 tion 16
Then we have by Equa-
Err( @0 ew) Err(#" (w) Acc(0): ( :ww)
Acc(O (se)) Ewe)
r/2
r/2
①b
①
T//2
30 dx
r/2-ca v
12TT
By plugging in t =
2lgg2/6 2/8
in Lemma A.2
we obtain that with probability at least 1 - we have
log log (2/5)
d-1 +
(1 1+
log(2/5)
Juin ==
1+
< s o
=: 7max
n
and equivalently Pmin E Pmaxx
15

---

Published as a conference paper at ICLR 2023
Now note the general fact that for all @ I 22r the density function f((:z) monotonically increasing in g
E p2 is
(r/ /2- - tse) so that f((;;)) 2 (1Pmin x)) for all By assumption the theorem, v S v2(r/2 (2)r/2-en)(r/2 E [r/2 T/2] and therefore
1
///
1
$7 dx 1
r/2
- (r/2 Etr
(r
E
dx
E
1/2-c0 V
8/2--ce VTT
(miin
Gmin
and the statement is proved.
A.2 PROOF OF COROLLARY
We now show that Theorem similar steps as in Equation 14, the en-robust max-margin solution for pi-petuuraations can be written
3. also holds for l -ball perturbations with at most radius Following
arg max min y (zz, yı sign (00) 101 (E(x)
(18)
where (0) arg max, 0,1 is the index of the maximum absolute value of 6. We now prove by contradiction that the robust max- solution for this perturbation set is equivalent to the solution 14 for the perturbation set We start by assuming that 0+ does not solve Equation 14 which is equivalent to assuming 1 $ (0*) by definition. We now show how this assumption leads to contradiction.
Define the shorthand j" = j" (0%x) 1. Since drh is the solution of 18 by definition, we have that 8 is also the max-margin solution of the shifted dataset De, = (x; y. +))
Further, note that by the assumption that 1 & j0**,, this dataset De, consists of input vectors Iz 5 2.
giew sign(0) +all+++))eee +1)e Hence via Lemma A.1 0 can be written as ge
) [r, 27.00u))
(19)
where jer is the normalized max-margin solution of D = (āt yiew sign000)))) Mc).
We now characterize 8 Note that by assumption, j = j (B = arg max, 07/1- Hence, the normalized max-margin solution 8 is the solution
e arg max min o: T; cir 0.10
(20)
pell_s1 sE(n]
Observe that the minimum margin this estimator 7 = min y (0 Tā; G decreases
with En as the problem becomes harder 8 S 5, where the latter equivalent to the margin of 0= for En = 0. Since r > 25max by assumption in the Theorem, by Lemma A.2 with probability at least
1 - 2E
we then have that r > 25 > 25 Given the closed form of 0h in Equation 19 it directly follows that 0j
(a)11)
25 - [0 |l and hence 1 E j* (0+).. This contradicts the original assumption 1 j (@rx) and hence established that 0 for the & perturbation set has the same closed form 14 as for the perturbation set 3
The final statement is proved by using the analogous steps as in the proof of 1. and 2. obtain the closed form of the robust accuracy of dtu
A.3 PROOF LEMMA A.I
We start by proving that 0 is of the form
a1 ag0
(21)
for a, 02 > 0. Denote by H() the plane through the origin with normal e. We define d ((x, y), H(0) the signed euclidean distance from the point (Ir D P, to the plane H(0). The signed
16

---

Published as a conference paper at ICLR 2023
euclidean distance is the defined as the euclidean distance from x to the plane if the point (z,y) is correctly predicted by 8. and the negative euclidean distance from I to the plane otherwise. We rewrite the definition of the max 12- la-margin classifier. It is the classifier induced by the normalized vector 0, such that
max min d ((2, v), H(()) = min ((Z, PERA (a-))ED
(æ.a)e
We that D is deterministic in its first coordinate and get
max min d ((z, v). H(()) max min (a.9)ED
111111111 + +0
(aNjjED
max 01
min yo J. (a.D))
Because r > 0, the maximum over all has I1] 2 0. Take any a > such that ||0|2 = a. By definition the max l2-margin classifier, 0, maximizes min d ((x, y). H())). Therefore, 0 is of the form of Equation 21
Note that all classifiers induced by vectors of the form of Equation 21 classify D correctly. Next, we aim find expressions for a₁ and 02 such that Equation 21 is the normalized max l2-mangin classifier. The distance from any x E to H())
d (())
|i az0 _ū.
Using that 2|1] =y and that the second term equals and
(z, (z, p(())) we get
d (x, (z))) |0 r + and (s, x(()) 01
r + v1 afd (z., H(())
(22)
2
Let (ū,y) E be the point closest Euclidean distance to Ō. This point also the closest point in Euclidean distance to (0), because by Equation 22 d xz, H())) is strictly decreasing for decreasing
x. H(()). We maximize the minimum margin d (z, H(()) with respect a1. Define the vectors
a [a1, ag] and v = [ d x. {. We find using the dual norm that
1 2
Plugging the expression of into Equation
21 yields that is given by
1
0
r. 2,,70
Vr2 + 452
For the second part of the lemma first decompose
P. Prem (YO
x 0)
_ p2
0 xx>0l1 =1
-
1
0 T 0'x00y==-1
We can further write
P Past
0
X 0
= -P Pix
X1 -011 Xp1 Y =
(23)
- Trest
PP
27
X1)
Y=
Tiaa
Tisst
Fiest
40
40
where & is the cumulative distribution function. The second equality follows by multiplying by the normalization constant on both sides and the third equality is due to the fact that Σ 0 X is
17

---

Published as a conference paper at ICLR 2023
zero-mean Gaussian with variance o²
[|0 (
o² since i is normalized. Correspondingly
write
2 d-1
X
Y = -1 = IP Pree
(11 < <- (- (-Peest
Plest
Y = -1 =
Festt
Yioi
2
4
(24)
that combine 23 and 23 and 24 to obtain Price (YO x = the proof of the lemma.
Fiee 407
This concludes
A.4 PROOF OF LEMMA
The proof plan is as follows. We start from the definition of the max m--margin of dataset. Then, we rewrite the max fg-margin as an expression that includes a random matrix with independent standard normal entries. This allows to prove the upper and lower bounds for the max-l2-marge in Sections I and A.4.2 respectively, using non-asymptotic estimates on the singular values of Gaussian random matrices.
Given the dataset Đ = { Hi we define the random matrix
X
(25)
where ū; ~ (O0 ola-).. Let V be the class of all perfect predictors of D. For matrix A and vector b also denote by |Ab] the vector whose entries correspond to the absolute values of the entries of Ab. Then, by definition
min [elg
min o [Qul
(26)
vEV.Iv||:=11Elo
vev.
where Q = 1 X is the scaled data matrix.
In the sequel we will use the operator norm of matrix A E gaxd-1
||A||2
sup HERM- 1|||||2==
||ul|
and denote the maximum singular value of a matrix as 5max(A) and the minimum singular value as 8min (A).
A.4.1
UPPER BOUND
Given the maximality the operator norm and since the minimum entry of the vector [Qul must be smaller than ||Q|aa we can upper bound y by
R sso
1Q12
Taking the expectation both sides with respect the draw of D and noting [|Q]|2 S Smax (Q), it follows from Corollary 5.35 of Vershynin (2010) that for all t 2 0:
P Vd 1+ vn + Smax (0) - 2c
Therefore, with probability greater than 1 - -2
t /
1+
Vn
18

---

Published as a conference paper at ICLR 2023
A.4.2 LOWER BOUND
By the definition in Equation 26 if find vector v E V with ||ol2 1 such that for an holds that milljen of 1| a, then 7 >
Recall the definition of the max-l2-margin as in Equation 25 As < d - 1. the random matrix Q is wide matrix, i.c. there are more columns than rows and therefore the minimal singular value Furthermore, Q has rank n almost surely and hence for all c > O, there exists a Rd-1 such that
oQu= = [n3C O,
(27)
where [mn) denotes the all ones vector of dimension n. The smallest non-zero singular value of Q, 8min, nonzero (Q), equals the smallest non-zero singular value of its transpose Q Therefore, there also exists V with ||l||2 such that
7
min of oQuln 2 d monnnorserrs jE[m]
(Q)
(28)
Vn'
where we used the fact that any vector in the span of non-zero eigenvectors satisfies ||Qul2 2 2 Smin, nonzeros (Q) and the existence a solution v for any right-hand side as in Equation 27 Taking the expectation on both sides, Corollary 5.35 of Vershynin (2010) yields that with a probability greater
22 than 2e x 2 O we have
Vd ( Vn
720
(29)
B BOUNDS ON THE SUSCEPTIBILITY SCORE
In Theorem 3.1 we give non-asymptotic bounds on the robust and standard error of linear classifier trained with adversarial logistic regression. Moreover, use the robust error decomposition in susceptibility and standard error to gain intuition about how adversarial training may hurt robust generalization. In this section, we complete the result of 1.11rrem.110 by also deriving non-asympiotic bounds on the susceptibility score of the max fg-margin classifier.
Using the results in Appendix [A] we can prove following Corollary bounds on the susceptibility score.
B.1 which gives asymptotic
Corollary B.1. Assume d 1 n. For the 6 su-susceptibiliity on test samples from T. with 26 <
and perturbation sets in Equation equation 3 and equation 9 the following holds:
For Crr < Tmmaso with probability at least 1 2E 2dd211
for any 0 < < 1, over the draw of dataset D with samples from Fr, the Ge fe-Susceptibilityy is upper and lower bounded by
(r
(E -1
))
(r (- 21)
Susc(Orr;:ad S
+
27max
27min
(30)
2 (r
(0r 1)) +b (r 20,r) Cxc ))
25min
27 T
We give the proof in Subsection B.1 Observe that the bounds on the susceptibility score in Corollary B.I consist of two terms each, where the second term decreases with but the first term increases. We recognise following two regimes: the max fg-margin classifier is close to the ground truth ej or not. Clearly, the ground truth classifier has zero susceptibility and hence classifiers close to the ground truth also have low susceptibility. close to the ground truth, then putting less weight the first coordinate increases invariance to the perturbations along the first direction. Recall that by Lemma A.I increasing érr, decreases the weight on the first coordinate of the max fa-margin classifier. Furthermore, in the low sample size regime, we are likely not close to the ground truth. Therefore, the regime where the susceptibility decreases with increasing 6, dominates in the low sample size regime.
the other hand, if the max 12-margin classifier is not
To confirm the result of Corollary B.I we plot the mean and standard deviation of the susceptibility score of 5 independent experiments. The results are depicted in Figure We see that for low standard
19

---

Published as a conference paper at ICLR 2023
error, when the classifier is reasonably close the optimal classifier, the susceptibility increases slightly with increasing adversarial budget. However, increasing the adversarial training budget, eu- further, causes the susceptibility score drop greatly. Hence, recognize both regimes and validate that, indeed, the second regime dominates the low sample size setting.
B.1 PROOF OF COROLLARY B.
We proof the statement by bounding the robustness of a linear classifier. Recall that the robustness of a classifier is the probability that a classifier does not change its prediction under an adversarial attack. The susceptibility score then given by
Susc Susc (w) = Rob (w)
(31)
The proof idea is as follows: since the perturbations are along the first basis direction, e1, compute the distance from the robust 12-max margin 0 to point (X, P. Then, note that the robustness of 0 is given by the probability that the distance along e1, from X to the decision plane induced by 0 is greater then 6te- Lastly, we use the non-asymptotic bounds of Lemma A.2
Recall, by Lemma A.1 the max 1a-margin classifier is of the form of
g
26 250
(32)
v(r 26t)² 2 + 472
Let (X, Y) P. The distance along ei from X the decision plane induced by 0en H(0)),, given by
deg (X, )) = X
X
[8]
[c]
fer i=2
Substituting the expression of 0h in Equation 32 yields
dey (X, H((0)))
((1) + 27 (r 6r))
0 X14
Let N be a standard normal distributed random variable. By definition ||0 and using that sum of Gaussian random variables is again a Gaussian random variable, we can write
de, (X, H((0))) = X + 27 ( N
The robustness of th is given by the probability that de, (X, H((Oe)) > Exc. Hence, using that X1 = = with probability 2.
I we get
Gor; (re) =
r + 25
N > 61
+ P
+ 24
N < - Eie
(33)
2
(r
2cx
(r Ea)
standard
robust
susceptibility
40
susceptibiity
error
20
+ experimental
upper and lower bound
2
adversarial budget 6)-
adversarial budget cr
(a) Susceptibility score decreases with éx
(b) Robust error decomposition
We set r deviation over independent experiments. Note how bounds closely predict the susceptibility score. (b) comparison, we also plot the robust emror decomposition in susceptibility and standard error. Even though the susceptibility decreases, the robust error increases with increasing adversarial budget
Figure 7:
1000, n and é (a) The average susceptibility score and standard
20

---

Published as a conference paper at ICLR 2023
We rewrite Equation 33 in the form
2) (r 2
+ P N - (r-2ea) (-Ec 2)
Rob(O";; 6x)
250
24/0
Recall, that N a standard normal distributed random variable and denote by & the cumulative standard normal density. By definition of the cumulative denisity function, we find that
(r
2 (Fw 2)
②b (r 2 (- 6 5)
Rob(O Eie) = +
250
2
Substituting the bounds on y of Lemma A.2 gives us the non-asynnn -on-asympooiic bounds on the robustness score and by Equation (aaaationn]]lss also the susceptibility score.
C EXPERIMENTAL DETAILS THE LINEAR MODEL.
In this section, we provide detailed experimental details to Figures and
We implement adversarial logistic regression using stochastic gradient descent with learning rate of 0.01. Note that logistic regression converges logarithmically to the robust max Iz-margin solution. As consequence of the slow convergence, train for up to 10 epochs. Both during training and test time we solve ((aze)) L(fo(()))) exactly. Hence, we exactly measure the robust error. Unless specified otherwise, set 1, 12 and tre 4.
Experimental details on Figure 3
(a) We draw 5 datasets with n = 50 samples and input dimen- sion d 1000 from the distribution P. We then run adversarial logistic regression on all datasets with adversarial training budgets,
= 1 to 5. To compute the resulting robust error gap of all the obtained classifiers, we use test set of size 10 Lastly, we compute the lower bound given in part 2. of Theorem 3.1 (b) We draw 5 datasets with different sizes between 50 and 104, We take an input dimension of 10 and plot the mean and standard deviation of the robust error after adversarial and standard logistic regression over the 5 samples. We again draw datasets for each d/n constellation and compute the robust error gap for each dataset.
Experimental details Figure 4 adversarial training budget (ew) from to 5. For every constellation of n and ftr, we draw 10 datasets and show the average and standard deviation of the resulting robust errors. In (b), we
For both (a) and (b) d 1000, éte 4, and vary the
50.
D EXPERIMENTAL DETAILS ON THE WATERBIRDS DATASET
In this section, we discuss the experimental details and construction of the Waterbirds dataset in more detail. We also provide ablation studies of attack parameters such as the size of the motion blur kernel, plots of the robust error decomposition with increasing and some experiments using early stopping.
D. THE WATERBIRDS DATASET
To build the Waterbirds dataset, we use the CUB-200 dataset Welinder et (2010), which contains images and labels of 200 bird species, and 4 background classes (forest, jungle/bamboo, water ocean, water lake natural) of the Places dataset Zhou et al. (2017). (201)).The The aim is to recognize whether not the bird, in given image, is waterbird (e.g. an albatros) or a landbird (e.g. woodpecker). To create the dataset, we randomly sample equally many water- as landbirds from the CUB-200 dataset. Thereafter, we sample for each bird image random background image. Then, use the segmentation provided in the CUB-200 dataset to segment the birds from their original images and paste them onto the randomly sampled backgrounds. The resulting images have size of 256 x 256. Moreover, resize the segmentations such that we have the correct segmentation profiles of the birds in the new dataset well. For the concrete implementation, use the code provided by Sagawa al. (2020).
also
21

---

Published as a conference paper at ICLR 2023
(a) Original
(b) M 5
(c) M 10
(d) M 15
(e) M 20
Figure 8: An ablation study of the motion blur kernel size, which corresponds to the severity level of the blur. For increasing M, the severity of the motion blur increases. In particular, note that for M 15 and even M 20, the bird remains recognizable: do not semantically change the class, i.e. the perturbations consistent.
D.2 EXPERIMETAL TRAINING DETAILS
Following the example of Sagawa et al. (020), we use ResNet50 or ResNet18 pretrained on the ImageNet dataset for all experiments the main text, a weight-decay of 10-4, and train for 300 epochs using the Adam optimizer. Extensive tne-ttning of the learning rate resulted in an optimal learning rate of 0.006 for all experiments using the adversarial illumination attack and pretrained ResNet50. For the experiments considering the adversarial illumnination attack using pretrained VGG19 or Densenet12) network, we found optimal learning rates of 0.001 and 0.002 respectively. Lastly, we found that for all experiments using the motion blur attack learning rate of 0.0011 was optimal. Adversarial training is implemented as suggested in Madry et al. (2018): : at each iteration we find the worst case perturbation with exact approximate method. In all our experiments,
the resulting classifier interpolates the training set. We plot the mean over all runs and the standard deviation of the mean.
D.3 SPECIFICS TO THE MOTION BLUR ATTACK
Fast moving objects or animals are hard to photograph due to motion blur. Hence, when trying to classify detect moving objects from images, is imperative that the classifier is robust against reasonable levels of motion blur. We implement the attack as follows. First, we segment the bird from the original image, then background. We are able to apply more severe blur, by enlarging the kernel of the filter. See Figure for ablation study of the kernel size.
blur filter and lastly, we paste the blurred bird back onto the
The motion blur filter is implemented as follows. We use a kernel of size A x M and build the filter as follows: we fill the row (M - 1)/2 of the kernel with the value 1/M. Thereafter, we use the 2D convolution implementation of OpenCV (filterDD) Bradski (200) to convolve the kernel with the image. Note that applying a rotation before the convolution to the kernel, changes the direction the resulting motion blur. Lastly, we find the most detrimental level of motion blur using list-search over all levels up to Mmax-
(a) 0.3 (b) c 0.2
(c)
0.1 (d) Original
(e) 0.1
(f) 0.2
(g) E = 0.3
Figure 9: An ablation study of the different lighting changes of the adversarial illumination attack. Even though the directed attack perturbs the signal component the image, the bird remains recognizable in all cases.
22

---

Published as a conference paper at ICLR 2023
30
standard training adversarial training
standard training adversarial training
standard error
standard training adversarial training
eiscepttbile
20-
10-
10
250
5000
730
1000
250
500
750
1000
250
500
750
1000
number of samples
number samples
number of samples
(a) Robust error
(b) Standard error
(c) Susceptibility
The plots depict the mean and standard deviation the mean over several independent experiments. We see that, in comparison to standard training, the reduction susceptibility for adversarial training is minimal in the low sample size regime. Moreover, the increase in standard error of adversarial training is quite severe, leading to an overall increase in robust error in the low sample size regime.
Figure 10: The robust error decomposition of the experiments depicted in Figure 10a
D.4 SPECIFICS TO THE ADVERSARIAL ILLUMINATION ATTACK
An adversary can hide objects using poor lightning conditions, which can for example arise from shadows or bright spots. To model poor lighting conditions on the object only (or targeted to the object),
the adversarial illumination attack. The attack is constructed as follows: First, segment the bird from their background. Then we apply an additive constant to the bird, where the absolute size of the constant satisfies Ee| E 0.3. Thereafter, we clip the values of the bird images to [0, IJ. and lastly, we paste the bird back onto the background. See Figure p for an ablation of the parameter of the attack. is non-trivial how to (approximately) find the worst perturbation. We find an approximate solution by searching over all perturbations with increments of size G Kmax Denote by seg, the segmentation profile of the image æ. We consider all perturbed images the form of
K
Zpert (1-seg)rr + sEg(æ + "mmxx
km 12252x2235 255), K [- Kmazi Kmax].
During training time we set Kmaz = 16 and therefore search over 33 possible images. During test time we search over 65 images (Kmar = 32).
D.5 EARLY STOPPING
In all our experiments on the Waterbirds dataset, a parameter search lead to an optimal weight-decay and learning of 10-4 and 0.006 respectively. Another common regularization technique is early stopping, where one stops training on the epoch where the classifier achieves minimal robust error on hold-out dataset. To understand early stopping can mitigate the effect of adversarial training aggregating robust generalization in comparison to standard training, we perform the following experiment. On the Waterbirds dataset of size n = 20 and considering the adversarial illumination attack, we compare standard training with early stopping and adversarial training (or with early stopping. Considering several independent experiments, early stopped adversarial training has an average robust error of 33.5 early stopped standard training 29.1. Hence, early stopping does decrease the robust error gap, but does not close it.
0.3)
D.6 ERROR DECOMPOSITION WITH INCREASING TV
In Figure 10a and 1la we see that adversarial training hurts robust generalization in the small sample size regime. For completeness, we plot the robust error composition for adversarial and standard training in Figure IO We see that the low sample size regime, the drop in susceptibility that adversarial training achieves in comparison to standard training, is much lower than the increase in standard error. Conversely, in the high sample regime, the drop of susceptibility from adversarial training over standard training is much bigger than the increase in standard error.
D.7 DIFFERENT ARCERNTARRCCHEEEE ARCHITECTURES
For completeness, we also performed similar experiments on the waterbirds dataset using the ad- versarial illumination attack with different network architectures with the pretrained ResNet50
23

---

Published as a conference paper at ICLR 2023
standard training aciversarial tr training
standard training
standard error
adversorial training
suscepttbilly
10-
standard training training
robust
120
20
30
100
150
150
56 number samples
100
150
number of samples
number of samples
(a) Robust error
(b) Standard error
(c) Susceptibility
The plots depict the mean and standard deviation the mean over several independent experiments. We see that, in comparison to standard training, the reduction susceptibility for adversarial training is minimal in the low sample size regime. Moreover, the increase in standard error of adversarial training is quite severe, leading to an overall increase in robust error in the low sample size regime.
Figure 11: The robust error dccomposition of the experiments depicted in Figure 6
network. In particular, we considered the following pretrained network architectures: VGG19 and Densenet121. See Figure 12 for the results. We observe that accros models, adversarial training hurts in the low sample size regime, but helps when enough data is available.
ST Rob
AT Rob
X
ST Std
ST Rob
Rob
X ST Std
30
30
error
error
20
20
10
25
75
100
100)
200
300
number of samples
number of samples
(a) VGG19
(b) Densenet121
Figure 12: The robust error of adversarial training and standard training with increasing sample size using the adversarial illumination attack with 6 0.3. We depict the mean and the standard deviation of the mean for multiple runs. Observe that across models, adversarial training hurts the low sample size regime, helps when enough samples are available.
D.8 UNDIRECTED ATTACKS ON THE WATERBIRDS DATASET
In this section, we analyse adversarial training for fg-and &oo -bal perturbations in the small sample size regime. We observe that while adversarial training hurts standard generalization, it helps robust generalization.
Adversarial training with 2alballs We train and test with small ba-balls, er 0.2, such that the networks trained with standard training achieve a non-zero robust accuracy and the networks trained with adversarial training achieve non-trivial standard accuracy. We see Figure [3] that adversarial training with bablls hurts standard generalization while increasing robust generalization. Moreover, in Figure [4] we see that also in the very small sample size regime, adversarial training with increasing éu increases the standard error, but reduces the susceptibility.
Advessaiial training with loc- -blls We also consider -bal perturbation. We see in Figure 15 2 standard training has robust error of 100 percent. that even smallest perturbation budget éte 255 On the other hand, adversarial training achieves low, but non-zero robust error.
We use an ImageNet pretrained ResNet34 and train for 300 epochs. More- over, for reliable robust error and susceptibility evaluation of the attacks we use AutoAttack Croce & Hein (2020). All networks were trained such that the network interpolates the training dataset and has low robust error with non-trivial standard error. For the networks trained using standard training we learning rate of 0.006 and for the networks trained with adversarial training used learning
Experimental details
24

---

Published as a conference paper at ICLR 2023
adversarial training standard training
adversarial training standard training
adversorial training
standard error
60
standard training
20-
ribust
10
40
50
75
100
25
30
100
25
100
number of samples
number of samples
number of samples
(a) Robust error
(b) Standard error
(c) Susceptibility
The robust error decomposition of adversarial training with fz-balls of size 0.2 and test adversaries with f2-balls of size Ee 0.2. The plots depict the mean and standard deviation of the mean over several independent experiments. We see that even though adversarial training hurts standard generalization, increases robust generalization as it decreases the susceptibility of the classifiers.
Figure 13:
6
40-
20
0.0
0.1
0.2
adversarial budget Cy
Figure 14:
The robust error decomposition adversarial training in function of éir in the small sample size regime n = 20. We see that even though adversarial training hurts standard generalization, it increases robust generalization it decreases the susceptibility of the classifiers with increasing We take 20 and consider test adversaries with fu-balls of size 6 0.2. The plots depict the mean and standard deviation of the mean over several independent experiments.
100
10-
adverssrial training
100-
standard eroor
standard training
80
adversarial training
rhhust
standard training
adversrial training
60
standard training
25
30
100
25
30
1.00
25
50
100
number of samples
number of samples
number of samples
(a) Robust error
(b) Standard error
(c) Susceptibility
The robust error decomposition of adversarial training with &oo balls of size 6r 255 and test adversaries with -bals of size 2 several independent experiments. We see that even though adversarial training hurts standard generalization, it increases robust generalization as it decreases the susceptibility of the classifiers.
Figure 15;
255 The plots depict the mean and standard deviation of the mcan over
rate 5- 10-4, We also trained with a weight decay of 10-4, batch size of 8 and a momentum of 0.9 for all networks. We train at least 3 networks for all settings and report the mean and standard deviation of the mean the standard error, robust error and susceptibility over the three runs.
E EXPERIMENTAL DETAILS ON CIFAR-10
In this section, we give the experimental details on the CIFAR-10-based experiments shown in Figures I and 17
In all our experiments we subsample CIFAR-10 to simulate the low sample size regime. We ensure that for all subsampled versions the number of samples of each class are equal. Hence, if we subsample to 500 training images, then each class has exactly 50 images, which are drawn uniformly from the 5k training images of the respective class.
Subsampling CIFAR-10
25

---

Published as a conference paper at ICLR 2023
Mask perturbation on CIFAR-10 On CIFAR-10, we consider the square black mask attack where the adversary can mask a square in the image of size Gu x G by setting the pixel values zero. To ensure that the mask cannot cover all the information about the true class in the image, the size of the masks to be at most 2 x 2, while allowing for all possible locations of the mask in the targeted image. For exact robust error evaluation, we perform full grid search over all possible locations during test time. We show an example of a black-mask attack on each of the classes in CIFAR-10 in Figure [6
restrict
During training, a full grid search is computationally intractable so that we use an approximate attack similar to Wu al. (2020) during training time: by identifying the K 16 most promising mask locations with heuristic as follows. First, we identify promising mask locations by analyzing the gradient, Va (x) y). of the cross-entropy loss with respect to the input. Masks that cover part of the image where the gradient is large, are more likely to increase the loss. Hence, we compute the K mask locations (i, j), where |V U) (=++). II the largest and take using full list-search the mask that incurs the highest loss. Our intuition from the theory predicts that higher K, and hence a more exact "defense", only increases the robust error of adversarial training, since the mask could then more efficiently cover important information about the class.
We show an example of a mask perturbation for all 10 classes of CIFAR-10. Even though the attack occludes part of the images, human can still easily classify all images correctly.
Figure 16:
Experimental training details For all our experiments on CIFAR-10. adjusted the code provided by Phan (202T). As typically done for CIFAR-IO, we augment the data with random cropping and horizontal flipping. For the experiments with results depicted in Figures [ and 17 we use ResNet18 network and train for 100 epochs. We tune the parameters learning rate and weight decay for low robust error. For standard standard training, we use learning rate of 0.01 with equal weight decay. For adversarial training, we use learning rate of 0.015 and weight decay of 10-1. We run each experiment three times for every dataset with different initialization seeds, and plot the average and standard deviation over the runs,
standard
robust
susceptibility
40
0
10
15
20
attack strength K
Figure 17:
The robust error decomposition in standard error and susceptibility for vary- ing attack strengths K. We see that the larger K, the lower the susceptibility, but the higher the standard error.
For the experiments in Figure D and 18 we use an attack strength of K 4. Recall that we perform full grid search at test time and hence have a good approximation of the robust accuracy and susceptibility score.
Increasing training attack strength We investigate the influence of the attack strength K on the robust error for adversarial training. We take &u = 2 and 500 and vary K. The results depicted in Figure 17] We see that for increasing K, the susceptibility decreases, but the standard error increases severely, resulting in increasing robust error.
Robust error decomposition In Figure D we see that the robust error increases for adversarial training compared standard training the low sample size regime, but the opposite holds when enough samples are available. For completeness, we provide a full decomposition of the robust error in standard error and susceptibility for standard and adversarial training. We plot the decomposition in Figure 18
F STATIC HAND GESTURE RECOGNITION
The goal of static hand gesture or posture recognition to recognize hand gestures such as a pointing index finger the okay-sign based static data such images Oudah et al. (2020); Yang al.
26

---

Published as a conference paper at ICLR 2023
(2013). The current use of hand gesture recognition is primarily in the interaction between computers and humans Oudah et al. (2020). More specifically, typical practical applications can be found the environment of games, assisted living. and virtual reality Mujahid al. (202T). In the following, conduct experiments on a hand gesture recognition dataset constructed by Mantecón et al. (2019), which consists of near-infrared stereo images obtained using the Leap Motion device. First, crop or segment the images after which we use logistic regression for classification. We see that adversarial logistic regression deteriorates robust generalization with increasing Eir.
Static hand-gesture dataset We the dataset made available by Mantecón et al. (2019). This dataset consists inar-infrared stereo images taken with the Leap Motion device and provides detailed skeleton data. We base our analysis the images only. The size of the images is 640 x 240 pixels. The dataset consists of 16 classes of hand poses taken by 25 different people. We note that the variety between the different people is relatively wide; there are and women with different posture and hand sizes. However, the different samples taken by the same person are alike.
We consider binary classification between the index-pose and L-pose, and take training set 30 images of the users 16 to 25. This results in a training dataset of 300 samples. We show two different class. Observe that
examples of the training dataset in Figure the near-ing iear-infrered images darken the background and successfully highlight the hand-pose As test dataset, we take 10 images of each of the two classes from the users to 10 resulting in a test dataset of size 200.
each corresponding
Cropping the dataset To speed up training and ease the classification problem, we crop the images from a size of 640 240 a size of 200 200. We crop the images using basic image segmentation technique to stay as close as possible to real-world applications. The aim is to crop the images such that the hand gesture is centered within the cropped image.
For every user in the training set, we crop an image of the L-pose and the index pose by hand. We call these images the training masks ( { 2020 We note that the more particular window of image resembles mask, the more likely that the window captures the hand gesture correctly. Moreover, the near-infrared images are such that the hands of a person are brighter than the surroundings of the person itself. Based on these two observations, upper left coordinates ji, for an image x as the solution to the following optimization problem:
define the best segment or window, defined by the
20
arg min
masks; - I (1::4
2 [ lla.
(34)
re[440] je (40)
Equation 34 is solved using full grid search. We the result to crop both training and test images. Upon manual inspection of the cropped images, close to all images were perfectly cropped. We replace the handful poorly cropped training images with hand-cropped counterparts.
Square-mask perturbations Since we use logistic regression, we perform a full grid search to find the best adversarial perturbation at training and test time. For completeness, the upper left coordinates
standard training adversarial training
60-
Rob Rob
standard error
standard training
30-
adversarial training
30
10
ST Std
201
30
5000
10000
15000
20000
3000
10000
15000
20000
5000
100000
15:00
20000
number samples
number samples
number of samples
(a) Robust error
(b) Standard error
(c) Susceptibility
Figure 18:
The robust error decomposition in standard error and susccptibility of the subsampled datasets of CIFAR-10 10 after adversarial and standard training. For small sample size, adversarial training has higher robust error then standard training.
27

---

Published as a conference paper at ICLR 2023
pose
Index pose
Figure 19: Examples of the original images of the considered hand-gestures. We recognize the "I. -I"" in Figure a9dland the index sign in Figure T9b) Observe that the near-infrared images highlight the hand pose well and blends much the non-usen con-useful or noisy background.
(a) Cropped L pose
Cropped index pose
(c) Black mask perturbation
Figure 20: Examples of the cropped hand- gesture images. We see that the hands are centered and the images have size of 200 x 200. In Figure 20c we show example of the square black mask perturbation.
of the optimal black-mant perturbation of size Gr X Grr can be found as the solution to arg
max ex) JE|200- end
Σ
(35)
L.mEled
The algorithm is rather slow we iterate over all possible windows. We show black-mask perturbation on an L-pose image in Figure 20c
Results We run adversarial logistic regression with square-mask perturbations on the cropped dataset and vary the adversarial training budget and plot the result in Figure 21 We observe attack that adversarial logistic regression deteriorates robust generalization.
Because we use adversarial logistic regression, we are able to visualize the classifier. Given the classifier induced by 0, we can visualize how classifies the images by plotting 0 0-min Oti
max;c ()
[0, 1]d. Recall that the class-prediction of our predictor for data point (x,, y) given by sign x) {+1]. The lighter parts of the resulting image correspond to the class with label 1 and the darker patches with the class corresponding label 1.
We plot the classifiers obtained by standard logistic re- gression and adversarial logistic regression with training adversarial budgets of 10 and 25 in Figure 22 The darker parts the classifier correspond to patches that are typically bright for the L-pose. Complementary, the lighter patches in the classifier correspond to patches that are typically bright for the index pose. We see that in the case of adversarial logistic regression, the background noise is much higher than for standard logistic regression. In other words, adversarial logistic regression puts more weight non-signal parts in the images to classify the
robust error
standard error
50
error
40
301
10
adversarial budget Eoe
The standard error and robust error for varying adversarial training budget fir We that the larger & the higher the robust error.
Figure 21:
28

---

Published as a conference paper at ICLR 2023
training dataset and hence exhibits worse performance the test dataset.
Eir 0
(b) Gr 10
(c) &z 25
We visualize the logistic regression solutions. In Figure 22a plot the vector that induces the classifier obtained after standard training. Figure 22b Figure 22c plot the vector obtained after training with square-mask perturbations
Figure 22:
10 and 25, respectively. We note the non-signal enhanced background correlations at the parts highlighted with the red circles in the image projection the adversarially trained classifiers.
29


Reviews:
[{"review_id": "wIv6KY-U_u", "reviewer": "Reviewer_7vDF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the impact of adversarial training on both the clean and robust error in the low-sample regime using a special type of adversarial attacks, termed directed attacks. The main message is that adversarial training may even hurt the robust accuracy, in addition to the well-known effect of reducing the clean accuracy. The authors analyze the linearly-separable binary classification problem, where directed attacks are easily defined in relation to the optimal separator, and use the robust max-$\\ell_2$-margin suggesting to decompose the robust error into a clean error term and a susceptibility term. The analysis, along with experiments, is aimed to elucidate the effect of the train/test budgets on each type of error. The findings of this analysis are then extended to the multi-class image classification problem using deep CNNs where directed attacks are generalized to encompass a number of image corruptions. Results on standard and customized image datasets are claimed to corroborate the theoretical and empirical findings from earlier.", "summary_of_the_review": "The authors clearly put significant work, evident by the abundance of content and auxiliary analyses. However, the presentation needs significant work to polish up the key contributions and make the results more accessible.\n\nFollowing my extended discussion with the authors, and taking the time to look up more of the recent literature, I'm updating my evaluation with the following requests:\n- Please revise the definition of directed attacks, involving notions of true signal and class information, with a clear and consistent linking to the Bayes optimal classifier.\n- Please include a careful derivation or additional empirical evidence to the qualification of the implemented attacks in the image context as instance of directed attacks.\n- Please cite and discuss related recent works in the same vein of the generalization and accuracy-robustness trade-offs of adversarial training, as I list below. It appears the present work is distinguished by more insightful empirical contributions, while similar works are either mostly theoretical or only include limited experiments.\n  - I recommend to discuss the following papers, which seemed most relevant to the main questions studied:\n>- Dobriban, Edgar, Hamed Hassani, David Hong, and Alexander Robey. \"Provable tradeoffs in adversarially robust classification.\" arXiv preprint arXiv:2006.05161 (2020).\n>- Min, Yifei, Lin Chen, and Amin Karbasi. \"The curious case of adversarially robust models: More data can help, double descend, or hurt generalization.\" In Uncertainty in Artificial Intelligence, pp. 129-139. PMLR, 2021.\n>- Dong, Chengyu, Liyuan Liu, and Jingbo Shang. \"Data Quality Matters For Adversarial Training: An Empirical Study.\" arXiv preprint arXiv:2102.07437 (2021).\n>- Mendonça, Marcele OK, Javier Maroto, Pascal Frossard, and Paulo SR Diniz. \"Adversarial training with informed data selection.\" In 2022 30th European Signal Processing Conference (EUSIPCO), pp. 608-612. IEEE, 2022.\n  - I recommend to include a brief citation of the following papers:\n>- Attias, Idan, Aryeh Kontorovich, and Yishay Mansour. \"Improved generalization bounds for robust learning.\" In Algorithmic Learning Theory, pp. 162-183. PMLR, 2019.\n>- Xing, Yue, Qifan Song, and Guang Cheng. \"On the generalization properties of adversarial training.\" In International Conference on Artificial Intelligence and Statistics, pp. 505-513. PMLR, 2021.\n\nWith a better understanding of the positioning of the presented contributions, I am increasing my score with residual concerns regarding the clarity of presentation and the anticipated impact given those clarity issues.", "strength_and_weaknesses": "- Strengths\n  - Brings attention to a theoretically-interesting aspect of adversarial training, that could be relevant in practice.\n  - Presents theoretical analysis in the linear 2-class case, verified by corresponding experiments.\n  - Applies the results to a restricted class of adversarial attacks on image datasets, (claiming to) corroborate the theory.\n- Weaknesses\n  - The presentation needs significant work ~~,to the point that I'm unable to grasp the core technical points or the empirical evaluations.~~\n    - The definition of directed attacks, involving notions of true signal and class information, needs to be solidified with a clear and consistent linking to the Bayes optimal classifier.\n    - The qualification of the implemented attacks in the image context as instance of directed attacks needs to be established more rigorously.\n    - Important recent work needs to be cited and discussed to better position the present contributions; see below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "jvQpIqPQRs", "reviewer": "Reviewer_SVdL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on adversarial training with direct attacks that effectively reduce the information about the true classes. It presents interesting observations that adversarial training might hurt robustness performance under limited training sample sizes. Theoretical analysis with main theorems provided on linear classifier case study and the generalizability is discussed. Experimental results on the customized Waterbirds,  CIFAR10, and hand-gesture datasets support the claims.", "summary_of_the_review": "Overall, the paper provides good observations on adversarial training with the directed attack. The authors provide solid theoretical analysis and sufficient empirical evaluations to support the claims.", "strength_and_weaknesses": "### Strength\n- The paper presents very interesting observations that can potentially arouse wide attention for the practical usage of adversarial training.\n- Solid theoretical analysis on linear classifiers with good intuitions provided.\n- Experimental results focus on a customized Waterbirds dataset with motion blur and illumination and the results confirm the claims. CIFAR10 and hand-gesture dataset with square mask perturbations are also investigated.\n\n### Weakness\n- More types of direct attacks could be potentially evaluated, e.g., physical attacks on stop signs, to make the claims more convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PatSmwqUxh", "reviewer": "Reviewer_udiw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the peculiar observation that, with very sample sizes, robust training can be detrimental for robust error. The authors show that in a classification task given by a linear model, the robust risk of the max-margin classier can increase as a function of the perturbation size used during robust training. Importantly, they study a new notion of robust training that they call \"directed attacks\", which specifically targets the informative features of the problem. The authors then show how, in more complex settings where analysis is harder (and not presented here) some of the intuitions gained for the linear case seem to translate as well. The paper is clearly written and organized.\n", "summary_of_the_review": "This paper studies a new (to my knowledge) behavior in adversarial training and shows interesting results.", "strength_and_weaknesses": "This is an interesting paper, which I enjoyed reading very much. The authors clearly study and characterize the simple linear setting, and their conclusions seem to extend, empirically, to more complicated scenarios.\n\nI have a few comments, mostly minor:\n\n1. The definition and analysis of \"directed attack\" is interesting, as it is a relative new concept (to my knowledge). I understand why restricting their analysis to these cases in particular is important. Yet, could the authors comment more on the differences between regular/traditional adversarial losses (say, L2 or Linfty bounded) and their \"directed attack\" model? It seems to me that in the linear case with only 1 informative feature, they would be equivalent (except in the Linfty case), but maybe I'm missing something.\n\n2. In their definition of logistic loss, immediately after Eq 2, i think the authors meant $e$ instead of $\\mathbb E$ (expectation operator).\n\n3. Commenting on Eq (7), the authors mention that this equation suggests that the robust error \"can only be small if both the standard error and susceptibility are small\". This is not true, as Eq. (7) is just an upper bound on the robust error. Probably the authors meant \"a small standard error and susceptibility imply a small robust error\". At the same time, it seems like that for the linear model studied here this bound could be tight?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IBw5-xeL5xD", "reviewer": "Reviewer_bvDH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle the problem of robust training. They show that when the training set is small robust training can hurt the robust performance of the model. They start by theoretically showing this result for linear classifiers. Using these results they also design directed attacks and show that robust error can indeed increase when the dataset is small. ", "summary_of_the_review": "This paper cautions ML practitioners against blindly applying ML techniques for robust learning. However, the scope of the application may not be large as most current ML techniques use a large dataset to train. This significantly takes away from the contributions of this paper.", "strength_and_weaknesses": "Strengths\n1. The results presented in this paper are interesting and does indeed caution user from blindly applying standard ML techniques for adversarial training. \n2. The theoretical aspect of the paper is sound and I prefer the way the authors draw inspiration from the linear classifiers and extend that to non-linear ones.\n3. The experimental results are also neat and clearly convey the message the authors are trying to provide.\n\nWeaknesses\n1. While the results are neat, I would have also liked to seen a more detailed discussion regarding in which scenario these learnings would be useful. The experimental results are for image classifiers which usually require a large amount of data both for training from scratch and for transfer learning. Thus it is unclear to me where the gain is useful. As the authors themselves mention, one of the main reasons for this degradation would be catastrophic overlearning which is intuitive on its own. \n2. One potential way to strengthen the findings of this paper would be to have a relation on when this robust error increase happens in terms of d and n. That aspect is currently missing from this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster