Title: TESTING ROBUSTNESS AGAINST UNFORESEEN ADVERSARIES

Abstract: Most existing defenses against adversarial attacks only consider robustness to L pbounded distortions. In reality, the specific attack is rarely known in advance and adversaries are free to modify images in ways which lie outside any fixed distortion model; for example, image rotations lie outside the set of L p -bounded distortions. In this work, we advocate measuring robustness against a much broader range of unforeseen attacks, attacks whose precise form is unknown during defense design. We propose several new attacks and a methodology for evaluating a defense against a diverse range of unforeseen distortions. First, we construct novel adversarial JPEG, Fog, Gabor, and Snow distortions to simulate more diverse adversaries. We then introduce UAR, a summary metric that measures the robustness of a defense against a given distortion. Using UAR to assess robustness against existing and novel attacks, we perform an extensive study of adversarial robustness. We find that evaluation against existing L p attacks yields redundant information which does not generalize to other attacks; we instead recommend evaluating against our significantly more diverse set of attacks. We further find that adversarial training against either one or multiple distortions fails to confer robustness to attacks with other distortion types. These results underscore the need to evaluate and study robustness against unforeseen distortions. Minimum Distortion JPEG Gabor Snow Maximum Distortion JPEG Gabor Snow (b) The JPEG, Gabor, and Snow attacks applied to adversarially trained models at εmin and εmax. Distortions are almost imperceptible at εmin, but make the image barely recognizable by humans at εmax.

1. INTRODUCTION:

Neural networks perform well on many benchmark tasks (He et al., 2016) yet can be fooled by adversarial examples (Goodfellow et al., 2014) or inputs designed to subvert a given model. Adversaries are usually assumed to be constrained by an L ∞ budget (Goodfellow et al., 2014; Madry et al., 2017; Xie et al., 2018) , while other modifications such as adversarial geometric transformations, patches, and even 3D-printed objects have also been considered (Engstrom et al., 2017; Brown et al., 2017; Athalye et al., 2017) . However, most work on adversarial robustness assumes that the adversary is fixed and known in advance. Defenses against adversarial attacks are often constructed in view of this specific assumption (Madry et al., 2017) .

In practice, adversaries can modify and adapt their attacks so that they are unforeseen. In this work, we propose novel attacks which enable the diverse assessment of robustness to unforeseen attacks. Our attacks are varied ( §2) and qualitatively distinct from current attacks. We propose adversarial JPEG, Fog, Gabor, and Snow attacks (sample images in Figure 1 ).

We propose an unforeseen attack evaluation methodology ( §3) that involves evaluating a defense against a diverse set of held-out distortions decoupled from the defense design. For a fixed, held-out distortion, we then evaluate the defense against the distortion for a calibrated range of distortion sizes whose strength is roughly comparable across distortions. For each fixed distortion, we summarize the robustness of a defense against that distortion relative to a model adversarially trained on that distortion, a measure we call UAR. We provide code and calibrations to easily evaluate a defense against our suite of attacks at https://github.com/iclr-2020-submission/ advex-uar.

By applying our method to 87 adversarially trained models and 8 different distortion types ( §4), we find that existing defenses and evaluation practices have marked weaknesses. Our results show

L ∞ L 2 L 1 Elastic New Attacks JPEG Fog Gabor Snow

Figure 1 : Attacked images (label "espresso maker") against adversarially trained models with large ε. Each of the adversarial images above are optimized to maximize the classification loss.

that existing defenses based on adversarial training do not generalize to unforeseen adversaries, even when restricted to the 8 distortions in Figure 1 . This adds to the mounting evidence that achieving robustness against a single distortion type is insufficient to impart robustness to unforeseen attacks (Jacobsen et al., 2019; Jordan et al., 2019; Tramèr & Boneh, 2019) .

Turning to evaluation, our results demonstrate that accuracy against different L p distortions is highly correlated relative to the other distortions we consider. This suggest that the common practice of evaluating only against L p distortions to test a model's adversarial robustness can give a misleading account. Our analysis demonstrates that our full suite of attacks adds substantive attack diversity and gives a more complete picture of a model's robustness to unforeseen attacks.

A natural approach is to defend against multiple distortion types simultaneously in the hope that seeing a larger space of distortions provides greater transfer to unforeseen distortions. Unfortunately, we find that defending against even two different distortion types via joint adversarial training is difficult ( §5). Specifically, joint adversarial training leads to overfitting at moderate distortion sizes.

In summary, we propose a metric UAR to assess robustness of defenses against unforeseen adversaries. We introduce a total of 4 novel attacks. We apply UAR to assess how robustness transfers to existing attacks and our novel attacks. Our results demonstrate that existing defense and evaluation methods do not generalize well to unforeseen attacks.

2. A SET OF DIVERSE AND NOVEL ADVERSARIAL ATTACKS:

We consider distortions (attacks) applied to an image x ∈ R 3×224×224 , represented as a vector of RGB values. Let f : R 3×224×224 → R 100 be a model mapping images to logits1 , and let (f (x), y) denote the cross-entropy loss. For an input x with true label y and a target class y = y, our adversarial attacks attempt to find x such that 1. the attacked image x is obtained by applying a constrained distortion to x, and 2. the loss (f (x ), y ) is minimized (targeted attack).

Adversarial training (Goodfellow et al., 2014) is a strong defense baseline against a fixed attack (Madry et al., 2017; Xie et al., 2018) which updates using an attacked image x instead of the clean image x at each training iteration.

We consider 8 attacks: L ∞ (Goodfellow et al., 2014) , L 2 (Szegedy et al., 2013; Carlini & Wagner, 2017) , L 1 (Chen et al., 2018) , Elastic (Xiao et al., 2018) , JPEG, Fog, Gabor, and Snow. We show sample attacked images in Figure 1 and the corresponding distortions in Figure 2 . The JPEG, Fog, (4.1m, 11.1k, 32) L2 (1.3m, 4.8k, 99) L1 (224k, 2.6k, 218) Elastic (3.1m, 15.2k, 253) New Attacks JPEG (5.4m, 18.7k, 255) Fog (4.1m, 13.2k, 89) Gabor (3.7m, 13.3k, 50) Snow (11.3m, 32.0k, 255) Figure 2 : Scaled pixel-level differences between original and attacked images for each attack (label "espresso maker"). The L 1 , L 2 , and L ∞ norms of the difference are shown after the attack name. Our novel attacks display behavior which is qualitatively different from that of the L p attacks. Attacked images are shown in Figure 1 , and unscaled differences are shown in Figure 9 , Appendix B.1.

Existing Attacks L∞

Gabor, and Snow attacks are new to this paper, and the L 1 attack uses the Frank-Wolfe algorithm to improve on previous L 1 attacks. We now describe the attacks, whose distortion sizes are controlled by a parameter ε. We clamp output pixel values to [0, 255].

Existing attacks. The L p attacks with p ∈ {1, 2, ∞} modify an image x to an attacked image x = x+δ. We optimize δ under the constraint δ p ≤ ε, where • p is the L p -norm on R 3×224×224 .

The Elastic attack warps the image by allowing distortions x = Flow(x, V ), where V : {1, . . . , 224} 2 → R 2 is a vector field on pixel space, and Flow sets the value of pixel (i, j) to the bilinearly interpolated original value at (i, j) + V (i, j). We construct V by smoothing a vector field W by a Gaussian kernel (size 25 × 25, std. dev. 3 for a 224 × 224 image) and optimize W under W (i, j) ∞ ≤ ε for all i, j. This differs in details from Xiao et al. (2018) but is similar in spirit.

Novel attacks. As discussed in Shin & Song (2017) for defense, JPEG compression applies a lossy linear transformation JPEG based on the discrete cosine transform to image space, followed by quantization. The JPEG attack imposes the L ∞ -constraint JPEG(x) -JPEG(x ) ∞ ≤ ε on the attacked image x . We optimize z = JPEG(x ) and apply a right inverse of JPEG to obtain x .

Original Initialization Optimized:

Figure 3 : Snow before and after optimization.

Our novel Fog, Gabor, and Snow attacks are adversarial versions of non-adversarial distortions proposed in the literature. Fog and Snow introduce adversarially chosen partial occlusions of the image resembling the effect of mist and snowflakes, respectively; stochastic versions of Fog and Snow appeared in Hendrycks & Dietterich (2019) . Gabor superimposes adversarially chosen additive Gabor noise (Lagae et al., 2009) onto the image; a stochastic version appeared in Co et al. (2019) . These attacks work by optimizing a set of parameters controlling the distortion over an L ∞ -bounded set. Specifically, values for the diamond-square algorithm, sparse noise, and snowflake brightness (Figure 3 ) are chosen adversarially for Fog, Gabor, and Snow, respectively.

Optimization. To handle L ∞ and L 2 constraints, we use randomly-initialized projected gradient descent (PGD), which optimizes the distortion δ by gradient descent and projection to the L ∞ and L 2 balls (Madry et al., 2017) . For L 1 constraints, this projection is more difficult, and previous L 1 attacks resort to heuristics (Chen et al., 2018; Tramèr & Boneh, 2019) . We use the randomly- initialized Frank-Wolfe algorithm (Frank & Wolfe, 1956) , which replaces projection by a simpler optimization of a linear function at each step (pseudocode in Appendix B.2).

3. MOTIVATION AND DESCRIPTION OF OUR METHODOLOGY:

We now propose a method to assess robustness against unforeseen distortions, which relies on evaluating a defense against a diverse set of attacks that were not used when designing the defense. Our method must address the following issues:

• The range of distortion sizes must be wide enough to avoid the misleading behavior in which robustness appears to transfer at low distortion sizes but not at high distortion sizes (Figure 4 ); • The set of attacks considered must be sufficiently diverse.

We first provide a method to calibrate distortion sizes and then use it to define a summary metric that assesses the robustness of a defense against a specific unforeseen attack. Using this metric, we are able to assess diversity and recommend a set of attacks to evaluate against.

Calibrate distortion size using adversarial training. As shown in Figure 4 , the correlation between adversarial robustness against different distortion types may look different for different ranges of distortion sizes. It is therefore critical to evaluate on a wide enough range of distortion size ε. We choose the minimum and maximum distortion sizes ε using the following principles; sample images at ε min and ε max are shown in Figure 5b .

1. The minimum distortion size ε min is the largest ε for which the adversarial validation accuracy against an adversarially trained model is comparable to that of a model trained and evaluated on unattacked data (for ImageNet-100, within 3 of 87). 2. The maximum distortion size ε max is the smallest ε which either (a) yields images which confuse humans when applied against adversarially trained models or (b) reduces accuracy of adversarially trained models (ATA below) to below 25.

In practice, we select ε min and ε max according to these criteria from a sequence of ε which is geometrically increasing with ratio 2. We choose to evaluate against adversarially trained models because attacking against strong defenses is necessary to produce strong visual distortions (Figure 5a ). We introduce the constraint that humans recognize attacked images at ε max because we find cases for L 1 , Fog, and Snow where adversarially trained models maintain non-zero accuracy for distortion sizes producing images incomprehensible to humans. An example for Snow is shown in Figure 5b .

UAR: an adversarial robustness metric. We measure a model's robustness against a specific distortion type by comparing it to adversarially trained models, which represent an approximate ceiling on performance with prior knowledge of the distortion type. other than ResNet-50 or ResNet-56, we recommend using the ATA values computed with these architectures to allow for uniform comparisons.

Given a set of distortion sizes {ε 1 , . . . , ε n }, we propose the summary metric UAR (Unforeseen Attack Robustness) normalizing the accuracy of a model M against adversarial training accuracy:

EQUATION

Here Acc(A, ε, M ) is the accuracy of M against distortions of type A and magnitude ε. We expect most UAR scores to be lower than 100 against held-out distortion types, as an UAR score greater than 100 means that a defense is outperforming an adversarially trained model on that distortion. The normalizing factor in (1) is required to keep UAR scores roughly comparable between distortions, as different distortions can have different strengths as measured by ATA at the chosen distortion sizes.

Having too many or too few ε k values in a certain range may cause an attack to appear artificially strong or weak because the functional relation between distortion size and attack strength (measured by ATA) varies between attacks. To make UAR roughly comparable between distortions, we evaluate at ε increasing geometrically from ε min to ε max by factors of 2 and take the subset of ε whose ATA values have minimum 1 -distance to the ATA values of the L ∞ attack at geometrically increasing ε.

For example, when calibrating Elastic in Table 1 , we start with ε min = 0.25 and ε max = 16 based on our earlier criteria. We then compute the ATAs at the 7 geometrically increasing ε values ε ∈ {0.25, 0.5, 1, 2, 4, 8, 16}. We consider size-6 subsets of those ATA values, view them as vectors of length 6 in decreasing order, and compute the 1 -distance between these vectors and the vector for L ∞ shown in the first row of Table 1 . Finally, we select the ε values for Elastic in Table 1 as those corresponding to the size-6 subset with minimum 1 -distance to the vector for L ∞ .

For our 8 distortion types, we provide reference values of ATA(A, ε) on this calibrated range of 6 distortion sizes on ImageNet-100 (Table 1 , §4) and CIFAR-10 (Table 3 , Appendix C.3.2). This allows UAR computation for a new defense using 6 adversarial evaluations and no adversarial training, reducing computational cost from 192+ to 6 NVIDIA V100 GPU-hours on ImageNet-100.

Evaluate against diverse distortion types. Since robustness against different distortion types may have low or no correlation (Figure 6b ), measuring performance on different distortions is important to avoid overfitting to a specific type, especially when a defense is constructed with it in mind (as with adversarial training). Our results in §4 demonstrate that choosing appropriate distortion types to evaluate against requires some care, as distortions such as L 1 , L 2 , and L ∞ that may seem different can actually have highly correlated scores against defenses (see Figure 6 ). We instead recommend evaluation against our more diverse attacks, taking the L ∞ , L 1 , Elastic, Fog, and Snow attacks as a starting point. We apply our methodology to the 8 attacks in §2 using models adversarially trained against these attacks. Our results reveal that evaluating against the commonly used L p -attacks gives highly correlated information which does not generalize to other unforeseen attacks. Instead, they suggest that evaluating on diverse attacks is necessary and identify a set of 5 attacks with low pairwise robustness transfer which we suggest as a starting point when assessing robustness to unforeseen adversaries.

Dataset and model. We use two datasets: CIFAR-10 and ImageNet-100, the 100-class subset of ImageNet-1K (Deng et al., 2009) containing every 10 th class by WordNet ID order. We use ResNet-56 for CIFAR-10 and ResNet-50 as implemented in torchvision for ImageNet-100 (He et al., 2016) . We give training hyperparameters in Appendix A.

Adversarial training and evaluation procedure. We construct hardened models using adversarial training (Madry et al., 2017) . To train against attack A, for each mini-batch of training images, we select a uniform random (incorrect) target class for each image. For maximum distortion size ε, we apply the targeted attack A to the current model with distortion size ε ∼ Uniform(0, ε) and update the model with a step of stochastic gradient descent using only the resulting adversarial images (no clean images). The random size scaling improves performance especially against smaller distortions. We use 10 optimization steps for all attacks during training except for Elastic, where we use 30 steps due to its more difficult optimization problem. When PGD is used, we use step size ε/ √ steps, the optimal scaling for non-smooth convex functions (Nemirovski & Yudin, 1978; 1983) .

We adversarially train 87 models against the 8 attacks from §2 at the distortion sizes described in §3 and evaluate them on the ImageNet-100 and CIFAR-10 validation sets against 200-step targeted attacks with uniform random (incorrect) target class. This uses more steps for evaluation than train- ing per best practices (Carlini et al., 2019) . We use UAR to analyze the results in the remainder of this section, directing the reader to Figures 10 and 11 (Appendix C.2) for exhaustive results and to Appendix D for checks for robustness to random seed and number of attack steps.

L ∞ L 2 L 1 J P E G E la s t ic F o g G a b o r S n o w Normal Training L ∞ ε = 32 L 2 ε = 4800 L 1 ε = 612000 JPEG ε = 2 Elastic ε = 16

Existing defense and evaluation methods do not generalize to unforeseen attacks. The many low off-diagonal UAR scores in Figure 6a make clear that while adversarial training is a strong baseline against a fixed distortion, it only rarely confers robustness to unforeseen distortions. Notably, we were not able to achieve a high UAR against Fog except by directly adversarially training against it. Despite the general lack of transfer in Figure 6a , the fairly strong transfer between the L p -attacks is consistent with recent progress in simultaneous robustness to them (Croce & Hein, 2019) .

Figure 6b shows correlations between UAR scores of pairs of attacks A and A against defenses adversarially trained without knowledge3 of A or A . The results demonstrate that defenses trained without knowledge of L p -attacks have highly correlated UAR scores against the different L p attacks, but this correlation does not extend to their evaluations against other attacks. This suggests that L pevaluations offer limited diversity and may not generalize to other unforeseen attacks.

The L ∞ , L 1 , Elastic, Fog, and Snow attacks offer greater diversity. Our results on L p -evaluation suggest that more diverse attack evaluation is necessary for generalization to unforeseen attacks. As the unexpected correlation between UAR scores against the pairs (Fog, Gabor) and (JPEG, L 1 ) in Figure 6b demonstrates, even attacks with very different distortions may have correlated behaviors. Considering all attacks in Figure 6 together results in signficantly more diversity, which we suggest for evaluation against unforeseen attacks. We suggest the 5 attacks (L ∞ , L 1 , Elastic, Fog, and Snow) with low UAR against each other and low correlation between UAR scores as a good starting point.

5. JOINT ADVERSARIAL TRAINING: DEFENDING AGAINST TWO DISTORTIONS:

A natural idea to improve robustness against unforeseen adversaries is to adversarially train the same model against two different types of distortions simultaneously, with the idea that this will cover a larger portion of the space of distortions. We refer to this as joint adversarial training (Jordan et al., 2019; Tramèr & Boneh, 2019) . For two attacks A and A , at each training step, we compute the attacked image under both A and A and backpropagate with respect to gradients induced by the image with greater loss. This corresponds to the "max" loss described in Tramèr & Boneh (2019) . We jointly train models for (L ∞ , L 2 ), (L ∞ , L 1 ), and (L ∞ , Elastic) using the same setup as before Transfer for jointly trained models. Figure 7 reports UAR scores for jointly trained models using ResNet-50 on ImageNet-100; full evaluation accuracies are in Figure 19 (Appendix E). Comparing to Figure 6a and Figure 12 (Appendix E), we see that, relative to training against only L 2 , joint training against (L ∞ , L 2 ) slightly improves robustness against L 1 without harming robustness against other attacks. In contrast, training against (L ∞ , L 1 ) is worse than either training against L 1 or L ∞ separately (except at small ε for L 1 ). Training against (L ∞ , Elastic) also performs poorly.

L ∞ L 2 L 1 J P E G E la s ti c Normal Training L∞ ε = 1, L2 ε = 300 L∞ ε = 2, L2 ε = 600 L∞ ε = 4, L2 ε = 1200 L∞ ε = 8, L2 ε = 2400 L∞ ε = 16, L2 ε = 4800

L ∞ L 2 L 1 J P E G E la s ti c Normal Training L∞ ε = 1, Elastic ε = 0.5 L∞ ε = 2, Elastic ε = 1 L∞ ε = 4, Elastic ε = 2 L∞ ε = 8, Elastic ε = 4 L∞ ε =

Joint training and overfitting. Jointly trained models achieve high training accuracy but poor validation accuracy (Figure 8 ) that fluctuates substantially for different random seeds (Table 4 , Appendix E.2). Figure 8 shows the overfitting behavior for (L ∞ , Elastic): L ∞ validation accuracy decreases significantly during training while training accuracy increases. This contrasts with standard adversarial training (Figure 8 ), where validation accuracy levels off as training accuracy increases. Overfitting primarily occurs when training against large distortions. We successfully trained against the (L ∞ , L 1 ) and (L ∞ , Elastic) pairs for small distortion sizes with accuracies comparable to but slightly lower than observed in Figure 11 for training against each attack individually (Figure 18 , Appendix E). This agrees with behavior reported by Tramèr & Boneh (2019) Evaluating robustness has proven difficult, necessitating detailed study of best practices even for a single fixed attack (Papernot et al., 2017; Athalye et al., 2018) . We build on these best practices by showing how to choose and calibrate a diverse set of unforeseen attacks. Our work is a supplement to existing practices, not a replacement-we strongly recommend following the guidelines in Papernot et al. (2017) and Athalye et al. (2018) in addition to our recommendations.

Some caution is necessary when interpreting specific numeric results in our paper. Many previous implementations of adversarial training fell prone to gradient masking (Papernot et al., 2017; Engstrom et al., 2018) , with apparently successful training occurring only recently (Madry et al., 2017; Xie et al., 2018) . While evaluating with moderately many PGD steps (200) helps guard against this, (Qian & Wegman, 2019) shows that an L ∞ -trained model that appeared robust against L 2 actually had substantially less robustness when evaluating with 10 6 PGD steps. If this effect is pervasive, then there may be even less transfer between attacks than our current results suggest.

For evaluating against a fixed attack, DeepFool Moosavi-Dezfooli et al. ( 2015) and CLEVER Weng et al. ( 2018) can be seen as existing alternatives to UAR. They work by estimating "empirical robustness", which is the expected minimum ε needed to successfully attack an image. However, these apply only to attacks which optimize over an L p -ball of radius ε, and CLEVER can be susceptible to gradient masking Goodfellow (2018) . In addition, empirical robustness is equivalent to linearly averaging accuracy over ε, which has smaller dynamic range than the geometric average in UAR.

Our results add to a growing line of evidence that evaluating against a single known attack type provides a misleading picture of the robustness of a model (Sharma & Chen, 2017; Engstrom et al., 2017; Jordan et al., 2019; Tramèr & Boneh, 2019; Jacobsen et al., 2019) . Going one step further, we believe that robustness itself provides only a narrow window into model behavior; in addition to robustness, we should seek to build a diverse toolbox for understanding machine learning models, including visualization (Olah et al., 2018; Zhang & Zhu, 2019) , disentanglement of relevant features (Geirhos et al., 2018) , and measurement of extrapolation to different datasets (Torralba & Efros, 2011) or the long tail of natural but unusual inputs (Hendrycks et al., 2019) . Together, these windows into model behavior can give us a clearer picture of how to make models reliable in the real world. A TRAINING HYPERPARAMETERS For ImageNet-100, we trained on machines with 8 NVIDIA V100 GPUs using standard data augmentation He et al. (2016) . Following best practices for multi-GPU training Goyal et al. (2017) , we ran synchronized SGD for 90 epochs with batch size 32×8 and a learning rate schedule with 5 "warm-up" epochs and a decay at epochs 30, 60, and 80 by a factor of 10. Initial learning rate after warm-up was 0.1, momentum was 0.9, and weight decay was 10 -4 . For CIFAR-10, we trained on a single NVIDIA V100 GPU for 200 epochs with batch size 32, initial learning rate 0.1, momentum 0.9, and weight decay 10 -4 . We decayed the learning rate at epochs 100 and 150.

B FURTHER ATTACK DETAILS B.1 FURTHER EXAMPLES OF ATTACKS:

We show the images corresponding to the ones in Figure 2 , with the exception that they are not scaled. The non-scaled images are shown in Figure 9 .

B.2 L 1 ATTACK

We chose to use the Frank-Wolfe algorithm for optimizing the L 1 attack, as Projected Gradient Descent would require projecting onto a truncated L 1 ball, which is a complicated operation. In contrast, Frank-Wolfe only requires optimizing linear functions g x over a truncated L 1 ball; this can be done by sorting coordinates by the magnitude of g and moving the top k coordinates to the boundary of their range (with k chosen by binary search). This is detailed in Algorithm 1.

C FULL EVALUATION RESULTS:

C.1 L 1 -JPEG AND L 2 -JPEG ATTACKS

We will present results with two additional versions of the JPEG attack which impose L 1 or L 2 constraints on the attack in JPEG-space instead of the L ∞ constraint discussed in Section 2. To avoid confusion, in this appendix, we denote the original JPEG attack by L ∞ -JPEG and these variants by Algorithm 1 Pseudocode for the Frank-Wolfe algorithm for the L 1 attack.

1: Input: function f , initial input x ∈ [0, 1] d , L 1 radius ρ, number of steps T . 2: Output: approximate maximizer x of f over the truncated L 1 ball B 1 (ρ; x) ∩ [0, 1] d centered at x. 3: 4: x (0) ← RandomInit(x) Random initialization 5: for t = 1, . . . , T do 6: g ← ∇f (x (t-1) )

Obtain gradient 7:

for k = 1, . . . , d do 8:

s k ← index of the coordinate of g by with k th largest norm 9:

end for 10:

S k ← {s 1 , . . . , s k }.

11: 12::

for i = 1, . . . , d do Compute move to boundary of [0, 1] for each coordinate.

13:

if g i > 0 then 14: b i ← 1 -x i 15: else 16: b i ← -x i 17: end if 18:

end for 19:

M k ← i∈S k |b i | Compute L 1 -perturbation of moving k largest coordinates. 20: k * ← max{k | M k ≤ ρ} Choose largest k satisfying L 1 constraint.

21:

for i = 1, . . . , d do Compute x maximizing g x over the L 1 ball. 22: if i ∈ S k * then 23: xi ← x i + b i 24: else if i = s k * +1 then 25: xi ← x i + (ρ -M k * ) sign(g i ) 26:

else 27:

xi ← x i 28:

end if

29::

end for 30:

x (t) ← (1 -1 t )x (t-1) + 1 t x

Average x with previous iterates 31: end for 32: x ← x (T ) we find that they have extremely similar results, so we omit L 1 -JPEG in the full analysis for brevity and visibility. Calibration values for these attacks are shown in Table 2 .

C.2 FULL EVALUATION RESULTS AND ANALYSIS FOR IMAGENET-100:

We show the full results of all adversarial attacks against all adversarial defenses for ImageNet-100 in Figure 11 . As described, the L p attacks and defenses give highly correlated information on heldout defenses and attacks respectively. Thus, we recommend evaluating on a wide range of distortion types. Full UAR scores are also provided for ImageNet-100 in Figure 12 .

We further show selected results in Figure 13 . As shown, a wide range of ε is required to see the full behavior.

L 2 -J P E G ε = 2 L 2 -J P E G ε = 1 6 L 2 -J P E G ε = 3 2 L 2 -J P E G ε = 6 4 L 2 -J P E G ε = 1 2 8 L 2 -J P E G ε = 2 5 6 L 1 -J P E G ε = 1 2 8 L 1 -J P E G ε = 1 0 2 4 L 1 -J P E G ε = 4 0 9 6 L 1 -J P E G ε = 1 6 3 8 4 L 1 -J P E G ε = 6 5 5 3 6 L 1 -J P E G ε = 1 3 1 0 7 2 Attack (evaluation) Normal training L ∞ ε = 1 L ∞ ε = 2 L ∞ ε = 4 L ∞ ε = 8 L ∞ ε = 16 L ∞ ε = 32 L 2 ε = 150 L 2 ε = 300 L 2 ε = 600 L 2 ε = 1200 L 2 ε = 2400 L 2 ε = 4800 L 1 ε = 9562.44 L 1 ε = 19125 L 1 ε = 38250.1 L 1 ε = 76500 L 1 ε = 153000 L 1 ε = 306000 L 1 ε = 612000 L ∞ -JPEG ε = 0.03125 L ∞ -JPEG ε = 0.0625 L ∞ -JPEG ε = 0.125 L ∞ -JPEG ε = 0.25 L ∞ -JPEG ε = 0.5 L ∞ -JPEG ε = 1 L ∞ -JPEG ε = 2 L ∞ -JPEG ε = 4 L 2 -JPEG ε = 2 L 2 -JPEG ε = 4 L 2 -JPEG ε = 8 L 2 -JPEG ε = 16 L 2 -JPEG ε = 32 L 2 -JPEG ε = 64 L 2 -JPEG ε = 128 L 2 -JPEG ε = 256 L 1 -JPEG ε = 128 L 1 -JPEG ε = 256 L 1 -JPEG ε = 512 L 1 -JPEG ε = 1024 L 1 -JPEG ε = 2048 L 1 -JPEG ε = 4096 L 1 -JPEG ε = 8192 L 1 -JPEG ε = 16384 L 1 -JPEG ε = 32768 L 1 -JPEG ε = 65536 L 1 -JPEG ε = 131072 Elastic ε = 0.25 Elastic ε = 0.5 Elastic ε = 1 Elastic ε = 2 Elastic ε = 4 Elastic ε = 8 Elastic ε = 16

Attack (adversarial training) 70 0 0 0 0 0 50 0 0 0 0 0 86 10 0 0 0 0 76 5 0 0 0 0 85 34 1 0 0 0 79 16 0 0 0 0 84 54 5 0 0 0 79 31 1 0 0 0 79 43 6 0 0 0 74 33 2 0 0 0 73 26 3 0 0 0  64 25 3 0 0 0  68 7 1 0 0 0  56 23 4 0 0 0   85 2 0 0 0 0  73 2 0 0 0 0  85 17 0 0 0 0  78 7 0 0 0 0  85 49 2 0 0 0  82 24 0 0 0 0  82 74 25 0 0 0  81 56 4 0 0 0  77 74 57 7 0 0  77 68 25 1 0 0  68 67 62 33 1 87 40 2 0 0 0 0 69 16 0 0 0 0 62 26 4 0 0 0 0 23 1 0 0 0 0 0 74 27 1 0 0 0 0 0 49 18 2 0 0 0 0 0 0 0 0 81 54 9 0 0 0 0 83 69 33 3 0 0 0 0 0 0 19 3 1 0 0 0 0 0 0 0 77 47 12 1 0 0 0 0 0 0 88 40 2 0 0 0 0 68 17 0 0 0 0 60 24 3 0 0 0 0 17 1 0 0 0 0 0 71 22 1 0 0 0 0 0 45 15 2 0 0 0 0 0 0 0 0 82 58 12 0 0 0 0 85 78 59 21 2 0 0 0 0 0 25 5 1 0 0 0 0 0 0 0 80 57 19 2 0 0 0 0 0 0 87 29 1 0 0 0 0 59 12 0 0 0 0 54 19 2 0 0 0 0 9 0 0 0 0 0 0 57 11 0 0 0 0 0 0 30 6 0 0 0 0 0 0 0 0 0 82 61 14 1 0 0 0 86 82 74 51 16 2 1 1 1 1 31 6 1 0 0 0 0 0 0 0 81 65 27 4 0 0 0 0 0 0 86 23 1 0 0 0 0 50 9 0 0 0 0 49 16 2 0 0 0 0 6 0 0 0 0 0 0 45 8 0 0 0 0 0 0 29 7 1 0 0 0 0 0 0 0 0 82 63 16 1 0 0 0 86 84 78 67 43 16 5 2 1 1 40 9 2 0 0 0 0 0 0 0 82 68 39 8 1 0 0 0 0 0 85 17 1 0 0 0 0 42 6 0 0 0 0 46 16 2 0 0 0 0 4 0 0 0 0 0 0 35 5 0 0 0 0 0 0 27 7 1 0 0 0 0 0 0 0 0 81 63 19 1 0 0 0 84 83 79 71 56 34 16 7 3 1 52 16 3 1 0 0 0 0 0 0 80 72 44 11 2 0 0 0 0 0 78 8 0 0 0 0 0 24 2 0 0 0 0 26 5 0 0 0 0 0 2 0 0 0 0 0 0 23 2 0 0 0 0 0 0 15 3 0 0 0 0 0 0 0 0 0 74 60 19 1 0 0 0 79 78 76 72 67 58 46 29 10 3 62 34 9 2 0 0 0 0 0 0 75 69 50 19 4 1 0 0 0 0 69 4 0 0 0 0 0 12 1 0 0 0 0 25 8 2 0 0 0 0 4 0 0 0 0 0 0 12 2 0 0 0 0 0 0 5 1 0 0 0 0 0 0 0 0 0 65 49 11 0 0 0 0 70 70 70 68 68 68 65 57 39 17 68 66 63 48 15 2 3 1 0 0 67 65 60 49 30 12 2 0 0 0 62 3 0 0 0 0 0 9 1 0 0 0 0 18 6 1 0 0 0 0 3 0 0 0 0 0 0 8 1 0 0 0 0 0 0 4 1 0 0 0 0 0 0 0 0 0 56 38 6 0 0 0 0 64 65 65 65 64 64 64 62 53 33 62 61 59 52 27 6 34 22 10 2 61 59 56 48 36 19 6 1 0 0 51 3 0 0 0 0 0 8 1 0 0 0 0 22 9 2 0 0 0 0 4 0 0 0 0 0 0 8 1 0 0 0 0 0 0 5 2 0 0 0 0 0 0 0 0 0 47 26 3 0 0 0 0 56 57 57 57 57 57 55 51 41 23 54 54 52 45 21 4 27 19 10 3 53 51 46 39 28 14 3 1 0 0 42 5 1 0 0 0 0 10 2 0 0 0 0 20 9 3 1 0 0 0 6 1 0 0 0 0 0 12 3 0 0 0 0 0 0 7 3 1 0 0 0 0 0 0 0 0 41 26 5 0 0 0 0 51 50 51 50 50 49 48 45 39 28 46 46 45 38 19 4 26 18 8 2 47 44 40 34 22 11 3 0 0 0 86 55 9 0 0 0 0 75 33 2 0 0 0 66 39 11 1 0 0 0 35 2 0 0 0 0 0 64 16 1 0 0 0 0 0 30 8 1 0 0 0 0 0 0 0 0 83 73 34 2 0 0 0 74 49 12 0 0 0 0 0 0 0 82 66 22 4 0 0 0 0 0 0 80 71 41 9 1 0 0 0 0 0 85 37 3 0 0 0 0 64 17 1 0 0 0 60 29 6 1 0 0 0 16 1 0 0 0 0 0 42 5 0 0 0 0 0 0 15 3 0 0 0 0 0 0 0 0 0 83 74 38 2 0 0 0 75 52 17 1 0 0 0 0 0 0 84 79 56 14 1 0 0 0 0 0 80 75 54 18 3 1 0 0 0 0 85 24 2 0 0 0 0 51 10 0 0 0 0 50 22 5 1 0 0 0 8 0 0 0 0 0 0 32 3 0 0 0 0 0 0 12 3 0 0 0 0 0 0 0 0 0 82 73 37 3 0 0 0 72 49 12 0 0 0 0 0 0 0 83 80 72 41 7 0 0 0 0 0 79 74 57 29 6 1 0 0 0 0 84 21 2 0 0 0 0 48 9 0 0 0 0 49 19 4 1 0 0 0 8 0 0 0 0 0 0 31 3 0 0 0 0 0 0 12 3 0 0 0 0 0 0 0 0 0 81 74 40 3 0 0 0 71 44 9 0 0 0 0 0 0 0 82 80 80 64 23 2 1 1 0 0 78 71 57 30 11 2 1 0 0 0 83 23 2 0 0 0 0 50 9 0 0 0 0 45 18 3 0 0 0 0 9 0 0 0 0 0 0 38 5 0 0 0 0 0 0 17 4 1 0 0 0 0 0 0 0 0 80 72 39 4 0 0 0 70 43 6 0 0 0 0 0 0 0 81 77 73 69 61 12 9 2 0 0 76 70 52 27 10 2 0 0 0 0 83 33 3 0 0 0 0 57 13 1 0 0 0 45 17 3 0 0 0 0 14 1 0 0 0 0 0 51 12 1 0 0 0 0 0 27 9 2 0 0 0 0 0 0 0 0 80 72 38 4 0 0 0 68 39 5 0 0 0 0 0 0 0 80 76 70 66 62 53 21 2 0 0 76 70 52 25 6 1 0 0 0 0 82 34 4 0 0 0 0 58 15 1 0 0 0 41 15 2 0 0 0 0 16 1 0 0 0 0 0 58 16 1 0 0 0 0 0 29 8 1 0 0 0 0 0 0 0 0 79 71 42 6 0 0 0 68 40 6 0 0 0 0 0 0 0 77 73 67 61 57 47 29 3 0 0 75 69 51 25 7 1 0 0 0 0 81 37 4 0 0 0 0 59 15 1 0 0 0 40 15 2 0 0 0 0 17 1 0 0 0 0 0 59 16 1 0 0 0 0 0 32 10 2 0 0 0 0 0 0 0 0 78 70 43 9 0 0 0 69 43 9 0 0 0 0 0 0 0 78 72 64 57 52 50 46 9 0 0 75 71 55 25 7 2 0 0 0 0 79 38 5 0 0 0 0 60 16 1 0 0 0 41 14 2 0 0 0 0 22 1 0 0 0 0 0 63 21 1 0 0 0 0 0 31 10 2 0 0 0 0 0 0 0 0 77 69 45 10 0 0 0 69 47 13 1 0 0 0 0 0 0 76 71 62 54 47 49 61 28 3 0 74 71 60 32 10 3 0 0 0 0 77 35 5 0 0 0 0 56 15 1 0 0 0 36 12 2 0 0 0 0 21 1 0 0 0 0 0 59 20 1 0 0 0 0 0 31 10 2 0 0 0 0 0 0 0 0 75 68 47 13 1 0 0 69 50 21 4 1 0 0 0 0 0 74 69 61 53 46 51 66 45 15 1 74 73 67 50 23 8 2 0 0 0 82 46 5 0 0 0 0 68 24 1 0 0 0 58 29 7 1 0 0 0 27 1 0 0 0 0 0 68 27 1 0 0 0 0 0 41 15 2 0 0 0 0 0 0 0 0 75 52 13 1 0 0 0 74 51 13 0 0 0 0 0 0 0 32 7 2 0 0 0 0 0 0 0 85 76 33 4 0 0 0 0 0 0 81 47 6 0 0 0 0 67 26 1 0 0 0 58 30 7 0 0 0 0 26 1 0 0 0 0 0 65 24 1 0 0 0 0 0 46 18 3 0 0 0 0 0 0 0 0 74 55 16 1 0 0 0 76 59 22 2 0 0 0 0 0 0 51 14 3 1 0 0 0 0 0 0 86 83 63 16 1 0 0 0 0 0 79 43 7 0 0 0 0 62 23 2 0 0 0 53 26 6 0 0 0 0 20 1 0 0 0 0 0 54 16 1 0 0 0 0 0 33 11 1 0 0 0 0 0 0 0 0 73 57 23 2 0 0 0 75 60 28 4 0 0 0 0 0 0 65 30 6 1 0 0 0 0 0 0 85 84 79 51 10 1 0 0 0 0 78 31 4 0 0 0 0 51 14 1 0 0 0 44 18 4 1 0 0 0 14 1 0 0 0 0 0 39 8 1 0 0 0 0 0 22 6 1 0 0 0 0 0 0 0 0 74 61 28 3 0 0 0 75 63 36 10 1 0 0 0 0 0 73 53 17 3 0 0 0 0 0 0 85 84 81 74 40 9 1 0 0 0 76 27 4 0 0 0 0 43 11 1 0 0 0 36 15 4 1 0 0 0 13 1 0 0 0 0 0 36 8 1 0 0 0 0 0 19 6 1 0 0 0 0 0 0 0 0 72 63 32 4 0 0 0 74 64 40 14 2 0 0 0 0 0 74 65 35 8 2 0 0 0 0 0 82 82 81 78 69 39 13 2 0 0 74 22 3 0 0 0 0 34 7 1 0 0 0 26 9 2 0 0 0 0 7 0 0 0 0 0 0 25 5 0 0 0 0 0 0 19 6 1 0 0 0 0 0 0 0 0 69 58 31 4 0 0 0 72 60 35 8 1 0 0 0 0 0 69 58 29 6 1 0 0 0 0 0 80 80 77 69 49 26 10 2 0 0 71 25 4 0 0 0 0 28 6 0 0 0 0 12 3 1 0 0 0 0 9 1 0 0 0 0 0 24 4 0 0 0 0 0 0 16 5 1 0 0 0 0 0 0 0 0 68 60 38 9 1 0 0 70 60 36 12 2 0 0 0 0 0 70 67 53 24 5 1 1 1 0 0 79 78 76 74 65 51 40 20 4 0 67 24 5 1 0 0 0 25 6 1 0 0 0 11 4 1 0 0 0 0 7 1 0 0 0 0 0 20 4 1 0 0 0 0 0 15 6 1 0 0 0 0 0 0 0 0 63 59 42 14 1 0 0 66 55 34 13 4 1 0 0 0 0 64 62 53 29 9 2 2 2 1 0 75 75 74 72 67 61 48 34 16 3 66 34 10 2 0 0 0 41 14 2 0 0 0 25 11 3 1 0 0 0 15 3 0 0 0 0 0 29 8 2 0 0 0 0 0 20 8 3 1 0 0 0 0 0 0 0 64 62 53 27 3 0 0 65 56 37 

L -J P E G = 1 L -J P E G = 2 L 2 -J P E G = 2 L 2 -J P E G = 4 L 2 -J P E G = 8 L 2 -J P E G = 1 6 L 2 -J P E G = 3 2 L 2 -J P E G = 6 4 L 2 -J P E G = 1 2 8 L 2 -J P E G = 2 5 6 L 1 -J P E G = 1 2 8 L 1 -J P E G = 2 5 6 L 1 -J P E G = 5 1 2 L 1 -J P E G =

L ∞ L 2 L 1 L ∞ -J P E G L 2 -J P E G E la st ic F o g G

L ∞ L 2 L 1 L ∞ -J P E G L 2 -J P E G E la st ic F o g G a b o r S n o w Normal Training L2 ε = 150 L2 ε = 300 L2 ε = 600 L2 ε = 1200 L2 ε = 2400 L2 ε = 4800 L2-JPEG ε = 8 L2-JPEG ε = 16 L2-JPEG ε = 32 L2-JPEG ε = 64 L2-JPEG ε = 128 L2-JPEG ε = 256

Gabor ε = 6.25

Gabor ε = 12.5

Gabor ε = 25

Gabor ε = 400

Gabor ε = 800

Gabor ε = 1600 We show the results of adversarial attacks and defenses for CIFAR-10 in Figure 14 . We experienced difficulty training the L 2 and L 1 attacks at distortion sizes greater than those shown and have omitted those runs, which we believe may be related to the small size of CIFAR-10 images.

7

L ∞ L 2 L 1 L ∞ -J P E G L 2 -J P E G E la st ic F o g G

C.3.2 ATA AND UAR FOR CIFAR-10:

The ε calibration procedure for CIFAR-10 was similar to that used for ImageNet-100. We started with the perceptually small ε min values in Table 3 and increased ε geometrically with ratio 2 until adversarial accuracy of an adversarially trained model dropped below 40. Note that this threshold 

D ROBUSTNESS OF OUR RESULTS:



D.1 REPLICATION:

We replicated our results for the first three rows of Figure 11 with different random seeds to see the variation in our results. As shown in Figure 16 , deviations in results are minor.

D.2 CONVERGENCE:

We replicated the results in Figure 11 with 50 instead of 200 steps to see how the results changed based on the number of steps in the attack. As shown in Figure 17 , the deviations are minor.

E FURTHER RESULTS FOR JOINT TRAINING E.1 FULL EXPERIMENTAL RESULTS:

We show the evaluation accuracies of jointly trained models in Figure 18 .

We show all the attacks against the jointly adversarially trained defenses in Figure 19 .

E.2 DEPENDENCE ON RANDOM SEED:

In Table 4 , we study the dependence of joint adversarial training to random seed. We find that at large distortion sizes, joint training for certain pairs of distortions does not produce consistent results over different random initializations. As a first test to understand the relationship between model capacity and overfitting, we trained ResNet-101 models using the same procedure as in Section 5. Briefly, overfitting still occurs, but ResNet-101 achieves a few percentage points higher than ResNet-50.

We show the training curves in Figure 20 and the training and validation numbers in Table 5 .

F COMPARING ADVERSARIAL DISTORTIONS AND COMMON CORRUPTIONS:

"Common" visual corruptions such as (non-adversarial) fog, blur, or pixelation have emerged as another avenue for measuring the robustness of computer vision models (Vasiljevic et al., 2016; Hendrycks & Dietterich, 2019; Geirhos et al., 2018) . Recent work suggests that robustness to such common corruptions is linked to adversarial robustness and proposes corruption robustness as an easily computed indicator of adversarial robustness (Ford et al., 2019) . We consider this alternative to our methodology by testing corruption robustness of our models on the ImageNet-C benchmark.

Experimental setup. We evaluate on the 100-class subset of the corruption robustness benchmark ImageNet-C introduced in (Hendrycks & Dietterich, 2019) with the same classes as ImageNet-100, which we call ImageNet-C-100. It is the ImageNet-100 validation set with 19 common corruptions at 5 severities. We use the JPEG files available at https://github.com/hendrycks/ robustness. We show average accuracies by distortion type in Figure 21 .

Adversarial training against small distortions increases corruption robustness. The first column of each block in Figure 21 shows that training against small adversarial distortions generally increases average accuracy compared to an undefended model. However, training against larger distortions often decreases average accuracy, largely due to the resulting decrease in clean accuracy.

Adversarial distortions and common corruptions can affect defenses differently. Our L p -JPEG and elastic attacks are adversarial versions of the corresponding common corruptions. While training against adversarial JPEG at larger ε improves robustness against adversarial JPEG attacks (Figure 12 in Appendix C.2), Figure 21 shows that robustness against common JPEG corruptions decreases as we adversarially train against JPEG at larger ε, though it remains better than for normally trained models. Similarly, adversarial Elastic training at large ε begins to hurt robustness to its common counterpart. This is likely because common corruptions are easier than adversarial distortions, hence the increased robustness does not make up for the decreased clean accuracy.

G ATTACKS AGAINST UNDEFENDED MODELS:

We show sample images of our attacks against undefended models trained in the normal way in Figure 22 . N o a tt a c k

L 2 - JP E G = 0 . 1 2 5 L 2 - JP E G = 0 . 2 5 L 2 - JP E G = 0 . 5 L 2 - JP E G = 1 L 2 - JP E G = 2 L 2 - JP E G = 4 L 2 - JP E G = 8 L 1 - JP E G = 1 L 1 - JP E G = 2 L 1 - JP E G = 4 L 1 - JP E G = 8 L 1 - JP E G = 1 6 L 1 - JP E G = 3 2 L 1 - JP E G = 6 4 L 1 - JP E G = 1 2 8 L 1 - JP E G = 2 5 6 L 1 - JP E G = 5 1 2 L 1 - JP E G =

L ∞ L 2 L 1 L ∞ -J P E G L 1 -J P E G E la s t ic Normal Training L ∞ ε = 1 L ∞ ε = 2 L ∞ ε = 4 L ∞ ε = 8 L ∞ ε = 16 L ∞ ε = 32 L ∞ -JPEG ε = 0.03125 L ∞ -JPEG ε = 0.0625 L ∞ -JPEG ε = 0.125 L ∞ -JPEG ε = 0.25 L ∞ -JPEG ε = 0.5 L ∞ -JPEG ε =

L ∞ L 2 L 1 L ∞ -J P E G L 1 -J P E G E la s t ic Normal Training L 2 ε = 40 L 2 ε = 80 L 2 ε = 160 L 2 ε = 320 L 2 ε = 640 L 2 ε = 2560 L 1 -JPEG ε = 2 L 1 -JPEG ε = 8 L 1 -JPEG ε = 64 L 1 -JPEG ε = 256 L 1 -JPEG ε = 512 L 1 -JPEG ε =

L ∞ L 2 L 1 L ∞ -J P E G L 1 -J P E G E la s t ic Normal Training L 1 ε = 195 L 1 ε = 390 L 1 ε = 780 L 1 ε = 1560 L 1 ε = 6240 L 1 ε = 49920 Elastic ε = 0.125 Elastic ε = 0.25 Elastic ε = 0.5 Elastic ε = 1 Elastic ε = 2 Elastic ε =

L ∞ ε = 1 L ∞ ε = 2 L ∞ ε = 4 L ∞ ε = 8 L ∞ ε = 1 6 L ∞ ε = 3 2 L 2 ε = 1 5 0 L 2 ε = 3 0 0 L 2 ε = 6 0 0 L 2 ε = 1 2 0 0 L 2 ε = 2 4 0 0 L 2 ε = 4 8 0 0 L 1 ε = 9 5 6 2 .4 4 L 1 ε = 1 9 1 2 5 L 1 ε = 7 6 5 0 0 L 1 ε = 1 5 3 0 0 0 L 1 ε = 3 0 6 0 0 0 L 1 ε = 6 1 2 0 0 0 L ∞ -J P E G ε = 0 .0 3 1 2 5 L ∞ -J P E G ε = 0 .1 2 5 L ∞ -J P E G ε = 0 .2 5 L ∞ -J P E G ε = 0 .5 L ∞ -J P E G ε = 1 L 2 -J P E G ε = 2 L 2 -J P E G ε = 1 6 L 2 -J P E G ε = 3 2 L 2 -J P E G ε = 6 4 L 2 -J P E G ε = 1 2 8 L 2 -J P E G ε = 2 5 6 E la st ic ε = 0 .2 5 E la st ic ε = 1 E la st ic ε = 2 E la st ic ε = 4 E la st ic ε = 8 E la st ic ε = 1 6

Attack (evaluation)

L∞ ε = 1 L∞ ε = 2 L∞ ε = 4 L∞ ε = 8 L∞ ε = 16 L∞ ε = 32 L2 ε = 150 L2 ε = 300 L2 ε = 600 L2 ε = 1200 L2 ε = 2400 L2 ε = 4800 L1 ε = 9562.44 L1 ε = 19125 L1 ε = 38250.1 L1 ε = 76500 L1 ε = 153000 L1 ε = 306000 L1 ε = 612000

Attack (adversarial training) 74 59 24 2 0 0 0 64 35 6 0 0 0 36 20 7 2 0 0 0 30 4 0 0 0 0 0 56 24 3 0 0 0 0 0 39 23 8 2 0 0 0 0 0 0 0 74 73 72 72 70 63 50 64 44 19 4 1 0 0 0 0 0 69 57 37 18 6 2 1 1 1 1 63 60 52 38 23 12 5 2 1 1 87 42 2 0 0 0 0 70 19 1 0 0 0 68 40 11 2 0 0 0 25 1 0 0 0 0 0 74 31 1 0 0 0 0 0 58 29 7 1 0 0 0 0 0 0 0 81 57 12 1 0 0 0 85 78 52 12 1 0 0 0 0 0 24 7 2 1 0 0 0 0 0 0 76 49 17 2 0 0 0 0 0 0 88 41 3 0 0 0 0 69 19 1 0 0 0 66 37 9 1 0 0 0 20 0 0 0 0 0 0 72 25 1 0 0 0 0 0 56 25 5 0 0 0 0 0 0 0 0 82 61 16 1 0 0 0 87 82 71 40 6 0 0 0 1 0 32 8 2 1 0 0 0 0 0 0 79 59 23 3 0 0 0 0 0 0 87 31 2 0 0 0 0 62 14 0 0 0 0 64 33 7 1 0 0 0 10 0 0 0 0 0 0 59 13 0 0 0 0 0 0 42 13 2 0 0 0 0 0 0 0 0 82 62 19 1 0 0 0 86 84 80 66 32 6 1 1 1 1 37 9 2 0 0 0 0 0 0 0 81 64 30 6 0 0 0 0 0 0 86 25 1 0 0 0 0 53 10 0 0 0 0 61 31 7 0 0 0 0 7 0 0 0 0 0 0 48 9 0 0 0 0 0 0 43 15 2 0 0 0 0 0 0 0 0 82 64 21 2 0 0 0 86 85 82 76 60 30 8 2 1 1 48 14 4 1 0 0 0 0 0 0 81 68 39 9 1 0 0 0 0 0 85 19 1 0 0 0 0 44 8 0 0 0 0 58 30 6 1 0 0 0 5 0 0 0 0 0 0 38 5 0 0 0 0 0 0 41 14 2 0 0 0 0 0 0 0 0 81 66 23 2 0 0 0 84 84 82 78 70 54 30 12 3 1 60 25 6 1 0 0 0 0 0 0 81 72 44 11 1 0 0 0 0 0 78 9 0 0 0 0 0 26 3 0 0 0 0 37 12 1 0 0 0 0 2 0 0 0 0 0 0 25 3 0 0 0 0 0 0 25 7 1 0 0 0 0 0 0 0 0 74 60 23 2 0 0 0 79 79 78 76 73 68 58 38 12 3 67 46 18 4 1 0 0 0 0 0 75 70 50 20 4 0 0 0 0 0 69 4 0 0 0 0 0 14 2 0 0 0 0 36 16 4 1 0 0 0 5 0 0 0 0 0 0 14 2 0 0 0 0 0 0 9 2 0 0 0 0 0 0 0 0 0 64 50 12 1 0 0 0 70 70 70 70 69 69 67 61 45 24 68 67 64 53 23 6 8 6 4 1 66 65 58 44 21 4 1 0 0 0 62 3 0 0 0 0 0 10 1 0 0 0 0 27 12 3 0 0 0 0 3 0 0 0 0 0 0 9 1 0 0 0 0 0 0 8 2 0 0 0 0 0 0 0 0 0 56 38 7 0 0 0 0 65 65 66 66 66 66 66 64 55 35 62 61 60 55 36 18 44 39 29 15 61 60 54 44 28 10 1 0 0 1 51 3 1 0 0 0 0 9 1 0 0 0 0 29 14 4 1 0 0 0 4 0 0 0 0 0 0 8 1 0 0 0 0 0 0 8 3 1 0 0 0 0 0 0 0 0 46 27 3 0 0 0 0 57 58 59 58 59 59 57 53 43 24 54 54 53 47 28 12 35 32 24 14 53 51 46 36 18 5 1 0 0 0 42 6 1 0 0 0 0 10 2 0 0 0 0 26 14 5 1 0 0 0 7 1 0 0 0 0 0 12 3 1 0 0 0 0 0 12 4 1 0 0 0 0 0 0 0 0 41 26 5 0 0 0 0 51 51 52 51 51 51 49 47 40 29 47 47 46 41 24 12 35 30 22 12 47 45 40 30 16 4 1 0 0 0 86 56 10 0 0 0 0 75 35 3 0 0 0 70 51 22 4 0 0 0 37 3 0 0 0 0 0 65 20 1 0 0 0 0 0 43 16 3 0 0 0 0 0 0 0 0 83 73 37 3 0 0 0 78 61 25 3 0 0 0 0 0 0 83 71 32 7 1 0 0 0 0 0 80 71 44 12 2 0 0 0 0 0 85 39 4 0 0 0 0 64 19 1 0 0 0 66 42 15 3 0 0 0 18 1 0 0 0 0 0 46 6 0 0 0 0 0 0 27 7 1 0 0 0 0 0 0 0 0 82 74 41 4 0 0 0 78 63 31 7 1 0 0 0 0 0 84 81 65 25 4 1 1 1 0 0 80 75 58 23 4 1 0 0 0 0 85 26 2 0 0 0 0 53 12 1 0 0 0 60 35 12 2 0 0 0 10 0 0 0 0 0 0 34 4 0 0 0 0 0 0 21 6 1 0 0 0 0 0 0 0 0 82 74 41 4 0 0 0 76 58 23 3 0 0 0 0 0 0 83 82 79 54 13 1 1 1 1 0 79 75 61 33 8 2 0 0 0 0 84 23 2 0 0 0 0 50 10 1 0 0 0 58 34 11 2 0 0 0 10 1 0 0 0 0 0 34 4 0 0 0 0 0 0 22 6 1 0 0 0 0 0 0 0 0 82 73 43 5 0 0 0 75 57 21 2 0 0 0 0 0 0 83 82 81 78 48 5 4 3 2 0 78 74 62 39 15 4 1 0 0 0 83 26 2 0 0 0 0 52 10 1 0 0 0 57 30 9 1 0 0 0 10 0 0 0 0 0 0 40 6 0 0 0 0 0 0 28 9 2 0 0 0 0 0 0 0 0 80 73 41 5 0 0 0 75 56 19 1 0 0 0 0 0 0 81 80 79 80 78 42 28 20 8 2 77 71 57 35 13 4 1 0 0 0 83 34 4 0 0 0 0 57 15 1 0 0 0 53 29 9 1 0 0 0 17 1 0 0 0 0 0 53 14 1 0 0 0 0 0 39 16 4 1 0 0 0 0 0 0 0 80 72 42 7 0 0 0 73 53 17 1 0 0 0 0 0 0 81 79 78 79 80 78 64 36 8 1 77 71 55 31 10 2 0 0 0 0 82 36 4 0 0 0 0 58 16 1 0 0 0 50 24 7 1 0 0 0 18 1 0 0 0 0 0 59 18 1 0 0 0 0 0 39 17 4 1 0 0 0 0 0 0 0 79 71 44 8 0 0 0 72 54 18 1 0 0 0 0 0 0 79 77 76 76 76 78 73 45 11 1 76 70 56 32 9 3 1 0 0 1 81 38 5 0 0 0 0 60 17 1 0 0 0 49 23 6 1 0 0 0 19 1 0 0 0 0 0 60 18 1 0 0 0 0 0 42 20 5 1 0 0 0 0 0 0 0 78 71 45 11 0 0 0 73 55 21 2 0 0 0 0 0 0 79 76 74 73 75 77 77 62 24 3 75 71 58 30 10 3 1 0 0 0 79 40 6 0 0 0 0 60 18 1 0 0 0 49 24 6 0 0 0 0 24 1 0 0 0 0 0 63 23 2 0 0 0 0 0 41 18 5 1 0 0 0 0 0 0 0 76 70 47 13 0 0 0 72 55 25 5 1 0 0 0 0 0 77 74 71 70 72 76 79 73 49 12 75 71 62 37 15 5 2 0 0 0 77 36 6 0 0 0 0 57 17 1 0 0 0 44 19 4 1 0 0 0 23 1 0 0 0 0 0 60 21 2 0 0 0 0 0 40 19 5 1 0 0 0 0 0 0 0 74 68 48 16 1 0 0 71 58 31 10 2 1 1 1 0 0 75 72 70 69 69 75 78 75 65 36 74 73 69 52 26 10 4 1 0 0 82 47 6 0 0 0 0 69 26 1 0 0 0 62 41 14 3 0 0 0 29 2 0 0 0 0 0 68 29 2 0 0 0 0 0 48 24 5 1 0 0 0 0 0 0 0 74 54 17 2 0 0 0 78 65 32 5 0 0 0 0 0 0 37 11 3 1 0 0 0 0 0 0 85 75 37 6 1 0 0 0 0 0 81 48 8 0 0 0 0 68 29 2 0 0 0 63 39 14 2 0 0 0 28 1 0 0 0 0 0 65 27 2 0 0 0 0 0 52 28 7 1 0 0 0 0 0 0 0 75 55 20 3 0 0 0 79 69 39 8 0 0 0 0 0 0 56 20 5 1 0 0 0 0 0 0 86 82 65 18 2 0 0 0 0 0 79 44 8 0 0 0 0 63 25 2 0 0 0 58 36 13 2 0 0 0 22 2 0 0 0 0 0 56 18 1 0 0 0 0 0 41 17 4 1 0 0 0 0 0 0 0 74 58 28 4 0 0 0 78 68 42 12 2 0 0 0 0 0 67 38 10 3 1 0 0 0 0 0 85 84 80 53 9 1 0 0 0 0 78 33 5 0 0 0 0 53 16 1 0 0 0 51 29 10 2 0 0 0 15 1 0 0 0 0 0 41 10 1 0 0 0 0 0 30 12 3 0 0 0 0 0 0 0 0 74 61 31 5 0 0 0 78 69 45 18 4 1 0 0 0 0 75 60 25 7 1 0 1 0 0 0 85 84 83 76 44 8 1 0 0 0 76 27 5 0 0 0 0 44 12 1 0 0 0 43 23 7 2 0 0 0 16 2 0 0 0 0 0 38 9 1 0 0 0 0 0 28 12 3 1 0 0 0 0 0 0 0 73 63 36 6 0 0 0 76 69 48 22 6 1 1 1 0 0 74 68 44 14 3 1 1 1 0 0 82 82 81 79 71 44 10 1 0 0 74 23 4 0 0 0 0 35 8 1 0 0 0 34 14 4 1 0 0 0 8 1 0 0 0 0 0 27 5 0 0 0 0 0 0 28 12 3 0 0 0 0 0 0 0 0 69 60 33 6 0 0 0 74 66 45 18 4 1 0 0 0 0 70 61 39 14 4 2 2 1 1 0 80 80 77 69 51 30 14 2 0 0 71 27 5 0 0 0 0 30 6 1 0 0 0 18 6 2 0 0 0 0 9 1 0 0 0 0 0 25 5 1 0 0 0 0 0 25 11 3 1 0 0 0 0 0 0 0 68 60 40 11 1 0 0 72 64 45 21 6 1 1 1 0 0 70 68 59 35 9 3 3 2 2 1 78 78 78 75 64 50 39 19 3 1 67 26 6 1 0 0 0 27 6 1 0 0 0 17 6 2 0 0 0 0 9 1 0 0 0 0 0 21 5 1 0 0 0 0 0 25 11 3 1 0 0 0 0 0 0 0 64 58 42 16 2 0 0 67 60 41 19 7 2 1 0 1 1 65 62 57 41 18 8 8 5 3 2 75 75 74 72 68 60 51 39 14 2 66 36 11 2 0 0 0 41 16 3 0 0 0 32 17 6 1 0 0 0 16 3 0 0 0 0 0 30 9 2 0 0 0 0 0 29 15 6 2 0 0 0 0 0 0 0 64 61 54 29 4 1 0 66 60 43 

:

We describe the attacks for ImageNet-100, but they can also be applied to CIFAR-10.

As explained in Figure 13 (Appendix C.2), this usually requires training at distortion size ε > ε because the typical distortion seen during adversarial training is sub-maximal.

We exclude defenses adversarially trained against A and A to ensure that attacks are unforeseen.

Figures and Tables:

Figure 4: Accuracies of L 2 and Elastic attacks at different distortion sizes against a ResNet-50 model adversarially trained against L 2 at ε = 9600 on ImageNet-100. At small distortion sizes, the model appears to defend well against Elastic, but large distortion sizes reveal a lack of transfer.

For distortion type A and size ε, let the Adversarial Training Accuracy ATA(A, ε) be the best adversarial accuracy on the test set that can be achieved by adversarially training a specific architecture (ResNet-50 for ImageNet-100, ResNet-56 for CIFAR-10) against A. 2 Even when evaluating a defense using an architecture clean vs. clean vs. ε = 2 vs. ε = 8 vs. ε = 16 vs. ε = 32 (a) The L∞ attack at ε = 32 applied to undefended model and models adversarially trained against L∞ at different distortion sizes. Attacking models trained against larger ε produces greater visual distortion.

Figure 5: Varying distortion size against adversarially trained models reveals full attack strength.

UAR scores for adv. trained defenses (rows) against attacks (columns) on ImageNet-100. See Figure12for more ε values and Appendix C.3.2 for CIFAR-10 results.

Correlation of UAR scores (b) Correlations between UAR scores in Figure6afor each attack (rows and columns). Correlation was computed over adversarial defenses in Figure6atrained without knowledge of the attacks (6 total per pair).

Figure 6: UAR scores demonstrate the need to evaluate against diverse attacks.

Figure 7: UAR scores for jointly adv. trained defenses (rows) against distortion types (columns).

Figure 8: Left: train and validation curves for joint training against L ∞ , ε = 8 and Elastic, ε = 4, Right: train and val curves for standard adversarial training for L ∞ , ε = 8. The joint validation accuracy of L ∞ decreases as training progresses, indicating overfitting.

Figure9: Differences of the attacked images and original image for different attacks (label "espresso maker"). The L 1 , L 2 , and L ∞ norms of the difference are shown in parentheses. As shown, our novel attacks display qualitatively different behavior and do not fall under the L p threat model. These differences are not scaled and are normalized so that no difference corresponds to white.

Figure 10: A comparison of L 1 -JPEG and L 2 -JPEG attacks.

L = 1 L = 2 L = 4 L = 8 L = 16 L = 32 L2 = 150 L2 = 300 L2 = 600 L2 = 1200 L2 = 2400 L2 = 4800 L1 = 9562.44 L1 = 19125 L1 = 38250.1 L1 = 76500 L1 = 153000 L1 = 306000 L1 = 612000 L -JPEG = 0.03125 L -JPEG = 0.0625 L -JPEG = 0.125 L -JPEG = 0.25 L -JPEG = 0.5 L -JPEG = 1 L -JPEG = 2 L2-JPEG = 2 L2-JPEG = 4 L2-JPEG = 8 L2-JPEG = 16 L2-JPEG = 32 L2-JPEG = 64 L2-JPEG = 128 L2-JPEG = 256 L1-JPEG = 128 L1-JPEG = 256 L1-JPEG = 512 L1-JPEG = 1024 L1-JPEG = 2048 L1-JPEG = 4096 L1-JPEG = 8192 L1-JPEG = 16384 L1-JPEG = 32768 L1-JPEG = 65536 L1-JPEG = 131072 Elastic = 0.25 Elastic = 74 56 6 0 77 77 76 73 50 2 77 76 75 71 57 23 1 77 76 75 63 9 0 0 77 76 76 74 57 7 0 0 77 76 73 68 52 25 5 1 0 0 0 76 74 68 38 4 0 0 59 30 6 1 1 1 0 1 0 0 76 74 61 16 1 0 0 0 0 0 70 68 58 35 10 2 0 0 0 0 68 68 68 67 61 28 1 68 68 68 67 59 20 69 68 68 66 61 44 13 68 68 67 64 37 2 0 68 68 68 67 62 33 1 0 68 68 68 66 61 49 30 12 4 2 3 68 66 63 47 11 1 0 49 23 5 1 1 1 0 1 0 0 68 67 61 30 3

Figure 11: Accuracy of adversarial attack (column) against adversarially trained model (row) on ImageNet-100.

Figure 12: UAR scores (multiplied by 100) for adv. trained defenses (rows) against distortion types (columns) for ImageNet-100.

Figure 14: Accuracy of adversarial attack (column) against adversarially trained model (row) on CIFAR-10.

Figure 15: UAR scores on CIFAR-10. Displayed UAR scores are multiplied by 100 for clarity.

Figure 16: Replica of the first three block rows of Figure 11 with different random seeds. Deviations in results are minor.

Figure 17: Replica of Figure 11 with 50 steps instead of 200 at evaluation time. Deviations in results are minor.

N o a tt a c k A v e ra g e G a u ss ia n N o is e S h o t N o is e Im p u ls e N o is e D e fo c u s B lu r G la ss B lu r M o ti o n B lu r Z o o m B lu r S n o w F ro st F o g B ri g h tn e ss C o n tr a st E la st ic P ix e la te J P E G S p e c k le N o is e G a u ss ia n B lu r S p a tt e r S a tu ra te Normal training L∞-JPEG ε = 0.03125 L∞-JPEG ε = 0.125 L∞-JPEG ε = 0.25 L∞-JPEG ε = 0.5L∞-JPEG ε = 1 L∞-JPEG ε = 2 L2-JPEG ε = 2 L2-JPEG ε = 16 L2-JPEG ε = 32 L2-JPEG ε = 64 L2-JPEG ε = 128 L2-JPEG ε = 256 Elastic ε = 0.25 Elastic ε = 1 Elastic ε = 2 Elastic ε = 4 Elastic ε = 8 Elastic ε =

Figure 21: Accuracies of defenses (rows) on ImageNet-C-100 corruptions (columns).

Calibrated distortion sizes and ATA values for different distortion types on ImageNet-100. ATA values for CIFAR-10 are shown in Table 3 (Appendix C.3.2).

Table content: <table><tr><td colspan="2">Attack ε1</td><td>ε2</td><td>ε3</td><td>ε4</td><td>ε5</td><td>ε6</td><td>ATA1 ATA2 ATA3 ATA4 ATA5 ATA6</td></tr><tr><td>L∞</td><td>1</td><td>2</td><td>4</td><td>8</td><td>16</td><td>32</td><td>84.6 82.1 76.2 66.9 40.1 12.9</td></tr><tr><td>L2</td><td>150</td><td>300</td><td colspan="5">600 1200 2400 4800 85.0 83.5 79.6 72.6 59.1 19.9</td></tr><tr><td>L1</td><td colspan="7">9562.5 19125 76500 153000 306000 612000 84.4 82.7 76.3 68.9 56.4 36.1</td></tr><tr><td colspan="4">Elastic 0.250 0.500 2</td><td>4</td><td>8</td><td>16</td><td>85.9 83.2 78.1 75.6 57.0 22.5</td></tr><tr><td colspan="6">JPEG 0.062 0.125 0.250 0.500 1</td><td>2</td><td>85.0 83.2 79.3 72.8 34.8 1.1</td></tr><tr><td>Fog</td><td>128</td><td>256</td><td colspan="5">512 2048 4096 8192 85.8 83.8 79.0 68.4 67.9 64.7</td></tr><tr><td colspan="5">Snow 0.062 0.125 0.250 2</td><td>4</td><td>8</td><td>84.0 81.1 77.7 65.6 59.5 41.2</td></tr><tr><td colspan="4">Gabor 6.250 12.500 25</td><td>400</td><td>800</td><td colspan="2">1600 84.0 79.8 79.8 66.2 44.7 14.6</td></tr><tr><td colspan="8">4 UAR REVEALS THE NEED TO EVALUATE AGAINST MORE DIVERSE ATTACKS</td></tr></table>

Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. arXiv preprint arXiv:1801.02612, 2018. Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. arXiv preprint arXiv:1812.03411, 2018. Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural networks. In International Conference on Machine Learning (ICML), 2019.

Table content: <table/>

ATA values for L 1 -JPEG and L 2 -JPEG on ImageNet-100.

Table content: <table><tr><td>Attack</td><td>ε1 ε2</td><td>ε3</td><td>ε4</td><td>ε5</td><td>ε6</td><td>ATA1 ATA2 ATA3 ATA4 ATA5 ATA6</td></tr><tr><td colspan="4">L2-JPEG 8 16 32 64</td><td colspan="2">128 256</td><td>84.8 82.5 78.9 72.3 47.5 3.4</td></tr><tr><td colspan="7">L1-JPEG 256 1024 4096 16384 65536 131072 84.8 81.8 76.2 67.1 46.4 41.8</td></tr></table>

Calibrated distortion sizes and ATA values for ResNet-56 on CIFAR-10 CIFAR-10 because there are fewer classes. The resulting ATA and UAR values for CIFAR10 are shown in Table3and Figure15. We omitted calibration for the L 2 -JPEG attack because we chose too small a range of ε for our initial training experiments, and we plan to address this issue in the future.

Table content: <table><tr><td>Attack</td><td>ε 1</td><td>ε 2</td><td>ε 3</td><td>ε 4</td><td>ε 5</td><td>ε 6</td><td>ATA 1 ATA 2 ATA 3 ATA 4 ATA 5 ATA 6</td></tr><tr><td>L ∞ L 2</td><td>1 40</td><td>2 80</td><td colspan="5">4 160 320 640 2560 90.1 86.4 79.6 67.3 49.9 17.3 8 16 32 91.0 87.8 81.6 71.3 46.5 23.1</td></tr><tr><td>L 1</td><td>195</td><td>390</td><td colspan="5">780 1560 6240 24960 92.2 90.0 83.2 73.8 47.4 35.3</td></tr><tr><td colspan="8">89.7 87.0 83.1 78.6 69.7 35.4 256 512 1024 91.4 88.1 80.2 68.9 56.3 37.7 L ∞ -JPEG 0.03125 0.0625 0.125 0.25 0.5 1 L 1 -JPEG 2 8 64</td></tr><tr><td>Elastic</td><td colspan="4">0.125 0.25 0.5 1</td><td>2</td><td>8</td><td>87.4 81.3 72.1 58.2 45.4 27.8</td></tr><tr><td>is higher for</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Train and val accuracies for joint adversarial training at large distortion are dependent on seed. For train and val, ε is chosen uniformly at random between 0 and ε, and we used 10 steps for L ∞ and L 1 and 30 steps for elastic. Single adversarial training baselines are also shown.

Table content: <table><tr><td>Training parameters (ResNet-50)</td><td colspan="4">L∞ train other train L∞ val other val</td></tr><tr><td>L∞ ε = 8, Elastic ε = 4, Seed 1</td><td>90</td><td>89</td><td>35</td><td>74</td></tr><tr><td>L∞ ε = 8, Elastic ε = 4, Seed 2</td><td>89</td><td>90</td><td>47</td><td>44</td></tr><tr><td>L∞ ε = 8, Elastic ε = 4, Seed 3</td><td>90</td><td>89</td><td>29</td><td>63</td></tr><tr><td colspan="2">L∞ ε = 16, L1 ε = 612000, Seed 1 86</td><td>87</td><td>22</td><td>16</td></tr><tr><td colspan="2">L∞ ε = 16, L1 ε = 612000, Seed 2 88</td><td>87</td><td>16</td><td>24</td></tr><tr><td>L∞ ε = 8</td><td>81</td><td>-</td><td>74</td><td>-</td></tr><tr><td>L∞ ε = 16</td><td>68</td><td>-</td><td>63</td><td>-</td></tr><tr><td>Elastic ε = 4</td><td>-</td><td>88</td><td>-</td><td>76</td></tr><tr><td>L1 ε = 612000</td><td>-</td><td>75</td><td>-</td><td>59</td></tr></table>

Training and validation numbers for ResNet-101 and ResNet-50 for joint training against L ∞ , ε = 8 and elastic, ε = 4.

Table content: <table><tr><td>Training parameters L ∞ ε = 8, Elastic ε = 4, ResNet-50 Seed 1 90 L ∞ train other train L ∞ val other val 89 35 74 90 47 44 L ∞ ε = 8, Elastic ε = 4, ResNet-50 Seed 2 89 L ∞ ε = 8, Elastic ε = 4 ResNet-101 90 91 49 46</td></tr><tr><td>E.3 OVERFITTING AND MODEL CAPACITY</td></tr></table>

Peer review (iclr2020):
The paper investigates empirically to what extent adversarial training with respect to a certain set of adversaries (e.g. Projected Gradient Descent (PGD) with a maximum l_inf perturbation of epsilon) also provides robustness against types of adversaries (e.g. PGD with different l_p norm constraints). While this problem has been studied before in terms of robustness transferability between different l_p constrained adversaries, this paper proposes a few novel attacks (*JPEG*, *Fog*, *Garbor*, *Snow*) to be included in such analysis. It shows that, while robustness from/against l_p constrained adversaries transfers relatively well, this is not the case for some of the newly proposed adversaries. It therefore proposes a framework, including a new canon of adversaries (including in particular the *Fog* attack), for a more broad assessment of empirical robustness against a variety of adversaries. The paper contains a lot of material that should be interesting to browse through for any researcher working the area of adversarial machine learning. The number of experiments carried out and reported on is impressive. The purpose of broadly and thoroughly assessing empirical robustness against a variety of adversarial attacks is important. While the paper makes a convincing case for the importance, some of the concrete steps that it*s trying to promote still seem preliminary to me. For instance, how complete is the set of adversaries in the proposed UAR framework? In the sense: isn*t it to be expected that the set of sources considered in Figure 6 (b) will have to be extended, e.g. once a new adversaries has been discovered that has low correlation with any of the known adversaries? In that sense, testing robustness against unforeseen/unknown adversaries still appears to be an open problem, unless the space of adversaries is constrained and it is established that any unknown adversary is necessarily correlated with the known ones. On the methodological side, the calibration of distortion sizes (p.4) was not clear to me. In 1.: what exactly means *comparable*? Does 2.a (*images which confuse humans*) involve surveys / user studies? And what does it mean in 2.b: *reduce accuracy of adversarially trained models to below 25*?