Year: 2020

Abstract:
Most existing adversarial defenses only measure robustness to Lp adversarial attacks. Not only are adversaries unlikely to exclusively create small Lp perturbations, adversaries are unlikely to remain fixed. Adversaries adapt and evolve their attacks; hence adversarial defenses must be robust to a broad range of unforeseen attacks. We address this discrepancy between research and reality by proposing a new evaluation framework called ImageNet-UA. Our framework enables the research community to test ImageNet model robustness against attacks not encountered during training. To create ImageNet-UA’s diverse attack suite, we introduce a total of four novel adversarial attacks. We also demonstrate that, in comparison to ImageNet-UA, prevailing L∞ robustness assessments give a narrow account of adversarial robustness. By evaluating current defenses with ImageNet-UA, we find they provide little robustness to unforeseen attacks. We hope the greater variety and realism of ImageNet-UA enables development of more robust defenses which can generalize beyond attacks seen during training.

heading: 1 INTRODUCTION
text: Neural networks perform well on many datasets (He et al., 2016) yet can be consistently fooled by minor adversarial distortions (Goodfellow et al., 2014). The research community has responded by quantifying and developing adversarial defenses against such attacks (Madry et al., 2017), but these defenses and metrics have two key limitations.
First, the vast majority of existing defenses exclusively defend against and quantify robustness to Lp-constrained attacks (Madry et al., 2017; Cohen et al., 2019; Raff et al., 2019; Xie et al., 2018). Though real-world adversaries are not Lp constrained (Gilmer et al., 2018) and can attack with diverse distortions (Brown et al., 2017; Sharif et al., 2019), the literature largely ignores this and evaluates against the Lp adversaries already seen during training (Madry et al., 2017; Xie et al., 2018), resulting in optimistic robustness assessments. The attacks outside the Lp threat model that have been proposed (Song et al., 2018; Qiu et al., 2019; Engstrom et al., 2017; Evtimov et al., 2017; Sharif et al., 2016) are not intended for general defense evaluation and suffer from narrow dataset applicability, difficulty of optimization, or fragility of auxiliary generative models.
Second, existing defenses assume that attacks are known in advance (Goodfellow, 2019) and use knowledge of their explicit form during training (Madry et al., 2017). In practice, adversaries can deploy unforeseen attacks not known to the defense creator. For example, online advertisers use attacks such as perturbed pixels in ads to defeat ad blockers trained only on the previous generation of ads in an ever-escalating arms race (Tramèr et al., 2018). However, current evaluation setups implicitly assume that attacks encountered at test-time are the same as those seen at train-time, which is unrealistic. The reality that future attacks are unlike those encountered during training is akin to a train-test distribution mismatch—a problem studied outside of adversarial robustness (Recht et al., 2019; Hendrycks & Dietterich, 2019)—but now brought to the adversarial setting.
The present work addresses these limitations by proposing an evaluation framework ImageNet-UA to measure robustness against unforeseen attacks. ImageNet-UA assesses a defense which may have been created with knowledge of the commonly used L∞ or L2 attacks with six diverse attacks (four of which are novel) distinct from L∞ or L2. We intend these attacks to be used at test-time only and not during training. Performing well on ImageNet-UA thus demonstrates generalization to a diverse set of distortions not seen during defense creation. While ImageNet-UA
does not provide an exhaustive guarantee over all conceivable attacks, it evaluates over a diverse unforeseen test distribution similar to those used successfully in other studies of distributional shift (Rajpurkar et al., 2018; Hendrycks & Dietterich, 2019; Recht et al., 2019). ImageNet-UA works for ImageNet models and can be easily used with our code available at https://github.com/ anon-submission-2020/anon-submission-2020.
Designing ImageNet-UA requires new attacks that are strong and varied, since real-world attacks are diverse in structure. To meet this challenge, we contribute four novel and diverse adversarial attacks which are easily optimized. Our new attacks produce distortions with occlusions, spatial similarity, and simulated weather, all of which are absent in previous attacks. Performing well on ImageNet-UA thus demonstrates that a defense generalizes to a diverse set of distortions distinct from the commonly used L∞ or L2.
With ImageNet-UA, we show weaknesses in existing evaluation practices and defenses through a study of 8 attacks against 48 models adversarially trained on ImageNet-100, a 100-class subset of ImageNet. While most adversarial robustness evaluations use only L∞ attacks, ImageNet-UA reveals that models with high L∞ attack robustness can remain susceptible to other attacks. Thus, L∞ evaluations are a narrow measure of robustness, even though much of the literature treats this evaluation as comprehensive (Madry et al., 2017; Qian & Wegman, 2019; Schott et al., 2019; Zhang et al., 2019). We address this deficiency by using the novel attacks in ImageNet-UA to evaluate robustness to a more diverse set of unforeseen attacks. Our results demonstrate that L∞ adversarial training, the current state-of-the-art defense, has limited generalization to unforeseen adversaries, and is not easily improved by training against more attacks. This adds to the evidence that achieving robustness against a few train-time attacks is insufficient to impart robustness to unforeseen test-time attacks (Jacobsen et al., 2019; Jordan et al., 2019; Tramèr & Boneh, 2019).
In summary, we propose the framework ImageNet-UA to measure robustness to a diverse set of attacks, made possible by our four new adversarial attacks. Since existing defenses scale poorly to multiple attacks (Jordan et al., 2019; Tramèr & Boneh, 2019), finding defense techniques which generalize to unforeseen attacks is crucial to create robust models. We suggest ImageNet-UA as a way to measure progress towards this goal.

heading: 2 RELATED WORK
text: Adversarial robustness is notoriously difficult to correctly evaluate (Papernot et al., 2017; Athalye et al., 2018a). To that end, Carlini et al. (2019a) provide extensive guidance for sound adversarial robustness evaluation. By measuring attack success rates across several distortion sizes and using a broader threat model with diverse differentiable attacks, ImageNet-UA has several of their recommendations built-in, while greatly expanding the set of attacks over previous work on evaluation.
We are only aware of a few prior works which evaluate on unforeseen attacks in specific limited circumstances. Wu et al. (2020) evaluate against physically-realizable attacks from Evtimov et al. (2017) and Sharif et al. (2016), though this limits the threat model to occlusion attacks on narrow datasets. Outside of vision, Pierazzi et al. (2020) proposes constraining attacks by a more diverse set of problem-space constraints in diverse domains such as text and malware or source code generation; however, even in this framework, analytically enumerating all such constraints is impossible.
Within vision, prior attacks outside the Lp threat model exist, but they lack the general applicability and fast optimization of ours. Song et al. (2018) and Qiu et al. (2019) attack using variational autoencoders and StarGANs, respectively, resulting in weaker attacks which require simple image distributions suitable for VAEs and GANs. Engstrom et al. (2017) apply Euclidean transformations determined by brute-force search. Zhao et al. (2019) use perceptual color distances to align human perception and L2 perturbations. Evtimov et al. (2017) and Sharif et al. (2016) attack stop signs and face-recognition systems with carefully placed patches or modified eyeglass frames, requiring physical object creation and applying only to specific image types.

heading: 3 NEW ATTACKS FOR A BROADER THREAT MODEL
text: There are few diverse, easily optimizable, plug-and-play adversarial attacks in the current literature; outside of Elastic (Xiao et al., 2018), most are Lp attacks such as L∞ (Goodfellow et al., 2014), L2 (Szegedy et al., 2013; Carlini & Wagner, 2017), L1 (Chen et al., 2018). We rectify this deficiency with four novel adversarial attacks: JPEG, Fog, Snow, and Gabor. Our attacks are differentiable and fast, while optimizing over enough parameters to be strong. We show example adversarial images in Figure 1 and compare stochastic and adversarial distortions in Figure 2.
Our novel attacks provide a range of test-time adversaries visually and semantically distinct from L∞ and L2 attacks. Namely, they cause distortions with large L∞ and L2 norm, but result in images that are perceptually close to the original. These attacks are intended as unforeseen attacks not used during training, allowing them to evaluate whether a defense can generalize from L∞ or L2 to a more varied set of distortions than current evaluations. Though our attacks are not exhaustive, performing well against them already demonstrates robustness to occlusion, spatial similarity, and simulated weather, which are absent from previous evaluations.
Our attacks create an adversarial image x′ from a clean image x with true label y. Let model f map images to a softmax distribution, and let `(f(x), y) be the cross-entropy loss. Given a target class y′ 6= y, our attacks attempt to find a valid image x′ such that (1) the attacked image x′ is obtained by applying a distortion (of size controlled by a parameter ε) to x, and (2) the loss `(f(x′), y′) is minimized. An unforeseen adversarial attack is a white- or black-box adversarial attack unknown to the defense designer which does not change the true label of x according to an oracle or human.

heading: 3.1 FOUR NEW UNFORESEEN ATTACKS
text: JPEG. JPEG applies perturbations in a JPEG-encoded space of compressed images rather than raw pixel space. More precisely, JPEG compression is a linear transform JPEG which applies colorspace conversion, the discrete cosine transform, and then quantization. Our JPEG attack imposes the L∞-constraint
‖JPEG(x)− JPEG(x′)‖∞ ≤ ε
on the attacked image x′. We optimize z = JPEG(x′) under this constraint to find an adversarial perturbation in the resulting frequency space. The perturbed frequency coefficients are quantized, and we then apply a right-inverse of JPEG to obtain the attacked image x′ in pixel space. We use ideas from Shin & Song (2017) to make this differentiable. The resulting attack is conspicuously distinct from Lp attacks.
Fog. Fog simulates worst-case weather conditions. Robustness to adverse weather is a safety critical priority for autonomous vehicles, and Figure 2 shows Fog provides a more rigorous stress-test than stochastic fog (Hendrycks & Dietterich, 2019). Fog creates adversarial fog-like occlusions by adversarially optimizing parameters in the diamond-square algorithm (Fournier et al., 1982) typically used to render stochastic fog effects.
This algorithm starts with random perturbations to the four corner pixels of the image. At step t, it iteratively perturbs pixels at the centers of squares and diamonds formed by those pixels perturbed at step t−1. The perturbation of a step t pixel is the average of the neighboring step t−1 perturbations plus a parameter value which we adversarially optimize. We continue this process until all pixels have been perturbed; the outcome is a fog-like distortion to the original image.
Snow. Snow simulates snowfall with occlusions of randomly located small image regions representing snowflakes. Because the distortions caused by snowflakes are not differentiable in their locations, we instead place occlusions representing snowflakes at randomly chosen locations and orientations and adversarially optimize their intensities. This choice results in a fast, differentiable, and strong attack. Compared to synthetic stochastic snow (Hendrycks & Dietterich, 2019), our adversarial snow is faster and includes snowflakes at differing angles. Figure 2 shows adversarial snow exposes model weaknesses more effectively than the easier stochastic, average-case snow.
Gabor. Gabor spatially occludes the image with visually diverse Gabor noise Lagae et al. (2009). Gabor noise is a form of band-limited anisotropic procedural noise which convolves a parameter mask with a Gabor kernel which is a product of a Gaussian kernel and a harmonic kernel. We choose the Gabor kernel randomly and adversarially optimize the parameters of the mask starting from a sparse initialization. We apply spectral variance normalization (Co et al., 2019) to the resulting distortion and add it to the input image to create the attack.

heading: 3.2 IMPROVING EXISTING ATTACKS
text: Elastic modifies the attack of Xiao et al. (2018); it warps the image by distortions x′ = Flow(x, V ), where V : {1, . . . , 224}2 → R2 is a vector field on pixel space, and Flow sets the value of pixel (i, j) to the bilinearly interpolated original value at (i, j) + V (i, j). We construct V by smoothing a vector field W by a Gaussian kernel (size 25 × 25, σ ≈ 3 for a 224 × 224 image) and optimize W under ‖W (i, j)‖∞ ≤ ε for all i, j. The resulting attack is suitable for large-scale images. The other three attacks are L1, L2, L∞ attacks, but we improve the L1 attack. For L∞ and L2 constraints, we use randomly-initialized projected gradient descent (PGD), which applies gradient descent and projection to the L∞ and L2 balls (Madry et al., 2017). Projection is difficult for L1, and previous L1 attacks rely on computationally intensive methods for it (Chen et al., 2018; Tramèr & Boneh, 2019). We replace PGD with the Frank-Wolfe algorithm (Frank & Wolfe, 1956), which
optimizes a linear function instead of projecting at each step (pseudocode in Appendix D). This makes our L1 attack more principled than previous implementations.
4 ImageNet-UA: MEASURING ROBUSTNESS TO UNFORESEEN ATTACKS
We propose the framework ImageNet-UA and its CIFAR-10 analogue CIFAR-10-UA to measure and summarize model robustness while fulfilling the following desiderata: (1) defenses should be evaluated against a broad threat model through a diverse set of attacks, (2) defenses should exhibit generalization to attacks not exactly identical to train-time attacks, and (3) the range of distortion sizes used for an attack must be wide enough to avoid misleading conclusions caused by overly weak or strong versions of that attack (Figure 3).
The ImageNet-UA evaluation framework aggregates robustness information into a single measure, the mean Unforeseen Adversarial Robustness (mUAR). The mUAR is an average over six different attacks of the Unforeseen Adversarial Robustness (UAR), a metric which assesses the robustness of a defense against a specific attack by using a wide range of distortion sizes. UAR is normalized using a measure of attack strength, the ATA, which we now define.
Adversarial Training Accuracy (ATA). The Adversarial Training Accuracy ATA(A, ε) estimates the strength of an attack A against adversarial training (Madry et al., 2017), one of the strongest known defense methods. For a distortion size ε, it is the best adversarial test accuracy against A achieved by adversarial training against A. We allow a possibly different distortion size ε′ during training, since this can improves accuracy, and we choose a fixed architecture for each dataset.
For ImageNet-100, we choose ResNet-50 for the architecture, and for CIFAR-10 we choose ResNet56. When evaluating a defense with architecture other than ResNet-50 or ResNet-56, we recommend using ATA values computed with these architectures to enable consistent comparison. To estimate ATA(A, ε) in practice, we evaluate models adversarially trained against distortion size ε′ for ε′ in a large range (we describe this range at this section’s end).
UAR: Robustness Against a Single Attack. The UAR, a building block for the mUAR, averages a model’s robustness to a single attack over six distortion sizes ε1, . . . , ε6 chosen for each attack (we describe the selection procedure at the end of this section). It is defined as
UAR(A) := 100× ∑6
k=1 Acc(A, εk,M)∑6 k=1 ATA(A, εk) , (1)
where Acc(A, εk,M) is the accuracy Acc(A, εk,M) of a model M after attack A at distortion size εk. The normalization in (1) makes attacks of different strengths more commensurable in a stable way. We give values of ATA(A, εk) and εk for our attacks on ImageNet-100 and CIFAR-10 in Tables 4 and 5 (Appendix B), allowing computation of UAR of a defense against a single attack with six adversarial evaluations and no adversarial training.
mUAR: Mean Unforeseen Attack Robustness. We summarize a defense’s performance on ImageNet-UA with the mean Unforeseen Attack Robustness (mUAR), an average of UAR scores
for the L1, Elastic, JPEG, Fog, Snow, and Gabor attacks:
mUAR := 1
6
[ UAR(L1)+UAR(Elastic)+UAR(JPEG)+UAR(Fog)+UAR(Snow)+UAR(Gabor) ] .
Our measure mUAR estimates robustness to a broad threat model containing six unforeseen attacks at six distortion sizes each, meaning high mUAR requires generalization to several held-out attacks. In particular, it cannot be achieved by the common practice of engineering defenses to a single attack, which Figure 4 shows does not necessarily provide robustness to different attacks.
Our four novel attacks play a crucial role in mUAR by allowing us to estimate robustness to a sufficiently large set of adversarial attacks. As is customary when studying train-test mismatches and distributional shift, we advise against adversarially training with these six attacks when evaluating ImageNet-UA to preserve the validity of mUAR, though we encourage training with other attacks.
Distortion Sizes. We explain the ε′ values used to estimate ATA and the choice of ε1, . . . , ε6 used to define UAR. This calibration of distortion sizes adjusts for the fact (Figure 3) that adversarial robustness against an attack may vary drastically with distortion size. Further, the relation between distortion size and attack strength varies between attacks, so too many or too few εk values in a certain range may cause an attack to appear artificially strong or weak according to UAR.
We choose distortion sizes between εmin and εmax as follows. The minimum distortion size εmin is the largest ε for which the adversarial accuracy of an adversarially trained model at distortion size ε is comparable to that of a model trained and evaluated on unattacked data (for ImageNet-100, within 3 of 87). The maximum distortion size εmax is the smallest ε which either reduces adversarial accuracy of an adversarially trained model at distortion size ε below 25 or yields images confusing humans (adversarial accuracy can remain non-zero in this case).
As is typical in recent work on adversarial examples (Athalye et al., 2018b; Evtimov et al., 2017; Dong et al., 2019; Qin et al., 2019), our attacks can be perceptible at large distortion sizes. We make this choice to reflect perceptibility of attacks in real world threat models per Gilmer et al. (2018).
For ATA, we evaluate against models adversarially trained with ε′ increasing geometrically from εmin to εmax by factors of 2. We then choose εk as follows: We compute ATA at ε increasing geometrically from εmin to εmax by factors of 2 and take the size-6 subset whose ATA values have minimum `1-distance to the ATA values of the L∞ attack in Table 4 (Appendix B.1). For example, for Gabor, (εmin, εmax) = (6.25, 3200), so we compute ATAs at the 10 values ε = 6.25, . . . , 3200. Viewing size-6 subsets of the ATAs as vectors with decreasing coordinates, we select εk for Gabor corresponding to the vector with minimum `1-distance to the ATA vector for L∞.
5 NEW INSIGHTS FROM ImageNet-UA
We use ImageNet-UA to assess existing methods for adversarial defense and evaluation. First, ImageNet-UA reveals that L∞ trained defenses fail to generalize to different attacks, indicating substantial weakness in current L∞ adversarial robustness evaluation. We establish a baseline for ImageNet-UA using L2 adversarial training which is difficult to improve upon by adversarial training alone. Finally, we show non-adversarially trained models can still improve robustness on ImageNet-UA over standard models and suggest this as a direction for further inquiry.

heading: 5.1 EXPERIMENTAL SETUP
text: We adversarially train 48 models against the 8 attacks from Section 3 and evaluate against targeted attacks. We use the CIFAR-10 and ImageNet-100 datasets for ImageNet-UA and CIFAR-10-UA. ImageNet-100 is a 100-class subset of ImageNet-1K (Deng et al., 2009) containing every tenth class by WordNet ID order; we use a subset of ImageNet-1K due to the high compute cost of adversarial training. We use ResNet-56 for CIFAR-10 and ResNet-50 from torchvision for ImageNet-100 (He et al., 2016). We provide training hyperparameters in Appendix A.
To adversarially train against attack A, at each mini-batch we select a uniform random (incorrect) target class for each training image. For maximum distortion size ε, we apply targeted attack A to the current model with distortion size ε′ ∼ Uniform(0, ε) and take a SGD step using only the attacked images. Randomly scaling ε′ improves performance against smaller distortions.
We train on 10-step attacks for attacks other than Elastic, where we use 30 steps due to a harder optimization. For Lp, JPEG, and Elastic, we use step size ε/ √ steps; for Fog, Gabor, and Snow, we use step size √
0.001/steps because the latent space is independent of ε. These choices have optimal rates for non-smooth convex functions (Nemirovski & Yudin, 1978; 1983). We evaluate on 200-step targeted attacks with uniform random (incorrect) target, using more steps for evaluation than training per best practices (Carlini et al., 2019b).
Figure 4 summarizes ImageNet-100 results. Full results for ImageNet-100 and CIFAR-10 are in Appendix E and robustness checks to random seed and attack iterations are in Appendix F.
5.2 ImageNet-UA REVEALS WEAKNESSESS IN L∞ TRAINING AND TESTING
We use ImageNet-UA to reveal weaknesses in the common practices of L∞ robustness evaluation and L∞ adversarial training. We compute the mUAR and UAR(L∞) for models trained against the L∞ attack with distortion size ε and show results in Figure 5. For small ε ≤ 4, mUAR and
UAR(L∞) increase together with ε. For larger ε ≥ 8, UAR(L∞) continues to increase with ε, but the mUAR decreases, a fact which is not apparent from L∞ evaluation.
The decrease in mUAR while UAR(L∞) increases suggests that L∞ adversarial training begins to heavily fit L∞ distortions at the expense of generalization at larger distortion sizes. Thus, while it is the most commonly used defense procedure, L∞ training may not lead to improvements on other attacks or to real-world robustness.
Worse, L∞ evaluation against L∞ adversarial training at higher distortions indicates higher robustness. In contrast, mUAR reveals that L∞ adversarial training at higher distortions in fact hurts robustness against a more diverse set of attacks. Thus, L∞ evaluation gives a misleading picture of robustness. This is particularly important because L∞ evaluation is the most ubiquitous measure of robustness in deep learning (Goodfellow et al., 2014; Madry et al., 2017; Xie et al., 2018).
5.3 LIMITS OF ADVERSARIAL TRAINING FOR ImageNet-UA
We establish a baseline on ImageNet-UA using L2 adversarial training but show a significant performance gap even for more sophisticated existing adversarial training methods. To do so, we evaluate several adversarial training methods on ImageNet-UA and show results in Table 1.
Our results show that L2 trained models outperform L∞ trained models and have significantly improved absolute performance, increasing mUAR from 14.0 to 50.7 compared to an undefended model. The individual UAR values in Figure 7 (Appendix E.1) improve substantially against all attacks other than Fog, including several (Elastic, Gabor, Snow) of extremely different nature to L2.
This result suggests pushing adversarial training further by training against multiple attacks simultaneously via joint adversarial training (Jordan et al., 2019; Tramèr & Boneh, 2019) detailed in Appendix C. Table 2 shows that, despite using twice the compute of L2 training, (L∞, L2) joint training only improves the mUAR from 50.7 to 50.9. We thus recommend L2 training as a baseline for ImageNet-UA, though there is substantial room for improvement compared to the highest UARs against individual attacks in Figure 4, which are all above 80 and often above 90.
5.4 ImageNet-UA ROBUSTNESS THROUGH NON-ADVERSARIAL DEFENSES
We find that methods can improve robustness to unforeseen attacks without adversarial training. Table 3 shows mUAR for SqueezeNet (Iandola et al., 2017), ResNeXts (Xie et al., 2016), and ResNets. For ImageNet-1K models, we mask 900 logits to predict ImageNet-100 classes.
A popular defense against average case distortions (Hendrycks & Dietterich, 2019) is Stylized ImageNet (Geirhos et al., 2019), which modifies training images using image style transfer in hopes of making networks rely less on textural features. Table 3 shows it provides some improvement on ImageNet-UA. More recently, Lopes et al. (2019) propose to train against Gaussian noise applied to small image patches, improving the mUAR by 3% over the ResNet-50 baseline. The second largest mUAR improvement comes from training a ResNeXt on approximately 1 billion images (Mahajan et al., 2018). This three orders of magnitude increase in training data yields a 5.4% mUAR
increase over a vanilla ResNeXt baseline. Finally, Hendrycks et al. (2020) create AugMix, which randomly mixes stochastically generated augmentations. Although AugMix did not use random nor adversarial noise, it improves robustness to unforeseen attacks by 10%.
These results imply that defenses not relying on adversarial examples can improve ImageNet-UA performance. They indicate that training on more data only somewhat increases robustness on ImageNet-UA, unlike many other robustness benchmarks (Hendrycks & Dietterich, 2019; Hendrycks et al., 2019) where more data helps tremendously (Orhan, 2019). While models with lower clean accuracy (e.g., SqueezeNet and ResNet-18) have higher UAR(L∞) and UAR(L2) than many other models, there is no clear difference in mUAR. Last, these non-adversarial defenses have minimal cost to accuracy on clean examples, unlike adversarial defenses. Much remains to explore, and we hope non-adversarial defenses will be a promising avenue toward adversarial robustness.

heading: 6 CONCLUSION
text: This work proposes a framework ImageNet-UA to evaluate robustness of a defense against unforeseen attacks. Because existing adversarial defense techniques do not scale to multiple attacks, developing models which can defend against attacks not seen at train-time is essential for robustness. Our results using ImageNet-UA show that the common practice of L∞ training and evaluation fails to achieve or measure this broader form of robustness. As a result, it can provide a misleading sense of robustness. By incorporating our 4 novel and strong adversarial attacks, ImageNet-UA enables evaluation on the diverse held-out attacks necessary to measure progress towards robustness more broadly.

heading: A TRAINING HYPERPARAMETERS
text: For ImageNet-100, we trained on machines with 8 NVIDIA V100 GPUs using standard data augmentation (He et al., 2016). Following best practices for multi-GPU training (Goyal et al., 2017), we ran synchronized SGD for 90 epochs with batch size 32×8 and a learning rate schedule with 5 “warm-up” epochs and a decay at epochs 30, 60, and 80 by a factor of 10. Initial learning rate after warm-up was 0.1, momentum was 0.9, and weight decay was 10−4. For CIFAR-10, we trained on a single NVIDIA V100 GPU for 200 epochs with batch size 32, initial learning rate 0.1, momentum 0.9, and weight decay 10−4. We decayed the learning rate at epochs 100 and 150.
B CALIBRATION OF ImageNet-UA AND CIFAR-10-UA
B.1 CALIBRATION FOR ImageNet-UA
Calibrated distortion sizes and ATA values are in Table 4.
B.2 CALIBRATION FOR CIFAR-10-UA
The ε calibration procedure for CIFAR-10 was similar to that used for ImageNet-100. We started with small εmin values and increased ε geometrically with ratio 2 until adversarial accuracy of an adversarially trained model dropped below 40. Note that this threshold is higher for CIFAR-10 than ImageNet-100 because there are fewer classes. The resulting ATA values for CIFAR-10 are shown in Table 5.

heading: C JOINT ADVERSARIAL TRAINING
text: Our joint adversarial training procedure for two attacksA andA′ is as follows. At each training step, we compute the attacked image under both A and A′ and backpropagate with respect to gradients induced by the image with greater loss. This corresponds to the “max” loss of Tramèr & Boneh (2019). We train ResNet-50 models for (L∞, L2), (L∞, L1), and (L∞,Elastic) on ImageNet-100.
Table 6 shows training against (L∞, L1) is worse than training against L1 at the same distortion size and performs particularly poorly at large distortion sizes. Table 7 shows joint training against
(L∞,Elastic) also performs poorly, never matching the UAR score of training against Elastic at moderate distortion size (ε = 2).

heading: D THE FRANK-WOLFE ALGORITHM
text: We chose to use the Frank-Wolfe algorithm for optimizing the L1 attack, as Projected Gradient Descent would require projecting onto a truncated L1 ball, which is a complicated operation. In contrast, Frank-Wolfe only requires optimizing linear functions g>x over a truncated L1 ball; this can be done by sorting coordinates by the magnitude of g and moving the top k coordinates to the boundary of their range (with k chosen by binary search). This is detailed in Algorithm 1.

heading: E FULL EVALUATION RESULTS
text: E.1 FULL EVALUATION RESULTS AND ANALYSIS FOR IMAGENET-100
We show the full results of all adversarial attacks against all adversarial defenses for ImageNet-100 in Figure 6. These results also include L1-JPEG and L2-JPEG attacks, which are modifications of the JPEG attack applying Lp-constraints in the compressed JPEG space instead of L∞ constraints. Full UAR scores are provided for ImageNet-100 in Figure 7.
E.2 FULL EVALUATION RESULTS AND ANALYSIS FOR CIFAR-10
We show the results of adversarial attacks and defenses for CIFAR-10 in Figure 8. We experienced difficulty training theL2 andL1 attacks at distortion sizes greater than those shown and have omitted those runs, which we believe may be related to the small size of CIFAR-10 images. Full UAR values for CIFAR-10 are shown in Figure 9.

heading: F ROBUSTNESS OF OUR RESULTS
text: F.1 REPLICATION
We replicated our results for the first three rows of Figure 6 with different random seeds to see the variation in our results. As shown in Figure 10, deviations in results are minor.
No at
tac k L
= 1 L =
2 L
= 4 L =
8 L
= 16 L =
32 L2 =
15 0 L2 =
30 0 L2 =
60 0
L2 =
12 00
L2 =
24 00
L2 =
48 00 L1 =
95 62
.44
L1 =
19 12
5
L1 =
38 25
0.1
L1 =
76 50
0
L1 =
15 30
00
L1 =
30 60
00
L1 =
61 20
00
L -JP
EG =
0.0 31
25
L -JP
EG =
0.0 62
5
L -JP
EG =
0.1 25
L -JP
EG =
0.2 5
L -JP
EG =
0.5
L -JP
EG =
1
L -JP
EG =
2 L2
-JP EG
= 2
L2 -JP
EG =
4
L2 -JP
EG =
8
L2 -JP
EG =
16
L2 -JP
EG =
32
L2 -JP
EG =
64
L2 -JP
EG =
12 8
L2 -JP
EG =
25 6
L1 -JP
EG =
12 8
L1 -JP
EG =
25 6
L1 -JP
EG =
51 2
L1 -JP
EG =
10 24
L1 -JP
EG =
20 48
L1 -JP
EG =
40 96
L1 -JP
EG =
81 92
L1 -JP
EG =
16 38
4
L1 -JP
EG =
32 76
8
L1 -JP
EG =
65 53
6
L1 -JP
EG =
13 10 72 Ela
stic =
0.2 5
Ela stic
= 0.5 Ela stic
= 1
Ela stic
= 2
Ela stic
= 4
Ela stic
= 8
Ela stic
= 16 Fog
= 12
8 Fog =
25 6 Fog =
51 2
Fog =
10 24
Fog =
20 48
Fog =
40 96
Fog =
81 92
Fog =
16 38
4
Fog =
32 76
8
Fog =
65 53 6 Ga bo r
=6 .25
Ga bo
r =1
2.5
Ga bo
r =2
5
Ga bo
r =5
0
Ga bo
r =1
00
Ga bo
r =2
00
Ga bo
r =4
00
Ga bo
r =8
00
Ga bo
r =1
60 0
Ga bo
r =3
20 0
Sn ow
= 0.0
31 25
Sn ow
= 0.0
62 5
Sn ow
= 0.1
25
Sn ow
= 0.2
5
Sn ow
= 0.5
Sn ow
= 1
Sn ow
= 2
Sn ow
= 4
Sn ow
= 8
Sn ow
= 16
No rm
al tr
ai ni
ng
L
= 1
L
= 2
L
= 4
L
= 8
L
= 16
L
= 32
L 2
= 15 0 L 2 = 30 0 L 2 = 60 0 L 2 = 12 00 L 2 = 24 00 L 2 = 48 00
L 1
= 95
62 .4 4 L 1 = 19 12 5 L 1 = 38 25 0. 1 L 1 = 76 50 0 L 1 = 15 30 00 L 1 = 30 60 00 L 1 = 61 20 00
L -JP
EG
= 0.
03 12 5 L -JP EG = 0. 06 25 L -JP EG = 0. 12 5 L -JP EG = 0. 25 L -JP EG = 0. 5 L -JP EG = 1 L -JP EG = 2 L 2 -JP EG = 2 L 2 -JP EG = 4 L 2 -JP EG = 8 L 2 -JP EG = 16 L 2 -JP EG = 32 L 2 -JP EG = 64 L 2 -JP EG = 12 8 L 2 -JP EG = 25 6 L 1 -JP EG = 12 8 L 1 -JP EG = 25 6 L 1 -JP EG = 51 2 L 1 -JP EG = 10 24 L 1 -JP EG = 20 48 L 1 -JP EG = 40 96 L 1 -JP EG = 81 92 L 1 -JP EG = 16 38 4 L 1 -JP EG = 32 76 8 L 1 -JP EG = 65 53 6 L 1 -JP EG = 13 10 72 El as tic = 0. 25 El as tic = 0. 5 El as tic = 1 El as tic = 2 El as tic = 4 El as tic = 8 El as tic = 16 Fo g = 12 8 Fo g = 25 6 Fo g = 51 2 Fo g = 10 24 Fo g = 20 48 Fo g = 40 96 Fo g = 81 92 Fo g = 16 38 4 Fo g = 32 76 8 Fo g = 65 53 6 Ga bo r = 6. 25 Ga bo r = 12 .5 Ga bo r = 25 Ga bo r = 50 Ga bo r = 10 0 Ga bo r = 20 0 Ga bo r = 40 0 Ga bo r = 80 0 Ga bo r = 16 00 Ga bo r = 32 00 Sn ow = 0. 03 12 5 Sn ow = 0. 06 25 Sn ow = 0. 12 5 Sn ow = 0. 25 Sn ow = 0. 5 Sn ow = 1 Sn ow = 2 Sn ow = 4 Sn ow = 8 Sn ow = 16
87 25
1 0
0 0
0 57
11 0
0 0
0 61
29 5
0 0
0 0
20 1
0 0
0 0
0 70
25 1
0 0
0 0
0 50
20 3
0 0
0 0
0 0
0 0
79 47
6 0
0 0
0 54
15 1
0 0
0 0
0 0
0 12
3 1
0 0
0 0
0 0
0 64
32 7
1 0
0 0
0 0
0
86 84
70 14
0 0
0 86
81 48
2 0
0 80
66 35
5 0
0 0
84 71
13 0
0 0
0 86
84 66
10 0
0 0
0 76
60 28
5 0
0 0
0 0
0 0
84 75
36 3
0 0
0 74
47 9
0 0
0 0
0 0
0 74
28 4
1 0
0 0
0 0
0 80
68 29
4 0
0 0
0 0 0 85 85 81 50 2 0 0 85 83 71 18 0 0 81 72 52 18 1 0 0 84 81 48 1 0 0 0 85 84 77 34 1 0 0 0 79 69 46 16 2 0 0 0 0 0 0 84 78 52 7 0 0 0 73 47 10 0 0 0 0 0 0 0 81 59 12 1 0 0 0 0 0 0 79 73 44 9 1 0 0 0 0 0 84 83 82 74 22 0 0 84 83 79 48 2 0 80 75 61 32 6 0 0 84 82 69 10 0 0 0 84 83 79 54 5 0 0 0 79 73 58 31 7 1 0 0 0 0 0 83 79 62 15 1 0 0 70 43 9 1 0 0 0 0 0 0 82 75 37 3 0 0 0 0 0 0 79 74 56 18 3 1 0 0 0 0 80 79 79 76 59 6 0 79 78 73 50 7 0 72 64 49 27 6 1 0 79 77 59 12 0 0 0 79 78 70 43 6 0 0 0 74 67 54 33 11 2 0 0 0 0 0 79 76 66 31 2 0 0 63 33 6 1 1 0 0 0 1 0 79 77 64 17 1 0 0 0 0 0 74 72 62 31 9 2 0 0 0 0 75 74 74 73 67 34 1 73 71 63 30 3 0 58 43 24 8 1 0 0 73 67 37 3 0 0 0 73 70 59 26 3 0 0 0 64 56 42 25 11 3 0 0 0 0 0 73 71 66 42 11 1 0 55 25 4 1 1 0 0 0 0 0 74 73 68 44 4 0 0 0 0 0 68 66 59 40 17 6 1 0 0 0 71 71 70 69 62 40 8 69 60 33 5 0 0 37 21 8 2 0 0 0 65 42 8 0 0 0 0 68 61 34 7 1 0 0 0 56 47 36 23 11 4 1 0 0 0 0 70 68 62 44 15 2 0 55 29 5 1 0 0 0 0 0 0 70 69 63 49 16 2 2 1 0 0 63 60 54 37 17 5 1 0 0 0 87 82 53 3 0 0 0 85 78 34 1 0 0 80 69 36 5 0 0 0 81 51 2 0 0 0 0 85 81 48 2 0 0 0 0 73 48 15 2 0 0 0 0 0 0 0 83 71 28 1 0 0 0 73 46 10 0 0 0 0 0 0 0 62 15 3 0 0 0 0 0 0 0 78 60 20 3 0 0 0 0 0 0 85 84 74 22 0 0 0 85 82 65 8 0 0 82 76 57 18 1 0 0 84 75 22 0 0 0 0 85 83 73 17 0 0 0 0 78 66 35 7 0 0 0 0 0 0 0 83 76 43 3 0 0 0 74 47 11 0 0 0 0 0 0 0 77 39 6 1 0 0 0 0 0 0 79 67 33 5 0 0 0 0 0 0 84 84 81 56 4 0 0 84 83 77 40 1 0 83 80 71 44 9 0 0 84 81 60 3 0 0 0 85 84 80 49 2 0 0 0 82 75 59 24 3 0 0 0 0 0 0 83 79 56 9 0 0 0 72 45 11 1 0 0 0 0 0 0 81 65 17 2 0 0 0 0 0 0 78 72 46 11 1 0 0 0 0 0 82 82 81 74 28 0 0 82 82 80 69 15 0 82 80 77 65 32 4 0 82 81 76 34 0 0 0 82 82 81 74 25 0 0 0 81 80 73 56 24 4 0 0 0 0 0 81 78 66 23 1 0 0 67 40 8 1 0 0 0 0 0 0 81 75 42 5 0 0 0 0 0 0 76 72 57 22 3 1 0 0 0 0 77 77 76 74 56 6 0 77 77 76 73 50 2 77 76 75 71 57 23 1 77 76 75 63 9 0 0 77 76 76 74 57 7 0 0 77 76 73 68 52 25 5 1 0 0 0 76 74 68 38 4 0 0 59 30 6 1 1 1 0 1 0 0 76 74 61 16 1 0 0 0 0 0 70 68 58 35 10 2 0 0 0 0 68 68 68 67 61 28 1 68 68 68 67 59 20 69 68 68 66 61 44 13 68 68 67 64 37 2 0 68 68 68 67 62 33 1 0 68 68 68 66 61 49 30 12 4 2 3 68 66 63 47 11 1 0 49 23 5 1 1 1 0 1 0 0 68 67 61 30 3 1 1 0 0 0 60 58 54 40 17 5 2 1 0 0 86 71 24 1 0 0 0 82 64 14 0 0 0 83 77 53 13 0 0 0 67 18 0 0 0 0 0 84 71 22 1 0 0 0 0 71 44 11 1 0 0 0 0 0 0 0 82 66 19 1 0 0 0 68 33 3 0 0 0 0 0 0 0 41 9 2 0 0 0 0 0 0 0 73 50 14 2 0 0 0 0 0 0 86 78 41 3 0 0 0 84 74 32 1 0 0 84 81 68 32 3 0 0 76 41 2 0 0 0 0 84 79 48 3 0 0 0 0 78 63 28 5 0 0 0 0 0 0 0 83 71 28 1 0 0 0 69 36 4 0 0 0 0 0 0 0 56 17 3 0 0 0 0 0 0 0 75 59 22 2 0 0 0 0 0 0 85 81 62 11 0 0 0 84 80 55 6 0 0 84 82 77 54 12 0 0 81 64 14 0 0 0 0 84 82 68 20 0 0 0 0 81 75 51 18 3 0 0 0 0 0 0 82 75 41 3 0 0 0 70 39 6 0 0 0 0 0 0 0 71 32 5 1 0 0 0 0 0 0 77 64 30 4 0 0 0 0 0 0 84 82 71 28 1 0 0 83 81 67 20 0 0 84 83 81 72 40 4 0 82 75 45 3 0 0 0 84 82 76 48 4 0 0 0 83 80 72 47 14 2 0 0 0 0 0 82 77 52 6 0 0 0 68 37 5 0 0 0 0 0 0 0 77 50 12 1 0 0 0 0 0 0 76 67 34 8 1 0 0 0 0 0 81 79 72 43 3 0 0 80 78 69 35 2 0 81 80 79 76 64 26 1 79 74 60 17 0 0 0 80 79 76 62 16 0 0 0 80 80 78 71 45 12 1 0 0 0 0 79 75 57 12 0 0 0 65 36 5 0 0 0 0 0 0 0 77 61 23 3 0 0 0 0 0 0 73 66 44 12 1 0 0 0 0 0 79 77 72 53 10 0 0 78 77 71 46 6 0 79 78 78 75 69 45 7 77 74 66 40 3 0 0 78 77 75 68 41 2 0 0 78 78 77 74 64 36 7 1 0 0 0 77 73 62 21 1 0 0 61 32 4 0 0 0 0 0 0 0 76 65 35 6 0 0 0 0 0 0 70 65 49 20 4 1 0 0 0 0 72 71 69 59 24 1 0 72 71 69 55 18 0 72 72 71 70 67 55 24 71 69 65 51 15 0 0 71 71 70 67 53 13 0 0 71 71 70 69 64 52 24 6 1 0 0 70 68 61 33 3 0 0 50 24 5 0 0 0 0 0 0 0 69 62 42 12 1 0 0 0 0 0 60 56 46 24 8 2 0 0 0 1 87 75 28 1 0 0 0 83 58 7 0 0 0 75 50 14 1 0 0 0 86 83 54 2 0 0 0 86 86 82 55 3 0 0 0 83 80 67 36 7 0 0 0 0 0 0 83 65 14 0 0 0 0 67 28 2 0 0 0 0 0 0 0 34 6 1 0 0 0 0 0 0 0 72 44 11 1 0 0 0 0 0 0 87 80 47 3 0 0 0 84 71 19 0 0 0 77 60 24 2 0 0 0 86 84 75 17 0 0 0 87 86 84 74 18 0 0 0 85 82 76 57 21 2 0 0 0 0 0 83 70 20 1 0 0 0 69 33 3 0 0 0 0 0 0 0 51 11 2 0 0 0 0 0 0 0 75 51 13 1 0 0 0 0 0 0 86 83 68 14 0 0 0 84 79 43 2 0 0 80 67 37 7 0 0 0 85 85 83 57 1 0 0 86 86 85 81 51 1 0 0 85 83 81 72 47 14 1 0 0 0 0 83 73 28 1 0 0 0 66 30 3 0 0 0 0 0 0 0 69 27 4 1 0 0 0 0 0 0 77 59 20 3 0 0 0 0 0 0 84 83 77 42 3 0 0 83 81 66 13 0 0 80 73 53 18 2 0 0 84 84 83 77 14 0 0 84 84 84 82 73 17 0 0 84 83 82 78 68 43 13 2 0 0 0 82 75 40 3 0 0 0 65 30 3 0 0 0 0 0 0 0 78 52 13 1 0 0 0 0 0 0 76 64 29 5 1 0 0 0 0 0 81 80 78 66 17 1 0 80 79 74 41 3 0 78 74 62 35 7 0 0 81 81 80 79 64 0 0 81 81 81 80 77 57 2 0 80 80 80 79 76 69 52 28 9 2 1 80 76 52 7 0 0 0 63 30 3 0 0 0 0 0 0 0 79 69 35 6 1 0 0 0 0 0 74 67 46 11 2 0 0 0 0 0 79 79 77 68 27 1 0 79 78 74 50 7 0 77 75 68 50 21 3 0 80 79 79 77 73 34 0 80 79 79 79 76 68 32 0 80 79 79 79 78 76 73 68 61 48 48 78 75 58 12 0 0 0 61 29 3 0 0 0 0 0 0 0 78 71 47 12 1 0 0 0 0 0 71 67 48 17 4 1 0 0 0 0 78 77 76 63 19 1 0 78 77 73 47 5 0 77 75 68 48 17 2 0 78 78 77 75 62 35 1 78 78 78 78 76 67 45 17 78 78 78 77 77 75 73 69 65 55 47 77 74 57 11 0 0 0 60 29 4 0 0 0 0 0 0 0 77 70 44 14 2 0 0 0 0 0 70 65 45 16 4 1 0 0 0 1 87 64 12 0 0 0 0 80 44 2 0 0 0 72 43 10 1 0 0 0 85 71 15 0 0 0 0 86 85 74 23 0 0 0 0 83 76 54 17 1 0 0 0 0 0 0 82 59 10 0 0 0 0 64 24 2 0 0 0 0 0 0 0 22 4 1 0 0 0 0 0 0 0 70 40 9 1 0 0 0 0 0 0 87 75 26 1 0 0 0 83 59 8 0 0 0 75 52 16 1 0 0 0 86 82 49 1 0 0 0 86 86 82 56 3 0 0 0 84 81 70 41 8 0 0 0 0 0 0 82 65 15 0 0 0 0 67 27 2 0 0 0 0 0 0 0 33 6 1 0 0 0 0 0 0 0 74 44 10 1 0 0 0 0 0 0 86 81 50 4 0 0 0 84 74 24 1 0 0 79 64 28 3 0 0 0 86 85 74 15 0 0 0 86 86 84 76 24 0 0 0 85 83 79 64 28 3 0 0 0 0 0 83 69 19 1 0 0 0 68 33 3 0 0 0 0 0 0 0 50 11 2 0 0 0 0 0 0 0 75 52 14 1 0 0 0 0 0 0 85 83 70 15 0 0 0 84 80 50 3 0 0 81 73 46 10 0 0 0 86 85 82 56 1 0 0 86 85 85 82 61 3 0 0 85 84 83 78 60 24 2 0 0 0 0 83 73 30 1 0 0 0 69 35 4 0 0 0 0 0 0 0 68 25 4 0 0 0 0 0 0 0 77 59 20 3 0 0 0 0 0 0 84 83 78 44 3 0 0 83 82 70 18 0 0 81 77 62 25 2 0 0 84 84 83 75 14 0 0 84 84 84 82 76 29 0 0 84 83 83 81 75 57 22 3 0 0 0 82 76 42 3 0 0 0 68 35 4 0 0 0 0 0 0 0 77 47 10 1 0 0 0 0 0 0 77 67 32 5 1 0 0 0 0 0 81 81 79 67 17 0 0 81 80 76 49 4 0 80 77 70 45 11 1 0 81 81 81 79 58 0 0 81 81 81 80 79 65 4 0 81 81 81 80 78 73 60 34 10 2 1 80 76 54 9 0 0 0 66 34 5 0 0 0 0 0 0 0 79 68 31 4 0 0 0 0 0 0 75 69 45 13 2 0 0 0 0 0 77 78 76 72 40 3 0 78 77 75 63 16 0 77 75 71 57 26 3 0 78 77 77 76 71 19 0 78 78 77 77 76 72 39 0 77 77 77 77 76 75 72 67 55 38 32 77 75 62 19 1 0 0 61 29 4 0 0 0 0 0 0 0 76 72 48 11 1 0 0 0 0 0 71 68 55 23 6 1 0 0 0 1 77 77 76 71 36 2 0 77 77 75 62 15 0 77 76 71 59 30 5 0 78 77 77 76 71 40 0 78 78 78 77 76 72 47 3 78 78 77 77 77 75 74 70 63 56 57 77 75 64 19 1 0 0 60 29 4 0 0 0 0 0 0 0 76 71 47 13 1 0 0 0 0 0 69 66 53 25 7 1 0 0 0 1 87 66 15 0 0 0 0 81 51 5 0 0 0 77 55 17 1 0 0 0 83 69 16 0 0 0 0 86 85 79 40 1 0 0 0 85 83 75 48 12 0 0 0 0 0 0 81 58 10 0 0 0 0 64 24 1 0 0 0 0 0 0 0 24 4 1 0 0 0 0 0 0 0 71 40 8 1 0 0 0 0 0 0 86 74 27 1 0 0 0 83 63 11 0 0 0 79 63 26 2 0 0 0 85 78 38 1 0 0 0 86 86 82 61 6 0 0 0 85 84 80 63 26 2 0 0 0 0 0 81 63 13 0 0 0 0 68 30 2 0 0 0 0 0 0 0 34 7 1 0 0 0 0 0 0 0 74 43 10 1 0 0 0 0 0 0 86 79 47 3 0 0 0 84 74 26 1 0 0 81 71 38 5 0 0 0 85 82 64 8 0 0 0 86 85 84 75 27 0 0 0 85 84 82 75 49 10 0 0 0 0 0 82 67 18 0 0 0 0 68 30 3 0 0 0 0 0 0 0 48 11 2 0 0 0 0 0 0 0 76 52 14 2 0 0 0 0 0 0 86 83 66 12 0 0 0 84 80 51 3 0 0 82 75 53 13 1 0 0 85 84 77 36 0 0 0 86 86 85 81 57 3 0 0 86 85 84 80 66 34 4 0 0 0 0 83 73 29 2 0 0 0 69 34 4 0 0 0 0 0 0 0 63 21 3 0 0 0 0 0 0 0 78 61 19 2 0 0 0 0 0 0 84 82 75 34 1 0 0 84 81 67 13 0 0 82 78 63 27 2 0 0 84 84 81 66 6 0 0 85 84 84 82 73 22 0 0 84 84 83 81 75 57 22 2 0 0 0 82 75 39 3 0 0 0 69 35 5 0 0 0 0 0 0 0 74 39 7 1 0 0 0 0 0 0 78 66 31 4 0 0 0 0 0 0 83 82 79 55 5 0 0 83 81 75 34 1 0 81 78 69 39 6 0 0 83 83 82 76 32 0 0 83 83 83 82 78 50 1 0 83 83 83 82 79 71 49 18 3 0 0 81 76 50 6 0 0 0 66 34 5 0 0 0 0 0 0 0 79 56 16 2 0 0 0 0 0 0 76 68 38 7 1 0 0 0 0 0 81 80 78 64 14 0 0 80 80 76 48 3 0 80 77 70 48 12 1 0 81 80 80 77 54 1 0 81 81 80 80 79 64 6 0 80 81 80 80 79 75 67 45 17 4 4 79 75 55 9 0 0 0 65 33 4 0 0 0 0 0 0 0 78 67 26 3 0 0 0 0 0 0 74 69 47 12 2 0 0 0 0 0 80 79 78 70 26 1 0 79 79 76 58 8 0 78 76 71 53 18 1 0 80 79 79 77 66 3 0 80 79 79 79 78 70 17 0 79 79 79 79 78 76 73 62 41 19 17 79 76 58 12 0 0 0 62 28 3 0 0 0 0 0 0 0 78 71 39 6 0 0 0 0 0 0 73 69 50 17 3 1 0 0 0 0 77 77 77 71 33 1 0 78 77 75 60 11 0 77 75 70 56 22 2 0 78 78 77 77 68 6 0 78 78 78 77 76 70 22 0 78 77 77 77 77 75 73 65 51 31 26 77 74 60 15 1 0 0 59 27 3 0 0 0 0 0 0 0 77 72 44 9 1 0 0 0 0 0 71 67 50 21 4 1 0 0 0 0 76 75 74 69 37 2 0 75 75 73 59 13 0 74 72 68 54 22 2 0 75 75 75 74 68 9 0 75 75 75 74 74 68 24 0 75 75 74 75 74 73 72 67 59 46 42 74 71 57 16 1 0 0 52 22 2 0 0 0 0 0 0 0 75 70 52 14 1 0 0 0 0 0 66 63 49 21 5 1 0 0 0 1 72 73 71 65 29 2 0 73 72 69 54 10 0 72 70 65 50 19 2 0 73 73 72 72 64 5 0 73 73 73 72 71 64 18 0 73 73 73 72 72 71 69 64 55 41 37 72 69 54 13 1 0 0 47 17 2 0 0 0 0 0 0 0 72 67 44 11 1 0 0 0 0 0 63 59 43 16 3 1 0 0 0 1 87 63 14 0 0 0 0 79 44 4 0 0 0 72 48 14 2 0 0 0 64 14 0 0 0 0 0 82 58 11 0 0 0 0 0 62 33 9 1 0 0 0 0 0 0 0 85 78 45 2 0 0 0 70 35 4 0 0 0 0 0 0 0 39 10 2 1 0 0 0 0 0 0 75 52 20 3 0 0 0 0 0 0 87 73 25 1 0 0 0 82 58 10 0 0 0 76 58 25 4 0 0 0 76 34 1 0 0 0 0 84 71 25 1 0 0 0 0 67 43 17 3 0 0 0 0 0 0 0 86 83 69 15 0 0 0 72 42 8 0 0 0 0 0 0 0 56 21 5 1 0 0 0 0 0 0 76 61 30 8 1 0 0 0 0 0 85 77 40 3 0 0 0 82 67 21 1 0 0 77 63 35 9 1 0 0 80 54 9 0 0 0 0 83 76 41 4 0 0 0 0 67 47 22 6 1 0 0 0 0 0 0 84 83 78 51 3 0 0 72 47 11 0 0 0 0 0 0 0 71 42 14 3 0 0 0 0 0 0 76 68 43 16 4 1 0 0 0 0 84 78 49 7 0 0 0 81 71 30 2 0 0 75 62 37 11 1 0 0 79 56 11 0 0 0 0 82 76 45 8 0 0 0 0 68 52 29 9 2 0 0 0 0 0 0 84 83 81 73 32 1 0 73 50 15 1 0 0 0 0 0 0 76 55 22 5 1 0 0 0 0 0 76 69 51 26 9 2 1 1 0 0 81 74 47 8 0 0 0 78 67 27 2 0 0 66 49 25 7 1 0 0 70 36 4 0 0 0 0 78 68 31 4 0 0 0 0 61 42 21 7 1 0 0 0 0 0 0 81 80 79 78 71 17 0 68 46 13 1 0 0 0 0 0 0 75 59 32 9 2 1 1 1 0 0 72 68 55 34 15 5 3 1 1 1 78 69 39 5 0 0 0 73 54 15 1 0 0 47 28 10 2 0 0 0 53 13 1 0 0 0 0 70 46 12 1 0 0 0 0 45 27 11 3 0 0 0 0 0 0 0 78 76 76 75 76 57 4 63 42 13 1 0 0 0 0 0 0 71 59 35 12 3 1 1 1 0 0 66 63 54 36 19 9 5 2 1 1 74 58 22 2 0 0 0 63 32 5 0 0 0 31 14 4 1 0 0 0 27 3 0 0 0 0 0 55 22 2 0 0 0 0 0 32 16 4 1 0 0 0 0 0 0 0 74 73 72 71 70 57 21 60 38 12 1 0 0 0 0 0 0 68 52 29 10 3 1 1 1 0 0 63 60 52 38 24 14 9 4 1 1 87 40 2 0 0 0 0 69 16 0 0 0 0 62 26 4 0 0 0 0 23 1 0 0 0 0 0 74 27 1 0 0 0 0 0 49 18 2 0 0 0 0 0 0 0 0 81 54 9 0 0 0 0 83 69 33 3 0 0 0 0 0 0 19 3 1 0 0 0 0 0 0 0 77 47 12 1 0 0 0 0 0 0 88 40 2 0 0 0 0 68 17 0 0 0 0 60 24 3 0 0 0 0 17 1 0 0 0 0 0 71 22 1 0 0 0 0 0 45 15 2 0 0 0 0 0 0 0 0 82 58 12 0 0 0 0 85 78 59 21 2 0 0 0 0 0 25 5 1 0 0 0 0 0 0 0 80 57 19 2 0 0 0 0 0 0 87 29 1 0 0 0 0 59 12 0 0 0 0 54 19 2 0 0 0 0 9 0 0 0 0 0 0 57 11 0 0 0 0 0 0 30 6 0 0 0 0 0 0 0 0 0 82 61 14 1 0 0 0 86 82 74 51 16 2 1 1 1 1 31 6 1 0 0 0 0 0 0 0 81 65 27 4 0 0 0 0 0 0 86 23 1 0 0 0 0 50 9 0 0 0 0 49 16 2 0 0 0 0 6 0 0 0 0 0 0 45 8 0 0 0 0 0 0 29 7 1 0 0 0 0 0 0 0 0 82 63 16 1 0 0 0 86 84 78 67 43 16 5 2 1 1 40 9 2 0 0 0 0 0 0 0 82 68 39 8 1 0 0 0 0 0 85 17 1 0 0 0 0 42 6 0 0 0 0 46 16 2 0 0 0 0 4 0 0 0 0 0 0 35 5 0 0 0 0 0 0 27 7 1 0 0 0 0 0 0 0 0 81 63 19 1 0 0 0 84 83 79 71 56 34 16 7 3 1 52 16 3 1 0 0 0 0 0 0 80 72 44 11 2 0 0 0 0 0 78 8 0 0 0 0 0 24 2 0 0 0 0 26 5 0 0 0 0 0 2 0 0 0 0 0 0 23 2 0 0 0 0 0 0 15 3 0 0 0 0 0 0 0 0 0 74 60 19 1 0 0 0 79 78 76 72 67 58 46 29 10 3 62 34 9 2 0 0 0 0 0 0 75 69 50 19 4 1 0 0 0 0 69 4 0 0 0 0 0 12 1 0 0 0 0 25 8 2 0 0 0 0 4 0 0 0 0 0 0 12 2 0 0 0 0 0 0 5 1 0 0 0 0 0 0 0 0 0 65 49 11 0 0 0 0 70 70 70 68 68 68 65 57 39 17 68 66 63 48 15 2 3 1 0 0 67 65 60 49 30 12 2 0 0 0 62 3 0 0 0 0 0 9 1 0 0 0 0 18 6 1 0 0 0 0 3 0 0 0 0 0 0 8 1 0 0 0 0 0 0 4 1 0 0 0 0 0 0 0 0 0 56 38 6 0 0 0 0 64 65 65 65 64 64 64 62 53 33 62 61 59 52 27 6 34 22 10 2 61 59 56 48 36 19 6 1 0 0 51 3 0 0 0 0 0 8 1 0 0 0 0 22 9 2 0 0 0 0 4 0 0 0 0 0 0 8 1 0 0 0 0 0 0 5 2 0 0 0 0 0 0 0 0 0 47 26 3 0 0 0 0 56 57 57 57 57 57 55 51 41 23 54 54 52 45 21 4 27 19 10 3 53 51 46 39 28 14 3 1 0 0 42 5 1 0 0 0 0 10 2 0 0 0 0 20 9 3 1 0 0 0 6 1 0 0 0 0 0 12 3 0 0 0 0 0 0 7 3 1 0 0 0 0 0 0 0 0 41 26 5 0 0 0 0 51 50 51 50 50 49 48 45 39 28 46 46 45 38 19 4 26 18 8 2 47 44 40 34 22 11 3 0 0 0 86 55 9 0 0 0 0 75 33 2 0 0 0 66 39 11 1 0 0 0 35 2 0 0 0 0 0 64 16 1 0 0 0 0 0 30 8 1 0 0 0 0 0 0 0 0 83 73 34 2 0 0 0 74 49 12 0 0 0 0 0 0 0 82 66 22 4 0 0 0 0 0 0 80 71 41 9 1 0 0 0 0 0 85 37 3 0 0 0 0 64 17 1 0 0 0 60 29 6 1 0 0 0 16 1 0 0 0 0 0 42 5 0 0 0 0 0 0 15 3 0 0 0 0 0 0 0 0 0 83 74 38 2 0 0 0 75 52 17 1 0 0 0 0 0 0 84 79 56 14 1 0 0 0 0 0 80 75 54 18 3 1 0 0 0 0 85 24 2 0 0 0 0 51 10 0 0 0 0 50 22 5 1 0 0 0 8 0 0 0 0 0 0 32 3 0 0 0 0 0 0 12 3 0 0 0 0 0 0 0 0 0 82 73 37 3 0 0 0 72 49 12 0 0 0 0 0 0 0 83 80 72 41 7 0 0 0 0 0 79 74 57 29 6 1 0 0 0 0 84 21 2 0 0 0 0 48 9 0 0 0 0 49 19 4 1 0 0 0 8 0 0 0 0 0 0 31 3 0 0 0 0 0 0 12 3 0 0 0 0 0 0 0 0 0 81 74 40 3 0 0 0 71 44 9 0 0 0 0 0 0 0 82 80 80 64 23 2 1 1 0 0 78 71 57 30 11 2 1 0 0 0 83 23 2 0 0 0 0 50 9 0 0 0 0 45 18 3 0 0 0 0 9 0 0 0 0 0 0 38 5 0 0 0 0 0 0 17 4 1 0 0 0 0 0 0 0 0 80 72 39 4 0 0 0 70 43 6 0 0 0 0 0 0 0 81 77 73 69 61 12 9 2 0 0 76 70 52 27 10 2 0 0 0 0 83 33 3 0 0 0 0 57 13 1 0 0 0 45 17 3 0 0 0 0 14 1 0 0 0 0 0 51 12 1 0 0 0 0 0 27 9 2 0 0 0 0 0 0 0 0 80 72 38 4 0 0 0 68 39 5 0 0 0 0 0 0 0 80 76 70 66 62 53 21 2 0 0 76 70 52 25 6 1 0 0 0 0 82 34 4 0 0 0 0 58 15 1 0 0 0 41 15 2 0 0 0 0 16 1 0 0 0 0 0 58 16 1 0 0 0 0 0 29 8 1 0 0 0 0 0 0 0 0 79 71 42 6 0 0 0 68 40 6 0 0 0 0 0 0 0 77 73 67 61 57 47 29 3 0 0 75 69 51 25 7 1 0 0 0 0 81 37 4 0 0 0 0 59 15 1 0 0 0 40 15 2 0 0 0 0 17 1 0 0 0 0 0 59 16 1 0 0 0 0 0 32 10 2 0 0 0 0 0 0 0 0 78 70 43 9 0 0 0 69 43 9 0 0 0 0 0 0 0 78 72 64 57 52 50 46 9 0 0 75 71 55 25 7 2 0 0 0 0 79 38 5 0 0 0 0 60 16 1 0 0 0 41 14 2 0 0 0 0 22 1 0 0 0 0 0 63 21 1 0 0 0 0 0 31 10 2 0 0 0 0 0 0 0 0 77 69 45 10 0 0 0 69 47 13 1 0 0 0 0 0 0 76 71 62 54 47 49 61 28 3 0 74 71 60 32 10 3 0 0 0 0 77 35 5 0 0 0 0 56 15 1 0 0 0 36 12 2 0 0 0 0 21 1 0 0 0 0 0 59 20 1 0 0 0 0 0 31 10 2 0 0 0 0 0 0 0 0 75 68 47 13 1 0 0 69 50 21 4 1 0 0 0 0 0 74 69 61 53 46 51 66 45 15 1 74 73 67 50 23 8 2 0 0 0 82 46 5 0 0 0 0 68 24 1 0 0 0 58 29 7 1 0 0 0 27 1 0 0 0 0 0 68 27 1 0 0 0 0 0 41 15 2 0 0 0 0 0 0 0 0 75 52 13 1 0 0 0 74 51 13 0 0 0 0 0 0 0 32 7 2 0 0 0 0 0 0 0 85 76 33 4 0 0 0 0 0 0 81 47 6 0 0 0 0 67 26 1 0 0 0 58 30 7 0 0 0 0 26 1 0 0 0 0 0 65 24 1 0 0 0 0 0 46 18 3 0 0 0 0 0 0 0 0 74 55 16 1 0 0 0 76 59 22 2 0 0 0 0 0 0 51 14 3 1 0 0 0 0 0 0 86 83 63 16 1 0 0 0 0 0 79 43 7 0 0 0 0 62 23 2 0 0 0 53 26 6 0 0 0 0 20 1 0 0 0 0 0 54 16 1 0 0 0 0 0 33 11 1 0 0 0 0 0 0 0 0 73 57 23 2 0 0 0 75 60 28 4 0 0 0 0 0 0 65 30 6 1 0 0 0 0 0 0 85 84 79 51 10 1 0 0 0 0 78 31 4 0 0 0 0 51 14 1 0 0 0 44 18 4 1 0 0 0 14 1 0 0 0 0 0 39 8 1 0 0 0 0 0 22 6 1 0 0 0 0 0 0 0 0 74 61 28 3 0 0 0 75 63 36 10 1 0 0 0 0 0 73 53 17 3 0 0 0 0 0 0 85 84 81 74 40 9 1 0 0 0 76 27 4 0 0 0 0 43 11 1 0 0 0 36 15 4 1 0 0 0 13 1 0 0 0 0 0 36 8 1 0 0 0 0 0 19 6 1 0 0 0 0 0 0 0 0 72 63 32 4 0 0 0 74 64 40 14 2 0 0 0 0 0 74 65 35 8 2 0 0 0 0 0 82 82 81 78 69 39 13 2 0 0 74 22 3 0 0 0 0 34 7 1 0 0 0 26 9 2 0 0 0 0 7 0 0 0 0 0 0 25 5 0 0 0 0 0 0 19 6 1 0 0 0 0 0 0 0 0 69 58 31 4 0 0 0 72 60 35 8 1 0 0 0 0 0 69 58 29 6 1 0 0 0 0 0 80 80 77 69 49 26 10 2 0 0 71 25 4 0 0 0 0 28 6 0 0 0 0 12 3 1 0 0 0 0 9 1 0 0 0 0 0 24 4 0 0 0 0 0 0 16 5 1 0 0 0 0 0 0 0 0 68 60 38 9 1 0 0 70 60 36 12 2 0 0 0 0 0 70 67 53 24 5 1 1 1 0 0 79 78 76 74 65 51 40 20 4 0 67 24 5 1 0 0 0 25 6 1 0 0 0 11 4 1 0 0 0 0 7 1 0 0 0 0 0 20 4 1 0 0 0 0 0 15 6 1 0 0 0 0 0 0 0 0 63 59 42 14 1 0 0 66 55 34 13 4 1 0 0 0 0 64 62 53 29 9 2 2 2 1 0 75 75 74 72 67 61 48 34 16 3 66 34 10 2 0 0 0 41 14 2 0 0 0 25 11 3 1 0 0 0 15 3 0 0 0 0 0 29 8 2 0 0 0 0 0 20 8 3 1 0 0 0 0 0 0 0 64 62 53 27 3 0 0 65 56 37 17 7 2 1 0 1 0 65 65 63 55 35 14 13 9 6 2 72 72 71 71 70 69 66 60 41 8 61 34 13 3 0 0 0 42 19 4 1 0 0 30 16 6 2 0 0 0 27 8 1 0 0 0 0 40 17 4 1 0 0 0 0 22 11 4 1 0 0 0 0 0 0 0 60 58 52 31 6 1 0 61 53 35 19 7 3 1 1 1 1 61 61 59 54 42 21 19 14 8 3 65 65 64 64 63 62 59 54 36 9
0. 0 0. 2 0. 4 0. 6 0. 8 1. 0
Adversarial accuracy
Fi gu
re 6:
A cc
ur ac
y of
ad ve
rs ar
ia la
tta ck
(c ol
um n)
ag ai
ns ta
dv er
sa ri
al ly
tr ai
ne d
m od
el (r
ow )o
n Im
ag eN
et -1
00 .
Algorithm 1 Pseudocode for the Frank-Wolfe algorithm for the L1 attack.
1: Input: function f , initial input x ∈ [0, 1]d, L1 radius ρ, number of steps T . 2: Output: approximate maximizer x̄ of f over the truncated L1 ball B1(ρ;x) ∩ [0, 1]d centered
at x. 3: 4: x(0) ← RandomInit(x) {Random initialization} 5: for t = 1, . . . , T do 6: g ← ∇f(x(t−1)) {Obtain gradient} 7: for k = 1, . . . , d do 8: sk ← index of the coordinate of g by with kth largest norm 9: end for
10: Sk ← {s1, . . . , sk}. 11: 12: {Compute move to boundary of [0, 1] for each coordinate.} 13: for i = 1, . . . , d do 14: if gi > 0 then 15: bi ← 1− xi 16: else 17: bi ← −xi 18: end if 19: end for 20: Mk ← ∑ i∈Sk |bi| {Compute L1-perturbation of moving k largest coordinates.} 21: k∗ ← max{k |Mk ≤ ρ} {Choose largest k satisfying L1 constraint.} 22: 23: {Compute x̂ maximizing g>x over the L1 ball.} 24: for i = 1, . . . , d do 25: if i ∈ Sk∗ then 26: x̂i ← xi + bi 27: else if i = sk∗+1 then 28: x̂i ← xi + (ρ−Mk∗) sign(gi) 29: else 30: x̂i ← xi 31: end if 32: end for 33: x(t) ← (1− 1t )x
(t−1) + 1t x̂ {Average x̂ with previous iterates} 34: end for 35: x̄← x(T )
F.2 CONVERGENCE
We replicated the results in Figure 6 with 50 instead of 200 steps to see how the results changed based on the number of steps in the attack. As shown in Figure 11, the deviations are minor.
No at
tac k
L =
1 L =2 L
= 4 L
=8 L =1
6 L
= 32
L2 =
10 L2 =2
0 L2
= 40 L2
=8 0 L2 =
16 0 L2 =
32 0 L2 =
64 0
L2 =
12 80 L2 =
25 60 L2 =
51 20 L
1 =1
95 L1
= 39 0 L1
= 78 0 L1
= 15 60 L1
= 31 20 L1
= 62
40
L1 =
12 48
0
L1 =
24 96
0
L1 =
49 92
0
L -JP
EG =
0.0 31
25
L -JP
EG =
0.0 62
5
L -JP
EG =
0.1 25
L -JP
EG =
0.2 5
L -JP
EG =
0.5
L -JP
EG =
1
L2 -JP
EG =
0.0 62
5
L2 -JP
EG =
0.1 25
L2 -JP
EG =
0.2 5
L2 -JP
EG =
0.5
L2 -JP
EG =
1
L2 -JP
EG =
2
L2 -JP
EG =
4
L2 -JP
EG =
8 L1
-JP EG
= 1
L1 -JP
EG =
2
L1 -JP
EG =
4
L1 -JP
EG =
8
L1 -JP
EG =
16
L1 -JP
EG =
32
L1 -JP
EG =
64
L1 -JP
EG =
12 8
L1 -JP
EG =
25 6
L1 -JP
EG =
51 2
L1 -JP
EG =
10 24 Ela stic
= 0.1
25
Ela stic
= 0.2
5
Ela stic
= 0.5 Ela stic
= 1 Ela stic
= 2 Ela stic
= 4 Ela stic
= 8
Ela stic
= 16
No rm
al tr
ai ni
ng
L
= 1
L
= 2
L
= 4
L
= 8
L
= 16
L
= 32
L 2
= 10
L 2
= 20
L 2
= 40
L 2
= 80
L 2
= 16 0 L 2 = 32 0 L 2 = 64 0 L 2 = 12 80 L 2 = 25 60 L 2 = 51 20 L 1 = 19 5 L 1 = 39 0 L 1 = 78 0 L 1 = 15 60 L 1 = 31 20 L 1 = 62 40
L 1
= 12
48 0
L 1
= 24
96 0
L 1
= 49
92 0
L -JP
EG
= 0.
03 12 5 L -JP EG = 0. 06 25 L -JP EG = 0. 12 5 L -JP EG = 0. 25 L -JP EG = 0. 5 L -JP EG = 1 L 2 -JP EG = 0. 06 25 L 2 -JP EG = 0. 12 5 L 2 -JP EG = 0. 25 L 2 -JP EG = 0. 5 L 2 -JP EG = 1 L 2 -JP EG = 2 L 2 -JP EG = 4 L 2 -JP EG = 8 L 1 -JP EG = 1 L 1 -JP EG = 2 L 1 -JP EG = 4 L 1 -JP EG = 8 L 1 -JP EG = 16 L 1 -JP EG = 32 L 1 -JP EG = 64 L 1 -JP EG = 12 8 L 1 -JP EG = 25 6 L 1 -JP EG = 51 2 L 1 -JP EG = 10 24 El as tic = 0. 12 5 El as tic = 0. 25 El as tic = 0. 5 El as tic = 1 El as tic = 2 El as tic = 4 El as tic = 8 El as tic = 16
93 59
9 0
0 0
0 91
83 49
5 1
3 4
3 3
1 85
71 39
9 0
0 0
0 0
21 0
0 0
0 0
92 91
83 50
5 0
2 2
89 82
61 24
2 0
0 0
0 0
0 9
0 0
0 0
0 0
0
93 89
79 37
1 0
0 92
92 89
74 25
0 2
2 2
1 90
86 73
44 9
0 0
0 0
83 50
3 0
0 0
92 92
91 88
72 22
0 0
91 90
84 66
28 3
0 0
0 0
0 67
11 0
0 0
0 0 0 93 91 86 64 12 0 0 92 92 90 83 50 4 0 2 2 1 90 87 78 54 17 1 0 0 0 87 69 14 0 0 0 92 92 91 90 81 41 1 0 92 91 86 73 39 7 0 0 0 0 0 78 26 1 0 0 0 0 1 91 90 88 78 40 1 0 91 91 90 85 67 17 0 0 1 0 90 88 80 62 28 3 0 0 0 88 79 40 1 0 0 91 90 90 89 84 59 8 0 90 89 86 77 52 16 1 0 0 0 0 83 50 3 0 0 0 0 0 89 88 87 82 63 13 0 88 88 87 85 74 33 0 0 0 0 87 86 80 65 35 5 0 0 0 86 80 53 4 0 0 88 88 88 87 83 66 16 0 88 87 85 77 57 24 3 0 0 0 0 83 66 12 0 0 0 0 0 83 83 82 80 71 39 2 83 83 82 81 75 49 4 0 0 0 83 81 78 69 48 16 1 0 0 81 78 64 20 0 0 83 83 83 82 80 71 33 1 83 82 81 76 65 39 11 1 0 0 0 80 72 34 3 0 0 0 1 84 83 82 77 66 47 23 84 84 82 80 69 43 10 0 0 0 81 78 73 63 48 27 8 1 0 82 77 62 29 6 1 84 84 84 83 79 66 36 8 84 83 81 76 66 49 32 18 9 4 1 80 66 22 2 0 0 0 1 92 81 48 4 0 0 0 91 89 80 42 3 3 5 5 3 1 89 82 63 27 2 0 0 0 0 59 10 0 0 0 0 92 91 89 79 37 1 2 3 90 87 74 42 8 0 0 0 0 0 0 38 1 0 0 0 0 0 0 94 88 67 13 0 0 0 93 92 87 62 8 0 3 3 2 1 91 85 70 35 4 0 0 0 0 74 22 0 0 0 0 93 93 91 86 55 4 0 1 92 89 80 53 12 0 0 0 0 0 0 51 3 0 0 0 0 0 0 93 90 81 42 1 0 0 93 92 90 78 32 1 3 3 2 1 92 88 79 53 15 0 0 0 0 85 55 4 0 0 0 93 92 92 89 76 26 0 1 92 90 85 69 30 3 0 0 0 0 0 69 13 0 0 0 0 0 0 92 90 86 66 13 0 0 92 91 90 85 61 9 0 3 2 1 91 89 84 68 33 4 0 0 0 88 77 30 0 0 0 92 92 91 90 85 58 6 0 91 90 87 78 51 14 1 0 0 0 0 79 32 1 0 0 0 0 0 90 89 87 77 38 1 0 90 90 89 86 75 33 0 0 1 0 89 88 86 77 52 16 1 0 0 88 84 60 7 0 0 90 90 90 89 87 75 25 0 90 89 87 82 64 31 5 0 0 0 0 82 55 5 0 0 0 0 0 87 86 85 80 60 11 0 87 86 86 85 80 57 8 0 0 0 86 86 84 80 67 36 5 0 0 85 84 76 37 1 0 86 86 86 86 85 81 56 6 86 86 85 83 76 58 24 3 0 0 0 82 66 18 1 0 0 0 0 80 79 79 77 68 34 1 80 80 80 79 77 67 32 0 0 0 80 79 79 77 73 57 24 1 0 79 79 75 62 14 0 79 79 80 79 79 77 69 30 79 79 79 78 76 69 51 23 4 0 0 77 70 38 6 1 0 0 1 73 73 73 71 66 49 14 73 73 73 73 71 66 50 15 0 0 73 73 72 71 69 62 47 21 1 73 72 70 64 47 14 73 73 73 73 72 72 67 54 73 73 73 72 71 68 61 50 37 25 16 72 67 51 23 6 2 2 3 69 69 68 67 63 49 33 69 68 68 68 67 62 50 32 17 4 68 68 68 67 65 58 46 34 24 68 68 66 61 48 34 69 69 68 68 68 67 64 52 68 68 68 68 67 64 58 49 40 35 32 68 64 52 34 21 11 6 7 77 76 74 68 55 30 7 77 76 75 74 67 54 24 3 0 0 76 76 74 71 64 50 31 16 5 76 74 69 52 16 1 77 77 76 76 75 72 61 26 76 76 76 74 71 64 47 27 11 4 2 73 64 33 4 1 0 2 4 93 84 55 6 0 0 0 92 90 84 52 5 1 4 4 3 1 91 88 75 44 8 0 0 0 0 68 17 0 0 0 0 93 92 91 84 50 4 0 2 91 89 81 55 15 1 0 0 0 0 0 41 2 0 0 0 0 0 0 94 86 62 13 0 0 0 93 92 85 60 10 0 3 3 3 1 92 90 83 63 28 3 0 0 0 74 29 1 0 0 0 93 93 91 86 62 11 0 1 92 91 87 73 38 7 0 0 0 0 0 47 3 0 0 0 0 0 0 93 73 31 2 0 0 0 92 87 70 25 1 1 2 3 2 1 92 90 83 66 31 3 0 0 0 55 10 0 0 0 0 93 92 89 76 35 2 0 1 92 90 84 67 29 3 0 0 0 0 0 34 1 0 0 0 0 0 0 94 56 14 1 0 0 0 91 82 48 8 0 0 1 2 2 1 92 89 83 68 37 8 0 0 0 40 4 0 0 0 0 92 91 86 64 19 1 0 1 91 89 83 69 43 11 0 0 0 0 0 25 0 0 0 0 0 0 0 94 49 15 1 0 0 0 88 74 40 8 0 0 1 2 2 1 90 87 81 70 45 14 0 0 0 38 5 0 0 0 0 92 91 85 61 20 1 0 0 91 87 77 58 28 6 0 0 0 0 0 26 1 0 0 0 0 0 0 93 54 13 1 0 0 0 89 78 43 7 0 0 0 1 2 1 90 86 79 62 35 9 1 0 0 39 6 0 0 0 0 92 90 84 61 19 1 0 0 90 87 79 63 38 14 2 0 0 0 0 35 1 0 0 0 0 0 0 92 66 20 1 0 0 0 90 82 55 12 0 0 0 0 1 1 90 87 82 70 45 15 1 0 0 44 9 0 0 0 0 91 90 84 62 22 2 0 0 89 86 77 57 30 10 2 0 0 0 0 46 3 0 0 0 0 0 0 89 77 48 12 1 0 0 88 84 71 39 8 0 0 0 1 0 88 86 83 74 55 27 4 0 0 67 34 7 1 0 0 89 88 85 78 53 17 2 0 88 86 82 71 52 28 10 2 0 0 0 63 19 1 0 0 0 0 0 46 46 47 44 37 20 2 45 45 46 46 42 34 16 2 0 0 46 46 46 47 47 47 42 26 4 48 49 45 39 25 3 47 48 48 49 49 48 45 35 50 52 54 57 57 56 54 46 30 14 6 40 34 23 8 1 0 0 1 92 88 75 31 1 0 0 91 90 86 64 14 0 3 3 2 1 88 81 62 27 3 0 0 0 0 90 83 50 2 0 0 91 91 91 90 87 70 17 0 91 90 88 82 63 29 5 0 0 0 0 57 5 0 0 0 0 0 1 91 88 79 46 3 0 0 90 89 87 74 30 1 1 2 2 1 88 83 68 38 7 0 0 0 0 90 87 76 28 0 0 90 90 90 90 89 84 56 5 90 90 89 87 80 61 30 5 0 0 0 66 11 0 0 0 0 0 0 89 87 81 59 10 0 0 88 88 86 78 44 4 0 2 1 1 87 83 73 48 13 1 0 0 0 88 87 83 63 5 0 89 89 89 88 88 86 77 37 89 88 88 87 85 78 63 32 5 0 0 71 21 0 0 0 0 0 0 86 84 80 65 22 1 0 85 85 83 78 56 12 0 1 1 0 84 81 74 56 23 2 0 0 0 85 85 83 75 30 0 86 86 86 85 85 84 80 63 86 85 85 85 84 81 73 54 24 4 0 72 33 1 0 0 0 0 0 83 81 79 70 39 4 0 82 82 81 78 64 27 1 0 0 0 82 80 75 63 37 7 0 0 0 82 82 82 79 61 3 83 83 83 82 82 82 80 74 83 83 82 82 82 80 77 69 52 28 11 73 47 5 0 0 0 0 0 80 79 77 69 44 6 0 79 79 78 75 65 30 2 0 0 0 79 78 74 62 38 9 0 0 0 79 79 78 77 70 35 80 80 80 79 79 79 77 73 80 80 80 79 79 78 76 74 69 62 53 72 51 8 0 0 0 0 0 94 74 23 0 0 0 0 92 87 63 11 0 2 2 2 2 1 86 70 35 5 0 0 0 0 0 50 4 0 0 0 0 93 93 90 75 25 0 1 1 91 86 71 36 4 0 0 0 0 0 0 14 0 0 0 0 0 0 1 93 77 30 0 0 0 0 92 88 70 18 0 2 3 3 2 0 86 72 40 7 0 0 0 0 0 69 15 0 0 0 0 93 93 91 84 48 2 0 2 91 88 78 51 11 0 0 0 0 0 0 21 0 0 0 0 0 0 1 93 86 58 7 0 0 0 93 91 81 42 2 1 2 2 2 1 88 78 52 15 0 0 0 0 0 83 48 2 0 0 0 93 93 92 89 74 22 0 1 92 90 85 67 28 3 0 0 0 0 0 37 1 0 0 0 0 0 0 93 88 71 20 0 0 0 92 91 86 61 9 0 2 2 2 1 89 82 63 25 2 0 0 0 0 89 76 27 0 0 0 93 93 92 91 87 60 6 0 92 91 89 82 60 22 2 0 0 0 0 53 2 0 0 0 0 0 1 91 86 69 21 0 0 0 91 89 85 63 13 0 2 2 2 1 88 82 66 31 3 0 0 0 0 90 85 62 7 0 0 92 91 91 91 88 80 39 1 91 91 90 87 78 54 17 1 0 0 0 58 5 0 0 0 0 0 0 90 87 80 51 5 0 0 89 88 86 76 38 2 1 2 2 1 87 84 73 46 11 0 0 0 0 89 87 79 44 1 0 90 89 89 89 88 85 72 20 90 89 89 88 85 77 55 19 1 0 0 67 14 0 0 0 0 0 0 88 86 81 61 15 0 0 88 87 85 79 52 7 0 1 2 0 86 84 77 56 22 2 0 0 0 87 87 84 70 15 0 88 88 88 88 87 86 81 56 88 88 88 87 86 82 73 49 15 1 0 71 26 0 0 0 0 0 0 85 83 80 68 30 2 0 85 84 83 78 62 19 0 0 1 0 84 82 77 63 34 5 0 0 0 85 84 83 78 48 0 85 85 85 85 84 84 81 72 85 85 85 84 83 82 79 69 47 16 3 73 40 2 0 0 0 0 0 93 79 37 1 0 0 0 92 89 74 26 1 1 3 3 2 1 87 77 50 13 0 0 0 0 0 71 21 0 0 0 0 93 93 91 86 58 6 0 1 92 90 85 66 26 2 0 0 0 0 0 24 0 0 0 0 0 0 0 93 85 60 10 0 0 0 92 90 83 50 5 1 4 4 3 1 89 82 63 26 2 0 0 0 0 83 52 5 0 0 0 93 92 91 89 79 35 1 2 92 91 89 81 57 17 1 0 0 0 0 43 2 0 0 0 0 0 0 91 78 44 4 0 0 0 90 87 77 40 2 0 2 2 2 1 87 80 60 21 1 0 0 0 0 88 75 29 0 0 0 92 92 92 91 86 67 14 0 92 91 90 87 76 45 9 0 0 0 0 43 1 0 0 0 0 0 0 91 87 75 34 1 0 0 90 89 86 70 23 0 1 2 1 1 88 85 72 42 8 0 0 0 0 89 83 59 9 0 0 91 91 90 90 88 80 47 2 91 90 90 88 83 69 37 5 0 0 0 60 8 0 0 0 0 0 0 89 87 79 47 4 0 0 89 88 86 76 37 1 1 3 2 1 88 85 76 51 14 0 0 0 0 88 86 75 34 0 0 89 89 89 89 88 84 69 21 89 89 89 88 86 79 60 22 1 0 0 66 13 0 0 0 0 0 0 88 85 79 54 8 0 0 87 87 85 77 46 4 0 2 1 1 86 84 77 56 20 1 0 0 0 87 85 80 55 5 0 88 88 88 87 87 85 77 43 88 87 87 87 85 82 71 47 12 0 0 68 18 0 0 0 0 0 0 88 86 81 60 14 0 0 87 87 85 79 53 7 0 2 2 1 86 84 78 60 25 2 0 0 0 87 86 83 67 16 0 88 88 88 87 87 85 80 58 88 88 87 87 86 83 76 59 27 3 0 71 25 0 0 0 0 0 0 86 84 81 64 20 0 0 86 85 84 79 58 12 0 0 1 0 85 83 78 63 30 3 0 0 0 86 85 82 72 32 0 86 86 86 86 85 85 81 66 86 86 86 85 84 83 79 69 46 15 2 72 31 1 0 0 0 0 0 85 84 81 67 27 1 0 85 84 83 79 62 18 0 0 1 0 84 83 78 65 34 5 0 0 0 85 84 82 75 45 1 85 85 85 85 85 84 81 70 85 85 85 85 84 83 80 73 59 36 13 72 38 2 0 0 0 0 0 84 83 80 71 38 2 0 84 83 82 79 67 28 0 0 0 0 83 82 78 68 43 9 0 0 0 84 83 82 77 56 4 84 84 84 84 84 83 81 74 84 84 84 84 83 82 80 76 66 49 26 75 48 4 0 0 0 0 0 82 81 79 71 40 3 0 82 82 81 78 67 31 1 0 0 0 81 80 77 67 44 11 0 0 0 82 82 81 76 59 9 82 82 82 82 82 81 79 73 82 82 82 82 82 81 79 75 69 56 38 74 50 6 0 0 0 0 0 93 85 60 13 0 0 0 92 90 82 50 8 0 2 2 2 1 89 82 64 32 6 0 0 0 0 65 19 1 0 0 0 92 91 87 69 24 2 0 0 88 79 57 23 3 0 0 0 0 0 0 86 60 5 0 0 0 0 0 91 84 62 17 1 0 0 90 88 81 54 12 0 0 0 0 0 87 82 67 38 10 1 0 0 0 70 29 2 0 0 0 90 89 86 71 32 3 0 0 86 79 58 26 4 0 0 0 0 0 0 87 78 33 0 0 0 0 0 87 81 65 25 2 0 0 87 85 79 57 18 1 0 0 0 0 84 79 66 42 15 2 0 0 0 73 43 8 0 0 0 87 86 84 74 45 9 0 0 84 77 61 35 10 1 0 0 0 0 0 85 81 62 8 0 0 0 0 82 78 67 36 6 0 0 82 80 76 59 23 3 0 0 0 0 77 70 56 36 15 3 0 0 0 73 52 19 3 0 0 81 81 80 74 54 20 2 0 79 74 64 45 20 6 1 0 0 0 0 81 79 72 40 7 1 0 0 81 75 65 42 15 1 0 80 78 70 48 17 2 0 0 0 0 70 59 45 31 18 9 3 1 0 68 47 20 4 0 0 78 78 77 70 50 20 3 0 77 73 64 47 26 10 3 1 0 0 0 79 76 71 58 37 19 11 7 76 71 64 45 19 2 0 75 73 66 47 18 3 0 0 0 0 63 54 39 25 15 8 3 0 0 64 46 19 4 1 0 74 74 73 66 49 21 4 0 72 69 61 46 27 11 4 1 0 0 0 74 72 68 58 45 33 23 10 68 65 59 41 14 1 0 68 66 59 42 17 3 0 0 0 0 54 45 33 20 10 3 1 0 0 58 41 16 3 0 0 68 67 65 59 43 19 4 0 65 62 54 40 24 11 4 1 0 0 0 68 67 63 53 42 34 28 15 61 58 53 38 13 1 0 60 59 53 39 15 2 0 0 0 0 50 41 31 19 9 3 0 0 0 53 37 14 2 0 0 60 59 58 53 39 17 3 0 58 55 49 37 21 8 2 1 0 0 0 61 60 58 49 39 29 23 18
0. 0 0. 2 0. 4 0. 6 0. 8 1. 0
Adversarial accuracy
Fi gu
re 8:
A cc
ur ac
y of
ad ve
rs ar
ia la
tta ck
(c ol
um n)
ag ai
ns ta
dv er
sa ri
al ly
tr ai
ne d
m od
el (r
ow )o
n C
IF A
R -1
0.
L ∞ L 2 L 1
L ∞
-J P
E G L 1 -J P E G E la st ic
Normal Training
L∞ ε = 1 L∞ ε = 2 L∞ ε = 4 L∞ ε = 8 L∞ ε = 16 L∞ ε = 32
L∞-JPEG ε = 0.03125
L∞-JPEG ε = 0.0625
L∞-JPEG ε = 0.125
L∞-JPEG ε = 0.25
L∞-JPEG ε = 0.5
L∞-JPEG ε = 1
17 16 48 5 25 3
110110110110110110
51 49 69 31 37 21
63 59 73 38 39 28
74 67 76 47 40 36
83 72 77 50 39 43
89 75 78 55 40 51
94 72 76 58 48 45
110110110110110110
48 43 61 51 42 17
54 50 66 63 49 21
59 55 69 74 58 25
63 59 71 81 64 29
68 64 73 88 79 34
68 64 71 94 99 35
L ∞ L 2 L 1
L ∞
-J P
E G L 1 -J P E G E la st ic
Normal Training
L2 ε = 40 L2 ε = 80 L2 ε = 160 L2 ε = 320 L2 ε = 640 L2 ε = 2560
L1-JPEG ε = 2 L1-JPEG ε = 8 L1-JPEG ε = 64 L1-JPEG ε = 256 L1-JPEG ε = 512 L1-JPEG ε = 1024
17 16 48 5 25 3
110110110110110110
53 53 74 32 38 22
64 63 80 44 40 30
73 73 84 54 41 38
80 81 88 64 46 45
84 86 88 70 51 52
87 85 86 78 71 66
110110110110110110
39 37 62 32 41 12
49 47 68 54 51 18
60 58 73 76 66 26
65 62 75 84 85 30
68 66 76 87 92 34
68 66 75 88 96 35
L ∞ L 2 L 1
L ∞
-J P
E G L 1 -J P E G E la st ic
Normal Training
L1 ε = 195 L1 ε = 390 L1 ε = 780 L1 ε = 1560 L1 ε = 6240 L1 ε = 49920
Elastic ε = 0.125
Elastic ε = 0.25
Elastic ε = 0.5
Elastic ε = 1
Elastic ε = 2
Elastic ε = 8
17 16 48 5 25 3
110110110110110110
36 38 70 19 34 12
40 41 79 24 39 13
26 26 79 15 37 9
18 15 80 10 37 7
17 13 77 10 36 10
49 47 61 47 51 29
110110110110110110
40 37 63 19 24 41
41 38 65 23 25 53
43 40 65 28 27 64
47 41 57 33 29 75
49 35 51 32 29 89
45 31 37 27 25 86
Attack (evaluation)
Figure 10: Replica of the first three block rows of Figure 6 with different random seeds. Deviations in results are minor.
No at
tac k L
= 1 L =
2 L
= 4 L =
8 L
= 16 L =
32 L2 =
15 0 L2 =
30 0 L2 =
60 0
L2 =
12 00
L2 =
24 00
L2 =
48 00 L1 =
95 62
.44
L1 =
19 12
5
L1 =
38 25
0.1
L1 =
76 50
0
L1 =
15 30
00
L1 =
30 60
00
L1 =
61 20
00
L -JP
EG =
0.0 31
25
L -JP
EG =
0.0 62
5
L -JP
EG =
0.1 25
L -JP
EG =
0.2 5
L -JP
EG =
0.5
L -JP
EG =
1
L -JP
EG =
2 L2
-JP EG
= 2
L2 -JP
EG =
4
L2 -JP
EG =
8
L2 -JP
EG =
16
L2 -JP
EG =
32
L2 -JP
EG =
64
L2 -JP
EG =
12 8
L2 -JP
EG =
25 6
L1 -JP
EG =
12 8
L1 -JP
EG =
25 6
L1 -JP
EG =
51 2
L1 -JP
EG =
10 24
L1 -JP
EG =
20 48
L1 -JP
EG =
40 96
L1 -JP
EG =
81 92
L1 -JP
EG =
16 38
4
L1 -JP
EG =
32 76
8
L1 -JP
EG =
65 53
6
L1 -JP
EG =
13 10 72 Ela
stic =
0.2 5
Ela stic
= 0.5 Ela stic
= 1
Ela stic
= 2
Ela stic
= 4
Ela stic
= 8
Ela stic
= 16 Fog
= 12
8 Fog =
25 6 Fog =
51 2
Fog =
10 24
Fog =
20 48
Fog =
40 96
Fog =
81 92
Fog =
16 38
4
Fog =
32 76
8
Fog =
65 53 6 Ga bo r
=6 .25
Ga bo
r =1
2.5
Ga bo
r =2
5
Ga bo
r =5
0
Ga bo
r =1
00
Ga bo
r =2
00
Ga bo
r =4
00
Ga bo
r =8
00
Ga bo
r =1
60 0
Ga bo
r =3
20 0
Sn ow
= 0.0
31 25
Sn ow
= 0.0
62 5
Sn ow
= 0.1
25
Sn ow
= 0.2
5
Sn ow
= 0.5
Sn ow
= 1
Sn ow
= 2
Sn ow
= 4
Sn ow
= 8
Sn ow
= 16
No rm
al tr
ai ni
ng
L
= 1
L
= 2
L
= 4
L
= 8
L
= 16
L
= 32
L 2
= 15 0 L 2 = 30 0 L 2 = 60 0 L 2 = 12 00 L 2 = 24 00 L 2 = 48 00
L 1
= 95
62 .4 4 L 1 = 19 12 5 L 1 = 38 25 0. 1 L 1 = 76 50 0 L 1 = 15 30 00 L 1 = 30 60 00 L 1 = 61 20 00
L -JP
EG
= 0.
03 12 5 L -JP EG = 0. 06 25 L -JP EG = 0. 12 5 L -JP EG = 0. 25 L -JP EG = 0. 5 L -JP EG = 1 L -JP EG = 2 L 2 -JP EG = 2 L 2 -JP EG = 4 L 2 -JP EG = 8 L 2 -JP EG = 16 L 2 -JP EG = 32 L 2 -JP EG = 64 L 2 -JP EG = 12 8 L 2 -JP EG = 25 6 L 1 -JP EG = 12 8 L 1 -JP EG = 25 6 L 1 -JP EG = 51 2 L 1 -JP EG = 10 24 L 1 -JP EG = 20 48 L 1 -JP EG = 40 96 L 1 -JP EG = 81 92 L 1 -JP EG = 16 38 4 L 1 -JP EG = 32 76 8 L 1 -JP EG = 65 53 6 L 1 -JP EG = 13 10 72 El as tic = 0. 25 El as tic = 0. 5 El as tic = 1 El as tic = 2 El as tic = 4 El as tic = 8 El as tic = 16 Fo g = 12 8 Fo g = 25 6 Fo g = 51 2 Fo g = 10 24 Fo g = 20 48 Fo g = 40 96 Fo g = 81 92 Fo g = 16 38 4 Fo g = 32 76 8 Fo g = 65 53 6 Ga bo r = 6. 25 Ga bo r = 12 .5 Ga bo r = 25 Ga bo r = 50 Ga bo r = 10 0 Ga bo r = 20 0 Ga bo r = 40 0 Ga bo r = 80 0 Ga bo r = 16 00 Ga bo r = 32 00 Sn ow = 0. 03 12 5 Sn ow = 0. 06 25 Sn ow = 0. 12 5 Sn ow = 0. 25 Sn ow = 0. 5 Sn ow = 1 Sn ow = 2 Sn ow = 4 Sn ow = 8 Sn ow = 16
87 28
2 0
0 0
0 58
13 0
0 0
0 70
44 13
2 0
0 0
22 1
0 0
0 0
0 72
28 2
0 0
0 0
0 59
32 9
1 0
0 0
0 0
0 0
80 50
9 0
0 0
0 70
37 6
0 0
0 0
0 0
0 18
5 2
1 0
0 0
0 0
0 64
37 10
1 0
0 0
0 0
0
86 84
70 15
0 0
0 85
82 50
3 0
0 81
70 43
10 1
0 0
84 72
16 0
0 0
0 86
84 68
13 0
0 0
0 77
64 36
10 1
0 0
0 0
0 0
84 75
39 4
0 0
0 78
60 23
3 0
0 0
0 0
0 75
34 6
1 0
0 0
0 0
0 80
67 33
7 1
0 0
0 0 0 85 85 80 51 3 0 0 85 83 71 21 0 0 81 74 57 24 3 0 0 84 81 49 2 0 0 0 85 84 77 38 1 0 0 0 79 71 52 22 4 0 0 0 0 0 0 84 79 53 10 0 0 0 76 56 19 2 1 1 0 1 0 0 81 62 16 2 0 0 0 0 0 0 80 73 47 14 2 0 0 0 0 0 84 84 82 74 24 0 0 84 83 79 50 3 0 81 76 64 39 9 1 0 83 82 70 13 0 0 0 84 83 80 56 6 0 0 0 79 74 62 38 11 1 0 0 0 0 0 83 79 62 18 1 0 0 73 50 15 2 1 1 1 1 1 0 82 77 41 5 0 0 0 0 0 0 79 76 59 23 4 1 1 0 0 0 80 79 79 76 59 7 0 79 78 73 52 9 0 73 65 53 31 9 1 0 80 77 60 15 0 0 0 79 78 71 46 8 0 0 0 74 70 59 39 16 4 1 0 0 0 0 79 77 66 34 4 0 0 66 40 9 1 1 1 1 1 1 1 79 78 65 18 1 0 0 0 0 0 75 73 65 38 11 3 1 1 1 1 75 74 74 73 67 35 1 73 71 64 32 4 0 59 46 27 10 2 0 0 73 68 39 5 0 0 0 74 70 58 29 4 0 0 0 64 58 46 31 15 5 1 0 0 0 0 73 71 65 44 12 1 0 58 29 6 1 1 1 1 1 1 0 74 73 69 46 5 1 1 1 0 0 68 66 61 44 21 8 2 1 0 0 71 71 70 69 63 43 14 68 60 35 7 0 0 39 24 9 3 0 0 0 65 45 11 1 0 0 1 68 60 35 9 2 0 1 1 59 51 41 27 15 7 2 0 0 0 0 70 68 63 46 16 2 0 58 33 9 1 1 1 1 1 1 0 70 69 64 52 18 3 3 2 1 1 63 61 56 41 21 7 2 1 0 0 87 82 53 4 0 0 0 85 78 36 1 0 0 80 71 43 10 1 0 0 81 53 3 0 0 0 0 85 81 51 3 0 0 0 0 75 55 23 4 0 0 0 0 0 0 0 84 72 32 2 0 0 0 77 56 19 2 0 0 0 0 0 0 65 21 4 1 0 0 0 0 0 0 78 61 24 4 0 0 0 0 0 0 85 84 73 23 0 0 0 85 82 66 10 0 0 83 77 61 26 3 0 0 84 76 25 0 0 0 0 85 84 73 20 0 0 0 0 79 69 42 11 1 0 0 0 0 0 0 84 76 45 5 0 0 0 76 55 20 2 0 0 0 0 0 0 78 45 8 1 0 0 0 0 0 0 79 69 35 7 1 0 0 0 0 0 84 84 81 58 4 0 0 84 83 78 41 1 0 83 80 73 50 13 0 0 84 81 61 4 0 0 0 84 84 80 52 3 0 0 0 81 76 61 30 6 0 0 0 0 0 0 83 79 58 12 0 0 0 75 52 16 2 0 0 0 0 0 0 81 67 20 2 0 0 0 0 0 0 79 73 50 14 2 0 0 0 0 0 82 82 81 74 29 0 0 82 82 80 68 17 0 81 80 77 67 38 6 0 82 81 76 37 0 0 0 82 82 81 74 28 0 0 0 81 80 74 59 28 6 0 0 0 0 0 81 78 67 24 2 0 0 69 45 12 1 0 1 1 1 0 1 81 76 45 6 0 0 1 0 0 0 76 72 60 26 5 1 0 0 1 0 77 77 76 73 57 7 0 77 76 76 72 49 3 77 76 75 71 58 26 3 76 76 75 64 13 0 0 77 77 76 74 58 8 0 0 77 76 74 69 56 29 8 1 0 0 0 76 74 68 39 5 1 0 61 34 8 1 1 0 0 1 0 1 76 74 61 19 1 0 1 1 0 0 69 68 61 38 12 3 1 1 1 1 68 68 68 67 61 28 1 68 68 68 67 60 20 68 68 68 66 61 46 16 68 68 67 64 38 2 0 68 68 68 67 63 36 2 0 68 68 68 66 62 51 33 15 6 3 3 68 67 63 49 12 1 0 51 27 6 1 1 1 1 1 1 1 68 67 61 31 3 1 1 1 1 1 60 59 55 42 19 6 2 1 1 1 86 71 26 1 0 0 0 82 65 16 0 0 0 83 79 61 21 1 0 0 69 20 1 0 0 0 0 84 72 25 1 0 0 0 0 74 53 20 3 0 0 0 0 0 0 0 82 67 22 1 0 0 0 74 48 10 1 0 0 0 0 0 0 47 13 3 1 0 0 0 0 0 0 73 53 18 2 0 0 0 0 0 0 86 78 43 3 0 0 0 84 75 34 1 0 0 84 81 72 41 5 0 0 77 43 3 0 0 0 0 84 79 49 5 0 0 0 0 80 68 40 10 1 0 0 0 0 0 0 83 71 32 2 0 0 0 74 49 12 1 0 0 0 0 0 0 61 22 4 1 0 0 0 0 0 0 75 58 23 3 1 0 0 0 0 0 85 81 63 13 0 0 0 84 80 57 7 0 0 85 83 78 58 18 1 0 81 66 19 1 0 0 0 84 82 69 23 1 0 0 0 82 77 59 26 5 0 0 0 0 0 0 83 75 43 4 0 0 0 74 51 13 1 0 0 0 0 0 0 73 38 7 1 0 0 0 0 0 0 76 65 32 6 1 0 0 0 0 0 84 82 71 31 1 0 0 83 81 69 23 0 0 84 83 81 73 43 5 0 82 75 48 5 0 0 0 84 82 77 51 5 0 0 0 83 81 74 53 19 3 0 0 0 0 0 82 78 53 8 0 0 0 72 48 12 1 0 0 0 0 0 0 78 55 16 2 0 0 0 0 0 0 76 67 39 9 1 0 0 0 0 0 81 79 73 46 4 0 0 80 79 70 39 2 0 81 80 79 77 64 24 1 79 76 63 20 1 0 0 80 79 77 64 20 1 0 0 80 79 78 70 44 13 1 0 0 0 0 79 75 58 14 0 0 0 69 45 11 0 0 0 0 0 0 0 78 63 27 4 0 0 0 0 0 0 72 66 46 15 2 0 0 0 1 1 79 77 73 54 12 0 0 78 77 72 49 8 0 79 79 78 76 71 49 8 77 75 69 47 4 0 0 78 77 75 69 45 4 0 0 78 78 77 74 66 38 9 1 0 0 0 77 74 62 23 1 0 0 64 39 9 1 0 0 0 0 0 0 75 67 39 8 1 0 0 0 0 0 69 64 49 21 5 1 1 0 1 1 72 71 69 60 28 1 0 72 71 68 57 22 1 72 72 71 70 67 57 24 71 70 66 54 18 0 0 72 71 70 67 56 17 0 0 71 71 70 69 65 50 25 7 1 1 0 70 68 61 35 4 0 0 53 30 8 1 1 0 1 0 0 0 69 62 43 14 2 1 1 0 0 0 60 56 46 25 9 2 1 1 1 1 87 75 29 1 0 0 0 83 60 9 0 0 0 77 58 25 4 0 0 0 86 83 57 3 0 0 0 86 86 82 57 4 0 0 0 84 81 71 46 13 1 0 0 0 0 0 82 66 18 1 0 0 0 75 48 11 1 0 0 0 0 0 0 41 9 2 1 0 0 0 0 0 0 73 47 13 2 0 0 0 0 0 0 87 81 48 3 0 0 0 84 71 21 0 0 0 79 65 33 6 0 0 0 86 85 76 20 0 0 0 87 86 84 74 21 0 0 0 85 83 78 62 30 5 0 0 0 0 0 83 71 22 1 0 0 0 75 51 13 1 0 0 0 0 0 0 55 15 3 1 0 0 0 0 0 0 74 51 18 3 0 0 0 0 0 0 86 83 68 16 0 0 0 84 78 44 3 0 0 80 71 44 11 1 0 0 86 85 83 60 2 0 0 86 86 85 81 55 2 0 0 85 83 81 74 51 20 3 0 0 0 0 83 73 31 2 0 0 0 75 51 13 1 0 0 0 0 0 0 71 33 6 1 0 0 0 0 0 0 76 58 23 4 1 0 0 0 0 0 84 83 77 43 3 0 0 83 81 66 16 0 0 80 74 56 24 3 0 0 84 84 83 77 20 0 0 84 84 84 82 73 21 0 0 84 83 82 79 70 49 18 4 1 0 0 83 75 43 4 0 0 0 73 49 13 1 0 0 0 0 0 0 78 56 18 3 0 0 0 0 0 0 76 65 34 6 1 0 0 0 0 1 81 80 78 66 19 1 0 80 79 74 42 3 0 79 75 65 39 10 1 0 81 81 80 79 66 1 0 81 81 80 80 77 59 3 0 81 80 80 79 76 71 57 35 13 3 2 79 75 53 10 0 0 0 70 46 12 1 0 0 0 0 0 0 79 72 41 8 1 0 0 0 0 0 74 68 48 15 2 1 0 0 0 0 79 79 77 68 30 2 0 79 78 74 53 9 0 77 75 69 54 24 4 0 80 79 79 77 73 41 0 80 80 80 79 76 69 39 1 80 79 79 78 77 76 74 69 62 53 48 79 75 59 14 1 0 0 67 44 11 1 0 0 0 0 0 0 78 73 50 16 3 1 1 0 0 0 71 68 53 21 5 1 1 0 0 1 78 77 76 64 20 1 0 78 77 73 50 8 0 77 75 69 52 22 4 0 78 78 78 76 70 61 13 78 78 78 78 77 72 65 53 78 78 78 78 77 76 74 72 69 65 60 77 74 57 13 1 0 0 65 42 10 1 0 0 0 0 0 0 77 71 49 18 4 1 1 1 0 0 70 65 49 20 6 2 1 0 1 1 87 64 14 0 0 0 0 80 45 3 0 0 0 75 53 19 2 0 0 0 84 72 17 0 0 0 0 86 85 76 26 0 0 0 0 83 78 61 27 5 0 0 0 0 0 0 81 61 14 1 0 0 0 75 50 12 1 0 0 0 0 0 0 28 6 2 0 0 0 0 0 0 0 71 44 13 2 0 0 0 0 0 0 87 75 28 1 0 0 0 82 60 9 0 0 0 77 60 25 4 0 0 0 86 81 51 2 0 0 0 87 86 82 59 4 0 0 0 84 82 74 50 15 1 0 0 0 0 0 82 66 18 1 0 0 0 75 48 11 0 0 0 0 0 0 0 38 9 2 1 0 0 0 0 0 0 74 45 14 2 0 0 0 0 0 0 86 81 50 4 0 0 0 84 74 26 1 0 0 80 67 37 7 0 0 0 86 84 75 19 0 0 0 86 86 84 77 27 0 0 0 85 83 80 68 38 7 0 0 0 0 0 83 70 23 1 0 0 0 75 50 13 1 0 0 0 0 0 0 54 15 3 1 0 0 0 0 0 0 75 55 19 2 0 0 0 0 0 0 85 83 70 17 0 0 0 84 81 52 4 0 0 81 75 52 16 1 0 0 85 85 82 57 1 0 0 85 85 85 82 63 4 0 0 85 84 82 79 63 30 5 0 0 0 0 83 74 33 2 0 0 0 75 51 14 1 0 0 0 0 0 0 70 30 6 1 0 0 0 0 0 0 76 60 25 4 0 0 0 0 0 1 84 82 78 46 3 0 0 83 81 70 20 0 0 81 77 64 30 4 0 0 84 84 83 76 17 0 0 84 84 84 83 77 33 0 0 84 83 83 81 75 60 28 6 1 0 0 82 75 44 4 0 0 0 74 48 13 1 0 0 0 0 0 0 78 52 14 2 0 0 0 0 0 0 77 66 36 7 1 0 0 0 0 0 81 81 79 67 19 0 0 81 80 76 50 4 0 79 78 70 49 15 1 0 81 81 81 79 60 1 0 81 81 81 80 79 66 6 0 81 81 80 80 78 74 62 40 15 3 1 80 76 56 10 1 0 0 70 44 11 1 0 0 0 0 0 0 80 69 35 6 1 1 0 0 0 0 75 70 49 15 2 1 0 0 1 1 77 77 76 71 40 3 0 77 77 76 64 18 0 76 75 72 60 30 5 0 78 78 77 76 72 26 0 78 77 77 77 76 72 44 1 77 77 77 77 76 75 72 68 57 41 33 77 74 62 21 2 0 0 65 38 9 1 0 0 0 0 0 0 77 73 51 13 2 1 1 1 0 0 70 68 56 27 7 2 1 0 1 1 77 77 76 71 38 2 0 77 77 75 61 16 0 77 75 72 60 34 7 0 78 77 77 76 72 48 0 78 78 77 77 76 73 55 14 78 77 77 77 76 75 73 71 64 58 55 77 75 64 22 1 0 0 64 38 10 1 0 1 0 1 0 0 76 72 50 15 2 1 0 0 0 0 70 67 57 30 9 2 1 0 1 1 87 67 17 1 0 0 0 80 53 6 0 0 0 78 63 27 3 0 0 0 84 70 19 0 0 0 0 86 85 79 43 2 0 0 0 86 84 76 56 18 2 0 0 0 0 0 81 60 12 1 0 0 0 73 44 8 0 0 0 0 0 0 0 31 7 2 0 0 0 0 0 0 0 70 44 10 2 0 0 0 0 0 0 86 74 30 1 0 0 0 83 64 13 0 0 0 81 68 36 5 0 0 0 85 78 41 2 0 0 0 86 85 83 63 8 0 0 0 85 84 81 68 35 5 0 0 0 0 0 82 63 16 0 0 0 0 74 48 10 0 0 0 0 0 0 0 41 10 2 0 0 0 0 0 0 0 73 49 13 2 0 0 0 0 0 0 86 79 49 4 0 0 0 83 74 29 1 0 0 81 72 45 9 1 0 0 85 82 65 10 0 0 0 86 85 84 76 31 0 0 0 85 85 83 75 54 17 1 0 0 0 0 82 68 22 1 0 0 0 75 48 11 1 0 0 0 0 0 0 54 16 3 1 0 0 0 0 0 0 74 53 16 2 0 0 0 0 0 0 86 82 67 13 0 0 0 85 80 50 4 0 0 83 77 57 19 1 0 0 86 84 78 39 1 0 0 86 85 85 82 59 4 0 0 86 85 84 80 70 40 7 1 0 0 0 82 73 32 2 0 0 0 75 50 14 1 0 0 0 0 0 0 67 27 5 1 0 0 0 0 0 0 76 62 23 4 1 0 0 0 0 0 84 83 75 34 1 0 0 84 81 67 14 0 0 82 78 66 33 4 0 0 84 84 81 66 8 0 0 85 85 84 82 74 25 0 0 84 84 83 82 76 61 28 5 1 0 0 82 75 41 4 0 0 0 74 49 14 1 0 0 0 0 0 0 75 43 9 1 0 0 0 0 0 0 77 66 34 6 1 0 0 0 0 0 83 82 79 56 6 0 0 83 82 75 35 1 0 81 78 70 44 9 0 0 83 83 82 76 36 0 0 83 83 83 82 79 52 1 0 83 83 82 82 79 72 53 22 4 1 0 81 76 52 8 0 0 0 71 45 13 1 0 0 0 0 0 0 79 60 20 2 0 0 0 0 0 0 76 69 43 10 1 0 0 0 0 0 81 80 78 66 15 0 0 80 80 75 49 3 0 79 77 72 52 16 1 0 81 81 80 78 58 1 0 81 81 80 80 78 65 9 0 81 80 80 80 79 76 68 49 22 6 3 79 75 55 11 0 0 0 69 43 11 1 0 0 0 0 0 0 78 68 32 5 1 0 0 0 0 0 75 70 50 16 3 1 0 0 0 1 80 79 78 70 27 1 0 79 79 76 58 9 0 78 77 72 55 23 2 0 80 79 79 78 68 7 0 80 79 79 79 78 71 25 0 80 79 79 79 78 77 73 64 44 23 18 78 75 59 14 1 0 0 66 40 9 1 1 0 0 0 0 0 78 72 43 9 1 0 0 0 0 0 74 69 52 21 4 1 1 0 1 1 77 77 76 71 34 2 0 77 77 75 61 13 0 77 76 71 58 26 3 0 78 78 78 77 70 12 0 78 78 78 77 76 71 32 0 78 77 77 77 77 75 73 67 54 36 28 77 74 60 17 1 0 0 64 37 7 1 0 1 0 0 0 1 77 73 48 12 1 0 0 0 0 0 70 68 53 25 6 2 1 1 0 1 76 75 74 69 38 2 0 75 74 73 60 15 0 74 73 68 57 28 4 0 75 75 75 74 69 22 0 75 75 75 75 74 69 40 1 75 75 75 75 74 74 72 68 61 50 45 74 71 58 17 1 0 0 58 31 6 1 1 1 0 1 1 0 75 72 56 19 2 1 1 0 0 0 67 64 52 24 7 2 1 1 1 1 72 73 71 64 31 2 0 73 72 69 54 11 0 72 71 66 52 24 4 0 73 73 73 72 66 17 0 73 73 73 73 71 66 36 1 73 73 73 73 72 71 69 65 57 46 42 72 69 54 14 1 0 0 52 25 5 1 1 1 0 1 1 1 72 68 49 15 2 1 1 0 0 0 64 60 47 21 5 1 1 0 1 1 87 63 15 0 0 0 0 79 45 4 0 0 0 75 58 26 5 0 0 0 65 16 1 0 0 0 0 82 60 13 0 0 0 0 0 68 44 17 3 0 0 0 0 0 0 0 85 78 47 4 0 0 0 77 53 15 1 0 0 0 0 0 0 45 15 4 1 0 0 0 0 0 0 74 55 26 5 1 0 0 0 0 1 87 74 28 1 0 0 0 82 60 12 0 0 0 78 64 37 11 1 0 0 77 36 2 0 0 0 0 84 72 28 2 0 0 0 0 70 52 26 7 1 0 0 0 0 0 0 86 83 69 19 0 0 0 77 57 22 2 0 0 0 0 0 0 60 25 7 2 0 0 0 0 0 0 75 62 34 11 2 1 0 0 0 0 85 77 42 4 0 0 0 82 68 24 1 0 0 77 68 46 19 3 0 0 80 57 11 0 0 0 0 83 76 44 6 0 0 0 0 71 56 32 10 2 0 0 0 0 0 0 84 83 78 52 4 0 0 76 58 24 3 0 0 0 0 0 0 73 48 18 4 1 0 0 0 0 0 76 68 49 21 6 2 1 0 0 0 84 78 51 9 0 0 0 82 72 34 3 0 0 76 66 46 19 4 0 0 79 58 14 0 0 0 0 82 76 49 10 1 0 0 0 71 59 38 17 3 0 0 0 0 0 0 84 83 81 74 30 1 0 75 59 25 4 0 0 0 0 0 0 76 59 29 7 1 1 1 1 0 0 76 69 54 29 10 4 1 1 1 1 81 75 49 10 0 0 0 78 67 29 3 0 0 68 54 32 12 2 0 0 70 39 5 0 0 0 0 78 68 34 5 0 0 0 0 64 50 30 11 2 0 0 0 0 0 0 81 80 79 78 67 14 1 72 53 22 3 0 0 0 0 0 0 75 63 38 13 3 1 1 1 1 1 72 68 55 35 15 6 2 1 1 1 78 69 42 6 0 0 0 73 56 17 1 0 0 52 35 16 5 1 0 0 55 16 1 0 0 0 0 71 48 13 1 0 0 0 0 51 34 18 5 1 0 0 0 0 0 0 77 77 76 76 75 54 15 67 49 21 4 0 0 0 0 0 0 72 62 42 17 4 1 1 1 1 1 66 63 54 38 20 9 4 2 1 1 74 59 24 2 0 0 0 64 35 6 0 0 0 36 20 7 2 0 0 0 30 4 0 0 0 0 0 56 24 3 0 0 0 0 0 39 23 8 2 0 0 0 0 0 0 0 74 73 72 72 70 63 50 64 44 19 4 1 0 0 0 0 0 69 57 37 18 6 2 1 1 1 1 63 60 52 38 23 12 5 2 1 1 87 42 2 0 0 0 0 70 19 1 0 0 0 68 40 11 2 0 0 0 25 1 0 0 0 0 0 74 31 1 0 0 0 0 0 58 29 7 1 0 0 0 0 0 0 0 81 57 12 1 0 0 0 85 78 52 12 1 0 0 0 0 0 24 7 2 1 0 0 0 0 0 0 76 49 17 2 0 0 0 0 0 0 88 41 3 0 0 0 0 69 19 1 0 0 0 66 37 9 1 0 0 0 20 0 0 0 0 0 0 72 25 1 0 0 0 0 0 56 25 5 0 0 0 0 0 0 0 0 82 61 16 1 0 0 0 87 82 71 40 6 0 0 0 1 0 32 8 2 1 0 0 0 0 0 0 79 59 23 3 0 0 0 0 0 0 87 31 2 0 0 0 0 62 14 0 0 0 0 64 33 7 1 0 0 0 10 0 0 0 0 0 0 59 13 0 0 0 0 0 0 42 13 2 0 0 0 0 0 0 0 0 82 62 19 1 0 0 0 86 84 80 66 32 6 1 1 1 1 37 9 2 0 0 0 0 0 0 0 81 64 30 6 0 0 0 0 0 0 86 25 1 0 0 0 0 53 10 0 0 0 0 61 31 7 0 0 0 0 7 0 0 0 0 0 0 48 9 0 0 0 0 0 0 43 15 2 0 0 0 0 0 0 0 0 82 64 21 2 0 0 0 86 85 82 76 60 30 8 2 1 1 48 14 4 1 0 0 0 0 0 0 81 68 39 9 1 0 0 0 0 0 85 19 1 0 0 0 0 44 8 0 0 0 0 58 30 6 1 0 0 0 5 0 0 0 0 0 0 38 5 0 0 0 0 0 0 41 14 2 0 0 0 0 0 0 0 0 81 66 23 2 0 0 0 84 84 82 78 70 54 30 12 3 1 60 25 6 1 0 0 0 0 0 0 81 72 44 11 1 0 0 0 0 0 78 9 0 0 0 0 0 26 3 0 0 0 0 37 12 1 0 0 0 0 2 0 0 0 0 0 0 25 3 0 0 0 0 0 0 25 7 1 0 0 0 0 0 0 0 0 74 60 23 2 0 0 0 79 79 78 76 73 68 58 38 12 3 67 46 18 4 1 0 0 0 0 0 75 70 50 20 4 0 0 0 0 0 69 4 0 0 0 0 0 14 2 0 0 0 0 36 16 4 1 0 0 0 5 0 0 0 0 0 0 14 2 0 0 0 0 0 0 9 2 0 0 0 0 0 0 0 0 0 64 50 12 1 0 0 0 70 70 70 70 69 69 67 61 45 24 68 67 64 53 23 6 8 6 4 1 66 65 58 44 21 4 1 0 0 0 62 3 0 0 0 0 0 10 1 0 0 0 0 27 12 3 0 0 0 0 3 0 0 0 0 0 0 9 1 0 0 0 0 0 0 8 2 0 0 0 0 0 0 0 0 0 56 38 7 0 0 0 0 65 65 66 66 66 66 66 64 55 35 62 61 60 55 36 18 44 39 29 15 61 60 54 44 28 10 1 0 0 1 51 3 1 0 0 0 0 9 1 0 0 0 0 29 14 4 1 0 0 0 4 0 0 0 0 0 0 8 1 0 0 0 0 0 0 8 3 1 0 0 0 0 0 0 0 0 46 27 3 0 0 0 0 57 58 59 58 59 59 57 53 43 24 54 54 53 47 28 12 35 32 24 14 53 51 46 36 18 5 1 0 0 0 42 6 1 0 0 0 0 10 2 0 0 0 0 26 14 5 1 0 0 0 7 1 0 0 0 0 0 12 3 1 0 0 0 0 0 12 4 1 0 0 0 0 0 0 0 0 41 26 5 0 0 0 0 51 51 52 51 51 51 49 47 40 29 47 47 46 41 24 12 35 30 22 12 47 45 40 30 16 4 1 0 0 0 86 56 10 0 0 0 0 75 35 3 0 0 0 70 51 22 4 0 0 0 37 3 0 0 0 0 0 65 20 1 0 0 0 0 0 43 16 3 0 0 0 0 0 0 0 0 83 73 37 3 0 0 0 78 61 25 3 0 0 0 0 0 0 83 71 32 7 1 0 0 0 0 0 80 71 44 12 2 0 0 0 0 0 85 39 4 0 0 0 0 64 19 1 0 0 0 66 42 15 3 0 0 0 18 1 0 0 0 0 0 46 6 0 0 0 0 0 0 27 7 1 0 0 0 0 0 0 0 0 82 74 41 4 0 0 0 78 63 31 7 1 0 0 0 0 0 84 81 65 25 4 1 1 1 0 0 80 75 58 23 4 1 0 0 0 0 85 26 2 0 0 0 0 53 12 1 0 0 0 60 35 12 2 0 0 0 10 0 0 0 0 0 0 34 4 0 0 0 0 0 0 21 6 1 0 0 0 0 0 0 0 0 82 74 41 4 0 0 0 76 58 23 3 0 0 0 0 0 0 83 82 79 54 13 1 1 1 1 0 79 75 61 33 8 2 0 0 0 0 84 23 2 0 0 0 0 50 10 1 0 0 0 58 34 11 2 0 0 0 10 1 0 0 0 0 0 34 4 0 0 0 0 0 0 22 6 1 0 0 0 0 0 0 0 0 82 73 43 5 0 0 0 75 57 21 2 0 0 0 0 0 0 83 82 81 78 48 5 4 3 2 0 78 74 62 39 15 4 1 0 0 0 83 26 2 0 0 0 0 52 10 1 0 0 0 57 30 9 1 0 0 0 10 0 0 0 0 0 0 40 6 0 0 0 0 0 0 28 9 2 0 0 0 0 0 0 0 0 80 73 41 5 0 0 0 75 56 19 1 0 0 0 0 0 0 81 80 79 80 78 42 28 20 8 2 77 71 57 35 13 4 1 0 0 0 83 34 4 0 0 0 0 57 15 1 0 0 0 53 29 9 1 0 0 0 17 1 0 0 0 0 0 53 14 1 0 0 0 0 0 39 16 4 1 0 0 0 0 0 0 0 80 72 42 7 0 0 0 73 53 17 1 0 0 0 0 0 0 81 79 78 79 80 78 64 36 8 1 77 71 55 31 10 2 0 0 0 0 82 36 4 0 0 0 0 58 16 1 0 0 0 50 24 7 1 0 0 0 18 1 0 0 0 0 0 59 18 1 0 0 0 0 0 39 17 4 1 0 0 0 0 0 0 0 79 71 44 8 0 0 0 72 54 18 1 0 0 0 0 0 0 79 77 76 76 76 78 73 45 11 1 76 70 56 32 9 3 1 0 0 1 81 38 5 0 0 0 0 60 17 1 0 0 0 49 23 6 1 0 0 0 19 1 0 0 0 0 0 60 18 1 0 0 0 0 0 42 20 5 1 0 0 0 0 0 0 0 78 71 45 11 0 0 0 73 55 21 2 0 0 0 0 0 0 79 76 74 73 75 77 77 62 24 3 75 71 58 30 10 3 1 0 0 0 79 40 6 0 0 0 0 60 18 1 0 0 0 49 24 6 0 0 0 0 24 1 0 0 0 0 0 63 23 2 0 0 0 0 0 41 18 5 1 0 0 0 0 0 0 0 76 70 47 13 0 0 0 72 55 25 5 1 0 0 0 0 0 77 74 71 70 72 76 79 73 49 12 75 71 62 37 15 5 2 0 0 0 77 36 6 0 0 0 0 57 17 1 0 0 0 44 19 4 1 0 0 0 23 1 0 0 0 0 0 60 21 2 0 0 0 0 0 40 19 5 1 0 0 0 0 0 0 0 74 68 48 16 1 0 0 71 58 31 10 2 1 1 1 0 0 75 72 70 69 69 75 78 75 65 36 74 73 69 52 26 10 4 1 0 0 82 47 6 0 0 0 0 69 26 1 0 0 0 62 41 14 3 0 0 0 29 2 0 0 0 0 0 68 29 2 0 0 0 0 0 48 24 5 1 0 0 0 0 0 0 0 74 54 17 2 0 0 0 78 65 32 5 0 0 0 0 0 0 37 11 3 1 0 0 0 0 0 0 85 75 37 6 1 0 0 0 0 0 81 48 8 0 0 0 0 68 29 2 0 0 0 63 39 14 2 0 0 0 28 1 0 0 0 0 0 65 27 2 0 0 0 0 0 52 28 7 1 0 0 0 0 0 0 0 75 55 20 3 0 0 0 79 69 39 8 0 0 0 0 0 0 56 20 5 1 0 0 0 0 0 0 86 82 65 18 2 0 0 0 0 0 79 44 8 0 0 0 0 63 25 2 0 0 0 58 36 13 2 0 0 0 22 2 0 0 0 0 0 56 18 1 0 0 0 0 0 41 17 4 1 0 0 0 0 0 0 0 74 58 28 4 0 0 0 78 68 42 12 2 0 0 0 0 0 67 38 10 3 1 0 0 0 0 0 85 84 80 53 9 1 0 0 0 0 78 33 5 0 0 0 0 53 16 1 0 0 0 51 29 10 2 0 0 0 15 1 0 0 0 0 0 41 10 1 0 0 0 0 0 30 12 3 0 0 0 0 0 0 0 0 74 61 31 5 0 0 0 78 69 45 18 4 1 0 0 0 0 75 60 25 7 1 0 1 0 0 0 85 84 83 76 44 8 1 0 0 0 76 27 5 0 0 0 0 44 12 1 0 0 0 43 23 7 2 0 0 0 16 2 0 0 0 0 0 38 9 1 0 0 0 0 0 28 12 3 1 0 0 0 0 0 0 0 73 63 36 6 0 0 0 76 69 48 22 6 1 1 1 0 0 74 68 44 14 3 1 1 1 0 0 82 82 81 79 71 44 10 1 0 0 74 23 4 0 0 0 0 35 8 1 0 0 0 34 14 4 1 0 0 0 8 1 0 0 0 0 0 27 5 0 0 0 0 0 0 28 12 3 0 0 0 0 0 0 0 0 69 60 33 6 0 0 0 74 66 45 18 4 1 0 0 0 0 70 61 39 14 4 2 2 1 1 0 80 80 77 69 51 30 14 2 0 0 71 27 5 0 0 0 0 30 6 1 0 0 0 18 6 2 0 0 0 0 9 1 0 0 0 0 0 25 5 1 0 0 0 0 0 25 11 3 1 0 0 0 0 0 0 0 68 60 40 11 1 0 0 72 64 45 21 6 1 1 1 0 0 70 68 59 35 9 3 3 2 2 1 78 78 78 75 64 50 39 19 3 1 67 26 6 1 0 0 0 27 6 1 0 0 0 17 6 2 0 0 0 0 9 1 0 0 0 0 0 21 5 1 0 0 0 0 0 25 11 3 1 0 0 0 0 0 0 0 64 58 42 16 2 0 0 67 60 41 19 7 2 1 0 1 1 65 62 57 41 18 8 8 5 3 2 75 75 74 72 68 60 51 39 14 2 66 36 11 2 0 0 0 41 16 3 0 0 0 32 17 6 1 0 0 0 16 3 0 0 0 0 0 30 9 2 0 0 0 0 0 29 15 6 2 0 0 0 0 0 0 0 64 61 54 29 4 1 0 66 60 43 22 9 3 1 1 1 1 65 65 63 58 42 25 23 20 16 11 72 72 72 71 70 69 67 59 34 9 61 35 14 3 1 0 0 43 20 5 1 0 0 36 22 10 4 1 0 0 29 9 2 0 0 0 0 41 18 4 1 0 0 0 0 30 17 7 2 1 0 0 0 0 0 0 60 58 51 32 7 1 0 62 56 39 22 9 4 1 1 1 1 61 61 59 55 46 33 31 28 24 17 65 65 64 64 63 62 59 51 34 8
0. 0 0. 2 0. 4 0. 6 0. 8 1. 0
Adversarial accuracy
Fi gu
re 11
:R ep
lic a
of Fi
gu re
6 w
ith 50
st ep
s in
st ea
d of
20 0
at ev
al ua
tio n
tim e.
D ev
ia tio
ns in
re su
lts ar
e m
in or
.

Review summaries:
This paper proposes four novel efficient adversarial attack methods beyond Lp threat models. Together with other two existing attack methods, these six attack methods combine as a framework to evaluate robustness of defenses against unforeseen attacks. In this framework, the novel measure is normalized with the performance of adversarial training. The experiments show that the Linf adversarially trained model may not lead to improvement of robustness against other threat models. It is expected that the framework could help test model robustness.