Title: UNIVERSALITY, ROBUSTNESS, AND DETECTABILITY OF ADVERSARIAL PERTURBATIONS UNDER ADVER-SARIAL TRAINING

Abstract: Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space. While adversarial training improves the robustness of classifiers against such adversarial perturbations, it leaves classifiers sensitive to them on a non-negligible fraction of the inputs. We argue that there are two different kinds of adversarial perturbations: shared perturbations which fool a classifier on many inputs and singular perturbations which only fool the classifier on a small fraction of the data. We find that adversarial training increases the robustness of classifiers against shared perturbations. Moreover, it is particularly effective in removing universal perturbations, which can be seen as an extreme form of shared perturbations. Unfortunately, adversarial training does not consistently increase the robustness against singular perturbations on unseen inputs. However, we find that adversarial training decreases robustness of the remaining perturbations against image transformations such as changes to contrast and brightness or Gaussian blurring. It thus makes successful attacks on the classifier in the physical world less likely. Finally, we show that even singular perturbations can be easily detected and must thus exhibit generalizable patterns even though the perturbations are specific for certain inputs.

1. INTRODUCTION:

While deep learning is relatively robust to random noise (Fawzi et al., 2016) , it can be easily fooled by so-called adversarial perturbations (Szegedy et al., 2014) . These perturbations are generated by adversarial attacks (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017b) that generate perturbed versions of the input which are misclassified by a classifier and, at the same time, remain quasi-imperceptible for humans. There have been different approaches for explaining properties of adversarial examples and why they exist in the first place (Goodfellow et al., 2015; Tanay & Griffin, 2016; Fawzi et al., 2017a; b) .

Adversarial perturbations often transfer between different network architectures (Szegedy et al., 2014) . Moreover, adversarial perturbations have been shown to be relatively robust against various kind of image transformations and can even be successful when being placed as artifacts in the physical world (Kurakin et al., 2017a; Sharif et al., 2016; Evtimov et al., 2017) . While adversarial perturbations are data-dependent in that they were generated to fool a classifier on a specific input, there also exist universal perturbations which mislead a classifier on the majority of the inputs (Moosavi-Dezfooli et al., 2017a; b; Metzen et al., 2017b) .

Several methods have been proposed for increasing the robustness of deep networks against adversarial examples such as adversarial training (Goodfellow et al., 2015; Kurakin et al., 2017b) , virtual adversarial training (Miyato et al., 2017) , ensemble adversarial training (Tramèr et al., 2017a) , defensive distillation (Papernot et al., 2016; Papernot & McDaniel, 2017) , stability training (Zheng et al., 2016) , robust optimization (Madry et al., 2017) , and Parseval networks (Cisse et al., 2017 ). An alternative approach for defending against adversarial examples is to detect and reject them as malicious (Metzen et al., 2017a) . While some of these approaches actually improve robustness against adversarial examples, the classifier remains vulnerable against adversarial perturbations on a non-negligible fraction of the inputs.

While adversarial training is arguably the most popular approach for increasing the robustness of deep networks, few works have investigated its effect on the resulting classifiers and adversarial perturbations. Recently, Tramèr et al. (2017b) investigated the effect of adversarial training on the classifier's decision boundary and found that it displaces the boundary slightly but not sufficiently for preventing black-box attacks (attacks where the adversarial perturbation is generated based on a source model that is different from the target model). While Tramèr et al. (2017b) focus on the transferability of perturbations in a black-box scenario, we investigate in this work properties of adversarial perturbations under adversarial training in a white-box scenario, where source and target model are identical.

We first study a property of adversarial perturbations which we denote as sharedness. We define a perturbation as shared if it fools the classifier on many inputs (at least 2%). We empirically find that adversarial training is effective in removing shared perturbations but does not improve robustness against singular perturbations, which only fool the classifier on a very small subset of the data. Moreover, we study how adversarial training affects the existence of universal perturbations, the robustness of adversarial perturbations against image transformations, and the detectability of adversarial perturbations. In summary, we find that adversarial training is very effective in removing universal perturbations and that the remaining (non-universal, singular) adversarial perturbations are less robust against most transformations. Thus, adversarial training is more promising in preventing certain kind of attacks on system safety, such as those performed in the physical world, than might have been assumed. Moreover, we find that adversarial training leaves the remaining singular adversarial perturbations easily detected. Thus, these perturbations must exhibit some generalizable patterns that can be detected. Understanding and exploiting those patterns might ideally lead to identifying new ways for improving adversarial training.

2. BACKGROUND:

Let ξ denote an adversarial perturbation for an input x and let x adv = x + ξ denote the corresponding adversarial example. The objective of an adversary is to find a perturbation ξ which changes the output of the model in a desired way. For instance, an untargeted attack would search for a perturbation that makes the true or predicted class less likely while a targeted attack would try to make a designated target class more likely. At the same time, the adversary typically tries to keep ξ quasi-imperceptible by, e.g., bounding its ∞ -norm. One of the first methods for generating adversarial examples was proposed by Szegedy et al. (2014) . While this method was able to generate adversarial examples successfully for many inputs and networks, it was also expensive to compute since it involved an L-BFGS-based optimization. Goodfellow et al. (2015) proposed a non-iterative and hence fast method for computing adversarial perturbations. This fast gradient-sign method (FGSM) defines an adversarial perturbation as the direction in image space which yields the highest increase of a linearization of the cost function J under ∞ -norm. This can be achieved by performing one step in the gradient sign's direction with step-width ε: ξ = ε sgn(∇ x J(θ, x, y true )). Here, ε is a hyper-parameter governing the distance between original image and adversarial image, θ denotes the classifier's parameters, and y true the true class for input x. As an extension, Kurakin et al. (2017a) introduced the basic iterative method (BI), an iterative version of FGSM. They apply FGSM several times with a smaller step size α and clip all pixels after each iteration to ensure results stay in the ε-neighborhood of the original image:

ξ (0) = 0; ξ (n+1) = Clip ε ξ (n) + α sgn(∇ x J(θ, x + ξ (n) , y true )) .

Adversarial training denotes a modification of the training procedure which has been shown to increase robustness of classifiers against adversarial perturbations and at the same acts as a regularizer (Goodfellow et al., 2015) . To this end, during adversarial training the objective function J is replaced by J(θ, x, y true ) = σJ(θ, x, y true ) + (1 -σ)J(θ, x + ξ(x), y true ). This can be seen as an online data augmentation, which replaces a fraction of the inputs with their adversarial counterparts. When using this version of adversarial training, Kurakin et al. (2017b) observed label leaking: adversarial perturbations may leak information regarding the true class into the data since their generation is based on the true label. We avoid this by using the class y pred predicted by the current classifier for input x in place of the true class, i.e., we use J(θ, x, y pred ) as objective in adversarial training.

3. EXPERIMENTS:

We perform our experiments on the CIFAR10 dataset (Krizhevsky, 2009) and use a pre-activation Residual Network (He et al., 2016) with 18 layers and 64-128-256-512 feature maps per level as classifier (results for other datasets and classifiers are contained in Appendix H and I). As adversary we use the basic iterative (BI) adversary with ε = 4 and α = 1 (relative to inputs with values between 0 and 255) and 10 iterations. We trained the classifier for 200 epochs without adversarial training using SGD and reached a test accuracy of 94.1%; however, accuracy on adversarial examples was merely 0.5%. Thereupon, we performed adversarial training for 250 epochs with σ = 0. Adversarial training converged to a classifier with roughly 90% accuracy on unchanged test inputs and 62% accuracy on adversarial inputs (see Appendix A). Notably, the accuracy on adversarial examples for the training data gets to 97.7% after 250 epochs. Thus, the main issue preventing adversarial training from reaching higher performance on the adversarial test examples is apparently overfitting to the perturbations of the training data and not an optimization issue or a lack of network capacity. Moreover, we did not found black-box attacks on the model to be more successful than white-box attacks (see Appendix A) as Tramèr et al. (2017a) observed and attributed to gradient masking. We believe this is due to using a stronger, iterative adversary during adversarial training. In the following sections, we take a closer look at what happens during adversarial training.

3.1. STABILITY OF ADVERSARIAL PERTURBATIONS:

In this section, we study why adversarial training does not increase the robustness of the classifier against adversarial perturbations of certain inputs. Two possible failure cases exist: (a) the model might remain vulnerable against the same perturbation of an input during adversarial training. We denote such adversarial perturbations as stable perturbations. (b) The model learns to become robust against an adversarial perturbation but at the same time becomes vulnerable to other perturbations. This might lead to an oscillation-like pattern, where for any perturbation there exist a point in time in which the model is robust against this perturbation, but there does not exist a point in time where the model is robust to all possible perturbations of an input.

For deciding which of the two failure cases is prevalent, we propose to compute a temporal stability profile P of a test input under T epochs of adversarial training. This profile P is a T × T matrix containing the probability of the true class predicted by the classifier after t 2 epochs for the adversarial examples generated for the classifier after t 1 epochs of adversarial training. More specifically, after t 1 epochs of adversarial training, we generate an adversarial perturbation ξ t1 for test input x and the current model parameters θ t1 . After t 2 epochs, we evaluate the predicted probability of the true class y for the perturbed input under the current model parameters θ t2 and set the profile to P t1,t2 (x) = p(y|x + ξ t1 ; θ t2 ). For such a profile, we compute both M axM in = min t1 max t2 P t1,t2 and M inM ax = max t2 min t1 P t1,t2 . Intuitively, MaxMin being large indicates that for any perturbation of the input, there exists a model during adversarial training that predicts the correct class with high confidence. MaxMin being small thus shows that there exist a stable perturbation of the input that fools all models during adversarial training. MinMax being large, in contrast, indicates that there is a single model which predicts the correct class with high confidence for all perturbations of the input. MaxMin being large and MinMax being small thus shows that the model oscillates between being vulnerable to different perturbations of the input.

Figure 1 shows the temporal stability profiles for three different test inputs. The first profile corresponds to an input where adversarial training makes the model robust against all perturbations of the input; the second profile corresponds to a stable perturbation, which fools the model at any point in time (small MaxMin), and the third profile to an oscillation-like pattern. Thus, both stable perturbations and oscillation between perturbations occur. However, we found stable perturbations to be more frequent and being the dominant failure case for adversarial training (see Appendix B).

What causes stable perturbations? We provide evidence that a good predictor of a perturbation being stable is that it is also singular. We denote a perturbation as singular if it fools the classifier on a specific input but leaves the classifier's prediction nearly unchanged for most other inputs (98% or more). In contrast, shared perturbations are effective for many different inputs. We define the perturbation's cost change on an input x and parameters θ as J ∆ (ξ, θ, x, y) = J(θ, x + ξ, y) -J(θ, x, y). For a perturbation ξ i,t of an input x i and model parameters θ t , where t indexes the epoch of adversarial training, we can now compute two quantities: (a) how well a perturbation transfers to other inputs under the same model parameters θ t (its sharedness), and (b) how well a perturbation transfers to other model parameters θ t2 under the same input (its stability).

More specifically, we randomly sample 1000 inputs x i from the test set and generate their respective perturbations ξ i,t=0 for the BI adversary and classifier parameters θ t=0 , i.e., the classifier before adversarial training. For (a), we randomly sample N = 10000 inputs x j from the training set and compute J ∆ current,train (ξ i,t=0 ) = 1/N j J ∆ (ξ i,t=0 , θ t=0 , x j , y j ). For (b), we sample a subset T of epochs in adversarial training and compute

J ∆ future,test (ξ i,t=0 ) = 1/|T | t∈T J ∆ (ξ i,t=0 , θ t , x i , y i ). Large values of J ∆

current,train indicate a shared perturbation since the perturbation increases cost on many inputs while small values indicate a singular perturbation. Similarly, large values under J ∆ future,test indicate a stable perturbation. Figure 2 shows a scatter plot of mean cost-change when transferring a perturbation to an input from the training set (under the same model parameters) versus the cost change when transferring a perturbation to future classifier parameters under adversarial training (for the same input). We note that the two are anti-correlated: the Pearson correlation coefficient in the log-log domain is -0.293 (p < 10 -18 ). Thus, increased sharedness of a perturbation results in decreased stability: adversarial training makes a classifier robust against shared perturbations which fool the classifier on many inputs but is less effective against singular perturbations. Appendix C shows that the remaining perturbations after adversarial training are in fact singular and rarely transfer between inputs. Fast Feature Fool, a method which is, in contrast to the former works, a data-independent approach for generating universal perturbations. Khrulkov & Oseledets (2017) show a connection between universal perturbations and singular vectors.

To the best of our knowledge, the effectiveness of universal perturbations to fool classifiers trained with adversarial training has not been investigated. Since we have observed in the previous section that adversarial training reduces the sharedness of adversarial perturbations, it appears likely that it is also effective against universal perturbations. Figure 3 In this section, we evaluate the effect of adversarial training on the robustness of adversarial perturbations against different transformations. Following Kurakin et al. (2017a) , we quantify the robustness of an adversarial example against a transformation via the destruction rate. For this let use denote adversarial perturbations which change the classifier's prediction from correct to incorrect as effective. The destruction rate is defined as the fraction of effective adversarial perturbations, which are no longer effective when a transformation is applied. Figure 5 In summary, adversarial training not only reduces the number of inputs where the classifier can be fooled but also makes the remaining adversarial examples less robust against typical transformations that would occur in physical world scenarios like brightness changes, contrast changes, or Gaussian blurring. However, there are also transformations like adding strong Gaussian noise where adversarial training appears to be counterproductive. Thus, while it seems likely that adversarial training helps in preventing physical world attacks since those would typically involve changes to brightness and contrast or blurring, further experiments are required for a final conclusion.

3.4. DETECTABILITY OF ADVERSARIAL PERTURBATIONS:

A different approach for defending against adversarial approaches is trying to identify and reject inputs which have been manipulated by an adversary. Many approaches have been proposed recently for detection of adversarial perturbations; we refer to Carlini & Wagner (2017a) for an overview of approaches and a cautionary note that detectors can themselves be fooled by an adversary which is aware of their existence. In this section, however, we are not primarily interested in whether detection of adversarial perturbation is an effective defense but rather whether adversarial perturbations are sufficiently regular for being detectable in principle. While this has been answered affirmatively in recent works for scenarios where the adversary does not actively try to fool the detector, it has not been investigated whether adversarial trained models exhibit the same property.

We focus on the approach for detecting adversarial perturbations proposed by Metzen et al. (2017a) : in this approach a separate convolutional neural network denoted as detector is trained on the binary classification task of distinguishing unchanged from adversarial inputs. We follow the network architecture proposed by Metzen et al. (2017a) and apply it directly to the inputs (and not to intermediate feature maps of the classifier). Figure 6 shows the detectability of adversarial perturbations during adversarial training. It can be seen that the detectability of adversarial perturbation, i.e., the accuracy of the detector, is well above 90% for any time during adversarial training. This indicates that even though adversarial perturbations become increasingly more singular during adversarial training, they must still exhibit generalizable patterns that make them detectable. To get an understanding of the patterns which a detector exploits for identifying adversarial example, we provide an illustration of those patterns in Figure 7 . More specifically, we adapted the method for generating universal perturbations from Appendix D such that it does not maximize the classifier's cost but maximizes the detector's predicted probability of the inputs being adversarial. For a model without adversarial training (epoch 0), the generated perturbation shows the same checker-board like pattern as the universal perturbation in Figure 4 . For epoch 1 or later, the generated perturbation look already similar to the universal perturbations of late epochs. These perturbations, however, also have a mean probability of being detected as adversarial of 12% or less when added to inputs. Since the detector identifies adversarial perturbations with high accuracy in general, we conclude that the detector does not pick up a global, linear pattern but rather input-dependent patterns (corresponding to the singular perturbations), which cancel out in Figure 7 .

3.5. GENERALITY OF RESULTS:

The results presented thus far have been for a specific classifier (a residual network) on a specific dataset (CIFAR10) against a specific adversary (basic iterative gradient sign method). How general are the results, that is: do they generalize to different classifiers, datasets, and adversaries? While we leave the details to the appendix, we would like to briefly summarize the main findings here: Appendix F shows that adversarial training also increases the destruction rate of perturbations generated by other adversaries (FGSM and DeepFool) under all image transformations except for adding Gaussian noise. Appendix G presents results that perturbations generated by FGSM and DeepFool are also well detectable at any time during adversarial training. Appendix H shows results for the same classifier on a different dataset (German Traffic Sign Recognition Benchmark (Stallkamp et al., 2012) ) and Appendix I shows results for a non-residual convolutional classifier on this dataset. While specific details differ, the main findings remain valid for different datasets and for different classifiers: adversarial training makes classifiers considerably more robust against universal perturbations, increases the destruction rate of perturbations considerably under most image transformations, and leaves perturbations well detectable.

4. DISCUSSION AND CONCLUSION:

We have empirically investigated the effect of adversarial training on adversarial perturbations. We have found that adversarial training increases robustness against shared perturbations and even more against universal perturbations. A non-negligible part of the inputs, however, exhibit singular perturbations, which are effective only for the specific input. While adversarial training is not successful in removing singular perturbations, we found that it makes the remaining singular perturbations less robust against image transformations such as changes to brightness, contrast, or Gaussian blurring. Because of the classifier's reduced vulnerability against universal perturbations and the reduced robustness of the remaining singular perturbations, adversarial training appears very promising for preventing attacks on system safety such as physical-world attacks presented for face recognition (Sharif et al., 2016) or road sign classification (Evtimov et al., 2017) . We strongly recommend that future work in those directions investigates the feasibility of attacks not only against undefended classifiers but also against classifiers that have undergone adversarial training or other defense mechanisms since our results indicate that adversarial training might affect the feasibility of such attacks severely.

Interestingly, while singular perturbations are very specific for certain inputs, we found them nevertheless to be sufficiently regular for being detectable. While detection of adversarial perturbations is not yet an effective defense mechanism (Carlini & Wagner, 2017a) , our results on detectability indicate that there seem to be stable patterns in adversarial examples but that adversarial training fails to make the classifier itself robust against those patterns. Understanding and using what makes up stable patterns in adversarial examples may be a promising direction for improving adversarial training and increasing robustness against safety-relevant attacks in the physical world. We leave a closer investigation of this to future work.

A LEARNING CURVES:

Learning curves of adversarial training are shown in Figure 8 : after an initial drop, the accuracy on unchanged test inputs reaches approximately 90%. The accuracy on adversarial inputs reaches approximately 68% for FGSM and approximately 62% for BI. Notably, the accuracy on adversarial examples for the training data gets close to 100%. The learning curves indicate that after approximately 25 epochs of adversarial training the classifier starts to overfit to particular perturbations of the training set. In the light of our results from Section 3.1, we hypothesize that the classifier learns to become robust against shared perturbations in the first epochs and (starting after approximately 25 epochs) starts to memorize singular perturbations on the training data. While increased robustness against shared perturbations generalizes to test data, memorizing singular perturbations obviously does not generalize. Figure 9 evaluates the error rate on training and test data for the BI adversary in white-box and different black-box attack scenarios. While the white-box attack is analogous to the results shown in Figure 8 (small differences are due to subsampling the test data to 256 inputs), the black-box attacks are performed as follows: adversarial examples for the classifiers obtained after 0, 5, 15, 51, and 251 epochs of adversarial training (the source models) are generated and tested for all classifiers obtained during adversarial training (the target models). Since typically source and target model are different, the attack is a black-box attack which relies on transferability between models. Tramèr et al. (2017a) observed that black-box attacks can be more effective than white-box attacks for adversarially trained models. They attributed this to gradient masking, i.e., adversarial training not making classifiers more robust but just adversarial examples harder to find with gradient-based approaches. However, we do not observe this effect on either training or test data. We believe this is due to using a stronger, iterative adversary during adversarial training.

We further note that on test data, adversarial examples generated for a classifier after 5 or 15 epochs of adversarial training remain effective against all later classifiers, with an error rate of 20% or more. This further indicates that there exist stable perturbations against which the classifier never becomes robust.

B MAXMIN VERSUS MINMAX SCATTER PLOT:

As discussed in Section 3.1, we can generate a temporal stability profile of adversarial training for each input. We can summarize this profile in two quantities: MaxMin and MinMax. Figure 10 uniform at random and generate their respective adversarial perturbations for the classifiers after 0, 1, 15, and 251 epochs of adversarial training. We then transfer these perturbations to 5000 other test inputs and compute the percentage of these inputs for which the perturbations are effective, i.e., change the prediction of the respective classifier (from the same epoch) from correct to incorrect.

The results are summarized in Figure 11 . Perturbations for the classifier without adversarial training (epoch 0), are shared: on average, they are also effective for 6.3% of the other inputs and at least for 3.3%. Already after 1 epoch of adversarial training, the sharedness is drastically reduced: on average, the perturbations are effective for 1.1% of the other inputs. This is further reduced to 0.8% after 15 epochs and 0.5% after 251 epochs. Notably, the most shared perturbation after 251 epochs is effective for less than 1% of the other inputs. In summary, adversarial training reduces sharedness considerably and most remaining perturbations can be considered as singular perturbations in the sense that they work only for a specific input or a small fraction of very similar inputs (less than 2%).

D GENERATING UNIVERSAL PERTURBATIONS:

We roughly follow the approach proposed by Metzen et al. (2017b) for generating universal perturbation. While they extend the basic iterative procedure to generating targeted universal perturbations for semantic image segmentation, we adapt the procedure to generating untargeted universal perturbations for image classification. Let Ξ denote the universal perturbation. We generate the universal perturbations as follows:

Ξ (0) = 0, Ξ (n+1) = Clip ε Ξ (n) + α K K k=1 sgn(∇ D (k) (Ξ (n) )) , with ∇ D (Ξ) = 1 m m j=1 ∇ x J(θ, x j + Ξ, y true j )

being the loss gradient averaged over a mini-batch of inputs sampled uniform randomly from the training set. In the experiments, we use = 10, α = 2.5, a mini-batch size of m = 128, K = 10 batches per update, and 200 iterations.

E UNIVERSAL PERTURBATIONS ON TEST DATA:

Figure 12 shows the accuracy of the classifier during adversarial training against universal perturbations generated on train and test data. Surprisingly, universal perturbations generated on the test data are no more effective than those generated on training data. This indicates that the classifier is actually more robust against universal perturbations and it is not just harder to generate universal perturbations on the training set that generalize to unseen data.

Figure 13 shows illustrations of universal perturbations generated on the test data. For the first epochs, the universal perturbations look visually similar to the perturbations generated on the training data. For later epochs, the test set perturbations are more noisy than the training set perturbations. 

G DETECTABILITY OF DIFFERENT ADVERSARIES:

Figure 16 shows the detectability (compare Section 3.4) for the FGSM and the DeepFool adversaries.

As has been noted by Metzen et al. (2017a) , in general FGSM is easier to detect and DeepFool is harder to detect than the basic iterative adversary. Besides that, the results reinforce the main finding 

H GERMAN TRAFFIC SIGN RECOGNITION BENCHMARK:

We repeat our experiments from Section 3 on the German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp et al., 2012) . Since we found that in general, robustness against adversarial perturbations is larger on GTSRB than on CIFAR10, we use ε = 8 during adversarial training and for generating image-dependent perturbations. We split the GTSRB training set in 35000 samples used for training the network and the remaining 4209 samples for evaluation. We do not use a pretrained classifier (in contrast to our experiments in Section 3) but use the first 25 epochs for vanilla, nonadversarial training. Thereupon, we perform 175 epochs of adversarial training. Otherwise, we use the same hyperparameters as in Section 3.

Figure 17 shows the learning curve during adversarial training. Despite the larger value of ε, accuracy under adversarial perturbations generated using BI reaches 95.2% on test data and 100% on training data. This shows that it is considerably easier to reach high levels of robustness on GT-SRB than on CIFAR10. Moreover, nearly perfect robustness can be achieved on training data (as on CIFAR10).

Figure 18 shows the accuracy of the classifier for universal perturbations of ε = 20 (please note that ε has been doubled compared to the experiments on CIFAR10). Adversarial training is again effective in increasing the robustness of a classifier against universal perturbations: accuracy on data containing universal perturbations is increased from approx. 40% to more than 98%. Figure 19 illustrates the corresponding universal perturbations: universal perturbations for classifiers without Figure 20 shows the destruction rate of adversarial perturbations under different image transformations on GTSRB. In summary, adversarial training increase the destruction rate under changes of brightness, reduction of contrast, and adding Gaussian noise. For Gaussian blur, adversarial training increases the destruction rate for small amount of blurring but reduces it slightly for strong blurring.

Figure 21 shows the detectability of adversarial perturbations on GTSRB. Since ε had to be increased from 4 to 8 to achieve a sufficient fooling rate on GTSRB, the detectability is also increased considerably. For a detector trained on the respective episode, it remains close to 100% and also de Figure 22 shows the learning curve during adversarial training. Despite the larger value of ε compared to CIFAR10, accuracy under adversarial perturbations generated using BI reaches 90% on test data and 97% on training data. Interestingly, the classifiers before adversarial training is turned on are more robust against adversarial perturbations than for the residual network. However, ultimately, the residual network becomes more robust than the non-residual after sufficient amount of adversarial training (see Figure 17 ).

Figure 23 shows the accuracy of the convolutional classifier for universal perturbations of ε = 20 (please note that ε has been doubled compared to the experiments on CIFAR10). Adversarial training is again effective in increasing the robustness of a classifier against universal perturbations: accuracy on data containing universal perturbations is increased from approx. 60% to more than 95%. Figure 24 illustrates the corresponding universal perturbations: in contrast to the results of a residual network, there is no qualitative change in the universal perturbation (from the very be- models from different epochs. This shows that there little qualitative change in adversarial perturbation during training on GTSRB, the change is mostly that many perturbations no longer fool the model.

:



Figures and Tables:

Figure 1: Profiles of three test inputs under adversarial training.White corresponds to a highconfidence correct classification (P t1,t2 ≈ 1) and black to an incorrect classification with high confidence (P t1,t2 ≈ 0). Note that MaxMin corresponds to first taking the maximum over each row, and then the minimum over the resulting column. MinMax, in contrast, takes first the minimum over the columns and then the maximum over the resulting row.

Figure 3: Accuracy under universal perturbations for ε = 10. Shown is the accuracy of the universal perturbation generated for the model from the respective epoch as well as the accuracy of the universal perturbations generated for the model in epoch 0, 5, and 51.

Figure 4: Illustration of universal perturbations for ε = 10 generated for different epochs of adversarial training. The perturbations are amplified by a factor of 10 for readability.

Figure 5: Destruction rate of adversarial perturbations for 4 kind of transformations: additive brightness changes, multiplicative contrast changes, Gaussian blurring, and additive Gaussian noise. Results are for basic iterative adversary, results for other adversaries can be found in Appendix F.

Moreover, also the detectors trained on the adversarial examples from epoch 1 and 251 of adversarial training detect adversarial examples generated at any other time during adversarial training with more than 90% accuracy. This indicates that the patterns that make adversarial examples detectable must stay relatively robust over time. This is in contrast to the universal perturbation, which change drastically over time. We would also like to note that a detector trained on the base model that was not adversarially trained (epoch 0) does not reliably detect adversarial perturbations in later epochs. Thus there is a principled change in the adversarial examples in the first epoch of adversarial training.

Figure 6: Detectability of adversarial perturbations on test data. Shown is the performance of detectors trained on the respective epoch as well as the performance of detectors trained in fixed epochs. Results are for basic iterative adversary, results for other adversaries can be found in Appendix G. Epoch: 000 Mean prob. 0.966 Epoch: 001 Mean prob. 0.117 Epoch: 005 Mean prob. 0.122 Epoch: 051 Mean prob. 0.098 Epoch: 101 Mean prob. 0.102 Epoch: 251 Mean prob. 0.095

Figure 8: Learning curve during adversarial training.

Figure 11: Sharedness of adversarial perturbations on test data during adversarial training.

Figure 14: Destruction rate of adversarial perturbations of FGSM adversary for 4 kind of transformations: additive brightness changes, multiplicative contrast changes, Gaussian blurring, and additive Gaussian noise.

Figure 15: Destruction rate of adversarial perturbations of DeepFool adversary for 4 kind of transformations: additive brightness changes, multiplicative contrast changes, Gaussian blurring, and additive Gaussian noise.

Figure 16: Detectability of adversarial perturbations on test data. Shown is the performance of detectors trained on the respective epoch as well as the performance of detectors trained in fixed epochs. Upper plot: Detectability of FGSM. Bottom plot: Detectability of DeepFool. The evaluation of the detectability of DeepFool has been restricted to the "respective epoch" setting because of the large computational cost of computing the respective perturbations.

Figure 17: Learning curve on GTSRB for a residual net. First 25 epochs are vanilla training, thereupon we perform adversarial training with ε = 8.

Figure 19: Illustration of universal perturbations for ε = 20 generated for different epochs of adversarial training. The perturbations are amplified by a factor of 5 for readability.

Figure 21: Detectability of adversarial perturbations on GTSRB. Shown is the performance of detectors trained on the respective epoch as well as the performance of detectors trained in fixed epochs. Results are for basic iterative adversary, results for other adversaries can be found in Appendix G.

Figure26: Detectability of adversarial perturbations on GTSRB for a convolutional, non-residual net. Shown is the performance of detectors trained on the respective epoch as well as the performance of detectors trained in fixed epochs. Results are for basic iterative adversary, results for other adversaries can be found in Appendix G.

Note that this is true for the universal perturbation generated for the model from the respective epoch as well as for perturbations from other epochs.Figure4illustrates the universal perturbations generated for models in different epochs of adversarial training. From the illustrations, it is apparent that universal perturbations change considerably: while the perturbation for a model without adversarial training (epoch 0) has a clear checker-board like pattern, adversarial training removes this pattern immediately. For the first 15 epochs of adversarial training, the perturbations instead have a black-and-white maze structure at the boundaries. Later on, also this pattern is removed by adversarial training and the universal perturbation consists mostly of large, constant-color areas. This probably corresponds to a situation where perturbations on different training inputs are so different that they effectively cancel out (an indication that they are singular). We would also like to note that also universal perturbations generated on test data are no more effective in fooling the classifier (see Appendix E); it is thus not just an issue of overfitting the perturbations to the training data.

Table content: <table><tr><td>Epoch: 000</td><td>Epoch: 001</td><td>Epoch: 002</td><td>Epoch: 005</td><td>Epoch: 015</td><td>Epoch: 026</td></tr><tr><td>Epoch: 051</td><td>Epoch: 101</td><td>Epoch: 151</td><td>Epoch: 201</td><td>Epoch: 246</td><td>Epoch: 251</td></tr></table>

shows the destruction rate of adversarial examples against four different transformations based on Kurakin et al. (2017a) for different models obtained during adversarial training. Clearly, adversarial training increases the destruction rate of adversarial examples under changes of brightness, contrast, and Gaussian blurring.It is notable that a model trained for 251 epochs of adversarial training has a considerably higher destruction rate under these transformations than a model trained for 51 epochs, even though the learning curves (see Appendix A) indicate that there are no improvements on test data for 50 or more epochs. The results for the destruction rate under additive Gaussian noise show a more complex picture: here a model without adversarial training has a smaller destruction rate for weak noise (σ < 5) but a considerably higher destruction rate for strong noise (σ > 7) than adversarially trained models. The reason for the latter is unclear and an interesting direction for future work.

Table content: <table/>

tectors trained on epoch 26 and on episode 195 (model after one epoch and at the end of adversarial training) transfer with nearly perfect detectability to models from different epochs. This shows that there is little qualitative change in adversarial perturbation during training on GTSRB, the change is mostly that many perturbations no longer fool the model.

Table content: <table><tr><td>Epoch: 010</td><td>Epoch: 025</td><td>Epoch: 030</td><td>Epoch: 035</td></tr><tr><td>Epoch: 050</td><td>Epoch: 100</td><td>Epoch: 150</td><td>Epoch: 195</td></tr></table>

Peer review (iclr2018):
 Summary: This paper empirically studies adversarial perturbations dx and what the effects are of adversarial training (AT) with respect to shared (dx fools for many x) and singular (only for a single x) perturbations. Experiments use a (previously published) iterative fast-gradient-sign-method and use a Resnet on CIFAR. The authors conclude that in this experimental setting: - AT seems to defend models against shared dx*s. - This is visible on universal perturbations, which become less effective as more AT is applied. - AT decreases the effectiveness of adversarial perturbations, e.g. AT decreases the number of adversarial perturbations that fool both an input x and x with e.g. a contrast change. - Singular perturbations are easily detected by a detector model, as such perturbations don*t change much when applying AT. Pro: - Paper addresses an important problem: qualitative / quantitative understanding of the behavior of adversarial perturbations is still lacking. - The visualizations of universal perturbations as they change during AT are nice. - The basic observation wrt the behavior of AT is clearly communicated. Con: - The experiments performed are interesting directions, although unfocused and rather limited in scope. For instance, does the same phenomenon happen for different datasets? Different models? - What happens when we use adversarial attacks different from FGSM? Do we get similar results? - The papers lacks a more in-depth theoretical analysis. Is there a principled reason AT+FGSM defends against universal perturbations? Overall: - As is, it seems to me the paper lacks a significant central message (due to limited and unfocused experiments) or significant new theoretical insight into the effect of AT. A number of questions addressed are interesting starting points towards a deeper understanding of *how* the observations can be explained and more rigorous empirical investigations. Detailed: -