Title: ROBUST SINGLE-STEP ADVERSARIAL TRAINING

Abstract: Deep learning models have shown impressive performance across a spectrum of computer vision applications including medical diagnosis and autonomous driving. One of the major concerns that these models face is their susceptibility to adversarial attacks. Realizing the importance of this issue, more researchers are working towards developing robust models that are less affected by adversarial attacks. Adversarial training method shows promising results in this direction. In adversarial training regime, models are trained with mini-batches augmented with adversarial samples. In order to scale adversarial training to large networks and datasets, fast and simple methods (e.g., single-step gradient ascent) are used for generating adversarial samples. It is shown that models trained using single-step adversarial training method (adversarial samples are generated using non-iterative method) are pseudo robust. Further, this pseudo robustness of models is attributed to the gradient masking effect. However, existing works fail to explain when and why gradient masking effect occurs during single-step adversarial training. In this work, (i) we show that models trained using single-step adversarial training method learns to prevent the generation of single-step adversaries, and this is due to over-fitting of the model during the initial stages of training, and (ii) to mitigate this effect, we propose a single-step adversarial training method with dropout scheduling to learn robust models. Unlike models trained using single-step adversarial training method, models trained using the proposed single-step adversarial training method are robust against both single-step and multi-step adversarial attacks, and achieve on-par results compared to the computationally expensive state-of-the-art multi-step adversarial training method, in white-box and blackbox settings.

1. INTRODUCTION:

Machine learning models are susceptible to adversarial samples: samples with imperceptible, engineered noise designed to manipulate model's output (Huang et al., 2011; Biggio et al., 2013; Szegedy et al., 2013; Biggio et al., 2014; Goodfellow et al., 2015; Papernot et al., 2016) . Further, Szegedy et al. (2013) observed that these adversarial samples are transferable across multiple models i.e., adversarial samples generated on one model might mislead other models. Due to which, models deployed in the real world are susceptible to black-box attacks (Liu et al., 2017; Papernot et al., 2017) , where limited or no knowledge of the deployed model is available to the attacker. Various schemes have been proposed to defend against adversarial attacks (e.g., (Szegedy et al., 2013; Papernot et al., 2015; Metzen et al., 2017) ), in this direction Adversarial Training (AT) procedure (Szegedy et al., 2013; Tramèr et al., 2018; Madry et al., 2018; Xie et al., 2019) shows promising results.

In adversarial training regime, models are trained with mini-batches containing adversarial samples typically generated by the model being trained. Adversarial sample generation methods range from simple methods (Goodfellow et al., 2015) to complex optimization methods (Moosavi-Dezfooli et al., 2016) . In order to scale adversarial training to large datasets, non-iterative methods such as Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015) are typically used for generating adversarial samples. Further, it has been shown that models trained using single-step adversarial training methods are pseudo robust (Tramèr et al., 2018 ):

• Although these models appears to be robust to single-step attacks in white-box setting (complete knowledge of the deployed model is available to the attacker), they are suscepti-ble to single-step attacks (non-iterative methods) in black-box attack setting (Tramèr et al., 2018) . • Further, these models are susceptible to multi-step attacks (iterative methods) in both whitebox setting (Kurakin et al., 2017) and black-box setting (Dong et al., 2018) . Tramèr et al. (2018) demonstrated that models trained using single-step adversarial training method exhibit Gradient Masking Effect. Single-step adversarial sample generation methods such as FGSM, compute adversarial perturbations based on the linear approximation of the model's loss function i.e., image is perturbed in the direction of the (sign of) gradient of loss with respect to the input image. Gradient masking effect causes this linear approximation of loss function to become unreliable for generating adversarial samples during single-step adversarial training. Further, Madry et al. (2018) demonstrated that the adversarially trained model can be made robust against white-box attacks (both single-step and multi-step attacks), if perturbations crafted while training maximizes the model's loss, and this is achieved by generating adversaries using Projected Gradient Descent (PGD) method. However, PGD method is an iterative method, due to which training time increases substantially. Therefore, this method is hard to scale for large datasets and networks. Though prior works have enabled to learn robust models, they fail to answer the following important questions: (i) Why models trained using single-step adversarial training method exhibit gradient masking effect? and (ii) At what phase of the single-step adversarial training, the model starts to exhibits gradient masking effect?

In this work, we attempt to answer these questions and propose a novel single-step adversarial training method to learn robust models. First, we show that models trained using single-step adversarial training method learn to prevent the generation of single-step adversaries, and this is due to overfitting of the model during the initial stages of training. This over-fitting of the model on single-step adversaries causes linear approximation of loss function to become unreliable for generating adversarial samples i.e., gradient masking effect. Finally, we propose a single-step adversarial training method with dropout scheduling to learn robust models. Note that, just adding dropout layer (typical setting: dropout layer with fixed dropout probability after FC+ReLU layer) does not help the model trained using single-step adversarial training method to gain robustness. Prior works observed no significant improvement in the robustness of models (with dropout layers in typical setting), trained using normal training and single-step adversarial training methods (Szegedy et al., 2013; Kurakin et al., 2017) . Results for these settings are shown in section 3.1. Unlike typical setting, we introduce dropout layer after each non-linear layer (i.e., dropout-2D after conv2D+ReLU, and dropout-1D after FC+ReLU) of the model, and further decay its dropout probability during training. Interestingly, we show that this proposed dropout setting has significant impact on the model's robustness. The major contributions of this work can be listed as follows:

• We show that models trained using single-step adversarial training method learns to prevent the generation of single-step adversaries, and this is due to over-fitting of the model during the initial stages of training. • Harnessing on the above observation, we propose a single-step adversarial training method with dropout probability scheduling. Unlike models trained using existing single-step adversarial training methods, models trained using the proposed method are robust against both single-step and multi-step attacks.

• The proposed single-step adversarial training method is much faster than multi-step adversarial training method i.e., PGD adversarial training method, and achieves on par results.

2. NOTATIONS:

Consider a neural network f trained to perform image classification task, and θ represents parameters of the neural network. Let x represents the image from the dataset and y true be its corresponding ground truth label. The neural network is trained using loss function J (e.g., cross-entropy loss), and ∇ x J represents the gradient of loss with respect to the input image x. Adversarial image x adv is generated by adding norm-bounded perturbation δ to the image x. Let, perturbation size ( ) represents the norm constraint on the generated adversarial perturbation i.e., ||δ|| ∞ ≤ for l ∞ norm constraint. Please refer to A.1 for details on adversarial attack generation methods.

3. OVER-FITTING AND ITS EFFECT DURING ADVERSARIAL TRAINING:

In this section, we show that models trained using single-step adversarial training method learns to prevent the generation of single-step adversaries, and this is due to over-fitting of the model during the initial stages of training. First, we discuss the criteria for learning robust models using adversarial training method, and then we show that this criteria is not satisfied during single-step adversarial training method. Most importantly, we show the reason for failure to satisfy this criteria is due to over-fitting. 

At each iteration, norm bounded adversarial perturbations that maximizes the training loss should be generated. Further, the model's parameters (θ) should be updated, so as to decrease the loss on such adversarial samples. Madry et al. (2018) solves the maximization step by generating adversarial samples using an iterative method named Projected Gradient Descent (PGD). In order to quantify the extent of inner maximization of Eq. ( 1), we compute loss ratio R using Eq. ( 2). Loss ratio is defined as the ratio of loss on the adversarial samples to the loss on its corresponding clean samples for a given perturbation size . The metric R captures the extent of inner maximization achieved by the generated adversarial samples i.e., factor by which loss has increased by perturbing the clean samples. By definition, adversarial perturbations are those perturbations which manipulate the model's predictions. Such manipulations can be achieved by generating perturbations that increase the loss on samples (Goodfellow et al., 2015) . Based on these facts, a perturbation is said to be an adversarial perturbation only when it causes loss to increase. This implies that the loss on the perturbed samples should be greater than the loss on the corresponding unperturbed samples i.e., loss adv > loss clean . With these facts, R can be interpreted in the following manner:

• Generated perturbation is said to be an adversarial perturbation if R >1 i.e., loss adv > loss clean • R <1 i.e., loss adv < loss clean , implies that the generated perturbation is not an adversarial perturbation. Further, the generation method is unable to generate adversarial perturbations for the given model.

We obtain the plot of R versus iteration for models trained using single-step adversarial training method (Goodfellow et al., 2015) and multi-step adversarial training method (Madry et al., 2018) .

Column-1 of Fig. 1 and Fig. 2 shows these plots obtained for LeNet+ (refer table 8) trained on MNIST dataset (LeCun) using single-step and multi-step adversarial training methods respectively. It can be observed that during single-step adversarial training, R initially increases and then starts to decay rapidly. Further R becomes less than one after 20 (×100) iterations. This implies that single-step adversarial sample generation method is unable to generate adversarial perturbations for the model, leading to adversarial training without useful adversarial samples.

We demonstrate this behavior of the model to prevent the inclusion of adversarial samples is due to over-fitting on the adversarial samples. Typically during normal training, loss on the validation set is monitored to detect over-fitting effect i.e., validation loss increases when the model starts to overfit on the training set. Unlike normal training, during adversarial training we monitor the loss on the clean and adversarial validation set. A normally trained model is used for generating adversarial validation set, so as to ensure that the generated adversarial validation samples are independent of the model being trained. Column-2 and column-3 of Fig. 1 shows the plot of loss versus iteration during training of LeNet+ on MNIST dataset using single-step adversarial training. It can be observed that, when R starts to decay, loss on the adversarial validation set starts to increase. This increase in the validation loss indicates over-fitting of the model on the single-step adversaries. Whereas, during multi-step adversarial training method, R initially increases and then saturates (column-1, Fig. 2 ). Further, no such over-fitting effect is observed for the entire training duration (column-3, 

3.1. EFFECT OF DROPOUT LAYER:

In the previous section, we showed that models trained using single-step adversarial training, learn to prevent the generation of single-step adversaries. Further, we demonstrated that this behavior of models is due to over-fitting. Dropout layer (Srivastava et al., 2014) has been shown to be effective in mitigating over-fitting during training, and typically dropout-1D layer is added after FC+ReLU layers in the networks. We refer to this setting as typical setting. Prior work (Kurakin et al., 2017) which used dropout layer during single-step adversarial training observed no significant improvement in the model's robustness. This is due to the use of dropout layer in typical setting. Whereas, we empirically show that it is necessary to introduce dropout layer after every non-linear layer of the model (proposed dropout setting i.e., dropout-2D after Conv2D+ReLU layer and dropout-1D after FC+ReLU layer) to mitigate over-fitting during single-step adversarial training, and to enable the model to gain robustness against adversarial attacks (single-step and multi-step attacks). We train LeNet+ with dropout layer in typical setting and in the proposed setting respectively, on MNIST dataset using single-step adversarial training method for different values of dropout probability. After training, we obtain the performance of these resultant models against PGD attack ( =0.3, step =0.01, steps=40). Column-1 of Fig. 3 shows the trend of accuracy of these models for PGD attack with respect to the dropout probability used while training. It can be observed that the gain in the robustness of adversarially trained model with dropout layer in the proposed setting is significantly better compared to the adversarially trained model with dropout layer in typical setting (FAT-TS). From column-2 of Fig. 3 , it can be observed that the robustness of adversarially trained model with dropout layer in the proposed setting, increases with the increase in the dropout probabil- ity (p) and reaches a peak value at p=0.4. Further increase in the dropout probability causes decrease in the accuracy on both clean and adversarial samples. Based on this observation, we propose an improved single-step adversarial training in the next subsection. Furthermore, we normally train LeNet+ with dropout layers in typical setting and in the proposed setting, on MNIST dataset. From column-1 of Fig. 3 , it can be observed that there is no significant improvement in the robustness of these normally trained models.

3.2. SADS: SINGLE-STEP ADVERSARIAL TRAINING WITH DROPOUT SCHEDULING:

We have demonstrated that existing single-step adversarial training does not make the model robust against adversarial attack, instead, the model learns to prevent the generation of single-step adversaries. Furthermore, we demonstrated this behavior of model trained using single-step adversaries is due to over-fitting on the adversarial samples during the initial stages of training. Column-1 of Fig. 3 indicates that use of dropout layer in typical setting is not sufficient to avoid over-fitting on adversarial samples, and we need severe dropout regime involving all the layers (i.e., proposed setting: dropout layer after Conv2D+ReLU and FC+ReLU layers) of the network in order to avoid over-fitting. For the proposed dropout regime, determining exact dropout probability is network dependent and is difficult. Further, having high dropout probability causes under-fitting of the model, and having low dropout probability causes the model to over-fit on the adversarial samples.

Based on these observations, we propose a single-step adversarial training method with dropout scheduling (Algorithm 1). In the proposed training method, we introduce dropout layer after each non-linear layer of the model to be trained. We initialize these dropout layers with a high dropout probability P d . Further, during training we linearly decay the dropout probability of all the dropout layers and this decay in the dropout probability is controlled by the hyper-parameter r d . The hyperparameter, r d is expressed in terms of maximum training iterations (e.g., r d =1/2 implies that dropout probability reaches zero when the current training iteration is equal to half of the maximum training iterations). In experimental section 4, we show the effectiveness of the proposed training method. Note that dropout layer is only used while training.

4. EXPERIMENTS:

In this section, we show the effectiveness of models trained using the proposed single-step adversarial training method (SADS) in white-box and black-box settings. We perform the tests described in Athalye et al. (2018) , in order to verify that models trained using SADS do not exhibit obfuscated 2018) demonstrated that models exhibiting obfuscated gradients are not robust against adversarial attacks). We show results on MNIST (LeCun), Fashion-MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al.) datasets. We use LeNet+ shown in table 8 for both MNIST and Fashion-MNIST datasets. For CIFAR-10 dataset, ResNet-34 (He et al., 2015) is used. These models are trained using SGD with momentum. Step-policy is used for learning rate scheduling. For all datasets, images are pre-processed to be in [0,1] range. For CIFAR-10, random crop and horizontal flip are performed for data-augmentation.

Evaluation: We show the performance of models against adversarial attacks in white-box and blackbox setting. For SADS, we report mean and standard deviation over three runs. Attacks: For l ∞ based attacks, we use Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015) , Iterative Fast Gradient Sign Method (IFGSM) (Kurakin et al., 2016) , Momentum Iterative Fast Gradient Sign Method (MI-FGSM) (Dong et al., 2018) and Projected Gradient Descent (PGD) (Madry et al., 2018) . For l 2 based attack, we use DeepFool (Moosavi-Dezfooli et al., 2016) and Carlini&Wagner (Carlini & Wagner, 2016) . Perturbation size: For l ∞ based attacks, we set perturbation size ( ) to the values described in Madry et al. (2018) i.e., =0.3, 0.1 and 8/255 for MNIST, Fashion-MNIST and CIFAR-10 datasets respectively. Comparisons: We compare the performance of the proposed single-step adversarial training method SADS with Normal training (NT), FGSM adversarial training (FAT) (Kurakin et al., 2017) , Ensemble adversarial training (EAT) (Tramèr et al., 2018) , and PGD adversarial training (PAT) (Madry et al., 2018) . Refer to A.2 in appendix for more details on these training methods. Note that, FAT, EAT and SADS (ours) are single-step adversarial training methods, whereas PAT is multi-step adversarial training method.

4.1. PERFORMANCE IN WHITE-BOX SETTING:

We train models on MNIST, Fashion-MNIST and CIFAR-10 datasets respectively, using NT, FAT, EAT, PAT and SADS (Algorithm 1) training methods (please refer to A.2 for details on training methods). These models are trained for 50, 50 and 100 epochs on MNIST, Fashion-MNIST and CIFAR-10 datasets respectively. For SADS, we set the hyper-parameter P d and r d to (0.8, 0.5), (0.8, 0.75) and (0.6, 0.6) for MNIST, Fashion-MNIST and CIFAR-10 datasets respectively. Table 1 , 2 and 3 shows the performance of these models against single-step and multi-step attacks in white-box setting, rows represent the training method and columns represent the attack generation method. It can be observed that models trained using the existing single-step adversarial training methods are not robust against multi-step attacks. Whereas, models trained using PAT and SADS are robust against both single-step and multi-step attacks. Unlike PAT, the proposed SADS method is a singlestep adversarial training method. Engstrom et al. (2018) demonstrated that the performance of models trained using certain adversarial training methods, degrades significantly with the increase in the number of steps of PGD attack. In order to verify that such behavior is not observed in models trained using SADS, we obtain the plot of classification accuracy on PGD test-set versus steps of PGD attack. Fig. 4 shows these plots obtained for models trained using PAT and SADS on MNIST, Fashion-MNIST and CIFAR-10 datasets respectively. It can be observed that the accuracy of models on PGD test set initially decreases slightly and then saturates. Even for PGD attack with step=1000, it can be observed that there is no significant degradation in the performance of models trained using PAT and SADS methods. In A.5 we show the effect of hyper-parameters of the proposed training method, and in A.4 we show the trend of R , train and validation loss during training of models using SADS.

4.2. PERFORMANCE IN BLACK-BOX SETTING:

In this subsection, we show the performance of models trained using different training methods against adversarial attacks in black-box setting. Typically, a substitute model (source model) is trained on the same task using normal training method, and this trained substitute model is used for generating adversarial samples. The generated adversarial samples are transferred to the deployed model (target model). We use FGSM and MI-FGSM methods for generating adversarial samples, since samples generated using these methods show good transfer rates (Dong et al., 2018) . Table 5 shows the performance of models trained using different methods, in black-box setting. It can be observed that the performance of models trained using PAT and SADS in black-box setting is better than that in white-box setting. Further, it can be observed that the performance of models trained on MNIST and CIFAR-10 datasets using FAT is worse in black-box setting when compared to whitebox setting.

4.3. PERFORMANCE AGAINST DEEPFOOL AND C&W ATTACKS:

DeepFool (Moosavi-Dezfooli et al., 2016) and C&W (Carlini & Wagner, 2016) attacks generates adversarial perturbations with minimum l 2 norm, that is required to fool the classifier. These methods measure the robustness of the model in terms of the average l 2 norm of the generated adversarial perturbations for the test set. For an undefended model, adversarial perturbation with small l 2 norm is enough to fool the classifier. Whereas for robust models, adversarial perturbation with relatively large l 2 norm is required to fool the classifier. Table 6 , shows the performance of models trained using NT, FAT, PAT and SADS methods, against DeepFool and C&W attacks. It can be observed that for models trained using PAT and SADS methods have relatively large average l 2 norm. Whereas, for models trained using NT and FAT have small average l 2 norm.

Table 6 : Performance of models trained using different training methods against DeepFool and C&W attacks. These attack methods measure the robustness of the model based on the average l 2 norm of the generated perturbations, higher the better. Success defines the percentage of samples of test set that has been misclassified. Note that, for models trained using PAT and SADS, perturbations with relatively large l 2 norm is required to fool the classifier. One-step attacks perform better than iterative attacks Black-box attacks are better than white-box attacks Unbounded attacks do not reach 100% success Increasing distortion bound does not increase success

We do not observe any of the above trends in models trained using SADS. From table 1, 2 and 3, it can be observed that iterative attacks (IFGSM and PGD) are stronger than non-iterative attack (FGSM) for models trained using SADS. Comparing results in Tables 1, 2 and 3 with results in Table 5 , it can be observed that white-box attacks are stronger than black-box attacks for models trained using SADS. Fig. 5 shows the plot of accuracy of the model on test set versus perturbation size of PGD attack, obtained for models trained using SADS. It can be observed that the model's accuracy falls to zero for large perturbation size ( ). Fig. 6 shows the plot of attack success rate (% of test set images that are misclassified by the model) versus perturbation size ( ) of PGD attack, obtained for models trained using SADS. It can be observed that attack success rate increases monotonically with increase in the attack perturbation size. 2018) showed that obfuscated gradients give a false of robustness, and broke seven out of nine defense papers (Buckman et al., 2018; Ma et al., 2018; Guo et al., 2018; Xie et al., 2018; Song et al., 2018; Samangouei et al., 2018; Madry et al., 2018; Ma et al., 2018; Dhillon et al., 2018) 2018) and Wong & Kolter (2017) propose a method to learn models that are provably robust against norm bounded adversarial attacks. However, scaling these methods to deep networks and large perturbation sizes is difficult. Whereas, in this work we show that it is possible to learn robust models using single-step adversarial training method, if over-fitting of the model on adversarial samples is prevented during training. We achieve this by introducing dropout layer after each non-linear layer of the model with a dropout schedule. 

:

CONCLUSIONIn this work, we have demonstrated that models trained using single-step adversarial training methods learn to prevent the generation of adversaries due to over-fitting of the model during the initial stages of training. In order to overcome this over-fitting, we have proposed a novel single-step adversarial training method with dropout scheduling to learn robust models. Unlike existing single-step adversarial training methods, models trained using the proposed method achieve robustness not only against single-step attacks but also against multi-step attacks. Further, the performance of models trained using the proposed method is on par with models trained using multi-step adversarial training method, and is much faster than multi-step adversarial training method.

Figures and Tables:

Figure 1: Single-step adversarial training: Trend of R , training loss, and validation loss during single-step adversarial training, obtained for LeNet+ trained on MNIST dataset. Column-1: plot of R versus training iteration. Column-2: training loss versus training iteration. Column-3: validation loss versus training iteration. Note that, when R starts to decay, loss on adversarial validation set starts to increase indicating that the model is over-fitting on the adversarial samples.

Figure 2: Multi-step adversarial training: Trend of R , training loss, and validation loss during multi-step adversarial training, obtained for LeNet+ trained on MNIST dataset. Column-1: plot of R versus training iteration. Column-2: training loss versus training iteration. Column-3: validation loss versus training iteration. Note that, for the entire training duration R does not decay, and no over-fitting effect can be observed.

Fig.2). In A.3, we show similar results for models trained on CIFAR-10 dataset. Note that, a normally trained model was used for generating FGSM ( =0.3) adversarial validation set, and we observe similar trend if a normally trained model of different architecture is used for generating FGSM adversarial validation set, please refer to A.3.

Figure 3: Column-1: Effect of dropout probability of dropout layers in typical setting and in the proposed setting on the model's robustness against PGD attack ( =0.3, step =0.01 and steps=40). Obtained for LeNet+ trained on MNSIT dataset. NT-TS: Normal training with dropout layer in typical setting. FAT-TS: Single-step adversarial training with dropout layer in typical setting. NT-PS: Normal training with dropout layer in the proposed setting. Proposed: Single-step adversarial training with dropout layer in the proposed setting. Column-2: Effect of dropout probability on the model's accuracy on clean and PGD adversarial validation set ( =0.3, step =0.01 and steps=40). Obtained for LeNet+ with dropout layer in the proposed setting, trained using single-step adversarial training method on MNIST dataset.

Single-step Adversarial training with Dropout Scheduling (SADS) Input: Training mini-batch size (m) Maximum training iterations (M ax itertion ) Hyper-parameters: P d , r d Initialization Randomly initialize network N iteration = 0 prob = P d Insert dropout layer after each non-linear layer of the network N Set dropout probability (p) of all the dropout layers with prob while iteration ≤ M ax itertion do Read minibatch B = {x 1 , .., x m } from training set Compute FGSM adversarial sample {x 1 adv , ..., x m adv } from corresponding clean samples {x 1 , ..., x m } using the current state of the network N Make new minibatch B * = {x 1 adv , ..., x m adv } /*Forward pass, compute loss, backward pass, and update parameters*/ Do one training step of Network N using minibatch B * /*Update dropout probability of Dropout-1D and Dropout-2D layers with prob*/ prob = max( 0, P d • (1 -iteration r d •M axitertion ) ) iteration = iteration + 1 end gradients. (Athalye et al. (

Figure 4: Plot of accuracy of the model trained using PAT and SADS, on PGD adversarial test set versus steps of PGD attack with fixed . For PGD attack we set ( , step ) to (0.3,0.01), (0.1,0.01) and (8/255,2/255) for MNIST, Fashion-MNIST and CIFAR-10 datasets. Note, x-axis is in logarithmic scale.

Figure 5: Plot of accuracy versus perturbation size of PGD attack, obtained for models trained using SADS. It can be observed that the accuracy of the model is zero for PGD attack with large perturbation size.

TIME COMPLEXITY In order to quantify the complexity of different training methods, we measure training time per epoch (seconds) for models trained using different training methods.

Figure 6: Plot of attack success rate versus perturbation size of PGD attack, obtained for models trained using SADS. Note that, the attack success rate increases monotonically with the increase in the attack perturbation size. 5 RELATED WORKS Following the findings of Szegedy et al. (2013), various attacks (e.g., Goodfellow et al. (2015); Moosavi-Dezfooli et al. (2016); Carlini & Wagner (2016); Mopuri et al. (2017); Dong et al. (2018) have been proposed. Further, in order to defend against adversarial attacks, various schemes such as adversarial training (e.g., Goodfellow et al. (2015); Kurakin et al. (2017); Madry et al. (2018)) and input pre-processing (e.g., Guo et al. (2018); Samangouei et al. (2018)) have been proposed.Athalye et al. (2018) showed that obfuscated gradients give a false of robustness, and broke seven out of nine defense papers(Buckman et al., 2018;Ma et al., 2018;Guo et al., 2018;Xie et al., 2018;Song et al., 2018;Samangouei et al., 2018;Madry et al., 2018;Ma et al., 2018;Dhillon et al., 2018) accepted to ICLR 2018. In this direction, adversarial training method(Madry et al., 2018), shows promising results for learning robust deep learning models.Kurakin et al. (2017) observed that models trained using single-step adversarial training methods are susceptible to multi-step attacks. Further,Tramèr et al. (2018) demonstrated that these models exhibits gradient masking effect, and proposed Ensemble Adversarial Training (EAT) method. During EAT, adversarial samples are generated by the model being trained or by one of the models from the fixed set of pre-trained models.Madry et al. (2018) demonstrated that adversarially trained model can be made robust against white-box attacks, if perturbation computed during training maximizes the loss. On the other hand, works such asRaghunathan et al. (2018) andWong & Kolter (2017) propose a method to learn models that are provably robust against norm bounded adversarial attacks. However, scaling these methods to deep networks and large perturbation sizes is difficult. Whereas, in this work we show that it is possible to learn robust models using single-step adversarial training method, if over-fitting of the model on adversarial samples is prevented during training. We achieve this by introducing dropout layer after each non-linear layer of the model with a dropout schedule.

accepted to ICLR 2018. In this direction, adversarial training method(Madry et al., 2018), shows promising results for learning robust deep learning models.Kurakin et al. (2017) observed that models trained using single-step adversarial training methods are susceptible to multi-step attacks. Further,Tramèr et al. (2018) demonstrated that these models exhibits gradient masking effect, and proposed Ensemble Adversarial Training (EAT) method. During EAT, adversarial samples are generated by the model being trained or by one of the models from the fixed set of pre-trained models.Madry et al. (2018) demonstrated that adversarially trained model can be made robust against white-box attacks, if perturbation computed during training maximizes the loss. On the other hand, works such asRaghunathan et al. (

Figure 9: MNIST: Trend of R , training loss, and validation loss during SADS training method, obtained for LeNet+ trained on MNIST dataset using SADS. Column-1: plot of R versus training iteration. Column-2: training loss versus training iteration. Column-3: validation loss versus training iteration. Note that, for the entire training duration R does not decay, and no over-fitting effect can be observed.

Figure 10: Fashion-MNIST: Trend of R , training loss, and validation loss during SADS training method, obtained for LeNet+ trained on Fashion-MNIST dataset using SADS. Column-1: plot of R versus training iteration. Column-2: training loss versus training iteration. Column-3: validation loss versus training iteration. Note that, for the entire training duration R does not decay, and no over-fitting effect can be observed.

Figure 11: CIFAR-10: Trend of R , training loss, and validation loss during SADS training method, obtained for ResNet-34 trained on CIFAR-10 dataset using SADS. Column-1: plot of R versus training iteration. Column-2: training loss versus training iteration. Column-3: validation loss versus training iteration. Note that, for the entire training duration R does not decay, and no overfitting effect can be observed.probability reaches zero and is expressed in terms of maximum training iteration. Row-2 of Fig.12shows the effect varying r d from 1/4 to 1. It can be observed that for r d < 0.5, there is degradation of model's robustness against multi-step attacks. During the initial stages of training learning rate is high and the model can easily over-fit to adversaries generated by single-step method.

White-Box setting: Classification accuracy (%) of models trained on MNIST dataset using different training methods. For all attacks =0.3 is used and for PGD attack step =0.01 is used. For both IFGSM and PGD attacks steps is set to 40.

Table content: <table><tr><td colspan="2">Training</td><td colspan="2">Attack Method</td></tr><tr><td colspan="4">Method Clean FGSM IFGSM PGD</td></tr><tr><td>NT</td><td/><td>99.24 11.65</td><td>0.31 0.01</td></tr><tr><td>FAT</td><td/><td>99.34 89.04</td><td>1.19 0.17</td></tr><tr><td/><td colspan="2">A 99.35 83.48</td><td>18.75 10.13</td></tr><tr><td>EAT</td><td colspan="2">B 99.31 80.16 C 99.20 82.48</td><td>48.13 37.85 4.00 1.29</td></tr><tr><td/><td colspan="2">D 97.66 56.85</td><td>0.87 0.29</td></tr><tr><td>PAT</td><td/><td>98.41 95.56</td><td>92.64 92.08</td></tr><tr><td>SADS</td><td/><td>98.89 94.78</td><td>89.35 88.51</td></tr><tr><td/><td/><td colspan="2">±0.01 ±0.19 ±0.09 ±0.22</td></tr></table>

White-Box attack: Classification accuracy (%) of models trained on Fashion-MNIST dataset using different training methods. For all attacks =0.1 is used and for PGD attack step =0.01 is used. For both IFGSM and PGD attacks steps is set to 40.

Table content: <table><tr><td colspan="2">Training</td><td colspan="2">Attack Method</td></tr><tr><td colspan="4">Method Clean FGSM IFGSM PGD</td></tr><tr><td>NT</td><td>91.42</td><td>6.46</td><td>1.01 0.16</td></tr><tr><td>FAT</td><td colspan="2">90.45 83.43</td><td>21.26 16.65</td></tr><tr><td/><td colspan="2">A 91.17 76.22</td><td>9.52 4.92</td></tr><tr><td>EAT</td><td colspan="2">B 90.18 73.48 C 90.18 73.48</td><td>7.59 3.56 7.59 3.56</td></tr><tr><td/><td colspan="2">D 83.81 47.41</td><td>24.23 18.26</td></tr><tr><td>PAT</td><td colspan="2">84.55 77.30</td><td>75.95 75.18</td></tr><tr><td>SADS</td><td colspan="2">85.21 75.81</td><td>71.14 69.51</td></tr><tr><td/><td colspan="3">±0.08 ±1.31 ±1.01 ±1.43</td></tr></table>

White-Box attack: Classification accuracy (%) of models trained on CIFAR-10 dataset using different training methods. For all attacks =8/255 is used and for PGD attack step =2/255 is used. For both IFGSM and PGD attacks steps is set to 7.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td colspan="3">Table 4: Comparison of training time</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">per epoch of models trained on MNIST</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">and CIFAR-10 datasets respectively,</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">obtained for different training meth-</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">ods. For PAT, steps=40 is used for</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">MNIST dataset and steps=7 is used for</td></tr><tr><td colspan="2">Training</td><td/><td colspan="2">Attack Method</td><td/><td colspan="3">CIFAR-10 dataset.  † For EAT, training</td></tr><tr><td colspan="2">Method</td><td colspan="3">Clean FGSM IFGSM</td><td>PGD</td><td colspan="3">time of pre-trained source models are</td></tr><tr><td>NT</td><td/><td>91.52</td><td>14.00</td><td>0.00</td><td>0.00</td><td colspan="2">not considered.</td></tr><tr><td>FAT</td><td colspan="2">92.42 A 90.80</td><td>98.58 82.14</td><td>0.09 10.56</td><td>0.05 4.69</td><td>Method</td><td colspan="2">Training time per epoch (sec.) MNIST CIFAR-10</td></tr><tr><td>EAT</td><td>B C</td><td>90.43 90.28</td><td>60.59 66.49</td><td colspan="2">32.76 28.90 36.49 29.41</td><td>NT FAT</td><td>∼ 2.7 ∼ 4.1</td><td>∼ 31 ∼ 53</td></tr><tr><td>PAT</td><td/><td>79.44</td><td>53.25</td><td colspan="2">50.53 50.08</td><td>EAT †</td><td>∼ 5.5</td><td>∼ 59</td></tr><tr><td>SADS</td><td/><td>75.93</td><td>48.16</td><td colspan="2">44.29 43.48</td><td>PAT</td><td>∼ 53.0</td><td>∼ 238</td></tr><tr><td/><td/><td>±0.28</td><td>±0.63</td><td colspan="2">±0.53 ±0.49</td><td>SADS</td><td>∼ 4.3</td><td>∼ 56</td></tr></table>

Black-box setting: Performance of models trained on MNIST, Fashion MNIST and CIFAR-10 datasets using different training method, against adversarial attacks in black-box setting. Source models are used for generating adversarial samples, and the target models are tested on these generated adversarial samples.

Table content: <table><tr><td/><td colspan="2">MNIST</td><td/><td/></tr><tr><td/><td>Source Model</td><td>NT</td><td>FAT</td><td>Target Model PAT</td><td>SADS</td></tr><tr><td>Model-A</td><td>FGSM ( =0.3) MI-FGSM ( =0.3, steps=40)</td><td colspan="4">29.09 79.49 96.01 95.06±0.055 10.69 72.44 95.83 94.80±0.160</td></tr><tr><td>Model-B</td><td>FGSM ( =0.3) MI-FGSM ( =0.3, steps=40)</td><td colspan="4">28.13 72.39 96.15 95.11±0.060 12.32 70.79 95.97 94.81±0.105</td></tr><tr><td/><td colspan="2">Fashion-MNIST</td><td/><td/></tr><tr><td>Model-A</td><td>FGSM ( =0.1) MI-FGSM ( =0.1, steps=40)</td><td colspan="4">36.66 88.26 81.32 80.86±0.303 33.04 88.36 81.20 80.68±0.375</td></tr><tr><td>Model-B</td><td>FGSM ( =0.1) MI-FGSM ( =0.1, steps=40)</td><td colspan="4">39.03 85.40 80.01 78.94±0.514 38.01 84.72 79.84 78.59±0.546</td></tr><tr><td/><td colspan="2">CIFAR-10</td><td/><td/></tr><tr><td>VGG-19</td><td colspan="5">FGSM ( =8/255) MI-FGSM ( =8/255, steps=7) 14.33 72.47 77.63 75.31±0.313 36.49 75.92 77.24 75.06±0.257</td></tr><tr><td>ResNet-18</td><td colspan="5">FGSM ( =8/255) MI-FGSM ( =8/255, steps=7) 4.65 30.77 74.07 77.66 75.19±0.210 68.14 77.62 75.36±0.325</td></tr></table>

Success Mean l 2 Success Mean l 2 Success Mean l 2 Success Mean l 2 Success Mean l 2

Table content: <table><tr><td/><td/><td>MNIST</td><td/><td/><td/><td colspan="2">F-MNIST</td><td/><td/><td colspan="2">CIFAR-10</td><td/></tr><tr><td>Method</td><td colspan="2">DeepFool</td><td>CW</td><td/><td colspan="2">DeepFool</td><td>CW</td><td/><td colspan="2">DeepFool</td><td>CW</td><td/></tr><tr><td colspan="3">Success Mean l 2 NT 99.35 1.837</td><td>100</td><td>1.659</td><td>93.73</td><td>0.796</td><td>100</td><td>0.709</td><td>94.11</td><td>0.162</td><td>100</td><td>0.102</td></tr><tr><td>FAT</td><td>99.37</td><td>1.455</td><td>100</td><td>0.798</td><td>93.11</td><td>1.514</td><td>100</td><td>1.167</td><td>94.66</td><td>0.168</td><td>100</td><td>0.058</td></tr><tr><td>PAT</td><td>85.68</td><td>4.633</td><td>99</td><td>2.779</td><td>90.29</td><td>2.635</td><td>100</td><td>1.572</td><td>88.72</td><td>1.156</td><td>100</td><td>0.710</td></tr><tr><td>SADS</td><td>95.89</td><td>3.692</td><td>100</td><td>2.321</td><td>90.68</td><td>2.305</td><td>100</td><td>1.308</td><td>87.85</td><td>1.089</td><td>100</td><td>0.682</td></tr><tr><td/><td colspan="2">±0.06 ±0.033</td><td colspan="4">0± ±0.027 ±0.26 ±0.102</td><td colspan="4">±0 ±0.188 ±0.22 ±0.033</td><td colspan="2">±0 ±0.006</td></tr></table>

Peer review (iclr2020):
Summary ======== This paper investigates why single-step adversarial training leads to gradient-masking, as shown in *Ensemble Adversarial Training* by Tramer et al. The authors suggest that this is due to overfitting, and propose a new type of single-step adversarial training method coupled with dropout, which appears to achieve comparable robustness to more expensive multi-step training methods. Overall, I find the findings over prior work to be somewhat limited, and the experimental evaluation is also not the most convincing. I thus lean towards rejection. Comments ========= Investigating why FGSM adversarial training leads to gradient masking is an interesting question. However, I*m not convinced that the experiments conducted by the authors give fundamentally new insights into this question. In particular, I*m not sure that *overfitting* is really the right characterization for what*s happening. The fact that the metric R_eps proposed by the authors satisfies R_eps < 1 was already hinted at in the work of Tramer et al. Specifically, Tramer et al. observe that for a model trained with single-step methods, the FGSM update step is less adversarial than a fully random step. The interesting observation here (Figure 1) is that the FGSM adversarial examples are initially good, but then perform worse as the model trains. However, I don*t think that characterizing this effect as overfitting is necessarily correct here, as the training data and validation data do not come from the same distribution anyways. The authors* experiment with the validation loss shows that as training progresses, the model becomes more vulnerable to adversarial examples transferred from another model. This high transferability was also observed by Tramer et al., and seems to have a simpler explanation: after the model *learns* to mask gradients, the FGSM examples it produces are not too different from the clean examples. So once gradient-masking kicks in, the adversarially trained model is trained similarly to the model that is the source of the transferred examples, which explains the increase in transferability. The combination of FGSM adversarial training with full-layer dropout is still interesting to consider. Yet, I do find some of the experimental results a bit hard to believe: For example, Figure 3 seems to suggest that FGSM + standard dropout (FAT-TS) achieves about 30% accuracy against PGD on MNIST. This seems clearly wrong. E.g., one of the models in the *Ensemble Adversarial Training* paper was trained in this way and did not achieve meaningful robustness against PGD. The step-size and number of steps chosen for PGD seem problematic here. With 40 steps of size 0.01, you can at most make a per-pixel change of size 0.4. Yet, because of the random start, every pixel could potentially be 0.6 away from its optimal value. Some numbers in Tables 1,2,3 are also suspicious: - In Table 1, EAT models A and B achieve non-trivial robustness to PGD and IFGSM. This suggests that the attacks are too weak. These models are not robust to iterative attacks. The same holds for models B & C in Table 3. - In Table 2, FAT and EAT model D also achieve non-trivial PGD robustness which is surprising. While the evaluation is fairly thorough, many of the evaluated attacks are redundant. E.g., FGSM can be dropped altogether, while IFGSM, MI-FGSM, and PGD are essentially all variants of each other. Adding a gradient-free / hard-label attack would be good practice. For the results in Table 6, were these obtained with models trained against l2 attacks? I don*t understand what Figure 6 is supposed to demonstrate that can*t already be inferred from Figure 5. Isn*t Figure 6 just a zoomed-in and inverted portion of the graph in Figure 5? Finally, with regards to time complexity (Section 4.5), the authors should consider recent works that show that multi-step adversarial training can be accelerated significantly by recycling gradient information. E.g., *Adversarial Training for Free!* by Shafahi et al. or * You Only Propagate Once* by Zhang et al.