Title: PIXEL REDRAWN FOR A ROBUST ADVERSARIAL DEFENSE

Abstract: Recently, an adversarial example becomes a serious problem to be aware of because it can fool trained neural networks easily. To prevent the issue, many researchers have proposed several defense techniques such as adversarial training, input transformation, stochastic activation pruning, etc. In this paper, we propose a novel defense technique, Pixel Redrawn (PR) method, which redraws every pixel of training images to convert them into distorted images. The motivation for our PR method is from the observation that the adversarial attacks have redrawn some pixels of the original image with the known parameters of the trained neural network. Mimicking these attacks, our PR method redraws the image without any knowledge of the trained neural network. This method can be similar to the adversarial training method but our PR method can be used to prevent future attacks. Experimental results on several benchmark datasets indicate our PR method not only relieves the over-fitting issue when we train neural networks with a large number of epochs, but it also boosts the robustness of the neural network.

1. INTRODUCTION:

Deep neural networks have exhibited very accurate results, but, are also vulnerable to adversarial examples (Szegedy et al., 2013) . Adversarial example is a generated output that induces a trained machine to misclassify a test datum with a high probability. In other words, the adversarial example can fool the trained machine to fail its task (Dhillon et al., 2018b; Goodfellow et al., 2014; Szegedy et al., 2013) . We call the method that generates adversarial examples as an attack technique. For instance, in image classification, an attack technique is creating an adversarial image by perturbing some pixels of the original image in which the adversarial image is remained to be perceptible by the human. Obviously, the lesser the pixels to be perturbed, the better the attack technique it is. Defense techniques to prevent those attacks usually need to produce a robust neural network model which can detect adversarial examples or label the adversarial examples correctly. There are many researchers who have studied both attacking and defense techniques, and it is generally believed that the defense techniques are more challenging than the attack techniques.

In this paper, we focus on the defense technique that generates a robust neural network. For this purpose, we have to understand the fundamental idea of generating the adversarial example. In image classification, usually, an adversarial image is generated by perturbing its pixels' value (for example, L 2 or L ∞ attack). If we regard the perturbed pixels to be formed by adding some noises, then the technique of creating the noises can be recognized as a dropout (Srivastava et al., 2014) (if the perturbed pixel value is zero) or other regularization techniques (Adeli & Wu, 1998; Krogh & Hertz, 1992; Nowlan & Hinton, 1992) . The neural network with the regularization method usually shows higher performance than the one without the regularization method (normal testing). From another viewpoint, we can regard adversarial examples as a subset from the training dataset universe. Due to the difficulty of collecting all possible examples in the world, we usually just use the clean (unperturbed) examples (Kurakin et al., 2016b; Yuan et al., 2017) as the representative of our training dataset.

In order to solve the problem, we propose Pixel Redrawn (PR) method, which is a pre-processing method that redraws a pixel value of the original image into a different pixel value. The PR method is motivated by two observations. The first observation is based on the lesson provided by Carlini and Wagner in (Carlini & Wagner, 2017) , which the randomization can increase the distortion required for attack. The paper has mentioned that the most effective defense technique so far is the dropout randomization which has increased nearly five times more difficulty in generating the adversarial examples on CIFAR dataset. We agree that the randomization method makes it difficult to compute the derivation during the back-propagation and it could replace the perturbed pixel with another value by some random chances during the feed-forward.

The second observation is that humans usually can recognize images regardless of the colors in which the image has been drawn (e.g. in various colors or in gray level). This indicates that we usually (human) may not need to have a colorful image just for image classification. We discuss this in more detail with our proposed method in Section 2.

In this study, we focus on the following:

• We analyze the influence of our PR method toward a normal deep neural network without any adversarial machine learning.

• We evaluate the effectiveness of our PR method applied in the deep neural network against the adversarial example.

• And, we compare our PR method with a random noise injection.

Our research contributions can be summarized as follows:

• Our PR method effectively generates adversarial training images that are not covered in the original dataset.

• Hence, our PR method increases the robustness in neural network training.

To promote reproducible research, we release the implementation of our defense.1 2 PIXEL REDRAWN METHOD

2.1. PRELIMINARIES:

Let X be the image for the input of neural networks. If the size of the image is m × n, then {x i |x i ∈ X}, where x i = x 1 , x 2 , . . . , x m×n is the pixel of the image. A classifier F (•) is a function to produce a predicted label Y for X. Let Y be the true label of image X.

An attack technique A(•) is a function to generate an adversarial image X adv from X with (or without) the knowledge of F (•).

2.2. OUR APPROACH:

As aforementioned, PR is a pre-processing method P R(•) which reproduces a new image by redrawing the pixels of the original image. We called the generated image as PR image X pr . We list the three main steps of PR method as follows:

1. Discretization Let C be the range of color C ∈ [0, 255]. Note that we use a normalized pixel value C ∈ [0, 1] in the experiment section (Section 4). If we discretize the color into k sets, then {c i |c i ∈ C}, where c i = c 1 , c 2 , . . . , c k is the sub-range of color in C. For each c i , the range starts from a minimum value s pr imin and ends with a maximum value s pr imax (e.g.

c 1 = [s pr 1min , s pr 1max ] = [0, 51] if k = 5 and C is equally divided).

2.. Prediction:

We train a PR model (describe more in Section 2.3) and then we use it to predict each pixel of an image.

ŷpr i = wx i + b (1)

where ŷpr i is a prediction label of a pixel from PR model, w is a weight of PR model, x i is a pixel of an image, and b is a bias of PR model (shown in 'PR process' of Fig. 1 ). 

3.. Mapping:

We redraw the pixel with the sub-range of color c i that has predicted from PR model.

EQUATION

x pr i = random ci (s pr imin , s pr imax )

where colorset(•) is the color set that has defined by the user, c i is the sub-range of color with a minimum value s pr imin and a maximum value s pr imax , and random ci (•) is a random function that generates a random value between s pr imin and s pr imax .

In Fig. 1 , we perform pre-processing (PR process) for the original image. We reassign each pixel of the image to a random value, such that the range of the random value is based on the chosen sub-range of color which is predicted from PR model. The newly generated PR image is then fed into a neural network.

We depict some examples from a real dataset (i.e. MNIST (LeCun et al., 1998) dataset) with PR method in Fig. 2 . The intuition of redrawing each pixel with a certain value in an accepted interval is that, for a certain training image, the training dataset does not include all possible combination of the training image with human-acceptably varying pixel values. With the PR method, we try to include those missing data in the training phase. Another advantage of using the PR method is that we can generate more accurate neural network models (discussed in more detail in Section 2.5).

In the following sub-sections, we explain PR model for effectively generating new pixel redrawn images by our PR method.

2.3. GENERATING PIXEL REDRAWN MODEL:

Given a pixel value from the original training image x i , PR model predicts the output pixel value ŷpr i , perturbed within an accepted range c i , for the corresponding pixel location of the new image x pr i . Since we redraw each pixel of the image (either grayscale or color) independently which the pixel is the only input for the PR model, and we need the PR model to produce several outputs, then single-layer Perceptron is qualified in this case. Firstly, we train the PR model by generating a batch of random examples with the corresponding labels initialized with the c i . After the PR model is converged, we predict each pixel of the original image with the newly trained PR model. Lastly, we assign a predicted output value of the winning node at the last layer in the trained PR model to the corresponding pixel location of the new image. Note that the output value is within the accepted range which is initialized by the user. The final output should look similar as the right-hand side image of a pair image in Fig. 2 ).

2.4. PSEUDO-CODE OF THE PIXEL REDRAWN METHOD:

In order to have a clear description of our proposed method, we provide the pseudo-code of the PR method in Algorithm 1. First, we initialize the settings of the PR models, which include the range of each color value c i , the number of the input (always 1) and output (k), the number of the epochs/iterations, and some random weights. Then, we train the PR model by generating some random inputs with a corresponding label which the label is based on the range of each color value. After we train the PR model, we generate a PR image (X pr ) by using the trained PR model to redraw all pixels of the original image (X). Next, we use the newly generated PR image (X pr ) as the input of the neural network during the training phase and testing phase. In the end, the trained network show the output label of the test data.

Algorithm 1 The pseudo-code of the Pixel Redrawn Method 

2.5. EXTENSION OF THE PIXEL REDRAWN MODEL:

One direct extension is using multiple PR models. We train d PR models and predict each pixel of the original image with one of the trained PR models. In other words, we randomly select one PR model out of d PR models to predict a pixel of the original image. Then, we repeat the same steps until all pixels are redrawn.

Another extension is partially converged PR models (less accurate) to redraw a pixel to a totally different color value. The partially converged PR model is generated when we prematurely stop the training phase on purpose after a few epochs so that the model is not fully trained. For example, if a black and white (k = 2) PR model has 70% accuracy to predict a pixel either black or white color, then a black color pixel can have 30% chance to be converted into a white color. Note that this may or may not influence the content of the image depending on the number of the pixels that are different from the original input. In the extreme case, PR method may able to change an eye color from black to blue for example, by using the partially converged PR models. The motivation of using the partially converged PR models is because the adversarial example changes a pixel value into a different value which the changed value is far from the original value (L ∞ attack).

In summary, using several PR models (including some partially converged PR models) increase the robustness of the neural network.

3. EXPERIMENTAL SETTINGS:



3.1. DATASET:

For experimental analysis of our propose methods, we use three public benchmark datasets in this study. The datasets include MNIST, Fashion MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al., 2010) . For the CIFAR-10 dataset, we generate another grayscale CIFAR-10 for the purpose of analyzing the influence of our proposed method on the color image. MNIST and Fashion MNIST have 60,000 training images and 10,000 test images associated with a label from ten classes. The size of each image is 28×28 grayscale. CIFAR-10, however, has 50,000 training images and 10,000 test images with ten classes. Each image is 32×32 color image.

We use a basic convolutional neural network (CNN) (Krizhevsky et al., 2012) as the neural network architecture for all datasets. The neural network architecture is shown in Fig. 1 . We use three convolutional layers and one fully connected layer. The convolutional layers are followed by the Rectifier Linear Unit (ReLU) activation function. The fully connected layer is followed by the Softmax activation function.

3.2. ATTACK TECHNIQUE:

There are two kinds of attacks, which are white-box attack and black-box attack. In the whitebox attack, attackers know all parameters of the attacked model, whereas in the black-box attack, attackers have no knowledge about the parameters of the attacked model. In this experiment, we run the white-box attack because it is the hardest to defend. The attack can be targeted attack or untargeted attack. In the targeted attack, the attacker tries to deceive the trained network to misclassify the datum as the targeted label. The untargeted attack, on the other hand, is fooling the trained model to misclassify the datum as any label except the true label. We choose the untargeted attack in the experiment because it is easier for the attacker than the targeted attack.

We use several state-of-the-art attack techniques for the evaluation of our proposed method. The attack techniques include Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) , basic iterative method (BIM) (Kurakin et al., 2016a) , momentum iterative method (MIM) (Dong et al., 2018) and (Carlini & Wagner, 2016) attack. During the experiments, when we apply with the FGSM, BIM, and MIM attacks, we set = 0.3 for the MNIST dataset and = 8 256 for the Fashion MNIST and CIFAR-10 datasets . For CW attack, we set 1,000 in the number of iterations to run the attack.

L 2 -Carlini & Wagner's (L 2 -CW)

FGSM It is a fast and simple attack technique to generate the adversarial example which has proposed by Goodfellow et al. (2014) . We set this technique as the baseline of attack techniques. The equation of FGSM is computed by

EQUATION

where is the maximum perturbation allowed for each pixel and L(X, y) is a loss function.

BIM It is an extension of FGSM by applying multiple iterations with small step size in order to obtain the least perturbations of the image. This technique is proposed by Kurakin et al. (2016a) .

The computation of the technique is shown in Eq. 5

X adv 0 = X, X adv N +1 = Clip X, {X adv N + α sign(∇ X L(X adv N , y))} (5)

where Clip X, is a function to clip the output image to be within the -ball of X.

MIM It is more advanced than BIM with momentum algorithm. This technique is proposed by Dong et al. (2018) . The computation of the technique is shown in Eq. 6

X adv t+1 = X adv t + α • g t+1 ||g t+1 || 2 (6)

where g t+1 is shown in Eq. 7.

g t+1 = µ • g t + J(x adv t , y) ||∇ x J(x adv t , y)|| 1 (7)

CW It is an efficient attack technique in finding the adversarial example with the smallest perturbations. The equation is shown as follow. 

EQUATION

Y = F (P R(A(F, X))) • Case B:

No PR method is used during the testing phase. The attackers have no knowledge of PR method but they knows the parameters of the trained neural networks. The attackers generate an adversarial image from an image with the parameters of the trained neural networks. The trained neural networks use the adversarial image as the input without being pre-processed by the PR method. Y = F (A(F, X)) • Case C: The attackers know both the PR method and the trained neural networks. The attackers produce an adversarial image from an image with the parameters of the PR method and trained neural network. The trained neural networks use the adversarial image as the input without being pre-processed by the PR method. Y = F (A(F, P R, X)) • Case D: The attackers know both the PR method and the trained neural networks. The attackers produce an adversarial image from an image with the parameters of the PR method and trained neural network. The defense mechanism receives the adversarial image as the input for the PR model and the newly created PR image is used as the input of the trained neural networks.

Y = F (P R(A(F, P R, X)))

We illustrate some adversarial images generated from the MNIST dataset for each case in Fig. 3 . Note that the adversarial example generated from the case C and D can cause high L 0 , L 2 and L ∞ distances because the attacker has used a distorted image instead of a clean image. In other words, the perturbed pixels from the adversarial image can be perceptible by the human.

4. EXPERIMENTAL RESULTS:

In this study, we perform several experiments based on the following questions:

1. What is the influence of applying the PR method in different phases with a legitimate test datum? 2. What is the influence of using the PR method in different case studies? 3. What is the difference between multiple PR models and a single PR model applying in the neural networks? 4. What is the difference between the PR method and the random noise injection? The objective of performing these experiments is to demonstrate that PR augmented training method has more advantages than common training methods. From Table 1 , we produce four different results by applying the PR method in different phases. These include (1) applying no PR method, denoted as non-PR (i.e. common training method), (2) applying PR method during the training phase, (3) testing phase (4) and both phases. We analyze that applying PR method (either during the training phase or both phases) is not significantly different with non-PR case in terms of the accuracy. This means that neural networks trained with PR method show comparable performance to those trained without PR method. From the results of "Testing Phase" in Table 1 , it can be seen that neural networks trained without PR (i.e. non-PR) are not sufficiently robust when they are exposed to images perturbed by PR method. In other words, we can use PR method as an attack technique to fool the trained neural networks. The accuracy for the MNIST and Fashion MNIST datasets are greatly decreased in Table 1 could be caused by over-fitting when a large number of epochs have passed. However, our method can relieve the over-fitting issue as shown in Table 1 when we apply PR method in the training phase only. The neural networks can classify clean images correctly after we train them with PR method. We conjecture PR method has performed regularization while training the neural networks. Common regularization methods (e.g. dropout, weight decay, etc.) usually are related with neural networks architecture, but our PR method is applied to the data themselves. The goal of operating these experiments is to validate whether PR method can be effectively used as a defense technique against the state-of-the-art attack techniques. We conduct five cases that we have discussed in Section 3.4 in this sub-section. From Table 2 , PR method can defend most attacks in the case A and D. One interesting observation that we have found in the case B is that the neural networks which have been trained with PR method during the training phase are robust to defend most attacks (except the L 2 -CW attack) even if we do not apply PR method during the testing phase. This shows that our PR method boosts the robustness of the neural networks. For the case C, the performance of the neural networks is significantly degraded due to large increase of L p distance (i.e. the distance between the original image and the adversarial image). As we know, the case C is not practical because the L p distance has to be as low as possible in order to create an adversarial example. Although the case C is impractical, we still perform the case D to validate our PR method can increase the robustness of the neural network (except MNIST dataset with FGSM technique because of high distortion of the generated adversarial image as shown in Fig. 3 ). We conjecture our method which can defend the attacks effectively because it has eliminated most adversarial pixels during the pre-processing phase. Furthermore, we can see that our PR method performs better in the grayscale dataset from the comparison between CIFAR-10 and CIFAR-10 (grayscale) dataset.

Table 2 : Results for several datasets in the case A, B, C, and D with 1,000 epochs and k=3. Note that in the case A, we classify as F (P R(A(F, X))). In the case B, we classify as F (A(F, X)). In the case C, we classify as F (A(F, P R, X)). In the case D, we classify as F (P R(A(F, P R, X))). In addition, we compare the performance between a neural network trained with PR method (during the training phase) and a neural network trained without PR method under four different attacks, shown in Table 3 . Note that we do not apply PR method during testing phase for the neural network that has been trained with PR method (case B). From Table 3 , neural networks trained with PR method show higher performance than non-PR neural networks except in the normal case. This result indicates our PR method can increase the robustness of the neural network.

4.3. EXPERIMENT TO ANSWER QUESTION # 3:

The target of implementing these experiments is to present the benefit of using the multiple PR models rather than the single PR model. As aforementioned, the case C is impractical since the case C will cause high L p distance. Hence, we do not include case C (and D) in this sub-section. From Table 4 , the one with the multiple PR models outperforms the one with a single PR model. In average of the accuracy, the multiple PR models produce 1.86% and 1.16% higher than the single PR model for case A and B respectively. This shows the multiple PR models produce a stronger defense than the single PR model. We leave the experiments for the parameters tuning of multiple PR models in the future. The aim of conducting these experiments are to distinguish between PR model and random noise injection. Note that the value of the random noise that we used in this sub-section is between -1.0 and 1.0 with the uniform distribution and we clip the value of the pixel between 0.0 and 1.0 (normalized pixel value). We list several outputs with the use of the PR method and random noise injection in Fig. 4 .

Besides that, we show the experiment results for the random noises injection with the legitimate data in Table 5 . We use a similar settings in Section 4.1 to only apply the random noises injection during the training phase, testing phase and both phases. Comparing the results of the PR method (shown in Table 1 ) and those of the random noise injection (shown in Table 5 ), we can see that the PR method outperforms the random noise injection in "Training Phase" and "Both Phases".

Figure 4 : Examples of using PR method and some random noises in MNIST, Fashion MNIST and CIFAR-10 datasets. For every three images from the left, the leftmost one is the original image, the middle one is the image after adding some random noises, and the rightmost one is the PR image.

Although the random noise injection has high performance in fooling the trained neural networks (in the case when we apply the random noise injection during the testing phase only), the L 2 distance is larger than our method since the image generated with the random noise injection cannot be easily recognized by the human. (shown in Fig. 4 ) In summary, the PR method is different from the random noise injection in terms of the performance in maintaining the accuracy when the number of the epoch is increased. Moreover, the PR method can maintain the newly generated image perceptible by the human. From the experiment results, we summarize the contribution of using the PR method as follows:

• The PR method relieves the over-fitting issue by generating different possible output of the image. (Q1)

• The PR method increases the robustness of the neural networks.(Q2)

• The defense level for the one with the multiple PR models is stronger than the one with the single PR model. (Q3)

• The PR model has higher performance than the random noise injection. (Q4)

5. CONCLUSION:

In this study, we propose a novel method, namely Pixel Redrawn method which regenerates the image to all possible forms without compromising human perception. We address four questions in our experiments: Q1 What is the influence of applying the PR method in different phases with a legitimate test datum?, Q2 What is the influence of using the PR method in different case studies?, Q3 What is the difference between multiple PR models and a single PR model applying in the neural networks?, and Q4 What is the difference between the PR method and the random noise injection? From the experimental results, our method shows that it can prevent the over-fitting problem while maintaining the performance of the neural network, it also improves the robustness of the neural network. Furthermore, our method is more reliable than using some random noises.

A APPENDIX:

A.1 EXPERIMENTS TO ANSWER QUESTION #2

In this appendix, we provide Table 6 , which is the extended version of Table 2 in Section 4.2. Note that more results with 50 and 500 epochs are added. We also provide Table 7 and 8 with k = 4 and k = 10 respectively. The purpose of the experiments is to analyze the influence of the k value (the number of colors) to the performance of the neural network. From Table 6 , 7 and 8, we observe that the bigger the k, the higher the accuracy when we test with the legitimate example, but the lower the accuracy when we test with the adversarial examples. In other words, we have to deal with a trade-off between legitimate examples and adversarial examples. Although we believe that multiple PR models could increase the accuracy for adversarial examples while using a large value of k, we leave these experiments in the future.

A.2 EXPERIMENTS TO ANSWER QUESTION #3

In this appendix, we show Table 10 as the extended version of Table 4 in Section 4.3 with the results of the multiple PR models and the single PR model with two epochs (i.e. 500 and 1,000 epochs). In Table 10 , it can be seen that the multiple PR models outperforms the single PR model, because the multiple PR models provide more new images to train as was discussed in Section 2.5.

A.3 EXPERIMENTS TO ANSWER QUESTION #4:

In this appendix, we show the comparison results between PR method (case A) and adversarial training with 1,000 epochs for several datasets in Table 11 . We apply MIM attack technique during the adversarial training. In Table 11 , it can be seen that our method outperforms adversarial training in most cases.

Table 6 : Results for several datasets in the case A, B, C, and D with 50, 500, and 1000 epochs and k=3. Note that in the case A, we classify as F (P R(A(F, X))). In the case B, we classify as F (A(F, X)). In the case C, we classify as F (A(F, P R, X)). In the case D, we classify as F (P R(A(F, P R, X))). Table 7 : Results for several datasets in the case A, B, C, and D with 50, 500, and 1000 epochs and k=4. Note that in the case A, we classify as F (P R(A(F, X))). In the case B, we classify as F (A(F, X)). In the case C, we classify as F (A(F, P R, X)). In the case D, we classify as F (P R(A(F, P R, X))).  and k=10 . Note that in the case A, we classify as F (P R(A(F, X))). In the case B, we classify as F (A(F, X)). In the case C, we classify as F (A(F, P R, X)). In the case D, we classify as F (P R(A(F, P R, X))). 

:

We modify and extend the scripts from CleverHans. Our scripts are available at https://github.com/canboy123/pixel-redrawn

Figures and Tables:

Figure 1: Pixel Redrawn concept with MNIST example.

Figure 2: Example of original images and PR images from MNIST dataset. From a pair of an image, the left-hand side is the original image and the right-hand side is the PR image.

Figure 3: The example of the input images for the classifier from MNIST dataset with FGSM ( = 0.3), BIM ( = 0.3), MIM ( = 0.3), and L 2 -CW attack techniques in five cases.

have no knowledge about the attack techniques and they try to generate a robust neural network model (e.g. stochastic activation pruning (SAP)(Dhillon et al., 2018a), input transformations(Guo et al., 2018), etc.). The white-box defense is usually performed better than the black-box defense because it can use the state-of-the-art attack technique to create adversarial examples for the training purpose and then the trained model can prevent the attacks with the similar level (or weaker levels) of the known attack technique. However, unlike the black-box defense, the white-box defense might not be able to defend strong attacks devised in the future. Due to the disability of the white-box defense in defending future attacks, the black-box defense is more reliable to be studied. Therefore, most state-of-the-art defense techniques are the black-box defense.3.4 CASE STUDY WITH DIFFERENT ATTACK SCENARIOSIn this paper, we evaluate our method in several cases with different attack scenarios. The case studies are included as follows:• Normal: No attack technique is used. Test the neural networks with a legitimate datum.Y = F (P R(X)) • Case A: The attackers have no knowledge of the PR method but they knows the parameters of the trained neural networks. The attackers create an adversarial image from an image with the parameters of the trained neural networks. The defense mechanism receives the adversarial image as the input for the PR model and then generate PR image for the input of the trained neural networks.

Table content: <table/>

Comparison results of applying pixel redrawn method during various phases.

Table content: <table><tr><td>Dataset</td><td>Epoch</td><td>Non-PR</td><td colspan="2">Accuracy Training Phase Testing Phase</td><td>Both Phases</td></tr><tr><td/><td>50</td><td>0.9921</td><td>0.9905</td><td>0.8097</td><td>0.9893</td></tr><tr><td>MNIST</td><td>500</td><td>0.9885</td><td>0.9824</td><td>0.4964</td><td>0.9879</td></tr><tr><td/><td>1000</td><td>0.9893</td><td>0.9671</td><td>0.2765</td><td>0.9865</td></tr><tr><td/><td>50</td><td>0.9106</td><td>0.8879</td><td>0.3538</td><td>0.8724</td></tr><tr><td>Fashion MNIST</td><td>500</td><td>0.9058</td><td>0.8795</td><td>0.3018</td><td>0.8670</td></tr><tr><td/><td>1000</td><td>0.8964</td><td>0.8788</td><td>0.2550</td><td>0.8632</td></tr><tr><td/><td>50</td><td>0.6179</td><td>0.5703</td><td>0.5144</td><td>0.5386</td></tr><tr><td>CIFAR-10</td><td>500</td><td>0.5989</td><td>0.5557</td><td>0.4864</td><td>0.5323</td></tr><tr><td/><td>1000</td><td>0.5873</td><td>0.5495</td><td>0.4739</td><td>0.5204</td></tr><tr><td>CIFAR-10 (grayscale)</td><td>50 500 1000</td><td>0.5697 0.5466 0.5369</td><td>0.5099 0.4864 0.4761</td><td>0.3764 0.3429 0.3370</td><td>0.4845 0.4609 0.4569</td></tr></table>

Comparison of using non-PR model and PR model (case B) in several datasets with 1,000 epochs under four different attacks.

Table content: <table><tr><td>Dataset</td><td>Model</td><td>Normal</td><td>FGSM</td><td>Accuracy BIM</td><td>MIM</td><td>L2-CW</td></tr><tr><td>MNIST</td><td>Non-PR PR (case B), k=3</td><td>0.9893 0.9870</td><td>0.1114 0.8787</td><td>0.1005 0.7930</td><td>0.0844 0.7504</td><td>0.1568 0.6621</td></tr><tr><td>Fashion</td><td>Non-PR</td><td>0.8964</td><td>0.6308</td><td>0.4221</td><td>0.2718</td><td>0.0680</td></tr><tr><td>MNIST</td><td>PR (case B), k=3</td><td>0.8632</td><td>0.7642</td><td>0.7271</td><td>0.6815</td><td>0.1048</td></tr><tr><td>CIFAR-10</td><td>Non-PR PR (case B), k=3</td><td>0.5873 0.5204</td><td>0.1807 0.2265</td><td>0.1819 0.2104</td><td>0.1761 0.1904</td><td>0.1612 0.1728</td></tr><tr><td>CIFAR-10</td><td>Non-PR</td><td>0.5369</td><td>0.1816</td><td>0.1876</td><td>0.1839</td><td>0.1670</td></tr><tr><td>(grayscale)</td><td>PR (case B), k=3</td><td>0.4569</td><td>0.2217</td><td>0.2079</td><td>0.1888</td><td>0.1845</td></tr></table>

The comparison of using a PR model and multiple PR models in several datasets with 1,000 epochs for the case A and B.

Table content: <table><tr><td/><td/><td/><td>Accuracy</td><td/><td/></tr><tr><td>Dataset</td><td>Attack</td><td>single PR</td><td/><td colspan="2">multiple PR</td></tr><tr><td/><td/><td>Case A</td><td>Case B</td><td>Case A</td><td>Case B</td></tr><tr><td/><td>FGSM, = 0.300</td><td>0.9689</td><td>0.8787</td><td>0.9752</td><td>0.9371</td></tr><tr><td>MNIST</td><td>BIM, = 0.300 MIM, = 0.300</td><td>0.9831 0.9680</td><td>0.7930 0.7504</td><td>0.9865 0.9706</td><td>0.8003 0.7693</td></tr><tr><td/><td>L2-CW</td><td>0.9853</td><td>0.6621</td><td>0.9875</td><td>0.5767</td></tr><tr><td/><td>FGSM, = 0.031</td><td>0.8426</td><td>0.7642</td><td>0.8564</td><td>0.8059</td></tr><tr><td>Fashion</td><td>BIM, = 0.031</td><td>0.8420</td><td>0.7271</td><td>0.8555</td><td>0.7703</td></tr><tr><td>MNIST</td><td>MIM, = 0.031</td><td>0.8340</td><td>0.6815</td><td>0.8501</td><td>0.7422</td></tr><tr><td/><td>L2-CW</td><td>0.8450</td><td>0.1048</td><td>0.8476</td><td>0.1615</td></tr><tr><td/><td>FGSM, = 0.031</td><td>0.3343</td><td>0.2265</td><td>0.3630</td><td>0.2181</td></tr><tr><td>CIFAR-10</td><td>BIM, = 0.031 MIM, = 0.031</td><td>0.3554 0.3025</td><td>0.2104 0.1904</td><td>0.3859 0.3353</td><td>0.2040 0.1853</td></tr><tr><td/><td>L2-CW</td><td>0.4667</td><td>0.1728</td><td>0.4964</td><td>0.1717</td></tr><tr><td/><td>FGSM, = 0.031</td><td>0.3371</td><td>0.2217</td><td>0.3655</td><td>0.2222</td></tr><tr><td>CIFAR-10</td><td>BIM, = 0.031</td><td>0.3487</td><td>0.2079</td><td>0.3812</td><td>0.2118</td></tr><tr><td>(grayscale)</td><td>MIM, = 0.031</td><td>0.3216</td><td>0.1888</td><td>0.3532</td><td>0.1948</td></tr><tr><td/><td>L2-CW</td><td>0.4290</td><td>0.1845</td><td>0.4526</td><td>0.1795</td></tr><tr><td colspan="3">4.4 EXPERIMENT TO ANSWER QUESTION # 4</td><td/><td/><td/></tr></table>

Comparison results of applying the some random noises on the image during the different phase.

Table content: <table><tr><td>Dataset</td><td>Epoch</td><td>Training Phase</td><td>Accuracy Testing Phase</td><td>Both Phases</td></tr><tr><td/><td>50</td><td>0.9871</td><td>0.1650</td><td>0.9596</td></tr><tr><td>MNIST</td><td>500</td><td>0.2789</td><td>0.1958</td><td>0.9705</td></tr><tr><td/><td>1000</td><td>0.1021</td><td>0.1695</td><td>0.9758</td></tr><tr><td/><td>50</td><td>0.8275</td><td>0.1435</td><td>0.8161</td></tr><tr><td>Fashion MNIST</td><td>500</td><td>0.7308</td><td>0.1357</td><td>0.8446</td></tr><tr><td/><td>1000</td><td>0.7263</td><td>0.1297</td><td>0.8449</td></tr><tr><td/><td>50</td><td>0.5112</td><td>0.2205</td><td>0.4779</td></tr><tr><td>CIFAR-10</td><td>500</td><td>0.4296</td><td>0.1883</td><td>0.4674</td></tr><tr><td/><td>1000</td><td>0.4194</td><td>0.4739</td><td>0.4718</td></tr><tr><td colspan="3">4.5 SUMMARY OF PR METHOD</td><td/><td/></tr></table>

Results for several datasets in the case A, B, C, and D with 50, 500, and 1000 epochs

Table content: <table/>

Comparison of using non-PR model and PR model (case B) in several datasets with 50, 500, and 1,000 epochs under four different attacks.

Table content: <table><tr><td>Dataset</td><td>Attack</td><td>Epoch</td><td>non-PR</td><td>Accuracy</td><td>PR (case B), k=3</td></tr><tr><td/><td/><td>50</td><td>0.9921</td><td/><td>0.9897</td></tr><tr><td/><td>Normal</td><td>500</td><td>0.9885</td><td/><td>0.9867</td></tr><tr><td/><td/><td>1000</td><td>0.9893</td><td/><td>0.9870</td></tr><tr><td>MNIST</td><td>FGSM, = 0.300 BIM, = 0.300</td><td>50 500 1000 50 500 1000</td><td>0.2004 0.2187 0.1114 0.0369 0.0576 0.1005</td><td/><td>0.8527 0.9045 0.8787 0.4485 0.8003 0.7930</td></tr><tr><td/><td>MIM, = 0.300</td><td>50 500 1000</td><td>0.0230 0.0549 0.0844</td><td/><td>0.3651 0.7574 0.7504</td></tr><tr><td/><td/><td>50</td><td>0.0156</td><td/><td>0.1617</td></tr><tr><td/><td>L2-CW</td><td>500</td><td>0.1018</td><td/><td>0.6758</td></tr><tr><td/><td/><td>1000</td><td>0.1568</td><td/><td>0.6621</td></tr><tr><td/><td/><td>50</td><td>0.9106</td><td/><td/></tr><tr><td/><td>Normal</td><td/><td/><td/><td/></tr><tr><td/><td/><td>1000</td><td>0.8964</td><td/><td>0.8632</td></tr><tr><td>Fashion</td><td>FGSM, = 0.031</td><td>50 500 1000</td><td>0.6083 0.6624 0.6308</td><td/><td>0.7366 0.7492 0.7642</td></tr><tr><td>MNIST</td><td>BIM, = 0.031</td><td>50 500 1000</td><td>0.4265 0.4064 0.4221</td><td/><td>0.6944 0.7038 0.7271</td></tr><tr><td/><td>MIM, = 0.031</td><td>50 500 1000</td><td>0.3566 0.2401 0.2718</td><td/><td>0.6662 0.6644 0.6815</td></tr><tr><td/><td/><td>50</td><td>0.0647</td><td/><td>0.0700</td></tr><tr><td/><td>L2-CW</td><td>500</td><td>0.0652</td><td/><td>0.0826</td></tr><tr><td/><td/><td>1000</td><td>0.0680</td><td/><td>0.1048</td></tr><tr><td/><td/><td>50</td><td>0.6179</td><td/><td>0.5386</td></tr><tr><td/><td>Normal</td><td>500</td><td>0.5989</td><td/><td>0.5323</td></tr><tr><td/><td/><td>1000</td><td>0.5873</td><td/><td>0.5204</td></tr><tr><td>CIFAR-10</td><td>FGSM, = 0.031 BIM, = 0.031</td><td>50 500 1000 50 500 1000</td><td>0.1801 0.1753 0.1807 0.1767 0.1751 0.1819</td><td/><td>0.1931 0.2053 0.2265 0.1876 0.1932 0.2104</td></tr><tr><td/><td>MIM, = 0.031</td><td>50 500 1000</td><td>0.1741 0.1723 0.1761</td><td/><td>0.1783 0.1775 0.1904</td></tr><tr><td/><td/><td>50</td><td>0.1603</td><td/><td>0.1646</td></tr><tr><td/><td>L2-CW</td><td>500</td><td>0.1576</td><td/><td>0.1626</td></tr><tr><td/><td/><td>1000</td><td>0.1612</td><td/><td>0.1728</td></tr><tr><td/><td/><td>50</td><td>0.5697</td><td/><td>0.4845</td></tr><tr><td/><td>Normal</td><td>500</td><td>0.5466</td><td/><td>0.4609</td></tr><tr><td/><td/><td>1000</td><td>0.5369</td><td/><td>0.4569</td></tr><tr><td>CIFAR-10</td><td>FGSM, = 0.031</td><td>50 500 1000</td><td>0.1781 0.1732 0.1816</td><td/><td>0.2216 0.2098 0.2217</td></tr><tr><td>(grayscale)</td><td>BIM, = 0.031</td><td>50 500 1000</td><td>0.1786 0.1768 0.1876</td><td/><td>0.2112 0.2005 0.2079</td></tr><tr><td/><td>MIM, = 0.031</td><td>50 500 1000</td><td>0.1745 0.1744 0.1839</td><td/><td>0.1971 0.1834 0.1888</td></tr><tr><td/><td/><td>50</td><td>0.1597</td><td/><td>0.1817</td></tr><tr><td/><td>L2-CW</td><td>500</td><td>0.1621</td><td/><td>0.1727</td></tr><tr><td/><td/><td>1000</td><td>0.1670</td><td/><td>0.1845</td></tr></table>

The comparison of using a PR model and multiple PR models in several datasets with 500 and 1,000 epochs respectively, for case A and case B.

Table content: <table><tr><td/><td/><td/><td/><td>Accuracy</td><td/><td/></tr><tr><td>Dataset</td><td>Attack</td><td>Epoch</td><td>single PR</td><td/><td colspan="2">multiple PR</td></tr><tr><td/><td/><td/><td>Case A</td><td>Case B</td><td>Case A</td><td>Case B</td></tr><tr><td/><td/><td>500</td><td>0.9698</td><td>0.9045</td><td>0.9724</td><td>0.9350</td></tr><tr><td/><td>FGSM</td><td>1000</td><td>0.9698</td><td>0.8787</td><td>0.9752</td><td>0.9371</td></tr><tr><td/><td/><td>500</td><td>0.9786</td><td>0.8003</td><td>0.9816</td><td>0.8224</td></tr><tr><td>MNIST,</td><td>BIM</td><td>1000</td><td>0.9831</td><td>0.7930</td><td>0.9865</td><td>0.8003</td></tr><tr><td>= 0.300</td><td/><td>500</td><td/><td>0.7574</td><td>0.9645</td><td>0.7806</td></tr><tr><td/><td>MIM</td><td>1000</td><td>0.9680</td><td>0.7504</td><td>0.9706</td><td>0.7693</td></tr><tr><td/><td/><td>500</td><td>0.9842</td><td>0.6758</td><td>0.9872</td><td>0.3273</td></tr><tr><td/><td>L2-CW</td><td>1000</td><td>0.9853</td><td>0.6621</td><td>0.9875</td><td>0.5767</td></tr><tr><td/><td/><td>500</td><td>0.8380</td><td>0.7492</td><td>0.8582</td><td>0.7903</td></tr><tr><td/><td>FGSM</td><td>1000</td><td>0.8426</td><td>0.7642</td><td>0.8564</td><td>0.8059</td></tr><tr><td>Fashion MNIST, = 0.031</td><td>BIM MIM</td><td>500 1000 500 1000</td><td>0.8360 0.8420 0.8302 0.8340</td><td>0.7038 0.7271 0.6644 0.6815</td><td>0.8527 0.8555 0.8503 0.8501</td><td>0.7575 0.7703 0.7310 0.7422</td></tr><tr><td/><td/><td>500</td><td>0.8444</td><td>0.0826</td><td>0.8487</td><td>0.1432</td></tr><tr><td/><td>L2-CW</td><td>1000</td><td>0.8450</td><td>0.1048</td><td>0.8476</td><td>0.1615</td></tr><tr><td/><td/><td>500</td><td>0.3256</td><td>0.2053</td><td>0.3470</td><td>0.2104</td></tr><tr><td/><td>FGSM</td><td>1000</td><td>0.3343</td><td>0.2265</td><td>0.3630</td><td>0.2181</td></tr><tr><td/><td/><td>500</td><td>0.3385</td><td>0.1932</td><td>0.3689</td><td>0.2013</td></tr><tr><td>CIFAR-10,</td><td>BIM</td><td>1000</td><td>0.3554</td><td>0.2104</td><td>0.3859</td><td>0.2040</td></tr><tr><td>= 0.031</td><td/><td>500</td><td>0.2898</td><td>0.1775</td><td>0.3189</td><td>0.1847</td></tr><tr><td/><td>MIM</td><td>1000</td><td>0.3025</td><td>0.1904</td><td>0.3353</td><td>0.1853</td></tr><tr><td/><td/><td>500</td><td>0.4708</td><td>0.1626</td><td>0.4947</td><td>0.1709</td></tr><tr><td/><td>L2-CW</td><td>1000</td><td>0.4667</td><td>0.1728</td><td>0.4964</td><td>0.1717</td></tr><tr><td/><td/><td>500</td><td>0.3357</td><td>0.2098</td><td>0.3665</td><td>0.2174</td></tr><tr><td/><td>FGSM</td><td>1000</td><td>0.3371</td><td>0.2217</td><td>0.3655</td><td>0.2222</td></tr><tr><td>CIFAR-10 (grayscale), = 0.031</td><td>BIM MIM</td><td>500 1000 500 1000</td><td>0.3534 0.3487 0.3234 0.3216</td><td>0.2005 0.2079 0.1834 0.1888</td><td>0.3750 0.3812 0.3526 0.3532</td><td>0.2083 0.2118 0.1917 0.1948</td></tr><tr><td/><td/><td>500</td><td>0.4363</td><td>0.1727</td><td>0.4483</td><td>0.1829</td></tr><tr><td/><td>L2-CW</td><td>1000</td><td>0.4290</td><td>0.1845</td><td>0.4526</td><td>0.1795</td></tr></table>

Comparison between PR method (case A) and adversarial training in several datasets with 1,000 epochs.

Table content: <table><tr><td>Dataset</td><td>Attack</td><td>PR</td><td>Accuracy</td><td>Adversarial Training</td></tr><tr><td>MNIST, = 0.300</td><td>FGSM BIM MIM</td><td>0.9691 0.9825 0.9672</td><td/><td>0.9712 0.9264 0.9287</td></tr><tr><td>Fashion MNIST, = 0.031</td><td>FGSM BIM MIM</td><td>0.8503 0.8489 0.8446</td><td/><td>0.8408 0.8190 0.8059</td></tr><tr><td>CIFAR-10, = 0.031</td><td>FGSM BIM MIM</td><td>0.3343 0.3554 0.3025</td><td/><td>0.3830 0.3495 0.3072</td></tr></table>

Peer review (iclr2019):
 This paper presents a new adversarial defense method. I found the paper difficult to understand, but as far as I understand, the method involves randomly perturbing the pixels of images in a dataset, and retraining the classifier to correctly classify these perturbed images as well. The perturbations are done independently per pixel by quantizing the pixel value, and then using a perceptron model to generate the full pixel value from just this quantized value. The perceptron is trained on randomly generated data mapping quantized pixel values to full pixel values (this random generation does not use the dataset statistics at all). They use both partially converged perceptron models as well as fully converged perceptron models. Pros: 1. The defense technique does not require knowledge of the attack method Cons: 1. The paper is incredibly difficult to understand due to the writing. 2. The performed experiments are insufficient to determine whether or not their technique works because: a. They don*t compare against any other defense techniques. In addition to at least adversarial training, I would like to also see comparisons to feature squeezing which I would expect to have a very similar effect. b. They do not show the results of an attack by an adversary that is aware of their technique. (i.e. F(PR(A(F,PR,x)))) Many alternative defense techniques will work much better if we assume the adversary does not know about the technique. c. Their comparison against random noise is not an apples-to-apples comparison. Instead of perturbing uniformly within a range, they perturb according to a normal distribution. The random noise perturbations also have a much larger L2 distance than the perturbations from their technique. To believe that their method is actually better than training with random noise I*d like to see an apples-to-apples comparison where these values are hyperparameter tuned as they presumably did for their method. d. They only show results for one value of epsilon and one value of "# of colors" for their technique. Presumably if there is a mismatch between these values then the results will be much worse (i.e. if they choose a large "# of colors"/small range per color and the attacker chooses a large epsilon). 3. Their use of machine learning models is quite ad-hoc. In particular they use a perceptron trained on algorithmically generated data. And their justification for using a perceptron, instead of just using the known underlying generation algorithm is that they also use a partially converged version of the perceptron as part of their model. 4. The authors partly deanonymize the paper through a github link 5. The main paper is 10 pages long, and the quality of the paper does not justify the additional length.