Title: BANDLIMITING NEURAL NETWORKS AGAINST ADVERSARIAL ATTACKS

Abstract: In this paper, we study the adversarial attack and defence problem in deep learning from the perspective of Fourier analysis. We first explicitly compute the Fourier transform of deep ReLU neural networks and show that there exist decaying but non-zero high frequency components in the Fourier spectrum of neural networks. We then demonstrate that the vulnerability of neural networks towards adversarial samples can be attributed to these insignificant but non-zero high frequency components. Based on this analysis, we propose to use a simple post-averaging technique to smooth out these high frequency components to improve the robustness of neural networks against adversarial attacks. Experimental results on the ImageNet and the CIFAR-10 datasets have shown that our proposed method is universally effective to defend many existing adversarial attacking methods proposed in the literature, including FGSM, PGD, DeepFool and C&W attacks. Our post-averaging method is simple since it does not require any re-training, and meanwhile it can successfully defend over 80-96% of the adversarial samples generated by these methods without introducing significant performance degradation (less than 2%) on the original clean images.
Although deep neural networks (DNN) have shown to be powerful in many machine learning tasks, Szegedy et al. (2013) found that they are vulnerable to adversarial samples. Adversarial samples are subtly altered inputs that can fool the trained model to produce erroneous outputs. They are more commonly seen in image classification task and typically the perturbations to the original images are so small that they are imperceptible to human eye.

:

attributed the adversarial samples to the low-probability blind spots in the manifold. In (Goodfellow et al., 2014) , the authors explained adversarial samples as "a result of models being too linear, rather than too nonlinear." In (Papernot et al., 2016) , the authors showed the transferability occurs across models with different structures and even different machine learning techniques in addition to neural networks. In summary, the general existence and transferability of adversarial samples are well known but the reason of adversarial vulnerability still needs further investigation.

Generally speaking, when we view neural network as a multivariate function f (x) of input x, if a small imperceptible perturbation ∆x leads to a huge fluctuation ∆f (x), the large quantity ∆f (x)/∆x essentially corresponds to high frequency components in the Fourier spectrum of f (x). In this paper, we will start with the Fourier analysis of neural networks and elucidate why there always exist some decaying but nonzero high frequency response components in neural networks. Based on this analysis, we show that neural networks are inherently vulnerable to adversarial samples due to the underlying model structure. Next, we propose a simple post-averaging method to tackle this problem. Our proposed method is fairly simple since it works as a post-processing stage of any given neural network models and it does not require re-training the networks at all. Furthermore, we have evaluated the post-averaging method against four popular adversarial attacking methods and our method is shown to be universally effective in defending all examined attacks. Experimental results on the ImageNet and the CIFAR-10 datasets have shown that our simple post-averaging method can successfully defend over 80-96% of the adversarial samples generated by these attacks with little performance degradation (less than 2%) on the original clean images.

2. FOURIER ANALYSIS OF NEURAL NETWORKS:

In order to understand the behaviour of adversarial samples, it is essential to find the Fourier transform of neural networks. Fortunately, for some widely used neural networks, namely fully-connected neural networks using ReLU activation functions, we may explicitly derive their Fourier transform under some minor conditions. As we will show, these theoretical results will shed light on how adversarial samples happen in neural networks.

2.1. FOURIER TRANSFORM OF FULLY-CONNECTED RELU NEURAL NETWORKS:

As we know, any fully-connected ReLU neural networks (prior to the softmax layer) essentially form piece-wise linear functions in input space. Due to space limit, we will only present the main results in this section and the proofs and more details may be found in Appendix.

Definition 2.1. A piece-wise linear function is a continuous function f : R n -→ R such that there are some hyperplanes passing through origin and dividing R n into M pairwise disjoint regions R m , (m = 1, 2, ..., M ), on each of which f is linear:

f (x) =        w 1 • x x ∈ R 1 w 2 • x x ∈ R 2 . . . w M • x x ∈ R M Lemma 2.2.

Composition of a piece-wise linear function with a ReLU activation function is also a piece-wise linear function.

Theorem 2.3. The output of any hidden unit in an unbiased fully-connected ReLU neural network is a piece-wise linear function. This is straightforward because the input to any hidden node is a linear combination of piece-wise linear functions and this input is composed with the ReLU activation function to yield the output, which is also piece-wise linear. However, each region R m is the intersection of a different number of half-spaces, enclosed by various hyperplanes in R n . In general, these regions R m (m = 1, • • • , M ) do not have simple shapes. For the purpose of mathematical analysis, we need to decompose each region into a union of some well-defined shapes having a uniform form, which is called infinite simplex.

Definition 2.4. Let V = {v 1 , v 2 , ..., v n } be a set of n linearly independent vectors in R n . An infinite simplex, R + V , is defined as the region linearly spanned by V using only positive weights:

EQUATION

Theorem 2.5. Each piece-wise linear function f (x) can be formulated as a summation of some simpler functions:

f (x) = L l=1 f l (x)

, each of which is linear and non-zero only in an infinite simplex as follows:

f l (x) = w l • x x ∈ R + V l 0 otherwise (2)

where V l is a set of n linearly independent vectors, and w l is a weight vector.

In practice, we can always assume that the input to neural networks, x, is bounded. As a result, for computational convenience, we may normalize all inputs x into the unit hyper-cube,

U n = [0, 1] n .

Obviously, this assumption can be easily incorporated into the above analysis by multiplying each f l (x) in eq.( 2) by n r=1 h(x r )h(1 -x r ) where h(x) is the Heaviside step function. Alternatively, we may simplify this term by adding n 2 additional hyperplanes to further split the input space to ensure all the elements of x do not change signs within each region R + Vq . In this case, within each region R + Vq , the largest absolute value among all elements of x is always achieved by a specific element, which is denoted as r q . In other words, the dimension x rq achieves the largest absolute value inside R + Vq . Similarly, the normalized piece-wise linear function may be represented as a summation of some functions: f (x) = Q q=1 g q (x), where each g q (x) (q = 1, 2, • • • , Q) has the following form:

g q (x) = w q • x h(1 -x rq ) x ∈ R + Vq 0 otherwise

For every V q , there exists an n × n invertible matrix A q to linearly transform all vectors of V q into standard basis vectors e i in R n . As a result, each function g q (x) may be represented in terms of standard bases V * = {e 1 , • • • , e n } as follows:

g q (x) = wq • xq h(1 -1 • xq ) xq ∈ R + V * 0 otherwise

where xq = xA T q , and wq = w q A -1 q . Lemma 2.6. Fourier transform of the following function:

s(x) = h(1 -1 • x) x ∈ R + V *

0 otherwise may be presented as:

S(ω) = -i √ 2π n n r=0 e -iωr r =r (ω r -ω r ) (3)

where ω r is the r-th component of frequency vector ω (r = 1, • • • , n), and ω 0 = 0.

Finally we derive the Fourier transform of fully-connected ReLU neural networks as follows.

Theorem 2.7. The Fourier transform of the output of any hidden node in a fully-connected unbiased 1 ReLU neural network may be represented as

Q q=1 w q A -1 q ∇S(ωA -1 q )

, where ∇ denote the differential operator. Obviously, neural networks are the so-called approximated bandlimited models as defined in (Jiang, 2019) , which have decaying high frequency components in Fourier spectrum. Theorem 2.7 further suggests that the matrices A -1 q may contribute to the high frequency components when the corresponding region R + Vq are too small. This is clear because the determinant of A q is proportional to the volume of R + Vq in R n . In summary, the high frequency components of neural networks are mostly attributed to these tiny regions in the input space. As we will show later, these small regions may be explicitly exploited to generate adversarial samples for neural networks.

2.2. UNDERSTANDING ADVERSARIAL SAMPLES:

As shown in Theorem 2.3, neural network may be viewed as a sequential division of the input space into many small regions, as illustrated in Figure 1 . Each layer is a further division of the existing regions from the previous layers, with each region being divided differently. Hence a neural network with multiple layers would result in a tremendous amount of sub-regions in the input space. For example, when cutting an n-dimensional space using N hyperplanes, the maximum number of regions may be computed as

N 0 + N 1 + • • • + N n .

For a hidden layer of N = 1000 nodes and input dimension is n = 200, the maximum number of regions is roughly equal to 10 200 . In other words, even a middle-sized neural network can partition input space into a huge number of sub-regions, which can easily exceed the total number of atoms in the universe. When we learn a neural network, we can not expect there is at least one training sample inside each region. For those regions that do not have any training sample, the resultant linear functions in them may be arbitrary since they do not contribute to the training objective function at all. Of course, most of these regions are extremely small in size. When we measure the expected loss function over the entire space, their contributions are negligible since the chance for a randomly sampled point to fall into these tiny regions is extremely small. However, adversarial attack is imposing a new challenge since adversarial samples are not naturally sampled. Given that the total number of regions is huge, those tiny regions are almost everywhere in the input space. For any data point in the input space, we almost surely can find such a tiny region in proximity where the linear function is arbitrary. If a point inside this tiny region is selected, the output of the neural network may be unexpected. We believe that these tiny unlearned regions may be a major reason why neural networks are vulnerable to adversarial samples.

In layered deep neural networks, the linear functions in all regions are not totally independent. If we use v (l) to denote the weight matrix in layer l, the resultant linear weight w k in eq.( 2) is actually the sum of all concatenated v (l) along all active paths. When we make a small perturbation ∆x to any input x, the fluctuation in the output of any hidden node can be approximated represented as:

∆f (x) ∝ N • l E |v (l) ij | (4)

where N denotes the total number of hyperplanes to be crossed when moving x to x + ∆x. In any practical neural network, we normally have at least tens of thousands of hyperplanes crossing the hypercube U n = [0, 1] n . In other words, for any input x in a high-dimensional space, a small perturbation can always easily cross a large number of hyperplanes to enter a tiny unlearned region.

When N is fairly large, the above equation indicates that the output of a neural network can still fluctuate dramatically even after all weight vectors are regularized by L 1 or L 2 norm. As a reference, we have verified this on some ImageNet data using a VGG16 model. When PGD is used to generate adversarial samples with average perturbation ||∆x|| 2 ≤ 0.35, which is extremely small perturbation since x has over a hundred thousand dimensions on ImageNet, we have observed that in average about N = 5278 hyperplanes are crossed per layer even after such a small perturbation is added.

At last, since the ubiquitous existence of unlearned tiny regions is an intrinsic property of neural networks given its current model structure, we believe that adversarial training strategies will not be sufficient to completely get rid of adversarial samples. In principle, neural networks must be strictly bandlimited to filter out those decaying high frequency components in order to completely eliminate all adversarial samples. We definitely need more research efforts to figure out how to do this effectively and efficiently for neural networks.

3 THE PROPOSED DEFENCE APPROACH: POST-AVERAGING

3.1. POST-AVERAGING:

In this paper, we propose a simple post-processing method to smooth out those high frequency components as much as possible, which relies on a simple idea similar to moving-average in onedimensional sequential data. Instead of generating prediction merely from one data point, we use the averaged value within a small neighborhood around the data point, which is called post-averaging here. Mathematically, the post-averaging is computed as an integral over a small neighborhood centered at the input:

f C (x) = 1 V C • • • x ∈C f (x -x ) dx (5)

where x is the input and f (x) represents the output of the neural network, and C denotes a small neighborhood centered at the origin and V C denotes its volume. When we choose C to be an n-sphere in R n of radius r, we may simply derive the Fourier transform of f C (x) as follows:

F C (ω) = F (ω) 1 V C • • • x ∈C e -ix •ω dx = F (ω) Γ( n 2 + 1) π n 2 J n 2 (r|ω|) (r|ω|) n 2 (6)

where J n 2 (•) is the first kind Bessel function of order n/2. Since the Bessel functions, J ν (ω), decay with rate 1/ √ ω as |ω| → ∞ (Watson, 1995), we have

F C (ω) ∼ F (ω) (r|ω|) n+1 2

as |ω| → ∞. Therefore, if r is chosen properly, the post-averaging operation can significantly bandlimit neural networks by smoothing out high frequency components. Note that the similar ideas have been used in (Jiang et al., 1999; Jiang and Lee, 2003) to improve robustness in speech recognition.

3.2. SAMPLING METHODS:

However, it is intractable to compute the above integral for any meaningful neural network used in practical applications. In this work, we propose to use a simple numerical method to approximate it. For any input x, we select K points in the neighborhood C centered at x, i.e. {x 1 , x 2 , • • • , x K } , to approximately compute the integral as

EQUATION

Obviously, in order to defend against adversarial samples, it is important to have samples outside the current unlearned tiny region. In the following, we use a simple sampling method based on directional vectors. To generate a relatively even set of samples for eq.( 7), we first determine some directional vectors v, and then move the input x along these directions using several step sizes within the sphere of radius r:

x = x + λ • v (8) where λ = [± r 3 , ± 2r 3 , ±r],

and v is a selected unit-length directional vector. For each selected direction, we generate six samples within C along both the positive and the negative directions to ensure efficiency and even sampling. We use this implementation for the convenience to extend with different types of sampling strategies.

We tried several direction sampling strategies, including using the directions towards the closest region boundaries, and found that the simple random direction sampling gives the best performance. In this sampling method, we fill the directional vectors with random numbers generated from a standard normal distribution, and then normalize them to have unit length.

4. EXPERIMENTS:

In this section, we evaluate the above post-averaging method on defending against several popular adversarial attacking methods.

4.1. EXPERIMENTAL SETUP:

• Dataset: We evaluated our method on both the ImageNet (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky et al., 2009) datasets. Since our proposed post-averaging method does not need to re-train neural networks, we do not need to use any training data in our experiments.

For evaluation purpose, we use the validation set of the ImageNet dataset. The validation set consists of 50000 images labelled into 1000 categories. For computational efficiency, we randomly choose 5000 images from the ImageNet validation set and evaluate our model on these 5000 images.

For the CIFAR-10 dataset, we use the full test set, which consists of 10000 images labelled into 10 categories. • Target model: For model on ImageNet, we use a pre-trained ResNet-152 (He et al., 2016) network that is available from PyTorch, while for CIFAR-10, we use a pre-trained ResNet-110 network from Yerlan Idelbayev2 . In our experiments, we directly use these pre-trained models without any modification. • Source of adversarial attacking methods: We use Foolbox (Rauber et al., 2017) , an open source tool box to generate adversarial samples using different adversarial attacking methods. In this work, we tested our method against four popular attacking methods in the literature: Fast Gradient Sign method (FGSM) (Goodfellow et al., 2014) , Projected Gradient Descent (PGD) method (Kurakin et al., 2016; Madry et al., 2017) , DeepFool (DF) attack method (Moosavi-Dezfooli et al., 2016) and Carlini & Wagner (C&W) L2 attack method (Carlini and Wagner, 2017a) . We used these attack methods in their default settings. • Threat model: In our experiments, we use an l ∞ norm to constrain the allowed perturbation distance.

4.2. EVALUATION CRITERIA:

For each experiment, we define:

• Clean set: The dataset that consists of the original images from ImageNet or CIFAR-10.

• Attacked set: For every correctly classified image in the Clean set, if an adversarial sample is successfully generated under the attacking criteria, the original sample is replaced with the adversarial sample; if no adversarial sample is found, the original sample is kept in the dataset. Meanwhile, all the misclassified images are kept in the dataset without any change.

Therefore the Attacked set has the same number of images as the clean set.

In our experiments, we evaluate the original network and the network defended by post-averaging on both the Clean and the Attacked sets. The performance is measured in terms of :

• Accuracy: number of correctly classified images over the whole dataset.

• Defence rate: number of successfully defended adversarial samples over the total number of adversarial samples in the Attacked set. By "successfully defended", it refers to the case where an adversarial sample is correctly classified after the original model is defended by the post-averaging approach. 

4.3. EXPERIMENTAL RESULTS:

Table 1 shows the performance of our defence approach against different attacking methods. In this table, the samples for post-averaging are selected within an n-sphere of radius r as in eq.( 8), with K = 15 different directions. Thus results in a total of 15 × 2 × 3 + 1 = 91 samples (including the input) for each input image to be used in eq.( 7). Moreover, all the adversarial samples generated are restricted to be within the perturbation range = 8 /255. We show the top-1 accuracy of the original model and the defended model on both the Clean and the Attacked set respectively, as well as the defence rate of the defended model. Besides, we also show the number of adversarial samples successfully generated by each attacking method in the last column.

From Table 1 , we can see that our proposed defence approach is universally robust to all of the attacking methods we have examined. It has achieved above 80-96% defence rates in all the experiments with only a minor performance degradation in the Clean set (less than 2%). Especially on the ImageNet dataset, our method is able to defend about 95% of the adversarial samples. However, an interesting observation from the experimental results is that the defence rate in the CIFAR-10 dataset is lower than the usually more challenging ImageNet dataset. We think this may be because data points are sparser in the ImageNet space than in the CIFAR-10 space, as ImageNet has a much larger dimensionality.

Generally, using a larger sampling radius r can increase the chance of moving out of the unlearned regions as we desired, but it will also introduce more noise that can harm the prediction accuracy; On the other hand, using a smaller sampling radius r can reduce the performance degradation but it may not be sufficient to defend against adversarial samples. The optimal value for r varies with different datasets due to their dimensionality and data sparsity. In experiments, we found that r = 30 for ImageNet and r = 6 for CIFAR-10 achieved relatively better performance. Figure 2 shows how the model defence rate on ImageNet varies with different r. As shown in the figure, the optimal value for r also varies in different attacking methods, but the performance variations are small. In general, our model retains high defence rate throughout the r range [15, 30] . We also tested the effect of K, the number of sampling directions used, on the model performance.

From Table 2 , we can see that our model performance is not very sensitive to K. It is able to achieve a good defence rate with only K = 6, that is, 37 samples used for each input image. In implementation, these samples can be easily packed into a mini-batch for fast computation in GPUs. When running on the same machine, we measured the averaged inference time for a single input image on the original network as 0.04 seconds, while the inference time for our models with different K are shown in Table 2 . By comparison, we can know that the inference time after adding post-averaging is roughly 2 3 K of the original inference time.

At last, we evaluated our post-averaging defence approach against attacks with different allowed perturbation ranges . The results are shown in Figure 3 . As we can see, our model retains very good attack defence rate up to = 32 /255. Note that the defence rate against PGD and C&W doesn't change much along the variation of , this is because PGD and C&W have already successfully generated adversarial samples for most of the correctly classified inputs when is small. Hence their generated adversarial samples will not change much when using larger . For FGSM, our method yields lower defending performance. The possible reason is that FGSM tends to generate much larger perturbations than other three stronger attacking methods under the same setting. A large perturbation is more likely to move samples across class-specific decision boundaries to generate much more confusing samples. In our opinion, this is a general phenomenon in pattern classification, not particular to adversarial attacks.

5. FINAL REMARKS:

In this paper, we have presented some theoretical results by Fourier analysis of ReLU neural networks. These results are useful for us to understand why neural networks are vulnerable to adversarial samples. Based on the results, we hypothesize that the inevitable and ubiquitous existence of tiny unlearned regions in the model function mapping may be a major reason for adversarial vulnerability. As a possible defence strategy, we have proposed a simple post-averaging method. Experimental results on the ImageNet and the CIFAR-10 datasets have demonstrated that our simple defence technique turns out to be very effective against many popular attack methods in the literature. Finally, it will be interesting to see whether our post-averaging method will be still robust against any new attack methods in the future. where ω r is the rth component of frequency vector ω (r = 1, • • • , n), and ω 0 = 0.

Proof. Alternatively, s(x) may be represented as:

EQUATION

Therefore, we need to compute Fourier transform of h(x)h(1 -x):

EQUATION

1 -e -iω ω (13)

By taking the inverse Fourier transform of the function:

( √ 2π) n-1 ∞ -∞ -i √ 2π 1 -e -iζ ζ δ n (ω -ζ1) dζ (14)

where δ n is n-dimensional Dirac Delta function, it can be shown that it is the Fourier transform of h(1 • x)h(1 -1 • x):

EQUATION

1 -e -iζ ζ δ n (ω -ζ1) dζ dω (16)

= 1 √ 2π ∞ -∞ -i √ 2π 1 -e -iζ ζ • • • R n e iω.

x δ n (ω -ζ1) dω dζ (17

EQUATION

1 -e -iζ ζ e iζ1.x dζ (18)

EQUATION

Now we can find the Fourier transform of s(x)

EQUATION

)

where * is convolution operator. The final integrand may be represented as: (25)

For mathematical convenience, we assume neural networks have no biases here. However, regular neural networks with biases may be reformulated as unbiased ones by adding another dimension of constants. Thus, the main results here are equally applicable to both cases. Note that regular neural networks with biases are used in our experiments in this paper.

https://github.com/akamaster/pytorch_resnet_cifar10/tree/master/ pretrained_models

Figures and Tables:

Figure 1: Illustration of input space divided into sub-regions by a biased neural network. The black lines are the hyperplanes for the first network layer, while the blue lines are for the second layer and the red lines are for the third layer. A small perturbation from point A to point B may possibly cross many hyperplanes.

Figure 2: Defence rate of post-averaging when using different sampling radius r on ImageNet.

Figure 3: Defence rate of post-averaging on different allowed perturbation distances on CIFAR-10.

ω r -ζ B⊆Ω (-1) |B| e -i(σ B -|B|ζ) (24) = n r=0 A r ω r -ζ B⊆Ω (-1) |B| e -i(σ B -(|B|-1)ζ)

Performance of post-averaging defending against different attacking methods ( = 8 /255, and K = 15).

Table content: <table><tr><td/><td/><td colspan="2">Original Model</td><td colspan="3">Defended by Post-Averaging</td></tr><tr><td/><td/><td colspan="2">Top-1 Accuracy</td><td colspan="2">Top-1 Accuracy</td><td>Defence</td></tr><tr><td>attack, defence</td><td>Dataset</td><td colspan="4">Clean Attacked Clean Attacked</td><td>Rate</td><td>#Adv</td></tr><tr><td>FGSM, random(r=30)</td><td/><td/><td>0.0750</td><td>0.7734</td><td>0.7488</td><td>0.9426</td><td>3500</td></tr><tr><td>PGD, random(r=30) DF, random(r=30)</td><td colspan="2">ImageNet 0.7750</td><td>0.0004 0.0350</td><td>0.7732 0.7730</td><td>0.7606 0.7614</td><td>0.9639 0.9639</td><td>3873 3710</td></tr><tr><td>C&amp;W, random(r=30)</td><td/><td/><td>0.0090</td><td>0.7734</td><td>0.7548</td><td>0.9554</td><td>3830</td></tr><tr><td>FGSM, random(r=6)</td><td/><td/><td>0.1816</td><td>0.9247</td><td>0.8022</td><td>0.8080</td><td>7552</td></tr><tr><td>PGD, random(r=6) DF, random(r=6)</td><td colspan="2">CIFAR-10 0.9368</td><td>0.0000 0.1968</td><td>0.9255 0.9254</td><td>0.8841 0.8626</td><td>0.9330 0.8872</td><td>9368 7413</td></tr><tr><td>C&amp;W, random(r=6)</td><td/><td/><td>0.0322</td><td>0.9257</td><td>0.8902</td><td>0.9367</td><td>9046</td></tr></table>

Performance of post-averaging on ImageNet with different number of sampling directions ( = 8 /255 and r = 30).

Table content: <table><tr><td/><td colspan="2">Original Model</td><td colspan="3">Defended by Post-Averaging</td><td/><td>Averaged</td></tr><tr><td/><td colspan="2">Top-1 Accuracy</td><td colspan="2">Top-1 Accuracy</td><td>Defence</td><td/><td>Inference</td></tr><tr><td>attack, defence</td><td colspan="4">Clean Attacked Clean Attacked</td><td>Rate</td><td>#Adv</td><td>Time</td></tr><tr><td>PGD, random(K=6)</td><td/><td>0.0004</td><td>0.7736</td><td>0.7606</td><td>0.9636</td><td>3873</td><td>0.18s</td></tr><tr><td>PGD, random(K=15) PGD, random(K=30)</td><td>0.7750</td><td>0.0004 0.0006</td><td>0.7730 0.7732</td><td>0.7604 0.7614</td><td>0.9636 0.9644</td><td>3873 3874</td><td>0.40s 0.86s</td></tr><tr><td>PGD, random(K=60)</td><td/><td>0.0004</td><td>0.7734</td><td>0.7618</td><td>0.9649</td><td>3874</td><td>1.74s</td></tr></table>

Peer review (iclr2020):
The paper makes three contributions: 1) analyzing the number of linear regions and the Fourier spectrum of fully-connected neural networks with ReLU activations; 2) proposing an ensemble method as a defense against adversarial evasion attacks; 3) analyzing the performance of that defense against state-of-the-art attacks on CIFAR-10 and ImageNet data. My main concern is regarding the originality and significance of this work. Most of the results regarding the piece-wise linear geometry of fully-connected neural networks with ReLU activations are known. The authors should reference and discuss the relation of their work in particular to [1]. The Fourier transformation results look straightforward to me. I am not convinced either that they are indeed *essential*, as the authors write, for analyzing the nature of adversarial samples. In fact, I find the discussion in Section 2.2 rather vague; for instance, what is the formal notion of *tiny unlearned regions* of neural networks? The authors mention an experiment in which they observed that an adversarial perturbation on ImageNet led to the crossing of N=5278 hyperplanes *per layer*, which sounds like an interesting finding, however, more detail is needed to understand how it was obtained. The proposed defense is not very original either; the authors should discuss it in comparison e.g. to [2] and [3] who also considered ensemble-based methods to defend against adversarial perturbations. The experimental results in Table 1 & 2 look promising at a first glance, however, [4] has demonstrated that ensemble-based, randomized defenses can be fairly easily defeated (e.g. using expectations over transformations when computing the gradients for adversarial sample generation). The authors need to report the performance of their defense under such white-box settings. The strong performance of the FGSM attack in Figure 3 is surprising. PGD is strictly stronger than FGSM, so its defense rate shouldn*t be higher than the one for FGSM. I disagree with the authors* explanation that FGSM *generates much larger perturbations*; if the epsilon parameter is set properly in the PGD attack, it should exhaust that perturbation budget (assuming that the step size and number of iterations are large enough). On the other hand, I don*t understand how the authors were able to constrain the L-inf norm of the C&W and DeepFool adversarial samples (since those attacks aim to minimize L2 norms). [1] Montufar et al., On the Number of Linear Regions of Deep Neural Networks. NeurIPS, 2014. [2] Liu et al., Towards robust neural networks via random self-ensemble. ECCV, 2018. [3] Cao & Gong, Mitigating evasion attacks to deep neural networks via region-based classification. ACSAC, 2017. [4] Athalye et al., Obfuscated gradients give a false sense of security. ICML, 2018.