Crafting Data-free Universal Adversaries with Dilate Loss
Authors: Deepak Babu Sam
ABINAYA K
Sudharsan K A
Venkatesh Babu RADHAKRISHNAN
Venue:  (2020)

Abstract: We introduce a method to create Universal Adversarial Perturbations (UAP) for a given CNN in a data-free manner. Data-free approaches suite scenarios where the original training data is unavailable for crafting adversaries. We show that the adversary generation with full training data can be approximated to a formulation without data. This is realized through a sequential optimization of the adversarial perturbation with the proposed dilate loss. Dilate loss basically maximizes the Euclidean norm of the output before nonlinearity at any layer. By doing so, the perturbation constrains the ReLU activation function at every layer to act roughly linear for data points and thus eliminate the dependency on data for crafting UAPs. Extensive experiments demonstrate that our method not only has theoretical support, but achieves higher fooling rate than the existing data-free work. Furthermore, we evidence improvement in limited data cases as well.

Under review as a conference paper at ICLR 2020
CRAFTING DRTE UNIVERSAL ADVERSARIES WITH DILATE Loss
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a method to create Universal Adversarial Perturbations (UAP) for a given CNN in data-free manner. Data-free approaches suite scenarios where the original training data is unavailable for crafting adversaries. We show that the adversary generation with full training data can be approximated to a formulation without data. This is realized through a sequential optimization of the adversarial perturbation with the proposed dilate loss. Dilate loss basically maximizes the Euclidean norm of the output before nonlinearity any layer. By doing so, the perturbation constrains the ReLU activation function at every layer to act roughly linear for data points and thus eliminate the dependency on data for crafting UAPs. Extensive experiments demonstrate that our method not only has theoretical sup- port, but achieves higher fooling rate than the existing data-free work. Further- more, we evidence improvement in limited data cases as well.
INTRODUCTION
Despite the phenomenal success of deep neural networks in many practical applications, adversarial attacks being constant plague. These attacks corrupt the input with small and usually im- perceptible structured noise causing the model to output incorrect predictions. The sole existence of such vulnerability not only raises concerns about the security of deep learning models, but also questions the robustness of the learned representations. To make it further worse, it has been shown that a single noise, called universal adversarial perturbation (UAP), can be added any image and fool the network. UAPs do not require any optimization on the input image at attack time, but the corruption effectively works for most of the images. Interestingly, such perturbations created for one model exhibit transferability of attack and induce high fooling on other models. One drawback of UAPs though, is the requirement of training data for crafting perturbations. This is increasingly infeasible as the datasets are becoming quite large and might not be publicly released due to privacy or copyright reasons. In such cases where the original data is not available, data-free methods are gaining traction. In the data-free setting, the perturbation is created only with the trained neural net- work. Such methods typically rely on the trained weights and the CNN structure to find vulnerable patterns that can maximally disturb the normal propagation of activations across the network. A higher transfer of attack across networks is observed for data-free UAPs as well, raising its practical utility. Moreover, the study of these perturbations might lead to new insights on how deep neural networks actually work.
In this paper, we propose a new method for crafting data-free UAPs for any given CNN using ReLU nonlinearity. The approach relies on finding the singular vectors of linearly approximated network (Section 3.I). A loss formulation is devised enable this approximation under certain conditions. Dilate loss forms the major component of the method, which generates a perturbation that maximizes the Fuclidean norm of the activation vector (before the nonlinearity) at a given layer (Section 3.2). We show that the perturbation crafted through dilation has the effect of lincarly approximating the ReLU layer responses for any data points. These dilations are done sequentially for all the layers from the input the last classification stage (Section 3.3). We argue that the sequential dilations results in a perturbation that aligns with the first singular vector of the linearly approximated network. Our approach outperforms the existing data-fre free method in fooling rates and the evaluation is also done for less data scenarios (Section
In summary, the work contributes the following:

---

Under review as a conference paper at ICLR 2020
Figure Demonstration of adversarial attack through perturbation generated from our proposed sequential dilation algorithm. The first row shows the clean images with the class predicted by the model, while the second row has the corresponding perturbed samples with the flipped labels.
A new method that can create universal adversarial perturbation without using data and achieve state-of-the-art data-free fooling rates.
A detailed theoretical analysis which formulates the proposed sequential dilation algorithm
by approximating the adversary generation with full training data under certain conditions.
2
RELATED WORK
The vulnerability of deep neural networks to adversarial samples first shown in Szegedy et al.
(2013). Following Szegedy et al. (2013), several methods (Goodtellow et al. 2014 Kurakin et al.
2016 Dong et al. 2018: Madry et al.)
2017
Moosavi-Dezfooll et al.) 2016 Brendel et al 2017
Athalye al. 2018 Carlin & Wagner 2017
being proposed to craft such adversarial samples.
One of the simplest method is the Fast Gradient Sign Method (FGSM) formulated in Goodfellow ct al. 2014). FGSM obtains the perturbation by single step gradient ascent of the loss function with respect to the input image. There are multi step variants to FGSM like iterative FGSM (Kurakin 2016), Momentum (Dong al.) 2018). Projected Gradient Descent (PGD) (Madry et al., 2017 Deepfool (Moosavi-Deztoolii et al., (2016), etc. These attacks are image-specific, where the perturbation is function of the input and requires a separate optimization for each image.
Moosavi-Dezfooli et al. (201)) introduce the idea of Universal Adversarial Perturbations (UAP), a single perturbation that can fool the model for most of the input images. UAP is obtained by jointly maximizing the training loss for dataset images. There are also generative approaches like NAG (Reddy Mopuri et al.) 2018b), AAA (Reddy Mopuri et al. 2018a), GAP (Poursaeed et al. 2018 for crafting universal adversaries. Khrulkov & Oseledets (018) propose method based singular vectors of Jacobian matrix to create universal adversaries. They show impressive fooling performance with very small set of training images, but the method is not data-free. Though the study of adversarial attacks started with the classification task, there are several works (Xie et al., 2017 Metzen et al.) 2017) that extend such attacks other tasks like segmentation, detection, etc. Further, adversarial examples are shown to generalize to the physical world in Kurakin et al. (201)). While most attacks changes each pixel in the image with small imperceptible noise, there methods (Sharif et al., 2016 Brown et al. 2017 Papernot et al. 2016) that perturb limited number of pixels with large noise as these are more practical in nature.
The attacks discussed so far, in general, rely on maximizing the training loss. In contrast, Mopuri al. (2018) devise generalizable data-free objective for crafting UAPs (GDUAP). GDUAP maxi- mizes the activations at the output of all convolutional layers corrupting the feature representations learned and hence fooling the model. Our method has similarity to GDUAP, but with the crucial difference that the Euclidean norm maximization is performed before the nonlinearity in our case. Further, we maximize the norms of the layers one after the other in sequential fashion as opposed to single joint optimization. We show theoretically and experimentally that these changes cause lot of difference in fooling performance. Morcover, no sound reasoning is available in Mopuri et al. 2018) to justify the formulation, whereas we provide theoretical explanation for the algorithm.
2

---

Under review as a conference paper at ICLR 2020
3 OUR APPROACH
3.1
CRAFTING A DATA- FREE ORJECTIVE
Consider deep neural network with L layers already trained for classification task. We assume the activation function employed in the network to be ReLU (Nair& & Hinton 2010), defined as
x
i 0
(
otherwise.
which basically matrix. Let fi (x) = Wi f2( W2za(W11)) ... fi(z)= WioR(... R20 Wi ...) be the outputs at different layers of the network for an input vector æ. Note that the output for the ith layer is taken before the nonlinearity and ft represents the pre-softmax neuron layer. We ignore
out all negative elements and retains the positive ones when applied on
the bias terms for mathematical simplicity. The weights (W,s) of the network are trained with input and label pairs from the dataset ID.
Our aim is to craft a perturbation vector p with Euclidean norm such that the network incorrectly classifies for most of the data samples. Mathematically. the optimization can be written as,
Σ
max p|=c
(xi (ED
where as is the softmax function. Note that the condition for the indicator function (Z) that checks for misclassification is dependent on the ground truth labels (y,s). Assuming a high clas- sification accuracy for the model, we approximate the condition to argmax(os(f2ll)))) (rgmax(os(f((x)))) Since softmax function monotonic, we further relax the objective to
+))) *
max P:PP
Σ (30 JDED
ff p) - fx.))))
which amounts to finding a that maximizes the network response after being added to the inputs. In other words, p should maximally disturb the output Jt. for all data points. Note that for some 8;s, maximizing [ (ar; p) - [x.() might not result in incorrect prediction. We assume such cases to be minority and the objective could lead to significant adversarial changes to ft responses for majority input samples. If X {wo, .i..... NN } denote the matrix formed by the assembling all the data samples as columns and represent a column vector of ones of appropriate size, then the optimization 2 can be rewritten as,
max [x(X -p11 - p: p
- fx())
We recognize that optimization proximation which does not require the ground truth labels. But still the training data X is essential for computation and needs to be eliminated for complete data-free approach. Now observe that if ft were to be a linear function, then the objective objectiveBrredees to,
not exactly equivalent to the original objective 1 but is ap-
max p: pipl
which means that p has to align along the first right singular vector of the linear fr map. The singular could potentially disturb the output fr more for all the æ,s than any other vector. Interestingly. data-free objective under the linear assumption of ft. However, fr. is nonlinear due to the presence of ReLU activation functions at every layer. Note that the formulation 4is valid even if fL not a complete linear map, but satisfies fz(X + p1") (X) + fz (p1") for some p. Hence, we devise an algorithm to seek a perturbation that can approximately induce the above additivity property to the ReL.U network.
note that the optimization
3.2 LINEARLY APPROXIMATING THE NETWORK
We start by noting that the only nonlinearity in the network is due the ReLU activation function at every layer. But ReLU is wiee linear; especially, observe that a æ - b) = dx((a) -)) + (b)) if vectors and are in the same orthant. Now consider the ReL.U nonlinearity after the first layer,
3

---

Under review as a conference paper at ICLR 2020
TR T + Wipl"), which becomes additive if column vectors in WiX are in the same orthant as Wip. We relax this criteria and favour the case of making the vectors as close as possible by,
max Pipl=C
(W1 = N(W (WiP),
(5)
where # stands for the mean of the N data samples X. The solution of the optimization 5 is expected to minimize the enror due to the additive approximation of the layer. In order to eliminate the data term from the objective, we make an assumption that the first singular vector of the weight matrices align along the mean vector of its corresponding input. In other words, the dot product of data mean I₁ with the singular vectors of W1 is maximum for the first singular vector. Now we use the following lemma argue that the objective 5 is maximum when p aligns with the first singular vector of Wi (proof available in Appendix A
Lemma If x has positive and larger scalar projection on the first singular vector of W than remaining singular vectors, then argmax, a W Wp argmax, W Wpp2 subject to |p|
Hence, the optimization problem 5 is equivalent to,
max Wipl? pipp==
which we call the dilation of the first layer. We justify the assumption based on the premise that the singular vectors of the weights must have captured the discriminatory modes of data samples while training. By discriminatory mode we refer to the components of X that are essential for the classification task and most likely extracted by the hierarchy of weights in the network. These does not correspond to the modes of variation of data points. The assumption essentially means that the first singular vector carries the most important features common most of the data points than the remaining singular directions. This taken to be valid for any layer weight Wi with difference that the mean vector 2; is averaged over the layer / output, i.e. Z; (ff))o for / >> 1.
Now consider the second layer of the network given by RR (R((WRRRR (Wix + Wip17)), where there two ReLU functions in action. Suppose the first ReLU function is linearly approximated with dilation objective 6 Consequently, the second layer output can be written as az (2 (
(W, + W2on W Wipl")) ))- Note that the second ReLU can be linearly approximated if column vectors in W ( X) are close to W20 (iPp). Considering the two approximations, we formulate the optimization as,
+1 max 1st 1 W (TWWWW))(WWWW... (W2oR + +1 (W, (W)xX)(((12)) (Wip).
p =C
max (Wa22) (W20 + (W1;))
(8)
p =|=c
Again, we leverage the assumption that the data mean projects more to the first singular vector of the weight matrix and with Lemma the problem becomes the dilation of the second layer,
max [W2) + |
pipp=c
We extend the same arguments to further layers and see that the dilations tends to make the net- work layers approximately additive with respect the generated perturbation vector. For the last layer, the dilation terms are added to objective to account for the errors introduced due to linear approximation of all the ReLU layers. Hence, the final optimization problem for UAP generation becomes,
max | (p) + pip
M(PP)
(10)
which is clearly completely data-free formulation.
3.3 SEQUENTIAL DILATION ALGORITHM
We leverage the theoretical intuitions from the previous Section to formulate algorithm for UAP generation in a data-free manner. Note that the direct implementation of optimization [0] through any gradient descent algorithm would lead to sub-optimal solutions as the chances of getting stuck
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: The sequential dilation algorithm for crafting data-free UAPs. The input is the multi-layer neural network f and the perturbation strength c. set adversarial perturbations {p:) one for each layer, is returned as the output. Note that À is the learning rate.
Po ~ U1 10. 10)
for 1 [1,2.... L) do
Pt Pt-1
while convergence do Pt + Av Ei= log(){((p)))²)
Set po =
end end
in local minimas is high. This is especially true since no data is used and the only variable being optimized is p with sources of randomness. Hence, perform the dilations of optimization T0 in sequential manner so as to avoid chances of reaching local minima solutions. Some more changes applied in the way the original optimization implemented, mainly for training stability and to compare fairly with existing methods. For numerical stability of the optimization, we follow Mopuri al.] (2018) and maximize logarithm of the Euclidean norm in the dilate loss. In order to compare with existing methods, 2x norm is restricted instead of the l2 in the problem [O] This constrains the maximum of absolute value of the adversarial noise.
Algorithm elucidates our proposed sequential dilation algorithm for ReL.U based neural networks. The procedure loops over all the layers the network. For the first layer, we find a vector p1 which maximizes the logarithm of l2 norm of WIPI. essentially finding the first singular vector of W1. After the dilation of the first layer, the perturbation p1 is used as an initialization for maximizing the Euclidean norm of second layer. But note the first loss term Wipp 2 is still kept in the dilation of second layer. This loss formulation tries to maximize the norm output at the current layer along with all the previous layers that feed into it. In short, dilation of ith layer starts the optimization with perturbation obtained from dilation of (1 - 1)th layer and involves the joint dilation of all layers. The method till the softmax layer of the network and the final perturbation Pt is UAP, created without using any training data and could potentially fool majority of input samples.
We only consider CNNs trained for classification task. The optimization is performed using standard ADAM optimizer (Kingma & Ba 2014) with a fixed learning rate schedule till the training loss sat- urates. Typical learning rate 0.1. At every step of the optimization, the values of the perturbation are clipped to limit the allowed range. The loo norm is set as 10 for all our experiments. Although, Fuclidean and maximum norms are not theoretically equivalent, practically we observe that the final perturbations are saturated, with roughly more than 78% of the values reaching +10. This implies the lo norm also to be approximately restricted under the saturation assumption. Once the perturba- tion gets saturated while optimization, the loss might saturate and could be stuck in local minimas.
Roshct-152
Incepllorvv
Inception
Figure 2: Data-free Universal Adversaries crafted using our sequential dilation algorithm.

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison of fooling rates (in percentage) obtained for approach with that of other works. We achieve higher fooling rate than the existing data-free method GDUAP (Mopuri et al., Note that UAP DMoosav-Deezoolli et al. 2017) is data dependent (shown for completeness).
2018
Network
UAP
Random
GDUAP
Ours
VGG16
77.80 80.80
8.20
44.79
51.35 + 1.16
VGG19
8.14
40.51
49.58 + 0.96
ResNet 50
10.01
32.81
60.10
0.63
ResNet 152
84.00
8,46
23.90
47.37 + 1,50
Inception vl
78.50
9.94
17.59
52.02 1.05
Inception v3
7.83
21.74
47.18 = 1.36
To prevent this, after dilation at every layer, we rescale the perturbation by dividing the pixel values by This does not make any difference to the procedure as only the magnitude is changed make room for further optimization. Ideally. we should do sequential dilations for all the convolutional and fully connected layers of the CNN from input side the end softmax classifier. But for very deep models like Inception and ResNet, the dilations are done only for every architectural blocks. Because of this the optimization might be more nonlinear than what being assumed in Section 3.1] with the maximum norm further inducing the clipping nonlinearity. Hence, we initialize the perturbation (po) with the values drawn randomly from uniform distribution U(-10, 10). Finally, note that absolutely no data in any form is used for creating adversarial noise, not a validation set is employed in contrast to Mopuri et al. (2018). The code for our approach is available for review seq dilate.
the anonymous link https::/githubb..o//oooooo
EXPERIMENTS
We benchmark our proposed sequential dilation method with the existing data-free approaches. All the experiments are performed on popular classification models like VGG (Simonyan & Zisserman
2014]. ResNet (He et al., 2016) and Inception (Szegedy et al. 2016)) These models are already 2009) dataset and delivers very high classification accuracy. Figure 2 shows the perturbations crafted using the proposed method for various networks. We follow other works (Moosavi-Dezfoolli et al.) 2017] Mopuri al. (018) and assess the performance of our adversarial attack using the fooling rate metric. Fooling rate is defined as the fraction of test images on which the network prediction differs before and after the addition of the adversarial noise. Table reports the fooling rates obtained by our method along with that of other works. The first comparison with the random baseline, which is the fooling incurred with just random noise. Second baseline is with the only existing data-free approach GDUAP Clearly, our proposed data-free objective achieves significantly higher fooling rates than the other data-free work. This indicates that sequential dilation algorithm, not only has theoretical backing, but also results in higher fooling rates in practice. Note that we have run our method ten times and the results produced in the table are mean fooling rate along with the standard deviation, to statistically validate the performance improvement.
trained on Imagenet (Deng
Now we ablate the different aspects of the sequential dilation algorithm to demonstrate the useful- ness of the design choices. Table 2 reports the results of the various ablative experiments. First
Table 2: Various ablative experiments regarding our approach. It is evident that the proposed quential dilation algorithm has higher performance than any of its variants.
Network
Single Dilation
PSM Maximization
Ours without accumulation
Ours
VGG16
49.46
24.24
31.49
51.35
VGG19
44.10
25.20
31.00
49.58
ResNet 50
25.62
16.57
28.52
60.10
ResNet 152
28.16
15.85
13,40
47.37
Inception vl
24.81
15.57
39.08
52.02
Inception v3
19.40
11.92
26.08
47.18

---

Under review as a conference paper at ICLR 2020
conv1 (29.95%)
conv1 (33.16%)
conv2 (34.37%)
ConV2 (35.30 %)
conv3 (36.11%)
conv3 (38.22%)
(40.62%)
(43.73%)
(46.01%)
conv5_ (48.68%)
(49.5_2
conv5_3 (51.32%)
(52.58%)
Figure 3: Intermediate perturbations (p:s) crafted for different layers using our sequential dilation algorithm for VGG-16 network with corresponding fooling rates obtained.
experiment is non-sequential version of dilation, listed as single dilation in the table. This single joint optimization maximizing the norm of activations before the non-linearity. PSM maxi- mization refers to simple maximization of the pre-softmax layer (fr.) alone, which is same as the approximated objective 4 As described in Section 3.3. our dilation of layer involves keeping the maximization terms all the previous layers. We empirically validate the necessity of such a scheme by just sequentially maximizing the layer norms without the cumulative loss term for the experiment Ours without accumulation in Table Note that each maximization starts with the per- turbation initialized from the previous optimization. The results for these ablations evidence that exact formulation of sequential dilation achieves higher fooling rates in data-free scenario. Further, Figure displays the perturbations obtained through sequential dilation at every layer for VGG-16 network, basically the pıs from Algorithm 1 We also indicate the corresponding fooling rates for each of the perturbation. is interesting observe that the fooling rate increases as we successively dilate layers and saturates towards the end, again emphasizing the need for the sequential process.
In many practical attack scenarios, the actual deployed model might not be available to generate the adversarial perturbation. Hence, the ability of the perturbation crafted for one network to cause reasonable fooling on another network often highly sought-after property. This setting is known as black-box, for which we compare our method with GDUAP (Mopuri et al., 2008)) in Table 3 The results evidence better black-box performance for our method than the existing data-free work, suggesting higher generalization for the perturbations from sequential dilation.
Table 3: Black-box testing of data-free sequential dilation approach. Rows indicate the networks
used craft the adversary and the columns specify the one for which the fooling rate is computed. Blue values are the fooling rates obtained by our method, while the red ones correspond to GDUAP.
Network
VGG16
VGG19
ResNet 50
ResNet 152
Inception
Inception v3
VGG16
52.2
38.5, 45.9
27.7, 37.8
23.9, 32.1
26.3, 34.9
20.1, 26.9
VGG19
47.5
40.5,
24.9,37.5
21.5,
25.7 33.7
18.7, 27.2
ResNet 50
15.2, 23.6
14.8.23.1
32,8,
17.9, 35.0
18.9,33.3
13.7, 21.6
ResNet 152
16.6, 27.1
15.8, 26.3
21.7,
23.9, 46.7
18.7,77.5
13.7, 18.5
Inception vl
14.4, 26.1
13.9, 26.5
16.4, 34.3
14.1, 27.2
17.6,
12.1,
Inception v3
11.8, 16.5
11.8, 16.3
13.2, 18.0
11.8, 14.6
13.9, 16.5
21.7,459

---

Under review as a conference paper at ICLR 2020
Table 4: Comparison of fooling rates obtained by method and GDUAP (Mouuri al. less data setting with varied number of images available for crafting the perturbation.
2018 in
GDUAP
Ours
Network
D=64
500
D=1000
D= 10000
D=64
D = 500
D=1000
D=10000
VGG16
71,25
75.16
76.20
75.62
74.25
77.66
78.12
79.09
VGG19
68.65
73.11
74.19
74.73
66.81
72.34
74.00
74.28
ResNet 50
57.94
71.98
72.71
71.64
70.33
74.52
74.54
76.36
ResNet 152
43,95
67.43
69.34
70.48
68.07
70.57
71.52
71.64
Inception v1
15.83
85.99
86.41
87.10
60.26
64.43
69.68
71.78
Inception v3
27.13
53.84
55.04
59.88
81.54
86.69
87.21
87.57
Benchmarking our approach in less data scenario with Singular Fool (Khrulkov & Os- eledets 2018) where the adversaries are crafted with just 64 images.
Table 5:
Ours
Singular Fool
Network
Imagenet
Pascal
Khrulkov & Oseledets 2018
VGG16
72.13
76.15
52
VGG19
66.56
68.87
60
ResNet 50
68.73
65.69
44
ResNet 152
66.84
66.47
Inception vl
46.74
40.86
Inception v3
76.42
76.70
The experiments performed so far show that proposed sequential dilate loss formulation achieves state-of-the-art fooling rates in data-free scenarios. We now consider the case where minimal train- ing data is available, called the less data setting. For this case, the sequential dilation is applied with the limited data. The input to the network any stage of the optimization is the image added expect with the current perturbation (x, pi for layer 1). With the help of some data points, the solution to approach more closer to the actual adversarial perturbation obtained with full data. Table 4 indicates the fooling rates of the less data setting with varied amount of training samples. Note that to compare with GDUAP, we also use a validation set to select the best perturbation while training. Our approach performs significantly better than GDUAP when data samples very less, increasing the practical utility of the method. We also observe that the fooling rates with less data, in general, have increased than data-free and became comparable full data UAP (see Table TT.
Furthermore, Table c compares our approach with Singular Fool Khrulkov & Oseledets 2018) in extremely less data scenario. For fair comparison with Khrulkov & Oseledets (2018). we use only 64 images for crafting the perturbation and no validation set is employed. The best perturbation is selected based on the training loss. As expected, our method achieves significantly higher fooling performance than Khrulkov & Oseledets (2018). Even more, we apply our algorithm with 64 ran- domly chosen images from Pascal VOC (Everingham et al. 201I). Interestingly, despite the models used are being trained for a different dataset, the fooling rates remain more less similar and is higher than that of Khrulkov & Oseledets (2018). This shows that our approach works well less data cases even when the available images not from the dataset which the model is trained.
CONCLUSIONS AND FUTURE WORK
In this paper, we have presented new algorithm, called the sequential dilation, to craft universal adversaries in data-free manner. The approach relies on finding the first singular vector of the linearly approximated neural network. The approximation is being enabled by optimizing with the proposed dilate loss. Elaborate experiments and ablations demonstrate that approach achieves superior data-free fooling performance. One promising direction for future research would be to modify the algorithm and generate targeted UAP, where the objective is fooling specific class.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of se- curity: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283, 2018.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-basee adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04225: 2017
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint 1712.0697 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition, pp. 9185-9193, 2018.
M Everingham, Van Gool, CKI Williams, J Winn, and A Zisserman. The pascal visual ob- ject classes challenge 2012 (voc2012) results (2012). In URL htp:://ww.. pascal-network, org/challenge//VCcvec211 I/workshop/index. html, 2011.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pailern recognition, pp- 770-778, 2016.
Valentin Khrulkov and Ivan Oseledets. Art of singular vectors and universal adversarial perturba- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3562-8570, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 2016.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706.000 2017.
Jan Hendrik Metzen, Mummadi Chaithanya Kumar, Thomas Brox, and Volker Fischer. Univer- sal adversarial perturbations against semantic image segmentation. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2774-2783. IEEE, 2017.
Seyed-Mohsen Moosavi-Dezfore Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puler vision and pattern recognition, pp. 2574-2582, 2016.
Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 1773, 2017.
Konda Reddy Mopuri. Aditya Ganeshan, and Venkatesh Babu Radhakrishnan. Generalizable data- free objective for crafting universal adversarial perturbations. IEEE transactions on pattern anal- ysis and machine intelligence, 2018

---

Under review as a conference paper at ICLR 2020
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations deep learning in adversarial settings. In 2016 IEEE European Sympo- sium on Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016.
Omid Poursaced, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial pertur- bations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4422-4431, 2018.
Konda Reddy Mopuri, Phani Krishna Uppala, and R Venkatesh Babu. Ask, acquire, and attack: Data-free uap generation using class impressions. Proceedings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a.
Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R Venkatesh Babu. Nag: Network for ad- versary generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 742-751, 2018b.
N. Silberman S. Guadarrama.
a lightweight library for defining, training and evaluating complex models 2016.
Tensorflow-slim.
tensorflow.
https///gihub..om//ess///// wwwp/e/ewsffloeeeeeoooooeeeeeeee
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to crime:
Real and stealthy attacks on state-of-the-art face recognition. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1528-1540) ACM, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818 2818-2826, 2016.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xic, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 1369-1378, 2017.
A PROOF OF LEMMA 1
If = USVT denote the singular value decomposition of W, then
max x' W''Wp' x VS Σ (a'
(11)
p: ppl
Since x V.V. x V., (by assumption) and Sn > Sy for all > 1, the solution to the optimization is cV. 1, scaled version of the first singular vector of W. The same solution can be obtained through the definition of the first singular vector as,
max Wpl p: =|
(12)
completing the proof.
B EXPERIMENTAL SETUP
All our experiments reported in the main paper NVIDIA DGX cluster (Dual 20-core Intel Xeon E5-2698 v4 2.2 GHz) within the Tensorflow docker. We use the pretrained classification models from Tensorflow Slim library S. Guadarrama 2016).
10

---

Under review as a conference paper at ICLR 2020
C DEMONSTRATION OF ADVERSARIAL ATTACK
Figures to demonstrate adversarial attack through perturbation generated from our proposed sequential dilation algorithm for various networks. The first row in the figures shows the clean images with the class predicted by the model, while the second row has the corresponding perturbed samples with the flipped prediction labels.
SON
Chocolate syrup
Joystick
Camura
Cup
Pinwoppe
RON
Lampohade
Cirg
Pajarn
Apron
Mask
Figure 4: Attack VGG-16 network.
Hammiiggbid
eagle
Magnetic compass
Figure 5: Attack on VGG-19 network.
Butterfhy
Cheetah
Abocus
Torch
Seer Bottle
Paper towel
Mitten
Wine bottle
Figure 6: Attack on ResNer-50 network.
11

---

Under review as a conference paper at ICLR 2020
Stole
Rifc
Grile
Balloon
Ping pong ball
Figure 7: Attack on ResNet- 152 network.
Brsincoral
Veetment
Harmer
Terrain
Rosehip
Figure Attack on Inception-v network.
Domselfy
Torch
Purse
Zebra
Chain
Fiowerpot
Figure 9: Attack on Inception-v3 network.
12


Reviews:
[{"review_id": "r1xcIWR35r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposed a method to generate universal adversarial perturbations without training data. This task is timely and practical. The proposed method maximizes the norm of the output before nonlinearity at any layer to craft the universal perturbation. A sequential dilation algorithm is designed to calculate UAPs. The experiments show that the proposed method outperforms GDUAP.\n\nMy major concern is that there is not much novelty in the proposed method compared with GDUAP. The dilate loss function (4) is similar to the objective function (3) in the GDUAP paper. This paper provides a theoretical explanation of the dilate loss function and an improvement on the non-linearity function, which, however, is not convincing. Equation 10 is derived based on many strong assumptions. See the comments below.\n\nPros:\n-\tThe theoretical analysis is clear.\n-\tThe proposed method performs better than GDUAP in the data-free and black-box setting.\n-\tThe writing is good. The paper is easy to follow.\n\nCons:\n-\tThe theoretical analysis is based on many strong assumptions/criteria. For example:\no\tTo derive equation (5), W1X and W1p must be in the same orthant. It is unclear how to satisfy the criteria In the algorithm. \no\tIn Lemma 1, problem (5) approximates problem (6) only if x has a very large projection on the first singular vector of W. However, x and W are fixed and independent of p. This assumption largely depends on the dataset and the weights of the model.\no\tIt would be better if the authors show that in what cases these assumptions can be satisfied.\n-\tOther factors such as batch normalization and max pooling used in Inception v3, may also affect the linearity of the model. It would be better if the authors provide theoretical analysis or an ablation study on these factors. \n-\tWhat’s the design principle behind Algorithm 1? Why can this algorithm solve the sub-optimal problem? The weights of different layers are not closely related. In the initialization part, why can we start learning p from the result of the previous layer? Would it be possible that the performance is improved due to the algorithm instead of the dilate loss?\n-\tThe proposed method performs worse than GDUAP does in some less data settings.\n-\tThe results in Table 4 and 5 are inconsistent. These two experiments use the same dataset (Imagenet) and the same number of images (D=64).\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1l5NeGTFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and easy to follow. In this paper, a new data-free method is proposed to create universal adversarial perturbation without using data. There are some similarities with GDUAP though, authors also make some crucial improvements. They perform Euclidean norm maximization before the non-linearity of each layer, which not only has theoretical backing but also brings better performance in practice. Meanwhile, they optimize the perturbations in each layer in a sequential manner instead of joint optimization to avoid chances of reaching local minima solutions.\n\nThe authors provide a detailed theoretical analysis and systematic experimental results to demonstrate their arguments, which is convincing. What’s more, the proposed method achieves state-of-the-art data-free fooling rates on the large-scale dataset, which strongly demonstrates the effectiveness of their method.\n\nIn section 3.2, (the top of page 4) “which becomes additive if column vectors in W1X are in the same orthant as W1p. We relax this criteria and favour the case of making the vectors as close as possible by”\nCould the authors provide more discussions about it?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeI1QS99B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a data free method for generating universal adversarial examples. Their method finds an input that maximizes the output of each layer by maximizing the dilation loss. They gave a well motivated derivation going from the data matrix, the data mean and to data free. The experiments results seems solid as the numbers show that their method is much better in many cases.\n\nI have 2 main issues:\n\n* The fooling rate experiments does not seem to control for how much distortion there really is. How do you make sure that different methods have similar level of distortion and not just similar l_\\inf.  Given that the authors says most of their method saturates all values, it is not clear that the baselines and competition really has a similar level of distortion. The fooling rate for random seems rather high. Why is random noise not mostly ignored by the model?\n\n* while the method is data free. It needs complete access to the model and relies on properties of ReLu. I am not sure how realistic this setting is, and how this compares to methods that has black box access to the model. While it is interesting, the paper did not establish that universal adversarial perturbation is well-motivated and why data free is more important that model free or targeted perturbations.  An attacker probably always see the input and probably wants to make it misclassified into a particular class, instead of just making the model wrong. \n\n ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BJxB-51Rtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a white-box (known network architecture, known network weight) data free (without need to access the data) adversarial attacking method. The main idea is to find a perturbation that maximizes the activations at different layers jointly. But the optimization is done sequentially, treating each layer’s activation (before ReLU) as a linear transformation output.\n\nThe method is compared with existing methods (only one existing approach for the problem, GDUAP by Mopuri et al. 2018) in terms of the fool rate. It shows significant improvement. Ablation study is carried out to compare with baselines like perturbation maximizing only first layer activation, only last layer activation, etc. Also on some other settings (black-box testing, less data) the proposed method outperforms GDUAP. \n\nThe problem of data-free white-box attack is very interesting and does make sense. The proposed method achieve significant improvement over the previous one (GDUAP). I do have the following concerns though.\n\n1), the novelty of the proposed idea seems relatively limited. The proposed idea seeks perturbation maximizing activations over all layers. It incur perturbation before ReLU. But overall, the flavor of the idea is not significantly different from GDUAP, despite the significant performance boost. \n\n2), it was mentioned that compare with GDUAP, this paper has more theoretical analysis. But this is not very convincing to me. There are many steps of approximation/relaxation from the original problem (Equation (1)) to the final formula (Equation (10)). Many assumptions are made over the steps. It is OK to use these steps to derive a heuristic. But these steps can hardly be called \"theoretical analysis\".\n\nI am particularly uncomfortable with Equation (5), which is the basis of the main idea. It assumes that all data in $W_1X$ are in the same orthant as $W_1p$. But this is unrealistic as different data in X will for sure incur different activation patterns. Did I misunderstand anything?\n\n3) I do like the experimental results. It looks impressive. But the baselines are really limited (granted, there are not many existing approaches). There is only one task (image classification). How about other tasks like segmentation etc shown in Mopuri et al. 2018? Also it would be nice to also show the results of other UAP methods, as it gives us a better sense of the gap between with and without data.\n\n4) I wonder how will the attack affect some model which has been trained with some defense mechanism, e.g., adversarial training.\n\nTypo:\nEquation (5), RHS missing a max\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on finding universal adversarial perturbations, that is, a single noise pattern that can be applied to any input to fool the network in many cases. Further more, it focuses on the data-free setting, where such a perturbation is found without having access to data (images) from the distribution that train- and test data comes from. 

The reviewers were very conflicted about this paper. Among others, the strong experimental results and the clarity of writing and analysis were praised. However, there was also criticism of the amount of novelty compared to GDUAP, on the strong assumptions needed (potentially limiting the applicability), and on some weakness in the theoretical analysis. 

In the end, the paper seems in current form not convincing enough for me to recommend acceptance for ICLR.  
Decision: Reject