Adversarial Reprogramming Revisited
Authors: Matthias Englert
Ranko Lazic
Venue: NeurIPS 2022 Accept (2022)
Keywords: adversarial reprogramming
adversarial examples
adversarial robustness
random networks
implicit bias

Abstract: Adversarial reprogramming, introduced by Elsayed, Goodfellow, and Sohl-Dickstein, seeks to repurpose a neural network to perform a different task, by manipulating its input without modifying its weights.  We prove that two-layer ReLU neural networks with random weights can be adversarially reprogrammed to achieve arbitrarily high accuracy on Bernoulli data models over hypercube vertices, provided the network width is no greater than its input dimension.  We also substantially strengthen a recent result of Phuong and Lampert on directional convergence of gradient flow, and obtain as a corollary that training two-layer ReLU neural networks on orthogonally separable datasets can cause their adversarial reprogramming to fail.  We support these theoretical results by experiments that demonstrate that, as long as batch normalisation layers are suitably initialised, even untrained networks with random weights are susceptible to adversarial reprogramming.  This is in contrast to observations in several recent works that suggested that adversarial reprogramming is not possible for untrained networks to any degree of reliability.

Adversarial Reprogramming Revisited
Matthias Englert" University of Warwick
Ranko Lazić"
University of Warwick
englert@warwick.ac ac.ukk
lazic@aarwiicc ck .u.uk
Abstract
Adversarial reprogramming, introduced by Elsayed, Goodfellow, and Sohl- Dickstein, seeks to repurpose a neural network to perform a different task, by manipulating its input without modifying its weights. We prove that two-layer ReLU neural networks with random weights can be adversarially reprogrammed to achieve arbitrarily high accuracy provided the network width is no greater than its input dimension. We also substan- tially strengthen a recent result of Phuong and Lampert directional convergence gradient flow, and obtain as a corollary that training two-layer ReLU neural networks orthogonally separable datasets ming to fail. We support these theoretical results by experiments that demonstrate that, as long as batch normalisation layers suitably initialised, even untrained networks with random weights are susceptible to adversarial reprogramming. This is in contrast observations in several recent works that suggested that adversarial reprogramming is not possible for untrained networks to any degree of reliability.
Bernoulli data models over hypercube vertices,
their adversarial reprogram-
Introduction
Elsayed, Goodfellow, and Sohl-Dickstein [2019] proposed adversarial reprogramming: given neural network N which performs task F' X Y and given an adversarial task G repurpose N to perform by finding mappings hin X + and host : Y + Y such that G as hout o Fohin. They focused on a setting where F and G are image classification tasks, X consists simple and computationally inexpensive: hin (z) = + I draws the input ë at the centre of the adversarial program and hout is hard coded mapping from the class labels Y to the class labels Y. Then the challenge of adversarial reprogramming find p such that hout a hin approximates well the adversarial task G.
Y.
of large images, X consists of small images, and his and hout
Remarkably, Elsayed et al. [2019] showed that a range of realistic neural networks can be adversarially reprogrammed to perform successfully several tasks. They considered six architectures trained on ImageNet [Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg, and Fei-Fei, 2015] and found adversarial programs that achieve very good accuracies on counting task, MNIST [LeCun, Bottou, Bengio, and Haffner, 1998], and CIFAR-10 10 [Krizhevsky, 2009]. In addition the basic setting, they investigated limiting the visibility of the adversarial program by restricting its size or scale, or even concealing both the input and the adversarial program by hiding them within normal image from ImageNet, and obtained good results.
Adversarial reprogramming can be seen as taking the crafting of adversarial examples [Biggio, Corona, Maiorca, Nelson, Šrndić, Laskov, Giacinto, and Roli, 2013, Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus, 2014] to next level. Like the universal adversarial perturbation of Moosavi-Deziee Fawzi, Fawzi, and Frossard [2017]. a single adversarial program combined with every input from the adversarial task, but in contrast the former where the goal is to cause
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

natural images to be misclassified with high probability, adversarial reprogramming seeks a high accuracy of classification for the given adversarial task.
Adversarial reprogramming is also related to transfer learning [Raina, Battle, Lee, Packer, and Ng, 2007, Mesnil, Dauphin, Glorot, Rifai, Bengio, Goodfellow, Lavoie, Muller, Desjardins, Warde-Farley, Vincent, Courville, and Bergstra, 2011]. however with important differences. Whereas transfer learning methods use the knowledge obtained from one task as a base for learning to perform another, and allow model parameters to be changed for the new task, in adversarial reprogramming the new task may bear no similarity with the old, and the model cannot be altered but has be manipulated through its input.
The latter features are suggestive of potential nefarious uses of adversarial reprogramming, and Elssyed et al. [2019] list several, such as repurposing computational resources to perform a task which violates the ethics code of system providers. However, its further explorations have demonstrated utility for virtuous deployments to medical image classification [Tsai, Chen, and Ho, 2020), natural language processing [[ambardzumyyn, Khachatrian, and May, 2021, Neekhara, Hussain, Dubnov, and Koushanfar, 2019, Neekhara, Hussain, Du, Dubnov, Koushanfar, and McAuley, 2022], molecule toxicity prediction [Vinod, Chen, and Das, 2020], and time series classification [Yang, Tsai, and Chen, 2021J. In the last three works, adversarial reprogramming was achieved between different domains, e.g. repurposing neural networks trained on ImageNet to perform classification of DNA sequences, and of natural language sentiments and topics.
In spite of the variety of fruitful applications, it is still largely mystery when adversarial reprogram- ming is possible and why. In the only work in this direction, Zheng, Feng, Xia, Jiang, Demontis, Pintor, Biggio, and Roli [2021] proposed the alignment ||9]l/ Σ; I of the gradients 9: of the inputs from the adversarial task with their average g as the main indication of whether adversarial reprogramming will succeed. However, their experiments did not show a significant correlation between accuracy after reprogramming and the gradient alignment before reprogramming, The correlation was statistically significant with the gradient alignment after reprogramming, but that is unsurprising and it is unclear how to use it for predicting success.
Specifically, central question on adversarial reprogramming is:
Can neural networks with random weights be adversarially reprogrammed, and more generally, how does training impact adversarial reprogrammability?
First, addressing this question is important for assessing scope of two claims made in the literature:
"[Adversarial] reprogramming usually fails when applied to unirained networks" [Zheng et al., 2021]. In addition to networks trained on ImageNet, Elsayed et al. [2019] experimented using the same architectures untrained, i.c. with random weights, and obtained generally poor results. Similarly, the experimental results of Neckhara et al. [2019] and Zheng et al. [2021] for random networks are significantly worse than their experimental results for trained networks. However, they remarked that this was surprising given the known richness of random networks (see e.g. He, Wang, and Hopcroft [2016c], Lee, Bahri, Novak, Schoenholz, Pennington, and Sohl-Dickstein [201 8)), and that it was possibly due to simple reasons such as poor scaling of the random weights.
"The original task the neural networks perform is important for adversarial reprogramming" [EI- sayed et al., 2019]. Nevertheless a number experimental results including those of Tsai et al. [2020], Neekhara et al. [2022], Vinod et al. [2020]. Yang et al. [2021]] Zheng et al. [2021] demonstrated successful adversarial reprogramming between tasks that are seemingly unrelated (e.g. repurposing networks trained on ImageNet to act as HCL2000 IZhang, Guo, Chen, and Li, 2009] classifiers) or
from different domains, so the interaction between the original and adversarial tasks remains unclear.
Second, the question above is important because of considerations:
implications for the following two applied
Disentangling architecture and training as factors in adversarial reprogrammability Clarify- ing the respective bearings on adversarial reprogramming success of the network architecture and of the task (if any) it was trained for would improve decision making, either to maximise the success in beneficial scenarios or to minimise it in detrimental ones.

---

Managing the cost of adversarial reprogramming. Understanding when training the network is not essential, and when training for longer does not help or even hinders adversarial reprogrammability, should make it possible to control better the economic and environmental costs.
1.1 Our contributions
We initiate theoretical study of adversarial reprogramming. In it we focus on two-layer neural networks with ReLU activation, and on adversarial tasks given by the Bernoulli distributions of Schmidt, Santurkar, Tsipras, Talwar, and Madry [2018] over hypercube vertices. The latter are binary classification data models in which the two classes are represented by opposite hypercube vertices, and when sampling a data point for a given class, we flip each coordinate of the corresponding class vertex with a certain probability. This probability is a parameter that controls the difficulty of the classification task. These data models are inspired by the MNIST dataset because MNIST images are close to binary (many pixels are almost fully black or white). The remaining parameters are the radius of the hypercube, the input dimension of the neural network, and its width.
We prove that, in this setting, for networks with random weights, adversarial programs exist that achieve arbitrarily high accuracy on the Bernoulli adversarial tasks. This holds for wide variety of parameter regimes, provided the network width is no greater than its input dimension. The adversarial programs we construct depend on the weights of the network and the class vertices (i.e. the direction) of the Bernoulli data model, and their Euclidean length is likely to be close to the square root of the input dimension. We present these results in Section 2.
We also prove that, in the same setting, training the network on orthogonally separable datasets can cause adversarial reprogramming to fail. Phuong and Lampert [2021] recently showed that, under several assumptions, training a two-layer ReLU network on such datasets by gradient flow makes it converge to linear combination of two maximun-margin neurons; and subsequently Wang and Pilanci [2022] obtained in a different manner the same conclusion under the same assumptions. We provide a simpler proof of significantly stronger result: we show that the assumptions in Phuong and Lampert [2021] and Wang and Pilanci [2022] of small initialisation, and of positive and negative support examples spanning the whole space, are not needed; and we generalise to the exponential loss function as well as the logistic one. We then observe that, for any Bernoulli data model whose direction in half-space of the difference of the maximum-margin neurons, and for any adversarial program, the accuracy tends to 1/2 (i.e. approaches guessing) under a mild assumption on the growth rate of the difficulty of the data model. We present these results in Section 3.
Both for the neworks with random weights and for the networks trained to infinity on orthogonally separable datasets, we then show that similar theoretical results can be obtained with different kind of adversarial task, namely those given by the Gaussian distributions also of Schmidt et al. [2018) 1). The latter are mixtures of two spherical multivariate Gaussians, one per data class. Please the appendix.
In the experimental part of our work, we demonstrate that, as long as batch normalisation layers are suitably initialised, even untrained networks with random weights are susceptible to adversarial reprogramming. Both the random weights and the batch normalisation layers are kept fixed throughout the finding of adversarial programs and their evaluation. Our experiments are conducted with six realistic network architectures and MNIST as the adversarial task. We investigate two different schemes to combine input images with adversarial programs: replacing the centre of the program by the image as was done by Elsayed al. [2019], and scaling the image the size of the program and then taking a convex combination of the two. Each of the two schemes has a ratio parameter, and we explore their different values. We find that the second scheme gives better results in our experiments, and that for some choices of the ratio parameter, the accuracies on the test set across all six architectures are not far below what Elsayed et al. [2019] reported for networks trained on ImageNet. Please see Section 4.
We conduct the same experiments also on the more challenging Fashion-MNISS [Xiao, Rasul, and Vollgraf, 2017] and Kuzushiji-Me -NIST [Clanuwat, Bober-Irizar, Kitamoto, Lamb, Yamamoto, and Ha, 2018] datasets, and obtain broadly similar results, however with lower test accuracies in several cases; they are reported in the appendix.
For further discussion of relations with other works, please see the appendix.
3

---

We conclude the paper in Section 5, where we discuss limitations of our work and suggest directions for future work.
2 Random networks
Basic notations. We write: [x]] for the set (1.....72 n)), || for the Euclidean length of vector v, <(v, v') for the angle between v and v', and HH for the d-dimensional unit hypercube { {=1/vd]
Two-layer ReLU networks. We consider two-layer neural networks N with the ReLU activation. We write d for the input dimension, & for the width, Wi..... We E R for the weights of the first layer, and a1...... ak E IR for the weights of the second layer. For an input R the output is thus
((x) Σ a
where (u) = max 0} is the ReLU function.
Random weights. In this section, we assume that the weights in A are random as follows:
each w consists of d independent centred Gaussians with variance 1/d, and
each aj independently uniformly distributed in (+1/VR].
This distribution is as in Bubeck, Cherapanamjeri, Gidel, and Tachet des Combes [2021], I, and standard for theoretical investigations. It is similar He's initialisation (He, Zhang, Ren, and Sun, 2015], with the second layer discretised for simplicity.
The variances of the weights are such that, for any input æ R2 Euclidean length vd, each w) æ a standard Gaussian, and for large widths the distribution of N(x) close to centred Gaussian with variance 1/2.
Bernoulli data models. Adapting from Schmidt et al. [2018], given a hypercube vertex & E H a radius p 0, and class bias parameter 0 < 1/2, define the (Φ, p, (6..,.))-Bemoulll -Beroulll distribution over (rr, y) E pll } as follows:
first draw the label y uniformly at random from (+1),
then sample the data point æ by taking ypо and flipping the sign of each coordinate independently with probability 1/2 1/2-T.
These binary classification data models on hypercube vertices are inspired by the MNIST dataset [Le- Cun et al., 1998]. The class bias parameter controls the difficulty of the classification task, which increases as tends to zero, i.c. as 1/T tends to infinity.
In this section and the next, we consider adversarial tasks that are ( p. r) -ermoullli data models, and investigate variations of the parameters Φ, p and т.
Adversarial program. In this section, we assume that & S d, i.c. the network width no greater than the input dimension, and define adversarial program p which depends the weights of the network A and on the direction & of the Bernoulli data model.
With probability 1, for all E [k], we have that a. Φ * 0. Let write K+ for {j E [k]
a Φ > 0}. and K- for {j E [A] aw Φ < 0). Then, for [k], let:
p = p d
if K
Since & < d, with probability the weights vectors w linearly independent, i.e. the k d matrix W whose w has positive smallest singular value 8miu (W). Hence p Rd exists such that
lp'' Smax W)
lpp'
p' = Wp and
5 |ppl
(1)
8 8min WW)

---

The neurons in K can be thought of as "helpful" for the adversarial task, and those in K as "unhelpful". This definition of an adversarial program p ensures that its effect to introduce as first-layer biases the entries of the vector p'. They are 0 (i.e. do nothing) for every "helpful" neuron, and the negative value - /d K- (i.e. reduce the contribution to the network output) for every "unhelpful" neuron.
From the inequalities in (1), we can expect ||p|| vd if k: o(d (see the appendix for details).
Expected reprogramming accuracy. The accuracy for an adversarial task D is the probability that the sign of the output of the reprogrammed network conforms to the input label, i.e.
IP ((aw)-D(yNV(p + x) > 0}
Our main result in this section that, for sufficiently large input dimensions and under mild restrictions on the growth rates of the network width k, the radius and the difficulty 1/r of the Bernoulli data model, in expectation over the random network weights, the reprogramming accuracy is at least (1 C1 - )"), which by tuning the probability parameters ?y and 7 can be arbitrarily close to 100%. The growth rate restrictions indicate that networks with smaller widths can be reprogrammed for tasks with larger radii, but that networks with larger widths can be reprogrammed for tasks that are more difficult. The proof proceeds by analysing concentrations of the underlying distributions and involves establishing theorem that bounds the reprogrammed network output; the details can be found in the appendix. Corollary 1. Suppose that
k = (dd))),,
p= Odd"oo), 1 (ddit)) and 1/r - wa(1),
where 1() M(o) J(r
[0, 1] are arbitrary constants that satisfy
The) < - 72)/2 and M[()) < 72/0/2
Then, for sufficiently large input dimensions d, the expected accuracy of the adversarially repro- grammed network N on the ( --eernoulll data model is arbitrarily close to 100%
3 Implicit bias
Orthogonally separable dataset. In this section, consider training the network on binary classification dataset = y1),..., ( yn)}
C IRd x [+1} which is orthogonally separa-
ble [Phuong and Lampert, 2021], ], i.e. for all 1' E [n] have:
x.x;; x. > 0 if 1/4 ye and x. TT. S if wr.
In other words, every data point act as a linear separator, although some data points from the opposite class may be exactly orthogonal to it.
Gradient flow with exponential or logistic loss. For two-layer ReLU networks with input dimen- sion and width as before (but without the assumption & S d, which is not needed in this section), we denote the vector of all weights by
0 (W1,..., Wk, 01, ag) kdd+1)
and we write No for the network whose weights are the coordinates of the vector 0.
The empirical loss of No on S is (()) Eln lexp (u) -u or the logistic Liog (u) In(1 + loss function.
where e is either the exponential
A trajectory of gradient flow is function O()) [0, oc) k(d+1) that is an arc, i.e. it is absolutely continuous on every compact subinterval, and that satisfies the differential inclusion
d0 dt
aC(((() for almost all 1. E [0, 20),
where OL denotes the Clarke subdifferential [Clarke, 1975] of the locally Lipschitz function L.
Gradient flow gradient descent with infinitesimal step size. We work with the Clarke subdifferential in order to handle the mon-differentiabiliyy of the ReLU function at zero: ayb(O) is the whole interval [0, 1]. At points of continuous differentiabilityy the Clarke subdifferential amounts to the gradient, e.g. aC(0)= (VL(O)) For some further background, the appendix.
5

---

Initialisation of network weights. In this section, we assume that the initialisation is
balanced: for all j [k], at time t = 0 we have |ayl = ||wy| 0; and
live: for both signs E } there exist i, E [n]] and ja E [k] such that y;, and at time / = 0
we have yi, "j. (u0). s...) > 0.
The balanced assumption has featured in previous work (see e.g. Phuong and Lampert [2021], Lyu, Li, Wang, and Arora [2021]). It ensures that it remains to hold throughout the training, and that the signs of the second-layer weights aj do not change (see the proof of Theorem 2). The live assumption (present probabilistically in Phuong and Lampert [2021]) is mild: it states that at least one positively initialised neuron is active for at least one positive input, and the same for negative ones.
Convergence of gradient flow. Our main result in this section establishes that the early phase of training necessarily reaches point where the empirical loss is less than E(O), which implies that then every input is classified correctly by the network. Perhaps surprisingly. small initialisation assumption is needed, however the proof makes extensive use of orthogonal separability of the dataset (see the appendix, which contains all proof' for this section).
Theorem 2. There exists a time lo such that L(((t))) < ((0).
This enables us to apply to the late phase recent results of Lyu and Li [2020], Ji and Telgarsky [2020], Lyu al. [2021] and obtain the next corollary, which is significantly stronger than the main result of Phuong and Lampert [2021], extending it to exponential loss, and showing that assumptions of small initialisation, and of positive and negative support examples spanning the whole space, not needed. The corollary establishes that each neuron converges to one of three types: scaling of the maximum-margin vector for the positive data class, a scaling of the maximum-margin vector for the negative data class, or zero. The two maximum-margin vectors are defined as follows: for both signs E [+1), let Ig- = {i E [n]
yi = s). and let v, be the unique minimiser of the quadratic problem
minimise
b ||a|
subject to Vi I. : vZ2
That a trajectory (t) converges in direction to a vector 0 means li O(t) 0 0/|
Corollary 3. As the time tends to infinity, we have that the empirical loss converges to zero, the Euclidean norm of the weights converges to infinity, and the weights converge in direction to some such that for all j E [k] we have lay] = =|w,]], and if a + 0 then
W
Vegu(ay)
sgn(aa)) (=pp((s) az
Thanks to homogeneity, the sign of the network output does not depend on the norm of the weights, only on their direction. Examining networks whose weights are directional limits as in Corollary is therefore informative of consequences for adversarial reprogramming of long training. The following result tells us that, for any Bernoulli data model whose direction is in a half-space of the difference of the maximum-marna margin vectors, and for any adversarial program, the accuracy tends to 1/2 provided that the difficulty 1/r of the data model increases slower than the square root of the input dimension d. The latter assumption is considerably weaker than in the results of Schmidt et al. [2018] on the Bernoulli data model, where the bound is in terms of the fourth root. The statement also tells us that this failure cannot be fixed by choosing in advance a different mapping from the class labels of the original task to the class labels of the adversarial task. Since we consider binary classification tasks here. the mapping can be represented by a multiplier m E {+1}.
Proposition 4. Suppose network weights are is in Corollary 3, class label mapping m E {+1} is arbitrary, data model D is any (d,, p. (T)-Benooull distribution such that m cos L(v) U- Φ) and adversarial program is arbitrary. Then we have that
P ((an)-Dmmy
0}
2dr cos² 2(01-0-1:0)

---

4 Experiments
Network architectures and initialisation. Our experiments are conducted using the following six network architectures: ResNet-50 [He, Zhang, Ren, and Sun, 2016a], ResNet-5002, ResNet- 1iie-0111 101 ResNet- 152V2 [He, Zhang, Ren, and Sun, 2016b], Inception-v3 [Szegedy, Vanhoucke, loffe, Shlens, and Wojna, 2016], and EfficientNet-BO [Tan and Le, 2019].
We use the networks exactly as implemented in Keras in TensorFlow 2 including the method for randomly initialising the trainable weights. For biases, this means they are initialised with O. For all other trainable weights, mostly, the Glorot uniform initialiser [Glorot and Bengio, 2010] is used in this implementation. EfficientNet is exception, where many layers instead use truncated normal distribution that has mean 0 and standard deviation / 2/number of output units.
All the networks we experiment with (ResNet-50, ResVet-50V2, ResNet- 101V2, ResNet- 152V2, Inception-vã. v3, and FfficientNet-BO) involve batch normalisation layers [loffe and Szegedy, 2015]. Every such layer maintains a moving mean and moving variance based on batches has seen during training. The inputs are then normalised accordingly: they are shifted by the recorded mean and scaled by the inverse of the recorded variance. Note that the moving mean and variance values are not trainable, i.e., they are not subject to updates by the optimiser during training. During inference, the moving mean and variance values are no longer updated, and the normalisation is performed based on the last values recorded during training.
Crucially, in the default implementation of these networks, these moving mean and moving variance values are initialised as 0 and 1, respectively. Therefore, an untrained network initialised in this way will behave as if the batch normalisation layers were not present.
To obtain more sensible random networks, i.e., ones that can still make use of batch normalisation, we initialise the moving mean and variance of batch normalisation layers differently. We generate a batch of 50 random images (each pixel value is chosen independently and uniformly at random in the allowed range). This single batch is then fed through the random network and each batch normalisation layer records the mean and variance values it sees at its input (and normalises its output accordingly). The trainable weights of the network are not changed during this process.
We should note that, in addition to the moving mean and variance, batch normalisation layers can have trainable weights, by means of which the output mean and variance can be tuned. Specifically, such a layer may have trainable weights 7y and 3, and will scale its otherwise normalised output by ? and shift it by B. If present, these trainable weights are initialised 1 and 0 respectively, and hence have no effect. Our initialisation procedure does not modify these trainable weights and they are therefore not used in our random networks.
For each network, after randomly initialising its weights as set out in Section 4 and its batch normalisation layers as described above, we keep it completely fixed: neither its weights nor its batch normalisation layers (i.e., their moving means and variances, and their weights if any) change in any way.
Combining input images with adversarial programs. Our adversarial programs are colour im- ages whose sizes match the expected input size of the networks. This is 224 x 224 for all networks except Inception-v3, where it is 299 x 299.
We use two different schemes to combine input images with the adversarial program. The first scheme, used by Flsayed et al. [2019], is to take the adversarial program and overwrite a portion of by the input image. We do this in such way that the input image is, up to rounding, centred in the adversarial program. We can vary the construction by scaling the input image up or down before applying this procedure. particular, we use parameter r E [0, 1] and scale the input image, using bilinear interpolation, in such way that times the width of the adversarial program is equal to the width of the scaled image. That we focus on the width is not important because all our inputs and programs are square. An illustration is shown in Figure 1.
Our second scheme involves scaling the image the same size as the adversarial program and then taking a convex combination of the two. We use a parameter v E [0, 1] to specify how much weight
We are making code to run the experiments available at https: t/gtthb..// com/ com//nglerttm// eiwersariallreprorraae
7

---

H
imput from adversarial task
scaled input. width r-L.
adversarial program, of width
combined input, fed to network
Figure 1: Scheme I for combining input images with adversarial programs. In this example, the width and height of the adversarial program are 224 and the parameter equals 2 2 z 0.214, so the input image is scaled width and height r 224 rounded, which is 48.
4
v
+ (1
imput from
scaled input
adversarial program
combined input, fed network
Figure 2: Scheme 2 for combining input images with adversarial programs. In this example, the sizes of the scaled input image and the adversarial program are 224 224. The parameter equals 2 2-40/9 0.046, so the weight of the input image in the convex combination with the adversarial program is approximately 4.6%, which makes it faintly visible.
the input image should get in this convex combination. Specifically, the combined image is obtained by calculating I + (1 - v) P. where / is the input image and P is the adversarial program. An illustration is shown Figure 2.
Adversarial task dataset. Elsayed al. [2019] evaluate adversarial reprogramming random networks using the MNIST [LeCun et al., 1998] dataset. In other words, they were asking whether it possible to repurpose a random network for the task of classifying the handwritten digits from the MNIST dataset. We use the same dataset, which consists of 60,000 training images and 10,000 test images, for our experiments. It available under the Creative Commons Attribution Share-Alike 3.0 licence.
The networks onto the first 10 of these classes.
classify inputs into 1,000 classes. We map the 10 labels of the MNIST dataset
For additional experimental results on the Fashion-MNIST and Kuzushiji-MNIST datasets, please see the appendix.
Finding and evaluating adversarial programs. Internally, represent adversarial programs using unconstrained weights. We then apply a softsign function to the weights to map them into the range (-1, 1), and further shift and scale the program such that the pixel values lie in the same range that is used for the input images. The program is initialised in such way that after the application of the softsign function, each value lies uniformly at random in (-1,
We use the 60,000 training images to run an Adam optimiser [Kingma and Ba, 2015] with learning rate 0.01 and batch size of 50 to optimise the unconstrained weights of the adversarial program. We report the accuracy on the 10,000 test images after 20 epochs, please see Figure 3.
The experiments were mainly run on two internal clusters utilising mix of NVIDIA GPUs such as GeForce RTX 3080 Ti, Quadro RTX 6000, GeForce RTX 3060, GeForce RTX 2080 Ti, and GeForce GTX 1080. Depending the network, optimising a single adversarial program for 20 epochs takes between minutes and 1.5 hours on standard desktop computer with two NVIDIA GeForce RTX 3080 Ti GPUs.

---

Scheme
Scheme 2
1
0.8
0.8
test accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0
0.1
0.001
0.01
0.1
r
ResNet-50
ResNet-50V2
ResNet-10112
Inception-v3
EfficientNet-BO
ResNet-152V2
Figure 3: The accuracy achieved by the adversarial program on the MNIST test set for different parameters the schemes of combining input images with adversarial programs. The horizontal axes are logarithmic. The values plotted are averages over 5 trials, which are listed together with the standard deviations in the appendix.
We did not explore different optimisers and learning rates, since our first choices already resulted in suitable adversarial programs for these random networks. We only reduced the batch size to 50 after first trying 100, in order to reduce the requirement on GPU memory and be able to easily run the experiments wider range of hardware. However, did extensively explore the two different schemes combing input images with adversarial programs, and different values for the respective parameters r and v. For each network, and each value of and ran 5 experiments, each with new random initialisation the network, and are reporting the average of the test accuracy.
Discussion. Overall, the second scheme of combining input images with adversarial programs appears to give better and more reliable results our experiments. For both schemes, the choice of parameters is important. Clearly, when On the other hand, when r or is 1, there no longer is an adversarial program. In most cases, best results achieved when the adversarial program is significantly larger (either by actual size in the first scheme, or in terms of pixel value ranges in the second scheme) than the input image.
is O, the input image is not visible to the network at all.
In the second scheme in particular, we see accuracies on the test set which are lower, but not much lower than what Elsayed et [2019] reported for networks trained on ImageNet. For v as 0.046 for example, we see accuracies of 91 18% 91.1% %, 90.9%, 90.2%, 88.8%, 72.0% for EfficientNer-Bo ResNet-50, ResNet-101V2, Inception-v3, ResNet-50V2, that, while training, say, ImageNet may impact the possibility of finding suitable adversarial programs, such a training may be less important than previously thought and other factors are of significant importance.
15222. respectively. This suggests
5 Conclusion and future work
We proved the first theoretical results on adversarial reprogrammability of neural networks, in which we focused on architectures with two layers and ReLU activations, and on Bernoulli and Gaussian adversarial tasks. Provided the input dimension is sufficiently large, and for wide variety of parameter regimes, our results show that: firstly, arbitrarily high reprogramming accuracies are achievable in expectation for networks with random weights; and secondly, reprogramming accuracies that are no better than guessing may be unavoidable for networks that were trained for many iterations with small learning rates orthogonally separable datasets.

---

In the theoretical results that conclude arbitrarily high expected reprogramming accuracies, we assumed that the width of the random network no grater than its input dimension, which is similar to the assumption existence of suitable adversarial programs by matrix inversion. Interesting directions for future work include relaxing this assumption, extending the whole theoretical analysis to deeper networks, and investigating derivation of adversarial programs by gradient methods.
widths in e.g. Daniely and Shacham [2020] and enabled us to show
It would also be interesting to consider more permissive data models; however, our theoretical results on the failure of adversarial reprogramming on networks that were trained to infinity rely on implicit bias properties of gradient methods, and in that area separability assumptions on training data are common and appear challenging lift (see e.g. Lyu al. [2021]).
The outcomes of our experiments, which are on six realistic convolutional network architectures designed for image classification, and three adversarial tasks provided by the MNIST, Fashion- MNIST and KuzushijiMNNIST datasets, are consistent with our theoretical results on high repro- gramming accuracies. Both the experimental plots of test accuracies and the theoretical bounds of network outputs indicate existence of "sweet spot" maximising parameter values." 3 Since in the experiments, the network architectures have widths, depths and other features that are currently beyond our theoretical assumptions, and the adversarial programs are derived by gradient descent, their outcomes provide further motivation for extending the theory as suggested above.
conclusion that emerges across our experimental results is that the EfficientNet-Bi architecture tends to be more susceptible adversarial reprogramming, and the ResNet-152V2 architecture tends to be less susceptible, than the remaining four. We suggest for future work investigating the causes of this, as well seeking to reprogram random networks for adversarial tasks that more difficult than MNIST, Fashion-MNIST and Kuzushiji-MNIST In the context of the latter, it would be interesting to consider also vision transformer [Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby, 2021] architectures.
Another direction for experimental work is verify that long training can cause adversarial repro- gramming to fail.
Acknowledgments and Disclosure of Funding
We are grateful to the anonymous reviewers, whose comments helped improve the paper. We also thank Maria Ovens for linguistic advice.
This research was partially supported by the Centre for Discrete Mathematics and Its Applications (DIMAP) at the University of Warwick.
References
Peter L. Bartlett, Sébastien Bubeck, and Yeshwanth Cherapanamjeri. Adversarial Examples in Multi-Layer Random ReLU Networks. In NeurlPS, pages 9241-9252, 2021.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Sindić, Pavel Laskov, Giorgio Giacinto, and Fahio Roli. Evasion Attacks against Machine learning Test Time. In ECML. PKDD, pages 387-402, 2013.
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial Patch. CoRR, abs/1712.0966s 2017.
Séhastien Bubeck, Yeshwanth Cherapanamjeri, Gauthier Gidel, and Rémi Tachet des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. NeurlPS, pages 10081-10091, 2021. 4
Tarin Clanuwat, Mikel Boher-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep Learning for Classical Japanese Literature. CoRR, ahs/ 1ii.01111,1118,, 2018. 3
Frank H. Clarke. Generalized gradients and applications. Trans. Amer. Math. Soc., 205:247-262. 1975. 5
'The varying of the scaling parameters r and v in the experiments corresponds in the Bernouli data model case to varying the radius p while keeping all other parameters tixed, and in the Gaussian data model case to varying the mean radius and the variance parameter while keeping fixed their ratio and all other parameters.
10

---

Amit Daniely and Hadas Shacham. Most ReLU Networks Suffer from E Adversarial Perturbations. In NeurlPS, 2020. 10
Damek Davis, Dmitriy Drusvyatskiy, Sham M. Kakade, and Jason D. Lee. Stochastic Subgradient Method Converges on Tame Functions, Found. Comput. Math., (0(1):119-154 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In /CLR, 2021. 10
Joydeep Dutta, Kalyanmoy Deb, Rupesh Tulshyan, and Ramnik Arora. Approximate KKT points and a proximity measure for termination. J. Glob. Optim., 56(4): 2013.
Gamaleldin F. Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial Reprogramming of Neural Networks. In ICLR, 2019. 3, 7,
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian J. Goodfellow. Adversarial Spheres. In /CLR (Workshop), 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pages 149-255, 2010. 7
Karen Hambardzumyan,, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgramming. In ACL/ICCLLP, pages 4921-4933, 2021. 2021. 2
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun, Delving Deep into Rectifiers: Surpassing Human- Level Performance on ImageNet Classification. In ICCV, pages 1026-1034, 2015. 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pages 770-778, 2016a. 7
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. In ECCV. pages 630-645, 2016b. 7
Kun He, Yan Wang, and John E. Hoperoft. A Powerful Generative Model Using Random Weights for the Deep Image Representation. In NeurlPS, pages 631-639, 2016c. 2
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Mądry. Adversarial Examples Are Not Bugs, They Are Features. In NeurlPS, pages 125-136, 2019.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, pages 448-456, 456, 2015.
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. In NeurlPS, 2020. 6
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015. 8
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Master's thesis, University of Toronto, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEFE, 1621222278- 1998. 4,
Jachoon Lec, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl- Dickstein. Deep Neural Networks as Gaussian Processes. ICLR, 2018.
Kaifeng Lyu and Jian Li. Gradient Descent Maximizes the Margin of Homogeneous Neural Networks. In ICLR, 2020. 6
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang. and Sanjeev Arora. Gradient Descent Two-layer Nets: Margin Maximization and Simplicity Bias. NeurlPS, pages 12978-12991, 2021. 6,
Grégoire Mesnil, Yann N. Dauphin, Xavier Glorot, Salah Rifai, Yoshua Bengio, Ian J. Goodfellow, Erick Lavoie, Xavier Muller, Guillaume Desjardins, David Farley, Pascal Vincent, Aaron C. Courville, and James Bergstra. Unsupervised and Transfer Learning Challenge: a Deep Learning Approach. In ICML Workshop on Unsupervised and Transfer Learning, pages 97-110, 2011. 2
Andrea Montanari and Yuchen Wu. Adversarial Examples in Random Neural Networks with General Activations. CoRR, abs/2203.17209, 2022.

---

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal Adversarial Perturbations. In CVPR, pages 86-94, 2017. 1
Paarth Neekhara, Shehzeen Hussain, Shlomo Dubnov, and Farinaz Koushanfar. Adversarial Reprogramming of
Text Classification Neural Networks. In EENLP-IJCNEL pages 2215-52211 2019. 2
Paarth Neckhara, Shehzeen Hussain, Jinglong Du, Shlomo Dubnov, Farinaz Koushanfar, and Julian J. McAuley.
Cross-modal Adversarial Reprogramming. In WACV, pages 2898-2906, 2022. 2
Mary Phuong and Christoph H. Lampert. The inductive bias of ReLU networks on orthogonally separable data. In ICLR, 2021. 3,5,
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y. Ng. Self-taught learning: transfer learning from unlabeled data. In ICML, pages 756,766, 2007.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bemstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis., 11(3))211 1-222,, 2015.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially Robust Generalization Requires More Data. In NeurIPS, pages 5019-5031, 2018. 3, 4, 6
Adi Shamir, Odelia Melamed, and Oriel BenShmuel. The Dimpled Manifold Model of Adversarial Examples in Machine Learning. CoRR, abs/2106. 10151, 2021.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to a Crime: Real and Stealthy Attacks on State-of-the of-the-Art Art Face Recognition. In CCS, pages 1528-155 1540,, 2016.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srehro. The Implicit Bias of Gradient Descent on Separable Data. J. Mach. Learn. Res., 19:70:1-70:57, 2018.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In CVPR, pages 2818 2826, 2016. 7
Mingxing Tan and Quac V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In ICML, pages 6105 61 14, 2019. 7
Yun-Yun Tsai, Pin-Yu Chen, and Tsung-Yi Yi Ho. Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources. ICMI,, pages 9614-9624, 2020.
Gal Vardi, Gilad Yehudai, and Ohad Shamir. Gradient Methods Provably Converge to Non-Robust Networks. In NewrlPS, 2022.
Roman Vershynin. Introduction the non-asymptotic analysis of random matrices. In Youina C. Eldar and Gitta Kutyniok, editors, Compressed Sensing, pages 210-268. Cambridge University Press, 2012. URL httt:///rxiv> wtwpo//////./..///./0/
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47 of Cambridge Series Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.
Ria Vinod, Pin-Yu Chen, and Payel Das. Reprogramming Language Models for Molecular Representation Learning. CoRR, abs/2012.03460, 2020.
Martin J. Wainwright. High--dmeensionll statistics: A non-asymptotic viewpoint, volume 48 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
Yifei Wang and Mert Pilanci. The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge Extreme Points of the Dual Convex Program. In ICI.R, 2022. 3
Yunjuan Wang, Enayat Ullah, Poorya Mianjy, and Raman Arora, Adversarial Robustness is at Odds with Lazy Training. In NeurlPS, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. CoRR, abs/ http///08.077777 2017. 3
Chao-Han Huck Yang, Yun- Yun Tsai, and Pin-Ye Yu Chen. Voice2Series: Reprogramming Acoustic Models for Time Series Classification. In ICML, pages 11808-10001 11819, 2021. 2
12

---

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Honggang Zhang, Jun Guo, Guang Chen, and Chun-Guang L.. HCL.2000 A Large-scale Handwritten Chinese Character Database for Handwritten Character Recognition. In ICDAR. pages 286-290, 2009.
Yang Zheng, Xiaoyi Feng, Zhaoqiang Xia, Xiaoyue Jiang, Ambra Demontis, Maura Pintor, Battista Biggio, and Fabio Roli. Why Adversarial Reprogramming Works, When It Fails, and How to Tell the Difference. CoRR, abs/2108.1 16673, 2021.
13


Reviews:
[{"review_id": "b3E0DZQ2bn", "reviewer": "Reviewer_iiJN", "summary": "This work studies the problem of \"Adversarial Reprogramming\" which is a task that is defined as follows. Suppose we have a machine learning model M that takes in inputs from a particular domain X to a label domain Y, while as a malicious actor imagine that we want to accomplish a task that takes in inputs from domain X' and outputs targets in domain Y'. Then the setting of adversarial reprogramming asks the question whether it is possible to find some simple functions h_in: X' -> X and h_out: Y -> Y' such that we can utilize h_out(M(h_in(x))) to solve this task. Note that finding a function h_out mapping from a discrete label space Y to Y' is a well studied problem in combinatorial optimization with several standard algorithms such as the Hungarian algorithm. Therefore, the main crux of the problem is to find the mapping h_in. Furthermore, in Adversarial Reprogramming a further assumption is made about h_in - that it be of the form of additive noise, i.e. h_in(x) = p + x, for some noise p which may depend on the model M. The goal then is to maximize accuracy on the task X' -> Y' without affecting the model parameters M (so one cannot fine-tune).\n\nThe authors study this problem for randomly initialized neural networks, i.e. when the model M is random. They show that in the 2-layer setting with the ReLU activation function and on a data distribution defined over the boolean hypercube one can find an adversarial program with arbitrarily high accuracy under several assumptions regarding network width etc. The authors also prove some theorems regarding when the dataset is orthogonally separable which I did not fully follow - e.g., Theorem 3 (are the convergence results about SGD or something else?). The authors also have some experiments which is hard to follow and I am not sure how their Theorems about 2 layer networks on boolean hypercubes translates to EfficientNets, Resnet-50 etc., on which they report experimental results. ", "questions": "1) While the assumption about boolean hypercube may be a bit contrived, it is still a reasonable starting assumption in my opinion. However, it is not clear to me why are we interested in Adversarially reprogramming a random neural network? The motivation for that is not clear to me, an attacker would usually be interested in reprogramming an already pre-trained model.\n \n2) The presentation of the paper is a little hard to follow, especially in Section 3  where someone not too familiar with the area will find it hard to follow. Moreover the consequence of Theorem 3 and Corollary 4 is not fully clear, what is the implication of this particular training dynamics?\n\n3) The relation of the experimental section with the theoretical contribution of the work is unclear and again difficult to follow. Is it being validated that the theorem's proposed noise function is successfully finding some inputs on which a randomly initialized network has some high accuracy?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper seems to be about a topic that appears to be important, i.e. whether one can repurpose a model trained on a certain domain to another (potentially unethical or malicious) domain.\n\nWeaknesses: The major weaknesses of the paper come from the several assumptions made about the Adversarial Reprogramming task:\n\nTypically given a model M and a dataset domain (X, Y) a malicious actor would most likely wish to use M for a specific task (X', Y'): for example consider a scenario where an image classification model is trained to classify everyday objects (such as say Imagenet) but the malicious user wants to use it to predict whether people of a certain group are more criminally inclined than the others. However, the assumption made throughout is that \"one can find an adversarial task\" and the theorems also seem to work only for specific kinds of adversarial task (e.g., the theorem seems would not go through if the noise added in the boolean dataset task is some other quantity). However, this obtained adversarial task/dataset itself may not be as interesting or even malicious or meaningful to an attacker.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B17fGP4yhA3", "reviewer": "Reviewer_TjuY", "summary": "This paper raises a fundamental question: what is the relationship between training and adversarial reprogrammability? The authors provide a theoretical analysis for adversarial reprogramming, where the conclusion shows that training has some but limited impact on reprogramming.", "questions": "Please see [Strengths And Weaknesses].", "limitations": "Please see [Strengths And Weaknesses].", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Reprogramming can have an important impact on many tasks, e.g. reprogramming for transfer learning, so a solid and fundamental investigation is promising.\n\n2. The conducted experiments provide good support to the theoretical analysis.\n\nWeakness:\n\n1. The authors mention several application scenarios of reprogramming, but the connection between the theoretical analysis and the mentioned applications seems weak.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5YVLdZ5gsdJ", "reviewer": "Reviewer_Fq7W", "summary": "The paper studies adversarial reprogramming (Elsayed et al 2019) that is the adversarial attack setting that can repurpose a network to conduct a task of interest. The paper provide theoretical proofs using simple random network and conclude that random networks can be adversarially repurposed as long as the input dimensionality is >= the network width. Elsayed et al 2019 managed to adversarially reprogram deep networks that are trained originally on ImageNet but failed to reprogram random deep networks. To address the possibility of training random networks in practice, the paper conducts experiments on multiple practical deep network architectures and show that the random networks can be repurposed to perform MNIST classification as long as batch normalization is suitably initialized, thus giving evidence that random networks could be trained in practice. The paper also explores two schemes for adding adversarial program (1) replacing the center by input images and (2) overlapping mixture of input images and adversarial program. The authors finds that the second scheme is more successful.", "questions": "Please see my above comments. ", "limitations": "I think discussion of limitations is somewhat lacking. The authors needs to explain clearly the limitation of the work in a separate section or at the conclusion section. Please consult my comments on the strengths and weaknesses above. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Work provide theoretical proofs, though in reduced setting, of conditions where it is possible to adversarially reprogram neural networks.\n- Authors explore two schemes of applying adversarial reprogramming of neural networks and demonstrate the superiority of one approach over the other.\n- Authors provide empirical study on multiple network architectures\n\n\nWeaknesses:\n- Clarity of presentation: The theory in the paper could be somewhat overwhelming. I encourage the authors to give intuitions in addition to the theoretical proofs. To give an example, the authors prove that the two layer network can be reprogrammed given that the network width <= input dimensions. This conclusion may be confusing to some as a wider network has more capacity and may be regarded as more capable to do adversarial tasks. Maybe an intuition here is that the adversarial program dimensionally scales with input dimensions and in fact a larger input dimension allows more space to explore suitable adversarial programs that can repurpose the network.\n\n- Theory limitations: To be able to provide theoretical proofs the authors needed to make simplifying assumption on the network architecture as well as the task. This although still an interesting contribution limit the impact of the work. I encourage authors can mention this limitation summary list of assumptions they needed to make. Perhaps also provide some suggestion for potential ways to extend the theory to more realistic networks and tasks. \n\n- Discussion of batch norm: the discussion in the experimental section on batch normalization is not very clear. It is not 100% clear whether batch normalization is trained or left at initialization values. Also, changing the initialization scheme for batch normalization kind of violates the adversarial reprogramming setting where an adversary is given a network as is and one can only design the adversarial program.  \n\n- Experiments limitations: the authors use only one task (MNIST). I think authors needs to provide consistent evidence from at least two datasets to give confidence on the conclusions made. Further, experiments in Figure 2 uses one sample. The experiments need to be repeated from different seeds to get an estimate of variations (eg STD or SEM).\n\n- It is unclear why scheme 2 of mixing adversarial programming with image is more successful. Is this because the dimensionality of useable adversarial program is higher than the scheme where the center of program is replaced with the image? It would be good to provide some explanation here.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept