A Target-Agnostic Attack on Deep Models: Exploiting Security Vulnerabilities of Transfer Learning
Authors: Shahbaz Rezaei
Xin Liu
Venue:  (2020)
Keywords: Machine learning security
Transfer learning
deep learning security
Softmax Vulnerability
Transfer learning Security

Abstract: Due to insufficient training data and the high computational cost to train a deep neural network from scratch, transfer learning has been extensively used in many deep-neural-network-based applications. A commonly used transfer learning approach involves taking a part of a pre-trained model, adding a few layers at the end, and re-training the new layers with a small dataset. This approach, while efficient and widely used, imposes a security vulnerability because the pre-trained model used in transfer learning is usually publicly available, including to potential attackers. In this paper, we show that without any additional knowledge other than the pre-trained model, an attacker can launch an effective and efficient brute force attack that can craft instances of input to trigger each target class with high confidence. We assume that the attacker has no access to any target-specific information, including samples from target classes, re-trained model, and probabilities assigned by Softmax to each class, and thus making the attack target-agnostic. These assumptions render all previous attack models inapplicable, to the best of our knowledge. To evaluate the proposed attack, we perform a set of experiments on face recognition and speech recognition tasks and show the effectiveness of the attack. Our work reveals a fundamental security weakness of the Softmax layer when used in transfer learning settings.

Published as a conference paper at ICLR 2020
A TARGET-AGNOSTIC ATTACK ON DEEP MODELS: EXPLOITING SECURITY VULNERABILITIES OF TRANSFER LEARNING
Shahbaz Rezaci Xin Liu Department of Computer Science University of California
Davis, CA 95616, USA {srezaei, xinliv] @ucdavisi edu
ABSTRACT
Due to insufficient training data and the high computational cost to train deep neural network from scratch, transfer learning has been extensively used in many mewuurl-nttwwrk-baaeeeeeee
applications. A commonly used transfer learning ap- proach involves taking a part of a pre-trained model, adding a few layers at the end, and re-training the new layers with a small dataset. This approach, while efficient and widely used, imposes security vulnerability because the pre-trained model used in transfer learning is usually publicly available, including poten- tial attackers. In this paper, we show that without any additional knowledge other than the pre-trained model, an attacker can launch an effective and efficient brute force attack that can craft instances of input to trigger each target class with high confidence. We assume that the attacker has no access to any target-specifie infor- mation, including samples from target classes, re-trained model, and probabilities assigned by Softmax to each class, and thus making the attack target-agnostic. These assumptions render all previous attack models inapplicable, to the best of our knowledge. To evaluate the proposed attack, we perform a set of experiments face recognition and speech recognition tasks and show the effectiveness of the attack. Our work reveals fundamental security weakness of the Softmax layer when used in transfer learning settings.
INTRODUCTION
Deep learning has been widely used in various applications, such as image classification Parkhi et al. (2013), image segmentation Chen al.] (2016), speech recognition Ji et al. (2018), machine translation Wu et al. (2016), network traffic classification Rezaei & Liu (2019b). etc. Because training deep model is expensive, time-consuming. and data intensive, it is often undesirable or impractical to train a model from scratch in many applications. In such cases, transfer learning is often adopted to overcome these hurdles.
A typical approach for transfer learning is to transfer a part of the network that has already been trained on similar task, add one or more layers at the end, and then re-train the model. Since large part of the model has already been trained on similar task, the weights are usually kept frozen and only the new layers are trained on the new task. Hence, the number of training parameters is considerably smaller than it is when training the entire model, which allows to train the model quickly with a small dataset. Transfer learning has been widely used in practice Rezaei Liu (2019c). including applications such as face recognition Parkhi et al. (2015), text-to-speech synthesis Jia et al. (2018), encrypted traffic classification Rezaci & Liu (2019a), and skin cancer detection Esteva et al. (2017).
One security vulnerability of transfer learning is that pre-trained models, also refereed to as teacher models, are often publicly available. For example, Google Cloud ML tutorial suggests using Google's Inception V3 model as pre-trained model and Microsoft Cognitive Toolkit (CNTK) sug- gests using ResNet18 a pre-trained model for tasks such as flower classification Wang et al.

---

Published as a conference paper at ICLR 2020
Normal Scenario
Attack Actack Scenario
AV
AV
Softmax
Softmax
Transferred Model
Transferred Model
Figure I: Example of activation vector and how the Softmax layer responses. The image on the left shows the activation vector of a natural face in the training set. The Softmax layer performs Softmax operation over the linear combination of such activation vectors and assigns high confidence to the corresponding class. The target-agnostic image (the image on the right) is crafted such that it activates one neuron in activation vector with extremely large value and all others are almost zero Such activation vector also fools the softmax layer to produce output with high confidence. Due to the lack of space, we only show the first 400 neurons of the activation vector.
2018). This means that the part of the model transferred from the pre-trained model is known to potential attackers.
In this paper, we show that an attacker can launch a target-agnostic attack and fool the network when only the pre-trained model is available the attacker. In our attack, the attacker only knows the pre- trained (teacher) model used to re-train the target (student) model. The attacker does not know the class labels, samples from any target class, the entire re-trained model, or probabilities the model assigns to each class, making it target-agnostic. To the best of our knowledge, these assumptions are more general than those used any previously proposed attack models, which renders the old models ineffective.
The target-agnostic attack can be adopted in scenarios where fingerprint, face, voice is used for authenticatioon/veriiaatooo In such cases, the attacker usually lacks access to fingerprint or voice samples, which could be used bypass cuutentiicaioonverifcclllt put that triggers any target class with high confidence. The attacker can also continue the crafting process to trigger all target classes. Such adversarial examples can used to easily bypass authen- tication/verification systems without having true sample of the target class. Our work develops a highly effective target-agnostic attack, exploiting the intrinsic characteristic Softmax in trans- fer learning settings. Our experiments face recognition and speech recognition demonstrate the effectiveness of our attack.
Our attack aims to craft an
In a typical transfer learning procedure, all neural network layers up the penultimate layer transferred to a new model and then a Softmax layer is added and re-trained on a new task. We call the scores the penultimate layer activation vector and the part of the model that produces the activation vector feature extractor. Hence, the Softmax layer basically computes the softmax operation
the linear combination of activation vector. The left side of Figure [ presents natural input and typical activation vector. Due to the use of linear combination of elements of activation vectors in Softmax layer, not only such patterns can trigger the corresponding classes, but also a large number of other unrelated patterns can also trigger Softmax layer in the same way. In this paper, we show that if we craft an image that produces an activation vector such that one neuron is large and others are almost zero (the right side of Figure D it triggers the class for which the weight associated to that neuron is higher the linear combination. finding all features that should be activated by feature extractor, we assign very large value only one neuron to compensate for other neurons that we do not activate.
other words, instead of
In summary, the contributions of this paper are follows:
Present a target-agnostic attack in transfer learning settings. We show that if the pre- pre-trained model used during transfer learning is available, attacker can craft set of universal
2

---

Published as a conference paper at ICLR 2020
adversarial images that can effectively fool any model re- re-trained the pre- pre-trained model. Our attack does not need any training sample from the target model or the target model itself for crafting images. Such a target-agnostic attack has two consequences: 1) the crafting time is irrelevant because adversarial images are crafted only once and then they can be used any model that used the pre-trained model during the transfer learning stage (that is why the attack is called target-agnostic), and II) it does not need to query the target model to craft images. Hence, an attack can craft set of adversarial images on VGG face model, as an example, and then uses them effectively on any re- re-trained model based on VGG face.
2. Design a simple approach to exploit the vulnerabilities of Softmax layer. We show that both threshold-based approach, where the model only accept the classification result if the confidence is high, and reject-class- class-bead approach, where the model is trained with an extra class, called reject/null class, to reject adversarial images are prone to our attack.
3. Evaluation of our attack on face recognition and speech recognition tasks, We study the effectiveness of our model in different scenarios and settings.
2 RELATED WORK
In general, there are two types of attacks on deep neural networks in literature: 1) evasion and 2) data poisoning. the evasion attack, an attacker aims to craft or modify an input to fool the neural network or force the model to predict specific target class Elsayed et al. (2018). Various methods have been developed to generate adversarial examples by iteratively modifying pixels in an image using gradient of the loss function with respect to the input to finally fool the network (Szegedy et al. 2013); Carlini & Wagner (2017a|b). These attacks usually assume that the gradient of the loss function is available to the attacker. In cases where the gradient is not available, it has been shown than one can still generate adversarial examples the top 3 (or any other number of) predicted class labels are available (hhrrif et al.) (2016). Interestingly, it has been shown that the adversarial examples are often universal, that is, an adversarial example generated for a model can often fool other models as well & Wagner (201a)). This allows an attacker to craft adversarial examples from a model she trained and use it on the target model provided that the training set is available.
The second type of attacks on deep neural networks is called data poisoning Shafahi et al. (2018). In the data poisoning attack, an attacker modifies the training dataset to create a backdoor that can be used later to trigger specific neurons which cause mlsssiassiiiaation In some papers, a specific pattern is generated and added to the training set to fool the network to associate the pattern with a specific target class (hhrrf et al. 2016); Chen et al. (201 Liao et al. (20T8): Liu et al. (0117). For instance, these patterns can be an eyeglass in face recognition task (Sharif et (2016) randomly chosen patterns Chen et al. (b)1 some specific watermarks or logos Liu et al. (2017), specific patterns to fool malware classifiers Muñoz-Gonzzleez et al. (2017). etc. In some extreme cases, it has been shown that by only modifying single bit to have a maximum or minimum possible value, one can create backdoor Allerti al. (2018). This happens due the operation of max pooling layer commonly used in convolutional neural networks. After the training phase, the backdoor can be used to fool the network to predict the class label associated with these patterns at inference time.
There are a few studies specifically focused on attacks in transfer learning scenarios Di et al. (2018); Wang et al. (2018). In WWang et al. (2018), the pre-trained model and an instance of target image are assumed to be available. Assuming that the attacker knows that the first A layers of the pre- trained model copied to the new model, the attacker perturb the source image such that the internal representation (activation vector) of the source image becomes similar to the internal representation of the target image layer k. using traitrdined model. In Ji al. (018)), first, a set of semantic neighbors are generated for given source and target input which are used to find the salient features of the source and target class. Then, similar to Wang et al. (2018), the pre-trained feature extractor is used to perturb the source image along the salient features such that their internal representation becomes close. However, these attacks do not work when no instances of the target class is available.
In this paper, we propose target-agnostic attack on transfer learning. We assume that only the pre- trained model (e.g.. VGG face or ResNet18) 18) is available to the attacker. We assume the re-training data and the re- re-trained model is unknown and not even single target class sample is available. Our attack model is more general than the previous studies, and thus renders previous attacks on transfer learning infeasible. Note that black-box attacks Sharif et al. (2016); Papernot et (201)) where an
3

---

Published as a conference paper at ICLR 2020
n-2
m-1
Input
Conv1
Conv2
Poola
Softmax
New target classes
Feature extractor (from VGG Face)
Classifier
Figure 2: Transfer learning on VGG Face
attacker only have access to the model output, can theoretically be applied in our transfer learning settings. However, successful black-box attack often needs hundreds to millions of queries the target model whereas the high effectiveness of our attack means it only needs a few query to the target model generate adversarial input.
SYSTEM MODEL
In this paper, assume that the transferred model trained source task publicly available. This is a reasonable assumption, which in fact is widely used in practice. For instance, Liu et al. (2017) used the VGG face model Parkhi al. (015) trained recognize 2622 identities to recognize new faces. The model is shown in Fig. 2 While our attack targets any
deep
models, use face recognition based on VGG face an example for explanation. Fig. shows
the typical transfer learning approach for face recognition Parkhi et al. (2015).
In transfer learning, the layers whose weights are transferred the new model are called feature extractor that outputs semantic (internal) representation of an input. The last few layers that are re-trained on the new task are called classifier. typical transfer learning attack scenarios, transferred model is publicly available, but the re-trained model is not known to attacker. In other words, the attacker only knows the feature extractor but not the classifier. The previous work on transfer learning Wang al. (2018); ; Ji et al. (2018) assumes that at least sample image from each target class is available because they aim to generate images that produce similar activation vector as the target samples produce. These approaches do not work without samples from the target class.
In this paper, assume that the attacker does not have access any samples of the new target classes. Our motivation of the attack is to craft images for models used in systems, such as authenti- cation/verification system, for which there is target sample available, otherwise the attacker could have just used those samples. In such cases, attackers do not have access to samples of the target classes and, consequently, the previous attacks do not work.
ATTACK DESIGN
Design Principle. To launch an attack with these restrictive assumptions, we need to approach the problem differently. Our attack exploits the key vulnerabilities of the Softmax layer which assigns high confidence labels to area of input space that are necessarily close to the training manifold Gunther et al. (2017). Softmax layer basically performs Softmax operation on the linear combination of activation vector. The activation vector of real image often shows certain pattern with several triggered neurons, as in Figure (on the left). However, the linear combination of the Softmax layer can also be triggered if only single neuron the activation vector has a large value. In other words, each neuron of the activation vector has direct and linear relation with one few target classes with different weights. Hence, the attacker can trigger these neurons one by one to see which one is highly associated with each target class.
The main attack idea is to activate the th neuron at the output of the feature extractor (n In-1"t layer), denoted by z- with high value and keep the other neurons at the same layer zero, similar to the Figure (n the right). After the feature extractor, the model has only a FC layer and a Softmax that outputs the probability of each class. Because of the linear combination used before Softmax
4

---

Published as a conference paper at ICLR 2020
operation, if there exists a neuron layer n that associated large weight x="' it will become large. Hence, the softmax will assign a high confidence to that class. In order to find an adversary image, we can iteratively try to trigger each neuron at the (n 1)th layer to find an adversary image.
Next, we further explain the attack intuition in more detail using a simple example. Let's assume that the output of feature extractor is layer (n 1)th and only have two target classes. Let's keep all neurons at layer (n 1)th zero except the 7th neuron, denoted by x Then, for the last layer, n²th, we have 2t W "T,,a,?-1 and x - and other terms zero. We omit b for simplicity. Now, W > 2 increasing x n- increases the difference between of and zz- Although the difference increases linearly with =;- the Softmax operation makes the difference exponential. In other words, by increasing -;-² can arbitrarily increase the confidence of the target class whose W higher, i.c., class in this example. That is the motivation of the proposed brute force attack.
Algorithm The target-agnostic brute force attack
Input: M (number of neurons at the output of feature extractor), Isng (initial input), K (number of iteration), F (known feature extractor), a (step constant), T (the target model on attack):
1: procedure ATTACK(/img imgs F, T)
2: for i from 1 to M do
3:
Y 0m
4:
Y
[ 1000;
Any sufficiently large number
5:
X limg
6:
for from 1 to K do
7:
= (F((((i) - Y(1)² + B(E relu -Y
8:
8 = 8L
X x a aδ
9:
(X) bypasses the authentication then return X return Ø
Algorithm Design. The brute force algorithm is shown in Algorithm I We first iterate through all neurons at the output of the feature extractor and set the target, Y, such that at each iteration only one neuron is triggered. We set all elements of Y to zero except for the 7t one which can be set to any sufficiently large number, e.g., 1000, in Algorithm| [ Note that Y is a target of the feature extractor, not that of the entire re-trained model. In the case of the VGG face, there are 4096 neurons at this layer. So, we only try 4096 times at maximum. In fact, we will show in the next section that we only need to try few times to trigger any class and we need way fewer than 4096 attempts to trigger all target classes at least once.
Inside the second loop, we use the derivative the loss with respect to an input and change the input gradually to decrease the loss. Note that in the loop only use the pre-trained model and the re- re-trained target model is not needed. We find that typical MSE loss between Y and feature extractor is very inefficient. For the target activation vector where ith neuron is large and all other neurons are close to zero, the modified loss is defined as follows:
L == Y(i)²² 3 (Flu((((x)1 Y(1)²),
where F(X) is the output of feature extractor (i.e. activation vector) and Y is the target activation vector. It is similar the regular MSE loss with two minor changes: D) Because the importance of 7th neuron is greater than all other 4095 neurons to our attack, we use y and 3 to control the influence of each part on the loss function. II) Because of the existence of relu function after each fully connected layer to provide non-linearit,, any value on the ( (-o0, 0] range becomes O. Hence, instead of crafting an image that has large value ith neuron and zero value all other neurons, we aim to craft an image that has large value in 7th neuron and any non-positive value in all other neurons. Not only the original MSE might not converge to an adversary example, our revised loss function defined in [1 is much more efficient since the loss function only focuses neurons that have positive value at each step and ignores the ones that are already negative. This goal is acheived by adding the relu function in the loss function.
5

---

Published as a conference paper at ICLR 2020
Implication. We call this type of attack target-agnostic because it does not exploit any information from target's ss classes, model, or samples. In fact, if the same pre-trained model is used to re- re-train two different target tasks (models), A and B, the proposed target-agnostic attack crafts similar adversarial inputs for both A and B since only uses the pre-trained model to craft inputs. The implication is that the attacker can craft set of adversarial inputs with the source model using the proposed attack and use it effectively to attack all re- re-trained models that use the same pre-trained model. This means that the attack crafting time is not important and one can create a database of likely-to-trigger inputs for each popular pre-trained model, such as the VGG face or ResNet18. Given the simplicity, remarkable effectiveness, and target-agnostic feature of the proposed algorithm, it poses a huge security threat to transfer learning.
5 EVALUATION
In this section, we evaluate the effectiveness of our approach using two test cases: Face recognition and speech recogmition (Appendix [..)).
We use Keras with Tensorflow backend and server with Intel Xeon W-2155 and Nvidia Titan Xp GPU using Ubuntu 16.04 We use two metrics to evaluate the proposed attack model: 1) Number of attempts to break all classes (NABAC): Assuming that the number of target classes are known, this metric shows how many adversarial input instances are queried, on average, to trigger all target classes at least once with above 99% confidence. 2) Effectiveness (X%): This metric shows the ratio of crafted inputs that trigger any target classes with X% confidence over the total number of crafted inputs. We use 95% and 99% confidence for effectiveness in this paper.
(.2))
5.1 CASE STUDY: FACE RECOGNITION
In this case study, we use the VGG face model Parkhi et al. (2015) as a pre-trained model. We remove the last FC layer and the softmax (SM) layer to make feature extractor. Then, we pair it with new FC and SM layer, and re- train the model (while fixing feature extractor) with labeled faces of vision lab UMass LWF (2016). During re- re-trrining, we train the model with Adam optimizer and cross entropy loss function. We set K 50000, a 0.1, B = 0.01 and ? 1. In some experiments, we add more FC layers before the SM layer, as explained later.
Number of Target Classes. Table [ shows the impact of number of target classes on the attack performance. We use 20 classes with the highest number of samples from UMass dataset LWF (2016). The largest class is George W Bush with 530 samples and the smallest one is Alejandro Toledo with 39 samples. A blank image used as an intial image. For 5, 10, and 15 classes, we randomly choose set from 20 classes and re- train and attack the model 50 times and average the results. For 20 classes, we only re-train and attack once. That is the reason we do not show the standard deviation in the table. Table 2 for test set and all other images for training set. Hence, the re-training dataset is imbalanced. To balance the dataset, we undersample all classes to have an equal training size, shown in Table []]
shows the result when use five images from each class
As it is shown, the effectiveness of the attack on an imbalanced model is higher. However, the NABAC is slightly worse. We find out that on average the weights of SM layer for the class with larger training samples are slightly higher than the other classes. Hence, it is easier to trigger that class with the proposed method which increases the effectiveness. However, it is much harder to trigger the smallest class which makes the NABAC larger. The impact of imbalance re-training dataset is studied in more detail in Appendix I. where we show that the probability of triggering a target class directly associated with the number of training samples of that class during re- re-training. Moreover, the effectiveness and the NABAC improves when the number of target classes decreases, as expected. Note that in all scenarios, the effectiveness is greater than 75%. It means that the first crafted image has more than 75% chance of bypassing the authentication system (or any other application). It basically means that the traditional approach of limiting the number of queries to prevent brute-force-baseee attack does not work here.
Number of Layers to Re-train. In previous experiments, assume that the weights of the feature extractor transferred from the pre-trained model are fixed during re-training and only the last FC layer is changed. One can tune more layers during re-training. Fig. 3 shows the impact of tuning
'The implementation is available in http://ghhn..ooommmeerrr/////gge

---

Published as a conference paper at ICLR 2020
Table 1:
Attack performance on balanced re-training dataset. Acc, NABAC, and Eff stands for accuracy, number of attempts to break all classes, and effectivvness, respectively.
Target classes
Balanced dataset
Acc
NABAC
(l5()
Eff((9%)
5
99.12%
.27
48.25 + 42.5
91.68% + 5.69
87.82%
6.98
10
98.43%
.23
149.97 + 132.15
88.87% + 2.46
83.07% 3.31
15
97.16%
+ 1.64
323.36 + 253.56
87.79% + 2.42
82.05% + 2.74
20
96.87%
413
87.17%
79.16%
Table 2: Attack performance on imbalanced re-training dataset. Ace, NABAC, and Eff stands for accuracy, number of attempts break all classes, and effectiveness, respectively.
Target classes
Imbalanced dataset
Acc
NABAC
Eff(
Eff(99%%
5
99.21% + .29
63.29 + 80.30
93.52% + 5.07
90.23% + 5.71
10
98.47%
_81
264.80 + 111.09
91.14% + 3.65
86.28%
5.40
15
98.01% + 1.39
451.45 + 244.31
90.41% + 1.89
85.31% + 2.48
20
97.07%
2836
88.72%
82.93%
more layers on the effectiveness and accuracy. Note that we assume that attacker does not know anything about the target model. Hence, in this experiment, the attacker still uses the pre-trained feature extractor up until the last FC layer. That means the pre-trained feature extractor that the attacker uses is slightly different from the re-trained model. In Fig. Aa.. X axis represents the layer from which we start tuning up to the last FC layer. Due to the small re-training dataset, as the number of tuning layers increases the accuracy drops. However, by tuning more layers, the pre-trained model that the attacker has access to becomes more different from the re- re-trained model. That is why the effectiveness of the attack decreases. Similarly, NABAC increases, shown in Fig. 3(b).. Despite the difference between the re- re-trained model and the model the attacker has access to, the attack is still effective, which means that the pre-trained model cannot be changed dramatically during re-training process and re-training more layers is not an effective defense strategy.
Number of New Layers in the Re-trained Model. Next, measure how adding and training more layers (pair of FC + Relu) after feature extractor can affect the proposed attack effectiveness. In this experiment, use balanced target classes. As shown in Table D] adding more layers decreases the accuracy of the re-trained model because the re- ee-training dataset is small and not enough to train more layers from scratch. The effectiveness of the attack decreases sightly as more new layers are tuned. When adding more new layers, not all target classes are affected equally and some classes may become harder to trigger. That why NABAC increases. The goal of our attack is have an activation vector with only one large value. However, each extra layer, added after feature extractor,
100%
800
90%
500
85%
75%
Etfactivenes(959 Elfectiveneea
100
Accurscy
70%
FC-8
Con-5-a
Con--511
Con--433
Gonv-4-1
Canv-3-
FC-7
FC-6
Com-5-
Can-S-1
Conv-43
Conv 1
Canv-33
Con-3-1
Canr
(a) Effeet re-training more layers
(b) Number of re-training layers vs NABAC
Figure Effect number of re-training layers

---

Published as a conference paper at ICLR 2020
Table 3: Effect of number of new layers in the re-trained model
# of new layers
Accuracy
NABAC
Effectiveness((55%)
Effectiveness(999)
99.12% 1 .27
48.25
42.5
91.68% + 5.69
87.82% = 6.98
98.24% + 2.10
51.87
39.94
91.57% 1 4.87
86.45% + 5.35
3
95.46% + 4.2
257.26 + 387.16
89.45% + 8.20
85.67% = 8.88
100
80
40
20
Accuracy
Effectivenass
100 200
300
800 900
Reject class training samples (k)
Figure 4: Number of reject training samples vs effectiivenes/aaccrraay
smooths out the single large value and distributes to more neurons in activation vector. That the reason the attack becomes less effective when more new layers are added.
Attack Effectiveness on A Classifier with Reject Class. has been shown that relying on threshold to reject accept the classification result is not accurate because for the vast space of unknown inputs Softmax provides high confidence scores Nguyen et al. (2015). Hence, we add an extra class to the softmax layer, similar to Hosseii al. (2017), called reject/unknown class. During re-training, we choose random sample images from entire UMass dataset, except the classes that we choose for target faces, and label them as reject class. We vary the number of training samples for reject class to see its effect on accuracy and effectiveness. Figure f Illustrates the trade-off between the accuracy of the re-trained model and the effectiveness of our attack. The lowest effectiveness, for which the accuracy is 87. 37.72%, is 41. 41.40% which is still high. Hence, the classifier with reject class option still prone our target-agnostic attack.
Attack Comparison with black-box attack and baseline attack. We compare our attack with black-box attack and baseline. For the baseline attack, we choose random face images from UMass dataset that have not been used for training. Interestingly, for the threshold beeseold-based model, there is 12.60% chance that random face image triggers an output class with high probability, as shown in T A model with reject class can effectively prevent baseline attack since none of the random images fool the model. Morcover, we use Zoo black-box attack Chen et al. 2017a) as comparison. The default configuration of untargeted Zoo attack yields very low effectiveness of 18. 20%.. After hyper-parameter tuning and setting the confidence the crafted images the algorithm to 0.99%, Zoo achieves its highest effectiveness of 76.12%. The effectiveness of all attacks higher against the threshold-basee model than the model with a reject class. Our attack needs only one query to the re-trained (student) model because crafts the images using the publicly available pre-trained (teacher) model. Any black-box attack, such as Zoo, that depends only on the student models needs a significant number of queries which is easy defend by limiting the number of queries.
DISCUSSION
In this paper, show that the public information from transfer learning settings can be exploited to fool Softmax-bast classifier. The main vulnerabilities the Softmax layer comes from the fact that it assigns high confidence output to inputs that are far away from the training input distribution. This drawback has been shown in studies that investigate open-set problem Bendale & Boult (2016); Gunther et al. (2017). In other words, Softmax-based models vulnerable to inputs with different
8

---

Published as a conference paper at ICLR 2020
Table 4: Attack Comparison. NQT. NQS. and Ef stands for number of query to the teacher model, number of query to the student model, and effectiveness, respectively.
Attack
Threshold-banee model
With reject class
type
NQT
NQS
Ef(9%%)
NQT
NQS
Ef
Our attack
50,000
87.82%
50,000
1 78.24%
Zoo (black-box)
1,036,800
76.12%
816,800
81.01%
Baseline (random)
12.60%
00.00%
distribution than their training set. To mitigate the problem and defeat our attack, we use recent novel classifier for the open-set problem, called extreme value machine (EVM) Rudd et al. (017).
that aims to fit a distribution to the activation vector (Figure [T) rather than a linear combination
based on Softmax operation. We follow the experimental setting similar to Gunther et al. (2017).
The accuracy of the EVM-based model is 95.60% which is lower than the nottmax-basee model in our experiments, and the model successfully defeat all our crafted images. The main reason that defense mechanism is that the activation vector of our crafted images are far
EVM can be used from the activation vector of any image in the training set. However, EVM has its own vulnerability: find out that by feeding images of random faces (UMass dataset in our study), there is 7.38% chance that the EVM-base model classifies the input as one of the target classes. Hence, more robust model is needed defend our attack and also work well in open-set scenarios.
Another approach to defend the vulnerability of the Softmax layer is to check all elements of acti- vation vector and avoid classification of suspicious inputs. In other words, if an activation element is significantly larger than what it should normally be, we can label the input image as malicious. In our experiment, we find that the average value of the largest neurons in activation vector around 23.86 for normal face images and the largest value we observed is 47.22. So, if we define a threshold for the maximum value in activation vector to be around 50, it is possible to detect crafted images with our attack. In our attack scenario, we craft inputs with the maximum value in activation vector of 1000, which is easily detectable, if checked. We perform an experiment to see our attack work when this value is much smaller and in normal range. By crafting images with max value in activa- tion vector of 50, instead of 1000, the effectiveness of our attack is dramatically reduced ( 0.0007%) However, this threshold may lead to large false positive in inference time. With the max value of 100 and 200, the effectiveness is 0.058% and 0.27%, respectively. Hence, if the threshold value for anomaly detection is chosen meticulously, it can serve as defense mechanism for our attack with the cost of increasing false positive (labeling some natural face images as malicious).
7 CONCLUSION
In this paper, we develop efficient brute force attack transfer learning for deep neural networks the attack exploits a fundamental vulnerability of the Softmax layer that can be casily exploited when transfer learning is used. We assume that the attacker only knows the transferred model and its weights, and does not have access to the re-trained model, the re-trained dataset, and the re-trained model's output. Our evaluations based on face recognition and speech recognition show that with a handful of attempts, the attacker can craft adversarial samples that can trigger all classes despite the fact that the attacker does not know the re-trained model and model's target classes. The target- agnostic feature of the attack allows the attacker to use the same set of crafted images for different re-trained models and achieve high effectiveness when the models use the same pre-traine pre-trained model. The proposed target-agnostic attack reveals fundamental challenge of Softmax layer in transfer learning settings: because the Softmax layer assign high confidence output to vast space of unseen inputs, a simple prute-force attack can operate surprisingly effective. To defeat the target-agnostic attack, the model should consider the distribution of the activation vector, like EVM method, not the linear combination alone, like Softmax layer. Nevertheless, there is fundamental trade-off between accuracy and robustness and it should be tuned based on the sensitivity of the application.
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation (NSF) under Grant CNS-1547461 Grant CNS-1718901, and Grant I1S-1838207.
9

---

Published as a conference paper at ICLR 2020
REFERENCES
Labeled faces in the wild, accessed 24-Mar-2019).
URL
wwt.:wwwww..uma...c..o www.sss.waass....u.....oo fw/
[Online;
Speech commands, 2017.
URL
sequences/ http//eccce/auddooeeeeoeeeeeee recognition
[Online; accessed 24-Mar-2019),
Pannous speech recognition,
2017.
URL lattps://github: wwwp://www.....coooo/a..ooooooao
[Online; accessed 24-Mar-2019].
Michele Alberti, Vinaychandran Pondenkandath, Marcel Wursch, Manuel Bouillon, Mathias Seuret, Rolf Ingold. and Marcus Liwicki. Are you tampering with my data? In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 2018.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017b.
Liang- Chich Chen, Yi Yang. Jiang Wang, Wei Xu, and Alan L Yuille. Attention to scale: Scale- aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3640-3649, 2016.
Pin-Yu Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15-26. ACM, 2017a.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:/712.05526, 2017b.
Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian Good- fellow, and Jascha Sohl-Dickstein. Adversarial examples that fool both human and computer 2018.
vision. arXiv preprint
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sehastian Thrun. Dermatologisttlevel classification of skin cancer with deep neural networks. Nature, 542(7639): 2017.
Manuel Gunther, Steve Cruz, Ethan M Rudd, and Terrance E Boult. Toward open-set face recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 7 71-80, 2017.
Hossein Hosseini, Yize Chen, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Block- ing transferability of adversarial examples in black-box learning systems. arXiv preprint arXiv:1703.04318, 2017.
Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. Model-reuse attacks on deep learning systems. In Proceedings of the 2018 ACM SIGSAC Conference Computer and Com- munications Security, pp. 349-363. ACM, 2018.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances Neural Information Processing Systems, pp. 4485 4495, 2018.
Cong Liao, Haoti Zhong, Anna Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. arXiv preprint arXiv: 10307, 2018.
10

---

Published as a conference paper at ICLR 2020
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chure Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. Proc. NDSS, 2017.
Luis
Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back- trck-grrdientt optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 27-38. ACM, 2017.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427 437,4,,64100 2015.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519. ACM, 2017.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. Deep face recognition. In BMVC, volume 1, pp. 6, 2015.
Shahbaz Rezaei and Xin Liu. How to achieve high classification accuracy with just few labels: A semi- supervised approach using sampled packets. In Advances in Data Mining Applications and Theoretical Aspects, 19th Industrial Conference, {ICDM) 2019, New York, USA, July 17 July 21, pp. 28 42, 2019a.
Shahbaz Rezaei and Xin Liu. Deep learning for encrypted traffic classification: An overview. IEEE communications magazine, 57(5):76-81, 2019b.
Shahbaz Rezaei and Xin Liu. Security of deep learning methodologies: Challenges and opportuni- ties. arXiv preprint
2019c.
Ethan M Rudd, Lalit P Jain, Walter J Scheirer, and Terrance E. Boult. The extreme value machine. IEEE transactions on pattern analysis and machine intelligence, 40(3):662-763 2017.
Ali Shafahi, Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. arXiv preprint 1xXiv:1804.07055 2018
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1528-1540. ACM, 2016.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1312iv:312..611 2013.
Bolun Wang, Yuanshun Yao, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. With great training comes great vulnerability: practical attacks against transfer learning. In 27th {USENIX) Security Symposium (f ({USENIX) Security 18), pp. 1281-1297 2018
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 1oXiv:6609.08144 2016.
A APPENDIX
A.1 CASE STUDY: FACE RECOGNITION
Choice of Initial Image. To generate adversarial images using Algorithm [ we need to start with an initial image. To find out whether the initial image we start with has any impact on the brute force attack, conduct 3 different experiments. We use random input, blank image (with all pixel set to one), and random images of celebrities. The results are shown in Fig. 3 First column shows crafted images starting from the random input. Second column illustrates crafted images from blank image.
11

---

Published as a conference paper at ICLR 2020
(a) Class 0 with random noise
(b) Class 0 with blank image
(c) Intact initial random face
(d) Class 0 with random face
(e) Class 0 train- ing sample
(I) Class random noise
(g) Class with
(h) Intact initial random face
(i) Class with random face
(i) Class train- ing sample
blank image
(k) Class 2 with random noise
(l) Class 2 with blank image
(m) Intact initial random face
(n) Class 2 with random face
(o) Class 2 train- ing sample
(p) Class with random noise
(q) Class 3 with blank image
(r) Intact initial random face
(8) Class 3 with random face
(t) Class train- ing sample
(u) Class 4 with random noise
(v) Class 4 with blank image
(w) Intact initial random face
(x) Class 4 with random face
(y) Class 4 train- ing sample
Figure 5: First column shows crafted images starting from the random input. Second column illus- trates crafted images from blank image. Third and fourth columns show the initial images and the crafted images from the initial image, respectively. The fifth column illustrates sample image from each class that is used for re- re-training.
12

---

Published as a conference paper at ICLR 2020
JSDyCrafted images, Unifomm diinibttion
0.4
0.35
0.8
0.3
0.25
0.2
0.4
0.2
0.05
0.05
0.1
0:15
0.2
Cless
Class Class Class Class
JSD(Training set, Unilorm distribution)
(a) A typical target class distribution
(b) Training dataset vs target class distribution
Figure 6: Target class distribution
Third and fourth columns show the initial images and the crafted images from the initial image, respectively. The fifth column illustrates sample image from each class that is used for re-training- In our experiment, the choice of initial image has negligible impact effectiveness of our attack.
Table5]show the result of using different initial images on attack performance. We only re-train model once with 5 randomly chosen faces and we achieve 99.38% accuracy. Then, we launch the attack on the same model 3 times, each with different initial image. Although using a face image marginally improves the attack performance, the impact is negligible and the other initial input cases are still considerably effective.
Table 5: Impact initial input on the attack Effectiveness(95%))
Initial input
NABAC
Effectiveness(99%))
Blank
18
98.37%
98.37%
Random
19
98.37%
97.22%
A face image
18
99.83%
99.19%
Distribution of Target Classes. Fig. ((a) illustrates a typical distribution of target classes triggered by crafted images of the proposed method. It is clear that the distribution is far from Uniform. basically means that more neurons in layer - are associated with class I and, hence, during brute force attack, more crafted images will trigger that class.
To measure the impact of re-training set the distribution of target classes, distance (JSD). Jensen-Shuan -hannnnn divergence measures the similarity between two distributions as follows:
Jensen-Shanne
JSD(PIIQ) = = D(PIIM) -(QII))))))
(2)
where D(.) is Kullback-Leible divergence and M = J Q). Square root of JSD is metric that we use to compare the similarity between the distribution of data samples in re-training dataset versus the distribution of triggered classes with adversarial inputs of our method.
We find that distribution of training samples during re-training can affect the target class distribu- tion. Fig. b(b) shows the JS distance of training set distribution and Uniform distribution versus JS distance of target class distribution and Uniform distribution. For each data point, we pick 5 random persons from UMass dataset and then re-train the VGG face model with. The line Fig. 6(b) repre- sents the linear regression of all data point. The figure shows that when the training set of re-training phase becomes more non-Uniform, the target class distribution becomes even more non- Uniform.
A.2 CASE STUDY: SPEECH RECOGNITION
In Ji et al. 2018), speech recognition model for digits were re-trained to detect speech commands. Following the same experiment, a model first pre-trained on the Pannous Speech dataset dig (2017) containing utterance of ten digits. Then, randomly pick 5 classes from speech command dataset
13

---

Published as a conference paper at ICLR 2020
Table 6: Effect of number of target classes the proposed attack
# of target classes
Accuracy
NABAC
Effectiveness(955))
Effectiveness(99%%)
97.38%
100.00%
98.21%
10
93.30%
114
95.80%
93.75%
15
85.72%
812
92.22%
84.17%
Table 7: Effect of re-training size
# of samples class
Accuracy
NABAC
(ffeectivenes(955)
Effectiveness(99%))
50
77.56%
13
97.48%
95.00%
100
82.46%
17
97.21%
95.23%
200
85.51%
21
98.25%
96.82%
1000
89.89%
17
98.60%
97.64%
2000
92.04%
17
98.60%
97.81% 81%
com (2017) re-train the model. 80% of the dataset is used for fine-tuning and 20% for inference. Due to the lack of space and similarity of the results with previous case study, we omit most experi- ments with similar results. We use 2D CNN model with 3 building block, each of which contains convolutional layers, Relu activation, and pooling layer, followed by FC layers and softmax layer the end. The input is the Mel-Frequency Cepstral Coefficients (MFCC) of the wave files. Similar to the previous case study, we replace the SM layer and re-train the model by only tuning the last FC and SM layer.
Number of Target Classes. Table shows the impact of number of target classes on the accuracy of the model and attack performance. Similar to face recognition experiment, we start with blank input (a 2D MFCC with 0 for all elements) and we use 70 and 0.1 for k and a, respectively. As expected, the accuracy drops when the number of target classes increases. Since ten classes representing digits exist in both the pre-training dataset (Pannous dig (017)) and the re-training dataset (speech command com (2017)) these classes are much easier for the target model to re-train with high accuracy in comparison with other classes, such stop left command. Hence, the re-trained model has more neuron connections to help classify digit classes which makes it harder for both the model classify the other classes and the proposed attack craft adversarial input for the non-digit classes. That why we observe more dramatic decrease in accuracy and attack performance when the number of target classes increases.
Re-training Sample Size. Unlike face recognition case study in which most re-training classes have fewer than 100 samples, speech command dataset ccom (2017) contains more than 2000 samples for each class. Hence, we conduct an experiment to study the effect of re-training sample size on model and attack performance. We choose six classes (commands) that the pre-trained model did not trained on, i.e., left, right, down, up, go, and stop speech commands. Table shows the impact of re-training set size on the model and attack performance. As expected, increasing the re-training set size improves the accuracy of the model. However, the accuracy of the re-trained model and the re-training set size have a negligible effect on the performance of proposed attack. By comparing Table 6 and Table 7 realize that the attack performance is directly affected by the number of target classes, but it is not significantly affected by the accuracy of the re-tre re-trained model.
14


Reviews:
[{"review_id": "rklYDMstKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed an attack scheme to any model that pretrained from a general model. The merit comes from that the attacker, by taking advantage of the vulnerability of softmax, has no access to examples from the target task to finetune the model.  \n\nPros:\n-\tThe setting where the authors focused on is much more practical, i.e., the attacker is blind to examples in a target task.  \n-\tThe blackdoor for attack in this work, namely the softmax layer, is novel and interesting to me, at least to my knowledge.\n-\tThe paper is well written and easy to follow.\n\nCons:\n-\tIn the experiments, the authors should still compare one or two black-box attacks which also use model outputs only, to demonstrate the effectiveness/efficiency of the proposed attack scheme. \n-\tConsidering the design of \\gamma and \\beta in Eqn. (1), it is expected to investigate the influence of both on the performance of the attack, as well as the vanilla version of Eqn. (1) by only considering MSE.\n-\tCurrently, it seems that the authors did not study the influence of different target datasets to finetune. It is highly expected that the same adversarial input crafted using Alg. 1 could fool all networks finetuned using different datasets. What if a wildly different dataset is used to finetune the pretrained model?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJgHO8bnFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a method of attacking deep neural networks that were trained using transfer learning. The primary claim is that the proposed technique only requires knowledge of the base model (i.e., if the frozen parameters taken from pretrained VGG, ResNET models are known, then that is sufficient) and doesn’t require any samples of target classes of the post-transfer application to successfully attack the DNN.\n\nCore idea is that Softmax layers used in classification can be exploited to craft a perturbation that will fool the network. The algorithm uses a brute force approach to iterate through each neuron on the final layer before the softmax. For each activated neuron, all other neurons are zeroed out and a regularized loss is computed between the activated feature and the neuron. The gradient of the loss function is then used to craft a perturbation using typical k-step gradient based method (as used in FGSM, PGD attacks). In other words, they are trying to craft a perturbation that will zero out all other class probability and put maximum weight on the target adversary class when applied to the network.\n\nExperiments were conducted on VGG-Face and Pannous Speech dataset.\n\n1. Paper is easy to follow and different empirical results show a few intuitive observations for a few different settings. \n\n2. The main issue is that the algorithmic is rather simplistic and seems impractical. Most security applications that motivate the paper can easily defend themselves by simply limiting the number of attempts. Even if this core motivation issue is discarded, the algorithmic contribution is very minimal, no analytical understanding either. \n\n3. Some comparison with Black Box models should be added. Although authors claim that Black Box models would require many queries, a specific comparison and contrasting with number of attempts and queries would help understand the critical advantages of the proposed technique. \n\n4. Table 1 provides a good overview of the results but it should include standard deviations for experiments that were averaged.\n\n5. Is there a notion of imperceptibility/attack budget here? Some of the attacks are clearly not imperceptible. This should be discussed. \n\n6. Section 6 seems a little brief and can use more details. Authors claims that using EVM, “the model successfully defeat all our crafted images” but “there is 7.38% chance that the EVM-base model classifies the input as one of the target classes”. This statement is a little confusing.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gIREuItr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting new adversarial attack concern: any machine learning that use a linear classifier on an off-the-shelf feature extractor, suffers the risk that the feature extractor has adversarial examples that output arbitrary feature vectors. The paper proposes a concrete attack where a set of samples is crafted to activate each feature vector. \n \n\nPro:\n\nThe setup is interesting and seems novel. To my knowledge, attack on the features used in transfer learning has not been considered in the literature. Several adversarial attack papers consider attacking the network layer by layer, or attack an intermediate layer, but they are different from the proposed setup in this paper. \n\nThe proposed attack is simple but effective (though unsurprisingly). \n\nThe experiments show reasonable performance even beyond the motivating setup, for example, with fine-tuning or with non-linear classifiers on the feature space. \n\nCon:\n\nI think the range of the input feature (to the linear classifier) should be reported. A simple check on the range of the input feature should be a good defense. Therefore, the input feature should not be unreasonably large. \n\nI think the experiments need comparison with the naive baseline by simply trying random faces. It seems that without any attack, it should be possible to try random faces and find one that is incorrectly classified. Assuming the model it well calibrated, among the time it says it is 99% confident, it should still make a mistake with approximately 1% probability. In addition, deep networks are known to be not calibrated and over confident. \n\nI do not think the attack is task-agnostic, because the attacker still needs to query the final linear layer as a black box to find which feature unit to activate. This is a special hybrid setup, where the feature layer is white-box while the final layers are black box. I think task-agnostic is somewhat an oversell. \n\n\nOverall I believe this paper is okay. It explores a novel task with a somewhat interesting approach, with sufficient empirical support. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were generally in agreement that the paper presents a valuable contribution and should be accepted for publication. However, I would strongly encourage the authors to carefully read over the reviews and address the suggestions and concerns insofar as possible for the final.
Decision: Accept (Poster)