Transferable Perturbations of Deep Feature Distributions
Authors: Nathan Inkawhich
Kevin Liang
Lawrence Carin
Yiran Chen
Venue:  (2020)
Keywords: adversarial attacks
transferability
interpretability

Abstract: Almost all current adversarial attacks of CNN classifiers rely on information derived from the output layer of the network. This work presents a new adversarial attack based on the modeling and exploitation of class-wise and layer-wise deep feature distributions. We achieve state-of-the-art targeted blackbox transfer-based attack results for undefended ImageNet models. Further, we place a priority on explainability and interpretability of the attacking process. Our methodology affords an analysis of how adversarial attacks change the intermediate feature distributions of CNNs, as well as a measure of layer-wise and class-wise feature distributional separability/entanglement. We also conceptualize a transition from task/data-specific to model-specific features within a CNN architecture that directly impacts the transferability of adversarial examples. 

Published as a conference paper at ICLR 2020
TRANSFERABLE PERTURBATIONS OF DEEP FEATURE DISTRIBUTIONS
Nathan Inkawhich, Kevin J Liang, Lawrence Carin & Yiran Chen Department of Electrical and Computer Engineering
Duke University
(nathan. inkawhich, kevin. liang, 1 Iearin, yiran. (aheen@duueeeee
ABSTRACT
Almost all current adversarial attacks of CNN classifiers rely on information de- rived from the output layer of the network. This work presents a new adversarial attack based on the modeling and exploitation of class-wise and layer-wise deep feature distributions. We achieve state-of-the-e targeted blackbox transfer-basee attack results for undefended ImageNet models. Further, we place a priority on explainability and interpretability of the attacking process. Our methodology af- fords an analysis of how adversarial attacks change the intermediate feature dis- tributions of CNNs, as well as measure of layer-wise and class-wise feature distributional separability/emaagleeeen tesk/daaa-specinie to model-specifie features within CNN architecture that di- rectly impacts the transferability of adversarial examples.
We also conceptualize a transition from
INTRODUCTION
Most recent adversarial attack literature has focused empirical demonstrations of how classifiers can be fooled by the addition of
gLenex
p-p(y cxlf(xx))
(X
gle-c
p-p(y (f)
quasi-imperceptible noise the input (Szegedy et al. 2014; Goodfellow et al., 2015 Carlini & Wagner [2017] Moosavi-Dezfool
y=y.re
al. 2016
Madry et al.] 2018 Kurakin et al. 2017). How- ever, adversarial attacks may be leveraged in other constructive ways to provide insights into how deep learning models learn data represen- tations and make decisions. In this work, we propose a new blackbox mansfer-based adver- sarial attack that outperforms sfate-o-thhe-ad art methods for undefended ImageNet classifiers. Importantly, this work provides broad ex- ploration into how different Deep Neural Net- work (DNN) models build feature representa- tions and conceptualize classes. The new attack methodology, which we call the Feature Distri- bution Attack (FDA), leverages class-wise and layer-wise deep feature distributions DNN to generate adversarial examples that are highly transferable to blackbox target DNN.
BCELoss
Pue=y
(GT=1)
ply = (x))
(top) Given a pre-trained whitebox model f. capture the layer-wise and class-wise feature distributions with binary neural networks Il.c> aiming model the probability that the layer features extracted from input x are from the class feature distribution (i.e. p(y c)))( (bot- tom) Forward pass for FDA targeted attack.
Figure 1:
substitute
One perspective on adversarial attacks that adversarial noise is a direction in which to "move" the natural data. In standard attacks which directly use the classification output, the noise points in the direction of the nearest decision boundary at the classification layer (Tramer et al.) 2017). work, our crafted noise points in direction that makes the data "look like" sample of another class intermediate feature space. Intuitively, if we can alter the representation in layer whose features representative of the data for the given task, but not specific to the model, the adversarial example may transfer better (to unobserved architectures) than attacks derived from logit-layer information.
this
Figure [oop) illustrates the feature distribution modeling of DNN, which is the core mechanism of the attack. ff is pre-trained substitute whitebox model to which we have full access. The true target blackbox model is not shown, but only assume limited query access and that it has been

---

Published as a conference paper at ICLR 2020
trained on ImageNet- 1k (Deng et al. 2009). Adversarial examples are then generated on the white- box model and transferred to the blackbox model. The novelty of the attack comes from the explicit use of class-wise and layer-wise feature distributions. In Figure top),, an auxiliary Neural Network (NN) It.. learns poy = cl c(f()))), which is the probability that the layer / features of the whitebox model, extracted from input image x, belong class The attack uses these learned distributions to generate targeted (or untargeted) adversarial examples by maximizing (or minimizing) the probabil- ity that the adversarial example is from particular class's 's feature distribution (Figure [ [bbotto)), We also use these learned distributions to analyze layer-wise and model-wist transfer properties, and to monitor how perturbations of the input change feature space representations. Thus, we gain insights on how feature distributions evolve with layer depth and architecture.
2 RELATED WORK
In blackbox attacks (Narodytska & Kasiviswanathan 2017 Su et al. 2017 Papernot et al. 2017 Tramer et al., 2017 Inkawhich et al., 2019 Dong ct al.) 2018 Zhou et al.
2018), knowledge of the target model is limited. In this work, the target model is blackbox in the sense that we do not have access to its gradients and make no assumptions about its architecture (Madry et al., 2018 Cheng et al. 2019). popular blackbox technique is transfer-based attacks, in which adversarial examples are constructed on the attackers' own whitebox model and transferred to the target model. Papernot et al. (2016) 2017) develop special methods for training the attackers' whitebox model to approximate the target model's decision boundaries. In this work, we only models that have been trained under standard configurations for ImageNet- Ik (Deng et al. 2009). Tramer et al. (2018) and Liu et al. (2177) bolster transferability by generating adversarial examples from ensemble of whitehox models, which helps the noise not overfit a single model architecture. Our methods also discourage overfitting of the generating architecture, but instead leverage feature space perturbations at the appropriate layer. In the 2017 NeurIPS blackbox attack competition (Kurakin) al. 2018), the winning method (Dong et al. 2018) used momentum in the optimization step, which helped to speed up the convergence rate and de-noise the gradient directions as to not be overly specific to the generating architecture. We also use this approach. Finally, [Tramer et al.) (017)) analyze why transferability occurs and find that well-trained models have similar decision boundary structures. We also analyze transferability, but in the context of how adversarial examples change model's internal representations, rather than only making observations at the output layer.
While all of the above methods generate adversarial examples using information from the classifica- tion layer of the model, there have been few recent works delving into the feature space of DNNs for both attacks and defenses. [Sabour et al. (2016) show that in whitebox settings, samples can be moved very close together while maintaining their original image- momae-d representations. Zhou et al. (2018) regularize standard untargeted attack objectives to maximize perturbations of (all) in- termediate feature maps and increase transferability. However, their primary objective is untargeted and still based on classification output information. Also, the authors do not consider which lay- ers are affected and how the regularization alters the intermediate representations. [nkawhich et al. (2019) show that driving source sample's 's feature representation towards target sample's 's repre- sentation at particular layers in deep feature space is an effective method of targeted transfer attack. However, the method is targeted only and relies on the selection of a single (carefully selected) sam- ple of the target class. Also, the attack success rate on ImageNet was empirically low. This work describes a more robust attack, with significantly better performance on ImageNet, and provides more detailed analysis
layer-wise transfer properties. For adversarial defenses, Xiie et al. (2019), Frosst et al. (2019), and Lin al. (2019) consider the effects of adversarial perturbations in feature space but do not perform a layer-wis wise analysis of how the internal representations are affected.
3 ATTACK METHODOLOGY
We assume to have set of training data and pre-trained model f from the same task as the target blackbox model (i.e. the ImageNet-Ik Ik training set and a pre-trained ImageNet model). To model the feature distributions for f. we identify a set of classes C {ci, CK} and a set of layers L= (4....,, Iv} that we are keen to probe. For each layer in L. we train a small, binary, one-versus- all classifier g for each of the classes in C, as shown in I a unique set of parameters, and referred to as an auxiliary model gt.c- The output of an auxiliary model represents the probability that the input feature map is from a specific class c E C. Thus,
Each binary classifier is given
2

---

Published as a conference paper at ICLR 2020
say that (. outputs ply clf((x)), where fi(z) is the layer / feature map of the pre- pre-trained model f given imput image z.
Once trained, we may leverage the learned feature distributions to create both targeted and untar- geted adversarial examples. Here, we focus mostly on targeted attacks, which are considered a harder problem (especially in the blackbox transfer case when do not have access to the target model's 's gradients) (Kurakin et al. 2018) Sharma et al.] 2018). Discussion of untargeted attacks is left to Appendix C. Recall, the goal of targeted attack is generate an adversarial noise ö that when added to clean sample x of class Yarc. the classification result of x x 5 is chosen class Yegt- The key intuition for our targeted methods that if sample has features consistent with the feature distribution of class at some layer of intermediate feature space, then it will likely be classified as class c. Although not shown the objective functions for simplicity, for all attacks the adversarial noise 8 is constrained by an lp norm (i.e. || S c), and the choice of layer I and target class label Vtgt are chosen prior to optimization.
FDA We propose three targeted attack variants. The most straightforward variant, called FDA, finds perturbation 5 of the "clean" input image x that maximizes the probability that the layer I features are from the target class Figt distribution:
max ply = Vigelfi + +)).
(1)
We stress that unlike standard attacks that use output layer information to directly cross decision boundaries of the whitebox, our FDA objective leverages intermediate feature distributions which do not implicitly describe these exact boundaries.
+ms In addition to maximizing the probability that the layer I features are from the target class distribution, the FDA variant also considers minimizing the probability that the layer I features are from the source class Vare distribution (ms = minimize source):
max Xp(y Ytgt fi(x 6)) (1 - A)p(y - fi + 6)).
Here, E (0, 1) weights the contribution of both terms and a fixed positive value.
FDA+fd Similarly, the FDA+fd variant maximizes the probability that the layer features from the target class distribution while also maximizing the distance of the perturbed features from the original features (fd feature-disuupioon):
max ply = Utge fi(x 5)) [i(( + +0) - f(x))22
llf((z)
In other words, the feature- disruptio term, with a fixed n E R4, prioritizes making the layer [ features of the perturbed sample maximally different from the original sample.
The additional terms FDA+ms and FDA+fd encourage the adversarial sample to move far away from the starting point, which may intuitively help in generating (targeted) adversarial examples. Also, notice that FDA+ms requires the modeling of both the source and target class distributions, whereas the others only require the modeling of the target class distribution.
Optimization Procedure. The trained auxiliary models afford a way to construct a fully differen- tiable path for gradient-based optimization of the objective functions. Specifically, compute FDA adversarial noise from layer I, we first build a composite model using the truncated whitebox model fi and the corresponding layer's auxiliary model gr.c= for the target class Vegt, as shown in Fig- (bbotto), The loss is calculated as the Binary Cross Entropy (BCELoss) between the predicted p(y Vigel.f (x))) and 1. Thus, we perturb the input image in the direction that will minimize the loss, in turn maximizing p(y ytgt= fi(x)). For optimization, we employ iterative gradient descent with momentum, as the inclusion
momentum term in adversarial attacks has proven effective Inkawhich et al., 2019 Dong et al. 2018). See Appendix D for more details.
4 EXPERIMENTAL SETUP
ImageNet models. For evaluation use popular CNN architectures designed for the ImageNet- Ik (Deng et al. 2009) classification task: VGG-19 with monnch-ommlizztionn (VGG19) (Simonyan &
Zisserman 2015), DenseNet-121 (DN121) (Huang al. 2017), and ResNet-50 (RN50) (He al.]
3

---

Published as a conference paper at ICLR 2020
2016). All models are pre-trained and found in the PyTorch Model Zoo. Note, our methods are in no way specific to these particular models/architectures.. We also emphasize transfers across different architectures rather than showing results between models from the same family.
Layer decoding scheme. Given a pre-trained model, we must choose a set of layers L to probe. For each model subsample the layers such that probe across the depth. For notation use relative layer numbers, so layer O of DN121 (DN121;=o) is near the input layer and DN121 is closer to the classification layer. For all models, the deepest layer probed is the logit layer. Appendix A decodes the notation for each model.
Auxiliary model training. We must also choose set of classes C that we interested in modeling. Recall, the number of auxiliary models required for a given base model is the number of layers probed multiplied by the number of classes we are interested in modeling. Attempting to model the feature distributions for all 1000 ImageNet classes for each layer is expensive, so we instead choose to run the majority of tests with a set of 10 randomly chosen classes (which are meant to be representative of the entire dataset): grey""""""""" 99;" 245:"bulldog", 344:"hipp"", ccnn:"ann 555: "frr-trucck" 661:"Modell-"";, 701: "01:"parachute". 802:"snowmobbie sign". Thus, for each layer of each model we train 10 auxiliary classifiers, one for each class. After identifying high performing attack settings, we then produce results for all 1000 classes.
"19restreet-
The architecture of all auxiliary models is the same, regardless of model, layer, class. Each is a 2-hidden layer NN with a single output unit. There are 200 neurons in each hidden layer and the number of input units matches the size of the input feature map. To train the auxiliary models, unbiased batches from the whole ImageNet- Ik training set are pushed through the truncated pre- trained model (fi), and the extracted features are used to train the auxiliary model parameters.
Experimental procedure. Since we have three pre-trained models, there are 6 blackbox transfer scenarios evaluate (no self-transfers). We the ImageNet-1K validation set the test dataset. Because FDA+ +ms requires both the source and target class distributions, for the targeted attack eval- uations we only use source samples from the 10 trained classes, and for each sample, target each of the other 9 classes. For baseline attacks, we use targeted random-start Projected Gradient Descent (1pgd) (Madry et al., 2018 Kurakin et al. 20188) targeted NeurIPS2017 competition winning mo- mentum iterative method (tmim) (Dong et al. (018)), and the Activation Attack (AA) (Inkawhich) et al. 2019). Further, all targeted adversarial examples are constrained by tx e = 16/ 255 as de- scribed in (Dong et al. 22118] Kurakin et al. (018). As experimentally found, À = 0.8 in (2) and 7 = 1e-6 in (3). Finally, as measured over the imitially correctly classified subset of the test dataset (by both the whitebox and blackbox models), attack success is captured in two metrics. Error is the percentage of examples that the blackbox misclassifies and Targeted Success Rate (tSuc) is the percentage of examples that the blackbox misclassifies as the target label.
EMPIRICAL RESULTS
5.1 10-CLASS IMAGENET RESULTS
The primary axis of interest is how attack success rate varies with the layer depth from which the feature distributions are attacked. F shows the transfer results between all pairs of whitebox and blackbox models. Each plot shows a metric of attack success versus relative layer depth of the generated attack. The notation DN121 + VGG19 indicates adversarial examples were generated with a DN121 whitebox model and transferred to a VGG19 blackbox model.
Similar to Inkawhich al. (2019), transferability trends for FDAs from a given whitebox model appear blackbox model agnostic (e.g. the shape of the curves from DN121 - RN50 are the same as DN121 - VGG19). This is positive property, as once the optimal transfer layer for a whitebox model is found, evidence shows it will be the same for any blackbox model architecture. Further, the most powerful transfers come from perturbations of intermediate features, rather than perturbations of classification layer information. Another global trend is that in tSuc, +DA+ performs best, FDA performs worst, +ms is in-between, and all FDAs significantly outperform the other baselines. In the error metric, FDA+fd is best in early layers, FDA+ms is best in later layers, and FDA routinely under cnder-performs the AA baseline. Although all attacks are targeted, it is relevant to report error as it is still an indication of attack strength. Also, it is clearly beneficial for the targeted attack objective to include a term that encourages the adversarial example to move far away from its starting place (FDA mss & FDA fd) in addition moving toward the target region.
4

---

Published as a conference paper at ICLR 2020
Error vs. Depth
1Suc vs Deptt
Error vs Depth
ISuc vs Depth
1.0
1.0
DW022 RRN30
0.3
0.6
0.6
0.4
0.4
0.1
0.1
0.2
0.0
0.0
1.0
1.0
0.03
VSG19- RM550
0.8
0.8
0.6
0.4
0.4
0.01
0.2
0.005
0.2
0.000
0.00
1.0
1.0
0.20
0.35
RV50 DD121
0.8
0.8
0.10
0.4
0.4
0.05
0.05
0.2
0.2
0.0
0.00
0.0
0.00
10
10
10
10
tpgd
AA
FDA+
togd
AA
FDA+mS
FDA
Figure 2: Targeted adversarial attack transfer results. The x-axis of each plot is the relative layer depth which the adversarial example was generated from. Each row is different whitebox model.
We now compare performance across whitebox models. For DN121 whitebox, FDA+fd from DN is the optimal targeted attack with an average tSuc of 34%. For DN121 - RN50, this attack outperforms the best baseline by 14% and 32% in eπror and tSuc, respectively. For VGG19 whitebox, FDA+ft from VGG19 is the optimal targeted attack with an average tSuc 2.3%. For VGG19 RN50, this attack outperforms the best baseline by 15% and 2% in error and tSuc, respectively. For a RN50 whitebox, FDA+fd from RN50,= is the optimal targeted attack with an average tSuc of 18%. For RN50 DN121, this attack outperforms the best baseline by 27% and 17% in error and tSuc, respectively.
5.2 1000-CLASS IMAGENET RESULTS
Table Transferability rates for 1000-class targeted attack tests using optimal layers.
DN121
VGG19
DN121 - RN50
RN50
DN121
RN50
VGG19
attack
eиror
tSuc
error
tSuc
error
tSuc
tSuc
tpgd 23.1
0.3
21.4
0.6
20.2
0.5
22.4
0.3
tmim
48.6
1.4
45.5
2.2
44.3
2.9
46.7
1.3
FDA
64.9
15.5
64.3
18.1
56.4
12.6
54.6
6.9
FDA+ms 91.9
21.7
91.9
23.4
87.3
15.9
85.3
10.2
FDA+fd 81.2
29.0
81.7
30,9
82.6
24.3
78.9
15.9
Recall, due to the computational complexity of training one auxiliary model per class per layer per model, we ran the previous experiments using 10 randomly sampled ImageNet- Ik classes. In reality, this may be realistic attack scenario because adversary would likely only be interested in attacking certain source-target pairs. However, to show that the chosen classes are not special, and the previously identified optimal transfer layers are still valid, we train all 1000 class auxiliary models for DN DN121 and R.V50, We exclude VGG19 because in previous tests. Table shows results for the four transfer scenarios. Attack success rates are all averaged over four random 10k splits and the standard deviation of all measurements is less than 1 1%. In these tests, for each source sample, random target class is chosen. As expected, the 1000-class results closely match the previously reported 10-class results.
its inferior performance
ANALYSIS OF TRANSFER PROPERTIES
We now investigate why a given layer and/or whitebox model is better for creating transferable adversarial examples. We also explore the hypothesis that the layer-wise transfer properties of
5

---

Published as a conference paper at ICLR 2020
DNN implicate the transition of intermediate features from task/data-specifee to model-specific. In- tuitively, early layers of DNNs trained for classification may be working to optimally construct a task/dtta-specifie feature set (Zeiler & Fergus, 2014; Yosinski et al. 2014). However, once the nec- essary feature hierarchy is built to model the data, further layers may perform extra processing to best suit the classification functionality of the model. This additional processing may be what makes the features model-specific. We posit that the peak of the iSuc curve for given transfer directly en- codes the inflection point from cask/dta--pecifee to model-specific features in the whitebox model. Instinctively, to achieve targeted attack success, the layer at which the attacks are generated must have captured the concepts of the classes for the general task of classification, without being overly specific to the architecture. Thus, layers prior to the inflection point may not have solidified the class concepts, whereas layers after the inflection point may have established the class concepts and are further processing them for the model output. This may also be considered an extension of the convergent learning theory Li al. 2016) and seneral-to-specin theory Yosinski et al. (2014).
INTERMEDIATE DISRUPTION
One way measure why and how adversarial attacks work is observe how the intermediate representations change as a result of perturbations to the input. Our trained auxiliary models afford novel way monitor the effects of such perturbations deep feature space. To measure how much a layer's features have changed as a result of a (targeted) adversarial perturbation, we define layer- wise disruption as the difference between the target class probability before and after perturbation, as measured in layer 1 of model f: disruption = p(v = Vtgt [i(( 6)) p(u Vige (t)e).((z)).
ssows the average disruption caused each transfer scenario, using both logit-based (tmim) and feature-based (FDA+fd) adversarial attacks. Each row plots the disruption versus layer depth from single whitebox model to each blackbox model (e.g. top row results from DN121 -> VGG19 and DN121 + RN50 transfers). Each line represents the average disruption caused by some adversarial attack, where all FDAs are FDA+fd. The first column of plots shows the impact of each attack on the whitebox model's feature distributions while the second and third columns shows impacts on the blackbox models' feature distributions. Bleckbonc VGG14
Whilebox: DN122
Blackboos: Rh50
tmim
DN121_L-13
0.8
0.8
L-11
DM121_1-9
0.6
0.6
DN121_L-7
DN121_1-5
D-121
0.2
UN121_Le0 L-0
0.0
0.0
Wiitabos:: VGGL9
Blackbox: DN121
Blackbox: RhS0
Imim
0.4
W8019 L-9
0.3
16619_1-7
V6619_1-5
0.6
0.2
0.2
V6619_1-3
V8619_L-0
8.1
0.3
0.2
0.0
0.0
V8619
RN50 L-12
IN6S L-10
RN56 1-6
0.4
0.4
RNSG RNSG_Lad Ltd
0.2
0.2
ANS0_1m2
R-0
0.0
0.0
10 12
12
Figure 3: Disruption versus layer depth for all transfer scenarios. Each row uses different whitebox model. Each line is a different attack, where all FDAs are #DA+fd.
It appears FDAs generated from early layers (e.g. DN121 VGG19 oN disrupt fea- tures the most in early layers and less so in deeper layers. Therefore, a sample resembling class Ytgt in early layer does not mean it will ultimately be classified figi- However, recall from Figure 2 that attacks from early layers create very powerful untargeted adversarial examples (error). This indicates that early layer perturbations amplified they proceed through the model (Lin

---

Published as a conference paper at ICLR 2020
et al. 2019). just not in class-specific manner. Next, expected, attacks that use information from the last layers of a whitebox model (e.g. miim, DN121 -G RN 10 create the largest disruption in the last layers of the whitebox, but not necessarily at the last layers of the blackhox models. However, the optimal transfer attacks (DN12))=7. VGG19 =2550(=)) have high disruption all throughout the models, not just the last layer. This is further evidence that perturbations of classification-laye features are overly model-specific and perturbations of optimal transfer-layer features are more specific to the data/task. Finally, notice that the maximum disruption caused in any blackbox model layer from VGG19 whitebox transfers is around 40% (row 2). For the DN121 and RN50 whitebox models, the maximum disruption is around 80%. This may explain VGGI9 19 being an inferior whitebox model to transfer from, as the perturbation of intermediate VGG features does not in-turn cause significant disruption of blackbox model features.
6.2 AUXILIARY MODEL CORRELATION WITH FULL MODEL
Another point of analysis to investigate the correla- tion/discrepancy between the auxiliary models at given layer and the output of the whitebox model. This may also indicate a transition from task/data-specific to model-specific features. We discuss discrepancy as an indication of how different the auxiliary model outputs are from the whitebox model outputs. Then correlation is the inverse of discrepancy so that when the
DN121
200
VGG19
Correlation
100
auxiliary model outputs align well with the whitebox model outputs, the discrepancy is low and correlation is high. To eval- uate discrepancy
50
given layer I for input æ, aggregate the logit values (i.e. prC-sigmoid/sottma/ for each class in C, as measured by the auxiliary models St.e and the whitebox model f. into separate vectors. Then, a softmax (smax) operation is performed on each vector establish two proper probability distributions over the classes in C. Discrepancy is then defined as the Kullback-Leible divergence (DKL) between the two distributions, or discrepancy = DKL (smax([gt.c (f()) f [=) is the class logit value from the whitebox model f given input Figure 4 shows the layer-wise auxiliary model correlations with the whitebox model outputs as measured from the av- erage discrepancy over 500 input samples of classes in C.
0
6
8
12
Layer Depth
Figure 4: Correlation of layer's auxiliary models with the white- box model output.
smax( smax((f())eel )). Here,
Note, the shapes of the curves are more informative than the actual values. Also, VGG19 layers have been shifted in notation by +4 that layer depth 13 is the logit layer of each model. As expected, the auxiliary models in early layers have little correlation with the model output, while auxiliary models in later layers have high correlation with the model output. Importantly, the optimal transfer layers (+) mark transition in the trendlines after which correlation increases sharply. This effect may directly explain why layers after the optimal layer are suboptimal, because the auxiliary models become highly correlated with the model output and begin to overfit the architecture. Since the auxiliary models are not highly-correlated with the output layer at the raaimaalrrassfer-lyyerr,. we may surmise that the captured features are still mostly task/data specific.
orig
DN121 DN121 L=3 DN121 L=5
DN121 L=7
DN121 L=9 DN121 DN121_L=11
L=13
Figure 5: Saliency maps of auxiliary models several interesting inputs across model depth.

---

Published as a conference paper at ICLR 2020
6.3 AUXILIARY MODEL SALIENCY
For more qualitative analysis, we may inspect the auxiliary model saliency maps. Given an image of class Ysrc. we visualize in Figure 5 what is salient to the Mere auxiliary models at several DN121 layer depths using SmoothGrad (Smilkov et al., 2017) (see Appendix for additional saliency ex- amples for RN50). Notice, an observable transition occurs at the high-performing transfer layers from Figure 2
(DN1211=5,7) The salient regions move from large areas around the whole image (e.g. (2 to specific regions that are also salient in the classification layer (DN1211=1)) The saliency and correlation transitions together show that the well-transferring layers have learned similar salient features as the classification layer while not being overly correlated with the model output. Therefore, perturbations focused on these salient regions significantly impact the final clas- sification without being too specific the generating architecture.
6.4 CLASS DISTRIBUTION SEPARABILITY
Finally, our trained auxiliary models afford new ways of measuring class-wise feature entangllementseeuurbbity for the purpose of explaining transfer performance. We adopt the def- inition of entanglement from Frosst et al. (2019) which states that highly entangled features have a "lack of separation of class manifolds in representation space," and define separa- bility as the inverse of entanglement. One way to measure the separability between class distributions in a layer using the auxiliary models is to gauge how far a sample has to "move" to enter region of high class confidence. We define intra-class distance as the distance a sample has to move to enter a high- confidence region of its source class's distribution. Similarly, we define inter-class distance as the distance a sample has to move to enter high-confidence region of a target class dis- tribution, where Yare * lhgt- Then, separability in a layer is the difference between average inter-class and intra-class distances. In practice, for a given sample, given model, chosen target class, and chosen layer, we iteratively perturb the sample using FDA with a small step size (e-g. 5e-5) until the confidence of the target distribution auxiliary network is over a threshold (e.g. 99.9%) The number of perturbation steps it takes to reach the confidence threshold encodes the distance of the sample to the targeted class's s distribution.
100
DN121
RN50
90
VGG19
Separability
80
70
60
50
40
0
6
8
12
Layer Depth
Figure 6: Class separability versus layer depth for each whitehox.
Results for the three whitebox models are shown in Figure 6 where the vertical axis is the separa- bility in units of perturbation steps, and the horizontal axis is the layer-depth of each model. We see that there is some separability in all layers, for all models, indicating that even features very close to the input layer are somewhat class- speccifc.. Further, VGG19's features are much less separable than DN121 and RN50, indicating why VGG19 may have performed much worse as whitebox model in the transferability tests. In the same vein, DN121 has generally the most separated features which further indicates why it may be superior whitebox model. Intuitively, if a model/layer has highly class-separable feature distributions, FDA attacks may be more transferable because there is less ambiguity between the target class's distribution and other class distributions during generation.
7 CONCLUSIONS
We present a new targeted blackbox transfer-based adversarial attack methodology that achieves state-of-the-art success rates for ImageNet classifiers. The presented attacks leverage learned class- wise and layer-wise intermediate feature distributions of modern DNNs. Critically, the depth at which features are perturbed has a large impact on the transferability of those perturbations, which may be linked to the transition from task/data-specific to model-specific features in an architecture. We further leverage the learned feature distributions to measure the entanglemeen//epparbblly of class manifolds in the representation space and the correlations of the intermediate feature distri- butions with the model output. Interestingly, we find the optimal attack transfer layers have fea- ture distributions that are class-specific and highly-separiiee whitebox model output. We also find that highly transferable attacks induce large disruptions in the intermediate feature space of the blackbox models.
but are not overly-correlate with the
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
The research was supported in part by AFRL (FA8870-18-2-0055)) DARPA, DOE, NIH, NSF and ONR.
REFERENCES
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In JEEE Symposium on Security and Privacy, pp. 39-57. IEEE Computer Society, 2017.
Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial attacks with a transfer-based prior. CoRR, abs/1906. /e///9066069999 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. IEEE Computer Society, 2009.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In CVPR, pp. 9185-9193. IEEE Computer Sociery, 2018.
Nicholas Frosst, Nicolas Papernot, and Geoffrey E. Hinton. Analyzing and improving representa- tions with the soft nearest neighbor loss. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 2012-2020. PMLR, 2019.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778. IEEE Computer Society, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pp. 2261-2269. IEEE Computer Society, 2017.
Nathan Inkawhich, Wei Wen, Hai Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In CVPR. IEEE Computer Society, 2019.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR. OpenReview.net, 2017.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang. Jianfeng Zhu, Xiaolin C. Hu, Cihang Xie, Jianyu Wang. Zhishuai Zhang, Zhou Ren, Alan Loddon Yuille, Sangxia Huang, Yao Zhao, Yuzhe Zhao, Zhonglin Han, Junjiajia Long, Yerkebulan Berdibekov, Takuya Akiba, Seiya Tokui, and Motoki Abe. Adversarial attacks and defences competition. arXiv, abs/1804 NS//100440077 2018.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E. Hoperoft. Convergent learning: Do different neural networks learn the same representations? In ICLR, 2016.
Ji Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness. In ICLR. OpenReview.net, 2019.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. In ICLR. OpenReview.net, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR. OpenReview.net, 2018.
Seyed-Mohsen Moosavi-Dezfoolh Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In CVPR, pp. 2574-2582, IEEE Computer Society, 2016.
Nina Narodytska and Shiva Prasad Kasiviswanathan, Simple black-box adversarial attacks on deep neural networks. In CVPR Workshops, pp. 1310-1318. IEEE Computer Society, 2017.

---

Published as a conference paper at ICLR 2020
Nicolas Papernot, Patrick D. McDaniel, and lan J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv, abs/1605.07277,, 2016.
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Anan- thram Swami. Practical black-box attacks against machine learning. In AsiaCCS, pp. 506-519. ACM, 2017.
Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial manipulation of deep representations. ICLR, 2016.
Yash Sharma, Tien-Dung Le, and Moustafa Alzantot. CAAD 2018: Generating transferable adver- sarial examples. CoRR, abs/1810.012666 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv, abs/1706.03825, 2017.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. arXiv, abs/1710.08864, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Florian Tramer, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. The space of transferable adversarial examples. arXiv, Abs/1104403453, 2017.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In ICLR. OpenReview.net, 2018.
Cihang Xie, Yuxin Wu, Laurens der Maaten, Alan L. Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In CVPR. IEEE Computer Society, 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, pp. 3320-332 33283,22.110 2014.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV (I), volume 8689 of Lecture Notes in Computer Science, pp. 318-833. Springer, 2014.
Wen Zhou, Xin Hou. Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In ECCV (14), volume 11218 Lecture Notes in Com- puter Science, pp. 471 -466. Springer, 2018
10

---

Published as a conference paper at ICLR 2020
APPENDIX
A. LAYER DECODING
Table 2: Whitebox Model Layer Decoding Table
Layer
DenseNet-121
VGG19bn
ResNet-50
0
6,2
512
3,1
6.10
512
3,2
2 6,12
512
3,3
6,12,2
512
3,4
4
6,12,14
512
3,4,1
6,12.20
512
3,4,2
6
6,12,22
512
3,4,3
7
6,12,24
512
3,4,4
6,12,24,2 8
FC2
3,4,5
9
6,12,24,8
FC3
3,4,6
10
6,12,24,12
3,4,6,1
11 6,12,24,14
3,4,6,2
12 6,12.24.16
3,4,6,3
13
6,12,24.16,FC
3,4,6,3,FC
Table 2 the layer number look-up-table that corresponds the layer notation used in the pa- per. DenseNet-121 (DN121), VGG19bn (VGG), and ResNet-50 (RN50) appear because they are model architectures used for the main results. The DN121 notation follows the implemen- tation here: http:::/gttthtttt
com/ www/pott.t..eeeeeeo http:/yytocchvv.i wwwissionblloo /tt////etttttttiieeeeeeeeeeeeeee moodess/densenet.. py english, layer 0 shows that the output of the truncated model comes from the nd denseblock of the 2nd denselayer. Layer 11 means the output of the truncated model comes from the 14th denseblock in the 4th denselayer. Layer 13 indicates the output comes from the final FC layer of the model.
The VGG model does not have denseblocks or dense layers so we use another nota- tion. In the implementation httptts:: / htt.gttthu.u..og www.thh..coooo.eeeooeoooooeeeeee http://pttorrt//iss////bbbb/e matoob/mmater////o torchvist s mode / /gg py the VGG19bn model is constructed from the layer array: [64,64, M'. 128, 128, M', 256, 256, 256, 256,' M'. 512, 512, 512, 512. M' 512, 512, 512, 512, "M', FC1, FC2, FC3), and we follow this convention in the table. In the array. each number cor- responds to a convolutional layer with that number of filters, the M's represent max-pooling layers, and the FCs represent the linear layers at the end of the model. Notice, in these tests we do not consider the first 1I layers of VGG19 as they were shown to have very little impact on classification when perturbed.
The RN50 notation follows the implementation here: httpt:://// //tt:::://ithusse cotp:/torc/.ttttcccooooooooooo.t vision/le onioooolll /
s iiviiiiinnn ooooonommeeee /tw.//ooee.../mm//oooo siteelssreeeees py As designed, the model has 4 layer groups with [3,4,6,3] Bottlenecks in each, respectively. Thus, layer 0 means the output of the truncated model comes from the 1xt Bottleneck of layer group 2. Layer 12 means the output comes from the 3rd Bottleneck of layer group 4, and layer 13 means the output comes from the final FC layer (i.e. output layer) of the model.

---

Published as a conference paper at ICLR 2020
B. FULL TARGETED TRANSFER RESULTS
Figure shows the full targeted attack transfer results from which the Figure 2 were extracted. These full results include two additional metries of attack success. Untargeted Transfer Rate (uTR) is the rate at which examples that fool the whitebox also fool the blackbox (encodes likelihood of misclassification). Targeted Transfer Rate (tTR) is the rate at which successful targeted examples on the whitebox also successful targeted examples the blackbox (encodes likelihood of targeted misclassification. Error and tSuc are described Section 4.
Error Depth
UTR vs. Depth
15uc vi Depth
CTR vs Depth
1.0
1.0
0.4
DI121 -..55
0.8
0.0
0.3
0.6
0.6
0.2
0.4
0.1
1.0
0.3
DV121 +V666
0.2
0.2
0.2
0.0
0.0
0.0
0.0
10
5
10
10
10
0.020
U6619 RN50
0.02
0.010
0.4
0.2
0.2
0.0
0.0
0.000
0.00
1.0
VSG19 -0V22
0.8
0.8
0.6
0.6
0.02
0.4
0.4
0.01
0.2
0.2
1.0
1.0
0.20
0.25
0.9
0.8
0.15
0.20
0.6
0.15
RA50
0.4
0.05
0.05
0.00
0.15
0.3
8050 .VGG19
0.10
0.2
0.05
0.2
0.2
0.0
0.0
0.00
0.0
10
10
10
10
Epgd
Figure 7: Full targeted adversarial attack transfer results. Each row is a unique transfer scenario and each column different attack success metric. The x-axis of each plot the layer depth at which the adversarial example was generated from. Note, top two rows are transfers from DN121 whitebox model, middle two rows are from VGG19 whitebox model, and bottow two rows are from RN50 whitebox.
12

---

Published as a conference paper at ICLR 2020
C. UNTARGETED FEATURE DISTRIBUTION ATTACKS
The goal of an untargeted attack is to generate an adversarial noise å that when added to clean sample of class Vere. the classification result of 8 is not Verc- The key intuition for feature distribution-base untargeted attacks is that if a sample's features are made to be outside of the feature distribution of class Yare at some layer of intermediate feature space, then will likely not be classified Yarc-
uFDA
The first untargeted attack variant is uFDA which is described as
min p(y = Vi(e + 6)).
uFDA minimizes the probability that the layer I features of the perturbed sample x + 5 are from the source class Verc distribution. Unlike the targeted samples which drive towards high confidence regions of a target class feature distribution, this objective drives the sample towards low confidence regions of the source class feature distribution.
uFDA+fd
The second untargeted variant uFDA+ff +fd is described as
min pi = Yare fttrelf((x:))) + 6)) fl + -) f
llf(2))l2
uFDA+fd also carries a feature disruption term so that the objective drives the perturbed sample towards low confidence regions of the source class feature distribution and maximal distance from the original sample's feature representation.
fd-only The final untargeted attack fd-only is described
[1 + 5) ft(z)]12
max 8
[1f((
fotice,fd-only is simply the feature disruption term and is a reasonable standalone untargeted attack objective because making features maximally different may intuitively cause misclassification
To test attack success, we generate untargeted adversarial examples from both DN121 and RN50 whiteboxes and test transfers to a VGG19 blackbox model. It is common evaluate untargeted attacks with a tighter noise constraint (Kurakin et al.) 2018.Dong et al. 2018) as the task is simpler, 16/255 used for targeted
so these tests we use fec e = 4/255 and 8/255 (rather than tests). For baselines, we use the Madry et al. (2018) random start PGD attack (upgd) and the Dong al. (2018) competition winning momentum iterative attack (umim). Similar the layer-wise targeted evaluations, each "clean" source sample belongs to the same set of 10 previously modeled classes. F 8 shows the error rates versus layer depth for the attacks.
-M121- V6G19
1.00
1.00
1.00
1.00
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
upgd
umim
UFDA
UFDA
foorlly
Figure 8: Error versus layer depth plots caused by untargeted adversarial attacks for DN121 > VGG19 and RN50 VGG19 transfer scenarios at two different attack strengths 4, 8.
As expected, the error rate increases with epsilon and the layer depth at which feature-based attacks are generated from has a large impact on attack success rate. In general, uFDA+fC + is the top per- former, followed by fd-only, then uFDA. However, uFDA often under- meder-pesforms the umim baseline, further indicating that for adversarial attacks in feature space it is beneficial to include a term that prioritizes feature disruption (e.g. uFDA+fd & fdlonly)..
On average across models, at = 4/ 4/255, the optimal layer afDA+fd has an untargeted error rate of 37%, which is 9% higher than the best baseline. At 8/255, the optimal layer =FDA+fd has an untargeted error rate of 79%, which is 27% higher than the best baseline. Also, both whitebox models perform similarly in terms of attack success rate, however the performance of fd-only varies
13

---

Published as a conference paper at ICLR 2020
between the two (especially feature map is the optimal attack for the DN121 whitebox (by a small margin). Finally, note that the optimal transfer layers from the targeted attacks (i.e. DN121 and R250(=8) are also high performing layers for the untargeted attacks.
8/255). Surprisingly, fd-onty which simply disrupts the original
D. ADVERSARIAL EXAMPLE GENERATION PROCESS
Recall, because the auxiliary models are NNs, the optimization objectives described for both the targeted and untargeted attacks can be solved with an iterative gradient descent procedure. For any version of the FDA attacks, we first build a "composite" model which includes the truncated whitebox model Si and the appropriate auxiliary model It.c, shown in Figure [bottom). An attack loss function LADA is then defined which includes a BCELoss term and any additional term which trivially incorporated (e.g. the feature disruption term). We then iteratively perturb the source image for K iterations using the sign of a collected momentum term. Similar to Dong et al. (2018) and Inkawhich al. (2019). momentum calculated
V LFDA(T::))
m
mi +
V LFDA (I4 O)lli
where mo 0 and T is the perturbed source image at iteration k. The perturbation method for this Lo constrained attack then
1x+1 Clip(Ia a sign(ms+1);;,.))
In this work, all attacks perturb for K 10 iterations and e/K.
E. ADDITIONAL SALIENCY
orig
RN50 L-0
RN50 L-3
RN50 L-4
RN50 L-8
RN50 L-9
RN50 L-12
RN50 L-13
Figure 9: SmoothGrad saliency maps for RN50 auxiliary models.
14


Reviews:
[{"review_id": "S1gWD_G9Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new adversarial attack for the targeted blackbox model Unlike previous approaches which use the output layer possibly with some additional terms and regularization, the proposed approaches only rely on intermediate features. In fact, the adversarial example is based on a single intermediate layer. The adversarial examples are built by training, for each target class, a binary classifier for the class based only on the features of that layer.\n\n- There are multiple similar prior works that use intermediate layers in some way, and the paper does a good job to explain the differences between the proposed approach and these.\n\nThe paper proposes three variants. The simplest one appears rather weak in numerical experiments. The two other methods seem to achieve a different tradeoff, between focusing on the general error rate or on the targeted success rate.\n\nIn particular, FDA+fd is similar in some ways to Zhou et al. (2018) which uses an additional loss term to maximize the distance of the features at various layers. A comparison with this method (which uses the output layer) would be interesting, and may constitute a relevant additional baseline, at least for non-targeted metrics like error (which is where FDA+ms shines).\n\nSimilarly, a comparison to AA is provided in Figure 2 in the case of 10 classes, but not in Table 1 for the 1000 classes experiment. This casts some doubt on the significance of the result in the 1000 classes setting. A major issue with the proposed method is that training binary classifiers in large multi-class settings is quite costly, which is bound to hinder one's ability to identify 'high performing attack settings' In particular, the paper mentions that the 10 classes are used for that, but no evidence is provided that these settings are actually good for all classes.\n\nThe conclusions are rather consistent accross different pairs of source/target network.\n\nThe paper does not mention adversarial training. As this has become a common defence practice in the literature, it would be interesting to understand to what extent the proposed methods are thwarted by adversarial training. This could increase the significance of the paper if working with intermediate feature representation makes the attack more robust to this type of defense.\n\nIn summary, I find the experiments satisfying, although they can definitely be made more convincing by adding additional strong baselines and demonstrating how much performance can really be attained in the 1000 classes scenario.\n\n- Generally, not using the output layer at all is an interesting approach that deserves in my opinion to be discussed and investigated further.\n\n- The paper is clearly written, and notation is rigorous.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJe0aM3sYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a new transfer attack method (on undefended models), called FDA. The method uses auxiliary class-wise binary classifiers attached on intermediate layers, and optimizes the targeted probability of the binary model. \n\nThe proposed method is simple and shown effective according to the results in Table 1. But I have some serious concerns about the method and results.\n\nConcerns\n1. One weakness is the paper didn't seem to justify why using binary classification models on intermediate features is better than just using the final classification result in the original model. The authors should have an apple-to-apple comparison with optimizing the probability using the original model, including the \"ms\" variant, and also explain why using the auxiliary model is better than that. Even if you use the original model it is still utilizing the distribution of intermediate features, just through the main branch, so this reason alone does not seem convincing to me.\n\n2. The baseline results seem too low (tpgd and tmim in Table 1, <10% target matching success rate). The FDA's result is ~20%. I'm not too familiar with the targeted adversarial attack literature, but one of the very early paper on targeted attack [1] seems to report much higher target matching rate (e.g., see their Table 3, ~70,80%). Why is this the case? What am I missing here? Is this just because they used an ensemble approach? If so you should also use ensemble to compare. Also the authors are welcome to want to point me to more recent papers' results and show their paper's result is better, given [1] is back in ICLR 2017.\n\n3. One drawback of the method is it needs to train a custom model by oneself first, with one binary model for each class we want to attack or target. This can be very cumbersome in practice. For 1000 classes in ImageNet it is not possible to cover all of them. But other conventional methods don't need to worry about this, they just operate on a pre-trained model and supports any source and target class. I doubt whether the method is practically useful. This point should be acknowledged in the paper.\n\nOverall, due to the concerns raised above (especially 2), I tend to vote a reject to the paper. I'm happy to reconsider my rating after the discussion period.\n\n[1] Delving into transferable adversarial examples and black-box attacks.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1l4i1BCtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an adversarial attack based on the feature representations at different layers given the classes. Instead of only looking at the final layers, class samples in intermediate feature space information is used to attack and increase transferability at the same time. Then the noise is optimized to perturb the input so that the a specific wrong output will be more likely to chosen. The paper is clear and well-written and different interesting experiments support the claims.\n\n1-\tIntermediate feature space is not fully independent of the architecture of  the model. However, the results in Figure 2 shows schematically the same behavior for tSuc for different models. I am wondering how different the results would be if adversarial attack was trained on the black box model. In other words, as a simple testing for example, how much higher the success rate would be in the case of DN121->RN50 if you trained the noise on RN50 itself?\n2-\tFor different scenarios explained in figure 2, the optimal layer is usually one of the intermediate ones, and it might change from case to case. To find the optimal layer, you needed to have access to the black box model. How possible do you think it is to make it black-box model agnostic? Maybe at the cost of less success?\n3-\tIn the Figure captions you have “Figure” and in the text it is Fig. please make it consistent.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers black box adversarial attacks based on perturbations of the intermediate layers of a neural network classifier, obtained by training a binary classifier for each target class.

Reviewers were happy with the novelty of the approach as well as the presentation, described the presentation as rigorous and were pleased with the situation of this method relative to the literature. R3 had concerns about evaluation, success rate, and that the procedure was "cumbersome".  Some of their concerns were addressed in rebuttal, but remained steadfast that the method was too cumbersome to be practical.

I agree with R1 & R2 that this approach is novel and interesting and disagree with R3 that it is too impractical. The paper could be stronger with the addition of adversarial training experiments (and I disagree with the authors that "there are currently no whitebox attacks that do well at attacking AT models", this is very much not the case), but I concur with R1 & R2 that this is interesting work that may stimulate further exploration, enough so to warrant acceptance.
Decision: Accept (Poster)