Transferable Perturbations of Deep Feature Distributions

1 INTRODUCTION Most recent adversarial attack literature has focused on empirical demonstrations of how classifiers can be fooled by the addition of quasi-imperceptible noise to the input ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ; Moosavi-Dezfooli et al. , 2016 ; Madry et al. , 2018 ; Kurakin et al. , 2017 ) . However , adversarial attacks may be leveraged in other constructive ways to provide insights into how deep learning models learn data representations and make decisions . In this work , we propose a new blackbox transfer-based adversarial attack that outperforms state-of-the-art methods for undefended ImageNet classifiers . Importantly , this work provides a broad exploration into how different Deep Neural Network ( DNN ) models build feature representa- tions and conceptualize classes . The new attack methodology , which we call the Feature Distribution Attack ( FDA ) , leverages class-wise and layer-wise deep feature distributions of a substitute DNN to generate adversarial examples that are highly transferable to a blackbox target DNN . One perspective on adversarial attacks is that adversarial noise is a direction in which to “ move ” the natural data . In standard attacks which directly use the classification output , the noise points in the direction of the nearest decision boundary at the classification layer ( Tramèr et al. , 2017 ) . In this work , our crafted noise points in a direction that makes the data “ look like ” a sample of another class in intermediate feature space . Intuitively , if we can alter the representation in a layer whose features are representative of the data for the given task , but not specific to the model , the adversarial example may transfer better ( to unobserved architectures ) than attacks derived from logit-layer information . Figure 1 ( top ) illustrates the feature distribution modeling of a DNN , which is the core mechanism of the attack . f is a pre-trained substitute whitebox model to which we have full access . The true target blackbox model is not shown , but we only assume limited query access and that it has been trained on ImageNet-1k ( Deng et al. , 2009 ) . Adversarial examples are then generated on the whitebox model and transferred to the blackbox model . The novelty of the attack comes from the explicit use of class-wise and layer-wise feature distributions . In Figure 1 ( top ) , an auxiliary Neural Network ( NN ) gl , c learns p ( y = c|fl ( x ) ) , which is the probability that the layer l features of the whitebox model , extracted from input image x , belong to class c. The attack uses these learned distributions to generate targeted ( or untargeted ) adversarial examples by maximizing ( or minimizing ) the probability that the adversarial example is from a particular class ’ s feature distribution ( Figure 1 ( bottom ) ) . We also use these learned distributions to analyze layer-wise and model-wise transfer properties , and to monitor how perturbations of the input change feature space representations . Thus , we gain insights on how feature distributions evolve with layer depth and architecture . 2 RELATED WORK . In blackbox attacks ( Narodytska & Kasiviswanathan , 2017 ; Su et al. , 2017 ; Papernot et al. , 2017 ; Tramèr et al. , 2017 ; Inkawhich et al. , 2019 ; Dong et al. , 2018 ; Zhou et al. , 2018 ) , knowledge of the target model is limited . In this work , the target model is blackbox in the sense that we do not have access to its gradients and make no assumptions about its architecture ( Madry et al. , 2018 ; Cheng et al. , 2019 ) . A popular blackbox technique is transfer-based attacks , in which adversarial examples are constructed on the attackers ’ own whitebox model and transferred to the target model . Papernot et al . ( 2016 ; 2017 ) develop special methods for training the attackers ’ whitebox model to approximate the target model ’ s decision boundaries . In this work , we only use models that have been trained under standard configurations for ImageNet-1k ( Deng et al. , 2009 ) . Tramèr et al . ( 2018 ) and Liu et al . ( 2017 ) bolster transferability by generating adversarial examples from an ensemble of whitebox models , which helps the noise not overfit a single model architecture . Our methods also discourage overfitting of the generating architecture , but we instead leverage feature space perturbations at the appropriate layer . In the 2017 NeurIPS blackbox attack competition ( Kurakin et al. , 2018 ) , the winning method ( Dong et al. , 2018 ) used momentum in the optimization step , which helped to speed up the convergence rate and de-noise the gradient directions as to not be overly specific to the generating architecture . We also use this approach . Finally , Tramèr et al . ( 2017 ) analyze why transferability occurs and find that well-trained models have similar decision boundary structures . We also analyze transferability , but in the context of how adversarial examples change a model ’ s internal representations , rather than only making observations at the output layer . While all of the above methods generate adversarial examples using information from the classification layer of the model , there have been a few recent works delving into the feature space of DNNs for both attacks and defenses . Sabour et al . ( 2016 ) show that in whitebox settings , samples can be moved very close together while maintaining their original image-domain representations . Zhou et al . ( 2018 ) regularize standard untargeted attack objectives to maximize perturbations of ( all ) intermediate feature maps and increase transferability . However , their primary objective is untargeted and still based on classification output information . Also , the authors do not consider which layers are affected and how the regularization alters the intermediate representations . Inkawhich et al . ( 2019 ) show that driving a source sample ’ s feature representation towards a target sample ’ s representation at particular layers in deep feature space is an effective method of targeted transfer attack . However , the method is targeted only and relies on the selection of a single ( carefully selected ) sample of the target class . Also , the attack success rate on ImageNet was empirically low . This work describes a more robust attack , with significantly better performance on ImageNet , and provides a more detailed analysis of layer-wise transfer properties . For adversarial defenses , Xie et al . ( 2019 ) , Frosst et al . ( 2019 ) , and Lin et al . ( 2019 ) consider the effects of adversarial perturbations in feature space but do not perform a layer-wise analysis of how the internal representations are affected . 3 ATTACK METHODOLOGY . We assume to have a set of training data and a pre-trained model f from the same task as the target blackbox model ( i.e . the ImageNet-1k training set and a pre-trained ImageNet model ) . To model the feature distributions for f , we identify a set of classes C = { c1 , ... , cK } and a set of layers L = { l1 , ... , lN } that we are keen to probe . For each layer in L , we train a small , binary , one-versusall classifier g for each of the classes in C , as shown in Figure 1 ( top ) . Each binary classifier is given a unique set of parameters , and referred to as an auxiliary model gl , c . The output of an auxiliary model represents the probability that the input feature map is from a specific class c ∈ C. Thus , we say that gl , c ( fl ( x ) ) outputs p ( y = c|fl ( x ) ) , where fl ( x ) is the layer l feature map of the pre-trained model f given input image x . Once trained , we may leverage the learned feature distributions to create both targeted and untargeted adversarial examples . Here , we focus mostly on targeted attacks , which are considered a harder problem ( especially in the blackbox transfer case when we do not have access to the target model ’ s gradients ) ( Kurakin et al. , 2018 ; Sharma et al. , 2018 ) . Discussion of untargeted attacks is left to Appendix C. Recall , the goal of a targeted attack is to generate an adversarial noise δ that when added to a clean sample x of class ysrc , the classification result of x+ δ is a chosen class ytgt . The key intuition for our targeted methods is that if a sample has features consistent with the feature distribution of class c at some layer of intermediate feature space , then it will likely be classified as class c. Although not shown in the objective functions for simplicity , for all attacks the adversarial noise δ is constrained by an ` p norm ( i.e . ||δ||p ≤ ) , and the choice of layer l and target class label ytgt are chosen prior to optimization . FDA We propose three targeted attack variants . The most straightforward variant , called FDA , finds a perturbation δ of the “ clean ” input image x that maximizes the probability that the layer l features are from the target class ytgt distribution : max δ p ( y = ytgt|fl ( x+ δ ) ) . ( 1 ) We stress that unlike standard attacks that use output layer information to directly cross decision boundaries of the whitebox , our FDA objective leverages intermediate feature distributions which do not implicitly describe these exact boundaries . FDA+ms In addition to maximizing the probability that the layer l features are from the target class distribution , the FDA+ms variant also considers minimizing the probability that the layer l features are from the source class ysrc distribution ( ms = minimize source ) : max δ λp ( y = ytgt|fl ( x+ δ ) ) − ( 1− λ ) p ( y = ysrc|fl ( x+ δ ) ) . ( 2 ) Here , λ ∈ ( 0 , 1 ) weights the contribution of both terms and is a fixed positive value . FDA+fd Similarly , the FDA+fd variant maximizes the probability that the layer l features are from the target class distribution while also maximizing the distance of the perturbed features from the original features ( fd = feature-disruption ) : max δ p ( y = ytgt|fl ( x+ δ ) ) + η ‖fl ( x+ δ ) − fl ( x ) ‖2 ‖fl ( x ) ‖2 . ( 3 ) In other words , the feature-disruption term , with a fixed η ∈ R+ , prioritizes making the layer l features of the perturbed sample maximally different from the original sample . The additional terms in FDA+ms and FDA+fd encourage the adversarial sample to move far away from the starting point , which may intuitively help in generating ( targeted ) adversarial examples . Also , notice that FDA+ms requires the modeling of both the source and target class distributions , whereas the others only require the modeling of the target class distribution . Optimization Procedure . The trained auxiliary models afford a way to construct a fully differentiable path for gradient-based optimization of the objective functions . Specifically , to compute FDA adversarial noise from layer l , we first build a composite model using the truncated whitebox model fl and the corresponding layer ’ s auxiliary model gl , c=ytgt for the target class ytgt , as shown in Figure 1 ( bottom ) . The loss is calculated as the Binary Cross Entropy ( BCELoss ) between the predicted p ( y = ytgt|fl ( x ) ) and 1 . Thus , we perturb the input image in the direction that will minimize the loss , in turn maximizing p ( y = ytgt|fl ( x ) ) . For optimization , we employ iterative gradient descent with momentum , as the inclusion of a momentum term in adversarial attacks has proven effective ( Inkawhich et al. , 2019 ; Dong et al. , 2018 ) . See Appendix D for more details .

Summary: This paper presents an adversarial attack based on the feature representations at different layers given the classes. Instead of only looking at the final layers, class samples in intermediate feature space information is used to attack and increase transferability at the same time. Then the noise is optimized to perturb the input so that the a specific wrong output will be more likely to chosen. The paper is clear and well-written and different interesting experiments support the claims.