Luring of transferable adversarial perturbations in the black-box paradigm

1 INTRODUCTION . Neural networks based systems have been shown to be vulnerable to adversarial examples ( Szegedy et al. , 2014 ) , i.e . maliciously modified inputs that fool a model at inference time . Many directions have been explored to explain and characterize this phenomenon ( Schmidt et al. , 2018 ; Ford et al. , 2019 ; Ilyas et al. , 2019 ; Shafahi et al. , 2019 ) that became a growing concern and a major brake on the deployment of Machine Learning ( ML ) models . In response , many defenses have been proposed to protect the integrity of ML systems , predominantly focused on an adversary in the white-box setting ( Madry et al. , 2018 ; Zhang et al. , 2019 ; Cohen et al. , 2019 ; Hendrycks et al. , 2019 ; Carmon et al. , 2019 ) . In this work , we design an innovative way to limit the transferability of adversarial perturbation towards a model , opening a new direction for robustness in the realistic black-box setting ( Papernot et al. , 2017 ) . As ML-based online API are likely to become increasingly widespread , and regarding the massive deployment of edge models in a large variety of devices , several instances of a model may be deployed in systems with different environment and security properties . Thus , the black-box paradigm needs to be extensively studied to efficiently protect systems in many critical domains . Considering a target model M that a defender aims at protecting against adversarial examples , we propose a method which allows to build the model T , an augmented version of M , such that adversarial examples do not transfer from T to M . Importantly , training T only requires to have access to M , meaning that no labeled data set is required , so that our approach can be implemented at a low cost for any already trained model . T is built by augmenting M with an additional component P ( with T = M ◦ P ) taking the form of a neural network trained with a specific loss function with logit-based constraints . From the observation that transferability of adversarial perturbations between two models occurs because they rely on similar non-robust features ( Ilyas et al. , 2019 ) , we design P such that ( 1 ) the augmented network exploits useful features of M and that ( 2 ) non-robust features of T and M are either different or require different perturbations to reach misclassification towards the same class . Our deception-based method is conceptually new as it does not aim at making M relying more on robust-features as with proactive schemes ( Madry et al. , 2018 ; Zhang et al. , 2019 ) , nor tries to anticipate perturbations which directly target the non-robust features ofM as with reactive strategies ( Meng & Chen , 2017 ; Hwang et al. , 2019 ) . Our contributions are as follows : • We present an innovative approach to thwart transferability between two models , which we name the luring effect . This phenomenon , as conceptually novel , opens a new direction for adversarial research . • We propose an implementation of the luring effect which fits any pre-trained model and does not require a label data set . An additional neural network is pasted to the target model and trained with a specific loss function that acts on the logits sequence order . • We experimentally characterize the luring effect and discuss its potentiality for black-box defense strategies on MNIST , SVHN and CIFAR10 , and analyze the scalability on ImageNet ( ILSVRC2012 ) . For reproducibility purposes , the code is available at https : //anonymous.4open . science/r/3c64e745-927d-4f51-b187-583e64586ff6/ . 2 LURING ADVERSARIAL PERTURBATIONS . 2.1 NOTATIONS . We consider a classification task where input-label pairs ( x , y ) ∈ X × Y are sampled from a distribution D. |Y| = C is the cardinality of the labels space . A neural network model Mφ : X → Y , with parameters φ , classifies an input x ∈ X to a label M ( x ) ∈ Y . The pre-softmax output function of Mφ ( the logits ) is denoted as hM : X → RC . For the sake of readability , the model Mφ is simply noted as M , except when necessary . 2.2 CONTEXT : ADVERSARIAL EXAMPLES IN THE BLACK-BOX SETTING . Black-box settings are realistic use-cases since many models are deployed ( in the cloud or embedded in mobile devices ) within secure environments and accessible through open or restrictive API . Contrary to the white-box paradigm where the adversary is allowed to use existing gradient-based attacks ( Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ; Chen et al. , 2018 ; Dong et al. , 2018 ; Madry et al. , 2018 ; Wang et al. , 2019 ) , an attacker in a black-box setting only accesses the output label , confidence scores or logits from the target model . He can still take advantage of gradient-free methods ( Uesato et al. , 2018 ; Guo et al. , 2019 ; Su et al. , 2019 ; Brendel et al. , 2018 ; Ilyas et al. , 2018 ; Chen et al. , 2020 ) but , practically , the number of queries requires to mount the attack is prohibitive and may be flagged as suspicious ( Chen et al. , 2019 ; Li et al. , 2020 ) . In that case , the adversary may take advantage of the transferability property ( Papernot et al. , 2017 ) by crafting adversarial examples on a substitute model and then transfering them to the target model . 2.3 OBJECTIVES AND DESIGN . Our objective is to find a novel way to make models more robust against transferable black-box adversarial perturbation without expensive ( and sometimes prohibitive ) training cost required by many white-box defense methods . Our main idea is based on classical deception-based approaches for network security ( e.g . honeypots ) and can be summarized as follow : rather than try to prevent an attack , let ’ s fool the attacker . Our approach relies on a network P : X → X , pasted to the already trained target network M before the input layer , such as the resulting augmented model will answer T ( x ) = M ◦ P ( x ) when fed with input x . The additional component P is designed and trained to reach a twofold objective : • Prediction neutrality : adding P does not alter the decision for a clean example x , i.e . T ( x ) = M ◦ P ( x ) = M ( x ) ; • Adversarial luring : according to an adversarial example x′ crafted to fool T , M does not output the same label as T ( i.e . M ◦P ( x′ ) 6= M ( x′ ) ) and , in the best case , x′ is inefficient ( i.e . M ( x′ ) = y ) . To explain the intuition of our method , we follow the feature-based framework proposed in Ilyas et al . ( 2019 ) where a feature f is a function from X to R , that M has learned to perform its predictions . Considering a binary classification task , f is said to be ρ-useful ( ρ > 0 ) if it satisfies Equation 1a and is γ-useful robust if under the worst perturbation δ chosen in a predefined set of allowed perturbations ∆ , f stays γ-useful under this perturbation ( Equation 1b ) . A ρ-useful feature f is said to be a non-robust feature if f is not robust for any γ ≥ 0 . ( a ) E ( x , y ) ∼D [ y · f ( x ) ] > ρ ( b ) E ( x , y ) ∼D [ inf δ∈∆ y · f ( x+ δ ) ] > γ ( 1 ) An adversary which aims at fooling a model will thus perform perturbations to the inputs to influence the useful features which are not robust with respect to the perturbation he is allowed to apply . We denote F∗M the set of ρ-useful features learned by M . We consider a set of allowed perturbations ∆ and γ > 0 such that we note F∗ , RM and F ∗ , NR M respectively the set of γ-robust and non-robust features learned by M ( relatively to ∆ ) . An adversary that aims at fooling M ◦P will alter function compositions of the form f ◦ P with f ∈ F∗M . These function compositions are the non-robust useful features of M ◦ P , whose set is denoted F∗ , NRM◦P . Based on the observations that transferability of adversarial perturbations between two models occurs because these models rely on similar non-robust features ( Ilyas et al. , 2019 ) , we consider f ◦ P ∈ F∗ , NRM◦P , and we derive two possibilities which ensure the lowest transferability between M ◦ P and M , regarding the robustness of f . If f is robust , f ∈ F∗ , RM , that means the adversarial perturbations from ∆ is sufficient to flip the augmented feature f ◦ P but is not efficient to directly impact f . This case is the optimal one , since the adversarial example is unsuccessful on the target model ( M ◦P ( x′ ) 6= y and M ( x′ ) = y ) . On the other hand , if f ∈ F∗ , NRM ( i.e . both f and f ◦ P are non-robust ) , restraining the transferability means that the additional model P impacts the way useful features vary with respect to input alterations so that the adversarial perturbation lead to two different labels . We encompass these two cases ( illustrated in Figure 1 ) within what we name the luring effect . The adversary is tricked into modifying input values in some way to flip useful and non-robust features of M ◦ P and these modifications are either without effect on the useful features of M , or flip the non-robust features of M in a different way ( and therefore are detectable , as presented in Section 4 ) . 2.4 TRAINING THE LURING COMPONENT . To reach our two objectives ( prediction neutrality and adversarial luring ) , we propose to train P with constraints based on the predicted labels order . For x ∈ X , let α and β be the labels corresponding respectively to the first and second highest confidence score given to x byM . The training of P is achieved with a new loss function that constraints α to ( still ) be the first class predicted by M ◦P ( prediction neutrality ) and that makes the logits gap between α and β the highest as possible for M ◦ P ( adversarial luring ) . To understand the intuition behind this loss function , let ’ s formalize the concepts learned by M and M ◦ P . The prediction given by M corresponds to ” class α is predicted , class β is the second possible class ” . Once P has been trained following this loss function , the prediction given byM ◦P corresponds to ” class α is predicted , the higher confidence given to class α , the smaller confidence given to class β ” . Concepts learned byM andM ◦P share the same goal of prediction , i.e . ” class α ” is predicted , but the relation between class α and class β is forced to be the most different as possible . As learned concepts are essentially different , then useful features learned by M and M ◦P to reach these concepts are necessarily different , and consequently display different types of sensitivity to the same input pixel modifications . In other words , as the direction of confidence towards classes is forced to be structurally different for M ◦ P and M , we hypothesize that useful features of the two classifiers should be different and behave differently to adversarial perturbations . The luring loss , designed to induce this behavior , is given in Equation 2 and the complete training procedure is detailed in Algorithm 1 . The parameters of P are denoted by θ , x ∈ X is an input and M is the target model . M has already been trained and its parameters are frozen during the process . hM ( x ) and hM◦P ( x ) denote respectively the logits of M and M ◦ P for input x. hMi ( x ) and hM◦Pi ( x ) correspond respectively to the values of h M ( x ) and hM◦P ( x ) for class i . The classes a and b correspond to the second maximum value of hM and hM◦P respectively . L ( x , M ( x ) ) =− λ ( hM◦PM ( x ) ( x ) − h M◦P a ( x ) ) +max ( 0 , hM◦Pb ( x ) − h M◦P M ( x ) ( x ) ) ( 2 ) The first term of Equation 2 optimizes the gap between the logits of M ◦ P corresponding to the first and second biggest unscaled confidence score ( logits ) given by M ( i.e . M ( x ) and a ) . This part formalizes the goal of changing the direction of confidence betweenM ◦P andM . The second term is compulsory to reach a good classification since the first part alone does not ensure that hM◦PM ( x ) ( x ) is the highest logit value ( prediction neutrality ) . The parameter λ > 0 , called the luring coefficient , allows to control the trade-off between ensuring good accuracy and shifting confidence direction . Algorithm 1 Training of the luring component Input : trained model M , training steps K , learning rate η , batch size B , luring coefficient λ Output : luring component P , with parameters θ 1 : hM◦Pc ( x ) denotes the logits of M ◦ P for input x and class c 2 : Randomly initialize the parameters θ of P 3 : for step = 1 . . .K : do 4 : { x1 , x2 , . . . , xB } : batch of training set examples 5 : for i = 1 . . . B : do 6 : ( a , b ) ← ( class of the 2nd max value of hM ( xi ) , class of the 2nd max value of hM◦P ( xi ) ) 7 : L ( xi , M ( xi ) ) ←−λ ( hM◦PM ( xi ) ( xi ) − h M◦P a ( xi ) ) +max ( 0 , h M◦P b ( xi ) − h M◦P M ( xi ) ( xi ) ) 8 : end for 9 : θ← θ − η ∑B i=1∇θL ( xi , M ( xi ) ) /B 10 : end for 11 : return P

Summary: In this paper, the authors present a novel approach to evade the transferability of adversarial examples between two models. Specifically, they design a luring loss to train model T, an augmented version of M, where the adversarial examples cannot transfer from T to M. The luring loss is designed to reach a twofold objective: (1) for a clean example, T and M yield the same prediction (2) for an adversarial example, T and M yield different predictions, and in the best case, M can provide correct prediction. The proposed approach can serve as a defense for both detect adversarial examples and defend adversarial examples. Experimental results show that the proposed defense can detect and defend adversarial examples better that the three baselines. In general, the paper is clearly written and easy to follow.