Learn2Weight: Weights Transfer Defense against Similar-domain Adversarial Attacks

1 INTRODUCTION . As machine learning models are applied to more and more real-world tasks , addressing machine learning safety is becoming an increasingly pressing issue . Deep learning algorithms have been shown to be vulnerable to adversarial examples ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Papernot et al. , 2016a ) . In particular , prior black-box adversarial attacks assume that the adversary is not aware of the target model architecture , parameters or training data , but is capable of querying the target model with supplied inputs and obtaining the output predictions . The phenomenon that adversarial examples generated from one model may also be adversarial to another model is known as adversarial transferability ( Szegedy et al. , 2013 ) . Motivated by adversarial transferability , we conjecture another black-box attack pipeline where the adversary does not even need to have access to the target model nor query labels from crafted inputs . Instead , as long as the adversary knows the task of the target , he can choose a similar domain , to build a substitute model and then attack the target model with adversarial examples that are generated from the attack domain . The similar-domain adversarial attack may be more practical than prior blackbox attacks as label querying from target model is not needed . This attack can be illustrated with the following example ( Figure 1b ) in medical insurance fraud ( Finlayson et al. , 2019 ) . Insurance companies may use hypothetical opioid risk models to classify the likelihood ( high/low ) of a patient to abuse the opioids to be prescribed , based on the patient ’ s medical history as text input . Physicians can run the original patient history through the attack pipeline to generate an adversarial patient history , where the original is more likely to be rejected ( ” High ” risk ) and the adversarial is more likely to be accepted ( ” Low ” risk ) . Perturbations in patient history could be , for example , a slight perturbation from ” alcohol abuse ” to ” alcohol dependence ” , and it may successfully fool the insurance company ’ s model . Based on domain adaption theory ( Ben-David et al. , 2010 ) , we conjecture that it is the domain-variant features that cause the success of the similar-domain attack . The adversarial examples with domainvariant features are likely to reside in the low-density regions ( far away from decision boundary ) of the empirical distribution of the target training data which could fool the target model ( Zhang et al. , 2019b ) . Literature indicates that worsened generalizability is a tradeoff faced by existing defenses such as adversarial training ( Raghunathan et al. , 2019 ) and domain generalization techniques ( Wang et al. , 2019 ) . In trying to increase robustness against adversarial inputs , a model faces a tradeoff of weakened accuracy towards clean inputs . Given that an adversarial training loss function is composed of a loss against clean inputs and loss against adversarial inputs , improper optimization where the latter is highly-optimized and the former weakly-optimized does not improve general performance in the real-world . To curb this issue , methods have been proposed ( Zhang et al. , 2019b ; Lamb et al. , 2019 ; Schmidt et al. , 2018 ) , such as factoring in under-represented data points in training set . To defend against this similar-domain adversarial attack , we propose a weight transfer network approach , Learn2Weight , so that the target model ’ s decision boundary can adapt to the examples from low-density regions . Experiments confirm the effectiveness of our approach against the similardomain attack over other baseline defense methods . Moreover , our approach is able to improve robustness accuracy without losing the target model ’ s standard generalization accuracy . Our contribution can be summarized as follows : • We are among the first to propose the similar-domain adversarial attack . This attack pipeline relaxes the previous black-box attack assumption that the adversary has access to the target model and can query the model with crafted examples . • We propose a defensive strategy for this attack based on domain adaptation theory . Experiment results show the effectiveness of our approach over existing defense methods , against the similar-domain attack . 2 RELATED WORK . Recent work in adversarial attack for NLP systems has attracted attention . See ( Zhang et al. , 2020 ) survey for an overview of the adversarial attack in NLP . Existing research proposes different attack methods for generating adversarial text examples ( Moosavi-Dezfooli et al. , 2016 ; Ebrahimi et al. , 2018 ; Wallace et al. , 2019 ) . The crafted adversarial text examples have been shown to fool the state-of-the-art NLP systems such as BERT ( Jin et al. , 2019 ) . A large body of adversarial attack research focuses on black-box attack where the adversary builds a substitute model by querying the target model with supplied inputs and obtaining the output predictions . The key idea behind such black-box attack is that adversarial examples generated from one model may also be mis-classified by another model , which is known as adversarial transferability ( Szegedy et al. , 2013 ; Cheng et al. , 2019 ) . While prior work examines the transferability between different models trained over the same dataset , or the transferability between the same or different model trained over disjoint subsets of a dataset , our work examines the adversarial transferability between different domains , which we call similar-domain attack . 3 SIMILAR-DOMAIN ADVERSARIAL ATTACK . 3.1 ADVERSARIAL ATTACK BACKGROUND . Adversarial attacks modify inputs to cause errors in machine learning inference ( Szegedy et al. , 2013 ) . We utilize the basic gradient-based attack method Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2014 ) and its variants , RAND-FGSM ( Tramèr et al. , 2017 ) and Basic Iterative Method ( BIM ) ( Kurakin et al. , 2016a ; b ; Xie et al. , 2018 ) . Other NLP adversarial generation algorithms could also be used , such as DeepFool ( Moosavi-Dezfooli et al. , 2016 ) , HotFlip ( Ebrahimi et al. , 2018 ) , universal adversarial trigger ( Wallace et al. , 2019 ) , and TextFooler ( Jin et al. , 2019 ) . To perform gradient-based perturbations upon discrete space data , we follow ( Yang et al. , 2018 ) to generate adversarial text . Our proposed similar-domain adversarial attack is in-variant to adversarial algorithm , meaning that the adversarial algorithm used would not affect the attack performance . Without losing generality , we denote Adv ( f , x ) as an NLP adversarial text generation method , defined as below . Definition 1 . NLP Adversarial Generation . Given a deep neural network model f built on text data X , an NLP adversarial generation method produces one adversarial instances x′ ← Adv ( f , x ) for x ∈ X , x′ ≈ x . The goal of the adversarial attack is to deviate the label to incorrect one f ( x′ ) 6= f ( x ) . 3.2 SIMILAR-DOMAIN ADVERSARIAL ATTACK . We present the architecture of similar-domain adversarial attack in Figure 1a . The defender , the target of the attack , constructs a target model trained on domain text data T ( 0 ) . An attacker , only having a rough idea about the target ’ s task but lacking direct access to the target data or target model parameters , collects attack data from a similar domain S and trains an attack model ( 1 ) . He runs the attack model on the test data ( 2 ) to obtain correctly-classified instances ( 3 ) . He chooses an adversarial attack algorithm and generates a set of adversarial samples A ( 4 ) . He exposes A to the target model , hoping A mislead the target model to produce an output of his choice ( 5 ) . This type of attack works best as an adversarial attack that compromises systems that base decision-making on one-instance . Definition 2 . Similar-domain Adversarial Attack . A target model f , built on target domain data T , is a deep neural network model with parameter weights WT that maps a text instance to a label : y ← f ( X , WT ) . An adversary chooses a source attack domain S , builds a substitute model fS , and generates a set of adversarial examples A from S using Adv ( fS , S ) , so that during an attack f ( A , WT ) = fS ( A ) . 3.3 DOMAIN SIMILARITY . Here , domain similarity refers to the similarity between attacker ’ s chosen domain and defender ’ s domain . SharedVocab measures the overlap of unique words , in each of the datasets ; a higher degree of overlapping vocabulary implies the two domains are more similar . We also use Transfer Loss , a standard metric for domain adaptation Blitzer et al . ( 2007 ) ; Glorot et al . ( 2011 ) , to measure domain similarity ; lower loss indicates higher similarity . The test error from a target model trained on target domain T and evaluated on attack domain S returns transfer error e ( S , T ) . The baseline error e ( T , T ) term is the test error obtained from target model trained on target domain ( train ) data T and tested on target domain ( evaluation ) data T . This computes the transfer loss , tf ( S , T ) = e ( S , T ) − e ( T , T ) . 4 IS THE ATTACK EFFECTIVE ? . 4.1 SETUP . Dataset . We simulate the similar-domain adversarial attack using Amazon ’ s multi-domain sentiment classification dataset ( Blitzer et al. , 2007 ) , a commonly-used dataset in cross-domain sentiment classification1 , with 1,000 positive and 1,000 negative reviews for each of the 25 product categories . Model . In practice , there could be unlimited choice for the attack model and target model , such as different deep learning architecture , different training parameters . To simplify the discussion , we choose Long Short-Term Memory ( LSTM ) network as a suitable baseline sentiment classification model ( Wang et al. , 2018 ) for our target model and attack model . The architecture consists of 64 LSTM cells , 80 % dropout , using a sigmoid activation function . Metrics . We first report the accuracy of the target models on the target domain test samples before the attack as the original accuracy . Then we measure the accuracy of the target models against adversarial samples crafted from the attack domain samples , denoted as the after-attack accuracy . Intra-attack accuracy denotes the after-attack accuracy where the attack domain is identical to the target domain . By comparing original and after-attack accuracy , we can evaluate the success of the attack . The greater the gap between the original and after-attack accuracy , the more successful the attack . Unperturbed accuracy measures the accuracy of the target model against the complete , unperturbed test set of the attack domain , to demonstrate that any drop in classification accuracy is not from domain shift alone but from adversarial transferability . 4.2 RESULTS . The similar-domain adversarial attack results are presented in Table 2 . We see a significant gap between original accuracy and after-attack accuracy indicating that this attack can impose valid threat to a target NLP system . After the similar-domain adversarial attack , the accuracy drops dramatically by a large margin . Take the book target domain for example , when the attack domain is magazine , the after-attack accuracy drops to 0.398 , and when the attack domain is baby , the accuracy is 0.421 . Moreover , we observe a positive correlation between transfer loss and after-attack accuracy , and a negative correlation between shared vocab and after-attack accuracy . 1Data is available from https : //www.cs.jhu.edu/˜mdredze/datasets/sentiment/

Summary: This paper is about generating adversarial examples for some target model and protecting from such attacks.  Authors consider a setting when an adversary has access to some "similar to target " domain data, and can use this data to generate a surrogate model. Using this surrogate model an adversary can generate adversarial examples, that apparently also fool the target model. Then authors also propose a defense mechanism from this type of attack, Learn2Weight. This is a learnt network that, for a given example, returns perturbation of weights to the target model which will be applied to the target before inference. This model is trained by a defender on synthetic domains generated as perturbations to the target data