Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data

1 INTRODUCTION . Robustness to adversarial perturbation has become an extremely important criterion for applications of machine learning in security-sensitive domains such as spam detection ( Stringhini et al. , 2010 ) , fraud detection ( Ghosh & Reilly , 1994 ) , criminal justice ( Berk & Bleich , 2013 ) , malware detection ( Kolter & Maloof , 2006 ) , and financial markets ( West , 2000 ) . Systematic methods for generating adversarial examples by small perturbations of original input data , also known as “ attack , ” have been developed to operationalize this criterion and to drive the development of more robust learning systems ( Dalvi et al. , 2004 ; Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . Most of the work in this area has focused on differentiable models with continuous input spaces ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Kurakin et al. , 2016 ) . In this setting , the proposed attack strategies add a gradient-based perturbation to the original input , resulting in a dramatic decrease in the predictive accuracy of the model . This finding demonstrates the vulnerability of deep neural networks to adversarial examples in tasks like image classification and speech recognition . We focus instead on adversarial attacks on models with discrete input data , such as text data , where each feature of an input sample has a categorical domain . While gradient-based approaches are not directly applicable to this setting , variations of gradient-based approaches have been shown effective in differentiable models . For example , Li et al . ( 2015 ) proposed to locate the top features with the largest gradient magnitude of their embedding , and Papernot et al . ( 2016 ) proposed to modify randomly selected features of an input through perturbing each feature by signs of the gradient , and project them onto the closest vector in the embedding space . Dalvi et al . ( 2004 ) attacked such models by solving a mixed integer linear program . Gao et al . ( 2018 ) developed scoring functions applicable for sequence data , and proposed to modify characters of the features selected by the scoring functions . Attack methods specifically designed for text data have also been studied recently . Jia & Liang ( 2017 ) proposed to insert distraction sentences into samples in a human-involved loop to fool a reading comprehension system . Samanta & Mehta ( 2017 ) added linguistic constraints over the pool of candidate-replacing words . Cheng et al . ( 2018 ) applied a gradient-based technique to attack sequence-to-sequence models . We propose a systematic probabilistic framework for generating adversarial examples for models with discrete input . The framework is a two-stage process , where the key features to be perturbed are identified in the first stage and are then perturbed in the second stage by values chosen from a dictionary . We present two instantiations of this framework—Greedy Attack and Gumbel Attack . Greedy Attack evaluates models with single-feature perturbed inputs in two stages , while Gumbel Attack learns a parametric sampling distribution for perturbation . Greedy Attack achieves higher success rate , while Gumbel Attack requires fewer model evaluations , leading to better efficiency in real-time or large-scale attacks . Table 1 systematically compares our methods with other methods . In summary , our contributions in this work are as follows : ( 1 ) We propose a general probabilistic framework for adversarial attacks on models with discrete data . ( 2 ) We show that Greedy Attack achieves state-of-the-art attack rates across various kinds of models . ( 3 ) We propose Gumbel Attack as a scalable method with low model-evaluation complexity . ( 4 ) We observe that character-based models in text classification are particularly vulnerable to adversarial attack . 2 FRAMEWORK . We assume a model in the form of a conditional distribution , Pm ( Y | x ) , for a response Y , supported on a set Y , given a realization of an input random variable X = x ∈ Wd , where W : = { w0 , w1 , . . . , wm } is a discrete space such as the dictionary of words or the space of characters . We assume there exists w0 ∈ W that can be taken as a reference point with no contribution to classification . For example , w0 can be the zero padding in text classification . Let x̃ denote a perturbation of the input variable x . The goal of the adversarial attack is to turn a given input x into x̃ through small perturbations , in such a way that Ỹ = 1 given x̃ , where Ỹ is the indicator of a successful attack : Ỹ | x̃ , x : = 1 { arg maxy Pm ( y | x̃ ) 6= arg maxy Pm ( y | x ) } . We restrict the perturbations to k features of x , and approach the problem through two stages . In the first stage , we search for the most important k features of x . In the second stage , we search for values to replace the selected k features : First stage : x̂ = arg max a∈S1 ( x , k ) P ( Ỹ = 1|a , x ) , ( 1 ) Second stage : x̃ = arg max a∈S2 ( x̂ , x ) P ( Ỹ = 1|a , x ) , ( 2 ) where S1 ( x , k ) : = { a ∈ Wd | ai ∈ { xi , w0 } for all i , d ( a , x ) ≤ k } is a set containing all the elements that differ from x by at most k positions , with the different features always taking value w0 , and S2 ( x̂ , x ) : = { a ∈ Wd | ai = x̂i if x̂i = xi ; ai ∈ W′ otherwise } . Here , we denote by xi , ai , x̂i the ith feature of x , a , x̂ , by d ( a , x ) the count of features different between a and x , and by W′ ⊆W a sub-dictionary of W chosen by the attacker . These two objectives are computationally intractable in general . We thus further propose a probabilistic framework to reformulate the objectives into a more tractable objective , as shown in Figure 1 . Let G be a random variable in Ddk : = { z ∈ { 0 , 1 } d : ∑ zi ≤ k } , the space of d-dimensional zeroone vectors with at most k ones , and let φ : Wd ×Ddk →Wd be a function such that φ ( x , g ) i = xi if gi = 0 and φ ( x , g ) i = w0 if gi = 1 . In the first stage , we let X̂ = φ ( X , G ) where G is generated from a distribution conditioned on X . We further add a constraint on P ( G|X ) , by defining k identical random one-hot random variables G1 , G2 , . . . , Gk ∈ Dd1 conditioned on X , and letting Gi : = maxs { Gsi } , with Gi and Gsi being the ith entries of the variables G and Gs respectively . We aim to maximize the objective P ( Ỹ = 1 | X̂ , X ) over the distribution of G given X , the probability of successful attack obtained by merely masking features : max P ( G|x ) EX [ P ( Ỹ = 1 | X̂ , X ) ] , s.t . Gs i.i.d.∼ P ( · | X ) , X̂ = φ ( X , G ) , Ỹ ∼ P ( Ỹ | X̂ , X ) . ( 3 ) The categorical distribution P ( Gs | x ) yields a rank over the d features for a given x . We define φG : Wd → Pk ( [ d ] ) to be the deterministic function that maps an input x to the indices of the top k features based on the rank from P ( Gs | x ) : φG ( x ) = { i1 , . . . , ik } . In the second stage , we introduce a new random variable H = ( H1 , . . . , Hd ) with each Hi being a one-hot random variable in D|W ′| 1 : = { z ∈ { 0 , 1 } |W ′| : ∑ zi = 1 } . Let Pk ( [ d ] ) be the set of subsets of [ d ] of size k. Let ψ : Wd × ( D|W ′| 1 ) d × Pk ( [ d ] ) → Wd be a function such that ψ ( x , h , φG ( x ) ) i is defined to be xi if i /∈ φG ( x ) , and is the value in W′ corresponding to the one-hot vector hi otherwise . The perturbed input is X̃ : = ψ ( X , H , φG ( X ) ) , whereH is generated from a distribution conditioned on X . We add a constraint on P ( H | X ) by requiring H1 , . . . , Hd to be independent of each other conditioned on X . Our goal is to maximize the objective P ( Ỹ = 1 | X̃ , X ) over the distribution of H given X : max P ( H|x ) EX , G [ P ( Ỹ = 1|X̃ , X ) ] , s.t . H ∼ P ( ·|X ) , X̃ = ψ ( X , H , φG ( X ) ) , Ỹ ∼ P ( Ỹ |X̃ , X ) . ( 4 ) For a given input x , the categorical distribution P ( Hi | x ) yields a rank over the values in W′ to be chosen for each feature i . The perturbation on x is carried out on the top k features φG ( x ) = { i1 , . . . , ik } ranked by P ( Gs | x ) ; each chosen feature is is assigned the top value in W′ selected by P ( His | x ) . 3 METHODS . In this section we present two instantiations of our general framework : Greedy Attack and Gumbel Attack . 3.1 GREEDY ATTACK . We motivate Algorithm 1 , Greedy attack , as optimizing the lower bounds of Problem ( 3 ) and Problem ( 4 ) . Let ei denote the d-dimensional one-hot vector whose ith component is 1 . To solve Problem ( 3 ) , we decompose the objective conditioned on a single instance x as : EG|X [ P ( Ỹ = 1 | X̂ , X ) | x ] = d∑ i=1 P ( G1 = ei | x ) EG ( 1 ) |X , G1 [ P ( Ỹ = 1 | X̂ ) | x , ei ] = d∑ i=1 P ( G1 = ei | x ) EG ( 1 ) |X , G1 [ P ( Ỹ = 1 | φ ( X , max { ei , G ( 1 ) } ) , X ) | x , ei ] , where max takes the elementwise maximum of k random vectors , andG ( 1 ) : = ( G2 , . . . , Gk ) . Thus , the objective in Problem 3 conditioned on a single instance x can be lower bounded as max P ( G|x ) EG|X [ P ( Ŷ = 1 | X̂ , X ) | x ] = max P ( G|x ) d∑ i=1 P ( G1 = ei | x ) EG ( 1 ) |X , G1 [ P ( Ŷ = 1 | φ ( X , max { ei , G ( 1 ) } ) , X ) | x , ei ] = max P ( G1|x ) d∑ i=1 P ( G1 = ei | x ) max P ( G ( 1 ) |x , ei ) EG ( 1 ) |X , G1 [ P ( Ŷ = 1 | φ ( X , max { ei , G ( 1 ) } ) , X ) | x , ei ] ≥ max P ( G1|x ) d∑ i=1 P ( G1 = ei | x ) P ( Ŷ = 1 | x ( i ) ) , ( 5 ) where the last inequality follows from the observation that P ( Ŷ = 1 | x ( i ) ) = P ( Ŷ = 1 | φ ( x , max { ei , G2 = ei , . . . , Gk = ei } ) ) ≤ max P ( G ( 1 ) |x , ei ) EG ( 1 ) |X , G1 [ P ( Ŷ = 1 | φ ( X , max { ei , G ( 1 ) } ) , X ) | x , ei ] We observe that the lower bound ( 5 ) is maximized if P ( G1 = ei | x ) ∝ P ( Ŷ = 1 | x ( i ) ) . ( 6 ) Similarly , we decompose the objective in Problem ( 4 ) by conditioning on Hi1 and employing the independence between G and H conditioned on X , and again use a similar argument : max P ( H|x , g ) EH|X , G [ P ( Ỹ = 1 | X̃ , X ) | x , g ] = max P ( Hi1 |x , g ) |W′|∑ j=1 P ( Hi1 = ej | x , g ) max P ( H ( i1 ) |x , ej ) EH ( i1 ) |X , G , Hi1 [ P ( Ỹ = 1 | X̃ , X ) | x , ei ] ≥ max P ( Hi1 |x , g ) |W′|∑ j=1 P ( Hi1 = ej | x , g ) P ( Ỹ = 1|x ( i1→wj ) ) , ( 7 ) The lower bound ( 7 ) is maximized when P ( Hi1 = ej | x ) ∝ P ( Ỹ = 1 | x ( i1→wj ) ) . ( 8 ) The same applies to i2 , . . . , ik . The algorithm Greedy Attack is built up from Equation ( 6 ) and Equation ( 8 ) in a straightforward manner . See Algorithm 1 for details .

Summary: The authors proposed a novel probabilistic framework to model adversarial attacks on deep networks with discrete inputs such as text. The proposed framework assumes a two step construction of an adversarial perturbation: 1) finding relevant features (or dimensions) to perturb (Eq. 3); 2) finding values to replace the features that are selected in step 1 (Eq. 4). The authors approximate some terms in these two equations to make the optimization easier. For example, it is *implicitly* assumed that given the i-th feature is removed from consideration, the probability of attack success does not change *on average* under probabilistic *adversarial* attack on other features (Eq. 5). It is not clear why that should hold and under what conditions that assumption would be reasonable (given that the attacks on other features are adversarial, although being probabilistic).