Policy-Driven Attack: Learning to Query for Hard-label Black-box Adversarial Examples

1 INTRODUCTION . It is widely known that deep neural networks ( DNNs ) are vulnerable to adversarial examples , which are crafted via perturbing clean examples to cause the victim model to make incorrect predictions . In a white-box setting where the adversaries have full access to the architecture and parameters of the victim model , gradients w.r.t . network inputs can be easily calculated via back-propagation , and thus first-order optimization techniques can be directly applied to craft adversarial examples in this setting ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ; Rony et al. , 2019 ) . However , in black-box settings , input gradients are no longer readily available since all model internals are kept secret . Over the past few years , the community has made massive efforts in developing black-box attacks . In order to gain high attack success rates , delicate queries to the victim model are normally required . Recent methods can be roughly categorized into score-based attacks ( Chen et al. , 2017 ; Ilyas et al. , 2018 ; Nitin Bhagoji et al. , 2018 ; Ilyas et al. , 2019 ; Yan et al. , 2019 ; Li et al. , 2020b ; Tu et al. , 2019 ; Du et al. , 2019 ; Li et al. , 2019 ; Bai et al. , 2020 ) and hard-label attacks ( a.k.a , decision-based attacks ) ( Brendel et al. , 2018 ; Cheng et al. , 2019 ; Dong et al. , 2019 ; Shi et al. , 2019 ; Brunner et al. , 2019 ; Chen et al. , 2020 ; Rahmati et al. , 2020 ; Li et al. , 2020a ; Shi et al. , 2020 ; Chen & Gu , 2020 ) , based on the amount of information exposed to the adversaries from the output of victim model . When the prediction probabilities of the victim model are accessible , an intelligent adversary would generally prefer score-based attacks , while in a more practical scenario where only the top-1 class prediction is available , the adversaries will have to resort to hard-label attacks . Since less information is exposed from such feedback of the victim model , hard-label attacks often bare higher query complexity than that of score-based attacks , making their attack process costly and time intensive . * The first two authors contributed equally to the work . Work was done when ZY was an intern at ByteDance AI Lab . In this paper , we aim at reducing the query complexity of hard-label black-box attacks . We cast the problem of progressively refining the candidate adversarial example ( by skillfully querying the victim model and analyzing its feedback ) into a reinforcement learning formulation . At each iteration , we search along a set of chosen directions to see whether there exists any new candidate adversarial example that is perceptually more similar to its benign counterpart , i.e. , in the sense of requiring less distortion . A reward is assigned to each of such search directions ( treated as actions ) , based on the amount of distortion reduction yielded after updating the adversarial example along that direction . Such a reinforcement learning formulation enables us to learn the non-differentiable mapping from search directions to their potential of refining the current adversarial example , directly and precisely . The policy network is expected to be capable of providing the most promising search direction for updating candidate adversarial examples to reduce the required distortion of the adversarial examples from their benign counterparts . As we will show , the proposed policy network can learn from not only the queries that had been performed following the evolving policy but also peer experience from other black-box attacks . As such , it is possible to pre-train the policy network on a small number of query-reward pairs obtained from the performance log of prior attacks ( with or without policy ) to the same victim model . Experiments show that our policy-driven attack ( PDA ) can achieve significantly lower distortions than existing state-of-the-arts under the same query budgets . 2 RELATED WORK . In this paper , we focus on the hard-label black-box setting where only the top-1 decision of the victim model is available . Since less information ( of the victim model ) is exposed after each query , attacks in this category are generally required to query the victim model more times than those in the white-box or score-based settings . For example , an initial attempt named boundary attack ( Brendel et al. , 2018 ) could require ∼million queries before convergence . It proposed to start from an image that is already adversarial , and tried to reduce the distortion by walking towards the benign image along the decision boundary . Recent methods in this category focused more on gradient estimation which could provide more promising search directions , while relying only on top-1 class predictions . Ilyas et al . ( 2018 ) advocated to use NES ( Wierstra et al. , 2014 ; Salimans et al. , 2017 ) to estimate the gradients over proxy scores , and then mounted a variant of PGD attack ( Madry et al. , 2018 ) with the estimated gradients . Towards improving the efficiency of gradient estimation , Cheng et al . ( 2019 ) and Chen et al . ( 2020 ) further introduced a continuous optimization formulation and an unbiased gradient estimation with careful error control , respectively . The gradients were estimated via issuing probe queries from a standard Gaussian distribution . To generate probes from some more powerful distributions , Dong et al . ( 2019 ) proposed to use the covariance matrix adaptation evolution strategy , while Shi et al . ( 2020 ) suggested to use customized distribution to model the sensitivity of each pixel . In contrast to these methods , our PDA proposes to use a policy network which is learned from prior intentions to advocate promising search directions to reduce the query complexity . We note that some works also proposed to exploit DNN models to generate black-box attacks . For example , Naseer et al . ( 2019 ) used DNNs to promote the transferability of black-box attacks , while several score-based black-box attacks proposed to train DNN models for assisting the generation of queries ( Li et al. , 2019 ; Du et al. , 2019 ; Bai et al. , 2020 ) . Our method is naturally different from them in problem settings ( score-based vs hard-label ) and problem formulations . In the autonomous field , Hamdi et al . ( 2020 ) proposed to formulate the generation of semantic attacks as a reinforcement learning problem to find parameters of environment ( e.g. , camera viewpoint ) that can fool the recognition system . To the best of our knowledge , our work is the first to incorporate reinforcement learning into the black-box attacking scenario for estimating perturbation directions , and we advocate the community to consider more about this principled formulation in the future . In addition to the novel reinforcement learning formulation , we also introduce a specific architecture for the policy network which enjoys superior generalization performance , while these methods adopted off-the-shelf auto-encoding architectures . 3 OUR POLICY-DRIVEN ATTACK . We study the problem of attacking an image classifier in the hard-label setting . The goal of the adversaries is to perturb an benign image x ∈ Rn to fool a k-way victim classifier f : Rn → Rk into making an incorrect decision : arg maxi f ( x ′ ) i 6= y , where x′ is the adversarial example generated by perturbing the benign image and y is the true label of x . The adversaries would generally prefer adversarial examples x′ with smaller distortions ‖x− x′‖2 achieved using less queries , since these properties make the attack less suspicious and also save the cost . In this section , we first briefly review some background information that motivate our method ( in Section 3.1 ) , and then detail our reinforcement learning formulation ( in Section 3.2 and Section 3.3 ) and the architecture of our policy network ( in Section 3.4 ) . 3.1 MOTIVATIONS . Most recent hard-label attacks followed a common pipeline of searching from a starting point which was already an adversarial image1 yet not close enough to the benign one . Unlike the white-box and score-based black-box setting in which the input gradients can be calculated and used as the most effective perturbation direction , in the concerned hard-label setting , outputs of the victim model only flip on the decision boundary while keeping constant away from the boundary , making it difficult to evaluate different directions almost everywhere . In this context , the search of promising perturbation directions was restricted into the regions near the decision boundary , since these regions are arguably more informative , and binary search was used to reach the decision boundary efficiently . Let us take a very recent attack named HopSkipJumpAttack ( Chen et al. , 2020 ) as an example . Given the current estimation x′s of the adversarial example at each iteration , HopSkipJumpAttack first performed binary search to project it onto the decision boundary of the victim model . Denote x′ as the updated example that was on the decision boundary already , HopSkipJumpAttack then sampled many probes around x′ from an isotropic Gaussian distribution , and issued these probes to the victim model as queries . The feedback of the victim model was utilized to estimate the gradient direction at x′ , and it was updated along this direction to obtain a new estimation of the adversarial example . This process was repeated many times until the query budget was exhausted . In comparison with boundary attack ( Brendel et al. , 2018 ) , HopSkipJumpAttack was in general far more query-efficient , though a large number of queries had to be consumed for probing the local geometrics of the decision boundary of the victim model . Its superiority came from using the estimated gradient directions as the search directions , which motivated us to explore even better search directions at each iteration of the attack . As will be shown in the appendices , drawing on some geometric insights , we found that the gradient directions are in fact not the optimal search directions in the framework of HopSkipJumpAttack . We also found that the task of performing hardlabel black-box attack could be naturally cast into a reinforcement learning task , thus we attempt to explore the possibility of developing a model-based method for predicting the most promising search directions for attacks . Feedbacks from the victim can provide supervision and thus the policy models in our reinforcement learning framework can be trained/fine-tuned on the fly during each attack process , such that little query is required once the model has been well-trained . 3.2 ATTACK AS REINFORCEMENT LEARNING PROBLEM . In this paper , we consider both targeted attacks and untargeted attacks . Given a benign example x , its label y , and the victim model f , an environment E ( x , y , f ) is naturally formed . The adversaries shall play the role of agent , trying to interact with the environment by issuing queries and collecting feedbacks , under a certain policy . The current example x′t on the decision boundary of the victim model ( or called the candidate adversarial example ) represents the state at each timestamp t. The agent uses a learnable policy network g which will be carefully introduced in Section 3.4 to guide its actions , and the action is to update the candidate adversarial example such that less distortions are required to fool the victim model . The action here incorporates searching along a promising direction at/‖at‖ where at ∈ Rn is sampled from an isotropic Gaussian distribution whose mean vector is given by the policy network µt = g ( x′t , y , y ′ ) ∈ Rn where y′ is the target label , and its covariance matrix is given by Σ = σI ∈ Rn×n , in which the value of σ ∈ R is set to be gradually increased as the attack on each sample progresses , and I ∈ Rn×n indicates the n × n identity matrix . With at , the agent searches along its direction at/‖at‖ to see whether any better candidate adversarial example can be found . For targeted attacks , the target label y′ is chosen by the agent from the beginning and kept unchanged during the attack process . For untargeted attacks , x′t should be on the decision boundary where one side is the ground-truth label y and the other side could be regarded as the “ target label ” y′ . As will be carefully introduced in Section 3.3 , a reward rt ∈ R 1In practice , it is performed by randomly sampling until the adversarial constraint is satisfied , i.e. , it is not classified as y by the victim model , or by directly choosing a benign sample from the adversarial class . Algorithm 1 Policy-Driven Attack Algorithm 1 : Input : the environment E ( x , y , f ) ; the target label y′ , initial adversarial image x′1 ∈ Rn which lies on the decision boundary ; the policy network g. 2 : Output : an adversarial example . 3 : Initialize the step index t← 1 . 4 : while the query count limit not reached do 5 : // Determine the baseline lt to evaluate the potential of different actions 6 : µt ← g ( x′t , y , y′ ) , z ← BS ( x′t + δ · µt‖µt‖2 , x , f ) , where BS ( · , · , · ) performs binary search 7 : Set the distortion reduction of z as baseline : lt ← max { ‖x′t − x‖2 − ‖z − x‖2 , lmin } 8 : 9 : // Collect actions and rewards , and update the policy network 10 : Sample M actions : at , i ∼ N ( µt , σtI ) , i ∈ { 1 , 2 , . . . , M } 11 : Assign rewards rt , i to each actions with our mechanism introduced in Section 3.3 12 : Update the policy network using one-step REINFORCE on M pairs : ( at , i , rt , i ) 13 : 14 : // Update adversarial image using predicted direction 15 : µt ← g ( x′t , y , y′ ) if ‖z − x‖2 ≤ ‖x′t − x‖2 , otherwise µt ← at , i∗ , i∗ = arg maxi rt , i 16 : x′t+1 ← BS ( x′t + · µt‖µt‖2 , x , f ) 17 : 18 : // Update other variables 19 : Double σt if all M rewards are zeros : σt+1 ← 2 · σt ; else keep it : σt+1 ← σt 20 : t← t+ 1 21 : end while 22 : return final adversarial image x′t based on the performance of each action and the corresponding at is given to the agent for updating the parameter of the policy network . All details of our PDA are summarized in Algorithm 1 . Powered by the reinforcement learning framework , we can use policy gradient algorithms to train the policy network g to generate promising search directions in a direct way . For simplicity , we use the one-step REINFORCE ( Williams , 1992 ) in the sequel of this paper and leave the exploration of more advanced policy gradient algorithms to future work . 3.3 REWARD AND ACTION Figure 1 illustrates how we assign the scalar reward rt given current candidate adversarial example x′t and an action at . The decision boundary is illustrated by a horizontal straight line ( denoted by B ) in the figure , the benign counterpart x is assumed to be below B , and the circle C centered at x′t with a small radius δ shows all possible locations after jumping along the directions of some actions by δ from x′t . As described earlier , the reward rt should be assigned based on the amount of potential distortion reduction brought by at . A direct evaluation can be achieved by jumping along the direction of at first and then projecting the updated example back onto the decision boundary via binary search , to see how much improvement is obtained . However , since we evaluateM actions at , i simultaneously at an iteration ( see Algorithm 1 ) and binary search needs to be performed for each of them , and the overall process would be prohibitively ( query- ) expensive . On this point , to efficiently assess the performance of an action , we instead evaluate whether the reduction of distortion by taking a particular action can exceed particular baselines . Concretely , we first evaluate µt = g ( x′t , y , y ′ ) as an action directly by using binary search in a way as just described . Suppose that it can reduce the required adversarial distortion2 by lt , then we setup two levels of baselines ‖x− x′t‖2 − β1 · lt and ‖x−x′t‖2−β2 · lt to see whether other actions can lead to adversarial examples with closer distance ( than these baselines ) from the benign example x , in which β1 = 0 and β2 = 0.25 . As shown in Figure 1 , for an action a ∈ { at , i } , we first obtain x′t + δ ·a/‖a‖2 and then move it towards x to see how much reward it can obtain . The two arcs V1 and V2 indicates where the same progress as the two baselines can be achieved , thus we can further project x′t + δ ·a/‖a‖2 onto the arcs to see if the projections ( i.e , xV1s and x V2 s ) are still adversarial . It can be seen that x V1 s is still adversarial yet x V2 s is not . We assign a reward 1 to such an action a . If both the projections are still adversarial we shall assign a reward of 2 , and if neither of them is adversarial , zero reward is assigned . Since xV1s is not adversarial could imply that xV2s is also not adversarial , such a way reduces the number of queries for assessing each action to at most 2 ( xV1s and x V2 s ) and makes our PDA more query-efficient .

Summary: This paper proposes a new hard-label black-box adversarial attack method based on reinforcement learning. The authors formulate the black-box attacking problem as a reinforcement learning problem, and design a policy network to learn the appropriate attack directions, in order to achieve more efficient attacks. The proposed policy-driven attack (PDA) algorithm is able to craft adversarial examples with lower distortions under the same query budgets. Experiment results show that with a pre-trained policy model, PDA outperforms two baseline methods, Boundary Attack and HopSkipJumpAttack.