On the exploitative behavior of adversarial training against adversarial attacks

1 INTRODUCTION . It is known in the literature that adversarial training can make deep neural networks more robust Madry et al . ( 2018 ) ; Shafahi et al . ( 2019 ) ; Wong et al . ( 2019 ) against adversarial attacks Goodfellow et al . ( 2014 ) ; Carlini & Wagner ( 2017 ) ; Moosavi-Dezfooli et al . ( 2016 ) ; Szegedy et al . ( 2013 ) . Arguably , adversarial training can be assumed as one of the most effective techniques for robustness improvement Athalye et al . ( 2018 ) . Moreover , it is empirically shown in Moosavi-Dezfooli et al . ( 2019 ) that adversarial training causes the boundary of the image classifiers to become flatter ( less curved ) compared to normally-trained ones . Adversarial attacks can be executed in the white-box setting Carlini & Wagner ( 2017 ) ; Goodfellow et al . ( 2014 ) ; Moosavi-Dezfooli et al . ( 2016 ) , score-based black-box setting Chen et al . ( 2017 ) ; Ilyas et al . ( 2018 ) ; Narodytska & Kasiviswanathan ( 2016 ) or decision-based black-box setting Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . The attacker ’ s level of information about the classifier plays a key role on the quality of the generated adversarial examples . In order to craft an adversarial perturbation in a decision-based black-box setting , the critical information is mostly the normal vector to the decision boundary . In this setting , the estimation of the normal vector is conducted with carefully designed fine-tuned queries at a boundary point of the image classifier , usually based on the linearization of the boundary . Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . The objective of such black-box attacks is typically to reduce the number of queries as much as possible with an efficient estimate of the normal vector . However , to make sure that this linearization approximation is valid at a boundary point , the ` 2 distance of these queries with the boundary point should be small enough . As a result , such similar queries can be detected using a k-nearest neighbours ( KNN ) similarity detector as in Chen et al . ( 2020 ) . Moreover , the efficient estimation of the normal to the decision boundary heavily relies on the assumption that the boundary of the image classifier has a low mean curvature in the vicinity of input samples Fawzi et al . ( 2016 ) ; Moosavi-Dezfooli et al . ( 2019 ) . Therefore , such estimators are expected to work better if the decision boundary is less curved . Interestingly , it is empirically shown that adversarial training leads to neural networks with flatter decision boundaries , compared to the boundaries learned through regular training methods Qin et al . ( 2019 ) ; Moosavi-Dezfooli et al . ( 2019 ) . We will show that this characteristic of the adversarially-trained networks indeed gives an edge to black-box attacks . The goal of this paper is to show some evidence that although the adversarial training improves the robustness of deep image classifiers effectively against the minimal-norm perturbation whitebox attacks , it becomes less effective in more practical attack settings . In particular , we propose a parallelizable attack against classifiers equipped with an ` 2 norm similarity detector to demonstrate that adversarially-trained networks might even be fooled with smaller ` 2 norm for a given query budget compared to regularly-trained networks due to their excessive linear behavior . That is , decision-based black-box attacks can exploit the excessive flatness caused by adversarial training . In addition , we define a metric called robustness gain as the ratio of ` 2 norm of adversarial perturbation required to fool the adversarially-trained network to that required for the regular network . We observe that the level of information available to the attacker about the classifier impacts the robustness gain ; in particular , the robustness gain increases when the information available to the attacker increases ( e.g. , from black-box to white-box , or by increasing the number of queries in the black-box setting ) . We summarize the contributions of this paper as follows : • We empirically show that there is an interesting trade-off between adversarial training and the attack ’ s effectiveness to fool the classifier . Moreover , we demonstrate that this trade-off is even more critical in certain black-box attacks which rely on the estimation of the normal vector at the boundary . • We introduce a highly parallelizable attack which is effective against a classifier equipped with a query similarity detector based on ` 2 norm . Using normal vectors estimated at multiple points on the boundary , we develop an attack which is , interestingly , more effective against an adversarially-trained network as compared to a regular network . • We define a metric called robustness gain as the ratio of ` 2 norm of adversarial perturbations required for the robust network to that for the regular network . We show that while adversarial training is an effective approach against minimum perturbation white-box attacks , it may not provide a good robustness gain against black-box attacks . The rest of the paper is organized as follows . In Section 3 , the problem setting , the similarity detector , and the multi-point normal vector estimator are introduced . In Section 4 , the performance of our proposed multi-point attack and the state of the art black-box attacks is analyzed for the adversarially-trained network compared to the regular one . In Section 5 , we evaluate the effectiveness of adversarial training against a minimum perturbation white-box attack and finally Section 6 concludes the paper . 2 RELATED WORK . Adversarial Training The basic idea of adversarial training is to create and then incorporate adversarial examples into the training process Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2014 ) . In Madry et al . ( 2018 ) , authors show an effective version of an adversarially-trained network to improve robustness against white-box attacks . In Shafahi et al . ( 2019 ) , the authors proposed a so-called “ free ” version of adversarial training with a cost nearly as equal as natural ( regular ) training . Their key idea is to update both the model parameters and image perturbations using one simultaneous backward-pass . Recently , in Wong et al . ( 2019 ) , the authors discovered that adversarial training can be conducted in a cheaper manner using the fast gradient sign method ( FGSM ) Goodfellow et al . ( 2014 ) added with random initialization . This approach can be useful to adversarially train large datasets such as ImageNet much faster . Adversarial attacks Adversarial attacks can be executed in different categories depending on the attacker ’ s level of information including white-box setting Carlini & Wagner ( 2017 ) ; Goodfellow et al . ( 2014 ) ; Moosavi-Dezfooli et al . ( 2016 ) ; Szegedy et al . ( 2013 ) , score-based black-box setting Chen et al . ( 2017 ) ; Ilyas et al . ( 2018 ) ; Narodytska & Kasiviswanathan ( 2016 ) or decision-based black-box scenarios Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . In order to craft an adversarial example , the critical information is the normal vector to the decision boundary of the classifier . The most successful black-box attacks directly estimate the normal vector with linearization of boundary . For example , the HSJA Chen et al . deploys the gradient direction estimation information . In Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) , authors locally approximated the decision boundary with a hyper-plane , and searched the closest point on the hyper-plane to the benign input as the perturbation . Black-box defenses The generation of an adversarial example requires the black-box attacker to submit multiple similar queries to the target model . Thus , the main idea of most of black-box defences is to detect such a similarity across multiple queries . In Chen et al . ( 2020 ) , authors propose to equip an existing classifier with a detection component , which stores the similarity vectors for all incoming queries , computed by a pre-trained similarity encoder . For each new query , it computes the k-nearest-neighbor distance between it and all other vectors in the memory . Blacklight Li et al . ( 2020 ) computes a compact set of one-way hash values for each query image that form a probabilistic fingerprint . The variants of an image make almost identical fingerprints , which makes it robust against manipulation . In Byun et al . ( 2021 ) , the authors introduce Small Noise Defense ( SND ) in which even a small additive input noise can neutralize most query-based attacks . 3 PROBLEM STATEMENT AND MULTI-POINT ATTACK . Motivation and background One of the most challenging settings to perform adversarial attack to image classifiers is when the attacker only has access to the top-1 label of the classifier , where the attacker ’ s level of information from the image classifier is the least . A query is a request that results in the top-1 label of an image classifier for a given input . The state-of-the-art attacks try to obtain the smallest possible ` p norm of the perturbations with efficient use of queries to the image classifier Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . An implicit assumption here is that the image classifier is naive enough to respond to multiple consecutive similar queries with no complain . This is a strong assumption which is in contrast with the common sense in terms of security . In practice , the defender can take advantage of such characteristic of the queries to detect the suspicious set of queries . In addition , all these attacks perform in an iterative manner which can be time consuming even for a powerful attacker with lots of processing power . Having a parallelizable attack can expedite the running time of the attack which can be critical in certain scenarios that attacker should act as quick as possible . Our goal is to propose a highly parallelizable attack to fool the classifier equipped with an ` 2 similarity detector by taking advantage of the low mean curvature of the decision boundary of state-of-the-art deep classifiers . Similarity detector , simple yet effective defense As in Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) , most of the state-of-the-art black-box attacks generate queries with additive Gaussian or Uniform noises to a boundary point to estimate the normal to the decision boundary . Inherently , these types of estimators need similar , i.e . very close queries in terms of ` 2 distance to make sure the linearization assumption is valid . However , such similar queries can be simply detected by a k nearest neighbour ( KNN ) similarity detector with k N , where N is the query budget for the attacker . Thus , the k nearest neighbours of the given query i are obtained among the queries stored in a buffer . In particular , for a given query i , the classifier computes the average ` 2 distance between the query i and its k nearest neighbors di defined as in Chen et al . ( 2020 ) : di = 1 k k∑ t=1 di , t , ( 1 ) where di , t is the ` 2 distance of the given query iwith its nearest neighbours twhere 1 ≤ t ≤ k. Then , by introducing the detection threshold δ , if di < δ , an attack is detected and the user is blocked . The value for δ should not be too small so that none of the queries get captured and also not too large to detect some clean queries as a false attack . Although , the computational complexity of of the KNN detector may be high , it is quite effective and simple in detecting the queries generated by the techniques deploying normal vector estimation as in HopSkipJump Chen et al. , GeoDA Rahmati et al . ( 2020 ) , Sign-OPT Cheng et al . ( 2019 ) , and qFool Liu et al . ( 2019 ) . We stress here that the goal of the paper is not to outperform the state-of-the-art black-box attacks , but rather to provide certain practical scenarios in which the flatter boundary of the adversarially-trained networks are exploitable by the black-box attacker . Therefore , the question is : Is there a way to design a parallelizable attack to generate a query-efficient ` 2-norm-minimized perturbation with totalN queries against a classifier equipped with a similarity detector of k nearest neighbours with threshold δ ? We show that the answer to this question is affirmative and introduce a multi-point normal estimator which evades the similarity detector . Problem statement We consider a trained L-class classifier with parameters θ represented as f : Rd → RL , where x ∈ Rd is the input image and k̂ ( x ) = argmaxk fk ( x ) is the top-1 classification label where fk ( x ) is the k-th component of f ( x ) corresponding to the k-th class . The attacker ’ s goal is to minimize the ` 2 norm as much as possible with a limited budget of N queries while it can not get detected with the similarity detector . We define the optimization problem as : min v ‖v‖2 ( 2 ) s.t . k̂ ( x+ v ) 6= k̂ ( x ) , di > δ , ∀i ≤ N. The last constraint ensures that the queries are not detected by the ` 2 similarity detector . Without the similarity detector constraint , problem equation 2 is already solved in the literature with different approaches Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . The main idea is to obtain a point on the boundary of the image classifier and estimate the normal to the decision boundary at this point . However , with the presence of the similarity detector , the normal vector estimation becomes more challenging for the attacker as it can not generate large number of queries with small ` 2 norms to the neural network .

Summary: This paper proposes a query-based black-box adversarial attack that can only query a model with the class output. The main consideration is to avoid sending very similar query points to evade the detection of a KNN-based detector. The proposed method is parallelizable and less likely to be detected by a similarity-based detector.