On the exploitative behavior of adversarial training against adversarial attacks

1 INTRODUCTION . It is known in the literature that adversarial training can make deep neural networks more robust Madry et al . ( 2018 ) ; Shafahi et al . ( 2019 ) ; Wong et al . ( 2019 ) against adversarial attacks Goodfellow et al . ( 2014 ) ; Carlini & Wagner ( 2017 ) ; Moosavi-Dezfooli et al . ( 2016 ) ; Szegedy et al . ( 2013 ) . Arguably , adversarial training can be assumed as one of the most effective techniques for robustness improvement Athalye et al . ( 2018 ) . Moreover , it is empirically shown in Moosavi-Dezfooli et al . ( 2019 ) that adversarial training causes the boundary of the image classifiers to become flatter ( less curved ) compared to normally-trained ones . Adversarial attacks can be executed in the white-box setting Carlini & Wagner ( 2017 ) ; Goodfellow et al . ( 2014 ) ; Moosavi-Dezfooli et al . ( 2016 ) , score-based black-box setting Chen et al . ( 2017 ) ; Ilyas et al . ( 2018 ) ; Narodytska & Kasiviswanathan ( 2016 ) or decision-based black-box setting Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . The attacker ’ s level of information about the classifier plays a key role on the quality of the generated adversarial examples . In order to craft an adversarial perturbation in a decision-based black-box setting , the critical information is mostly the normal vector to the decision boundary . In this setting , the estimation of the normal vector is conducted with carefully designed fine-tuned queries at a boundary point of the image classifier , usually based on the linearization of the boundary . Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . The objective of such black-box attacks is typically to reduce the number of queries as much as possible with an efficient estimate of the normal vector . However , to make sure that this linearization approximation is valid at a boundary point , the ` 2 distance of these queries with the boundary point should be small enough . As a result , such similar queries can be detected using a k-nearest neighbours ( KNN ) similarity detector as in Chen et al . ( 2020 ) . Moreover , the efficient estimation of the normal to the decision boundary heavily relies on the assumption that the boundary of the image classifier has a low mean curvature in the vicinity of input samples Fawzi et al . ( 2016 ) ; Moosavi-Dezfooli et al . ( 2019 ) . Therefore , such estimators are expected to work better if the decision boundary is less curved . Interestingly , it is empirically shown that adversarial training leads to neural networks with flatter decision boundaries , compared to the boundaries learned through regular training methods Qin et al . ( 2019 ) ; Moosavi-Dezfooli et al . ( 2019 ) . We will show that this characteristic of the adversarially-trained networks indeed gives an edge to black-box attacks . The goal of this paper is to show some evidence that although the adversarial training improves the robustness of deep image classifiers effectively against the minimal-norm perturbation whitebox attacks , it becomes less effective in more practical attack settings . In particular , we propose a parallelizable attack against classifiers equipped with an ` 2 norm similarity detector to demonstrate that adversarially-trained networks might even be fooled with smaller ` 2 norm for a given query budget compared to regularly-trained networks due to their excessive linear behavior . That is , decision-based black-box attacks can exploit the excessive flatness caused by adversarial training . In addition , we define a metric called robustness gain as the ratio of ` 2 norm of adversarial perturbation required to fool the adversarially-trained network to that required for the regular network . We observe that the level of information available to the attacker about the classifier impacts the robustness gain ; in particular , the robustness gain increases when the information available to the attacker increases ( e.g. , from black-box to white-box , or by increasing the number of queries in the black-box setting ) . We summarize the contributions of this paper as follows : • We empirically show that there is an interesting trade-off between adversarial training and the attack ’ s effectiveness to fool the classifier . Moreover , we demonstrate that this trade-off is even more critical in certain black-box attacks which rely on the estimation of the normal vector at the boundary . • We introduce a highly parallelizable attack which is effective against a classifier equipped with a query similarity detector based on ` 2 norm . Using normal vectors estimated at multiple points on the boundary , we develop an attack which is , interestingly , more effective against an adversarially-trained network as compared to a regular network . • We define a metric called robustness gain as the ratio of ` 2 norm of adversarial perturbations required for the robust network to that for the regular network . We show that while adversarial training is an effective approach against minimum perturbation white-box attacks , it may not provide a good robustness gain against black-box attacks . The rest of the paper is organized as follows . In Section 3 , the problem setting , the similarity detector , and the multi-point normal vector estimator are introduced . In Section 4 , the performance of our proposed multi-point attack and the state of the art black-box attacks is analyzed for the adversarially-trained network compared to the regular one . In Section 5 , we evaluate the effectiveness of adversarial training against a minimum perturbation white-box attack and finally Section 6 concludes the paper . 2 RELATED WORK . Adversarial Training The basic idea of adversarial training is to create and then incorporate adversarial examples into the training process Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2014 ) . In Madry et al . ( 2018 ) , authors show an effective version of an adversarially-trained network to improve robustness against white-box attacks . In Shafahi et al . ( 2019 ) , the authors proposed a so-called “ free ” version of adversarial training with a cost nearly as equal as natural ( regular ) training . Their key idea is to update both the model parameters and image perturbations using one simultaneous backward-pass . Recently , in Wong et al . ( 2019 ) , the authors discovered that adversarial training can be conducted in a cheaper manner using the fast gradient sign method ( FGSM ) Goodfellow et al . ( 2014 ) added with random initialization . This approach can be useful to adversarially train large datasets such as ImageNet much faster . Adversarial attacks Adversarial attacks can be executed in different categories depending on the attacker ’ s level of information including white-box setting Carlini & Wagner ( 2017 ) ; Goodfellow et al . ( 2014 ) ; Moosavi-Dezfooli et al . ( 2016 ) ; Szegedy et al . ( 2013 ) , score-based black-box setting Chen et al . ( 2017 ) ; Ilyas et al . ( 2018 ) ; Narodytska & Kasiviswanathan ( 2016 ) or decision-based black-box scenarios Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . In order to craft an adversarial example , the critical information is the normal vector to the decision boundary of the classifier . The most successful black-box attacks directly estimate the normal vector with linearization of boundary . For example , the HSJA Chen et al . deploys the gradient direction estimation information . In Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) , authors locally approximated the decision boundary with a hyper-plane , and searched the closest point on the hyper-plane to the benign input as the perturbation . Black-box defenses The generation of an adversarial example requires the black-box attacker to submit multiple similar queries to the target model . Thus , the main idea of most of black-box defences is to detect such a similarity across multiple queries . In Chen et al . ( 2020 ) , authors propose to equip an existing classifier with a detection component , which stores the similarity vectors for all incoming queries , computed by a pre-trained similarity encoder . For each new query , it computes the k-nearest-neighbor distance between it and all other vectors in the memory . Blacklight Li et al . ( 2020 ) computes a compact set of one-way hash values for each query image that form a probabilistic fingerprint . The variants of an image make almost identical fingerprints , which makes it robust against manipulation . In Byun et al . ( 2021 ) , the authors introduce Small Noise Defense ( SND ) in which even a small additive input noise can neutralize most query-based attacks . 3 PROBLEM STATEMENT AND MULTI-POINT ATTACK . Motivation and background One of the most challenging settings to perform adversarial attack to image classifiers is when the attacker only has access to the top-1 label of the classifier , where the attacker ’ s level of information from the image classifier is the least . A query is a request that results in the top-1 label of an image classifier for a given input . The state-of-the-art attacks try to obtain the smallest possible ` p norm of the perturbations with efficient use of queries to the image classifier Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . An implicit assumption here is that the image classifier is naive enough to respond to multiple consecutive similar queries with no complain . This is a strong assumption which is in contrast with the common sense in terms of security . In practice , the defender can take advantage of such characteristic of the queries to detect the suspicious set of queries . In addition , all these attacks perform in an iterative manner which can be time consuming even for a powerful attacker with lots of processing power . Having a parallelizable attack can expedite the running time of the attack which can be critical in certain scenarios that attacker should act as quick as possible . Our goal is to propose a highly parallelizable attack to fool the classifier equipped with an ` 2 similarity detector by taking advantage of the low mean curvature of the decision boundary of state-of-the-art deep classifiers . Similarity detector , simple yet effective defense As in Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) , most of the state-of-the-art black-box attacks generate queries with additive Gaussian or Uniform noises to a boundary point to estimate the normal to the decision boundary . Inherently , these types of estimators need similar , i.e . very close queries in terms of ` 2 distance to make sure the linearization assumption is valid . However , such similar queries can be simply detected by a k nearest neighbour ( KNN ) similarity detector with k N , where N is the query budget for the attacker . Thus , the k nearest neighbours of the given query i are obtained among the queries stored in a buffer . In particular , for a given query i , the classifier computes the average ` 2 distance between the query i and its k nearest neighbors di defined as in Chen et al . ( 2020 ) : di = 1 k k∑ t=1 di , t , ( 1 ) where di , t is the ` 2 distance of the given query iwith its nearest neighbours twhere 1 ≤ t ≤ k. Then , by introducing the detection threshold δ , if di < δ , an attack is detected and the user is blocked . The value for δ should not be too small so that none of the queries get captured and also not too large to detect some clean queries as a false attack . Although , the computational complexity of of the KNN detector may be high , it is quite effective and simple in detecting the queries generated by the techniques deploying normal vector estimation as in HopSkipJump Chen et al. , GeoDA Rahmati et al . ( 2020 ) , Sign-OPT Cheng et al . ( 2019 ) , and qFool Liu et al . ( 2019 ) . We stress here that the goal of the paper is not to outperform the state-of-the-art black-box attacks , but rather to provide certain practical scenarios in which the flatter boundary of the adversarially-trained networks are exploitable by the black-box attacker . Therefore , the question is : Is there a way to design a parallelizable attack to generate a query-efficient ` 2-norm-minimized perturbation with totalN queries against a classifier equipped with a similarity detector of k nearest neighbours with threshold δ ? We show that the answer to this question is affirmative and introduce a multi-point normal estimator which evades the similarity detector . Problem statement We consider a trained L-class classifier with parameters θ represented as f : Rd → RL , where x ∈ Rd is the input image and k̂ ( x ) = argmaxk fk ( x ) is the top-1 classification label where fk ( x ) is the k-th component of f ( x ) corresponding to the k-th class . The attacker ’ s goal is to minimize the ` 2 norm as much as possible with a limited budget of N queries while it can not get detected with the similarity detector . We define the optimization problem as : min v ‖v‖2 ( 2 ) s.t . k̂ ( x+ v ) 6= k̂ ( x ) , di > δ , ∀i ≤ N. The last constraint ensures that the queries are not detected by the ` 2 similarity detector . Without the similarity detector constraint , problem equation 2 is already solved in the literature with different approaches Brendel et al . ( 2018 ) ; Chen et al . ; Cheng et al . ( 2019 ) ; Liu et al . ( 2019 ) ; Rahmati et al . ( 2020 ) . The main idea is to obtain a point on the boundary of the image classifier and estimate the normal to the decision boundary at this point . However , with the presence of the similarity detector , the normal vector estimation becomes more challenging for the attacker as it can not generate large number of queries with small ` 2 norms to the neural network .

Summary: This paper proposes a new black box adversarial examples generation algorithm and uses it to draw conclusions about the decision boundary of adversarially trained models.   The algorithm developed is in response to a recent defense mechanism that stops black box attacks by detecting queries that are similar to each other. Therefore, this work proposes a new method that keeps similar queries to a minimum. The crux of the algorithm’s operation is that it estimates the normal vector to the decision boundary of the model – similar to prior work. Unlike previous papers, however, this one does so by querying at different spots along the decision boundary rather than focusing all the queries on the same point. This helps defeat similarity-based defenses.   The paper then applies this algorithm to both regularly trained networks and adversarially trained ones. This serves as the basis for claims about the geometry of the decision boundaries of each of those network types. To make these claims, it uses two metrics. First, this work observes the magnitude of the perturbation added to starting points in the adversarial example generation process against the number of queries the black box algorithm has used. Second, it introduces a new metric – robustness gain – that computes the ratio of that magnitude for adversarially robust networks and regular networks.   The conclusions of the work are all based on the difference of those metrics between adversarially trained and regular networks. Most interestingly, this paper claims that adversarial training is a weak defense against black box attacks of the type it develops.