Random Noise Defense Against Query-Based Black-Box Attacks

1 Introduction . Deep neural networks ( DNNs ) have been successfully applied in many safety-critical tasks , such as autonomous driving , face recognition and verification , etc . However , it has been shown that DNN models are vulnerable to adversarial examples [ 18 , 21 , 26 , 29 , 48 ] , which are indistinguishable from natural examples but make a model produce erroneous predictions . For real-world applications , the DNN model as well as the training dataset , are often hidden from users . Instead , only the model feedback for each query ( e.g. , labels or confidence scores ) is accessible . In this case , the product providers mainly face severe threats from query-based black-box attacks , which don ’ t require any knowledge about the attacked models . In this work , we focus on efficient defense techniques against query-based black-box attacks , of which the main challenges are 1 ) the defender should not significantly influence the model ’ s feedback to normal queries , but it is difficult to know whether a query is normal or malicious ; 2 ) the defender has no information about what kinds of black-box attack strategies adopted by the attacker . Considerable efforts have been devoted to improving the adversarial robustness of DNNs [ 13 , 34 , 49 , 50 ] . Among them , adversarial training ( AT ) is considered as the most effective defense techniques [ 3 , 49 ] . However , the improved robustness from AT is often accompanied by significant degradation of the †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . clean accuracy . Besides , the training cost of AT is much higher than that of standard training , and then it also often suffers from poor generalization to new samples or adversarial attacks [ 20 , 39 , 45 , 46 ] . Thus , we argue that AT-based defense is not a very suitable choice for black-box defense . In contrast , we expect that a good defense technique should satisfy the following requirements : well keeping clean accuracy , lightweight , and plug-and-play . To this end , we study a lightweight defense strategy , dubbed Random Noise Defense ( RND ) against query-based black-box attacks . For query-based attacks [ 1 , 2 , 7 , 9 , 11 , 19 , 23 , 26 , 27 , 32 , 35 ] , the core is to find an attack direction by gradient estimation or random search based on the exact feedback of consecutive queries , which leads to a decrease of the designed objective . RND is realized by adding random noise to each query at the inference time . Therefore , the returned feedback with randomness results in poor gradient estimation or random search and slows down the attack process . To better understand the effectiveness of RND , we provide a theoretical analysis of the defense performance of RND against query-based attacks and adaptive attacks . Our theoretical results reveal that the defense performance of RND is determined by the magnitude ratio between the noise induced by RND and the noise added by the attackers for gradient estimation or random search . The attack efficiency is significantly affected by a large magnitude ratio . The attackers need more queries to find the adversarial examples or fail to find a successful attack under limited query settings . That is , the larger ratio leads to the better defense performance of RND . Apart from standard attacks , the adaptive attack ( EOT ) [ 3 ] has been considered as an effective strategy to mitigate the random effect induced by the defenders . We also conduct theoretical analysis about the defense effect of RND against EOT attacks , and demonstrate that the magnitude ratio is also crucial to mitigate adaptive attacks and the adaptive attacks have the limited impact of evading the RND . On the other hand , large random noises to each query may lead to degradation of clean accuracy . To achieve a better trade-off between the defense effect and the clean accuracy while maintaining training time efficiency , we further propose combining RND with a lightweight Gaussian augmentation Fine-tuning ( RND-GF ) . RND-GF enables us to adopt larger noise in inference time to disturb the query process better . We conduct extensive experiments on CIFAR-10 and ImageNet . The experimental results verify our theoretical results and demonstrate the effectiveness of RND-GF . It is worth noting that RND is plug-and-play and easy to combine with existing defense methods such as AT to boost the defense performance further . To verify these , we evaluate the performance of RND combined with AT [ 22 ] and find that the RND can improve the robust accuracy of AT by up to 23.1 % against the SOTA black-box attack Square attack [ 2 ] with maintaining clean accuracy . The main contributions of this work are four-fold . • We study a lightweight random noise defense ( RND ) against black-box attacks theoretically and empirically . Our theoretical results reveal that the effectiveness of RND is determined by the magnitude ratio between the noise induced by RND and the noise added by the attackers for gradient estimation and random search . • We theoretically analyze the performance of RND against the adaptive attack ( EOT ) and demonstrate that EOT has the limited effect of evading the RND . • Leveraging our theoretical analysis , we further propose an efficient and stronger defense strategy RND-GF by combining the Gaussian augmentation Fine-tuning and RND towards a better trade-off between clean and adversarial performance . • Extensive experiments verify our theoretical analysis and show the effectiveness of our defense methods against several state-of-the-art query-based attacks . 2 Related Work . Query-based Methods . Here we mainly review the query-based black-box attack methods , which can be categorized into two classes , including gradient estimation and search-based methods . Gradient estimation methods are based on zero-order ( ZO ) optimization algorithms . The attacker utilizes a direct search strategy to find the search direction in search-based methods instead of explicitly estimating gradient . Furthermore , for our theoretical analysis , we focus on score-based queries . The continuous score ( e.g. , the posterior probability or the logit ) for each query is returned , in contrast to the decision-based queries which return hard labels . Specifically , [ 26 ] proposed the first limited query-based attack method by utilizing the Natural Evolutionary Strategies ( NES ) to estimate the gradient . [ 32 ] proposed the ZOsignSGD algorithm , which is similar to NES . Based on Bandit Optimization , [ 27 ] proposed to combine the time and data-dependent gradient prior with gradient estimation , which dramatically reduced the number of queries . For the ` 2 norm constrain , SimBA [ 23 ] randomly sampled a perturbation from orthonormal basis . SignHunter [ 1 ] focuses on estimating the sign of gradient and flipped the sign of perturbation to improve query efficiency . Square attack [ 2 ] is the state-of-art query-based attack method that selects localized square-shaped updates at random positions of images . Black-Box Defense . Compared with the defense for white-box attacks , the defense specially designed for black-box attacks has not been well studied . Two recent works [ 8 , 31 ] proposed to detect malicious queries based on the comparison with the history queries since the malicious queries are more similar with each other compared with normal queries . AdvMind [ 37 ] proposed to infer the intent of the adversary , and it also needed to store the history queries . However , suppose the attacker adopted the strategy of long-interval malicious queries . In that case , the defender has to store a long history , with a very high cost of storage and comparison . [ 42 ] proposed the first black-box certified defense method , dubbed denoised smoothing . [ 5 ] also showed that adding random noise can defend against query-based attacks through experimental evaluations , without theoretical analysis of the defense effect . There are also a few randomization-based defenses . R & P [ 51 ] proposed a random input transform-based method . RSE [ 33 ] added large Gaussian noise to both the input and activation of each layer . PNI [ 25 ] combined adversarial training with adding Gaussian noise to the input or weight of each layer . However , these methods significantly sacrificed the accuracy of benign examples and also have a huge training cost . PixelDP [ 30 ] and random smoothing [ 13 , 41 ] proposed to train classifiers with large Gaussian noise to get certified robustness under the ` 2 norm . However , they require to obtain a majority prediction of this query . Obviously , that places a huge burden on model inference and even helps the attacker get the more accurate gradient estimation . The too large Gaussian noise used in those methods also sacrifices clean accuracy . In contrast , RND maintains a good clean accuracy and only perturbs each query once , without any extra burden . [ 40 ] showed that the model with Gaussian augmentation training achieves the state-of-art defense against common corruptions , while the defense to black-box attacks is not evaluated . 3 Preliminaries . 3.1 Score-based Black-Box Attack . In this work , we mainly focus on the score-based attacks . The analysis and evaluation of decisionbased attacks are shown in supplementary materials . We denote the attacked model asM : X → Y with X being the input space and Y being the output space . Given a benign data ( x , y ) ∈ ( X , Y ) , the goal of adversarial attack is to generate an adversarial example xadv that is similar with x , but to enforce prediction ofM to be different with the true label y ( i.e. , untargeted attack ) or to be a target label h ( i.e. , targeted attack ) . The optimization of untargeted attack can be formulated as follows , min xadv∈NR ( x ) f ( xadv ) = min xadv∈NR ( x ) ( My ( xadv ) −max j 6=y Mj ( xadv ) ) . ( 1 ) For targeted attack , the objective function is maxj 6=hMj ( xadv ) −Mh ( xadv ) .Mj ( xadv ) denotes the logit or softmax output w.r.t . class j. NR ( x ) = { x′| ‖x′ − x‖p ≤ R } indicates a ` p ball around x ( p is often specified as 2 or∞ ) , with R > 0 . In attack evaluation , as long as L ( xadv ) is less than 0 , attackers consider the attack to be successful . The score-based attack algorithms commonly adopt the projected gradient descent algorithms , xt+1 = ProjNR ( x ) ( xt − ηtg ( xt ) ) . ( 2 ) For white-box attacks , g ( xt ) is the gradient of f ( x ) w.r.t . xt . However , for black-box attacks , the gradient g ( xt ) can not be directly obtained . So the attackers utilize the gradient estimation or random search to conduct the update g ( xt ) . 3.2 Zero-Order Optimization for Black-Box Attack . Zero-order optimization ( ZO ) has become the mainstream of black-box attacks , dubbed ZO attacks . The general idea of ZO attack methods is to estimate the gradient according to the objective values returned by querying . The gradient estimator widely used in score-based and decision-based attacks [ 7 , 10 , 12 , 12 , 16 , 26 , 27 , 32 , 36 ] is gµ ( x ) = f ( x+ µu ) − f ( x ) µ u , ( 3 ) where u ∼ N ( 0 , I ) , and µ ≥ 0 . Based on this gradient estimator , the update of attack becomes xt+1 = ProjNR ( x ) ( xt − ηtgµ ( xt ) ) . ( 4 )

Summary: This paper proposes a simple but theoretically supported defense method against query-based black-box attack, called as Random Noise Defense. The core idea is to add Gaussian noise to a given input image at each query so it can be considered as quite simple. However, this paper shows theoretical analyses of RND against zero-order optimization, adaptive and search-based attacks. The overall finding through each theoretical analysis is that the number of queries for successful attacks by an attacker changes with the ratio of nu/mu. To overcome trade-off between clean accuracy and larger nu, the authors propose Gaussian augmentation fine-tuning for the target model. From the extensive experiments using CIFAR-10 and ImageNet against various black-box attacks, the authors demonstrated that, not only the results of theoretical analysis are confirmed by experiments but RND achieves the superior performance. Also, they showed RND can be easily combined with other defense methods and it achieves the state-of-the-art defense performance when it is combined with the adversarially-trained model.