ScaleCert: Scalable Certified Defense against Adversarial Patches with Sparse Superficial Layers

1 Introduction . Despite the promising opportunities and great success of deep neural network ( DNN ) techniques in computer vision tasks [ 9 , 13 ] , DNN techniques are vulnerable to adversarial attacks that craft the input data with adversarial perturbations to manipulate the output [ 2 , 20 , 26 ] . Recent studies reach a consensus that compared to the adversarial example attack , adversarial patch attack is more practical and well-demonstrated in physical environment with good effectiveness and transferability in both classification and detection systems [ 5 , 14 , 15 , 23 , 25 , 28 ] . The key difference between the adversarial example attack and patch attack is the constraint of perturbation . For adversarial example attack , the adversary perturbs all pixels in the image with the ` p-norm of the perturbation in prescribed bounds , while the adversarial patch attack only changes the pixels in a confined region without bounded constraints but free to choose the values . Consider the real-world condition , it ’ s hard to define the ` p-norm constraint of the perturbation due to the environment , viewpoint and object variation . Hence , the adversarial patch attack is much more appropriate to be mounted in the physical world . The success of practical adversarial patch attacks raises the importance and urgency of defense methodologies . Existing defending methodologies can be classified into two categories , heuristic defenses and certified defenses . For heuristic defense approaches , they have good efficiency ∗ Equal contribution . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . but lack robustness against a strong adaptive attack . For example , digital watermarking ( DW ) [ 8 ] utilizes the magnitude of the saliency maps to detect unusually dense regions and mask them out of the input . Local gradient smoothing ( LGS ) [ 19 ] pre-processes the image gradient of the classification function with a normalization and a thresholding step and then suppress the adversarial noise based on the gradient . However , these empirical defenses may be invalidated when confront with the strong adaptive attacker that has the white-box knowledge of the defense [ 3 ] . For certified defenses , they must be proved that have provable robustness , like [ 3 ] proposes the first certified defense based on the interval bound propagation ( IBP ) [ 7 ] that are commonly used in adversarial robustness certification problems . De-randomized smoothing ( DS ) [ 15 ] proposes the defense method extending randomized smoothing robustness schemes [ 4 ] with structured ablation . Moreover , a provable robustness defense based on clipped BagNet [ 1 ] with small reception field is proposed by [ 33 ] . However , all of these certified studies achieve distinctly low certified accuracy for large scale datasets such as ImageNet [ 13 ] . A certified accuracy of 20.5 % -36.3 % 1 from these methods can hardly be applied in the real world systems . It is a dilemma that certified robustness against such a practical and pernicious attack model can be hardly scaled for realistic usage . To address this issue , in this paper , we propose the defenses with both certified robustness and good scalability for large scale dataset that used in practical scenarios . Inspired by the existing analysis on the unique distribution of the activation map yielded by adversarial patch attacked input [ 16 , 19 , 25 ] , and substantial neural network compression techniques without sacrificing natural accuracy [ 10 , 31 ] , we observe that the adversarial patches rely on the localized superficial important neurons ( SINs ) to poison the output and can be exceedingly alleviated by utilizing the pruning techniques . We then propose the scalable certified ( ScaleCert ) defense with SIN-based neural network sparsity approach . The results show that the certified accuracy of ScaleCert on ImageNet dataset achieves 60.4 % against 1 % -pixel patch , significantly surpassing the highest record of 36.3 % in state-of-the-art certified detection methods [ 24 ] . This is the first certified work that largely reduces the gap between clean and certified accuracy , retaining the high certified accuracy with good computing efficiency , which largely pushes the certified defenses for common practice in real adoption . 2 Problem Setup . In this section , we describe the formulation and important terminology of adversarial patch attack and certified defenses against it . 2.1 Adversarial Patch Attack Formulation . Given an input x ∈ X ⊂ [ 0 , 1 ] W×H×3 with a true label y∗ , adversarial patch attack can manipulate the output label of victim model , M ( · ) , by adding the patch perturbation on x . The goal of adversarial patch attack is to find an adversarial patch x̂ that can be added on x to generate image x′ ∈ A ( x ) satisfying a constraint A such thatM ( x′ ) = y′ 6= y∗ . y′ can be a targeted label ( targeted attack ) or any arbitrary label not equal to the benign label y∗ ( untargeted attack ) . The adversarial patch x̂ can be generated by solving the follow equation : x̂ = arg max x′∈A ( x ) EX [ logPr ( M ( x′ ) = y′|x ) ] ( 1 ) The constrain A ( x ) is determined by the threat model . In this paper , the adversary is allowed to arbitrarily change pixels within a square contiguous region and locate this region anywhere on the image , which is similar to the previous works [ 15 , 18 , 25 ] . Formally , we use a binary pixel block p ∈ { 0 , 1 } W×H to represent the restricted square region , where the pixels located in the region are set to one , otherwise zero . For the purpose of stealthiness , p is extremely smaller than x , usually has the size of 1 % to 5 % of the x . The constraint set A ( x ) can be expressed as : A ( x ) ∈ { x′ = ( 1− p ) x+ p x̂ } ( 2 ) where refers to the element-wise product operator , and x̂ is the adversarial patch we generated from Eq . ( 3 ) . 1Covering both certified recovery and detection methodologies . See Table 2 for more details . 2.2 Certified Defenses . Bounding the range of a neural network outputs given a ` p-norm of input perturbation has become an important theme for neural network verification and certified adversarial defense [ 4 , 7 , 21 , 22 , 27 , 32 ] . Recently , researchers also shed light on generalizing the conventional input perturbation constraint to the adversarial patch formulation . Existing certified defenses against adversarial patch attacks can be classified into two major categories with different certification problems : 1 ) Certified defense for attack recovery . The classifier returns the assurance that the classification result will provably not change under any distortion within an adversarial constraint set for the input . It is a heritage from certified defenses towards adversarial example attacks . Formally , the classifier ensuresM ( x0 ) =M ( x′ ) = y0 to produce a certificate for an clean input x0 for any adversarial example x′ ∈ A ( x0 ) , where A ( x0 ) is the adversarial constraint set defined in Eq . ( 2 ) . Many existing studies extend the certified techniques in adversarial example attacks with the new constraints of patch perturbation to propose the certified defense against patch attacks . For example , [ 3 ] proposes their certified defense by extending interval bound propagation ( IBP ) [ 7 ] defense on patch attacks . [ 15 ] proposes the derandomized smoothing methodology based on structured ablation scheme by extending the randomized smoothing technique [ 4 ] . PatchGuard [ 25 ] leverages the DNN models with small reception fields to localize the adversarial features and proposes the robust masking defense to remove adversarial effect of these features . 2 ) Certified defense for attack detection . The classifier returns the assurance that the image is clean or alert when the image is an adversarial input . It is a relaxed certified problem compared to certified recovery . However , it is also an important and effective defense because occluding the adversarial patches can eliminate the adversarial effect . MRD [ 18 ] is the first work that proposes the certified attack detection study . By examining the pattern of the prediction labels that occlude the windows of the images , the adversarial images can be certified detected . PatchGuard++ [ 24 ] aims to improve the certified detection rate by extending PatchGuard with the same insights of MRD . For both certified recovery and detection defenses , the state-of-the-art approaches obtain low certified accuracy on large scale datasets and can be hardly adopted for practical use . As shown in Table 1 , the best certified accuracy in recent literature is about 36.3 % when the patch contains 1 % pixels of the input image . Therefore , we confront with the dilemma of practicality and low certified provable robustness for the harmful and practical adversarial patch attacks . To address this issue , we aim to propose a defense methodology that bridges the gap between clean and certified accuracy and achieves both provable robustness and good scalability for large scale datasets . 3 ScaleCert Defense . In this section , we first use an empirical study to motivate the use of superficial important neurons ( SIN ) distribution to distinguish the adversarial patch and eliminate the adversarial effect and noises . Then we propose the ScaleCert to defend against adversarial patch attack with SIN-based sparsification methodology . To note , Section 3.1 focuses on empirical analysis based on the results of some specific adversarial patch attack methodology . However , the ScaleCert ’ s robustness applies to any arbitrary adversarial patch attack . The provable robustness of this defense is demonstrated and analyzed in Section 3.3 . 3.1 SIN in Patched and Benign Images . In observing that the adversarial patch determines the prediction results with a very small region of pixels for a relatively broad range of input images , we thus perform superficial feature importance distribution analysis of patched images and benign images . Though neuron importance analysis has been widely used for abnormal input detection [ 6 , 16 , 29 , 30 ] , the metric in our methodology ( superficial feature importance ) is distinct from previous studies ( deep feature importance ) . The latter focuses on the neurons that contribute significantly to the inference output , while superficial important neurons refers to the neurons that contribute significantly to the shallow feature map values in the first ( several ) layer ( s ) . Compared to deep important neurons ( DIN ) , SIN has good discrimination to distinguish patched images , more straightforward correlation with the input adversarial regions , and smaller computing overhead without requirement for gradient calculation . More comparison between SIN and DIN is in Appendix A.3 . Specifically , we discover that the SIN of patched images exhibits extremely localized pattern and the adversarial patch effects can be eliminated if localized SINs are removed . For benign images , however , the prediction results do not rely on the localized SINs and thus being more stable . The intuition is that the prediction of typical DNN inference on the benign inputs depends on the deep features of the input image , while adversarial inputs have to strengthen the superficial features . Otherwise , its adversarial effect will fade out through the computation of all DNN layers . We quantitatively evaluate the impact of SIN during predictions of benign and patched images as follows . 1 ) Localized SIN Feature Analysis : SINs in patched images are much more localized than that in benign images . We show the Top-200 neurons in the activation map of the first layer for benign images and patched images in Figure 1 ( a ) . Then , we compare the standard deviation of distance between the Top-200 neurons to the central neuron of benign images and patched images for randomly-selected 10,000 images in ImageNet dataset . The patched images ’ distance deviation is much smaller than benign images , as shown in Figure 1 ( b ) . Additionally , we also cluster the Top-k neurons with the classic MeanShift clustering algorithm and observe the cluster number of SINs in patched images is much less than that in benign images . As shown in Figure 1 ( c ) , for patched images , about 75 % of cases have only one cluster and 99 % of cases have less than two clusters . While for benign images , more than 75 % of the cases have more than three clusters . The results indicate that SINs are much more localized in patched images in terms of both the distance and cluster number . 2 ) Prediction Stability Analysis : By occluding the localized SINs in a small window , the prediction results of patched images are unstable , while benign images seldom change . Specifically , we examine when moving the localized SIN features , how the prediction results are affected in patched images and benign images . We compare the prediction accuracy of benign objects and patched objects before and after moving out the localized SINs ( l-SIN ) , as shown in Figure 1 ( d ) . In this experiment , we consider 10000 randomly-selected ImageNet images that can be correctly classified by the model to eliminate the interference of other factors . The results show that for benign images , the prediction accuracy is not sensitive to the localized SIN features . For the patched images , originally the prediction accuracy is extremely low because of the patch effects . After moving the localized important features , the detection rate increases drastically , which indicates that the patch effect has been effectively eliminated . The results show that the nature images perform robust classification without relying on extremely localized SIN features , while patches rely on the extremely localized SIN features to deceive and induce the object detector to output the incorrect results . 3 ) Adaptive Attack Against SIN-based Detection : The adversarial patch attacks confront with the dilemma of either introducing large SINs or hardly affecting the prediction results . We perform the adaptive attacks on ImageNet dataset and the results are shown in Figure 1 ( e ) . To steer clear of the detection based on SINs , the adversary trains the adversarial patch by taking the superficial activation value into consideration . Compared to Equation ( 1 ) , a penalty loss of activation value is considered with the parameter α , when α is larger , the attack pays more attention to the stealthiness other than the attack effectiveness . In this way , the adversary aims to build the adversarial patch with both good poisoning effects , but also trying to escape from the adversary candidate searching . Adversary detection rate refers to the rate that detector correctly identifies the adversarial region in the adversarial inputs or the benign input with no adversarial region . The results show that , it is indeed that the adversary detection rate decrease to 82.1 % when adversary attempts to reduce the activation magnitude in the first layer ( α increases from 0 to 0.01 ) . However , compared to the gentle slope of adversary detection rate , the attack success rate decreases much more drastically . When α is increased from 0 to 0.01 , the adversarial attack success rate drops to 7.9 % . These results indicate that the adversary can not maintain the two goals of high attack success rate and good stealthiness in SINs simultaneously . The detailed attack setup and the datasets are covered in Appendix A.1 . These results show the empirical evidence to the hypothesis that the adversarial patch attacks rely on the strengths of localized SINs to effectively damage the prediction results of victim models . We leverage such insights to design ScaleCert certified defense methodology . Different from the motivation of previous studies [ 25 ] , we don ’ t constrain the reception field to localize the adversarial feature effect in the deep features , but focus on the elimination of poisoned SIN features . More comparison and discussion between these two design philosophies are provided in Appendix A.3 .

Summary: The proposed approach presents scalable certified defense against adversarial patch attack.  The method relies on pruning unimportant neurons and identifying regions in the input image which contain the adversarial patch. Once the patch location is identified, the label is predicted is using prediction consistency. A certification algorithm is provided which shows significant improvements over previous methods on ImageNet.