Imbalanced Gradients: A New Cause of Overestimated Adversarial Robustness

1 INTRODUCTION . Deep neural networks ( DNNs ) are vulnerable to adversarial examples , which are input instances crafted by adding small adversarial perturbations to natural examples . Adversarial examples can fool DNNs into making false predictions with high confidence , and transfer across different models ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) . A number of defenses have been proposed to overcome this vulnerability . However , a concerning fact is that many defenses have been quickly shown to have undergone incorrect or incomplete evaluation ( Carlini and Wagner , 2017 ; Athalye et al. , 2018 ; Engstrom et al. , 2018 ; Uesato et al. , 2018 ; Mosbach et al. , 2018 ; He et al. , 2018 ) . One common pitfall in adversarial robustness evaluation is the phenomenon of gradient masking ( Papernot et al. , 2017 ; Tramèr et al. , 2018 ) or obfuscated gradients ( Athalye et al. , 2018 ) , leading to weak or unsuccessful attacks and false signals of robustness . To demonstrate “ real '' robustness , newly proposed defenses claim robustness based on results of white-box attacks such as PGD ( Madry et al. , 2018 ) , and at the same time , demonstrate that they are not a result of obfuscated gradients . In this work , we show that the robustness may still be overestimated even when there are no obfuscated gradients . Specifically , we identify a new situation called Imbalanced Gradients that exists in several state-of-the-art defense models and can cause highly overestimated robustness . Imbalanced gradients is a new type of gradient masking effect where the gradient of one loss term dominates that of other terms . This causes the attack to move toward a suboptimal direction . Different from obfuscated gradients , imbalanced gradients are more subtle and are not detectable by the detection methods used for obfuscated gradients . To exploit imbalanced gradients , we propose a novel attack named Margin Decomposition ( MD ) attack that decomposes the margin loss into two separate terms , and then exploits the attackability of these terms via a two-stage attacking process . We derive MD variants of traditional attacks like PGD and MultiTargeted ( MT ) ( Gowal et al. , 2019 ) , and deploy these MD attacks to re-examine the robustness of 12 adversarial training-based defense models . We find that 6 of them are susceptible to imbalanced gradients , and their robustness originally evaluated by the PGD attack drops significantly against our MD attacks . Our key contributions are : • We identify a new type of subtle effect called imbalanced gradients , which can cause highly overestimated adversarial robustness and can not be detected by detection methods for obfuscated gradients . Especially , We highlight that label smoothing is one of the major causes of imbalanced gradients . • We propose Margin Decomposition ( MD ) attacks to exploit imbalanced gradients . MD leverages the attackability of the individual terms in the margin loss in a two-stage attacking process . We also introduce two variants of MD for existing attacks PGD and MT . • We conduct extensive evaluations on 12 state-of-the-art defense models and find that 6 of them suffer from imbalanced gradients and their PGD robustness drops by more than 9 % against our MD attacks . Our MD attacks exceed state-of-the-art attacks when imbalanced gradients occur . 2 BACKGROUND . We denote a clean sample by x , its class by y ∈ { 1 , · · · , C } withC the number of classes , and a DNN classifier by f . The probability of x being in the i-th class is computed as pi ( x ) = ezi/ ∑C j=1 e zj , where zi is the logits for the i-th class . The goal of adversarial attack is to find an adversarial example xadv that can fool the model into making a false prediction ( e.g . f ( xadv ) 6= y ) , and is typically restricted to be within a small -ball around the original example x ( e.g . ‖xadv − x‖∞ ≤ ) . Adversarial Attack . Adversarial examples can be crafted by maximizing a classification loss ` by one or multiple steps of adversarial perturbations . For example the one-step Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) and the iterative FGSM ( I-FGSM ) attack ( Kurakin et al. , 2017 ) . Projected Gradient Descent ( PGD ) ( Madry et al. , 2018 ) attack is another iterative method that projects the perturbation back onto the -ball centered at x when it goes beyond . Carlini and Wagner ( CW ) ( Carlini and Wagner , 2017 ) attack generates adversarial examples via an optimization framework . Whilst there exist other attacks such as Frank-Wolfe attack ( Chen et al. , 2018a ) , distributionally adversarial attack ( Zheng et al. , 2019 ) and elastic-net attacks ( Chen et al. , 2018b ) , the most commonly used attacks for robustness evaluations are FGSM , PGD , and CW . Several recent attacks have been proposed to produce more accurate robustness evaluations than PGD . This includes Fast Adaptive Boundary Attack ( FAB ) ( Croce and Hein , 2019 ) , MultiTargeted ( MT ) attack ( Gowal et al. , 2019 ) , Output Diversified Initialization ( ODI ) attack ( Tashiro et al. , 2020 ) , and AutoAttack ( AA ) ( Croce and Hein , 2020 ) . FAB finds the minimal perturbation necessary to change the class of a given input . MT ( Gowal et al. , 2019 ) is a PGD-based attack with multiple restarts and picks a new target class at each restart . ODI provides a more effective initialization strategy with diversified logits . AA attack is a parameter-free ensemble of four attacks : FAB , two Auto-PGD attacks , and the black-box Square Attack ( Andriushchenko et al. , 2019 ) . AA has demonstrated to be one of the state-of-the-art attacks to date ( Croce and Hein , 2020 ) . Adversarial Loss . Many attacks use Cross Entropy ( CE ) as the adversarial loss : ` ce ( x , y ) = − logpy . The other commonly used adversarial loss is the margin loss ( Carlini and Wagner , 2017 ) : ` margin ( x , y ) = zmax − zy , with zmax = maxi 6=y zi . Shown in ( Gowal et al. , 2019 ) , CE can be written in a margin form ( e.g . ` ce ( x , y ) = log ( ∑C i=1 e zi ) − zy ) , and in most cases , they are both effective . While FGSM and PGD attacks use the CE loss , CW and several recent attacks such as MT and ODI adopt the margin loss . AA has one PGD variant using the CE loss and the other PGD variant using the Difference of Logits Ratio ( DLR ) loss . DLR can be regarded as a “ relative margin ” loss . In this paper , we identify a new effect that causes overestimated adversarial robustness from the margin loss perspective and propose new attacks by decomposing the margin loss . Adversarial Defense . In response to the threat of adversarial attacks , many defenses have been proposed such as defensive distillation ( Papernot et al. , 2016 ) , feature/subspace analysis ( Xu et al. , 2017 ; Ma et al. , 2018 ) , denoising techniques ( Guo et al. , 2018 ; Liao et al. , 2018 ; Samangouei et al. , 2018 ) , robust regularization ( Gu and Rigazio , 2014 ; Tramèr et al. , 2018 ; Ross and Doshi-Velez , 2018 ) , model compression ( Liu et al. , 2018 ; Das et al. , 2018 ; Rakin et al. , 2018 ) and adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) . Among them , adversarial training via robust min-max optimization has been found to be the most effective approach ( Athalye et al. , 2018 ) . A number of new techniques have been proposed to further enhance the adversarial training ( Wang et al. , 2019 ; Zhang et al. , 2019 ; Carmon et al. , 2019 ; Alayrac et al. , 2019 ; Wang and Zhang , 2019 ; Zhang and Wang , 2019 ; Zhang and Xu , 2020 ; Wang et al. , 2020 ; Kim and Wang , 2020 ; Ding et al. , 2018 ; Chan et al. , 2020 ) . We will discuss and evaluate these adversarial training-based defenses with our proposed attacks in Section 4 . 3 IMBALANCED GRADIENTS AND MARGIN DECOMPOSITION ATTACK . We first give a toy example of imbalanced gradients and show how regular attacks can fail in such a situation . We then empirically verify their existence in deep neural networks , particularly for some adversarially-trained models . Finally , we propose the Margin Decomposition attack to exploit the imbalanced gradients . Since CE and margin loss are the two commonly used loss functions for adversarial attack and CE can be written in a margin form ( Gowal et al. , 2019 ) , here we focus on the margin loss to present the phenomenon of imbalanced gradients . Imbalanced Gradients . The gradient of the margin loss ( e.g . ` margin ( x , y ) = zmax − zy ) is the combination of the gradients of its two individual terms ( e.g . ∇x ( zmax−zy ) = ∇xzmax+∇x ( −zy ) ) . Imbalanced Gradients is the situation where the gradient of one loss term dominates that of other term ( s ) , pushing the attack towards a suboptimal direction . Toy Example . Consider a one-dimensional classification task and a binary classifier with two outputs z1 and z2 ( like logits of a DNN ) , Figure 1 illustrates the distributions of z1 , z2 and z2 − z1 around x = 0 . The classifier predicts class 1 when z1 ≥ z2 , otherwise class 2 . We consider an input at x = 0 with correct prediction y = 1 , and a maximum perturbation constraint = 2 ( e.g . perturbation δ ∈ [ −2 , +2 ] ) . The attack is successful if and only if z2 > z1 . In this example , imbalanced gradients occurs at x = 0 , where the gradients of the two terms ∇xz2 and ∇x ( −z1 ) have opposite directions , and the attack is dominated by the z1 term as∇x ( −z1 ) is significantly larger than ∇xz2 . Thus , attacking x with the margin loss will converge to +2 , where the sample is still correctly classified . However , for a successful attack , x should be perturbed towards -2 . In this particular scenario , the gradient ∇xz2 < 0 alone can provide the most effective attack direction . Note that this toy example was motivated by the loss landscape of DNNs when imbalanced gradient occurs . 3.1 IMBALANCED GRADIENTS IN DNNS . The situation can be extremely complex for DNNs with high-dimensional inputs , as imbalanced gradients can occur at each input dimension . It thus requires a metric to quantitatively measure the degree of gradient imbalance . Here , we propose such a metric named Gradient Imbalance Ratio ( GIR ) to measure the imbalance ratio for a single input x , which can then be averaged over multiple inputs to produce the imbalance ratio for the entire model . Definition of GIR . To measure the imbalance ratio , we focus on the input dimensions that are dominated by one loss term . An input dimension xi is dominated by a loss term ( e.g . zmax ) means that 1 ) the gradients of loss terms at xi have different directions ( ∇xizmax · ∇xi ( −zy ) < 0 ) , and 2 ) the gradient of the dominant term is larger ( e.g . |∇xizmax| > |∇xi ( −zy ) | ) . According to the dominant term , we can split these dimensions into two subsets xs1 and xs2 where xs1 are dominated by the zmax term , while xs2 are dominated by the −zy term . The overall dominance effect of each loss term can be formulated as r1 = ∥∥∇xs1 ( zmax − zy ) ∥∥1 and r2 = ∥∥∇xs2 ( zmax − zy ) ∥∥1 . Here , we use the L1-norms instead of L0-norms ( i.e . the number of dominated dimensions ) to also take into consideration the gradient magnitude . To keep the ratio larger than 1 , GIR is computed as : GIR = max { r1 r2 , r2 r1 } ( 1 ) GIR is defined by the ratio between the L1-norms of the gradients of two groups : Note that the GIR metric is not a general measure of imbalance . Rather , it is designed only for assessing gradient imbalance for adversarial robustness evaluation . GIR focuses specifically on the imbalanced input dimensions , and uses the L1 norm to also take into account the influence of these dimensions to the final output . The ratio reflects how far away the imbalance towards one direction than the other . GIR of both Naturally- and Adversarial-trained DNNs . With the GIR metric , we next investigate 6 DNN models including a naturally-trained ( Natural ) model and 5 adversarially-trained models using standard adversarial training ( Madry et al. , 2018 ) ( SAT ) , sensible adversarial training ( Kim and Wang , 2020 ) ( Sense ) , feature scattering-based adversarial training ( Zhang and Wang , 2019 ) ( FeaScatter ) , bilateral adversarial training ( Wang and Zhang , 2019 ) ( Bilateral ) , and adversarial interpolation training ( Zhang and Xu , 2020 ) ( AdvInterp ) . We present these defense models here because they represent different levels of gradient imbalance ( a complete analysis of more models can be found in Appendix F ) . Natural , SAT and Sense are WideResNet-34-10 models , while others are WideResNet-28-10 models . We train Natural and SAT following typical settings in ( Madry et al. , 2018 ) while others use their officially released models . We compute the GIR scores of the 6 models based on 1000 randomly selected test samples , and show them in Figure 2a . One major observation is that some defense models can have a much higher imbalance ratio than either naturally-trained or SAT model . This confirms that gradient imbalance does exist in DNNs , and some defenses tend to train the model to have highly imbalanced gradients . We will show , in Section 4 , that this situation of imbalanced gradients may cause highly overestimated robustness when evaluated using a traditional PGD attack . Imbalanced Gradients Reduce Attack Effectiveness . When there are imbalanced gradients , the attack can be pushed by the dominant term to produce weak attacks , and the non-dominant term alone can lead to more successful attacks . To illustrate this , in Figure 2b , we show the success rates of PGD attack on the above 5 defense models ( Natural has zero robustness against PGD ) with different losses : CE loss , margin loss , and the two individual margin terms . We consider 20-step PGD ( PGD-20 ) attacks with step size /4 and = 8/255 on all CIFAR-10 test images . Intuitively , the two margin terms could lead to less effective attacks , as they only provide partial information about the margin loss . This is indeed the case for the low gradient imbalance model SAT . However , for highly imbalanced models Sense , FeaScatter , Bilateral and AdvInterp , attacking the zmax term produces even more powerful attacks than attacking the margin loss . This indicates that the gradient of the margin loss is shifted by the dominant term ( e.g . −zy in this case ) towards a less optimal direction , which inevitably causes less powerful attacks . Compared between attacking CE loss and attacking −zy , they achieve a very close performance on imbalanced models . This shows a stronger dominant effect of −zy in CE loss ( ` ce ( x , y ) = log ( ∑C i=1 e zi ) − zy ) . It is worth mentioning that , while both GIR and this individual term-based test can be used to check whether there are significantly imbalanced gradients in a defense model , GIR alone can not fully reflect the attack success rate . Figure 2c shows an example of how the −zy term leads the attack to a suboptimal direction : the margin loss is flat at the ∇x ( −zy ) direction , yet increases drastically at an opposite direction . In this example , the attack can actually succeed if it increases ( rather than decreases ) zy . Gradients can be Balanced by Attacking Individual Loss Terms . Here , we show that , interestingly , imbalanced gradients can be balanced by attacking the non-dominant term . Consider the AdvInterp model tested above as an example , the dominant term is −zy . Figure 3 illustrates the GIR values of 5 randomly selected CIFAR-10 test images by attacking them using PGD-20 with different margin terms or the full margin loss . As can be observed that , for all three losses , the GIRs are effectively reduced after the first few steps . However , only the non-dominant term zmax manages to stably reduce the imbalance ratio to around 2 . This indicates optimizing the individual terms separately can help avoid the situation of imbalanced gradients and the attack can indeed benefit from more balanced gradients ( see the higher success rate of zmax in Figure 2b ) .

Summary: The paper introduces and analyses the possibility that the effectiveness of PGD-based adversarial attacks might be reduced by imbalanced gradients between the terms of the margin losses commonly used. As a remedy, it also proposed a new scheme for PGD attack, where for the first half of the iterations a single-term loss is optimized, before falling back on the the usual margin loss. The authors test the hypothesis of imbalanced gradients, introducing a new metric, GIR, and the newly proposed attack in two versions, MD and MDMT, on several defenses based on adversarial training.