Imbalanced Gradients: A New Cause of Overestimated Adversarial Robustness

1 INTRODUCTION . Deep neural networks ( DNNs ) are vulnerable to adversarial examples , which are input instances crafted by adding small adversarial perturbations to natural examples . Adversarial examples can fool DNNs into making false predictions with high confidence , and transfer across different models ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) . A number of defenses have been proposed to overcome this vulnerability . However , a concerning fact is that many defenses have been quickly shown to have undergone incorrect or incomplete evaluation ( Carlini and Wagner , 2017 ; Athalye et al. , 2018 ; Engstrom et al. , 2018 ; Uesato et al. , 2018 ; Mosbach et al. , 2018 ; He et al. , 2018 ) . One common pitfall in adversarial robustness evaluation is the phenomenon of gradient masking ( Papernot et al. , 2017 ; Tramèr et al. , 2018 ) or obfuscated gradients ( Athalye et al. , 2018 ) , leading to weak or unsuccessful attacks and false signals of robustness . To demonstrate “ real '' robustness , newly proposed defenses claim robustness based on results of white-box attacks such as PGD ( Madry et al. , 2018 ) , and at the same time , demonstrate that they are not a result of obfuscated gradients . In this work , we show that the robustness may still be overestimated even when there are no obfuscated gradients . Specifically , we identify a new situation called Imbalanced Gradients that exists in several state-of-the-art defense models and can cause highly overestimated robustness . Imbalanced gradients is a new type of gradient masking effect where the gradient of one loss term dominates that of other terms . This causes the attack to move toward a suboptimal direction . Different from obfuscated gradients , imbalanced gradients are more subtle and are not detectable by the detection methods used for obfuscated gradients . To exploit imbalanced gradients , we propose a novel attack named Margin Decomposition ( MD ) attack that decomposes the margin loss into two separate terms , and then exploits the attackability of these terms via a two-stage attacking process . We derive MD variants of traditional attacks like PGD and MultiTargeted ( MT ) ( Gowal et al. , 2019 ) , and deploy these MD attacks to re-examine the robustness of 12 adversarial training-based defense models . We find that 6 of them are susceptible to imbalanced gradients , and their robustness originally evaluated by the PGD attack drops significantly against our MD attacks . Our key contributions are : • We identify a new type of subtle effect called imbalanced gradients , which can cause highly overestimated adversarial robustness and can not be detected by detection methods for obfuscated gradients . Especially , We highlight that label smoothing is one of the major causes of imbalanced gradients . • We propose Margin Decomposition ( MD ) attacks to exploit imbalanced gradients . MD leverages the attackability of the individual terms in the margin loss in a two-stage attacking process . We also introduce two variants of MD for existing attacks PGD and MT . • We conduct extensive evaluations on 12 state-of-the-art defense models and find that 6 of them suffer from imbalanced gradients and their PGD robustness drops by more than 9 % against our MD attacks . Our MD attacks exceed state-of-the-art attacks when imbalanced gradients occur . 2 BACKGROUND . We denote a clean sample by x , its class by y ∈ { 1 , · · · , C } withC the number of classes , and a DNN classifier by f . The probability of x being in the i-th class is computed as pi ( x ) = ezi/ ∑C j=1 e zj , where zi is the logits for the i-th class . The goal of adversarial attack is to find an adversarial example xadv that can fool the model into making a false prediction ( e.g . f ( xadv ) 6= y ) , and is typically restricted to be within a small -ball around the original example x ( e.g . ‖xadv − x‖∞ ≤ ) . Adversarial Attack . Adversarial examples can be crafted by maximizing a classification loss ` by one or multiple steps of adversarial perturbations . For example the one-step Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) and the iterative FGSM ( I-FGSM ) attack ( Kurakin et al. , 2017 ) . Projected Gradient Descent ( PGD ) ( Madry et al. , 2018 ) attack is another iterative method that projects the perturbation back onto the -ball centered at x when it goes beyond . Carlini and Wagner ( CW ) ( Carlini and Wagner , 2017 ) attack generates adversarial examples via an optimization framework . Whilst there exist other attacks such as Frank-Wolfe attack ( Chen et al. , 2018a ) , distributionally adversarial attack ( Zheng et al. , 2019 ) and elastic-net attacks ( Chen et al. , 2018b ) , the most commonly used attacks for robustness evaluations are FGSM , PGD , and CW . Several recent attacks have been proposed to produce more accurate robustness evaluations than PGD . This includes Fast Adaptive Boundary Attack ( FAB ) ( Croce and Hein , 2019 ) , MultiTargeted ( MT ) attack ( Gowal et al. , 2019 ) , Output Diversified Initialization ( ODI ) attack ( Tashiro et al. , 2020 ) , and AutoAttack ( AA ) ( Croce and Hein , 2020 ) . FAB finds the minimal perturbation necessary to change the class of a given input . MT ( Gowal et al. , 2019 ) is a PGD-based attack with multiple restarts and picks a new target class at each restart . ODI provides a more effective initialization strategy with diversified logits . AA attack is a parameter-free ensemble of four attacks : FAB , two Auto-PGD attacks , and the black-box Square Attack ( Andriushchenko et al. , 2019 ) . AA has demonstrated to be one of the state-of-the-art attacks to date ( Croce and Hein , 2020 ) . Adversarial Loss . Many attacks use Cross Entropy ( CE ) as the adversarial loss : ` ce ( x , y ) = − logpy . The other commonly used adversarial loss is the margin loss ( Carlini and Wagner , 2017 ) : ` margin ( x , y ) = zmax − zy , with zmax = maxi 6=y zi . Shown in ( Gowal et al. , 2019 ) , CE can be written in a margin form ( e.g . ` ce ( x , y ) = log ( ∑C i=1 e zi ) − zy ) , and in most cases , they are both effective . While FGSM and PGD attacks use the CE loss , CW and several recent attacks such as MT and ODI adopt the margin loss . AA has one PGD variant using the CE loss and the other PGD variant using the Difference of Logits Ratio ( DLR ) loss . DLR can be regarded as a “ relative margin ” loss . In this paper , we identify a new effect that causes overestimated adversarial robustness from the margin loss perspective and propose new attacks by decomposing the margin loss . Adversarial Defense . In response to the threat of adversarial attacks , many defenses have been proposed such as defensive distillation ( Papernot et al. , 2016 ) , feature/subspace analysis ( Xu et al. , 2017 ; Ma et al. , 2018 ) , denoising techniques ( Guo et al. , 2018 ; Liao et al. , 2018 ; Samangouei et al. , 2018 ) , robust regularization ( Gu and Rigazio , 2014 ; Tramèr et al. , 2018 ; Ross and Doshi-Velez , 2018 ) , model compression ( Liu et al. , 2018 ; Das et al. , 2018 ; Rakin et al. , 2018 ) and adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) . Among them , adversarial training via robust min-max optimization has been found to be the most effective approach ( Athalye et al. , 2018 ) . A number of new techniques have been proposed to further enhance the adversarial training ( Wang et al. , 2019 ; Zhang et al. , 2019 ; Carmon et al. , 2019 ; Alayrac et al. , 2019 ; Wang and Zhang , 2019 ; Zhang and Wang , 2019 ; Zhang and Xu , 2020 ; Wang et al. , 2020 ; Kim and Wang , 2020 ; Ding et al. , 2018 ; Chan et al. , 2020 ) . We will discuss and evaluate these adversarial training-based defenses with our proposed attacks in Section 4 . 3 IMBALANCED GRADIENTS AND MARGIN DECOMPOSITION ATTACK . We first give a toy example of imbalanced gradients and show how regular attacks can fail in such a situation . We then empirically verify their existence in deep neural networks , particularly for some adversarially-trained models . Finally , we propose the Margin Decomposition attack to exploit the imbalanced gradients . Since CE and margin loss are the two commonly used loss functions for adversarial attack and CE can be written in a margin form ( Gowal et al. , 2019 ) , here we focus on the margin loss to present the phenomenon of imbalanced gradients . Imbalanced Gradients . The gradient of the margin loss ( e.g . ` margin ( x , y ) = zmax − zy ) is the combination of the gradients of its two individual terms ( e.g . ∇x ( zmax−zy ) = ∇xzmax+∇x ( −zy ) ) . Imbalanced Gradients is the situation where the gradient of one loss term dominates that of other term ( s ) , pushing the attack towards a suboptimal direction . Toy Example . Consider a one-dimensional classification task and a binary classifier with two outputs z1 and z2 ( like logits of a DNN ) , Figure 1 illustrates the distributions of z1 , z2 and z2 − z1 around x = 0 . The classifier predicts class 1 when z1 ≥ z2 , otherwise class 2 . We consider an input at x = 0 with correct prediction y = 1 , and a maximum perturbation constraint = 2 ( e.g . perturbation δ ∈ [ −2 , +2 ] ) . The attack is successful if and only if z2 > z1 . In this example , imbalanced gradients occurs at x = 0 , where the gradients of the two terms ∇xz2 and ∇x ( −z1 ) have opposite directions , and the attack is dominated by the z1 term as∇x ( −z1 ) is significantly larger than ∇xz2 . Thus , attacking x with the margin loss will converge to +2 , where the sample is still correctly classified . However , for a successful attack , x should be perturbed towards -2 . In this particular scenario , the gradient ∇xz2 < 0 alone can provide the most effective attack direction . Note that this toy example was motivated by the loss landscape of DNNs when imbalanced gradient occurs . 3.1 IMBALANCED GRADIENTS IN DNNS . The situation can be extremely complex for DNNs with high-dimensional inputs , as imbalanced gradients can occur at each input dimension . It thus requires a metric to quantitatively measure the degree of gradient imbalance . Here , we propose such a metric named Gradient Imbalance Ratio ( GIR ) to measure the imbalance ratio for a single input x , which can then be averaged over multiple inputs to produce the imbalance ratio for the entire model . Definition of GIR . To measure the imbalance ratio , we focus on the input dimensions that are dominated by one loss term . An input dimension xi is dominated by a loss term ( e.g . zmax ) means that 1 ) the gradients of loss terms at xi have different directions ( ∇xizmax · ∇xi ( −zy ) < 0 ) , and 2 ) the gradient of the dominant term is larger ( e.g . |∇xizmax| > |∇xi ( −zy ) | ) . According to the dominant term , we can split these dimensions into two subsets xs1 and xs2 where xs1 are dominated by the zmax term , while xs2 are dominated by the −zy term . The overall dominance effect of each loss term can be formulated as r1 = ∥∥∇xs1 ( zmax − zy ) ∥∥1 and r2 = ∥∥∇xs2 ( zmax − zy ) ∥∥1 . Here , we use the L1-norms instead of L0-norms ( i.e . the number of dominated dimensions ) to also take into consideration the gradient magnitude . To keep the ratio larger than 1 , GIR is computed as : GIR = max { r1 r2 , r2 r1 } ( 1 ) GIR is defined by the ratio between the L1-norms of the gradients of two groups : Note that the GIR metric is not a general measure of imbalance . Rather , it is designed only for assessing gradient imbalance for adversarial robustness evaluation . GIR focuses specifically on the imbalanced input dimensions , and uses the L1 norm to also take into account the influence of these dimensions to the final output . The ratio reflects how far away the imbalance towards one direction than the other . GIR of both Naturally- and Adversarial-trained DNNs . With the GIR metric , we next investigate 6 DNN models including a naturally-trained ( Natural ) model and 5 adversarially-trained models using standard adversarial training ( Madry et al. , 2018 ) ( SAT ) , sensible adversarial training ( Kim and Wang , 2020 ) ( Sense ) , feature scattering-based adversarial training ( Zhang and Wang , 2019 ) ( FeaScatter ) , bilateral adversarial training ( Wang and Zhang , 2019 ) ( Bilateral ) , and adversarial interpolation training ( Zhang and Xu , 2020 ) ( AdvInterp ) . We present these defense models here because they represent different levels of gradient imbalance ( a complete analysis of more models can be found in Appendix F ) . Natural , SAT and Sense are WideResNet-34-10 models , while others are WideResNet-28-10 models . We train Natural and SAT following typical settings in ( Madry et al. , 2018 ) while others use their officially released models . We compute the GIR scores of the 6 models based on 1000 randomly selected test samples , and show them in Figure 2a . One major observation is that some defense models can have a much higher imbalance ratio than either naturally-trained or SAT model . This confirms that gradient imbalance does exist in DNNs , and some defenses tend to train the model to have highly imbalanced gradients . We will show , in Section 4 , that this situation of imbalanced gradients may cause highly overestimated robustness when evaluated using a traditional PGD attack . Imbalanced Gradients Reduce Attack Effectiveness . When there are imbalanced gradients , the attack can be pushed by the dominant term to produce weak attacks , and the non-dominant term alone can lead to more successful attacks . To illustrate this , in Figure 2b , we show the success rates of PGD attack on the above 5 defense models ( Natural has zero robustness against PGD ) with different losses : CE loss , margin loss , and the two individual margin terms . We consider 20-step PGD ( PGD-20 ) attacks with step size /4 and = 8/255 on all CIFAR-10 test images . Intuitively , the two margin terms could lead to less effective attacks , as they only provide partial information about the margin loss . This is indeed the case for the low gradient imbalance model SAT . However , for highly imbalanced models Sense , FeaScatter , Bilateral and AdvInterp , attacking the zmax term produces even more powerful attacks than attacking the margin loss . This indicates that the gradient of the margin loss is shifted by the dominant term ( e.g . −zy in this case ) towards a less optimal direction , which inevitably causes less powerful attacks . Compared between attacking CE loss and attacking −zy , they achieve a very close performance on imbalanced models . This shows a stronger dominant effect of −zy in CE loss ( ` ce ( x , y ) = log ( ∑C i=1 e zi ) − zy ) . It is worth mentioning that , while both GIR and this individual term-based test can be used to check whether there are significantly imbalanced gradients in a defense model , GIR alone can not fully reflect the attack success rate . Figure 2c shows an example of how the −zy term leads the attack to a suboptimal direction : the margin loss is flat at the ∇x ( −zy ) direction , yet increases drastically at an opposite direction . In this example , the attack can actually succeed if it increases ( rather than decreases ) zy . Gradients can be Balanced by Attacking Individual Loss Terms . Here , we show that , interestingly , imbalanced gradients can be balanced by attacking the non-dominant term . Consider the AdvInterp model tested above as an example , the dominant term is −zy . Figure 3 illustrates the GIR values of 5 randomly selected CIFAR-10 test images by attacking them using PGD-20 with different margin terms or the full margin loss . As can be observed that , for all three losses , the GIRs are effectively reduced after the first few steps . However , only the non-dominant term zmax manages to stably reduce the imbalance ratio to around 2 . This indicates optimizing the individual terms separately can help avoid the situation of imbalanced gradients and the attack can indeed benefit from more balanced gradients ( see the higher success rate of zmax in Figure 2b ) .

Summary: This work highlights the existence of imbalanced gradients as a phenomenon that may hinder optimization of gradient-based adversarial attacks and, thus, give a false sense of robustness. Imbalanced gradients may occur as the attack objective consists of the difference of two terms (typically, the outputs of the network on two different classes). When the gradients of these two terms have opposite directions, the attack optimization may get easily stuck in a suboptimal local optimum, thus decreasing the attack effectiveness.