Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples

Evaluating robustness of machine-learning models to adversarial examples is a1 challenging problem . Many defenses have been shown to provide a false sense of2 security by causing gradient-based attacks to fail , and they have been broken under3 more rigorous evaluations . Although guidelines and best practices have been sug-4 gested to improve current adversarial robustness evaluations , the lack of automatic5 testing and debugging tools makes it difficult to apply these recommendations in6 a systematic manner . In this work , we overcome these limitations by ( i ) defining7 a set of quantitative indicators which unveil common failures in the optimization8 of gradient-based attacks , and ( ii ) proposing specific mitigation strategies within9 a systematic evaluation protocol . Our extensive experimental analysis shows that10 the proposed indicators of failure can be used to visualize , debug and improve11 current adversarial robustness evaluations , providing a first concrete step towards12 automatizing and systematizing current adversarial robustness evaluations.13 1 Introduction14 Neural networks are now deployed in settings where it is important that they behave reliably and15 robustly [ 19 , 15 , 33 , 3 ] . Unfortunately , these systems are vulnerable to adversarial examples [ 29 , 4 ] ,16 i.e. , inputs intentionally crafted to mislead machine-learning classifiers at test time . These attacks17 are especially important in settings where classifiers have security-critical consequences , including18 autonomous driving , automated medical diagnoses , and cybersecurity-related tasks such as spam and19 malware detection , web-page ranking and network protocol verification [ 27 , 18 , 26 , 2 , 28 , 15 ] .20 This vulnerability has caused a strong reaction from the community , with many proposed defenses [ 33,21 22 , 31 , 25 ] . Early defenses often argued robustness by showing the defense could prevent prior22 attacks , but not attacks tailored to that particular defense . As a result , most of these defenses have23 turned out to only provide a false sense of security , i.e. , to be broken when targeted by an adaptive24 attack that tailors the attack strategy to the particular defense [ 11 , 1 ] . More recent work has tried to25 evaluate using such adaptive attacks . Unfortunately , even this has proven difficult ; recent work has26 shown that 13 published defenses proposed in the last year are ineffective despite almost all of them27 containing an analysis to adaptive attacks [ 30 ] .28 The reason why adversarial example defense evaluations are incomplete comes down to the difficulty29 of performing an adaptive attack , and diagnosing when they go wrong . Adversarial examples are30 typically generated through gradient descent : the adversary first constructs a loss function so that a31 minimum for that function is an adversarial example . While gradient-based attacks are highly effective32 at finding adversarial examples on undefended classifiers with smooth loss functions , many defenses33 substantially hinder the attack optimization by obfuscating gradients or by exhibiting harder-to-34 optimize loss functions . In particular , most attempted defenses to adversarial examples only succeed35 at increasing the difficulty of solving the minimization formulation , and not at actually increasing the36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Algorithm 1 : Our framework for computing adversarial attacks Input : x , the initial point ; y , the true class of the initial point ; n , the number of iterations ; α , the learning rate ; f , the target model ; ∆ , the considered region . Output : x ? , the solution found by the algorithm 1 x0 ← initialize ( x ) . Initialize starting point 2 θ̂ ← approximation ( θ ) . Approximate model parameters 3 δ0 ← 0 . Initial δ 4 for i ∈ [ 1 , n ] do 5 δ′ ← δi − α∇xiL ( x0 + δi , y ; θ̂ ) . Compute optimizer step 6 δi+1 ← apply-constraints ( x0 , δ′ , ∆ ) . Apply constraints ( if needed ) 7 δ ? ← best ( δ0 , ... , δn ) . Choose best perturbation 8 return δ ? robustness of the underlying classifier ( i.e. , increasing the actual distance of the decision boundary37 from the input sample ) [ 10 , 11 , 1 , 30 ] . Moreover , even though guidelines and best practices have38 been suggested to improve current adversarial robustness evaluations , the lack of automatic testing39 and debugging tools makes it difficult to apply these recommendations in a systematic manner . These40 difficulties have perpetuated a constant cat-and-mouse game where defenders propose new schemes,41 and attackers find that actually the defense was only increasing the difficulty of solving the underlying42 minimization problem [ 5 , 3 ] .43 This paper directly addresses these limitations by ( i ) developing quantitative indicators of failure,44 i.e. , metrics designed to help debug optimization of gradient-based attacks for generating adversarial45 examples , and ( ii ) suggesting a systematic evaluation protocol to improve current robustness eval-46 uations by applying a sequence of specific mitigation strategies . In four case studies of published47 defenses that have been shown to be ineffective against stronger adaptive attacks , we show ( i ) that48 our indicators would have highlighted different failure modes in the original evaluations , and ( ii ) how49 these failures could have been easily overcome by following our suggested mitigation strategies.50 To summarize , we make the following contributions : ( i ) we introduce a unified attack framework51 that captures the predominant styles of existing gradient-based attack methods , and allows us to52 categorize the five main causes of failure that may arise during their optimization ( Sect . 2 ) ; ( ii ) we53 propose five indicators of attack failures ( IoAF ) , i.e. , metrics and principles that help understand why54 and when gradient-based attack algorithms fail ( Sect . 3 ) ; ( iii ) we empirically evaluate the utility of55 our metrics on four recently-published defenses , showing how their robustness evaluations could56 have been improved by monitoring the IoAF values and following our evaluation protocol ( Sect . 4 ; 57 and ( iv ) we provide open-source code and data we used in this paper for reproducing resources . Our58 code is available at https : //github.com/ioaf-todo.1 We conclude by discussing related work59 ( Sect . 5 ) , along with the limitations of our work and future research directions ( Sect . 6 ) .60 2 Adversarial Robustness : Gradient-based Attacks and Failures61 We argue here that optimizing adversarial examples amounts to solving a multi-objective optimization:62 min δ∈∆ ( L ( x+ δ , y ; θ ) , ‖δ‖p ) , ( 1 ) where x ∈ [ 0 , 1 ] d is the input sample , y ∈ { 1 , . . . , c } is either its label ( for untargeted attacks ) or63 the label of the target class ( for targeted attacks ) , and δ ∈ ∆ is the perturbation optimized to have64 the perturbed sample x′ = x + δ misclassified as desired , within the given input domain . The65 target model is parameterized by θ . The given problem presents an inherent tradeoff : minimizing L66 amounts to finding an adversarial example with large misclassification confidence and perturbation67 size , while minimizing ‖δ‖p penalizes larger perturbations ( in the given ` p norm ) at the expense of68 decreasing misclassification confidence.2 Typically the attacker loss L is defined as the Cross-Entropy69 ( CE ) loss , or the logit difference [ 11 ] .70 1Anonymized for submission . 2Note that the sign of L may be adjusted internally in our formulation to properly account for both untargeted and targeted attacks . Multiobjective problems can be solved by establishing a different tradeoff between the given objectives71 along the Pareto frontier , by either using soft- or hard-constraint reformulations . For example , Carlini-72 Wagner ( CW ) [ 11 ] is a soft-constraint attack , which reformulates the aforementioned multiobjective73 problem as an unconstrained optimization : minδ ‖δ‖p + c · min ( L ( x + δ , y , θ ) , −κ ) , where the74 hyperparameters κ and c tune the trade-off between misclassification confidence and perturbation75 size . Hard-constraint reformulations instead aim to minimize one objective while constraining the76 other . They include maximum-confidence attacks like Projected Gradient Descent ( PGD ) [ 17 ] , which77 is formulated as minδ L ( x+ δ , y ; θ ) s.t . ‖δ‖p ≤ , and minimum-norm attacks like Brendel-Bethge78 ( BB ) [ 6 ] and Decoupling-Direction-Norm ( DDN ) [ 24 ] , which can be formulated as minδ ‖δ‖p s.t.79 L ( x+δ , y ; θ ) ≤ k. In these cases , and k upper bound the perturbation size and the misclassification80 confidence , respectively , thereby optimizing a different tradeoff between these two quantities.81 The aforementioned attacks often need to use an approximation θ̂ of the target model , since the latter82 may be either non-differentiable , or not sufficiently smooth [ 1 ] , hindering the gradient-based attack83 optimization process . In this case , once the attacker loss has been optimized on the surrogate model84 θ̂ , the attack is considered successful if it evades the target model θ.85 Attack Algorithm . According to the previous discussion , even if different attacks minimize different86 objectives or require different constraints , all of them can be seen as solutions to a common multiob-87 jective problem , based on gradient descent . Thus , their main steps can be summarized as detailed in88 Algorithm 1 . First , an initialization point ( line 1 ) needs to be set , and this can be achieved by directly89 using the input point x , a randomly-perturbed version of it , or even a sample from the target class [ 6 ] .90 Then , if the target model θ is difficult to deal with , or it is non-differentiable , the attacker must chose91 a surrogate model θ̂ that approximates the real target θ ( line 2 ) . The attack then iteratively updates92 the initial point searching for a better and better adversarial example ( line 4 ) , computing in each93 iteration one ( or more ) gradient descent steps ( line 5 ) using the initial point and the perturbation δi94 computed so far . Hence , the new perturbation δi+1 is obtained by enforcing the constraints defined in95 the problem ( line 6 ) , that can be updated accordingly to the chosen strategy [ 23 , 24 ] . For maximum96 confidence approaches , the attack can not exit the ∆ region , and samples are projected accordingly on97 this ball when reaching the constraints . Similarly , we consider minimum distance attacks successful98 only if they found adversarial examples inside the ∆ region . At the end of the iterations , the attacker99 has collected all the perturbations along the iterations , formalized as the attack path . The final result100 of the algorithm is the the best perturbation contained in the attack path , w.r.t . the loss they are101 minimizing ( line 7 ) .102 2.1 Attack failures103 We can now isolate four failures that can be encountered while optimizing adversarial attacks using104 Algorithm 1 , and we bound each of them to specific steps of such procedure.105 F1 : Implementation Problems . If no adversarial examples are found by the attack , it might be106 possible that the used implementation include errors or bugs . For example , we isolated a bug inside107 the procedure proposed by Madry et al . [ 17 ] . The attack as described returns the adversarial example108 only by looking at the last point of the attack path ( line 7 of Algorithm 2 ) , as shown in Fig . 1a , but109 would not return an adversarial example if one was found during search and then passed over.110 F2 : Non-converging attack . When performing gradient descent based attacks , a common problem111 is that attacks do not converge to any local minimum , as shown in Fig . 1b . This problem can be112 caused by either the setup of the attack , and in Algorithm 1 , this is reflected on the values of α and n,113 i.e . the step size of the attack , and the number of iterations . If α is too small , the gradient update step114 is not exploring the space ( line 5 of Algorithm 1 ) , while using too few iterations n might cause an115 early stopping of the attack ( line 4 of Algorithm 1 ) . An example of this failure can be found in the116 evaluation of the defense proposed by Buckman et al . [ 7 ] , where the authors only used 7 steps of117 PGD for testing the robustness of their defense , or by the one proposed by Pang et al . [ 21 ] , where118 the defense has been evaluated with only 10 steps of PGD . Also , this failure might be triggered119 either by a too-large step size , that lead the optimizer to keep overshooting the local minimum , or120 the presence of gradient obfuscation techniques [ 31 ] that alter the gradients of the model to point to121 random directions , leading the descent to fail.122 F3 : Bad local optimum . Once the attack reached convergence , the computed point might not be123 adversarial , since the optimizer has reached a region where it can not update anymore the adversarial124 perturbation , as shown in Fig . 1c . There are few reasons that might lead to such failure . One of them125 is again caused by the presence of gradient obfuscation , where the optimizer is unable to continue the126 descent , since it arrived in a region where the norms of gradients are ( nearly ) zero ( i.e . flat regions ) ,127 or again because the gradients are noisy , and the optimization lands on a bad local optimum ( line 5128 of Algorithm 1 ) . An example of such failure is detected inside the defense proposed by Papernot129 et al . [ 22 ] , where the model is trained to have signal in correspondence of samples , and producing130 regions with no gradient all around them . Another reason might be triggered by the choice of the131 initialization point itself ( line 1 of Algorithm 1 ) , that leads the optimizer into a region where no132 adversarial examples can be found . The latter has been detected by the analysis conducted by Tramèr133 et al . [ 30 ] against the defense proposed by Pang et al . [ 21 ] , where a different initialization point lead134 the attack to find a better solution.135 F4 : Non-adaptive attack . The loss function that the attacker optimizes does not match the actual loss136 of the target system , and this is caused by a bad choice of the surrogate model ( line 2 of Algorithm 1 ) ,137 as shown in Fig . 1d . This issue manifests when either the attack is computed on an undefended138 model , and later tested against the defense , or the target model is not differentiable and the surrogate139 is not really approximating it . Since we consider both cases , we differ from the literature , where the140 term non-adaptive has been used only for attacks that were not specifically designed to target a given141 defense [ 30 ] . An examples of this failure is found in the defense proposed by Yu et al . [ 32 ] , where the142 attack has been computed against the undefended model , and then evaluated against the defense later.143 To maximize the likelihood of creating successful attacks and hence avoiding such failures , current144 recommendations [ 30 ] suggest to ( i ) select the strongest attacks against the model that is being tested ; 145 ( ii ) state the precise threat model being considered ; ( iii ) select the correct hyperparameters for the146 attack being used ; and ( iv ) compute charts to understand how the attacks behave by varying the size147 of the perturbation . Indeed useful , such are only qualitative recommendations that require ad-hoc148 inspection of each failed attack.149 3 Indicators of Attack Failure150 In this section we describe our Indicators of Attack Failures , i.e . tests that help an analyst debug a151 failing attack . Each of these tests outputs a value bounded between 0 and 1 , where values towards 1152 implies the presence of the failure described by the test . Informed by the results of the indicators,153 we propose potential mitigations that can resolve the presence of the detected failure . An overview154 of such approach can be appreciated in Fig . 2 , where we connect failures with the indicators that155 quantify them , along with possible mitigations.156 I1 : Silent Success . This indicator is designed as a binary flag that157 triggers when the attack is failing , but a legitimate adversarial exam-158 ple is found inside the attack path , as described by the implementa-159 tion problem failure ( F1 ) .160 I2 : Break-point angle . This indicator is designed to quantify the161 non-convergence of the attack ( F2 ) caused by the choice of too small162 hyperparameters . We normalize the loss along the attack path and163 the iteration , to fit the loss in the domain [ 0 , 1 ] × [ 0 , 1 ] , and , ideally , a164 well-converged loss should approximate a triangle in that domain , as165 shown in Fig . 3 . To create that triangle , we connect the first and the166 last point in the loss curve , and we conclude the shape by considering167 the point of the loss curve that is further to such conjunction . We are168 interested in the amplitude of the basis β angle , since it is the one169 that characterizes the shape of the triangle : when β ≈ π , the triangle is flat , implying that the loss is170 still decreasing . For this reason , the indicator computes 1− |cosβ| , matching such intended behavior.171 On the other hand , this indicator is close to 0 when the triangle is close to be right , hence β ≈ π2 .172 p1 p2p3 β 1 1 1 1 At ta ck er L os s 0 0 At ta ck er L os s IterationsIterations Figure 4 : I3 indicator . I3 : Increasing loss . This indicator is designed to quantify either the173 non-convergence of the attack ( F2 ) , or the inability of converging174 to a good local optimum ( F3 ) , both caused by the presence of noisy175 gradients , where the loss of the attack is increasing while optimizing.176 To characterize such behavior , we normalize the loss of the attack177 and the iterations as we did in I2 , and we extract from it only the178 portions where it increases , and we compute its area , as shown in179 Fig . 4 . When this indicator is close to 1 , the values of the loss are180 fluctuating around its maximum value , difficult to be decreased by181 the optimizer.182 I4 : Zero gradients . This indicator is designed to quantify the183 bad-local optimum failure ( F3 ) , caused by the absence of gradi-184 ent information . For this reason , we compute how many times,185 along the attack path , the gradients of the loss function are zero:186 1 n+1 ∑n i=0 1‖∇x+δiL‖=0 . This indicator is close to 1 when most of the norms of the gradient are 0,187 causing the attack step to fail.188 I5 : Non-transferability . This indicator is designed to quantify the non-adaptive failure ( F4 ) , by189 measuring if the optimized attack fails against the real target model , while succeeding against the190 surrogate one . If the attack transfers successfully , the indicator is set to 0 , otherwise it is set to 1.191 3.1 Mitigate the Failures of Security Evaluations192 Once the robust accuracy of a model has been computed , the attacker should now check the feedback193 of the indicators and mitigate accordingly the detected failures.194 M1 : Fix the implementation . If I1 is active , the attack is considered failed , but there exists an195 adversarial point inside the computed path that satisfies the attack objective . Hence , the resulting196 robust accuracy must be lowered to reflect this patch accordingly . Also , the attacker would want to197 run again their evaluations using another library , or a patched version of the same attack.198 M2 : Tune the hyperparameters . If I2 activates , it means that the optimization can be improved , and199 hence both the step size and iteration hyperparameters can be increased . Otherwise , if I3 activates,200 the attack should consider a smaller step size , since the loss might be overshooting local minima.201 M3 : Use a different loss function . If I3 activates , and the decrement of the step size did not work,202 the attack should change the loss to be optimized [ 30 ] , preferring one that has a smoother behavior . If203 I4 activates , the attack should consider loss functions that do not saturate ( e.g . avoid the softmax ) [ 9 ] ,204 or also increase the step size of the attack to avoid regions with zero gradients.205 M4 : Consider different restarts for the attack . If I3 or I4 activates , the attack might also consider206 to repeat the experiments with more initialization points and restarts , as the failure could be the result207 of added randomness or an unlucky initialization.208 M5 : Perform adaptive attacks . Lastly , if none of the above applied , the attack might be optimizing209 against a bad surrogate model . If I5 is active , the attack should be repeated by changing the surrogate210 to better approximate the target , or include the defense inside the attack itself [ 30 ] . This step implies211 repeating the evaluation , as the change of the surrogate might trigger other previously-fixed failures.212 When attacks fail even after the application of recommended mitigations , it would be easy to assume213 that the evaluated defense is strong against adversarial attacks . However , the only thing known is214 that baseline attacks , properly tested , are not working against the defense . Hence , the designer of the215 defense should try as hard as possible to break the proposed defense with further investigations [ 12 ] ,216 and by performing sanity checks , e.g. , ensuring that the robust accuracy drops to 0 % when the217 perturbation size is unbounded , or by trying different attack strategies , e.g. , using gradient-free218 attacks or attacks designed by reversing the defense mechanism.219 4 Experiments220 We now exhibit the results of our experiments , by showing the correlation between the feedback of221 our indicators , and the false sense of security given by badly-evaluated defenses.222 Experimental setup . We run our attacks on an Intel R© Xeon R© CPU E5-2670 v3 , with 48 cores , 126223 GB of RAM , and equipped with an Nvidia Quadro M6000 with 24 GB of memory . All the attacks224 and models have been wrapped and run by using the SecML library [ 20 ] . We select four defenses that225 have been reported as failing , and we show that our indicators would have detected such evaluation226 errors . For each of them , we set the hyperparameters for the attack as done in the original evaluation,227 in order to collect similar results.228 k-Winners-Take-All ( kWTA ) , the defense proposed by Xiao et al . [ 31 ] uses only the top-k outputs229 from each layer , generating many discontinuities in the loss landscape , and hence resulting in the230 non-converging failure due to noisy gradients ( F2 ) . We use the implementation provided by Tramèr231 et al . [ 30 ] , trained on CIFAR10 , and we test its robustness by attacking it with ` ∞-PGD [ 17 ] with a232 step size of α = 0.003 , maximum perturbation = 8/255 and 50 iterations , with 5 restarts for each233 attack , scoring a robust accuracy of 58 % on 100 samples.234 Distillation , the defense proposed by Papernot et al . [ 22 ] , works by training a model to have zero235 gradients around the training points , leading gradient-based attacks towards bad local optimum ( F3 ) .236 We re-implemented such defense , by training a distilled classifier on the MNIST dataset to mimic the237 original evaluation . Then , we apply ` ∞-PGD [ 17 ] , with step size α = 0.01 , maximum perturbation238 = 0.3 for 50 iterations on 100 samples , resulting in a robust accuracy of 94,2 % .239 Ensemble diversity , the defense proposed by Pang et al . [ 21 ] is composed with different neural240 networks , trained with a regularizer that encourages diversity . We adopt the implementation provided241 by Tramèr et al . [ 30 ] . Then , following its original evaluation , we apply ` ∞-PGD [ 17 ] , with step size242 α = 0.001 , maximum perturbation = 0.01 for 10 iterations on 100 samples , resulting in a robust243 accuracy of 38 % .244 Turning a Weakness into a Strenght ( TWS ) , the defense proposed by Yu et al . [ 32 ] , applies a mechanism245 for detecting the presence of adversarial examples on top of an undefended model , measuring how246 much the decision changes locally around a sample . Even if the authors also apply other rejection247 mechanisms , we take into account only the described one , as we wish to show that attacks optimized248 neglecting such term will trigger the non-adaptive attack failure ( F4 ) . We apply this defended on249 a WideResNet model trained on CIFAR10 , provided by RobustBench [ 14 ] . We attack this model250 with ` ∞-PGD [ 17 ] , with step size α = 0.1 , maximum perturbation = 0.3 for 50 iterations on 100251 samples , and then we query the defended model with all the computed adversarial examples . While252 the attacks works against the standard model , some of them are rejected by the defense , resulting253 in a robust accuracy of 35 % , highlighted by the trigger of the I5 indicator . In this case , we consider254 an attack unsuccessful if the original sample is not misclassified and the adversarial point is either255 belonging to the same class , or it is labeled as rejected.256 Each of these attacks have been executed with 5 random restarts . We also attack all these models with257 the version of AutoPGD ( APGD ) [ 13 ] that uses the difference of logit ( DLR ) as a loss to optimize.258 This strategy will take care to automatically tune its hyperparameters while optimizing , reducing259 possible errors that occur while deciding the values of step size , and iterations . Lastly , we compute260 attacks that take into account all the mitigations we prescribed , and they will be analyzed further on261 in the paper.262 Identifying failures . We want now to understand if our in-263 dicators are correlated with faults of the security evaluations264 of defenses . We collect the results of all the attacks against265 the selected targets , and we compute our indicators , by listing266 their values in Table 1 , along with their mean score . With267 a glance , it is possible to grasp that out hypothesis is right:268 the detection of a failure is linked with higher values for the269 robust accuracy , and also the opposite . Each original evalua-270 tion is characterized by high values of one or more indicator,271 while the opposite happens for stronger attacks . For instance,272 APGD automatically tunes its hyperparameter while optimiz-273 ing , hence it is able to apply some mitigations directly during274 the attack . To gain a quantitative evaluation of out hypothe-275 sis , we compute both the p-value and the correlation between276 the average score of the indicators and the robust accuracy,277 depicting this result in Fig . 5 . Both p-value and correlation278 suggest a strong connection between these analyzed quantities,279 confirming our initial belief.280 Mitigating failures . We can now use our indicators to im-281 prove the quality of the security evaluations , and we apply282 the following pipeline : ( i ) we test the defense with a set of283 points with the original attack strategy proposed by the author284 of the defense ; ( ii ) we select the failure cases and inspect the285 feedback of our indicators per-sample ; ( iii ) for each cause of failure , we apply the specific remediation286 suggested by the metric ; and ( iv ) we show that the attack now succeeds , thus reducing the robust287 accuracy of the target model , and also the values of the indicators.288 We report all the results of this process in Table 2 , where each row shows the original robust accuracy,289 and how it is decreased , mitigation after mitigation . Also , all the individual values of each indicator290 computed on these patched attacks can be found in Table 1 , marked as PGD ? .291 Mitigating k-WTA failures . For many failing attacks , the I1 indicator triggers , implying that the292 attack found an adversarial example inside the path . We then apply mitigation M1 , and we lower293 accordingly the robust accuracy of the model to 36,4 % . We then analyze the feedback of the I3294 indicator , the one that detects the presence of noisy gradients . We apply mitigation M3 , and we295 change the loss of the attack as described by Tramèr et al . [ 30 ] . This loss is computed by averaging296 the gradient of each single point of the attack path with the information of the surrounding ones . The297 resulting direction is then able to correctly descent toward a minimum . We run ` ∞-PGD with the298 same parameters , but smoothing the gradients by averaging 100 neighboring points from a normal299 distribution N ( µ = xi , σ = 0.031 ) , where xi is a point in the attack path . After such mitigation , the300 robust accuracy drops to 6 , 4 % , and so follows the indicator ( Fig . 6a ) .301 Mitigating Distillation failures . All the attacks fail because of the absence of gradient information,302 leading the attack to a bad local optimum ( F3 ) , and such is highlighted by the feedback of the I3303 indicator . We apply mitigation M3 , and we change the loss optimized during the attack , following the304 strategy applied by Carlini et al . [ 9 ] , that computes the loss of the attack on the logit of the model305 rather than the final softmax layer . We repeat the PGD attack with such fix , and the robust accuracy306 drops to 0 % , along with the indicator I3 ( Fig . 6b ) .307 Mitigating Ensemble diversity failures . Firstly , the I1 indicator highlighted the presence of F1,308 implying that some failing attacks are due to the implementation itself . We apply mitigation M1 , and309 the robust accuracy decreases to 36 % . Also , I2 indicator is active , implying that the loss of of failing310 attacks could be optimized more . For this reason , we apply mitigation M2 , and we increase the step311 size to 0.05 and the iterations to 50 . This patch is enough for lowering the robust accuracy to 9 % .312 ( Fig . 6c ) .313 Mitigating TWS failures . The detector is rejecting adversarial attacks successfully computed on the314 undefended model , triggering the I5 indicator . Hence we apply mitigation M5 , and we adapt the attack315 to consider also the rejection class . This version of PGD minimizes the usual loss function of the316 attacker , but it also minimizes the score of the rejection class when encountered , allowing it to evade317 the rejection . We run such attack , and we obtain a new robust accuracy of 0 % ( Fig . 6d ) .318 5 Related Work319 Other systematic analysis on robustness evaluations . There have been a number of prior papers320 evaluating the robustness of particular defense schemes [ 10 , 1 , 30 ] . These papers focus on under-321 standing whether the robustness claims of particular defenses are true , often by performing one-off322 attacks or by proposing new general attack approaches that can be used to break future defenses . In323 contrast our goal is not to break any particular defense , but rather to help researchers understand324 when their evaluation may have gone wrong . In this way our paper is related to Carlini et al . [ 12 ] 325 that systematizes various suggestions from the literature for how to ensure that adversarial robust-326 ness evaluations are performed thoroughly . We imagine that our tests could be included in future327 recommendations for robustness evaluations.328 Benchmarks . Related to this work , there are a number of attack benchmarks that have been329 constructed . Instead of measuring the robustness of individual schemes as the prior papers do , these330 benchmarks aim to provide a complete evaluation framework that can be applied to any future defense331 as well . Ling et . al [ 16 ] proposed DEEPSEC , a benchmark that tests several attacks against a wide332 range of defenses . However , this framework was shown to be flawed by several implementation issues333 and problems in the configuration of the attacks [ 8 ] . Croce et al . [ 14 ] propose RobustBench [ 14 ] ,334 that accepts state-of-the-art models as submissions , and it tests their robust accuracy by applying335 AutoAttack [ 13 ] . However , this benchmark suite only works on CIFAR-trained models , and it is not336 able to determine which are the possible causes of such scored performance.337 Hence , these benchmark would benefit from our indicators , since they might provide useful insight338 that can be autonomously computed . Here we imagine that our framework could be used to help339 these tools automatically detect when their evaluations are incomplete , so that they could warn the340 operator that there was a potential error that should be investigated.341 6 Contributions , Limitations and Future Work342 We propose the Indicators of Attack Failures ( IoAF ) , quantitative tests that help the debugging of343 faulty-conducted security evaluations , and we propose a pipeline for mitigating their issues , leading to344 a fairer evaluation . We select defenses that have been previously shown to be weak against adversarial345 attacks , and we evaluate them with the lens of our indicators , showing that we could have detected346 their misconduct in advance . We empirically prove that these test are correlated with wrongly high347 robust accuracy , while they drop when attacks are successful.348 On top of these contributions , we acknowledge some limitations in our methodology . We do not349 provide a fully-autonomous way for deciding how to turn an attack into its adaptive version against350 a particular defense ( e.g . gradient obfuscation ) , but we provide quantitative tools for helping the351 decision among all the possible solutions that the attacker could come up with . Another limitation352 lurks in the choice of the attack itself , since some unknown-and-adaptive attack could behave very353 differently w.r.t . standard one , triggering some indicator in the process . However , these tests can be354 patched accordingly to take care of these newly-proposed patched attacks , and still being used as355 debugging tools . Lastly , as already discussed in Sect . 3 , if the evaluated defense is not triggering any356 indicators it does not imply it is secure , but rather it forces the application of other sanity checks [ 12 ] .357 We believe some part of this last process can be automatized with additional indicators , however we358 leave this as a future work.359 We hope that future work will include our indicators during the evaluation phase of new methods , in360 order to identify when attacks are failing for known reasons , and thus contributing to the creation of361 better defense mechanisms . Also , this work pose a preliminary step towards the creation of interactive362 dashboards that can be inspected as a web application . Finally , it would be insightful to attach363 our pipeline of indicators and mitigations to already-available benchmarks ( i.e . RobustBench [ 14 ] ) ,364 possibly detecting other failures in security evaluations we did not covered in our experiments.365 References366 [ 1 ] A. Athalye , N. Carlini , and D. A. Wagner . Obfuscated gradients give a false sense of security:367 Circumventing defenses to adversarial examples . In ICML , volume 80 of JMLR Workshop and368 Conference Proceedings , pages 274–283 . JMLR.org , 2018.369 [ 2 ] M. Barreno , B. Nelson , A. Joseph , and J. Tygar . The security of machine learning . Machine370 Learning , 81:121–148 , 2010.371 [ 3 ] B. Biggio and F. Roli . Wild patterns : Ten years after the rise of adversarial machine learning.372 Pattern Recognition , 84:317–331 , 2018.373 [ 4 ] B. Biggio , I. Corona , D. Maiorca , B. Nelson , N. Šrndić , P. Laskov , G. Giacinto , and F. Roli.374 Evasion attacks against machine learning at test time . In H. Blockeel , K. Kersting , S. Nijssen,375 and F. Železný , editors , Machine Learning and Knowledge Discovery in Databases ( ECML376 PKDD ) , Part III , volume 8190 of LNCS , pages 387–402 . Springer Berlin Heidelberg , 2013.377 [ 5 ] B. Biggio , G. Fumera , and F. Roli . Security evaluation of pattern classifiers under attack.378 IEEE Transactions on Knowledge and Data Engineering , 26 ( 4 ) :984–996 , April 2014 . ISSN379 1041-4347.380 [ 6 ] W. Brendel , J. Rauber , M. Kümmerer , I. Ustyuzhaninov , and M. Bethge . Accurate , reliable and381 fast robustness evaluation , 2019.382 [ 7 ] J. Buckman , A. Roy , C. Raffel , and I. Goodfellow . Thermometer encoding : One hot way to383 resist adversarial examples . In International Conference on Learning Representations , 2018.384 [ 8 ] N. Carlini . A critique of the deepsec platform for security analysis of deep learning models,385 2019.386 [ 9 ] N. Carlini and D. Wagner . Defensive distillation is not robust to adversarial examples , 2016.387 [ 10 ] N. Carlini and D. A. Wagner . Adversarial examples are not easily detected : Bypassing ten388 detection methods . In B. M. Thuraisingham , B. Biggio , D. M. Freeman , B. Miller , and A. Sinha,389 editors , 10th ACM Workshop on Artificial Intelligence and Security , AISec ’ 17 , pages 3–14,390 New York , NY , USA , 2017 . ACM.391 [ 11 ] N. Carlini and D. A. Wagner . Towards evaluating the robustness of neural networks . In IEEE392 Symposium on Security and Privacy , pages 39–57 . IEEE Computer Society , 2017.393 [ 12 ] N. Carlini , A. Athalye , N. Papernot , W. Brendel , J. Rauber , D. Tsipras , I. Goodfellow , A. Madry,394 and A. Kurakin . On evaluating adversarial robustness , 2019.395 [ 13 ] F. Croce and M. Hein . Reliable evaluation of adversarial robustness with an ensemble of diverse396 parameter-free attacks . In ICML , 2020.397 [ 14 ] F. Croce , M. Andriushchenko , V. Sehwag , N. Flammarion , M. Chiang , P. Mittal , and398 M. Hein . Robustbench : a standardized adversarial robustness benchmark . arXiv preprint399 arXiv:2010.09670 , 2020.400 [ 15 ] S. G. Finlayson , J. D. Bowers , J. Ito , J. L. Zittrain , A. L. Beam , and I. S. Kohane . Adversarial401 attacks on medical machine learning . Science , 363 ( 6433 ) :1287–1289 , 2019.402 [ 16 ] X. Ling , S. Ji , J. Zou , J. Wang , C. Wu , B. Li , and T. Wang . Deepsec : A uniform platform for403 security analysis of deep learning model . In 2019 IEEE Symposium on Security and Privacy404 ( SP ) , pages 673–690 , 2019. doi : 10.1109/SP.2019.00023.405 [ 17 ] A. Madry , A. Makelov , L. Schmidt , D. Tsipras , and A. Vladu . Towards deep learning models406 resistant to adversarial attacks . In ICLR , 2018.407 [ 18 ] A. McCallum and K. Nigam . A comparison of event models for naive bayes text classification.408 In Proc . AAAI Workshop on learning for text categorization , pages 41–48 , 1998.409 [ 19 ] P. McDaniel , N. Papernot , and Z . B. Celik . Machine learning in adversarial settings . IEEE410 Security & Privacy , 14 ( 3 ) :68–72 , May 2016.411 [ 20 ] M. Melis , A. Demontis , M. Pintor , A. Sotgiu , and B. Biggio . secml : A python library for secure412 and explainable machine learning . arXiv preprint arXiv:1912.10013 , 2019.413 [ 21 ] T. Pang , K. Xu , C. Du , N. Chen , and J. Zhu . Improving adversarial robustness via promoting414 ensemble diversity . In K. Chaudhuri and R. Salakhutdinov , editors , Proceedings of the 36th415 International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning416 Research , pages 4970–4979 . PMLR , 09–15 Jun 2019 . URL http : //proceedings.mlr.417 press/v97/pang19a.html.418 [ 22 ] N. Papernot , P. McDaniel , X. Wu , S. Jha , and A. Swami . Distillation as a defense to adversarial419 perturbations against deep neural networks . In 2016 IEEE Symposium on Security and Privacy420 ( SP ) , pages 582–597 , May 2016. doi : 10.1109/SP.2016.41.421 [ 23 ] M. Pintor , F. Roli , W. Brendel , and B. Biggio . Fast minimum-norm adversarial attacks through422 adaptive norm constraints , 2021.423 [ 24 ] J. Rony , L. G. Hafemann , L. S. Oliveira , I . B. Ayed , R. Sabourin , and E. Granger . Decoupling424 direction and norm for efficient gradient-based l2 adversarial attacks and defenses , 2019.425 [ 25 ] K. Roth , Y. Kilcher , and T. Hofmann . The odds are odd : A statistical test for detecting426 adversarial examples . In International Conference on Machine Learning , pages 5498–5507.427 PMLR , 2019.428 [ 26 ] B. I. Rubinstein , B. Nelson , L. Huang , A. D. Joseph , S.-h. Lau , S. Rao , N. Taft , and J. D. Tygar.429 Antidote : understanding and defending against poisoning of anomaly detectors . In Proceedings430 of the 9th ACM SIGCOMM Internet Measurement Conference , IMC ’ 09 , pages 1–14 , New York,431 NY , USA , 2009 . ACM.432 [ 27 ] M. Sahami , S. Dumais , D. Heckerman , and E. Horvitz . A bayesian approach to filtering junk433 e-mail . AAAI Technical Report WS-98-05 , Madison , Wisconsin , 1998.434 [ 28 ] C. Smutz and A. Stavrou . Malicious pdf detection using metadata and structural features . In435 Proceedings of the 28th Annual Computer Security Applications Conference , ACSAC ’ 12 , pages436 239–248 , New York , NY , USA , 2012 . ACM.437 [ 29 ] C. Szegedy , W. Zaremba , I. Sutskever , J. Bruna , D. Erhan , I. Goodfellow , and R. Fergus . Intrigu-438 ing properties of neural networks . In International Conference on Learning Representations,439 2014 . URL http : //arxiv.org/abs/1312.6199.440 [ 30 ] F. Tramer , N. Carlini , W. Brendel , and A. Madry . On adaptive attacks to adversarial example441 defenses . Advances in Neural Information Processing Systems , 33 , 2020.442 [ 31 ] C. Xiao , P. Zhong , and C. Zheng . Resisting adversarial attacks by k-winners-take-all . 2020.443 [ 32 ] T. Yu , S. Hu , C. Guo , W. Chao , and K. Weinberger . A new defense against adversarial444 images : Turning a weakness into a strength . In Proceedings of the 33rd Conference on Neural445 Information Processing Systems ( NeurIPS 2019 ) , Oct. 2019.446 [ 33 ] X. Yuan , P. He , Q. Zhu , and X. Li . Adversarial examples : Attacks and defenses for deep447 learning . IEEE Transactions on Neural Networks and Learning Systems , 30 ( 9 ) :2805–2824,448 2019. doi : 10.1109/TNNLS.2018.2886017.449 Checklist450 1 . For all authors ... 451 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s452 contributions and scope ? [ Yes ] 453 ( b ) Did you describe the limitations of your work ? [ Yes ] We discuss the limitations in454 Sect . 6455 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] In Sect . 6,456 we specified that the aim of our work is not to break defenses in an harmful way . Our457 purpose is only to help researchers to improve their security evaluation.458 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to459 them ? [ Yes ] 460 2 . If you are including theoretical results ... 461 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 462 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 463 3 . If you ran experiments ... 464 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-465 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] The code466 will be submitted as supplementary material , and the instructions for reproducing the467 experiments are described in Sect . 4468 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they469 were chosen ) ? [ Yes ] We describe the experimental protocol in Sect . 4470 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-471 ments multiple times ) ? [ N/A ] 472 ( d ) Did you include the total amount of compute and the type of resources used ( e.g.,473 type of GPUs , internal cluster , or cloud provider ) ? [ Yes ] The resourced used for the474 experiments are listed in Sect . 4475 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 476 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] We cited all the477 existing assets used for the experiments.478 ( b ) Did you mention the license of the assets ? [ Yes ] We cited the authors of the assets , and479 we provide the list of external assets along with the code.480 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 481 We provide the code for computing the metrics as supplementary material.482 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re483 using/curating ? [ N/A ] All the assets we used are publicly available.484 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable485 information or offensive content ? [ N/A ] 486 5 . If you used crowdsourcing or conducted research with human subjects ... 487 ( a ) Did you include the full text of instructions given to participants and screenshots , if488 applicable ? [ N/A ] 489 ( b ) Did you describe any potential participant risks , with links to Institutional Review490 Board ( IRB ) approvals , if applicable ? [ N/A ] 491 ( c ) Did you include the estimated hourly wage paid to participants and the total amount492 spent on participant compensation ? [ N/A ] 493

Summary: This paper proposes a systematic method for understanding the failures of adversarial attacks as means towards thorough evaluation of robustness defenses in the research community. The devised framework relies of three connected aspects: (i) analysis of the main causes why gradient-based adversarial attacks fail, (ii) metrics for identifying their reason of failure and (iii) practical mitigations to ensure attack efficiency when evaluating defenses. All these aspects are illustrated and confirmed on four existing defenses that were initially subjected to limited evaluation.