Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples

Evaluating robustness of machine-learning models to adversarial examples is a1 challenging problem . Many defenses have been shown to provide a false sense of2 security by causing gradient-based attacks to fail , and they have been broken under3 more rigorous evaluations . Although guidelines and best practices have been sug-4 gested to improve current adversarial robustness evaluations , the lack of automatic5 testing and debugging tools makes it difficult to apply these recommendations in6 a systematic manner . In this work , we overcome these limitations by ( i ) defining7 a set of quantitative indicators which unveil common failures in the optimization8 of gradient-based attacks , and ( ii ) proposing specific mitigation strategies within9 a systematic evaluation protocol . Our extensive experimental analysis shows that10 the proposed indicators of failure can be used to visualize , debug and improve11 current adversarial robustness evaluations , providing a first concrete step towards12 automatizing and systematizing current adversarial robustness evaluations.13 1 Introduction14 Neural networks are now deployed in settings where it is important that they behave reliably and15 robustly [ 19 , 15 , 33 , 3 ] . Unfortunately , these systems are vulnerable to adversarial examples [ 29 , 4 ] ,16 i.e. , inputs intentionally crafted to mislead machine-learning classifiers at test time . These attacks17 are especially important in settings where classifiers have security-critical consequences , including18 autonomous driving , automated medical diagnoses , and cybersecurity-related tasks such as spam and19 malware detection , web-page ranking and network protocol verification [ 27 , 18 , 26 , 2 , 28 , 15 ] .20 This vulnerability has caused a strong reaction from the community , with many proposed defenses [ 33,21 22 , 31 , 25 ] . Early defenses often argued robustness by showing the defense could prevent prior22 attacks , but not attacks tailored to that particular defense . As a result , most of these defenses have23 turned out to only provide a false sense of security , i.e. , to be broken when targeted by an adaptive24 attack that tailors the attack strategy to the particular defense [ 11 , 1 ] . More recent work has tried to25 evaluate using such adaptive attacks . Unfortunately , even this has proven difficult ; recent work has26 shown that 13 published defenses proposed in the last year are ineffective despite almost all of them27 containing an analysis to adaptive attacks [ 30 ] .28 The reason why adversarial example defense evaluations are incomplete comes down to the difficulty29 of performing an adaptive attack , and diagnosing when they go wrong . Adversarial examples are30 typically generated through gradient descent : the adversary first constructs a loss function so that a31 minimum for that function is an adversarial example . While gradient-based attacks are highly effective32 at finding adversarial examples on undefended classifiers with smooth loss functions , many defenses33 substantially hinder the attack optimization by obfuscating gradients or by exhibiting harder-to-34 optimize loss functions . In particular , most attempted defenses to adversarial examples only succeed35 at increasing the difficulty of solving the minimization formulation , and not at actually increasing the36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Algorithm 1 : Our framework for computing adversarial attacks Input : x , the initial point ; y , the true class of the initial point ; n , the number of iterations ; α , the learning rate ; f , the target model ; ∆ , the considered region . Output : x ? , the solution found by the algorithm 1 x0 ← initialize ( x ) . Initialize starting point 2 θ̂ ← approximation ( θ ) . Approximate model parameters 3 δ0 ← 0 . Initial δ 4 for i ∈ [ 1 , n ] do 5 δ′ ← δi − α∇xiL ( x0 + δi , y ; θ̂ ) . Compute optimizer step 6 δi+1 ← apply-constraints ( x0 , δ′ , ∆ ) . Apply constraints ( if needed ) 7 δ ? ← best ( δ0 , ... , δn ) . Choose best perturbation 8 return δ ? robustness of the underlying classifier ( i.e. , increasing the actual distance of the decision boundary37 from the input sample ) [ 10 , 11 , 1 , 30 ] . Moreover , even though guidelines and best practices have38 been suggested to improve current adversarial robustness evaluations , the lack of automatic testing39 and debugging tools makes it difficult to apply these recommendations in a systematic manner . These40 difficulties have perpetuated a constant cat-and-mouse game where defenders propose new schemes,41 and attackers find that actually the defense was only increasing the difficulty of solving the underlying42 minimization problem [ 5 , 3 ] .43 This paper directly addresses these limitations by ( i ) developing quantitative indicators of failure,44 i.e. , metrics designed to help debug optimization of gradient-based attacks for generating adversarial45 examples , and ( ii ) suggesting a systematic evaluation protocol to improve current robustness eval-46 uations by applying a sequence of specific mitigation strategies . In four case studies of published47 defenses that have been shown to be ineffective against stronger adaptive attacks , we show ( i ) that48 our indicators would have highlighted different failure modes in the original evaluations , and ( ii ) how49 these failures could have been easily overcome by following our suggested mitigation strategies.50 To summarize , we make the following contributions : ( i ) we introduce a unified attack framework51 that captures the predominant styles of existing gradient-based attack methods , and allows us to52 categorize the five main causes of failure that may arise during their optimization ( Sect . 2 ) ; ( ii ) we53 propose five indicators of attack failures ( IoAF ) , i.e. , metrics and principles that help understand why54 and when gradient-based attack algorithms fail ( Sect . 3 ) ; ( iii ) we empirically evaluate the utility of55 our metrics on four recently-published defenses , showing how their robustness evaluations could56 have been improved by monitoring the IoAF values and following our evaluation protocol ( Sect . 4 ; 57 and ( iv ) we provide open-source code and data we used in this paper for reproducing resources . Our58 code is available at https : //github.com/ioaf-todo.1 We conclude by discussing related work59 ( Sect . 5 ) , along with the limitations of our work and future research directions ( Sect . 6 ) .60 2 Adversarial Robustness : Gradient-based Attacks and Failures61 We argue here that optimizing adversarial examples amounts to solving a multi-objective optimization:62 min δ∈∆ ( L ( x+ δ , y ; θ ) , ‖δ‖p ) , ( 1 ) where x ∈ [ 0 , 1 ] d is the input sample , y ∈ { 1 , . . . , c } is either its label ( for untargeted attacks ) or63 the label of the target class ( for targeted attacks ) , and δ ∈ ∆ is the perturbation optimized to have64 the perturbed sample x′ = x + δ misclassified as desired , within the given input domain . The65 target model is parameterized by θ . The given problem presents an inherent tradeoff : minimizing L66 amounts to finding an adversarial example with large misclassification confidence and perturbation67 size , while minimizing ‖δ‖p penalizes larger perturbations ( in the given ` p norm ) at the expense of68 decreasing misclassification confidence.2 Typically the attacker loss L is defined as the Cross-Entropy69 ( CE ) loss , or the logit difference [ 11 ] .70 1Anonymized for submission . 2Note that the sign of L may be adjusted internally in our formulation to properly account for both untargeted and targeted attacks . Multiobjective problems can be solved by establishing a different tradeoff between the given objectives71 along the Pareto frontier , by either using soft- or hard-constraint reformulations . For example , Carlini-72 Wagner ( CW ) [ 11 ] is a soft-constraint attack , which reformulates the aforementioned multiobjective73 problem as an unconstrained optimization : minδ ‖δ‖p + c · min ( L ( x + δ , y , θ ) , −κ ) , where the74 hyperparameters κ and c tune the trade-off between misclassification confidence and perturbation75 size . Hard-constraint reformulations instead aim to minimize one objective while constraining the76 other . They include maximum-confidence attacks like Projected Gradient Descent ( PGD ) [ 17 ] , which77 is formulated as minδ L ( x+ δ , y ; θ ) s.t . ‖δ‖p ≤ , and minimum-norm attacks like Brendel-Bethge78 ( BB ) [ 6 ] and Decoupling-Direction-Norm ( DDN ) [ 24 ] , which can be formulated as minδ ‖δ‖p s.t.79 L ( x+δ , y ; θ ) ≤ k. In these cases , and k upper bound the perturbation size and the misclassification80 confidence , respectively , thereby optimizing a different tradeoff between these two quantities.81 The aforementioned attacks often need to use an approximation θ̂ of the target model , since the latter82 may be either non-differentiable , or not sufficiently smooth [ 1 ] , hindering the gradient-based attack83 optimization process . In this case , once the attacker loss has been optimized on the surrogate model84 θ̂ , the attack is considered successful if it evades the target model θ.85 Attack Algorithm . According to the previous discussion , even if different attacks minimize different86 objectives or require different constraints , all of them can be seen as solutions to a common multiob-87 jective problem , based on gradient descent . Thus , their main steps can be summarized as detailed in88 Algorithm 1 . First , an initialization point ( line 1 ) needs to be set , and this can be achieved by directly89 using the input point x , a randomly-perturbed version of it , or even a sample from the target class [ 6 ] .90 Then , if the target model θ is difficult to deal with , or it is non-differentiable , the attacker must chose91 a surrogate model θ̂ that approximates the real target θ ( line 2 ) . The attack then iteratively updates92 the initial point searching for a better and better adversarial example ( line 4 ) , computing in each93 iteration one ( or more ) gradient descent steps ( line 5 ) using the initial point and the perturbation δi94 computed so far . Hence , the new perturbation δi+1 is obtained by enforcing the constraints defined in95 the problem ( line 6 ) , that can be updated accordingly to the chosen strategy [ 23 , 24 ] . For maximum96 confidence approaches , the attack can not exit the ∆ region , and samples are projected accordingly on97 this ball when reaching the constraints . Similarly , we consider minimum distance attacks successful98 only if they found adversarial examples inside the ∆ region . At the end of the iterations , the attacker99 has collected all the perturbations along the iterations , formalized as the attack path . The final result100 of the algorithm is the the best perturbation contained in the attack path , w.r.t . the loss they are101 minimizing ( line 7 ) .102 2.1 Attack failures103 We can now isolate four failures that can be encountered while optimizing adversarial attacks using104 Algorithm 1 , and we bound each of them to specific steps of such procedure.105 F1 : Implementation Problems . If no adversarial examples are found by the attack , it might be106 possible that the used implementation include errors or bugs . For example , we isolated a bug inside107 the procedure proposed by Madry et al . [ 17 ] . The attack as described returns the adversarial example108 only by looking at the last point of the attack path ( line 7 of Algorithm 2 ) , as shown in Fig . 1a , but109 would not return an adversarial example if one was found during search and then passed over.110 F2 : Non-converging attack . When performing gradient descent based attacks , a common problem111 is that attacks do not converge to any local minimum , as shown in Fig . 1b . This problem can be112 caused by either the setup of the attack , and in Algorithm 1 , this is reflected on the values of α and n,113 i.e . the step size of the attack , and the number of iterations . If α is too small , the gradient update step114 is not exploring the space ( line 5 of Algorithm 1 ) , while using too few iterations n might cause an115 early stopping of the attack ( line 4 of Algorithm 1 ) . An example of this failure can be found in the116 evaluation of the defense proposed by Buckman et al . [ 7 ] , where the authors only used 7 steps of117 PGD for testing the robustness of their defense , or by the one proposed by Pang et al . [ 21 ] , where118 the defense has been evaluated with only 10 steps of PGD . Also , this failure might be triggered119 either by a too-large step size , that lead the optimizer to keep overshooting the local minimum , or120 the presence of gradient obfuscation techniques [ 31 ] that alter the gradients of the model to point to121 random directions , leading the descent to fail.122 F3 : Bad local optimum . Once the attack reached convergence , the computed point might not be123 adversarial , since the optimizer has reached a region where it can not update anymore the adversarial124 perturbation , as shown in Fig . 1c . There are few reasons that might lead to such failure . One of them125 is again caused by the presence of gradient obfuscation , where the optimizer is unable to continue the126 descent , since it arrived in a region where the norms of gradients are ( nearly ) zero ( i.e . flat regions ) ,127 or again because the gradients are noisy , and the optimization lands on a bad local optimum ( line 5128 of Algorithm 1 ) . An example of such failure is detected inside the defense proposed by Papernot129 et al . [ 22 ] , where the model is trained to have signal in correspondence of samples , and producing130 regions with no gradient all around them . Another reason might be triggered by the choice of the131 initialization point itself ( line 1 of Algorithm 1 ) , that leads the optimizer into a region where no132 adversarial examples can be found . The latter has been detected by the analysis conducted by Tramèr133 et al . [ 30 ] against the defense proposed by Pang et al . [ 21 ] , where a different initialization point lead134 the attack to find a better solution.135 F4 : Non-adaptive attack . The loss function that the attacker optimizes does not match the actual loss136 of the target system , and this is caused by a bad choice of the surrogate model ( line 2 of Algorithm 1 ) ,137 as shown in Fig . 1d . This issue manifests when either the attack is computed on an undefended138 model , and later tested against the defense , or the target model is not differentiable and the surrogate139 is not really approximating it . Since we consider both cases , we differ from the literature , where the140 term non-adaptive has been used only for attacks that were not specifically designed to target a given141 defense [ 30 ] . An examples of this failure is found in the defense proposed by Yu et al . [ 32 ] , where the142 attack has been computed against the undefended model , and then evaluated against the defense later.143 To maximize the likelihood of creating successful attacks and hence avoiding such failures , current144 recommendations [ 30 ] suggest to ( i ) select the strongest attacks against the model that is being tested ; 145 ( ii ) state the precise threat model being considered ; ( iii ) select the correct hyperparameters for the146 attack being used ; and ( iv ) compute charts to understand how the attacks behave by varying the size147 of the perturbation . Indeed useful , such are only qualitative recommendations that require ad-hoc148 inspection of each failed attack.149 3 Indicators of Attack Failure150 In this section we describe our Indicators of Attack Failures , i.e . tests that help an analyst debug a151 failing attack . Each of these tests outputs a value bounded between 0 and 1 , where values towards 1152 implies the presence of the failure described by the test . Informed by the results of the indicators,153 we propose potential mitigations that can resolve the presence of the detected failure . An overview154 of such approach can be appreciated in Fig . 2 , where we connect failures with the indicators that155 quantify them , along with possible mitigations.156 I1 : Silent Success . This indicator is designed as a binary flag that157 triggers when the attack is failing , but a legitimate adversarial exam-158 ple is found inside the attack path , as described by the implementa-159 tion problem failure ( F1 ) .160 I2 : Break-point angle . This indicator is designed to quantify the161 non-convergence of the attack ( F2 ) caused by the choice of too small162 hyperparameters . We normalize the loss along the attack path and163 the iteration , to fit the loss in the domain [ 0 , 1 ] × [ 0 , 1 ] , and , ideally , a164 well-converged loss should approximate a triangle in that domain , as165 shown in Fig . 3 . To create that triangle , we connect the first and the166 last point in the loss curve , and we conclude the shape by considering167 the point of the loss curve that is further to such conjunction . We are168 interested in the amplitude of the basis β angle , since it is the one169 that characterizes the shape of the triangle : when β ≈ π , the triangle is flat , implying that the loss is170 still decreasing . For this reason , the indicator computes 1− |cosβ| , matching such intended behavior.171 On the other hand , this indicator is close to 0 when the triangle is close to be right , hence β ≈ π2 .172 p1 p2p3 β 1 1 1 1 At ta ck er L os s 0 0 At ta ck er L os s IterationsIterations Figure 4 : I3 indicator . I3 : Increasing loss . This indicator is designed to quantify either the173 non-convergence of the attack ( F2 ) , or the inability of converging174 to a good local optimum ( F3 ) , both caused by the presence of noisy175 gradients , where the loss of the attack is increasing while optimizing.176 To characterize such behavior , we normalize the loss of the attack177 and the iterations as we did in I2 , and we extract from it only the178 portions where it increases , and we compute its area , as shown in179 Fig . 4 . When this indicator is close to 1 , the values of the loss are180 fluctuating around its maximum value , difficult to be decreased by181 the optimizer.182 I4 : Zero gradients . This indicator is designed to quantify the183 bad-local optimum failure ( F3 ) , caused by the absence of gradi-184 ent information . For this reason , we compute how many times,185 along the attack path , the gradients of the loss function are zero:186 1 n+1 ∑n i=0 1‖∇x+δiL‖=0 . This indicator is close to 1 when most of the norms of the gradient are 0,187 causing the attack step to fail.188 I5 : Non-transferability . This indicator is designed to quantify the non-adaptive failure ( F4 ) , by189 measuring if the optimized attack fails against the real target model , while succeeding against the190 surrogate one . If the attack transfers successfully , the indicator is set to 0 , otherwise it is set to 1.191 3.1 Mitigate the Failures of Security Evaluations192 Once the robust accuracy of a model has been computed , the attacker should now check the feedback193 of the indicators and mitigate accordingly the detected failures.194 M1 : Fix the implementation . If I1 is active , the attack is considered failed , but there exists an195 adversarial point inside the computed path that satisfies the attack objective . Hence , the resulting196 robust accuracy must be lowered to reflect this patch accordingly . Also , the attacker would want to197 run again their evaluations using another library , or a patched version of the same attack.198 M2 : Tune the hyperparameters . If I2 activates , it means that the optimization can be improved , and199 hence both the step size and iteration hyperparameters can be increased . Otherwise , if I3 activates,200 the attack should consider a smaller step size , since the loss might be overshooting local minima.201 M3 : Use a different loss function . If I3 activates , and the decrement of the step size did not work,202 the attack should change the loss to be optimized [ 30 ] , preferring one that has a smoother behavior . If203 I4 activates , the attack should consider loss functions that do not saturate ( e.g . avoid the softmax ) [ 9 ] ,204 or also increase the step size of the attack to avoid regions with zero gradients.205 M4 : Consider different restarts for the attack . If I3 or I4 activates , the attack might also consider206 to repeat the experiments with more initialization points and restarts , as the failure could be the result207 of added randomness or an unlucky initialization.208 M5 : Perform adaptive attacks . Lastly , if none of the above applied , the attack might be optimizing209 against a bad surrogate model . If I5 is active , the attack should be repeated by changing the surrogate210 to better approximate the target , or include the defense inside the attack itself [ 30 ] . This step implies211 repeating the evaluation , as the change of the surrogate might trigger other previously-fixed failures.212 When attacks fail even after the application of recommended mitigations , it would be easy to assume213 that the evaluated defense is strong against adversarial attacks . However , the only thing known is214 that baseline attacks , properly tested , are not working against the defense . Hence , the designer of the215 defense should try as hard as possible to break the proposed defense with further investigations [ 12 ] ,216 and by performing sanity checks , e.g. , ensuring that the robust accuracy drops to 0 % when the217 perturbation size is unbounded , or by trying different attack strategies , e.g. , using gradient-free218 attacks or attacks designed by reversing the defense mechanism.219 4 Experiments220 We now exhibit the results of our experiments , by showing the correlation between the feedback of221 our indicators , and the false sense of security given by badly-evaluated defenses.222 Experimental setup . We run our attacks on an Intel R© Xeon R© CPU E5-2670 v3 , with 48 cores , 126223 GB of RAM , and equipped with an Nvidia Quadro M6000 with 24 GB of memory . All the attacks224 and models have been wrapped and run by using the SecML library [ 20 ] . We select four defenses that225 have been reported as failing , and we show that our indicators would have detected such evaluation226 errors . For each of them , we set the hyperparameters for the attack as done in the original evaluation,227 in order to collect similar results.228 k-Winners-Take-All ( kWTA ) , the defense proposed by Xiao et al . [ 31 ] uses only the top-k outputs229 from each layer , generating many discontinuities in the loss landscape , and hence resulting in the230 non-converging failure due to noisy gradients ( F2 ) . We use the implementation provided by Tramèr231 et al . [ 30 ] , trained on CIFAR10 , and we test its robustness by attacking it with ` ∞-PGD [ 17 ] with a232 step size of α = 0.003 , maximum perturbation = 8/255 and 50 iterations , with 5 restarts for each233 attack , scoring a robust accuracy of 58 % on 100 samples.234 Distillation , the defense proposed by Papernot et al . [ 22 ] , works by training a model to have zero235 gradients around the training points , leading gradient-based attacks towards bad local optimum ( F3 ) .236 We re-implemented such defense , by training a distilled classifier on the MNIST dataset to mimic the237 original evaluation . Then , we apply ` ∞-PGD [ 17 ] , with step size α = 0.01 , maximum perturbation238 = 0.3 for 50 iterations on 100 samples , resulting in a robust accuracy of 94,2 % .239 Ensemble diversity , the defense proposed by Pang et al . [ 21 ] is composed with different neural240 networks , trained with a regularizer that encourages diversity . We adopt the implementation provided241 by Tramèr et al . [ 30 ] . Then , following its original evaluation , we apply ` ∞-PGD [ 17 ] , with step size242 α = 0.001 , maximum perturbation = 0.01 for 10 iterations on 100 samples , resulting in a robust243 accuracy of 38 % .244 Turning a Weakness into a Strenght ( TWS ) , the defense proposed by Yu et al . [ 32 ] , applies a mechanism245 for detecting the presence of adversarial examples on top of an undefended model , measuring how246 much the decision changes locally around a sample . Even if the authors also apply other rejection247 mechanisms , we take into account only the described one , as we wish to show that attacks optimized248 neglecting such term will trigger the non-adaptive attack failure ( F4 ) . We apply this defended on249 a WideResNet model trained on CIFAR10 , provided by RobustBench [ 14 ] . We attack this model250 with ` ∞-PGD [ 17 ] , with step size α = 0.1 , maximum perturbation = 0.3 for 50 iterations on 100251 samples , and then we query the defended model with all the computed adversarial examples . While252 the attacks works against the standard model , some of them are rejected by the defense , resulting253 in a robust accuracy of 35 % , highlighted by the trigger of the I5 indicator . In this case , we consider254 an attack unsuccessful if the original sample is not misclassified and the adversarial point is either255 belonging to the same class , or it is labeled as rejected.256 Each of these attacks have been executed with 5 random restarts . We also attack all these models with257 the version of AutoPGD ( APGD ) [ 13 ] that uses the difference of logit ( DLR ) as a loss to optimize.258 This strategy will take care to automatically tune its hyperparameters while optimizing , reducing259 possible errors that occur while deciding the values of step size , and iterations . Lastly , we compute260 attacks that take into account all the mitigations we prescribed , and they will be analyzed further on261 in the paper.262 Identifying failures . We want now to understand if our in-263 dicators are correlated with faults of the security evaluations264 of defenses . We collect the results of all the attacks against265 the selected targets , and we compute our indicators , by listing266 their values in Table 1 , along with their mean score . With267 a glance , it is possible to grasp that out hypothesis is right:268 the detection of a failure is linked with higher values for the269 robust accuracy , and also the opposite . Each original evalua-270 tion is characterized by high values of one or more indicator,271 while the opposite happens for stronger attacks . For instance,272 APGD automatically tunes its hyperparameter while optimiz-273 ing , hence it is able to apply some mitigations directly during274 the attack . To gain a quantitative evaluation of out hypothe-275 sis , we compute both the p-value and the correlation between276 the average score of the indicators and the robust accuracy,277 depicting this result in Fig . 5 . Both p-value and correlation278 suggest a strong connection between these analyzed quantities,279 confirming our initial belief.280 Mitigating failures . We can now use our indicators to im-281 prove the quality of the security evaluations , and we apply282 the following pipeline : ( i ) we test the defense with a set of283 points with the original attack strategy proposed by the author284 of the defense ; ( ii ) we select the failure cases and inspect the285 feedback of our indicators per-sample ; ( iii ) for each cause of failure , we apply the specific remediation286 suggested by the metric ; and ( iv ) we show that the attack now succeeds , thus reducing the robust287 accuracy of the target model , and also the values of the indicators.288 We report all the results of this process in Table 2 , where each row shows the original robust accuracy,289 and how it is decreased , mitigation after mitigation . Also , all the individual values of each indicator290 computed on these patched attacks can be found in Table 1 , marked as PGD ? .291 Mitigating k-WTA failures . For many failing attacks , the I1 indicator triggers , implying that the292 attack found an adversarial example inside the path . We then apply mitigation M1 , and we lower293 accordingly the robust accuracy of the model to 36,4 % . We then analyze the feedback of the I3294 indicator , the one that detects the presence of noisy gradients . We apply mitigation M3 , and we295 change the loss of the attack as described by Tramèr et al . [ 30 ] . This loss is computed by averaging296 the gradient of each single point of the attack path with the information of the surrounding ones . The297 resulting direction is then able to correctly descent toward a minimum . We run ` ∞-PGD with the298 same parameters , but smoothing the gradients by averaging 100 neighboring points from a normal299 distribution N ( µ = xi , σ = 0.031 ) , where xi is a point in the attack path . After such mitigation , the300 robust accuracy drops to 6 , 4 % , and so follows the indicator ( Fig . 6a ) .301 Mitigating Distillation failures . All the attacks fail because of the absence of gradient information,302 leading the attack to a bad local optimum ( F3 ) , and such is highlighted by the feedback of the I3303 indicator . We apply mitigation M3 , and we change the loss optimized during the attack , following the304 strategy applied by Carlini et al . [ 9 ] , that computes the loss of the attack on the logit of the model305 rather than the final softmax layer . We repeat the PGD attack with such fix , and the robust accuracy306 drops to 0 % , along with the indicator I3 ( Fig . 6b ) .307 Mitigating Ensemble diversity failures . Firstly , the I1 indicator highlighted the presence of F1,308 implying that some failing attacks are due to the implementation itself . We apply mitigation M1 , and309 the robust accuracy decreases to 36 % . Also , I2 indicator is active , implying that the loss of of failing310 attacks could be optimized more . For this reason , we apply mitigation M2 , and we increase the step311 size to 0.05 and the iterations to 50 . This patch is enough for lowering the robust accuracy to 9 % .312 ( Fig . 6c ) .313 Mitigating TWS failures . The detector is rejecting adversarial attacks successfully computed on the314 undefended model , triggering the I5 indicator . Hence we apply mitigation M5 , and we adapt the attack315 to consider also the rejection class . This version of PGD minimizes the usual loss function of the316 attacker , but it also minimizes the score of the rejection class when encountered , allowing it to evade317 the rejection . We run such attack , and we obtain a new robust accuracy of 0 % ( Fig . 6d ) .318 5 Related Work319 Other systematic analysis on robustness evaluations . There have been a number of prior papers320 evaluating the robustness of particular defense schemes [ 10 , 1 , 30 ] . These papers focus on under-321 standing whether the robustness claims of particular defenses are true , often by performing one-off322 attacks or by proposing new general attack approaches that can be used to break future defenses . In323 contrast our goal is not to break any particular defense , but rather to help researchers understand324 when their evaluation may have gone wrong . In this way our paper is related to Carlini et al . [ 12 ] 325 that systematizes various suggestions from the literature for how to ensure that adversarial robust-326 ness evaluations are performed thoroughly . We imagine that our tests could be included in future327 recommendations for robustness evaluations.328 Benchmarks . Related to this work , there are a number of attack benchmarks that have been329 constructed . Instead of measuring the robustness of individual schemes as the prior papers do , these330 benchmarks aim to provide a complete evaluation framework that can be applied to any future defense331 as well . Ling et . al [ 16 ] proposed DEEPSEC , a benchmark that tests several attacks against a wide332 range of defenses . However , this framework was shown to be flawed by several implementation issues333 and problems in the configuration of the attacks [ 8 ] . Croce et al . [ 14 ] propose RobustBench [ 14 ] ,334 that accepts state-of-the-art models as submissions , and it tests their robust accuracy by applying335 AutoAttack [ 13 ] . However , this benchmark suite only works on CIFAR-trained models , and it is not336 able to determine which are the possible causes of such scored performance.337 Hence , these benchmark would benefit from our indicators , since they might provide useful insight338 that can be autonomously computed . Here we imagine that our framework could be used to help339 these tools automatically detect when their evaluations are incomplete , so that they could warn the340 operator that there was a potential error that should be investigated.341 6 Contributions , Limitations and Future Work342 We propose the Indicators of Attack Failures ( IoAF ) , quantitative tests that help the debugging of343 faulty-conducted security evaluations , and we propose a pipeline for mitigating their issues , leading to344 a fairer evaluation . We select defenses that have been previously shown to be weak against adversarial345 attacks , and we evaluate them with the lens of our indicators , showing that we could have detected346 their misconduct in advance . We empirically prove that these test are correlated with wrongly high347 robust accuracy , while they drop when attacks are successful.348 On top of these contributions , we acknowledge some limitations in our methodology . We do not349 provide a fully-autonomous way for deciding how to turn an attack into its adaptive version against350 a particular defense ( e.g . gradient obfuscation ) , but we provide quantitative tools for helping the351 decision among all the possible solutions that the attacker could come up with . Another limitation352 lurks in the choice of the attack itself , since some unknown-and-adaptive attack could behave very353 differently w.r.t . standard one , triggering some indicator in the process . However , these tests can be354 patched accordingly to take care of these newly-proposed patched attacks , and still being used as355 debugging tools . Lastly , as already discussed in Sect . 3 , if the evaluated defense is not triggering any356 indicators it does not imply it is secure , but rather it forces the application of other sanity checks [ 12 ] .357 We believe some part of this last process can be automatized with additional indicators , however we358 leave this as a future work.359 We hope that future work will include our indicators during the evaluation phase of new methods , in360 order to identify when attacks are failing for known reasons , and thus contributing to the creation of361 better defense mechanisms . Also , this work pose a preliminary step towards the creation of interactive362 dashboards that can be inspected as a web application . Finally , it would be insightful to attach363 our pipeline of indicators and mitigations to already-available benchmarks ( i.e . RobustBench [ 14 ] ) ,364 possibly detecting other failures in security evaluations we did not covered in our experiments.365 References366 [ 1 ] A. Athalye , N. Carlini , and D. A. Wagner . Obfuscated gradients give a false sense of security:367 Circumventing defenses to adversarial examples . In ICML , volume 80 of JMLR Workshop and368 Conference Proceedings , pages 274–283 . JMLR.org , 2018.369 [ 2 ] M. Barreno , B. Nelson , A. Joseph , and J. Tygar . The security of machine learning . Machine370 Learning , 81:121–148 , 2010.371 [ 3 ] B. Biggio and F. Roli . Wild patterns : Ten years after the rise of adversarial machine learning.372 Pattern Recognition , 84:317–331 , 2018.373 [ 4 ] B. Biggio , I. Corona , D. Maiorca , B. Nelson , N. Šrndić , P. Laskov , G. Giacinto , and F. Roli.374 Evasion attacks against machine learning at test time . In H. Blockeel , K. Kersting , S. Nijssen,375 and F. Železný , editors , Machine Learning and Knowledge Discovery in Databases ( ECML376 PKDD ) , Part III , volume 8190 of LNCS , pages 387–402 . Springer Berlin Heidelberg , 2013.377 [ 5 ] B. Biggio , G. Fumera , and F. Roli . Security evaluation of pattern classifiers under attack.378 IEEE Transactions on Knowledge and Data Engineering , 26 ( 4 ) :984–996 , April 2014 . ISSN379 1041-4347.380 [ 6 ] W. Brendel , J. Rauber , M. Kümmerer , I. Ustyuzhaninov , and M. Bethge . Accurate , reliable and381 fast robustness evaluation , 2019.382 [ 7 ] J. Buckman , A. Roy , C. Raffel , and I. Goodfellow . Thermometer encoding : One hot way to383 resist adversarial examples . In International Conference on Learning Representations , 2018.384 [ 8 ] N. Carlini . A critique of the deepsec platform for security analysis of deep learning models,385 2019.386 [ 9 ] N. Carlini and D. Wagner . Defensive distillation is not robust to adversarial examples , 2016.387 [ 10 ] N. Carlini and D. A. Wagner . Adversarial examples are not easily detected : Bypassing ten388 detection methods . In B. M. Thuraisingham , B. Biggio , D. M. Freeman , B. Miller , and A. Sinha,389 editors , 10th ACM Workshop on Artificial Intelligence and Security , AISec ’ 17 , pages 3–14,390 New York , NY , USA , 2017 . ACM.391 [ 11 ] N. Carlini and D. A. Wagner . Towards evaluating the robustness of neural networks . In IEEE392 Symposium on Security and Privacy , pages 39–57 . IEEE Computer Society , 2017.393 [ 12 ] N. Carlini , A. Athalye , N. Papernot , W. Brendel , J. Rauber , D. Tsipras , I. Goodfellow , A. Madry,394 and A. Kurakin . On evaluating adversarial robustness , 2019.395 [ 13 ] F. Croce and M. Hein . Reliable evaluation of adversarial robustness with an ensemble of diverse396 parameter-free attacks . In ICML , 2020.397 [ 14 ] F. Croce , M. Andriushchenko , V. Sehwag , N. Flammarion , M. Chiang , P. Mittal , and398 M. Hein . Robustbench : a standardized adversarial robustness benchmark . arXiv preprint399 arXiv:2010.09670 , 2020.400 [ 15 ] S. G. Finlayson , J. D. Bowers , J. Ito , J. L. Zittrain , A. L. Beam , and I. S. Kohane . Adversarial401 attacks on medical machine learning . Science , 363 ( 6433 ) :1287–1289 , 2019.402 [ 16 ] X. Ling , S. Ji , J. Zou , J. Wang , C. Wu , B. Li , and T. Wang . Deepsec : A uniform platform for403 security analysis of deep learning model . In 2019 IEEE Symposium on Security and Privacy404 ( SP ) , pages 673–690 , 2019. doi : 10.1109/SP.2019.00023.405 [ 17 ] A. Madry , A. Makelov , L. Schmidt , D. Tsipras , and A. Vladu . Towards deep learning models406 resistant to adversarial attacks . In ICLR , 2018.407 [ 18 ] A. McCallum and K. Nigam . A comparison of event models for naive bayes text classification.408 In Proc . AAAI Workshop on learning for text categorization , pages 41–48 , 1998.409 [ 19 ] P. McDaniel , N. Papernot , and Z . B. Celik . Machine learning in adversarial settings . IEEE410 Security & Privacy , 14 ( 3 ) :68–72 , May 2016.411 [ 20 ] M. Melis , A. Demontis , M. Pintor , A. Sotgiu , and B. Biggio . secml : A python library for secure412 and explainable machine learning . arXiv preprint arXiv:1912.10013 , 2019.413 [ 21 ] T. Pang , K. Xu , C. Du , N. Chen , and J. Zhu . Improving adversarial robustness via promoting414 ensemble diversity . In K. Chaudhuri and R. Salakhutdinov , editors , Proceedings of the 36th415 International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning416 Research , pages 4970–4979 . PMLR , 09–15 Jun 2019 . URL http : //proceedings.mlr.417 press/v97/pang19a.html.418 [ 22 ] N. Papernot , P. McDaniel , X. Wu , S. Jha , and A. Swami . Distillation as a defense to adversarial419 perturbations against deep neural networks . In 2016 IEEE Symposium on Security and Privacy420 ( SP ) , pages 582–597 , May 2016. doi : 10.1109/SP.2016.41.421 [ 23 ] M. Pintor , F. Roli , W. Brendel , and B. Biggio . Fast minimum-norm adversarial attacks through422 adaptive norm constraints , 2021.423 [ 24 ] J. Rony , L. G. Hafemann , L. S. Oliveira , I . B. Ayed , R. Sabourin , and E. Granger . Decoupling424 direction and norm for efficient gradient-based l2 adversarial attacks and defenses , 2019.425 [ 25 ] K. Roth , Y. Kilcher , and T. Hofmann . The odds are odd : A statistical test for detecting426 adversarial examples . In International Conference on Machine Learning , pages 5498–5507.427 PMLR , 2019.428 [ 26 ] B. I. Rubinstein , B. Nelson , L. Huang , A. D. Joseph , S.-h. Lau , S. Rao , N. Taft , and J. D. Tygar.429 Antidote : understanding and defending against poisoning of anomaly detectors . In Proceedings430 of the 9th ACM SIGCOMM Internet Measurement Conference , IMC ’ 09 , pages 1–14 , New York,431 NY , USA , 2009 . ACM.432 [ 27 ] M. Sahami , S. Dumais , D. Heckerman , and E. Horvitz . A bayesian approach to filtering junk433 e-mail . AAAI Technical Report WS-98-05 , Madison , Wisconsin , 1998.434 [ 28 ] C. Smutz and A. Stavrou . Malicious pdf detection using metadata and structural features . In435 Proceedings of the 28th Annual Computer Security Applications Conference , ACSAC ’ 12 , pages436 239–248 , New York , NY , USA , 2012 . ACM.437 [ 29 ] C. Szegedy , W. Zaremba , I. Sutskever , J. Bruna , D. Erhan , I. Goodfellow , and R. Fergus . Intrigu-438 ing properties of neural networks . In International Conference on Learning Representations,439 2014 . URL http : //arxiv.org/abs/1312.6199.440 [ 30 ] F. Tramer , N. Carlini , W. Brendel , and A. Madry . On adaptive attacks to adversarial example441 defenses . Advances in Neural Information Processing Systems , 33 , 2020.442 [ 31 ] C. Xiao , P. Zhong , and C. Zheng . Resisting adversarial attacks by k-winners-take-all . 2020.443 [ 32 ] T. Yu , S. Hu , C. Guo , W. Chao , and K. Weinberger . A new defense against adversarial444 images : Turning a weakness into a strength . In Proceedings of the 33rd Conference on Neural445 Information Processing Systems ( NeurIPS 2019 ) , Oct. 2019.446 [ 33 ] X. Yuan , P. He , Q. Zhu , and X. Li . Adversarial examples : Attacks and defenses for deep447 learning . IEEE Transactions on Neural Networks and Learning Systems , 30 ( 9 ) :2805–2824,448 2019. doi : 10.1109/TNNLS.2018.2886017.449 Checklist450 1 . For all authors ... 451 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s452 contributions and scope ? [ Yes ] 453 ( b ) Did you describe the limitations of your work ? [ Yes ] We discuss the limitations in454 Sect . 6455 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] In Sect . 6,456 we specified that the aim of our work is not to break defenses in an harmful way . Our457 purpose is only to help researchers to improve their security evaluation.458 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to459 them ? [ Yes ] 460 2 . If you are including theoretical results ... 461 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 462 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 463 3 . If you ran experiments ... 464 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-465 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] The code466 will be submitted as supplementary material , and the instructions for reproducing the467 experiments are described in Sect . 4468 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they469 were chosen ) ? [ Yes ] We describe the experimental protocol in Sect . 4470 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-471 ments multiple times ) ? [ N/A ] 472 ( d ) Did you include the total amount of compute and the type of resources used ( e.g.,473 type of GPUs , internal cluster , or cloud provider ) ? [ Yes ] The resourced used for the474 experiments are listed in Sect . 4475 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 476 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] We cited all the477 existing assets used for the experiments.478 ( b ) Did you mention the license of the assets ? [ Yes ] We cited the authors of the assets , and479 we provide the list of external assets along with the code.480 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 481 We provide the code for computing the metrics as supplementary material.482 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re483 using/curating ? [ N/A ] All the assets we used are publicly available.484 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable485 information or offensive content ? [ N/A ] 486 5 . If you used crowdsourcing or conducted research with human subjects ... 487 ( a ) Did you include the full text of instructions given to participants and screenshots , if488 applicable ? [ N/A ] 489 ( b ) Did you describe any potential participant risks , with links to Institutional Review490 Board ( IRB ) approvals , if applicable ? [ N/A ] 491 ( c ) Did you include the estimated hourly wage paid to participants and the total amount492 spent on participant compensation ? [ N/A ] 493

Summary: This paper reasons about complications that may cause authors to not recognized that their newly proposed defense is ineffective. The authors identify failure motivations, propose 5 indicators (measurable) that are indicative of an attack failure, and also propose appropriate mitigations. The paper very methodically and experimentally presents this findings, and offers a more concrete support towards adaptive attacks evaluations whenever new defenses are proposed.