Title: SuperDeepFool: a new fast and accurate minimal adversarial attack
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Learning

Adversarial Attacks

Robustness

Interpretable AI

ML Security
Abstract: Deep neural networks have been known to be vulnerable to adversarial examples, which are inputs that are modified slightly to fool the network into making incorrect predictions. This has led to a significant amount of research on evaluating the robustness of these networks against such perturbations. One particularly important robustness metric is the robustness to minimal $\ell_{2}$ adversarial perturbations. However, existing methods for evaluating this robustness metric are either computationally expensive or not very accurate. In this paper, we introduce a new family of adversarial attacks that strike a balance between effectiveness and computational efficiency. Our proposed attacks are generalizations of the well-known DeepFool (DF) attack, while they remain simple to understand and implement. We demonstrate that our attacks outperform existing methods in terms of both effectiveness and computational efficiency. Our proposed attacks are also suitable for evaluating the robustness of large models and can be used to perform adversarial training (AT) to achieve state-of-the-art robustness to minimal $\ell_{2}$ adversarial perturbations.

SuperDeepFool: a new fast and accurate minimal adversarial attack

Abstract

Deep neural networks have been known to be vulnerable to adversarial examples, which are inputs that are modified slightly to fool the network into making incorrect predictions. This has led to a significant amount of research on evaluating the robustness of these networks against such perturbations. One particularly important robustness metric is the robustness to minimal ℓ 2 adversarial perturbations. However, existing methods for evaluating this robustness metric are either computationally expensive or not very accurate. In this paper, we introduce a new family of adversarial attacks that strike a balance between effectiveness and computational efficiency. Our proposed attacks are generalizations of the well-known DeepFool (DF) attack, while they remain simple to understand and implement. We demonstrate that our attacks outperform existing methods in terms of both effectiveness and computational efficiency. Our proposed attacks are also suitable for evaluating the robustness of large models and can be used to perform adversarial training (AT) to achieve state-of-the-art robustness to minimal ℓ 2 adversarial perturbations.

1 Introduction

Deep learning has achieved breakthrough improvement in numerous tasks and has developed as a powerful tool in various applications, including computer vision [32] and speech processing [35]. Despite their success, deep neural networks are known to be vulnerable to adversarial examples, carefully perturbed examples perceptually indistinguishable from original samples [54]. This can lead to a significant disruption of the inference result of deep neural networks. It has important implications for safety and security-critical applications of machine learning models.

Our goal in this paper is to introduce a parameter-free and simple method for accurately and reliably evaluating the adversarial robustness of deep networks in a fast and

0.75

0.50

1.00

1.25

1.50

Norm of Perturbations

10 2

10 3

Number of Gradient Computations (Log Scale)

DF

SDF

DDN

ALMA

FMN

FAB

geometrically-based fashion. Most of the current attack methods rely on general-purpose optimization

techniques, such as Projected Gradient Descent (PGD) [33] and Augmented Lagrangian [49], which are oblivious to the geometric properties of models. However, deep neural networks' robustness to adversarial perturbations is closely tied to their geometric landscape [14, 29, 38, 41]. Given this, it would be beneficial to exploit such properties when designing and implementing adversarial attacks. This allows to create more effective and computationally efficient attacks on classifiers. Formally, for a given classifier ˆ k and input x , we define an adversarial perturbation as the minimal perturbation r that is sufficient to change the estimated label ˆ k ( x ) :

̸

DeepFool (DF) [36] was among the earliest attempts to exploit the 'excessive linearity' [23] of deep networks to find minimum-norm adversarial perturbations. However, more sophisticated attacks were later developed that could find smaller perturbations at the expense of significantly greater computation time.

In this paper, we exploit the geometric characteristics of minimum-norm adversarial perturbations to design a family of fast yet simple algorithms that achieves a better trade-off between computational cost and accuracy in finding ℓ 2 adversarial perturbations (see Fig. 1). Our proposed algorithm, guided by the characteristics of the optimal solution to Eq. (1), enhances DF to obtain smaller perturbations, while maintaining simplicity and computational efficiency that are only slightly inferior to those of DF. Our main contributions are summarized as follows:

We introduce a novel family of fast yet accurate algorithms to find minimal adversarial perturbations. We conduct a comprehensive evaluation of our algorithms against state-of-theart (SOTA) adversarial attack methods across multiple scenarios. Our findings demonstrate that our algorithm identifies minimal yet accurate perturbations with significantly greater efficiency than competing SOTA approaches (4).

Our algorithms are developed in a systematic and well-grounded manner, based on theoretical analysis (3).

We further improve the robustness of state-of-the-art image classifiers to minimum-norm adversarial attacks via adversarial training on the examples obtained by our algorithms (4.3).

We significantly improve the time efficiency of the state-of-the-art Auto-Attack (AA) [12] by adding our proposed method to the set of attacks in AA (4.3).

We revisit the importance of minimal adversarial perturbations as a proxy to demystify deep neural network properties (Appendix G, Appendix O).

Related works. It has been observed that deep neural networks are vulnerable to adversarial examples [23, 36, 54]. To exploit this vulnerability, a range of methods have been developed for generating adversarial perturbations for image classifiers. These attacks occur in two settings: whitebox, where the attacker has complete knowledge of the model, including its architecture, parameters, defense mechanisms, etc.; and black-box, where the attacker's knowledge is limited, mostly relying on input queries to observe outputs [10, 44]. Further, adversarial attacks can be broadly categorized into two categories: bounded-norm attacks (such as FGSM [23] and PGD [33]) and minimumnorm attacks (such as DF and C & W[6]) with the latter aimed at solving Eq. (1). In this work, we specifically focus on white-box minimum ℓ 2 -norm attacks.

The authors in [54] studied adversarial examples by solving a penalized optimization problem. The optimization approach used in [54] is complex and computationally inefficient; therefore, it cannot scale to large datasets. The method proposed in [23] applied a single-step of the input gradient to generate adversarial examples efficiently. DF was the first method to seek minimum-norm adversarial perturbations, employing an iterative approach. It linearizes the classifier at each step to estimate the minimal adversarial perturbations efficiently. C & Wattack [6] transform the optimization problem in [54] into an unconstrained optimization problem. C & Wleverages the first-order gradient-based optimizers to minimize a balanced loss between the norm of the perturbation and misclassification confidence. Inspired by the geometric idea of DF, FAB [11] presents an approach to minimize the norm of adversarial perturbations by employing complex projections and approximations while maintaining proximity to the decision boundary. By utilizing gradients to estimate the local geometry of the boundary, this method formulates minimum-norm optimization without the need for tuning a weighting term. DDN [48] uses projections on the ℓ 2 -ball for a given perturbation budget ϵ . FMN[40]

extends the DDN attack to other ℓ p -norms. By formulating (1) with Lagrange's method, ALMA [49] introduced a framework for finding adversarial examples for several distances.

Why does ℓ 2 white-box adversarial robustness matter? The reasons for using ℓ 2 norm perturbations are manifold. We acknowledge that ℓ 2 threat model may not seem particularly realistic in practical scenarios (at least for images); however, it can be perceived as a basic threat model amenable to both theoretical and empirical analyses, potentially leading insights in tackling adversarial robustness in more complex settings. The fact that, despite considerable advancements in AI/ML, we are yet to solve adversarial vulnerability, motivates part of our community to return to the basics and work towards finding fundamental solutions to this issue [9, 25, 34]. In particular, thanks to their intuitive geometric interpretation, ℓ 2 perturbations provide valuable insights into the geometry of classifiers. They can serve as an effective tool in the "interpretation/explanation" toolbox to shed light on what/how these models learn. Moreover, it has been demonstrated that [19, 38], ℓ 2 robustness has several applications beyond security (for more details on the necessity of robustness to ℓ p norms, please refer to Appendix O).

2 DeepFool (DF) and Minimal Adversarial Perturbations

In this section, we first discuss the geometric interpretation of the minimum-norm adversarial perturbations, i.e., solutions to the optimization problem in Eq. (1). We then examine DF to demonstrate why it may fail to find the optimal minimum-norm perturbation. Then in the next section, we introduce our proposed method that exploits DF to find smaller perturbations.

Let f : R d → R C denote a C -class classifier, where f k represents the classifier's output associated to the k th class. Specifically, for a given datapoint x ∈ R d , the estimated label is obtained by ˆ k ( x ) = argmax k f k ( x ) , where f k ( x ) is the k th component of f ( x ) that corresponds to the k th class. Note that the classifier f can be seen as a mapping that partitions the input space R d into classification regions, each of which has a constant estimated label (i.e., ˆ k ( . ) is constant for each such region). The decision boundary B is defined as the set of points in R d such that f i ( x ) = f j ( x ) = max k f k ( x ) for some distinct i and j . Additive ℓ 2 -norm adversarial perturbations are inherently related to the geometry of the decision boundary. More formally, Let x ∈ R d , and r ∗ ( x ) be the minimal adversarial perturbation defined as the minimizer of Eq. (1). Then:

2 Its norm, i.e., ∥ r ∗ ( x ) ∥ 2 measures the Euclidean distance between x and B , that is x + r ∗ lies

Properties of minimal adversarial perturbation → r ∗ ( x ) : 1 It is orthogonal to the decision boundary of the classifier B . on B .

We aim to investigate whether the perturbations generated by DF satisfy the aforementioned two conditions. Let r DF denote the perturbation found by DF for a datapoint x . We expect x + r DF to lie on the decision boundary. Hence, if r is the minimal perturbation, for all 0 < γ < 1 , we expect the perturbation γ r to remain in the same decision region as of x and thus fail to fool the model.

Fig. 2 illustrates the two conditions discussed in Section 2. In the figure, n 1 and n 2 represent two orthogonal vectors to the decision boundary. The optimal perturbation vector r ∗ aligns parallel to n 2 . On the other hand, a non-optimal perturbation r DF forms an angle α with n 1 .

In Fig. 3 (left), we consider the fooling rate of γ r DF for 0 . 2 < γ < 1 . For a minimum-norm perturbation, we expect an immediate sharp decline for γ close to one. However, in Fig. 3 (top-left) we cannot observe such a decline (a sharp decline happens close to γ = 0 . 9 , not 1). This is a confirmation that DF typically finds an overly perturbed point. One potential reason for this is the fact that DF stops when a misclassified point is found, and this point might be an overly perturbed one within the adversarial region, and not necessarily on the decision boundary.

Now, let us consider the other characteristic of the minimal adversarial perturbation. That is, the perturbation should be orthogonal to the decision boundary. We measure the angle between the found

perturbation r DF and the normal vector orthogonal to the decision boundary ( ∇ f ( x + r DF ) ). To do so, we first scale r DF such that x + γ r DF lies on the decision boundary. It can be simply done via performing a line search along r DF. We then compute the cosine of the angle between r DF and the normal to the decision boundary at x + γ r DF (this angle is denoted by cos( α ) ). A necessary condition for γ r DF to be an optimal perturbation is that it must be parallel to the normal vector of the decision boundary. In Fig. 3 (right) , we show the distribution of cosine of this angle. Ideally, we wanted this distribution to be accumulated around one. However, it clearly shows that this is not the case, which is a confirmation that r DF is not necessarily the minimal perturbation.

0.0

1.0

Fooling rate of

scaled

SDF

& DF

perturbations

0.8

0.6

0.4

0.2

0.2

0.4

0.6

0.8

1.0

DF

SDF

0.0

Orthognality of SDF

& DF

perturbations

0.2

0.4

0.6

0.8

1.0

SDF

DF

3 SuperDeepFool: Efficient Algorithms to Find Minimal Perturbations

In this section, we propose a new class of methods that modifies DF to address the aforementioned challenges in the previous section. The goal is to maintain the desired characteristics of DF, i.e., computational efficiency and the fact that it is parameter-free while finding smaller adversarial perturbations. We achieve this by introducing an additional projection step which its goal is to steer the direction of perturbation towards the optimal solution of Eq. (1). FMN Let us first briefly recall how DF finds an adversarial per- turbations for a classifier f . Given the current point x i , DF updates it according to the following equation:

Here the gradient is taken w.r.t. the input. The intuition is that, in each iteration, DF finds the minimum perturbation for a linear classifier that approximates the model around x i . The below proposition shows that under certain conditions, repeating this update step eventually converges to a point on the decision boundary.

C&W

Proposition 1 Let the binary classifier F 1 : R d → R be continuously differentiable and its gradient ∇F is β -Lipschitz. For a given input sample x 0 , suppose B ( x 0 , ε ) is a ball centered around x 0 with radius ε , such that there

∈ B ∥∇F∥ ≥ ∈ B iterations converge to a point on the decision boundary.

Proof: We defer the proof to the Appendix.

Notice while the proposition guarantees the perturbed sample to lie on the decision boundary, it does not state anything about the orthogonality of the perturbation to the decision boundary.

To find perturbations that are more aligned with the normal to the decision boundary, we introduce an additional projection step that steers the perturbation direction towards the optimal solution of

Eq. (1). Formally, the optimal perturbation, r ∗ , and the normal to the decision boundary at x 0 + r ∗ , ∇ f ( x 0 + r ∗ ) , should be parallel. Equivalently, r ∗ should be a solution of the following maximization problem:

which is the cosine of the angle between r and ∇ f ( x 0 + r ) . A necessary condition for r ∗ to be a solution of Eq. (3) is that the projection of r ∗ , i.e, ( P S ) on the subspace orthogonal to ∇ f ( x 0 + r ∗ ) should be zero. Then, r ∗ can be seen as a fixed point of the following iterative map:

The scalar multiplier on the right-hand side of Eq. (4) represents the norm of the projection of the vector r i along the gradient direction. The following proposition shows that this iterative process can converge to a solution of Eq. (3).

Proposition 2 For a differentiable f and a given r 0 , r i in the iterations Eq. (4) either converge to a solution of Eq. (3) or a trivial solution (i.e., r i → 0 ).

Proof: We defer the proof to the Appendix.

Intuitively, by the geometrical properties of a decision boundary ( B ), a small portion of the boundary can be enclosed between two affine parallel hyperplane. The following proposition from ([5]) states that the angle between ∇ f ( x ) and the optimal direction ∇ f ( x + r ∗ ) can be bounded in a neighborhood of the boundary B .

Proposition 3 ([5]) Given a radius r > 0 and Ψ r is the set of all samples whose distance from the decision boundary B is less than r . For each angle | θ | ∈ ( 0 , π 2 ) , there exists a distance ˜ r ( θ ) , such that, for all x ∈ Ψ ˜ r ( θ ) , the following inequality holds:

where P S is the unique projection of x on the B .

Proof: We defer the proof to the Appendix.

3.1 A Family of Adversarial Attacks

Finding minimum-norm adversarial perturbations can be seen as a multi-objective optimization problem, where we want f ( x + r ) = 0 and the perturbation r to be orthogonal to the decision boundary. So far we have seen that DF finds a solution satisfying the former objective and the iterative map Eq. (4) can be used to find a solution for the latter. A natural approach to satisfy both objectives is to alternate between these two iterative steps, namely Eq. (2) and Eq. (4). We propose a family of adversarial attack algorithms, coined SuperDeepFool, by varying how frequently we alternate between these two steps. We denote this family of algorithms with SDF ( m,n ) , where m is the number of DF steps Eq. (2) followed by n repetition of the projection step Eq. (4). This process is summarized in Algorithm 1. One interesting case is SDF ( ∞ , 1) which, in each iteration, continues DF steps till a point on the decision boundary is found and then applies the projection step.

This particular case has a resemblance with the strategy used in [44] to find black-box adversarial perturbations. This algorithm can be interpreted as iteratively approximating the decision boundary with a hyperplane and then analytically calculating the minimal adversarial perturbation for a linear

classifier for which this hyperplane is the decision boundary. It is justified by the observation that the decision boundary of state-of-the-art deep networks has a small mean curvature around data samples [21, 22]. A geometric illustration of this procedure is shown in Figure 5.

3.2 SDF Attack

We empirically compare the performance of SDF ( m,n ) for different values of m and n in Section 4.1. Interestingly, we observe that we get better attack performance when we apply several DF steps followed by a single projection. Since the standard DF typically finds an adversarial example in less than four iterations for state-of-the-art image classifiers, one possibility is to continue DF steps till an adversarial example is found and then apply a single projection step. We simply call this particular version SDF ( ∞ , 1) of our algorithm SDF, which we will extensively evaluate in Section 4.

SDF can be understood as a generic algorithm that can also work for the multi-class case by simply substituting the first inner loop of Algorithm 1 with the standard multi-class DF algorithm. The label of the obtained adversarial example determines the boundary on which the projection step will be performed. A summary of multi-class SDF is presented in Algorithm 2. Compared to the standard DF, this algorithm has an additional projection step. We will see later that such a simple modification leads to significantly smaller perturbations.

Table 1 demonstrates that SDF family outperforms DF in finding more accurate perturbations, particularly SDF( ∞ ,1) which significantly outperforms DF at a small cost.

Like any other gradient-based optimization method tackling a non-convex problem, providing a definitive explanation for why one algorithm outperforms others is not straightforward. We have the following speculation on why SDF ( ∞ , 1) consistently outperforms the other configurations: Note that each projection step reduces the perturbation, while each DF step moves the perturbation nearer to the boundary. So when projection is repeated multiple times ( n > 1 ),

it might undo the progress made by DF, potentially slowing down the algorithm's convergence. On the other hand, by first reaching a boundary point through multiple DF steps and then applying the projection operator just once, we at least ensure that the algorithm has reached intermediate adversarial examples. Each subsequent outer loop is hoped to incrementally move the adversarial example closer to the optimal point (see 5).

4 Experimental Results

In this section, we conduct extensive experiments to demonstrate the effectiveness of our method in different setups and for several natural and adversarially trained networks. We first introduce our experimental settings, including datasets, models, and attacks. Next, we compare our method with state-of-the-art ℓ 2 -norm adversarial attacks in various settings, demonstrating the superiority of our simple yet fast algorithm for finding accurate adversarial examples. Moreover, we add SDF to the collection of attacks used in AutoAttack, and call the new set of attacks AutoAttack++. This setup

meaningfully speeds up the process of finding norm-bounded adversarial perturbations. We also demonstrate that a model adversarially training using the SDF perturbations becomes more robust compared to the models 2 trained using other minimum-norm attacks. Please refer to Appendix B for details of the experimental setup and metrics.

4.1 Comparison with DeepFool (DF)

In this part, we compare our algorithm in terms of orthogonality and size of the ℓ 2 -norm perturbations especially with DF. Assume r is the perturbation vector obtained by an adversarial attack. First, we measure the orthogonality of perturbations by measuring the inner product between ∇ f ( x + r ) and r . As we explained in Section 2, a larger inner product between r and the gradient vector at f ( x + r ) indicates that the perturbation vector is closer to the optimal perturbation vector r ∗ . We compare the orthogonality of different members of the SDF family and DF.

The results are shown in Table 2. We observe that DF finds perturbations orthogonal to the decision boundary for low-complexity models such as LeNet, but fails to perform effectively when evaluated against more complex ones. In contrast, attacks from the SDF family consistently found perturbations with a larger cosine of the angle for all three models.

Verifying optimality conditions for SDF. We validate the optimality conditions of the perturbations generated by SDF using the procedure outlined in Section 2. Comparing Fig. 3 DF and SDF, it becomes

evident that our approach effectively mitigates the two issues we previously highlighted for DF. Namely, the alignment of the perturbation with the normal to the decision boundary and the problem of over-perturbation. We can see that unlike DF, the cosine of the angle for SDF is more concentrated around one, which indicates that the SDF perturbations are more aligned with the normal to the decision boundary. Moreover, Fig. 3 shows a sharper decline in the fooling rate (going down quickly to zero) when γ decreases. This is consistent with our expectation for an accurate minimal perturbation attack.

4.2 Comparison with minimum-norm attacks

We now compare SDF with SOTA minimum ℓ 2 -norm attacks: C&W, FMN, DDN, ALMA, and FAB. For C&W, we use the same hyperparameters as in [48]. We use FMN, FAB, DDN, and ALMA with budgets of 100 and 1000 iterations and report the best performance. For a fair comparison, we clip the pixel-values of SDF-generated adversarial images to [0 , 1] , consistent with the other minimum-norm attacks. We report the average number of gradient computations per sample, as these operations are computationally intensive and provide a consistent metric unaffected by hardware differences. We also provide a runtime comparison (Appendix Table 19).

We evaluate the robustness of the IBP model, which is adversarially trained on the MNIST dataset, against SOTA

attacks in Table 3. We choose this robust model as it allows us to have a more nuanced comparison between different adversarial attacks. SDF and ALMA are the only attacks that achieve a 100% percent fooling rate against this model, whereas C&W is unsuccessful on most of the data samples. The fooling rates of the remaining attacks also degrade when evaluated with 100 iterations. For instance, FMN's fooling rate decreases from 89% to 67 . 8% when the number of iterations is reduced from 1000 to 100 . This observation shows that, unlike SDF, selecting the necessary number of iterations is critical for the success of fixed-iteration attacks. Even for ALMA which can achieve

a nearly perfect FR, decreasing the number of iterations from 1000 to 100 causes the median norm of perturbations to increase fourfold. In contrast, SDF is able to compute adversarial perturbations using the fewest number of gradient computations while still outperforming the other algorithms, except ALMA, in terms of the perturbation norm. However, it is worth noting that ALMA requires twenty times more gradient computations compared to SDF to achieve a marginal improvement in the perturbation norm.

Table 4 compares SDF with SOTA attacks on the CIFAR10 dataset. The results show that SOTA attacks have a similar norm of perturbations, but an essential point is the speed of attacks. SDF finds more accurate adversarial perturbation very quickly rather than other algorithms.

We also evaluated all attacks on an adversarially trained model for the CIFAR10 dataset. SDF achieves smaller perturbations with half the gradient calculations than other attacks. SDF finds smaller adversarial perturbations for adversarially trained networks at a significantly lower cost than other at-

tacks, requiring only 20% of FAB's cost and 50% of DDN's and ALMA's (See Tables 11, 19 in the Appendix).

Table 5 demonstrates the performance of SDF on a naturally and adversarially trained models on ImageNet dataset. Unlike models trained on CIFAR10, where the attacks typically result in perturbations with similar norm, the differences between attacks are more nuanced for ImageNet models.

In particular, FAB, DDN, and FMN performance degrades when the dataset changes. In contrast, SDF achieves smaller perturbations at a signifi-

cantly lower cost than ALMA. This shows that the geometric interpretation of optimal adversarial perturbation, rather than viewing (1) as a non-convex optimization problem, can lead to an efficient solution. On the complexity aspect, the proposed approach is substantially faster than the other methods. In contrast, these approaches involve a costly minimization of a series of objective functions. We empirically observed that SDF converges in less than 5 or 6 iterations to a fooling perturbation; our observations show that SDF consistently achieves SOTA minimum-norm perturbations across different datasets, models, and training strategies, while requiring the least number of gradient computations. This makes it readily suitable to be used as a baseline method to estimate the robustness of very deep neural networks on large datasets.

4.3 SDF Adversarial Training (AT)

In this section, we evaluate the performance of a model adversarially trained using SDF against minimum-norm attacks and AutoAttack. Our experiments provide valuable insights into the effectiveness of adversarial training with SDF and sheds light on its potential applications in building more robust models. Adversarial training requires computationally efficient attacks, making costly options such as C & Wunsuitable. Therefore, an attack that is parallelizable (both on batch size and gradient computation) is desired for successful adversarial train-

ing. SDF possesses these crucial properties, making it a promising candidate for building more robust models.

We adversarially train a WRN28 -10 on CIFAR10. Similar to the procedure followed in [48], we restrict ℓ 2 -norms of perturbation to 2 . 6 and set the maximum number of iterations for SDF to 6 .

We train the model on clean examples for the first 200 epochs, and we then fine-tune it with SDF generated adversarial examples for 60 more epochs. Since a model trained using DDN-generated samples [48] has demonstrated greater robustness compared to a model trained using PGD [33], we compare our model with that one (for more details about AT please refer to Appendix O). Our model reaches a test accuracy of 90 . 8% while the model by [48] obtains 89 . 0% . SDF adversarially trained model does not overfit to SDF attack because, as Table 6 shows, SDF obtains the smallest perturbation. It is evident that SDF adversarially trained model can significantly improve the robustness of model against minimum-norm attacks up to 30% . In terms of comparison of these two adversarially trained models with AA, our model outperformed the [48] by improving about 8 . 4% against ℓ ∞ -AA, for ε = 8 / 255 , and 0 . 6% against ℓ 2 -AA, for ε = 0 . 5 .

Furthermore, compared to a network trained on DDN samples, our adversarially trained model has a smaller input curvature (Table 7). The second column shows the average spectral-norm of the Hessian w.r.t. input, ∥∇ 2 f ( x ) ∥ 2 , and the third column shows the average of the same quantity normalized by the norm of the input gradient, C f ( x ) = ∥∇ 2 f ( x ) ∥ 2 / ∥∇ f ( x ) ∥ 2 . The standard deviation is denoted by numbers enclosed in brackets.

This observation corroborates the idea that a more robust network will exhibit a smaller input curvature [1, 37, 39, 42, 47, 53].

AutoAttack++

Although it is not the primary focus of this paper, in this section we notably enhance the time efficiency of the AA [12] by incorporating SDF method into the set of attacks in AA.

We introduce a new variant of AA by introducing AutoAttack++ (AA++). AA is a reliable and powerful ensemble attack that contains three types of white-box and a strong black-box attacks. AA evaluates the robustness of a trained model to adversarial perturbations whose ℓ 2 / ℓ ∞ -norm is bounded by ε . By substituting SDF with the attacks in the AA, we significantly increase the performance of AA

Table 8: Analysis of robust accuracy for various defense strategies against AA++ and AA with ε = 0 . 5 for six adversarially trained models on CIFAR10. All models are taken from the RobustBench library [13].

in terms of computational time . Since SDF is an ℓ 2 -norm attack, we use the ℓ 2 -norm version of AA as well. We restrict maximum iterations of SDF to 10 . If the norm of perturbations exceeds ε , we renormalize the perturbation to ensure its norm stays ≤ ε . In this context, we have modified the AA algorithm by replacing APGD ⊤ [12] with SDF due to the former's cost and computation bottleneck in the context of AA (See Appendix F.1 for more details). Our decision to replace APGD ⊤ with SDF was primarily motivated by the former being a computational bottleneck in AA. As it is shown in Table 8, AA and AA++ achieve similar fooling rates, with AA++ being notably faster. We compared the sets of points that were fooled or not fooled by SDF/APGD ⊤ across 1000 samples ( ε = 0 . 5 ). The results indicate that both algorithms fool approximately the same set of points, differing only in a handful of samples for this epsilon value. Therefore, the primary benefit of using SDF is the reduction in computation time. We compare the fooling rate and computational time of AA++ and AA on the models from the RobustBench. In Table 8, we observe that AA++ is up to three times faster than AA. In an alternative scenario, we added the SDF to the beginning of the AA set, resulting in a version that is up to two times faster than the original AA, despite now containing five attacks (See Appendix F). This outcome highlights the efficacy of SDF in finding adversarial examples. These experiments suggest that leveraging efficient minimum-norm and non-fixed iteration attacks, such as SDF, can enable faster and more reliable evaluation of the robustness of deep models.

5 Conclusion and Future Works

In this work, we have introduced a family of parameter-free, fast, and parallelizable algorithms for crafting optimal adversarial perturbations. Our proposed algorithm, SDF, consistently finds smaller

norm perturbations on various networks and datasets with only a small additional computation cost compared to DF (which is still significantly faster than all SOTA attacks). Furthermore, we have shown that adversarial training using the examples generated by SDF builds more robust models. While our primary focus in this work has been on minimal ℓ 2 attacks, there exists potential for extending SDF families to other threat models, including general ℓ p -norms and targeted attacks. In the Appendix, we have demonstrated straightforward modifications that highlight the applicability of SDF to both targeted and ℓ ∞ -norm attacks. However, a more comprehensive evaluation remains a direction for future work. Moreover, further limitations of our proposed method are elaborated upon in Appendix N. In the end, by revisiting the necessity of ℓ p -norm robustness and characterizing a toy example on robustness-free phenomena, we underscore the pivotal role of minimum-norm attacks in ensuring secure AI systems.

6 Acknowledgments

We want to thank Kosar Behnia and Mohammad Azizmalayeri for their helpful feedback. We are very grateful to Fabio Brau and Jérôme Rony for providing code and models and answering questions on their papers.

References

Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training, 2020. 9

Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning , pages 274-283. PMLR, 2018. 17

Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in-and out-distribution improves explainability. In European Conference on Computer Vision , pages 228-245. Springer, 2020. 16

Yutong Bai, Jieru Mei, Alan Yuille, and Cihang Xie. Are transformers more robust than cnns?, 2021. 21

Fabio Brau, Giulio Rossolini, Alessandro Biondi, and Giorgio Buttazzo. On the minimal adversarial perturbation for deep neural networks with provable estimation error. arXiv preprint arXiv:2201.01235 , 2022. 5, 15

Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) , pages 39-57. Ieee, 2017. 2, 6, 23

Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705 , 2019. 23

Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness, 2022. 18

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024. 3

Jianbo Chen, Michael I Jordan, and Martin J Wainwright. Hopskipjumpattack: A query-efficient decision-based attack. In 2020 ieee symposium on security and privacy (sp) , pages 1277-1294. IEEE, 2020. 2

Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning , pages 2196-2205. PMLR, 2020. 2

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning , pages 2206-2216. PMLR, 2020. 2, 9

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670 , 2020. 9, 16

Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional nonconvex optimization. Advances in neural information processing systems , 27, 2014. 2

Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. arXiv preprint arXiv:1812.02637 , 2018. 9

Junhao Dong, Seyed-Mohsen Moosavi-Dezfooli, Jianhuang Lai, and Xiaohua Xie. The enemy of my enemy is my friend: Exploring inverse adversaries for improving adversarial training, 2022. 23

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. 16, 21

Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL https://github.com/MadryLab/robustness . 16

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations, 2019. 3

Christian Etmann, Sebastian Lunz, Peter Maass, and Carola-Bibiane Schönlieb. On the connection between adversarial robustness and saliency map interpretability, 2019. 23

Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. The robustness of deep networks: A geometrical perspective. IEEE Signal Processing Magazine , 34(6):50-62, 2017. 6

Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3762-3770, 2018. 6

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014. 2, 18

Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593 , 2020. 9

Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Querybased adversarial prompt generation, 2024. 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 18

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision , pages 630-645. Springer, 2016. 16

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017. 18

Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Geometric robustness of deep networks: analysis and improvement, 2017. 2, 24

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017. 18

Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient-based learning. In Shape, contour and grouping in computer vision , pages 319-345. Springer, 1999. 16

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015. 1

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017. 2, 9, 18, 22, 23

Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gardner. Adversarial prompting for black box foundation models. arXiv preprint arXiv:2302.04237 , 1(2), 2023. 3

Tomáš Mikolov, Anoop Deoras, Daniel Povey, Lukáš Burget, and Jan ˇ Cernock` y. Strategies for training large scale neural network language models. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding , pages 196-201. IEEE, 2011. 1

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 2, 22

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9078-9086, 2019. 9, 19

Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Optimism in the face of adversity: Understanding and improving deep learning through adversarial robustness, 2021. 2, 3

Guillermo Ortiz-Jimenez, Pau de Jorge, Amartya Sanyal, Adel Bibi, Puneet K. Dokania, Pascal Frossard, Grégory Rogez, and Philip Torr. Catastrophic overfitting can be induced with discriminative non-robust features. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=10hCbu70Sr . Expert Certification. 9

Maura Pintor, Fabio Roli, Wieland Brendel, and Battista Biggio. Fast minimum-norm adversarial attacks through adaptive norm constraints. Advances in Neural Information Processing Systems , 34:20052-20062, 2021. 2, 22

Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems , 29, 2016. 2

Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. Advances in Neural Information Processing Systems , 32, 2019. 9

Rahul Rade and Seyed-Mohsen Moosavi-Dezfooli. Helper-based adversarial training: Reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In ICML 2021 Workshop on Adversarial Machine Learning , 2021. 16, 17, 18, 23

Ali Rahmati, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, and Huaiyu Dai. Geoda: a geometric framework for black-box adversarial attacks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8446-8455, 2020. 2, 5

Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946 , 2021. 9, 21

Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning , pages 8093-8104. PMLR, 2020. 9

Elias Abad Rocamora, Fanghui Liu, Grigorios G. Chrysos, Pablo M. Olmos, and Volkan Cevher. Efficient local linearity regularization to overcome catastrophic overfitting, 2024. 9

Jerome Rony, Luiz G. Hafemann, Luiz S. Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 2, 6, 7, 8, 9, 16, 17, 22, 23

Jérôme Rony, Eric Granger, Marco Pedersoli, and Ismail Ben Ayed. Augmented lagrangian adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7738-7747, 2021. 2, 3

Kevin Roth, Yannic Kilcher, and Thomas Hofmann. Adversarial training is a form of datadependent operator norm regularization, 2020. 23

Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Improving adversarial robustness using proxy distributions. CoRR , abs/2104.09425, 2021. 9

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 18

Suraj Srinivas, Kyle Matoba, Himabindu Lakkaraju, and François Fleuret. Efficient training of low-curvature neural networks. In Advances in Neural Information Processing Systems . 9, 19

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013. 1, 2

Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems , 2020. 23

Jonathan Uesato, Brendan O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning , pages 5025-5034. PMLR, 2018. 18

Dongxian Wu, Shu tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization, 2020. 23

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016. 16

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy, 2019. 23

Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv:1906.06316 , 2019. 16

A Appendix

A.1 Proofs

Proof of Proposition 1.

Since ∇F ( x ) is Lipschitz-continuous, for x , y ∈ B ( x 0 , ε ) , we have:

DeepFool updates the new x n in each according to the following equation:

Hence if we substitute x = x n and y = x n -1 in (6), we get:

Now, let s n := || x n -x n -1 || . Using (8) and DeepFool's step, we get:

Using the assumptions of the theorem, we have βε ζ 2 < 1 , and hence s n converges to 0 when n →∞ . We conclude that { x n } is a Cauchy sequence . Denote by x ∞ the limit point of { x n } . Using the continuity of and Eq.(8), we obtain

F

Which concludes the proof of the theorem.

Proof of Proposition 2. Let us denote the acute angle between ∇ f ( x 0 + r i ) and r i by θ i ( 0 ≤ θ i ≤ π/ 2 ). Then from (4) we have | r i +1 | = | r i | cos θ i . Therefore, we get

Now there are two cases, either θ i → 0 or not. Let us first consider the case where zero is not the limit of θ i . Then there exists some ϵ 0 > 0 such that for any integer N there exists some n > N for which we have θ n > ϵ 0 . Now for ϵ 0 , we can have a series of integers n i where for all of them we have θ n i > ϵ 0 . Since we have 0 ≤ | cos θ | ≤ 1 , we have the following inequality:

The RHS of the above inequality goes to zero which proves that r i → 0 . This leaves us with the other case where θ i → 0 . This means that cos θ i → 1 which is the maximum of Eq. (3), this completes the proof.

Proof of proposition 3 ([5]) We use assumptions discussed in proposition 1 (the continuity of ∇ f ). We derive that there exists a distance r such that ∥∇ f ( x ) ∥ ̸ = 0 in ¯ Ψ r (the smallest closed set containing Ψ r ), and so we derive that ∇ f ∥∇ f ∥ is uniformly continuous in ¯ Ψ r . Hence, for each ε , there exists a distance r ε ≤ r such that, for each x , y ∈ ¯ Ψ r and ∥ x -y ∥ < r ε , the following inequality holds:

from triangle inequality for norms, we can derive:

In conclusion, by taking y = P S ( x ) and by choosing ε = √ 2 -2 cos( θ ) , we achieve upper bound for cos( θ ) where ˜ r ( θ ) = min( r max , r ε ) . Where r max is a maximum distance such that for each x in the Ψ r max there exists a P S ( x ) ∈ B solves the minimum-norm optimization problem.

B Setup

We test our algorithms on architectures trained on MNIST, CIFAR10, and ImageNet datasets. For MNIST, we use a robust model called IBP from [60] and naturally trained model called SmallCNN. For CIFAR10, we use three models: an adversarially trained PreActResNet-18 [27] from [43], a regularly trained Wide ResNet 28-10 (WRN-28-10) from [58] and LeNet [31]. These models are obtainable via the RobustBench library [13]. On ImageNet, we test the attacks on two ResNet-50 (RN50) models: one regularly trained and one ℓ 2 adversarially trained, obtainable through the robustness library [18]. We additionally evaluate the robustness of Vision Transformers (ViT-B-16 [17]) and reevaluate the comparative analysis between ViTs and CNNs.

C On the benefits of line search

As we show in Figure 3, DF typically finds an overly perturbed point. SDF's gradients depend on DF, so overly perturbing DF is problematic. Line search is a mechanism that we add to the end of our algorithms to tackle this problem. For a fair comparison between adversarial attacks, we add this algorithm to the end of other algorithms to investigate the effectiveness of line search. As shown in Table 9, we observe that line search can increase the performance of the DF significantly. However, this effectiveness for SDF is a little. We now measure the effectiveness of line search for other attacks. As observed from Table 10, line search effectiveness for DDN and ALMA is small.

D Comparison on CIFAR10 with the AT PRN18

In this section, we compare SDF with other minimum-norm attacks against an adversarially trained network [43]. In Table 11, SDF achieves smaller perturbation compared to other attacks, whereas it costs only half as much as other attacks.

E Performance comparison of adversarially trained models versus Auto-Attack (AA)

Evaluating the adversarially trained models with attacks used in the training process is not a standard evaluation in the robustness literature. For this reason, we evaluate robust models with AA. We perform this experiment with two modes; first, we measure the robustness of models with ℓ ∞ norm, and in a second mode, we evaluate them in terms of ℓ 2 norm. Tables 12 and 13 show that adversarial training with SDF samples is more robust against reliable AA than the model trained on DDN samples [48].

F Another variants of AA++

As we mentioned, in an alternative scenario, we added the SDF to the beginning of the AA set, resulting in a version that is up to two times faster than the original AA. In this scenario, we do not exchange the SDF with APGD. We add SDF to the AA configuration. So in this configuration, AA has five attacks (SDF, APGD, APGD ⊤ , FAB, Square). By this design, we guarantee the performance of AA. An interesting phenomenon observed from these tables is that when the budget increases, the speed of the AA++ increases. We should note that we restrict the number of iterations for SDF to 10 .

F.1 Why do we replace SDF with APGD ⊤ ?

It is well established that AutoAttack (AA) is a robust method for evaluating model robustness, unaffected by gradient obfuscation [2]. The primary limitation of AA, however, is its computational intensity. To thoroughly evaluate a model, it must be subjected to four distinct attacks sequentially.

(a) R1

(b) S

Our empirical analysis identified the APGD ⊤ attack as the main computational bottleneck in AA. For example, when attacking a standard WRN-28-10 model trained on CIFAR-10, APGD ⊤ requires approximately 4310 backward passes to achieve a 100 % fooling rate. Similarly, for an adversarially trained WRN-28-10 [8] model on CIFAR-10, APGD ⊤ necessitates around 5660 backward passes to attain a 100 % fooling rate. To address this issue, rather than simply replacing SDF with another minimum-norm attack such as FAB in AA, we mitigate the bottleneck by employing a faster minimumnorm attack like SDF.

G Why do we need stronger minimum-norm attacks?

Bounded-norm attacks like FGSM [23], PGD [33], and momentum variants of PGD [56], by optimizing the difference between the logits of the true class and the best non-true class, try to find an adversarial region with maximum confidence within a given, fixed perturbation size. Bounded-norm attacks only evaluate the robustness of deep neural networks; this means that they report a single scalar value as robust accuracy for a fixed budget. The superiority of minimum-norm attacks is to report a distribution of perturbation norms, and they do not report a percentage of fooling rates (robust accuracy) by a single scalar value. This critical property of minimum-norm attacks helps to accelerate to take an in-depth intuition about the geometrical behavior of deep neural networks.

We aim to address a phenomenon we observe by using the superiority of minimum-norm attacks. We observed that a minor change within the design of deep neural networks affects the performance of adversarial attacks. To show the superiority of minimum-norm attacks, we show how minimum-norm attacks verify these minor changes rather than bounded-norm attacks.

Modeling with max-pooling was a fundamental aspect of convolutional neural networks when they were first introduced as the best image classifiers. Some state-of-the-art classifiers such as [26, 30, 52] use this layer in network configuration. We use the pooling layers to show that using the max-pooling and Lp-pooling layer in the network design leads to finding perturbation with a bigger ℓ 2 -norm.

Assume that we have a classifier f . We train f in two modes until the training loss converges. In the first mode, f is trained in the presence of the pooling layer in its configuration, and in the second mode, f does not have a pooling layer. When we measure the robustness of these two networks with regular budgets used in bounded-norms attacks like PGD ( ε = 8 / 255 ), we observe that the robust accuracy is equal to 0% . This is precisely where bounded-norm attacks such as PGD mislead robustness literature in its assumptions regarding deep neural network properties. However, a solution to solve the problem of bounded-norm attack scan be proposed: " Analyzing the quantity of changes in robust accuracy across different epsilons reveal these minor changes." Is this case, the solution is costly. This is precisely where the distributive view of perturbations from worst-case to best-case of minimum-norm attacks detects this minor change.

To show these changes, we trained ResNet18 and Mobile-Net [28] in two settings. In the first setting, we trained them in the presence of a pooling layer until the training loss converged, and in the second setting, we trained them in the absence of a pooling layer until the training loss converged. We should note that we remove all pooling-layers in these two settings. For a fair comparison, we train models until they achieve zero training loss using a multi-step learning rate. We use max-pooling and Lp-pooling, for p = 2 , for this minor changes.

Table 14 shows that using a pooling layer in network configuration can increase robustness. DF has an entirely different behavior according to the presence or absence of the pooling layer; max-pooling affects up to 50% of DF performance. This effect is up to 9% for DDN and FMN. ALMA and SDF show a 4% impact in their performance, which shows their consistency compared to other attacks.

As shown in Table 15, we observe that models with pooling-layers have more robust accuracy when facing adversarial attacks such as AA and PGD. It should be noted that using regular epsilon for AA and PGD will not demonstrate these modifications. For this reason, we choose an epsilon for AA and PGD lower ( ε = 2 / 255 ) than the regular format ( ε = 8 / 255 ).

Table 14 and 15 demonstrate that pooling-layers can affect adversarial robustness of deep networks. Powerful attacks such as SDF and ALMA show high consistency in these setups, highlighting the need for powerful attacks.

G.1 Max-pooling's effect on the decision boundary's curvature

Here, we take a step further and investigate why max-pooling impacts the robustness of models. In order to perform this analysis, we analyze gradient norms, Hessian norms, and the model's curvature. The curvature of a point is a mathematical quantity that indicates the degree of non-linearity. It has been observed that robust models are characterized by their small curvature [37], implying smaller Hessian norms. In order to investigate robustness independent of non-linearity, [53] propose normalized curvature , which normalizes the Hessian norm at a given input x by its corresponding gradient norm. They defined normalized curvature for a neural network classifier f as C f ( x ) = ∥∇ 2 f ( x ) ∥ 2 / ( ∥∇ f ( x ) ∥ 2 + ε ) . Where ∥∇ f ( x ) ∥ 2 and ∥∇ 2 f ( x ) ∥ 2 are the ℓ 2 -norm of the gradient and the spectral norm of the Hessian, respectively, where ∇ f ( x ) ∈ R d , ∇ 2 f ( x ) ∈ R d × d , and ε > 0 is a small constant to ensure the proper behavior of the measure. In Table 16, we measure these quantities for two trained models, one with max-pooling and one without. It clearly shows that the model incorporating max-pooling exhibits a smaller curvature. This finding corroborates the observation that models with greater robustness tend to have a smaller curvature value.

H Model geometry for AT models

In this section we provide curvature analysis of our adversarially trained networks, SDF AT, and DDN AT model. Table 17 shows that our AT model decreases the curvature of network more than DDN AT model.

I CNN architecture used in Table 1

The architecture used to compare SDF variants and DF (Table 1) is summarized in above Table.

J ViT-B-16 for CIFAR-10

Given our available computational resources, we conduct experiments on a ViT-B-16 [17] trained on CIFAR-10, achieving 98.55 % accuracy. The results are summarized in the following table:

As seen, this transformer model does not exhibit significantly greater robustness compared to CNNs, with only a negligible difference of 0.01 compared to a WRN-28-10 trained on CIFAR-10. These results support the notion that there might not be a substantial disparity between the adversarial robustness of ViTs and CNNs. This aligns with the findings of [4]. They argue that earlier claims of transformers being more robust than CNNs stems from an unfair comparison and evaluation methods. We believe that thorough evaluations using minimum norm attacks could be helpful in resolving this debate.

K Natural (Regular) Trained MNIST Model

In Table 18 we show the results of evaluating adversarial attacks on naturally trained SmallCNN on MNIST dataset. Our algorithm demonstrates a higher rate of convergence compared to other algorithms, as the perturbations for all algorithms are generally similar.

L Runtime Comparison

We report the number of gradient computations as a main proxy for computional cost comparison. In Table 19, we have compared the runtime of different attacks for a fixed hardware. SDF is significantly faster.

M Query-Distortion Curves

20

40

60

80

100

120

Numbers of Forward Passes

1

.

0

1

.

1

1

.

2

1

.

3

1

.

4

‖

r

‖

Query-Distortion Curve

DF

SDF

Unlike FMN and ALMA, SDF (and DF) does not allow control over the number of forward and backward computations. They typically stop once a successful adversarial example is found. Terminating the process prematurely could prevent them from finding an adversarial example. Hence, we instead opted to plot the median norm of achievable perturbations for a given maximum number of queries (Figure 7) Although this is not directly comparable to the query-distortion curves in [40], it provides a more comprehensive view of the query distribution than the median alone.

N Limitations

In this section, we discuss some limitations and potential extensions of SDF.

Extension to other ℓ p -norms and targeted attacks. The proposed attack is primarily designed for ℓ 2 -norm adversarial perturbations. Moreover, our method, similar to DeepFool (DF), is non-targeted. Though there are potential approaches for adapting SDF to targeted and ℓ p attacks, these aspects remain largely unexplored in our work.

Nevertheless, we here demonstrate how one could possibly extend SDF to other p -norms. A simple way is to replace the ℓ 2 projection (Line 5 of Algorithm 2) with a projection operator minimizing ℓ p norm similar to the derivations used in [36]. In particular, for p = ∞ , the following projection would replace the line 5 of Algorithm 2:

In Table 20, we compare the performance of this modified version of SDF, named SDF ℓ ∞ with FMN, FAB, and DF, on two pretrained networks M1 [33] and M2 [48] on CIFAR-10 dataset. Our findings indicate that SDF ℓ ∞ also exhibits superior performance compared to other algorithms in discovering smaller perturbations.

Furthermore, we can convert SDF to a targeted attack by replacing the line 3 of Algorithm 2 with the targeted version of DeepFool, and the line 4 with the following:

where t is the target label. We followed the procedure outlined in [6] to measure the performance in the targeted setting. The result is summarized in Table 21. While SDF is effective in quickly finding smaller perturbations, it does not achieve a 100% fooling rate. Further analysis is required to understand the factors preventing SDF from converging in certain cases. This aspect remains an area for future work.

Convergence guarantees. A common challenge for all gradient-based optimization methods applied to non-convex problems is the lack of a guarantee in finding globally optimal perturbations for SotA neural networks. Obtaining even local guarantees is not trivial. Nevertheless, in Propositions 1 and 2 we worked towards this goal. We have established local guarantees showing the convergence of each individual operation, namely the DeepFool step and projection step. However, further analysis is needed to establish local guarantees for the overall algorithm.

Adaptive attacks. It is known that gradient-based attacks, ours included, are prone to gradient obfuscation/masking [7]. To counter this challenge, adaptation, as outlined in [55], is needed. It is also important to recognize that adapting geometric attacks such as SDF, does not follow a one-sizefits-all approach, as opposed to loss-based ones such as PGD. While this might be perceived as a weakness, it actually underscores a broader trend in the community. The predominant focus has been on loss-based attacks. This emphasis has inadvertently led to less exploration and development in the realm of geometric attacks.

O Vanila Adversarial Training

Vanila Adversarial Training without Additional Regularization. Our primary objective was to evaluate which adversarial attacks technique most effectively enhances robustness among PGD [33], DDN [48], and SDF. This focus differs from comparing various adversarial training strategies such as TRADES [59], TRADES-AWP [57], HAT [43], and UIAT [16]. These strategies often include additional regularization techniques to enhance Madry's method using PGD adversarial examples. Therefore, our assertion is not aimed at developing a state-of-the-art robust model. Instead, we aim to demonstrate that vanilla AT, when combined with minimum-norm attacks like SDF, can potentially outperform PGD-based models. Accordingly, we selected vanilla adversarial training with SDF-generated samples for our study and compared its effectiveness against a network trained with DDN samples. While TRADES or similar AT strategies could also integrate SDF, exploring this combination will be addressed in future research endeavors.

Why ℓ p norm is Critical? The existing literature has explored a variety of approaches to understanding adversarial examples. For example, training on ℓ p -norm adversarial examples has been identified as a form of spectral regularization [50], and adversarial perturbations, seen as counterfactual explanations, have been connected to saliency maps in image classifiers [20]. The rapid and accurate generation of these perturbations is critical for the empirical investigation of such phenomena. Moreover, minimal ℓ p adversarial perturbations are often considered "first order approximations of the decision boundary," illuminating the local geometric characteristics of models near data samples. This insight underscores the need for quick and precise methods for such explorations. Additionally, these

minimal perturbations provide a data-dependent, worst-case analysis of certain test-time corruptions, facilitating worst-case evaluations not only in the input space but also in the transformation space [29]. Within the context of Large Language Models (LLMs), these perturbations could potentially act as probing tools within their embedding space to examine their geometric properties. However, it is important to note that our interest in these topics was driven more by academic curiosity than by their practical applications in this specific study.

P Multi-class algorithms for SDF (1,3) and SDF (1,1)

Algorithm (3,4) summarizes pseudo-codes for the multi-class versions of SDF (1 , 1) and SDF (1 , 3) .

̸

̸

̸

̸

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Guidelines: The code to reproduce our experiments can be found at https://github. com/alirezaabdollahpour/SuperDeepFool

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper deals with fundamental questions regarding our understanding of deep networks. In this sense, it is subject to the same ethical concerns as the machine learning field as a whole, which makes it hard to identify potential direct risks or benefits associated to our empirical findings.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SuperDeepFool (SDF), a new family of adversarial attacks aimed at testing the robustness of deep neural networks against minimal L2 perturbations. The proposed attacks generalize the DeepFool (DF) attack, improving both computational efficiency and effectiveness. The authors show that SDF surpasses existing methods in both effectiveness and efficiency, making it ideal for evaluating large models and improving adversarial training to achieve state-of-the-art robustness.\n\nSTRENGTHS:\n1. The paper introduces a novel method that leverages the geometric properties of minimal L2 adversarial perturbations, offering an innovative perspective on adversarial attacks.\n2. The proposed method outperforms current state-of-the-art attacks by identifying smaller perturbations more efficiently, demonstrating superior effectiveness and computational efficiency.\n3. The authors conduct solid and comprehensive experimental validation, illustrating the superiority of SDF across various scenarios and benchmarks.\n4. The proposed method enhances the robustness of image classifiers through adversarial training, highlighting its practical applicability.\n\nWEAKNESSES:\n1. The paper focuses exclusively on L2 perturbations, which may not be the most practical or relevant threat model in real-world scenarios.\n2. The proposed method builds upon DeepFool, an existing state-of-the-art white-box attack, which may limit the perceived novelty and contribution of the work.\n\nQUESTIONS:\n1. Can the insights gained from the proposed approach be extended to develop efficient black-box attacks?\n2. Are there realistic and practical scenarios where the proposed attack can effectively evaluate the robustness of neural networks?\n\nLIMITATIONS:\n1. The paper focuses on a white-box threat model, which is less practical and realistic than the more challenging black-box threat model.\n2. The paper focuses on L2 norm attacks, which may not adequately assess network robustness against realistic perturbations such as physical attacks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SuperDeepFool (SDF), a new adversarial attack algorithm designed to evaluate the robustness of deep neural networks against L2-norm adversarial attacks. SDF generalizes the DeepFool (DF) attack by incorporating a projection step to find smaller perturbations while maintaining computational efficiency. The authors demonstrate that SDF outperforms many L2-norm attacks in terms of both effectiveness and computational cost on MNIST, CIFAR10 and ImageNet.\n\nSTRENGTHS:\nStrengths:\n\n1.\tEfficiency and theory justification: Based on the experimental results, SDF achieves a better trade-off between computational cost and attack efficiency compared to other baselines. Theoretical analysis is provided to support that such algorithm can converge to a point on the boundary.\n2.\tExperiments: The paper provides experimental results across three datasets, including a large one (ImageNet). It also shows that it can be combined with AutoAttack to further improve its strength.\n3.\tRobustness Improvement: The paper demonstrates that adversarial training using SDF-generated examples can better enhance the robustness image classifiers compared to DDN.\n\nWEAKNESSES:\nWeaknesses:\n\n1.\tClarity Issues: Generally speaking, it is well-organized, but there are some minor things make it not very clear. For example, on Line 123, the sentence “on about the geometrical…” misses the beginning part. In Section 3, $f$ represents a binary classifier when explaining the theory and algorithms in 3.1, but it was defined as a C-class classifier in Section 2. It is better to distinguish the two using different notations.\n2.\tNovelty: In fact, leveraging the geometry of decision boundary to improve the efficiency of adversarial attacks is not a new idea. This has been explored in many blackbox attacks, such as [1], [2]. A discussion of the relationship between the proposed idea and previous works would help improve the work.\n\n[1] Cheng, M., Singh, S., Chen, P., Chen, P. Y., Liu, S., & Hsieh, C. J. “Sign-opt: A query-efficient hard-label adversarial attack.” ICLR. 2020.\n\n[2] Chen, Jinghui, and Quanquan Gu. \"Rays: A ray searching method for hard-label adversarial attack.\" KDD. 2020.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nA discussion of limitation is not found in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel, parameter-free and computationally efficient minimal-$L_2$ adversarial attack. Building on the DeepFool attack and incorporating a novel geometric perspective, the SuperDeepFool attack achieves state-of-the-art success rates on selected MNIST, CIFAR10, and ImageNet classifiers. The authors demonstrate that SuperDeepFool maintains computational efficiency while achieving higher success rates in fooling neural networks compared to similar attacks. Adversarial training experiments show that this approach improves the evaluation of neural network robustness against adversarial attacks.\n\nSTRENGTHS:\nThe paper is clear, well-organized, and provides excellent geometric intuition and presentation. The illustrations are particularly effective in conveying complex concepts. The novel SuperDeepFool method demonstrates significant improvements over existing techniques by identifying the optimal adversarial point orthogonal to the decision boundary and employing an alternating optimization strategy. The paper's approach of combining the DeepFool attack with orthogonality optimization leads to higher success rates while maintaining computational efficiency. Additionally, the comprehensive comparisons and experiments highlight the effectiveness of SuperDeepFool across different models and datasets.\n\nWEAKNESSES:\nThe paper has several areas that need further elaboration. The measurements to test optimality are simple and strightfoward, particularly regarding whether a $\\gamma$ < 1 factor makes the adversarial perturbation fail. Hence, it is not clear whether a simple gamma optimization with DF could outperform SuperDeepFool (SDF). Comparisons in tables (e.g., Tables 2 and 3) lack clarity on success rates and how averages are computed. The comparison with Adversarial Training (AT) is very interesting but too brief, raising questions about the chosen perturbation size and the network's optimization against minimum-norm attacks. Additionally, the discussion on the comparison with auto attack is also too brief, and the rationale for switching from multitarget APGD (or other attacks) to single target SDF without evaluating the multitarget approach is unclear.\n\nQUESTIONS:\n1. Optimality Measurements: I believe comparing DF with simple $\\gamma$ optimization against SuperDeepFool (SDF) is essential to start with, since it's much faster and is presented as a measure of optimality.\n2. Comparison Tables: In tables such as 2 and 3, it is not clear if both DF and SDF achieve 100% success rates. Could you clarify the success rates and explain how averages are computed?\nHow do you account for variations across different models when presenting average success rates?\n3. Adversarial Training (AT) Comparison: The comparison in the Adversarial Training section is quite brief. Could you elaborate on whether the evaluated network is state-of-the-art in terms of robustness and how it is optimized to defend against minimum-norm attacks?\nThe perturbation size of 2.6 chosen for SDF seems arbitrary. Could you explain the rationale behind this choice, especially given the convention of using 0.5-norm attacks (other AT networks don't claim to be robust to larger perturbations)?\n4. Auto Attack Comparison: The discussion on the comparison with auto attack is also brief. Why did you choose to switch from multitarget APGD to single target SDF without evaluating the multitarget approach?\nDoes multitarget SDF achieve better robust accuracy?\n\nLIMITATIONS:\nThe authors discussed the limitations of their experimental work - the limited evaluation of the SDF methods on a big variance of robust and non-robust classifiers, and extensions to targeted attacks and different norms. (no justifications for limitations on the checklist)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper introduces a new family of adversarial attacks called SuperDeepFool (SDF) attack, extending the well-known DeepFool (DF) attack. This novel approach strikes an proper balance between effectiveness and efficiency, and consistently outperforms existing methods in terms of both. Additionally, the method can be adapted for robustness evaluation and adversarial training.

STRENGTHS:
1. The concept of integrating DF with minimal adversarial perturbations is both novel and intriguing.
2. The theoretical analysis is sound: DF iterations converge to a point on the decision boundary, while SDF iterations converge to a point orthogonal to the decision boundary.
3. Comprehensive experiments demonstrate SDF's remarkable performance compared to existing methods, in terms of both Fooling Rate (FR) and the number of gradient computations (Grads).
4. The experiments consistently show improvements over DF and minimum-norm attacks, together with its potential for adaptation in adversarial training (AT).

WEAKNESSES:
1. The method is limited to the $\ell_2$-norm.
2. Most of the experiments are conducted with $\varepsilon=0.5$.

QUESTIONS:
1. Can the method be generalized to the $\ell_p$-norm?
2. Do we expect similar performance for smaller $\varepsilon$, e.g. $8/255$?
3. Is the case SDF(m,$\infty$) also interesting?
4. How do we determine the number of iterations in SDF beforehand?

LIMITATIONS:
The potential limitations are outlined in the weaknesses and questions sections.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12