How to compare adversarial robustness of classifiers from a global perspective
Authors: Niklas Risse
Jan Philip Göpfert
Christina Göpfert
Venue:  (2021)
Keywords: adversarial robustness
robustness
adversarial defense
adversarial example

Abstract: Adversarial robustness of machine learning models has attracted considerable attention over recent years. Adversarial attacks undermine the reliability of and trust in machine learning models, but the construction of more robust models hinges on a rigorous understanding of adversarial robustness as a property of a given model. Point-wise measures for specific threat models are currently the most popular tool for comparing the robustness of classifiers and are used in most recent publications on adversarial robustness. In this work, we use robustness curves to show that point-wise measures fail to capture important global properties that are essential to reliably compare the robustness of different classifiers. We introduce new ways in which robustness curves can be used to systematically uncover these properties and provide concrete recommendations for researchers and practitioners when assessing and comparing the robustness of trained models. Furthermore, we characterize scale as a way to distinguish small and large perturbations, and relate it to inherent properties of data sets, demonstrating that robustness thresholds must be chosen accordingly. We hope that our work contributes to a shift of focus away from point-wise measures of robustness and towards a discussion of the question what kind of robustness could and should reasonably be expected. We release code to reproduce all experiments presented in this paper, which includes a Python module to calculate robustness curves for arbitrary data sets and classifiers, supporting a number of frameworks, including TensorFlow, PyTorch and JAX.

Under review as a conference paper at ICLR 2021
HOW TO COMPARE ADVERSARIAL ROBUST- NESS OF CLASSIFIERS FROM A GLOBAL PER- SPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial robustness of machine learning models has attracted considerable attention over recent years. Adversarial attacks undermine the reliability of and trust in machine learning models, but the construction of more robust models hinges on a rigorous understanding of adversarial robustness as a property of a given model. Point-wist measures for specific threat models are currently the most popular tool for comparing the robustness of classifiers and are used in most recent publications adversarial robustness. In this work, show that point-wise measures fail to capture important global properties that are essential reliably compare the robustness of different classifiers. We introduce new ways in which robustness curves can be used to systematically uncover these properties and provide concrete recommendations for researchers and practitioners when assessing and comparing the robustness of trained models. Furthermore, we characterize scale as a way to distinguish small and large perturbations, and relate it to inherent properties of data sets, demonstrating that robustness thresholds must be chosen accordingly. We hope that our work contributes to a shift of focus away from point-wise measures of robustness and towards discussion of the question what kind robustness could and should reasonably be expected. We release code reproduce all experiments presented in this paper, which includes Python module to calculate robustness curves for arbitrary data sets and classifiers, supporting number of frameworks, including TensorFlow, Py PyTorch and JAX.
robustness curves
INTRODUCTION
Despite their astonishing success in a wide range of classification tasks, deep neural networks can be lead to incorrectly classify inputs altered with specially crafted adversarial perturbations (Szegedy et al. 2014; Goodfellow et al. 2015). These perturbations can be so small that they remain almost imperceptible to human observers (J. P. Göpfert al. 2020). Adversarial robustness describes model's s ability to behave correctly under such small perturbations crafted with the intent to mislead the model. The study of adversarial robustness with its definitions, their implications, attacks, and defenses - has attracted considerable research interest. This is due to both the practical importance of trustworthy models as well the intellectual interest in the differences between decisions of machine learning models and our human perception. A crucial starting point for any such analysis is the definition of what exactly small input perturbation function to measure perturbation size, and (b) the choice of particular seale to distinguish small and large perturbations. Together, these two choices determine a threat model that defines exactly under which perturbations model is required to be robust.
requiring (a) the choice of a distance
The most popular choice of distance function is the class of distances induced by l norms (Szegedy et al. 2014; Goodfellow et al. 2015; Carlini, Athalye, et al. 2019), in particular ε,, &2 and foc. although other choices such as Wasserstein distance have been explored as well (Wong, Schmidt, et al. 2019). Regarding scale, the current default is to pick some perturbation threshold without providing concrete reasons for the exact choice. Analysis then focuses on the robust error of the model, the proportion of test inputs for which the model behaves incorrectly under some perturbation up to size e. This means that the scale defined as a binary distinction between small and large perturbations based on the perturbation threshold. A of canonical thresholds have emerged in

---

Under review as a conference paper at ICLR 2021
the literature. For example, in the publications referenced in this section, the MNIST data set is typically evaluated at a perturbation threshold E E {0.1, 0.3) y for the foe norm, while CIFAR-10 is evaluated at E E f2/255, 4/255, 8/255), stemming from the three 8-bit color channels used to represent images.
Based these established threat models, researchers have developed specialized methods to minimize the robust error during training, which results in more robust models. Popular approaches include specific data augmentation, sometimes used under the umbrella term adversarial training (Guo et al. 2017; Madry et al. 2018; Carmon et al. 2019; Hendrycks et al. 2019), training under regularization that encourages large margins and smooth decision boundaries in the learned model (Hein and Andriushchenko 2017; Wong and Kolter 2018; Croce, Andriushchenko, and Hein 2019; Croce and Hein 2020), and post-hoe processing or randomized smoothing of predictions in a learned model (Lecuyer et al. 2019; Cohen et al. 2019).
In order to show the superiority of a new method, robust accuracies of differently trained models are typically compared for a handful of threat models and data sets, eg., (s 0.1) and (2 0.3) for MNIST. Out of 22 publications on adversarial robustness published at NeurlPS 2019, ICLR 2020, and ICML 2020, 12 publications contain results for only single perturbation threshold. In five publications, robust errors are calculated for at least two different perturbation thresholds, but still, only arbitrary number of thresholds is considered. Only in five out of the total 22 publications do we find extensive considerations of different perturbation thresholds and the respective robust errors. Out of these five, three are analyses of randomized smoothing, which naturally gives rise to certification radii (B. Li et al. 2019; Carmon et al. 2019; Pinot et al. 2019). Najafi et al. (2019) follow a learning-theoretical motivation, which results in an error bound as a function of the perturbation threshold. Only Maini et al. (2020) do not rely on randomization and still provide a complete, empirical analysis of robust error for varying perturbation thresholds'.
Our contributions: In this work, we demonstrate that point-wise measures of ép robustness are not sufficient to reliably and meaningfully compare the robustness of different classifiers. We show that, both in theory and practice, results of model comparisons based on point- point-wise measures may fail to generalize to threat models with even slightly larger or smaller & and that robustness curves avoid this pitfall by design. Furthermore, we show that point-wise measures are insufficient to meaningfully compare the efficacy of different defense techniques when distance functions are varied, and that robustness curves, again, are able to reliably detect and visualize this property. Finally, we analyze how scale depends on the underlying data space, choice of distance function, and distribution. Based on our findings we suggest that robustness curves should become the standard tool when comparing adversarial robustness of classifiers, and that the perturbation threshold of threat models should be selected carefully in order to be meaningful, considering inherent characteristics of the data set. We release code to reproduce all experiments presented in this paper", which includes a Python module with an easily accessible interface (similar to Foolbox, Rauber et al. (2017)) calculate robustness curves for arbitrary data sets and classifiers. The module supports classifiers written in most of the popular machine learning frameworks, such as TensorFlow, Py and JAX.
METHODS
An adversarial perturbation for a classifier f and input-output pair (a., y) is small perturbation 8 with f(x + 5) # y. Because the perturbation 8 is small, it is assumed that the label would still be the correct prediction for x + 8. The resulting point x + is called an adversarial example. The points vulnerable to adversarial perturbations are the points that are either already misclassified when unperturbed, or those that lie close to a decision boundary.
One tool to visualize and study the robustness behavior of a classifier are robustness curves, first used by Wong and Kolter (2018) and later formalized by C. Göpfert et al. (2020). A robustness curve
SSingee thresholds: (Mao et al. 2019; Tramer and Boneh 2019; Alayrac et al. 2019; Brendel et al. 2019; Qin et al. 2019; Wang et al. 2020; Song et al. 2020; Croce and Hein 2020; Xie and Yuille 2020; Rice et al. 2020; Zhang al. 2020; Singla and Feizi 2020). multiple thresholds: (Lee et al. 2019; Mahloujifar al. 2019; Hendrycks al. 2019; Wong, Rice, et al. 2020; Roopathy al. 2020), full analysis: (Pinot al. 2019; Carmon et al. 2019 9; B. Li et al. 2019; Najafi et al. 2019; Maini et al. 2020).
2The full code is available at https: : / / thub.com wttphh..wo/.annnnoss///2a Annoymouus23844022334441 compa re adwersaaddrsr wwwustrnsssssse.........o of cll fiers- fww.-a-oo---aaaeea obal perspective.
2

---

Under review as a conference paper at ICLR 2021
2s
1.0
Smouth
scurially
0.6
0.4
0.2
0.0
Eg perturbation threshold
Figure 1: Excerpt of toy data set with two decision boundaries (left) and respective robustness curves (right). The data is separated perfectly by one smooth boundary (blue robustness curve), and squiggly boundary (orange robustness curve). We indicate margins around the boundaries at distances and 2e. Selecting single perturbation threshold is not sufficient to decide which classifier more robust.
captures the distribution of shortest distances between a of points and the decision boundaries of classifier:
Definition 1. Given an inpul space X and label set y, distance function d on X x X, and classifier J: + y. Assume (x, y) miid. P for some distribution P on X Y. Then the d-robustness curve
for I is the graph of the function
R1(() == P (((x.)) S... 3x' ((x,, (
+1)
model's robustness curve shows how data points are distributed in relation the decision bound- aries of the model, essentially visualizing simultaneously an extremely large number of point-wise measures. This allows us take a step back from robustness regarding a specific perturbation thresh- old and instead compare global robustness for different classifiers, distributions and distance functions. To see why this relevant, consider Figure 1, which shows toy data along with two possible classifiers that perfectly separate the data. For perturbation threshold of e, the blue classifier has robust error 0.5, while the orange classifier is perfectly robust. However, for a perturbation threshold 2s, the orange classifier has robust error 1, while the blue classifier remains 0.3. By freely choosing single perturbation threshold for comparison, it is therefore possible to make either classifier appear to be much better than the other, and single threshold can capture the whole picture. In fact, for any two disjoint sets of perturbation thresholds, it is possible to construct a data distribution and two classifiers f. J', such that the robust error of f is lower than that of f' for all perturbation thresholds in the first set, and that of f' is lower than that of f for all perturbation thresholds in the second set. See Appendix A for constructive proof. This shows that even computing multiple point-wise measures to compare two models may give misleading results.
3 EXPERIMENTS
In the following, we empirically evaluate the robustness of a number of recently published models, and demonstrate that the weaknesses of point-wise measures described above are not limited to toy examples, but occur for real-world data and models.
3.1 EXPERIMENTAL. SETUP
We evaluate and compare the robustness models obtained using the following training methods:
1. Standard training (ST), i.e training without specific robustness considerations.
2. Adversarial training (AT) (Madry et al. 2018).
3. Training with robust loss (KW) (Wong and Kolter 2018).
4. Maximum margin regularization for a single L, norm together with adversarial training
(MMR AT) (Croce, Andriushchenko, and Hein 2019).
5. Maximum margin regularization simultancously for Ex and l margins (MMR-UNIC (Croce
and Hein 2020).

---

Under review as a conference paper at ICLR 2021
Table Three point-wise measures for different threat models. All threat models use the loo distance function, but differ in choice of perturbation threshold (denoted by e). Each row contains the robust test errors for one point-wise measure. Each column contains the robust test errors for one model, trained with specific training method (marked by column title). The lower the number, the better the robustness for the specific threat model. Each point-wise measure results a different relative ordering of the classifiers based on the errors. The order is visualized by different tones of gray in the background of the cells.
6
ST
AT
KW
MMR - AT
MMR-UNIV
1/255
0.60
0,38
0.43
0.42
0.54
4/255
0.99
0.68
0.57
0.63
0.74
8/255
1.00
0.92
0.73
0.84
0.91
Together with each training method, we state the threat model the trained model is optimized to defend against, eg., fo = 0.1) for perturbations lx norm with perturbation threshold e 0.1, if any. The trained models are those made publicly available by Croce, Andriushchenko, and Hein (2019)² and Croce and Hein (2020) The network architecture is a convolutional network with two convolutional layers, two fully connected layers and ReLU activation functions. The evaluation is based on six real-world datasets: MNIST, Fashion-MNT (FMNIST) (Xiao et al. 2017), German Traffic Signs (GTS) (Houben al. 2013), CIFAR- (Krizhevsky 2009), Tiny-Imagenet- 200 (TINY- IMG) (F.FF Li et al. 2016), and Human Activity Recognition (HAR) (Anguita et al. 2013). For specifics on model training (hyperparameters, architecture details), refer to Appendix Models are generally trained on the full training set for the corresponding data set, and robustness curves evaluated on the full test set, unless stated otherwise.
For complex models, calculating the exact distance of a point to the closest decision boundary, and thus estimating the true robustness curve, is computationally very intensive, not intractable. Therefore we bound the true robustness curve from below using strong adversarial attacks, which is consistent with the literature on empirical evaluation adversarial robustness and also applicable to many different types of classifiers. We base our selection of attacks on the recommendations by Carlini, Athalye, et al. (2019). Specifically, we use the lz-attack proposed by (Carlini and Wagner 2017) for &g robustness curves and PGD (Madry et al. 2018) for &o robustness curves. For both attacks, we use the implementations of Foolbox (Rauber et al. 2017). See Appendix C for information on adversarial attack hyperparameters. In the following, "robustness curve" refers this empirical approximation of the true robustness curve.
3.2 THE WEAKNESSES OF POINT- WISE MEASURES
Point-wise measures are used to quantify robustness of classifiers by measuring the robust test error for specific distance function and perturbation threshold (eg., (s 4/255)). In Table 1 we show three point-wise measures to compare the robustness five different classifiers on CIFAR-10. If we compare the robustness of the four robust training methods (latter four columns of the table) based on the first point-wise threat model Ex (s 1/255) (first row of the table), we can see that the classifier trained with AT is the most robust, followed by MMR AT, followed by KW, and MMR-UNIV results in the least robust classifier. However, if we increase the E of our threat model to 4/255 (second row of the table), KW is more robust than AT. For even larger E (third row of the table), we would conclude that MMR-UNTV is preferable over AT, and that AT results in the least robust classifier. All three statements are true for the particular perturbation threshold (c), and the magnitude of all perturbation thresholds is reasonable: publications CIFAR- 10 on perturbation thresholds < 10/255 for foo perturbations. Meaningful conclusions on the robustness of the classifiers relative each other can not be made without taking all possible into account. In other words, a global perspective is needed.
adversarial robustness typically evaluate
'hhe models trained with sT, KW, AT and MMR + AT are avaible at www.gi t thub.com/ boooo//a---a/ ///mmxxaamnn maax-andr/ae andr/provabl e- wwweeneesssssnneeeeeee "The models trained with MMR-UNIV V are avaible www.github.com..cc fra31 /tmr//niveerreeeee versal.
4

---

Under review as a conference paper at ICLR 2021
1.0
1.0
Standard Training
0.66
E 0.1)
0.4
Coo
2/265)
0.44
Egit
MMR+AT
MMR-AT
MMR UNIVERSAL
Coo. 2/2655
0.2
MMR UNIVERSAL Sp rerms
Fale
Ep nomm
0.0
1/300
1/205
77203
0.0
0.00
0.06
0.00
0.10
D:15
0.25
0.30
Ego perturbation threshold
(2 perturbation threshold e
Figure 2: fo robustness curves (left plot) and &g robustness curves (right plot) resulting from different training methods (indicated by label), optimized for different threat models (indicated by label). The dashed vertical lines visualize the three point-wise measures from Table 1. The models trained and evaluated on the full training-/test sets of CIFAR-IE -c. The curves allow us to reliably compare the robustness of the classifiers, unbiased by choice of perturbation threshold.
3.2.1 A GLOBAL PERSPECTIVE
Figure 2 shows the robustness of different classifiers for the too (right plot) and & (left plot) distance functions from global perspective using robustness curves. The plot reveals why the three point- wise measures (marked by vertical black dashed lines in the left plot) lead to different results the relative ranking of robustness of the classifiers. Both for the classifiers trained to be robust against attacks in too distance (left plot) and lg distance (right plot), we can observe multiple intersections of robustness curves, corresponding changes the relative ranking of the robustness of the compared classifiers. The robustness curves allow us to reliably compare the robustness of classifiers for all possible perturbation thresholds. Furthermore, the curves clearly show the perturbation threshold intervals with strong and weak robustness for each classifier, and are not biased by an arbitrarily chosen perturbation threshold.
3.2.2 OVERFITTING TO SPECIFIC PERTURBATION THRESHOLDS
In addition to the problem of robustness curve intersection, relying on point-wise robustness mea- sures to evaluate adversarial robustness is prone to overfitting when designing training procedures. Figure shows lx robustness curves for MMR AT with foo threat model provided by Croce, An- driushchenko, and Hein (2019). The models trained on MNIST and FMNIS' ST both show change in slope, which could be sign of overfitting to the specific threat models for which the classifiers were optimized for, since the change of slope occurs approximately at the chosen perturbation threshold e. This showcases a potential problem with the use point-wise measures during training. The binary separation of "small" and "large" perturbations based on the perturbation threshold is not sufficient to capture the intricacies human perception under perturbations, but a simplification based on the idea that perturbations below the perturbation threshold should almost certainly not lead to change in classification. If training procedure moves decision boundaries so that data points lie just beyond this threshold, it may achieve low robust error, without furthering the actual goals of adversarial robustness research. Using robustness curves for evaluation cannot prevent this effect, but can be used to detect it.
3.2.3 TRANSFER OF ROBUSTNESS ACROSS DISTANCE FUNCTIONS
In the following, we analyze to which extent properties of robustness curves transfer across different choices of distance functions. If properties transfer, may not be necessary to individually analyze robustness for each distance function.
In Figure 4 we compare the robustness of different models for the loo (left plot) and &a (right plot) distance functions. The difference to Figure 2 is that the models (indicated by colour) are the same models in the left plot and in the right plot. We find that for MMR + AT, the for threat model leads to better robustness than the & threat model both for foo and lg robustness curves. In fact, MMR + AT with the too threat model even leads to better Eoo and & robustness curves than MMR-UNIV which is specifically designed to improve robustness for all ép norms. Overall, the plots visually similar.
5

---

Under review as a conference paper at ICLR 2021
1.0
MNIST
FMNIST
CIFARLO
FMNIST
CIFAR10
0.0
0.2 0.6
0.2
0.8
0.00
0.05
0:20
foo perturbation threshold e
8oc perturbation threshold
6sc
perturbation threshold
Figure 3: fo robustness curves for multiple data sets. Each curve is calculated for a different model and a different test data set. The data sets are indicated by the labels. The models are trained with MMR AT, Threat Models: MNI ST: (E 0.1), FMNIST: foo (e 0.1), GTS: (e 4/255), CIFAR-10: foo (E 2/255). The curves for MNIST and FMNIST I both show change in slope, which can not captured with point-wise measures and could be sign of overfitting to the specific threat models for which the classifiers were optimized for.
1.0
1.0
Training
Training MMR--T
MMR+AT
2/255)
MMR++T
Fr
MMR UNIVERSAL
MMR UNIVERSAL
all Ep nomss
Ep norms
0.0 2/30
0.0
2:28
0.00
0.02
0.00
0.00 0.05 0.10
0.20 0.25
For perturbation threshold
f2 perturbation threshold 6
Figure 4: &or robustness curves (left plot) and & robustness curves (right plot) resulting from different training methods (indicated by color and label), optimized for different threat models (indicated by label). The models are trained and evaluated on the full training- sets of CIFAR-100 The curves allow us to reliably compare the transfer of robustness of the classifiers across distance functions, unbiased by choice of threat model.
However, since both plots contain multiple robustness curve intersections, the ranking of methods remains sensitive to the choice of perturbation threshold. For example, a perturbation threshold of 3/255 (vertical black dashed line) for the foc distance function (left subplot) shows that the classifier trained with MMR + AT (E 0.1)) is approximately as robust as the classifier trained with MMR-UNIV. The same perturbation threshold for the f2 distance function (right subplot) shows that the classifier trained with MMR AT is more robust than the classifier trained with MMR-UNIV for & threat models. Using typical perturbation thresholds from the literature for each distance function does not alleviate this issue: At perturbation threshold
2/255 for fo distance, the classifier trained with MMR AT (E2(E= = 0.1)) is more robust than the one trained with MMR-UNIV, while at perturbation threshold robustness curves across various distance functions are qualitatively similar, this may be obscured by the choice of threat model(s) to compare
0.1 for & distance, the opposite is true. This shows that even when
We also emphasize that general, robustness curves across various distance functions may be qualitatively dissimilar. In particular:
For linear classifiers, the shape of robustness curve is identical for distances induced by different f, norms. This follows from Theorem 2 in Appendix B, which is extension of weaker result in C. Göpfert et al. (2020). For non-linear classifiers, different Ép norms may induce different robustness curve shapes. See C. Göpfert et al. (2020) for example.
2. Even for linear classifiers, robustness curve intersections do not transfer between distances induced by different l, norms. That is, for two linear classifiers, there may exist p. y/ such that the robustness curves for the f, distance intersect, but not the robustness curves for the by distance. See Appendix A for an example.

---

Under review as a conference paper at ICLR 2021
1.00
percentage of paints
PUNST GTS
TRY IMAGEEET BAR CINAROO
MUST
MRAST
GT3
20
4000
distance Roo
distance in
distance &1
Figure 5: Minimum inter-class distances of all data sets considered in this work, measured in foo (left), & (middle), and & (right) norm. See Table for size and dimensionality. The shapes of the curves and the threshold from which any classifier must necessarily trade of between accuracy and robustness differ strongly between data sets.
3.3 ON THE RELATIONS BETWEEN SCALE AND DATA
As the previous sections show, robustness curves can used to reveal properties of robust models that may be obscured by point-wise measures. However, some concept of scale, that is, some way to judge whether a perturbation is small or large, remains necessary. Especially when robustness curves intersect, it is crucial to be able to judge how critical it is for model to be stable under the given perturbations. For many pairs of distance function and data set, canonical perturbation thresholds have emerged in the literature, but to the best of our knowledge, reasons for these choices given.
Since the assumption behind adversarial examples is that small perturbations should not affect classification behavior, the question of scale cannot be answered independently of the data distribution. In order to understand how to interpret difffernnt perturbation sizes, can be helpful to understand how strongly the data point would need to be perturbed to actually change the correct classification. We call this the inter-class distance and analyze the distribution of inter-class distances for several popular data sets.
In Figure we compare the inter-class distance distributions in loo. l2. and & norm for all data sets considered in this work. We observe that for the & and l2 norms, the shape of the curves is similar across data sets, but their extent determined by the dimensionality of the data space, the too norm, vastly different curves emerge for the different data sets. We hypothesize that, because the inter-class distance distributions vary more strongly for foo distances than for ε distances, the results of robustifying model r. foo distances may depend more strongly the underlying data distribution than the results of robustifying w. r. & distances. This is an interesting avenue for future work.
When we look at the smallest inter-class distances in the loo norm (where all distances lie in the interval [0, 1)), we make several observations. Because the smallest inter-class distance for MNIST in the for norm is around 0.9, we can see that transforming an input from one class to one from a different class almost always requires completely flipping least one pixel from almost-black to almost-white or vice versa. For the other datasets, the inter-class distance distributions are more spread than the inter-class distance distribution of MNIST. We observe that for CIFAR-1 with lo perturbations of size > 0.25, it becomes possible to transform samples from different classes into each other, so starting from this threshold, any classifier must necessarily trade off between accuracy and robustness. The shapes of the curves and the threshold from which any classifier must necessarily trade of between accuracy and robustness differ strongly between data sets refer to Table for exact values for the threshold.
In Table 2, we summarize the smallest and largest inter-class distances different norms together with additional information about the size, number of classes, and dimensionality of the all the data sets we consider this work. The values correspond directly to Figure 5, but even in this simplified view,
quickly make out key differences between the data sets. Compare, for example. MN IST and GTS: While it appears reasonable to expect foo robustness of 0.3 for MNIST, the same threshold for GTS is not possible. Relating Table 2 and Figure find entirely plausible the strong robustness
7

---

Under review as a conference paper at ICLR 2021
Table Smallest and largest inter-class distances for subsets of several data sets, measured in loo l2, and L norm, together with basic contextual information about the data sets. All data has been been normalized to lie within the interval [0, I], and duplicates and corrupted data points have been removed. Apart from HAR, all data sets contain images the dimensionality reported specifies their sizes and number of channels.
Inter-class Distance
Smallest
Largest
Dataset
Samples
Classes
Dimensionality
1o
12
4
1
12
/
MNIST
10000
10
28 x 28 x
0.88
3.03
19.16
1.00
10.18
132.38
TINY-IMG
98 139
200
64 x 64 x 3
0.27
5.24
369.29
0.71
47 49
4184.37
FM
10000
10
28 x
28 x
0.36
2.00
21.87
1.00
10.70
194.29
GTS
10000
43
32
32
x 3
0.07
0.90
31.46
0.62
19.54
833.22
CIFAR-10 10
10000
32
0.27
3.61
130.77
0.70
18.57
831.44
HAR
2947
6
561
0.26
1.26
12.95
0.87
73.19
results for MNIST. and the small perturbation threshold for GTS. Based inter-class distances also expect less & robustness for CIFAR-10 than for FMNIST, but not as seen in Figure 3. In any case, it is safe to say that, when judging the robustness of model by certain threshold, that numher must be set with respect to the distribution the model operates on.
Overall, the strong dependence of robustness curves the data and the chosen norm, emphasizes the necessity of informed and conscious decisions regarding robustness thresholds. We provide an easily accessible reference in the form of Table 2, that should prove useful while judging scales in threat model.
4 DISCUSSION
We have demonstrated that comparisons of robustness of different classifiers using point-wise measures can be heavily biased by the choice of perturbation threshold and distance function of the threat model, and that conclusions about rankings of classifiers with regards to their robustness based on point-wise measures therefore only provide a narrow view of the actual robustness behavior the classifiers. Further, we have demonstrated different ways of using robustness curves to overcome the shortcomings of point-wise measures, and therefore recommend using them as the standard tool for comparing the robustness of classifiers. Finally, we have demonstrated how suitable perturbation thresholds necessarily depend on the data they pertain to.
It is our hope that practitioners and researchers alike will use the methodology proposed in this work, especially when developing and comparing adversarial defenses, and carefully motivate any concrete threat models they might choose, taking into account all available context.
Computing approximate robustness curves for state-of-the-ar classifiers and large data sets is computationally very intensive, due to the need of computing approximate minimal adversarial perturbations with strong adversarial attacks. Developing adversarial attacks which are both strong and fast is ongoing challenge in the field of adversarial robustness.
Limitations:
One way to reduce the computational cost is to approximate the robustness curves by computing set of point-wise measures. However, since robustness curves may intersect at arbitrarily many points, this may give misleading results. It would be interesting to investigate how closely robustness curves need be approximated order estimate the number of intersections, any, and their location, with high certainty.
Another limitation of our work the focus on a small group of distance functions (mainly foo and l norms). Even though it does intuitively make sense that models should at least be robust against these types of perturbations, a more general evaluation able consider more distance functions simultaneously could be advantageous.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jean- Caan-Bapiiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli (2019). "Are Labels Required for Improving Adversarial Robustness?" In: Advances in Neural Information Processing Systems 32, pp. 12214-12223.
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and J Reyes-Ortiz (Jan. 2013). "A Public Domain Dataset for Human Activity Recognition using Smartphones". In: ESANN.
Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Cynthia Liu, Pin-Yu Chen, Shiyu Chang, and Luca Daniel (2020). "Proper Network Interpretability Helps Adversarial Robustness in Classification", en. In: Proceedings of the International Conference on Machine Learning 1.
Wieland Brendel, Jonas Rauber, Matthias Kümmerer, Ivan Ustyuzhaninov, and Matthias Bethge (2019). "Accurate, reliable and fast robustness evaluation". In: Advances in Neural Information Processing Systems 32. pp. 12861-12871.
Nicholas Carlini, Anish Athalye, et al. (2019). On Evaluating Adversarial Robustness. arXiv: 1,902 06705.
Nicholas Carlini and David A. Wagner (2017). "Towards Evaluating the Robustness of Neural Networks". In: 2017 IEEE Symposium Security and Privacy (SP), arXiv: 1608. 04644.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi (2019). Unlabeled Data Improves Adversarial Robustness. arXiv: 1905. 13736.
Jeremy Cohen, Elan Rosenfeld, and 7ico Kolter (2019). "Certified Adversarial Robustness via Randomized Smoothing". In: Proceedings of the 36th International Conference on Machine Learning, ICML. Vol. 97, pp. 1310-1320.
Francesco Croce, Maksym Andriushchenko, and Matthias Hein (2019). "Provable Robustness of ReLU networks via Maximization of Linear Regions". " In: Proceedings of Machine Learning Research. arxiv: 1810. 07481.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Nicolas Flammarion, Mung Chiang. Prateck Mittal, and Matthias Hcin (2020). "RobustBench: standardized adversarial robustness benchmark". In: arXiv preprint PrXiv:2010.09677
Francesco Croce and Matthias Hein (2020). "Provable robustness against all adversarial 4prttturrbttions for
p > 1", In: International Conference on Learning Representations. arXiv: 1905. 11213.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy (2015). "Explaining and Hamessing Adversarial
Examples". In: 3rd International Conference on Learning Representations, arXiv: 1412.6616 6572.
Christina Göpfert, Jan Philip Göpfert, and Barbara Hammer (2020). "Adversarial Robustness Curves". In:
Machine Learning and Knowledge Discovery in Databases. arXiv: 9908 00096.
Jan Philip Göpfert, André Artelt, Heiko Wersing, and Barbara Hammer (2020). "Adversarial attacks hidden in plain sight". : In: Symposium on Intelligent Data Analysis. arXiv: 1902. 09286.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten (2017). Countering Adversarial
Images using Inpul Transformations. arXiv: 1711.00117. 00117.
Matthias Hein and Maksym Andriushchenko (2017). Formal Guarantees the Robustness of a Classifier against Adversarial Manipulation. arXiv: 1705. 08475.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song (2019). "Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty". In: Advances in Neural Information Processing Systems 32, pp. 35663-15674 arXiv: 1901 09960.
Sebastian Houben, Johannes Stallkamp. Jan Salmen, Mare Schlipsing, and Christian Igel (2013). "Detection of Traffic Signs in Real-World Images: The German Traffic Sign Delection Benchmark" In: IJCNN. 1288.
Diederik Kingma and Jimmy Ba (2014), Adam: A Method for Stochastic Optimization. arXiv: 1412. 6980.
Alex Krizhevsky (2009). Learning multiple layers of features from tiny images. Tech. rep.
M. Lecuyer. V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana (2019). "Certified Robustness to Adversarial Examples with Differential Privacy". In: 2019 IEEE Symposium on Security and Privucy (SP), pp. 656-672. arXiv: 1802000000 034
Guang-Ho Lee, Yang Yuan, Shiyu Chang, and Tommi Jaakkola (2019). "Tight Certificates of Adversarial Robustness for Randomly Smoothed Classifiers". In: Advances Neural Information Processing Systems 32, pp. 4910-

---

Under review as a conference paper at ICLR 2021
Bai Li. Changyou Chen, Wenlin Wang, and Lawrence Carin (2019). "Certified Adversarial Robustness with Additive Noise". In: Advances in Neural Information Processing Systems 32. pp. 9464 9474.
Fei-Fei Li, Andrej Karpathy, and Justin Johnson (2016). CS231m: Convolutional Neural Networks for Visual
Recognition, [Online; accessed March 28, 2020]. URL:
http://cs23ln, Sttanford.edu//00 wdw.edu///e. 201d 6/ propjctt....../opo .ttefecttt....... html.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu (2018), "Towards Deep Learning Models Resistant to Adversarial Attacks". In: ICLR. arXiv: 1706. . 06083.
Saeed Mahloujifar, Xiao Zhang, Mohammad Mahmoody, and David Evans (2019). "Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness". In: Advances in Neural Information Processing Systems 32, pp. 5220.
Pratyush Maini, Eric Wong, and Zico Kolter (2020). "Adversarial Robustness Against the Union of Multiple Threat Models". en. In: Proceedings of the International Conference on Machine Learning 1.
Chengzhi Mao, Ziyuan Zhong. Junfeng Yang, Carl Vondrick, and Baishakhi Ray (2019). "Metric Learning Adversarial Robustness". In: Advances in Neural Information Processing Systems 32, pp. 180-4911 arXiv: 1909. 00000.
Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato (2019). "Robustness to Adversarial Perturbations in Learning from Incomplete Data". In: Advances in Neural Information Processing Systems 32, pp. 5541-5551.
Rafacl Pinot, Laurent Mcunier, Alexandre Araujo, Hisashi Kashima, Florian Yger, Cedrie Gouy-Paile Pailler, and Jamal Atif (2019). "Theoretical evidence for adversarial robustness through randomization". In: Advances in Neural Information Processing Systems 32, pp. 1 11838-1 1848.
Chongli Qin al. (2019). " "Adversarial Robustness through Local Linearization", In: Advances in Neural Information Processing Systems 32, pp. 13847-13855
Jonas Rauber, Wicland Brendel, and Matthias Bethge (2017). Foolbox: A Python toolbox to benchmark the robustness of machine learning models. arXiv: 1777.0411. .01311.
Leslie Rice, Eric Wong, and Zico Kolter (2020). "Overfitting in adversarially robust deep learning". en. In: Proceedings of the International Conference Machine Learning 1.
Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana (2020). HYDRA: Pruning Adversarially Robust Neural Networks. arXiv: 2002. 0509 [cs. CCV]
Sahil Singla and Soheil Feizi (2020). "Second- "Second-Order Provable Defenses against Adversarial Attacks", en. In: Proceedings of the International Conference on Machine Learning 1.
Chuanbiao Song, Kun He, Jiadong Lin, Liwei Wang, and John E. Hopcroft (Apr. 2020). "Robust Local Features for Improving the Generalization of Adversarial Training". In.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus (2014), Intriguing properties of neural networks. arXiv: 1 1312. 6199.
Florian Tramer and Dan Boneh (2019). "Adversarial Training and Robustness for Multiple Perturbations". In: Advonces in Neural Information Processing Systems 32, pp. 5866-5876.
Yisen Wang. Difan Zou, Jinfeng Yi, James Bailcy, Xingjun Ma, and Quanquan Gu (Apr. 2020). "Improving Adversarial Robustness Requires Revisiting Misclassified Examples", en. In.
Eric Wong and Zico Kolter (2018), "Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope". In: Proceedings of the 35th International Conference on Machine Learning. arXiv: 1711. 00851.
Eric Wong, Leslie Rice, and J. Zico Kolter (Apr. 2020). "Fast is better than free: Revisiting adversarial training". In.
Eric Wong, Frank R. Schmidt, and J. Zico Kolter (2019). "Wasserstein Adversarial Examples via Projected Sinkhorn Iterations". In: Proceedings of the 36th International Conference on Machine Learning, ICML. Vol. 97. Proceedings of Machine Learning Research, pp. 6808-6817
Dongxian Wu, Shu-tao Xia, and Yisen Wang (2020). Adversarial Weight Perturbation Helps Robust Generalization. arXiv: 2004. 05884.
Han Xiao, Kashif Rasul, and Roland Vollgraf (2017). Fashion MNNIST a Novel Image Dataset for Benchmarking Machine Learning Algorithmss. arXiv: 1708. 07747.
10

---

Under review as a conference paper at ICLR 2021
20
1.0
1.0
10
0.4
R
0.4
0.0
0.0
8oo perturbation threshold
Eg perturbation threshold
Figure 6: Example of data distribution and two linear classifiers such that the l robustness curves intersect, but not the fo robustness curves.
Cihang Xie and Alan Yuille (Apr. 2020). "Intriguing Properties of Adversarial Training at Scale". en. In.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli (2020). "Attacks Which Do Not Kill Training Make Adversarial Learning Stronger". en. In: Proceedings of the International Conference on Machine Learning 1.
A ROBUSTNESS CURVES WITH ARBITRARY INTERSECTIONS
Theorem 1. Let Tı, T2 C R20 be two disjoint finite sets. Then there exists distribution P x (0.1) and two classifiers 01,2 + {0,1) such that (t)()) < (t(() for all t E and (t) (t) (t) for all t T2.
Proof. Without loss of generality, assume that Ti .i,...,,nn. and T2 , (/) with t < & < f441 for E [I....,n], We will construct such that the robustness curves ()), (2(())))))))))))))) intersect at exactly the points (f, + /22 and (1, + 1/22 on the interval (4 (f,,,). Let
= 1, and
to t
0) -P p(a+
4 4+41111
2
P
-d
2
4n +
and
P -d 1/
1
4n +
Let ci (I) = 1 a and (2 =
1 add Both classifiers have perfect accuracy on P, meaning that [2 (0) = 0. The closest point to the decision boundary of is -d - 4 with weight In so R
(()) (4)
Tn+I The second-closest point is -d 4 with weight an+II so R2 ( = in+I
and so on. Mcanwhile, the closest point to the decision boundary of is d 4+4 with weight - so R (
2
2 an+I the second-closest point is d d22+22 with weight 2
int1 so R l2t%) 4n+11
and
Example 1. To see that robustness curve intersections do not transfer between different E, norms, consider the example in Figure 6. The blue and orange linear classifiers both perfectly separate the displayed data. The Lo robustness curves of the classifiers do not intersect. meaning that the robust error of the blue classifier is always better than that the orange classifier. In l2 distance, the robustness curves intersect, so that there is a range of perturbation sizes where the orange classifier has better robust error than the blue classifier.
11

---

Under review as a conference paper at ICLR 2021
B ROBUSTNESS CURVE DEPENDENCE OF SHAPE ON DISTANCE FUNCTION
Theorem 2. Let [() sgn(w= x b) be linear classifier. Then the shape of the robustness curve for f regarding an Ey norm- norm-induced distance does not depend on the choice of p. It holds that
| 1
RY (E) Rl (C.E) f
Pi
(1)
| lez
p:
Lemma 1. Ler x E R with w² # Letp p E [1, o0] and such that
1
+ 1 1. where we
p
take 1 = 0. Then
Iw +
m sgu(w" (x + + b) + sgn( (w x + b]}
(2)
|
and the minimum attained by
w x-b
sgn(w))ees} = argmax, |wal
p=1
(3)
w (sgn(w:)) [wa > y
p E (1,
w
where x: -- = x0 and ej is the j-th unit vector
Proof of Theorem 2. By Hölder's inequality, for any
Σ w < ||01||||l|l.
(4)
For such that sgn(w" (x + 5) b) * sgn(w' (w"
TT x + b) it follows that
m
|w xr +
|111 >
(5)
||/||/
Using the identity Equation (3),
P p-I it is easy to check that for every E [1, oo], with ò as defined in
1. u - x b, that w. wT(x + 5) b 0, and
2. |80|p
Ta ||l|
Item 1 shows that is a feasible point, while Item 2 in combination with Equation (5) shows that ||ō|| minimal.
Using Lemma 1, we ready to prove Theorem 2.
Proof. By definition,
RC (E) -[((((())) s.t. 38: ||0| < 5) v))
(6)
R (e)
can split R,, (e) into the disjoint sets
(I, 9) : f(x)(( + m}
(x, v) s.t. 38: |80|, < <eAg= f(x) + f(x + 8))
(9)
=B, (e)
12

---

Under review as a conference paper at ICLR 2021
Choose 91 92 such that
1 Pi 2:
+ 1 1. By Lemma and using that f(x) sgn(wn (w" b),
1 x + bl
Bpf (e) {1x, v) sgn(w² + b) y
{)
(10)
1| l1
(1x, v) sgn(w" x +bb) - |w- x + bl
| a1 })) || |
(11)
110 l1z
|
Bp
(12)
||
This shows that
RI (5) = P(M) ((Bp((E)
(13)
| |
= =((M) P
Bp
(14)
|
Ripp [Ilwlllss
(15)
||we||92
C EXPERIMENTAL DETAILS
C.I MODEL TRAINING
We use the same model architecture as Croce, Andriushchenko, and Hein (2019) and Wong and Kolter (2018). Unless explicitly stated otherwise, the trained models are taken from Croce, Andriushchenko, and Hein (2019). The exact architecture of the model is: Convolutional layer (number of filters: 16, size: 4x4, stride: 2), ReLu activation function, convolutional layer (number of filters: 32, size: 4x4,
stride: 2), ReLu activation function, fully connected layer (number of units: 100), ReLu activation function, output layer (number of units depends on the number of classes). All models are trained with Adam Optimizer (Kingma and Ba 2014) for 100 epochs, with batch size 128 and default learning rate of 0.001. More information on the training can be found in the experimental details section of the appendix of Croce, Andriushchenko, and Hein (2019). The trained models those made publicly available by Croce, Andriushchenko, and Hein (2019)* and Croce and Hein (2020)
C.2 APPROXIMATED ROBUSTNESS CURVES
We use state-of the--ar adversarial attacks to approximate the true minimal distances of input data- points to the decision boundary of classifier for adversarial robustness curves (see Definition 1). We base our selection of attacks on the recommendations of Carlini, Athalye, et al. (2019). Specifi- cally,
the following attacks: For l2 robustness curves use the lg fz-attack proposed by Carlini and Wagner (2017) and for fxx. robustness curves we use PGD (Madry et al. 2018). For both attacks, the implementations of Foolbox (Version 2.4) (Rauber et al. 2017). 7). For the foo attack, the implementation of Foolbox automatically performs hyperparameter search over different epsilon and uses the smallest resulting adversarial perturbation. For the rest of the hyperparameters, we use the standard values of the Foolbox implementation. For the & attack, we increase the number of binary search steps that are used find the optimal radeoff-constant between distance and confidence from 5 to 10, which we found empirically improve the results. For the rest of the hyperparameters, we again use the standard values the Foolbox implementation.
C.3 COMPUTATIONAL ARCHITECTURE
We executed all programs on an architecture with 2 Intel Xeon(R) CPU ES-2640 v4 @ 2.4 GHz, x Nvidia GeForce GTX 1080 TI 12G and 128 GB RAM.
$ 'The models trained with ST, KW, AT and MMR + AT are avaible at
www.gi t chub.com/ httmmmmaaaapoooooooooaa paovarrovallllo e- robustness weaessssx--xl- ine---iiaer--eiioonn 'The models trained with MMR-UN are available www.gi www.github.com thub.com/ fra31 /mmr-untiiinmm/m versal.
13

---

Under review as a conference paper at ICLR 2021
Original
Adversarial Example
Figure 7: Visualization of four images from CIFAR-10 (top row), together with adversarial examples (bottom row), calculated with PGD (Madry et al. 2018) for model trained with MMR + AT, Threat Model: (e 2/255). The resulting perturbation sizes of the adversarial examples are (from left to right) 17/255, 18/255, 18/255, 18/255. Even for perturbation sizes far greater than popular choices of point-wise measures, adversarial examples can very hard to detect for humans.
1.0
0.6
R
Srandard Training 200kaa2000Hyhha
0.0
50333
0225
X/250
0.00
0.01
0.06
0.08
0.10
0.14
perturbation
e
Figure 8: loo robustness curves for state-of-the-ar robust models with large architecture (WideResNeet-2-100)) The labels indicate the training method (Sehwag2020yydra: (Sehwag et al. 2020), Wuz0Adversarial: (Wu et al. 2020)). The trained models are taken from Croce, An- driushchenko, Sehwag, et al. (2020). The models are trained on the full training set of CIFAR-10. and robustness curves based on a sample of 1000 points from the test set.
D VISUALIZATION OF ADVERSARIAL EXAMPLES
As we pointed out Section 1, adversarial robustness of classifiers trained on CIFAR-10 usually evaluated perturbation threshold (2/255, 4/255, 8/255] for the foo norm. Robustness curves allow us to investigate robustness of classifiers for perturbation thresholds beyond those which are used in the literature, It should not be necessary for the model to invariant under large perturbations, if these perturbations are clearly perceptible or change the "correct" classification of the input. However, the thresholds that models are currently optimized for are small enough that even larger perturbations may not be perceptible. Figure 7 shows four images of CTFAR-10 C (top row), together with adversarial examples (bottom row). With perturbation sizes & E (17/255, 18 /2/555.. the perturbations are more than two times larger than the biggest perturbation threshold used in the literature, and still almost imperceptible for untrained humans.
E ROBUSTNESS CURVES FOR LARGER MODELS
In Section 3, we demonstrate the usefulness of robusiness curves on small convolutional network architecture used by Croce, Andriushchenko, and Hein (2019). The choice small architecture allows to compute robustness curves for large number of different defensive strategies with limited computational resources. Figure 8 shows approximate robustness curves for two state-of-the- art robust models with large network architecture (WideRes.Net-28-100) computed for sample of
14

---

Under review as a conference paper at ICLR 2021
1000 data points from CIFAR-10. Due to the small number of points used, the approximation may be rough, so the following observations should be taken with a grain salt.
1. Both robust models are indeed much more robust than the model obtained by standard training even for perturbation thresholds that are significantly larger than the threshold of 8/255 that the models are optimized for. This observation may help decide whether is worthwhile to stop using conventionally trained model, sacrificing accuracy for robustness.
2. Wu et al. (2020) has slightly worse accuracy than Sehwag et al. (2020) roughly up to perturbation size 1/255. This is a trade-off for better accuracy between perturbation sizes 4/255 and 0.1. From perturbation size 0)1 onward, Sehwag et al. (2020) appears to have slightly better accuracy than Wu et (2020). This observation may help decide which of the two robust models is preferable, based on the robustness requirements of concrete application.
3.
The gap between the performance of Wu et (2020) and Sehwag et al. (2020) is even wider at perturbation size 0.04 than 8/255, but overall, the robustness curves of the robust models are quite similar. This observation may help decide whether it worthwhile to switch from one model to the other, if one of the models is already in use or preferable for other reasons.
15


Reviews:
[{"review_id": "qKYZoWeYJhe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors advocate for the use of Robustness curves, plotting the adversarial accuracy as a function of the size of the neighbourhood region of allowed perturbation. The problem that they identify is that if you only evaluate adversarial accuracy at some numbers of threshold, you might conclude that some models (and the method that was used to train them) are more robust than others while it would be incorrect at other thresholds.\n\nGeneral comment:\nThe paper clearly describes the problem it deals with and reads easily. On the other hand, I am not entirely convinced by the importance of the problem. If you have a particular specification that you care about  (\"Robustness against l_inf attack for eps=0.1\"), you can just verify that. On the other hand, if your goal is \"I want my network to be robust\", then it's not properly defined, so of course it's hard to evaluate. Robustness curves will help there but there is still the problem that you might want to be robust to L_infinity, L_1, L_2, brightness difference, Wasserstein difference, changes of small patches... and then the robustness curves will not help you (unless of course you compute one for each difference). At the same time, they are quite a bit more costly to compute that simple point measures. While I agree that you are going to get more information if you compute a full robustness curve than if you sample it at a bunch of points, I'm not convinced that it is worth the effort.\n\nOne thing that I would recommend the authors is to make clearer the distinction between robustness curves as they described them (based on finding the closest adversarial example) vs. plotting on a robustness curve the pointwise measures and interpolating through them. This would be a much cheaper solution (and is essentially what reporting experimental results for a few chosen eps achieves). For example, the results the authors give in their Table 1. based on point wise measures wessentially achieves what the authors want to show: no defense strictly dominate the others.\n\nSpecific Comments:\n- The toy dataset example presented in Figure 1. is great and provides a great explanation of the problem that the authors identify. The constructive proof in Appendix A. is also quite interesting and really drives the point that the authors want to make.\n\n- The authors argue that robustness curves allow to compare \"global robustness properties and their dependence on a given classifier, distribution and distance function\". In practice, does it really give insights global robustness property? If I look at the L_infinity robustness curves, it does not tell me much about the robustness to L_2 perturbations.\n\n- I don't understand how the robustness curves are generated for the L-infinity case? If PGD is used to find adversarial examples, it's not likely to be the \"closest\" adversarial examples that is going to be found, in all likelihood it's going to be one that matches the epsilon given as input to the PGD attack (due to the projection)? There is nothing that is even encouraging the sample to be close to the input beyond the constraints used for projection.\nEven for the L2 distance, there is still the usual problem that, although the CW attack encourages to find the closest sample, there is no guarantee that it will, and the effectiveness it will have at doing so might depend from model to model. As a result, it's hard to decouple the robustness curve from the attack that it used internally.\n\nMinor Notes & Typos:\n- To improve the look of the paper, it should be possible to include manual linebreaks in the title so that it's not broken in every line. \n- The authors talk in the introduction about \"recently proposed robustness curves\" and cite a paper from 2020 for them, but it seems like those curves were already in use before that. \"On the effectiveness of interval bound propagation for training verifiably robust models\", Gowal et al. had some in 2018.; \"Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope\" had some (transposed) in 2017.\n- At the end of the introduction, the author say: \"It is our belief that the continued use of single perturbation thresholds in the adversarial robustness literature is due to a lack of awareness of the shortcomings of these measures\".  This seems overtly harsh. You could make the same point about training algorithms and say that authors only reporting on only a few datasets due it just out of lack of awareness of the fact that the relative performance of different algorithms will vary depending on the dataset. Given that computing robustness curves needs computing the closest adversary to a point, this is much more expensive so maybe computational cost might be the differentiating factor rather than \"lack of awareness\"?", "extra_scores": {}, "extra_text": {}}, {"review_id": "GUwAw2WV_rL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper surveys various adversarial defense methods on their performance when the perturbation distortion epsilon is increased. The author argues that robustness for a specific epsilon may not be enough and suggests robustness curves as an alternative. I think in general this paper provides some interesting empirical studies. My detailed comments are as follows.\n1. The overfitting of specific epsilon value is expected but interesting to see and I think this is one of the main reasons why a robustness curve is necessary. However, in this adversarial robustness community, the status quo is that researchers compare with each other on some specific datasets with some specific epsilon, for example, 0.3 for MNIST, 8/255 for CIFAR. I think one reason for choosing these values is that studying robustness under perturbation with larger distortion is kind of unnecessary because then the noise added is no longer imperceptible, which is at odds with adversarial examples' definition. I think the authors may need to provide more discussion on why studying robustness with epsilon>0.6 for MNIST is necessary, since we may misclassify those images as human. \n2. The authors suggest a robustness curve as an evaluation metric. However, I haven't seen any works on improving the robustness for all epsilon values globally. One possibility is that there is a trade-off between small epsilon performance and large epsilon performance, similar to the trade-off between robustness and accuracy (epsilon=0 performance). My suggestion is that the authors may define an \"area under the curve\" value just like in ROC curves for better comparison.\n3. (Minor) Please refrain from only using color to distinguish curves in figures as it may not be friendly to readers with color blindness. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9yBqshCrrPB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper showed that point-wise measures fail to capture important properties that are essential to compare the robustness of different classifiers. The authors introduced the recently proposed robustness curves to provide a global perspective including the scale as a way to distinguish small and large perturbations. \n\nPros:\n(1) How to compare the robustness is a very important question for current machine learning models. The author introduced a better criterion for this important question. \n(2) The paper is well written and easy to read.  The experiments and its discussion are strong proofs to support that the robustness curve should be the better criterion.\n(3) The authors released code to reproduce all the experiments for the current popular frameworks. It will be very helpful for the researchers to the advantages of this curve over point-wise measures. \n\nCons:\nOverall, this paper is impressive. \nThe only concern the reviewer has is the contribution compared to the previous work who proposed the robustness curve (C. Gopfert et al. 2020).  it seems this paper's contribution is highly based on  the proposal of robustness curve, and providing more explanations and discussions. (But this will not affect the importance of this paper)\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3P_HB7Etsy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a theoretical scenario where point-wise measure of adversarial robustness falls short in comparing model robustness, then conduct experiments to show that robustness curve is a more meaningful evaluation metric from a global perspective.\n\n\nPros:\n\n\n+ The motivation is well explained. I mainly agree with the authors on the argument that point-wise measurement of robustness may be insufficient in explaining model robustness. Computing and visualizing robustness curves seems to be more meaningful and rigorous from a security perspective. \n\n+ Relating the choice of the perturbation strength to the underlying property of the data distribution is useful. The inter-class distances demonstrated in Table could potentially be used as a reference on determining the right scale of perturbation strength.\n\n\nCons:\n\n\n- The robustness results presented in Table 1 seems far below the state-of-the-art robustness. For instance, in the last row (\\epsilon=8/255), the robust test error of AT is 0.92, which is much higher than the reported statistics in (Madry et al., 2018). The author uses a very small 4-layer convolutional neural network for CIFAR-10 experiments, whereas the state-of-the-art robustness results are achieved using a much larger network, such as a ResNet architecture or a WideResNet architecture (refer to [1] for the current best robustness results on CIFAR-10). Thus, I recommend authors to rerun these experiments using a larger network.\n\n- Similar architecture is used for the robustness curves in Figures 3 and 4. This suboptimal choice of network architecture makes the argument 'it contains multiple intersections in the robustness curve' unconvincing. \n\n\nOther Questions or Comments:\n\n1. Most of the existing defenses against adversarial examples are typically trained using a specifically-chosen perturbation strength. If adopting robustness curves (or global robustness) as the evaluation criteria instead of point-wise robustness, how will this affect the existing adversarial training procedure?\n\n2. What does the distance statistics presented in Table 2 suggest for the typical choice of perturbation strength used in existing literature? \n\n3. The global robustness considered in this paper is robustness for varying perturbation strength. Is there a way to define the perturbation strength for different input locations based on your computed inter-class statistics?\n\n4. The bibliography style of the reference is not standard. Check if you are using the correct file.\n\n\n[1] Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks, Francesco Croce and Matthias Hein, ICML 2020\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study "robustness curves" which are plots of the robust error versus the radius used in the corresponding l_p-ball threat model.

Pro: I completely agree with the authors that the current evaluation purely based on evaluation for a single radius is insufficient
and one should report the complete curve. 

Con: The authors are overclaiming that they have come up with robustness curves. Very early papers e.g. even in the adversarial
training paper of Madry there are plots of robust accuracy versus chosen threshold. Moreover, I agree with one of the reviewers that using PGD for the purpose of a robustness curve is inaccurate and in particular inefficient as several attacks for different radii have to be done. There have been several attacks developed which aim to find the adversarial sample with minimum norm and thus compute the robustness curve in one run.

The additional insights e.g. intersection of robustness curves are partially to be expected and I don't find them sufficient to move the paper over the bar for ICLR.  As these insights are additionally  only shown for relatively small models which seem far away from the state of the art, it is unclear if they generalize. However, I encourage to follow some of the reviewer's suggestions to improve the paper.
Decision: Reject