Perceptual Adversarial Robustness: Defense Against Unseen Threat Models
Authors: Cassidy Laidlaw
Sahil Singla
Soheil Feizi
Venue: ICLR 2021 Poster (2021)

Abstract: A key challenge in adversarial robustness is the lack of a precise mathematical characterization of human perception, used in the definition of adversarial attacks that are imperceptible to human eyes. Most current attacks and defenses try to get around this issue by considering restrictive adversarial threat models such as those bounded by $L_2$ or $L_\infty$ distance, spatial perturbations, etc. However, models that are robust against any of these restrictive threat models are still fragile against other threat models, i.e. they have poor generalization to unforeseen attacks. Moreover, even if a model is robust against the union of several restrictive threat models, it is still susceptible to other imperceptible adversarial examples that are not contained in any of the constituent threat models. To resolve these issues, we propose adversarial training against the set of all imperceptible adversarial examples. Since this set is intractable to compute without a human in the loop, we approximate it using deep neural networks. We call this threat model the neural perceptual threat model (NPTM); it includes adversarial examples with a bounded neural perceptual distance (a neural network-based approximation of the true perceptual distance) to natural images. Through an extensive perceptual study, we show that the neural perceptual distance correlates well with human judgements of perceptibility of adversarial examples, validating our threat model.

Under the NPTM, we develop novel perceptual adversarial attacks and defenses. Because the NPTM is very broad, we find that Perceptual Adversarial Training (PAT) against a perceptual attack gives robustness against many other types of adversarial attacks. We test PAT on CIFAR-10 and ImageNet-100 against five diverse adversarial attacks: $L_2$, $L_\infty$, spatial, recoloring, and JPEG. We find that PAT achieves state-of-the-art robustness against the union of these five attacks—more than doubling the accuracy over the next best model—without training against any of them. That is, PAT generalizes well to unforeseen perturbation types. This is vital in sensitive applications where a particular threat model cannot be assumed, and to the best of our knowledge, PAT is the first adversarial training defense with this property.

Code and data are available at https://github.com/cassidylaidlaw/perceptual-advex

Published as a conference paper at ICLR 2021
PERCEPTUAL ADVERSARIAL ROBUSTNESS:
DEFENSE AGAINST UNSEEN THREAT MODELS
Cassidy Laidlaw University of Maryland claid all awe umd. edu
Sahil Singla
Soheil Feizi
University of Maryland ss i ng aêcs umd edu
University of Maryland s afe z @cs. umd. edu
ABSTRACT
A key challenge in adversarial robustness is the lack of a precise mathematical characterization of human perception, used in the definition of adversarial attacks that are imperceptible to human eyes. Most current attacks and defenses try to avoid this issue by considering restrictive adversarial threat models such as those bounded by L2 or Ti distance, spatial perturbations, etc. However, models that are robust against any of these restrictive threat models are still fragile against other threat models, i.e. they have poor generalization to unforeseen attacks. Moreover, even if a model robust against the union of several restrictive threat models, it is still susceptible to other imperceptible adversarial examples that not contained in any of the constituent threat models. To resolve these issues, we propose adversarial training against the of all imperceptible adversarial examples. Since this set is intractable to compute without human in the loop, we approximate it using deep neural networks. We call this threat model the neural perceptual threat model (NPTM); it includes adversarial examples with a bounded neural perceptual distance (a neural network-based approximation of the true perceptual distance) to natural images. Through an extensive perceptual study, we show that the neural perceptual distance correlates well with human judgements of perceptibility of adversarial examples, validating our threat model. Under the NPTM, we develop novel perceptual adversarial attacks and defenses. Because the NPTM is very broad, find that Perceptual Adversarial Training (PAT) against perceptual attack gives robustness against many other types of adversarial attacks. We test PAT CIFAR-IO and NeageNet-100 against five diverse adversarial attacks: T2, Tox. spatial, recoloring, and JPEG. We find that PAT achieves state-of-the-an robustness against the union of these five attacks- more than doubling the accuracy over the next best model-without training against any of them. That is, PAT generalizes well unforeseen perturbation types. This is vital in sensitive applications where a particular threat model cannot be assumed, and to the best of our knowledge, PAT is the first adversarial training defense with this property.
INTRODUCTION
Many modern machine learning algorithms are susceptible to adversarial examples: carefully crafted inputs designed to fool models into giving incorrect outputs (Biggio et al. 2013 Szegedy et al. 2014 Kurakin al. 2016a Xie al. 2017). Much research has focused on increasing classifiers' robustness against adversarial attacks (Goodfellow et al. 2013) Madry et al. 2018 Zhang et al.) 2019a). However, existing adversarial defenses for image classifies generally consider simple threat models. An adversarial threat model defines a set of perturbations that may be made to an image in order to produce an adversarial example. Common threat models include 7.2 and T. threat models, which constrain adversarial examples to be close to the original image in L2 or Loo distances. Some work has proposed additional threat models which allow spatial perturbations (Engstrom et al. 20177] Wong et al. 2019 Xiao al.) 2018), recoloring (Hosseini and Poovendran 2018 Laidlaw and Feizi 2019 Bhattad et al. 2019), and other modifications (Song et al. 2018 Zeng et al. 2019 of an image.
There are multiple issues with these unrealistically constrained adversarial threat models. First, hardening against one threat model assumes that an adversary will only attempt attacks within that threat model. Although classifier may be trained to be robust against La attacks, for instance,

---

Published as a conference paper at ICLR 2021
L2
StAdv
Unrestricted threat model
0.0%
0.5%
Patch
9.7%
24.9%
1.3%
Tpp
True perceptual threat model
Spatial
26.5%
Neural perceptual
threat model
Neural Perceptual Attack (ours)
Figure 2: Area-proportional Venn diagram val- idating our threat model from Figure 1. Each ellipse indicates set of vulnerable ImageNet- 100 examples to of three attacks: T.2, StAdv spatial (Xiao et al. 2018), and our neural per- ceptual attack (LPA, Section 4. Percentages indicate the proportion of test examples success- fully attacked. Remarkably, the NPTM encom- passes both other types of attacks and includes additional examples not vulnerable either.
Figure 1: Relationships between various adver- sarial threat models. Lp and spatial adversarial attacks are nearly contained within the percep- tual threat model, while patch attacks may be perceptible and thus are not contained. In this pa- per, we propose a neural perceptual threat model (NPTM) that is based on an approximation the true perceptual distance using neural networks.
an attacker could easily generate a spatial attack to fool the classifier. One possible solution is to train against multiple threat models simultancously (Jordan al. 2019 Laidlaw and Feizi 2019 Maini et al., 2019; Tramer and Boneh, 2019). However, this generally results in a lower robustness against any of the threat models when compared to hardening against that threat model alone. Furthermore, not all possible threat models may be known at training time, and adversarial defenses do not usually generalize well to unforeseen threat models (Kang al. 2019).
The ideal solution to these drawbacks would be a defense that robust against wide, unconstrained threat model. We differentiate between two such threat models. The unrestricted adversarial threat model (Brown et al. 2018) encompasses any adversarial example that is labeled as one class by classifier but different class by humans. On the other hand, we define the perceptual adversarial threat model as including all perturbations of natural images that are imperceptible to a human. Most existing narrow threat models such as L2. Los. etc. are near subsets of the perceptual threat model (Figure) T]. Some other threat models, such as adversarial patch attacks (Brown et al. (2088) may perceptibly alter an image without changing its true class and as such are contained in the unrestricted adversarial threat model. In this work, we focus the perceptual threat model.
The perceptual threat model can be formalized given the true perceptual distance d' (x1, x2) between images xı and X2, defined as how different two images appear to humans. For some threshold e*, which we call the perceptibility threshold, images x and x' are indistinguishable from one another as long as d' (x, x') S c". Note that in general c* may depend on the specific input. Then, the perceptual threat model for a natural input x includes all adversarial examples X which cause misclassification but are imperceptibly different from x, i.e. d' (x, x) Se"
The true perceptual distance d' (::)), however, cannot be easily computed or optimized against. To solve this issue, we propose to use a neural perceptual distance, an approximation of the true perceptual distance between images using neural networks. Fortunately, there have been many surrogate perceptual distances proposed in the computer vision literature such SSIM (Wang et al. 2004). Recently, Zhang et al. (2018) discovered that comparing the internal activations of convolutional neural network when two different images are passed through provides measure, Learned Perceptual Image Patch Similarity (LPIPS), that correlates well with human perception. We propose to use the LPIPS distance d(...) in place of the true perceptual distance d² (::.) to formalize the neural perceptual threat model (NPTM).
We present adversarial attacks and defenses for the proposed NPTM. Generating adversarial examples bounded by the neural perceptual distance is difficult compared to generating L, adversarial examples because of the complexity and con-convessess of the constraint. However, we develop two attacks for the NPTM, Perceptual Projected Gradient Descent (PPGD) and Lagrangian Perceptual Attack (LPA) (see Section 4 for details). We find that LPA is by far the strongest adversarial attack at a given level of perceptibility (see Figure 4,, reducing the most robust classifier studied to only 2.4%
2

---

Published as a conference paper at ICLR 2021
accuracy ImageNet- 100 (a subset of ImageNet) while remaining imperceptible. LPA also finds adversarial examples outside of any of the other threat models studied (see Fi Thus, even if a model is robust to many narrow threat models (Lp, spatial, etc.), LPA can still cause serious errors.
In addition to these attacks, which are suitable for evaluation of a classifier against the NPTM, we also develop Fast-LPA, more efficient version of LPA that we use in Perceptual Adversarial Training (PAT). Remarkably, using PAT to train a neural network classifier produces a single model with high robustness against a variety of imperceptible perturbations, including Lox. L2, spatial, recoloring, and JPEG attacks, on CIFAR- 10 and ImageNet- 100 (Tables [nd For example, PAT on ImageNet- 100 gives 32.5% accuracy against the union of these five attacks, whereas L and Le adversarial training give 0.5% and 12.3% accuracy, respectively (Table D.. PAT achieves more than double the accuracy against this union of five threat models despite not explicitly training against any of them. Thus, generalizes well to unseen threat models.
Does the LPIPS distance accurately reflect human perception when it is used to evaluate adversarial examples? We performed a study on Amazon Mechanical Turk (AMT) to determine how perceptible 7 different types of adversarial perturbations such as Los, L2. spatial, and recoloring attacks are at multiple threat-specific bounds. We find that LPIPS correlates well with human judgements across all the different adversarial perturbation types we examine. This indicates that the NPTM closely matches the true perceptual threat model and reinforces the utility of our perceptual attacks to measure adversarial robustness against an expansive threat model. Furthermore, this study allows calibration of a variety of attack bounds to a single percepti- bility metric. We have released our dataset of adversarial examples along with the annotations made by participants for further study"
ImageNet- 100 ac- curacies against the union of five threat models (Too. Lg, JPEG, StAdv, and Re- ColorAdv) for different ad- versarially trained (AT) clas- sifiers and single model trained using PAT.
Table 1:
Training
Accuracy
PGD Lo
0.5%
PGD L PAT (ours)
12.3%
32.5%
RELATED WORK
Adversarial robustness Adversarial robustness has been studied extensively for L2 or Loo threat
models (Goodfellow et al. 2015 Carlini and Wagner 2017:
Madry et al. (0088) and non-L, threat
models such as spatial perturbations (Engstrom et al. [2017] Xiao et al. 2018 Wong et al. 2019).
recoloring of an image (Hosseini and Poovendran] [2188] Laidlaw and Feizi| [019]] Bhattad et al.. 2019), and perturbations in the frequency domain (Kang et 2019). The most popular known adversarial defense for these threat models adversarial training Kurakin et al. (2016b); Madry et al.)
(2018): (Zhng et al. (2019a) where neural network is trained to minimize the worst-case loss in region around the input. Recent evaluation methodologies such as Unforeseen Attack Robustness (UAR) (Kang et al. 2019) and the Unrestricted Adversarial Examples challenge (Brown al. 2018) have raised the problem of finding an adversarial defense which gives good robustness under more general threat models. (hhrrif al. (2018) conduct perceptual study showing that Lp threat models are a poor approximation of the perceptual threat model. Dunn et al. (2020) and Xu et al. (2020) have developed adversarial attacks that manipulate higher-level, semantic features. Jin and Rinard (020) train with a manifold regularization term, which gives some robustness to unseen perturbation types. Stutz et al. (2020) also propose a method which gives robustness against unseen perturbation types, but requires rejecting (abstaining on) some inputs.
Perceptual similarity Two basic similarity measures for images are the L2 distance and the Peak Signal-to-Notee Noise Ratio (PSNR). However, these similarity measures disagree with human vision on perturbations such as blurring and spatial transfornations, which has motivated others including SSIM (Wang et al. 2004), MS-SSIM (Wang et al. (2003), CW-SSIM (Sampat et al.) 2009). HDR-VDD - (anniiuk et al., 201T) and LPIPS (Zhang et al.] 2018). MAD competition (Wang and Simoncelli 2008) uses constrained optimization technique related to our attacks to evaluate perceptual measures.
Perceptual adversarial robustness Although LPIPS was previously proposed, it has mostly been used for development and evaluation of generative models (Huang et al. 2018 Karras et al.) 2019). Jordan et al. (2019) first explored quantifying adversarial distortions with LPIPS distance. However, to the best of our knowledge, we are the first to apply more accurate perceptual distance to the
"oode and data can be downloaded at http:://hhh......o..oo////aa/aaa
3

---

Published as a conference paper at ICLR 2021
Original
Self-bd. LPA
External-bd. LPA
Original
Self-bd. LPA
External-bd. LPA
Figure 3: Adversarial examples generated using self- self-bounded and extermally-bounti LPA perceptual adversarial attack (Section 4 with large bound. Original images are shown in the left column and magnified differences from the original are shown to the right of the examples. See also Figure 7
problem of improving adversarial robustness. As we show, adversarial defenses based on Le or Lo attacks are unable to generalize to more diverse threat model. Our method, PAT, is the first adversarial training method we know of that can generalize to unforeseen threat models without rejecting inputs.
NEURAL PERCEPTUAL THREAT MODEL (NPTM)
Since the true perceptual distance between images cannot be efficiently computed, we use approxi- mations of it based on neural networks, i.c. neural perceptual distances. In this paper, we focus on the LPIPS distance (Zhang al. 2018) while we note that other neural perceptual distances can also be used in our attacks and defenses.
Let g X' + y be convolutional image classifier network defined on images X. Let g have layers, and let the internal activations (outputs) of the I-th layer of g(x) for an input x be denoted as gi (x). /Zhang al. 2018) have found that normalizing and then comparing the internal activations of convolutional neural networks correlates well with human similarity judgements. Thus, the first step in calculating the LPIPS distance using the network g(.) )) is to normalize the internal activations across the channel dimension such that the L2 norm over channels at each pixel is one. Let 9r (x) denote these channel-normalize activations at the 1-th layer of the network. Next, the activations
are normalized again by layer size and flattened into single vector d(x) A Viezhy
(x(
where wt and hy are the width and height of the activations in layer 1, respectively. The function : X + A thus maps the inputs x E X' of the classifier 9(.) to the resulting normalized, flattened internal activations @(x) A, where A C I refers to the space of all possible resulting activations. The LPIPS distance d(x1, X2) between images X1 and x2 is then defined as:
d(x1, x2)
[lx(x)) o(xx)ll2-
In the original LPIPS implementation, /Zhag et al. (2018) learn weights to apply to the normalized activations based on a dataset of human perceptual judgements. However, they find that LPIPS is
good surrogate for human vision even without the additional learned weights; this is the version
use since it avoids the need to collect such a dataset.
Now let f X } be classifier which maps inputs X to labels f(x) E V f()) could be the same as g(.), or it could be a different network; we experiment with both. For given natural input x with the true label y, neural perceptual adversarial example with perceptibility bound is input
E X' such that x must be perceptually similar to x but cause ff to misclassify:
f(x) +
and
d(x, x) = [1x(()) (x) S
4 PERCEPTUAL ADVERSARIAL ATTACKS
We propose attack methods which attempt to find adversarial example with small perceptual distortion. Developing adversarial attacks that utilize the proposed neural perceptual threat model is more difficult than that of standard Lp threat models, because the LPIPS distance constraint is more complex than Lp constraints. In general, we find an adversarial example that satisfies (2) by maximizing loss function L within the LPIPS bound. The loss function is similar to the
4

---

Published as a conference paper at ICLR 2021
margin loss from Carlini and Wagner (2017). defined as
[(f(x),) _ max (z:(x) - 2 (x));
*Øy
where 2, x) is the i-th logit output of the classifier f()). This gives the constrained optimization
max
())())
subject to d(x, x) ||o((x ox)]l12 S.
(3)
Note that in this attack problem, the classifier network f(-) and the LPIPS network g()) are fixed. These two networks could be identical, in which case the same network that is being attacked is used to calculate the LPIPS distance that bounds the attack; we call this a self-bounded attack. If a different network is used to calculate the LPIPS bound, we call it
eeternally-bounder attack.
Based on this formulation, we propose two perceptual attack methods, Perceptual Projected Gradient Descent (PPGD) and Lagrangian Perceptual Attack (LPA). See Figures 3 and 7 for sample results.
Perceptual Projected Gradient Descent (PPGD) The first of our two attacks is analogous to the PGD (Madry et al., 2018) attacks used for Lp threat models. In general, these attacks consist of iteratively performing two steps on the current adversarial example candidate: (a) taking a step of certain size under the given distance that maximizes first-order approximation of the misclassification loss, and (b) projecting back onto the feasible set of the threat model.
Identifying the ideal first-order step is easy in La and Los threat models; it is the gradient of the loss function and the sign of the gradient, respectively. However, computing this step is not straightforward with the LPIPS distance, because the distance metrie itself is defined by neural network. Following
3 we desire to find step 8 to maximize (xf(x 5), y) such that d(x 6, x) =
|1ő(x + 6) <xx) 2| 1J, where 7) is the step size. Let f(x) [( (x), v) for an input E X. Let be the Jacobian of p(-) x and V be the gradient of f(-) at x. Then, we can approximate using a first-order Taylor's approximation of d and f as follows:
subject to || n. We show that this constrained optimization can be solved in a closed form:
max f(x) + TT
(4)
Lemma 1. Let J denote the pseudoinverse of J. Then the solution to (4) is given by
8 = (JTJ)-
See Appendix(A.1] [A.] for the proof. This solution is still difficult efficiently compute, since calculating J and inverting J J are computationally expensive. Thus, we approximately solve for 5* using the conjugate gradient method; see Appendix A. for details.
Perceptual PGD consists of repeatedly finding first-order optimal 8* to add to the current adversarial example x for a number of steps. Following each step. the current adversarial example is outside the LPIPS bound, we project X back onto the threat model such that d(X, x) S e. The exact projection again difficult due to the non-convexity of the feasible set. Thus, we solve approximately with a technique based on Newton's 's method; see Algorithm 4 in the appendix.
Lagrangian Perceptual Attack (LPA) The second of our two attacks uses Lagrangian relaxation of the attack problem 33 similar to that used by Carlini and Wagner (2017) for constructing L2 and Los adversarial examples. We call this attack the Lagrangian Perceptual Attack (LPA). derive the attack, the following Lagrangian relaxation of (3:
max X
[(f(x),) v) A Amax 00 ([)((( ((x) (2x))
(5)
The perceptual constraint cost, multiplied by À in (5). is designed to be 0 as long as the adversarial example within the allowed perceptual distance; i.e. d(xx, x) S c; once d(xx x) > c, however, increases linearly by the LPIPS distance from the original input x. Similar to Lp attacks Carlini and Wagner (2017), we adaptively change À to find an adversarial example within the allowed perceptual distance; see Appendix A.2 for details.
5 PERCEPTUAL ADVERSARIAL TRAINING (PAT)
The developed perceptual attacks can be used to harden a classifier against a variety of adversarial attacks. The intuition, which we verify in Section 7 is that if model is robust against neural
5

---

Published as a conference paper at ICLR 2021
perceptual attacks, it can demonstrate enhanced robustness against other types of unforeseen adversarial attacks. Inspired by adversarial training used to robustify models against Lp attacks, we propose method called Perceptual Adversarial Training (PAT).
Suppose we wish to train a classifier f()) over a distribution of inputs and labels (x, y) ~ D such that it is robust to the perceptual threat model with bound Let L denote the cross entropy (negative log likelihood) loss and suppose the classifier f(-) is parameterized by 8,. Then, PAT consists of optimizing f()) in manner analogous Lp adversarial training (Madry et al. 2018):
min
E max L (fe d(xxx){+
(6)
(x:))=
The training formulation attempts to minimize the worst-case loss within a neighborhood of each training point x. In PAT, the neighborhood is bounded by the LPIPS distance. Recall that the LPIPS distance is itself defined based on a particular neural network classifier. We refer to the normalized, flattened activations of the network used to define LPIPS as Φb.)) and 0g to refer its parameters. We explore two variants of PAT differentiated by the choice of the network used to define 6(-). In externally- bounded PAT, separate, pretrained network is used to calculate ((-). the LPIPS distance d(-,-). In self- self-bounded PAT, the same network which is being trained for classification is used to calculate the LPIPS distance, i.e. 0g S 0y. Note that in self-bounded PAT the definition of the LPIPS distance changes during the training as the classifier is optimized.
The inner maximization in (6) intractable to compute exactly. However, we can use the perceptual attacks developed in Section 4 to approximately solve it. Since the inner maximization must be solved repeatedly during the training process, we use an inexpensive variant of the I.AA attack called Fast-I.PA. In contrast to LPA, Fast-LPA does not search over values of X. It also does not include projection step at the end of the attack, which means it may sometimes produce adversarial examples outside the training bound. While this makes it unusable for evaluation, it is fine for training. Using Fast-LPA, PAT is nearly as fast as adversarial training; see Appendix A.3 and Algorithm for details.
PERCEPTUAL EVALUATION
We conduct a thorough perceptual evaluation of our NPTM and attacks to ensure that the resulting adversarial examples are imperceptible. We also compare the perceptibility of perceptual adversarial attacks five narrow threat models: Lox: and L2 attacks. JPEG attacks (Kang et al. 20199, spatially transformed adversarial examples (StAdv) (Xiao et al.) 2018). and functional adversarial attacks (ReColorAdy (Laidlaw and Feizi, 2019). The comparison allows us to determine if the LPIIS distance is a good surrogate for human comparisons of similarity. It also allows to set bounds across threat models with approximately the same level of perceptibility.
To determine how perceptible a particular threat model is at particular bound (e.g. Loe attacks at c = 8/ /255), we perform an experiment based on just noticeable differences (JND). We show pairs of images participants on Amazon Mechanical Turk (AMT), an online crowdsourcing platform. In each pair, one image is a natural image from ImageNet-100 100 and one image is an adversarial perturbation of the natural image, generated using the particular attack against classifier hardened to that attack. One of the images, chosen randomly, is shown for one second, followed by a blank screen for 250ms, followed by the second image for one second. Then, participants must choose whether they believe the images are the same or different. This procedure is identical to that used by Zhang et al. (2018) to originally validate the LPIPS distance. We report the proportion of pairs for which participants report the images are "different" as the perceptibility of the attack. In addition to adversarial example pairs, we also include sentinel image pairs which are exactly the same; only 4.1 % of these were annotated as "different."
We collect about 1,000 annotations of image pairs for each of 3 bounds for all five threat models, plus our PPGD and LPA attacks (14k annotations total for 2.8k image pairs). The three bounds for each attack are labeled as small, medium, and large; bounds with the same label have similar perceptibility across threat models (see Appendix D'addixxDTaaie Table The dataset of image pairs and associated annotations is available for use by the community.
To determine if the LPIPS threat model is a good surrogate for the perceptual threat model, use various classifiers to calculate the LPIPS distance d(..-) between the pairs of images used in the perceptual study. For each classifier, determine the correlation between the mean LPIIS distance

---

Published as a conference paper at ICLR 2021
80%
80%
Loo
Perceptibilltyy correlation
1.0
La JPEG
Perceptibility
60%
60%
0.8
40%
40%
StAdv
ReColorAdv
0.6
20%
20%
PPGD (ours)
L
SSIM
AlexNet
(ini))
ResNet-50
(init)
LPA (ours)
0%
0%
0.5
1.0
1.5
0%
50%
100%
(a) Mean LPIPS distance
(b) Strength (succ. against AT)
(c) Distance models
Figure 4: Results of the perceptual study described in Section 6 across five narrow threat models and our two perceptual attacks, each with three bounds. (a) The perceptibility of adversarial examples correlates well with the LPIPS distance (based on AlexNet) from the natural example. (b) The Lagrangian Perceptual Attack (LPA) and Perceptual PGD (PPGD) are strongest at given percepti- bility. Strength is the attack success rate against an adversarially trained classifier.
(c) Correlation between the perceptibility of attacks and various distance measures: L2, SSIM (Wang et al. 2004)) and LPIPS (Zhang et al. 2018) calculated using various architectures, trained and at initialization.
assigns to image pairs from each attack and the perceptibility of that attack (Figure 4p). We find that AlexNet (Krizhevsky al. 2012). trained normally on ImageNet (Russakovsky al. 2015). correlates best with human perception of these adversarial examples (r = 0.94); this agrees with Zhang al. (2018) who also find that AlexNet-based LPIPS correlates best with human perception (Figure 4 A normally trained ResNet-50 correlates similarly, but not quite as well. Because AetxNet the best proxy for human judgements of perceptual distance, we use it for all externally-bout -bounded evaluation attacks. Note that even with an untrained network at initialization, the LPIPS distance correlates with human perception better than the L2 distance. This means that even during the first few epochs of self-bounded PAT, the training adversarial examples are perceptually-aligne
We use the results of the perceptual study to investigate which attacks are strongest at a particular level of perceptibility. We evaluate each attack on a classifier hardened against that attack via adversarial training, and plot the resulting success rate against the proportion of correct annotations from the perceptual study. Out of the narrow threat models, perceptibility. However, our proposed PPGD and LPA attacks reduce a PAT-trained classifier to even lower accuracies (8.2% for PPGD and 0% for LPA), making it the strongest attack studied.
find that L2 attacks are the strongest for their
EXPERIMENTS
We compare Perceptual Adversarial Training (PAT) to adversarial training against narrow threat models (Lg, spatial, etc.) CIFAR-10 Krizhevsky and Hinton| (2099) and ImageNet- (the subset of ImageNet (Russakovsky et al. 2015) containing every tenth class by WordNet ID order). We find that PAT results in classifiers with robustness against a broad range of narrow threat models. We also show that our perceptual attacks, PPGD and LPA, are strong against adversarial training with narrow threat models. We evaluate with externally- bounnded PPGD and LPA (Section 4). using AlexNet to determine the LPIPS bound because it correlates best with human judgements (Figure Fc). For L2 and Lox robustness evaluation we use AutoAttack (Croce and Hein 2020). which combines four strong attacks, including two PGD variants and a black box attack, to give reliable evaluation.
Evaluation metrics For both datasets, we evaluate classifiers' robustness to a range of threat models using two summary metrics. First, compute the union accuracy against all narrow threat models (Lx. L2, StAdv, ReColorAdy, and JPEG for ImageNet-100y, this is the proportion of inputs for which classifier is robust against all these attacks. Second, we compute the unseen mean accuracy, which is the mean of the accuracies against all the threat models not trained against; this measures how well robustness generalizes to other threat models.
We test ResNet-50s trained on the CIFAR-10 dataset with PAT and adversarial training (AT) against six attacks (see Table 2 Teo and L2 AutoAttack, StAdy (Xiao et al.) 2018), ReCol-
CIFAR-10
7

---

Published as a conference paper at ICLR 2021
Table 2: Accuracies against various attacks for models trained with adversarial training and Perceptual Adversarial Training (PAT) variants CIFAR-10 10. Attack bounds are 8/255 for Les, 1 for L2. 0.5 for PPGD/LPA (bounded with AlexNet), and the original bounds for StAdw/ReColorAdk, Manifold regularization is from Jin and Rinard (2020). See text for explanation of all terms.
Union
Unseen
Narrow threat models
NPTM
Training
mean
Clean
Loo
L2
StAdv
ReColor
PPGD
LPA
Normal
0.0
0.1
94.8
0.0
0.0
0.0
0.4
0.0
0.0
AT Loo
1.0
19.6
86.8
49.0
19.2
4.8
54.5
1.6
0.0
TRADES Loo
23.3
84.9
52.5
23.3
60.6
2.0
0.0
AT L2
4.0
25.3
85.0
39.5
47.8
7.8
53.5
6.3
0.3
AT StAdy
0.0
1.4
86.2
0.1
0.2
53.9
5.1
0.0
0.0
AT ReColorAdy
0.0
3.1
93.4
8.5
3.9
0.0
65.0
0.1
0.0
AT all (random)
0.7
85.2
22.0
23.4
1.2
46.9
1.8
0.1
AT all (average)
14.7
86.8
39.9
39.6
20.3
64.8
10.6
1.1
AT all (maximum)
21.4
84.0
25.7
30.5
40.0
63.8
8.6
1.1
Manifold reg.
21.2
36.2
72.1
36.8
43.4
28.4
63.1
8.7
9.1
PAT-self PAT-AlexNet
21.9
45.6
82.4
30.2
34.9
46.4
71.0
13.1
2.1
27.8
48.5
71.6
28.7
33.3
64.5
67.5
26.6
9.8
Table 3: Comparison of adversarial training against narrow threat models and Perceptual Adversarial Training (PAT) on ImageNet-100, Accuracies are shown against seven attacks with the medium bounds from Table PAT greatly improves accuracy (33% vs 12%) against the union of the narrow threat models despite not training against any of them. See text for explanation of all terms.
Union
Unseen mean
Narrow threat models
NPTM
Training
Clean
Lx
L2
JPEG
StAdv
ReColor
PPGD LPA
Normal
0.0
0.1
89.1
0.0
0.0
0.0
0.0
2.4
0.0
0.0
Loc
0.5
11.3
81.7
55.7
3.7
10.8
4.6
37.5
1.5
0.0
L2
12.3
31.5
75.3
46.1
41.0
56.6
22.8
31.2
22.0
0.5
JPEG
0.1
7.4
84.8
13.7
1.8
74.8
0.3
21.0
0.5
0.0
StAdy
0.6
2.1
77.1
2.6
1.2
3.7
65.3
2.9
0.6
0.0
ReColorAdy
0.0
0.1
90.1
0.2
0.0
0.1
0.0
69.3
0.0
0.0
All (random)
0.9
78.6
38.3
26.4
61.3
1.4
32.5
16.1
0.2
PAT-self
32.5
46.4
72.6
45.0
37.7
53.0
51.3
45.1
29.2
2.4
PAT-AlexNet
25.5
44.7
75.7
46.8
41.0
55.9
39.0
40.8
31.1
1.6
orAdv (Laidlaw and Feizi] 2019), and PPGD and LPA. This allows us determine if PAT gives robustness against range of adversarial attacks. We experiment with using various models to calculate the LPIPS distance during PAT. We try using the same model both for classification and to calculate the LPIPS distance (self-bounded PAT). We also use AlexNet trained on CIFAR- - prior to PAT siteernally-boundee PAT). We find that PAT outperforms Ip adversarial training and TRADES (Zhang et al. 2019a), improving the union accuracy from <5% to 220, and nearly doubling mean accuracy against unseen threat models from 26% to 49%. Surprisingly, we find that PAT even outperforms threat-specific AT against StAdv and ReColorAdv; see Appendix F.3 for more details. Ablation studies PAT are presented in Appendix F.I
ImageNet-100 We compare ResNet-50s trained on the ImageNet-100 dataset with PAT and adversarial training (Table Classifiers are tested against seven attacks at the medium bound from the perceptual study (see Section and Appendix Table 4.. Self- and externally-bunnde similar results. Both produce more than double the next highest union accuracy and also significantly increase the mean robustness against unseen threat models by around 15%.
PAT give
Perceptual attacks On both CIFAR-10 and ImageNet-100, we find that Perceptual PGD (PPGD) and Lagrangian Perceptual Attack (LPA) are the strongest attacks studied. I.AA is the strongest,
8

---

Published as a conference paper at ICLR 2021
Distances of adv. examples
(a) Los distance
(b) Le distance
(c) LPIPS (self)
(d) LPIPS (AlexNet)
1.0
1.5
2
75
1.0
0.5
50
25
I
0.5
0.0
0
0.0
L2
JPEG
StAdv
ReColorAdy
Figure We generate samples via different adversarial attacks using narrow threat models in ImageNet-100 and measure their distances from natural inputs using Lp and LPIPS metrics. The distribution of distances for each metric and threat model shown as a violin plot. (a-b) Lp metrics assign vastly different distances across perturbation types, making it impossible train against all of them using Lp adversarial training. (c-d) LPIPS assigns similar distances to similarly perceptible attacks, single training method, PAT, give good robustness across different threat models.
reducing the most robust classifier 9.8% accuracy CIFAR-10 and 2.4% accuracy ImageNet- 100. Also, models most robust to LPA in both cases are those that have the best union and unseen mean accuracies. This demonstrates the utility of evaluating against LPA robustness against range of threat models. See Appendix for further attack experiments.
proxy for adversarial
Comparison to other defenses against multiple attacks Besides the baseline of adversarially training against single attack, we also compare PAT to adversarially training against multiple attacks (Tramer and Boneh 2019 Maini al. 2019). We compare three methods for multiple-attack training: choosing a random attack at each training iteration, optimizing the average loss across all attacks, and optimizing the maximum loss across attacks. The latter two methods are very expensive. increasing training time by factor equal to the number of attacks trained against, so we only evaluate these methods on CIFAR- 10. As in Tramer and Boneh (019)), we find that the maximum loss strategy leads to the greatest union accuracy among the multiple-attace training methods. However, PAT performs even better on CIFAR-10, despite training against none of the attacks and taking one fourth of the time to train. The random strategy, which is the only feasible one ImageNet- 100, performs much worse than PAT. Even the best multiple-atte unseen neural perceptual attacks, PPGD and LPA, achieving much lower accuracy than PAT.
training strategies still fail to generalize to the
On CIFAR-10 10, we also compare PAT to manifold regularization (MR) (Jin and Rinard 2020), non-adversarial training defense. MR gives union accuracy close to PAT-self, but much lower clean accuracy; for PAT-AlexNet, which gives similar clean accuracy to MR, the union accuracy is much higher.
Threat model overlap In Figure 2 we investigate how the sets of images vulnerable to L2, spatial, and perceptual attacks overlap. Nearly all adversarial examples vulnerable L2 or spatial attacks also vulnerable to LPA. However, there is only partial overlap between the examples vulnerable to L2 and spatial attacks. This helps explain why PAT results in improved robustness against spatial attacks (and other diverse threat models) compared to L2 ad adversarial training.
Why does PAT work better than Lp adversarial training? In Figure we give further expla- nation of why PAT results in improved robustness against diverse threat models. We generate many adversarial examples for the Lx.. L2, JPEG, StAdv, and ReColorAdy threat models and measure their distance from the corresponding natural inputs using I.p distances and the neural perceptual distance, L.IPS. While I,p distances vary widely, L.IPS gives remarkably comparable distances to different types of adversaria examples. Covering all threat models during Loo L2 adversarial training would require using huge training bound, resulting in poor performance. In contrast, PAT obtain robustness against all the narrow threat models at reasonable training bound.
Robustness against common corruptions In addition to evaluating PAT against adversarial examples, also evaluate its robustness to random perturbations in the CIFAR-10-C and ImageNet- datasets (Hendrycks and Dietterich, 2019). We find that PAT gives increased robustness (lower relative mCE) against these corruptions compared to adversarial training; see Appendix G for details.

---

Published as a conference paper at ICLR 2021
8 CONCLUSION
We have presented attacks and defenses for the neural perceptual threat model (realized by the LPIPS distance) and shown that it closely approximates the true perceptual threat model, the set of all perturbations to natural inputs which fool a model but are imperceptible to humans, Our work provides novel method for developing defenses against adversarial attacks that generalize to unforeseen threat models. Our proposed perceptual adversarial attacks and PAT could be extended to other vision algorithms, or even other domains such as audio and text.
ACKNOWLEDGMENTS
This project was supported in part by NSF CAREER AWARD 1942230, HR 001 09911990077, HR001 12090132, HR001 1990002, NIST 60NANB20D134, AWS Machine Learning Research Award and Simons Fellowship on "Foundations of Deep Learning."
REFERENCES
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Sindié, Pavel Laskow, Giorgio Giacinto, and Fabio Roli. Evasion Attacks against Machine Learning at Test Time. In Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Zelezný, editors, Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, pages 387-402, Berlin, Heidelberg, 2013. Springer. ISBN 978-3-642-40994-3 doi: 10.
25. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Frhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL http::/ arxiv. org/ abs/ 1312. 6199
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1rXiv:1607.02533 201
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings the IEEE International Conference on Computer Vision, pages 1369-1378, 2017.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations, 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, International Conference on Learning Representations, 2018.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael I. Jordan. Theoreti- cally Principled Trade-off Between Robustness and Accuracy. In JCML, 2019a.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, Aleksander Madry. Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations. arXiv preprint arXiv:1712.02779, 2017.
Eric Wong, Frank Schmidt, and Zico Kolter. Wasserstein Adversarial Examples via Projected Sinkhomn Iterations. arXiv preprint 0tt:::9902.0900000 February 2019. arXiv: 1902.07906.
Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially Transformed Adversarial Examples. arXiv preprint or 2018.
Hossein Hosseini Radha Poovendran. Semantic Adversarial Examples. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition Workshops, pages 1614-1619, 2018.
Cassidy Laidlaw and Soheil Feizi. Functional Adversarial Attacks. In NeurlPS, 2019.
Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, David Forsyth, Big but Imperceptible Adversarial Perturbations via Semantic Manipulation. arXiv preprint arXiv: 1904.0664444 2019.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing Unrestricted Adversarial Examples with Generative Models. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8322-832-8 8333, Curran Associates Inc., 2018.
10

---

Published as a conference paper at ICLR 2021
Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang. Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi Keung Tang. and Alan L. Yuille. Adversarial Attacks Beyond the Image Space. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition, 2019, arXiv: 1711.07183.
Matt Jordan, Naren Manoj. Surbhi Goel, and Alexandros G. Dimakis. Quantifying perceptual distortion of adversarial examples, 2019.
Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial Robustness Against the Union of Multiple Perturbation Models, arXiv: 1909,00999448888 [cs, stat/, September 2019. URL/http://axxx.ccm abs/ 1909. 04068
arXiv: 1909.04068.
Florian Tramer and Dan Bonch. Adversarial Training and Robustness for Multiple Pertur- bations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dttextquotesinglc Alché-Buc, E. Fox, and R. Gareett, editors, Advances in Neural Information Processing Systems 32, pages 5866-5876. Curran Associates, Inc., 2019. URL http::/papers.nips.../// robustness- 8821 -tt-aarrrrarral--rainnnnn--- -oseaaa--s-aiinn-n--oooossssssss for l odf
Daniel Kang, Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. Testing Robustness Against Unforescen Adversaries. arXiv preprint arXiv:1908.0000 0800,, 2019.
B. Brown, N. Carlini, C. Zhang. Olsson, P. Christiano, and Goodfellow. Unrestricted adversarial examples. arXiv preprint arXiv: 1eXXiv:1809008300 08052, 2018.
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. arXiv: [cs], May 2018. [tRp:ttt://axxixi httpaaawwiv.....raa/////a htrgg/bs/1711 wwt.s.11..... 09665
Adversarial Patch. arXiv: 1712.09665.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visihility to structural similarity. IEEE Transactions on Image Processing, 1304)600-612, April 2004, ISSN 1941-0042. doi: 1099T..0/.220 20......000..198888 819861. Conference Name: IEEE Transactions on Image Processing.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features Perceptual Metric. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57. IEEE, 2017.
Alexey Kurakin, Tan J. Goodfellow, and Samy Bengio. Adversarial machine learning scale. ArXiv, abs/1611..0126. 2016b.
Mahmood Sharif, Lujo Bauer, and Michael K. Reiter. On the Suitability of Lp-norms for Creating and Preventing Adversarial Examples. 1rXiv:1802.0065: 09653 [cx], July 2018. URL http:::/aacxvv..orr//////////// arxiv: 1802.09653.
09653
Isaac Dunn, Tom Melham, and Daniel Kroening. Semantic Adversarial Perturraton using Learnt Represen- tations. arXiv:2001.1100 [cs], January 2020. UtL.:stpp///arrbb..rr//a///D...0
arXiv:
1055.
Qiuling Xu, Guanhong Tao, Siyuan Cheng, Lin Tan, and SlangyuZaang Towards Feature Space Adversarial Attack. arXiv:2004.13385 [cs, eess]. April 2020. http::::///x.v../////a/../ 2004.12388
arXiv:
Charles Jin and Martin Rinard. Manifold Regularization for Locally Stable Deep Neural Networks. March 2020.
URL httpts:t/t//tttt //tpivv.aiivvvv//a org/ abs /200. 04286v2]
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-Calibratted Adversarial Training: Generalizing to
Unseen Attacks. 1rXiv:1910.06253 [cs, stut]. February 2020. http:////axx http://axwivv..r//////11
06259
arXiv: 1910.06259.
Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference Signals, Systems Computers, 2003, volume 2, pages 1398-1402 Vol.2, November 2003. doi: 10. 10.1109//ASSC.200...22000
Mehul P. Sampat, Zhou Wang. Shalini Gupta, Alan Conrad Bovik, and Mia K. Markey. Complex Wavelet Structural Similarity: A New Image Similarity Index. IEEE Transactions on Image Processing, 18(1)) 2385 2401, November 2009. ISSN 1941-0042, doi: 10. 10.1009/IP22099222522. Conference Name: IEEE Transactions on Image Processing.
11

---

Published as a conference paper at ICLR 2021
Rafal Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. IDDR-VDP-2: calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on Graphics, 30 10:11, July 2011. ISSN 0730-0301. doi: 041144520100344199999999 UR/httpp:://6o..rg
10.1145/2010330 201022110322.4.0 1964935
Zhou Wang and Eero P. Simoncelli. Maximum diffcrentiation (MAD) competition: A methodology for
comparing computational models of perceptual quantities. Journal of Vision, 8 (-)))888, September 2008.
ISSN 1534-7362. doi: 10.1 10.7167/8.11 12.8. URLpttps:: http:://p///oo //jov j org/ article..... aspx?
articleid=2199
Publisher: The Association for Research in Vision and Ophthalmology.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal Muliinnniiiiindaannnnniiiiiii Image-to-image Translation.
pages 172-189, 2018. URL. http:///cpp//gpa.aaaa...eeeeoooa
RCCV 20t_v__011hhhm
Xun Xun_Huunge Hww.Haaan_____ moddal wwwpaaaunnuuuuurrueeeee www.appprrrisseeaaaaaae--------e orgpap/oooooooiooooooooooooooooo http:ooeeeinneeee____eeee Bac_s_pappe_eee
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial
Networks. pages 4401-4410, 2019. URL
thecvff www.://teeccttttceee/ee CVPR
A Style Based Generator Architecture for Generative Adversarial Networks
2019/html/Karte
(acpll_pppep.._ html
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In E. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097 1105. Curran Associates, Inc., 2012. URL http::/papers. nips ccppppprrrrpppp
4824 htg.::/--.aasatiatttto.ooo.....-
pdt
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)211-255) 2015. doi: 10. 126-0-/1126--115-081666
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, Citeseer, 2009.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with ensemble of diverse parameter-free attacks. In 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Hossein Hosseini, Baicen Xiao, Mayoore Jaiswal, and Radha Poovendran. On the Limitation of Convolutional Neural Networks in Recognizing Negative Images. In 16th IEFE International Conference Machine Learning and Applications (ICMLA), pages 352-358. IEEF, 2017.
Huan Zhang, Hongge Chen, Zhao Song, Duane S. Boning, Inderjit S. Dhillon, and Cho-Jui Hsich. The Limitations of Adversarial Training and the Blind-Spot Attack. International Conference on Learning Representations, 2019b.
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradcoff's in neural nets. arXiv: 1eXivv/910.000 1910.085// fcs, stat), October 2019. Bttpbbt:://axxx.. /bs/ 1910.08051
arXiv: 1910.08051.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Rohustness May Be at Odds with Accuracy. arXiv: 12152 [cs, star], September 2019. URL http::/arxiv arg/abs/ 805. 12152 arXiv: 1805.12152.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic Differentiation in PyTorch. In NIPS- W. 2017.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019.
URL httl:htts:///ithhttt thw.. com http://hhhuubt..ommmaaaaaaaara/p wtw..ryyy..bbrroout.eeesss
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. September 2014. URL BtLLLttpsss//aaaw.ooo///asss// 155666
12

---

Published as a conference paper at ICLR 2021
APPENDIX
A PERCEPTUAL ATTACK ALGORITHMS
A.I PERCEPTUAL PGD
Recall from Section 4 that Perceptual PGD (PPGD) consists of repeatedly applying two steps: first-order step in LPIPS distance to maximize the loss, followed by a projection into the allowed set of inputs. Here, we focus on the first-order step; see Appendix A.4 for how we perform projection onto the LPIPS ball.
We wish to solve the following constrained optimization for the step d given the step size 1 and current input x:
max L(f(x 5),9) subject || 12 S 1
Let f(x) = ((f(x),)) y) for an input x E X. Let be the Jacobian of (()) at x and V be the gradient of f(.) x.
Lemma 1, The first-order approximation of 7 is
max f(x) + (V
subject | S
(8)
and can be solved in closed-form by
(J (JTJ)- [[J+)](()))) (ViIl2
8" = 11
where J+ is the pseudoinverse of
Proof. We solve (8) using Lagrange multipliers. First, we take the gradient of the objective:
V [
Vf
We rewrite the constraint by squaring both sides obtain
8 J J8 2 0
Taking the gradient of the constraint gives
V _ J JÓ c²1 2
Now, we gradient multiple of the other and solve for 5:
J8 J
J
X(V
(9)
8 6= X(J J)-
(10)
Substituting into the constraint from g
gives
||J8||2 = 1
[V 1
([ = 7
17
{|[[]) (V)-)
17
X
((J+))(VÍiII)
We substitute this value of À into
10 to obtain
8" (JTJ)-
(1I)
((J++T(V.))) (v/)22
13

---

Published as a conference paper at ICLR 2021
"shopping basket"
f(x) E y
x X
((x) A
LPIPS
< E
"fiddler crab"
d(x, x)
| o(x) |
((x) + f(x)
X X
6(x) A
Predicted
Classifier
Original input Adv. example
LPIPS network
Normalized activations
LPIPS distance
label
network
Figure 6: Creating an adversarial example in the LPIPS threat model.
Solution with conjugate gradient method Calculating IT] directly is computationally intractable for most neural networks, since inverting J J and calculating the pseudoinverse of J are expensive. Instead, we approximate d" by using the conjugate gradient method to solve the following linear system, based on (9):
J = v
(12)
V is easy to calculate using backpropagation. The conjugate gradient method does not require calculating fully JJ J; instead, it only requires the ability perform matrix-vector products J/ Ju for various vectors v.
We can approximate Ju using finite differences given a small, positive value h:
Ju ~ <(x + hv) o(x)
Then, calculate J Ju by introducing additional variable and using autograd:
(
dá du
(x + m) T
Vu [0(x + u))² Ju
Ju + (ő(x + u))²
Ju
u=0
du
1mm0
deb du (so + 11)
T
Ju (Ø(x + u))
u==
de (z)) du
T
Ju JJ
This allows us to efficiently approximate the solution of iterations the conjugate gradient algorithm practice.
12 to obtain (JT We use
From there, it easy to solve for X, given that (J+) (- VV. Then, 8* can be calculated via (10). See Algorithm I for the full attack.
Computational complexity PPGD's running time scales with the number of steps T and the number of conjugate gradient iterations K. It also depends whether the attack is self-bounded (the same network is used for classification and the LPIPS distance) or externally-bout bounded (different networks are used).
For each of the T steps, (xx), VxC(f(x), y). and @(x + hōg) must be calculated once (lines 4 and 15 in Algorithm [ This takes 2 forward passes and backward pass for the self-bounded case, and forward passes and backward pass for the externally- boundddd case.
In addition, J Ju needs be calculated (in the MULTIPLYJACOBIAN routine) K + times. Each calculation of JT Ju requires forward and backward pass, assuming o(x) is already calculated.
Finally, the projection step takes + forward passes for iterations of the bisection method (see
14

---

Published as a conference paper at ICLR 2021
Section A.4).
In all, the algorithm requires T(K + + 4) forward passes and T(K + +3) backward passes in the self-bounder case. In the externally-boundee bounded case, it requires T(K + 5) forward passes and the same number backward passes.
Algorithm Perceptual PGD (PPGD)
1: procedure PPGD(classifier f(-), LPIPS network <(-), imput x, label bound step 17) X x 0.01 + 1)
2:
initialize perturbations with random Gaussian noise > T is the number of steps
3:
for ł in 1..... T do
4:
vf Vx((I(X),w)
5:
dg +-
6:
ro < Vf MULTTPLYYACCOBIANG X, 66)
7:
Po < To
8:
for in 0, ......,,,, do
conjugate gradient algorithm; we use K iterations
rT
9:
Og
MULTIPLYACOOIAN
10:
6k-1 < 6% axpk
rk+1 < Tk ag MULTTIPLYJACOBIANN X,Ps)
3x + 1
12:
TE
13:
Pa+1 Fk+1 + BEPK
14:
end for
15:
m ||<((x + h10x) (13))1//
||Jox|| for small h; we use 10-3
16:
x - (n/m)ō
+ (R,
18:
end for return
20:
end procedure
22:
procedure MUL.TPLYYJACOBIAN((() &, v)
D calculates JT Jv; J is the Jacobian of at x
23:
Jv + (xx -xx)
>> a small positive value; we use 10-3
JJ V. [6(x+ u=0
25: return JT
end procedure
A.2
LAGRANGIAN PERCEPTUAL. ATTACK (LPA)
Our second attack, Lagrangian Perceptual Attack (LPA), optimizes Lagrangian relaxation of the perceptual attack problem B
[(f(x),) Amax (o, (1, - o(x) -2
max
(13)
To optimize we use variation of gradient descent over x, starting at x with a small amount of noise added. We perform our modified version of gradient descent for T steps. We use a step size which begins at e and decays exponentially to e/10.
At each step, we begin by taking the gradient of (13) with respect to x; let A refer to this gradient. Then, we normalize A to have L2 norm I, i.e. Á |/|| of Â of size di in LPIPS distance. If wanted to take step of size v) in T-2 distance, just take the step n'.. However, taking a step of particular size in LPIPS distance harder. We assume that the LPIPS distance is approximately linear the direction Å. We can approximate the directional derivative of the LPIPS distance in the direction Å using finite differences:
12. We wish to take step in the direction could
d de Here, h is small positive value, and assign the approximation the directional derivative m. Now, we can write the first-order Taylor expansion of the perceptual distance torwards the direction
d x hA)
d(x,x +
m.
15

---

Published as a conference paper at ICLR 2021
Å follows:
d X + a②) = d(x ) + ma = ma.
we want to take a step of size 1). Plugging in and solving, we obtain
d(x, X + aL)
1 R ma
n/ m R
So, the approximate step we should take is (n/m) We take this step at each of the T iterations of our modified gradient descent method.
We begin with 10 10-2. After performing gradient descent, if d(x.x) > c (i.e. the adversarial
example is outside the constraint) we increase À by a factor of 10 and repeat the optimization. We repeat this entire process five times, meaning we search over À E (10-2 10-1, 10 10 102). Finally, if the resulting adversarial example still outside the constraint, we project it into the threat model; see Appendix
Computational complexity LPA's running time scales with the number iterations S used to search for À as well as the number of gradient descent steps T. ((x) may be calculated once during the entire attack, which speeds it up. Then, each step of gradient descent requires forward and backward passes in the self-bounded case, and 3 forward and 2 backward passes in the externally- bounded case.
The projection at the end of the attack requires forward passes for iterations of the bisection method (see Section A.4).
In total, the attack requires 2ST + + forward passes and ST + + backward passes in the self-bounded case, and 3ST + + forward passes and 2ST + + backward passes in the externally-bounded case.
Algorithm 2 Lagrangian Perceptual Attack (LPA)
procedure LPA(classifier network f(-). LPIPS distance d(-,)) input x, label y. bound
2:
À - 0.01
3: x x 0.01 * 1))
initialize perturbations with random Gaussian noise
4:
for i in ,,..., do
use S 5 iterations to search for the best value of X
5:
for t I..... .T do
>> ' is the number of steps
6:
A Vx [C(F(X))))) v) maxx (0,d(x,x) c)]
take the gradient of (5)
7:
Å A/||a|2
b normalize the gradient
8:
7 * (0.1)//T
D the step size 7 decays exponentially D m RJ derivative of d(x, in the direction of Á; h = 0.1 take a step of size in LPIPS distance
9:
m 4 d(x, x + ha)/h
10:
x x (n/m)A
11:
end for
12:
if d( x) > then
13:
< 10A end if end for
b increase À if the attack goes outside the bound
14:
15:
16:
x PROJECT(d, x,,
17:
return x
end procedure
A.3 FAST LAGRANGIAN PERCEPTUAL ATTACK
We use the Fast Lagrangian Perceptual Attack (Fast-LPA) for Perceptual Adversarial Training (PAT, see Section 55. Fast-LPA is similar to LPA (Appendix A.2). with two major differences. First, Fast-LPA does not search over values; instead, during the gradient descent steps, is increased exponentially from 1 to 10. Second, we remove the projection step at the end of the attack. This means that Fast-LPA may produce adversarial examples outside the threat model. This means that Fast-LPA cannot be used for evaluation, but it is fine for training.
16

---

Published as a conference paper at ICLR 2021
Computational complexity Fast-LPA's running time can be calculated similarly to LPA's (see Section A.2), except that = and there is no projection step. Let T be the number of steps taken during the attack. Then Fast-LPA requires 2T' + 1 forward passes and T + backward passes for the self-bounded case, and 37 + forward passes and 27 + backward passes for the externally-boundee case.
In comparison, PGD with T iterations requires 7 forward passes and T backward passes. Thus, Fast-LPA slightly slower, requiring + more forward passes and more backward passes.
Algorithm 3 Fast Lagrangian Perceptual Attack (Fast-LPA)
procedure FASTLPA network f(), LPIPS distance d(; -), input x, label y. bound c)
2:
X <- 0.01 * 1), for 1, I......' T do
D initialize perturbations with random Gaussian noise T is the number of steps
3:
4:
<- 10t/T
> À increases exponentially take the gradient of (5)
5:
A <- Vx [A(S(X)))) X Amax (0, - -c]]
6:
Å A/||A||2
normalize the gradient the step size v) decays exponentially
7:
1 = c (0.1)./T
8:
m f d(E, & + hā)/h
b m R derivative of d(x,) in the direction of A = 0.1 take a step of size in LPIPS distance
9:
X x (n/ m//m)
end for
11: return x
end procedure
A.4 PERCEPTUAL PROJECTION
We explored two methods of projecting adversarial examples into the LPIPS thread model. The method we use throughout the paper is based on Newton's 's method and is shown in algorithm 4 However, we also experimented with the bisection root finding method, shown Algorithm (also see Appendix D.)
(also
In general, given an adversarial example x, original input x, and LPIPS bound e, we wish to find projection x' of x such that d(X', x) S e. Assume for this section that d(X, x) > i.c. the current adversarial example X is outside the bound. d(X, x) I e, then we can just let X = X and done.
Newton's method The second projection method we explored uses the generalized New- ton- Raphssn method to attempt to find the closest projection X' to the current adversarial example x such that the projection is within the threat model, i.e. d(X', x) < c. To find such a projection, again define a function r(.) and look for its roots:
r(X) = d(x',xx x) -
If can find a projection X close to X such that r(x') O, then this projection will be contained within the threat model, since
rxx) < d(x'' x)
To find such a root, we use the generalized Newton Waaton-Raphson method, an iterative algorithm. Begin- ning with X = I, update x' iteratively using the step
X
X [v({3)
(r (+xx)+ +s).
where A denotes the pscudoinverse of A. and is small positive constant (the "overshoot"), which helps the algorithm converge. We continue this process until r(x)) < 0, at which point the projection complete.
This algorithm usually takes 2-3 steps to converge with s = 10 10-2 Each step requires forward and 1 backward pass calculate r r(x)) and its gradient. The method also requires forward pass the beginning to calculate @(x).
17

---

Published as a conference paper at ICLR 2021
Algorithm 4 Perceptual Projection (Newton's Method)
procedure PROJECT(LPIPS distance d(.;)) adversarial example x, original input x, bound e)
X
for i in 0..- ... do r(x) <- d(x,, x) -
(x)) S then
return X
end if
K4 = [r(x3)
[ (F(x))
s is the "overshoot"; we use 10-2
end for end procedure
Bisection method The first projection method we explored (and the one we use throughout the paper) attempts to find projection x' along the line connecting the current adversarial example x x. Then we can represent our final projection X' a point between
and original input x. Let d and x as
x = x + ad, if xf
for some [0,1]. If 0, x'
X. Now, define function r: |0, 1] R
r()) = d(x (x + αő, x) -c
This function has the following properties:
r(0) < 0, since r(0) d(x, x) - - -6,
2. r(1) 0, since r(1) d(xx x) - because d(X, x) e.
3. r(a) O iff d(x", x) =
We use the bisection root finding method to find a root a* of r()) on the interval [0, 1], which exists since r(-) is continuous and because of items and 2 above. By item 3, at this root, the projected adversarial example is within the threat model:
d(x', x) = d(x + a d, = r(aaa) +c="
We use 10 iterations of the bisection method to calculate ax, This requires + 1 forward passes through the LPIPS network, since ((x) must be calculated once, and ø(x + ad) must be calculated times. See Algorithm 5 for the full projection algorithm.
Algorithm 5 Perceptual Projection (Bisection Method)
procedure PROJECT(LPIPS distance d(.,)), adversarial example X, original input x, bound e)
O'nnny max <- 0,1 5 x
for l...., n2 do
(0min Ormax) 1/2
+- x + ad
if d(x, x') then
Qmax + a
else
Amain + a
end if
end for return x'
end procedure
B ADDITIONAL RELATED WORK
Here, we expand on the related work discussed in Section 2 discuss some additional existing work on adversarial robustness.
18

---

Published as a conference paper at ICLR 2021
Much of the initial work adversarial robustness focused perturbations to natural images which were bounded by the L2 or Lee distance (Carlini and Wagner 2017 Goodfellow et al. 2015 Madry et al., 2018). However, recently the community has discovered many other types of perturbations that are imperceptible and can be optimized to fool a classifier, but are outside T.p threat models. These include spatial perturbations using flow fields (Xiao al., (018). translation and rotation (Engstrom et al. (017), and Wassterstein distance bounds (Wong et al. 2019). Attacks that manipulate the colors images uniformly also been proposed (Hosseini and Poovendran 2018 Hosseini et al. 2017) Zhang et al. (019b) and have been generalized into "functional adversarial attacks" by] Laidlaw and Feizi| (2019).
Adversarial attacks
A couple papers have proposed adversarial threat models that do not focus on simple, manually defined perturbation type. Dunn et al. (2020) use generative model of images; they perturb the features at various layers in the generator to create adversarial examples, Xu et al. (2020) train an autoencoder and then perturb images in representation space rather than pixel space.
C ADDITIONAL ADVERSARIAL EXAMPLES
Original
Self-bd. PPGD
External-bd. PPGD
Self-bd. LPA
External-bdl LPA
Figure 7: Adversarial examples generated using self-bounded and Enternally-boundee PPGD and LPA perceptual adversarial attacks (Section 4 with large bound. Original images shown in the left column and magnified differences from the original are shown to the right of the examples.
19

---

Published as a conference paper at ICLR 2021
D ADDITIONAL PERCEPTUAL STUDY RESULTS
Table 4: Bounds and results from the perceptual study. Each threat model was evaluated with small, medium, and large bound. Bounds for 1.2, T.ac and JPHG attacks (first three rows) are given assuming input image is in the range [0, 255]. Perceptibility (perc.) is the proportion of natural input-adversarial example pairs annotated as "different" by participants. Strength (str.) is the success rate when attacking classifier adversarially trained against that threat model (higher is stronger). Perceptual attacks (PPGD and LPA, see Section 4 are externally bounded with AlexNet. All experiments on ImageNet- 100.
Threat model
Small bound
Medium bound Perc.
Large bound Perc.
Bound
Perc.
Str.
Bound
Str.
Bound
Str.
Loo
2
7%
30.4%
4
11%
44.3%
8
39%
55.7%
L2
600
8%
40.6%
1200
12%
59.0%
2400
51%
88.5%
JPEG-L.x
0.0625
6%
19.5%
0.125
16%
25.2%
0.25
51%
44.0%
StAdy
0.025
8%
28.1%
0.05
17%
34.7%
0.1
37%
55.4%
ReColorAdy
0.03
14%
14.8%
0.06
42%
30.8%
0.12
75%
71.1%
PPGD
0.25
7%
47.9%
0.5
9%
68.6%
25%
91.8%
LPA
0.25
14%
76.5%
0.5
27%
98.4%
45%
100.0%
PERCEPTUAL ATTACK EXPERIMENTS
We experiment with variations of the validation attacks, PPGD and LPA, described in Section 4 As described in Appendix A.4 we developed two methods for projecting candidate adversarial examples into the LPIPS ball surrounding a natural input. We attack single model using PPGD and LPA with both projection methods. We also compare self-bounded to externall-boundde attacks.
We find that LPA tends to more powerful than PPGD. Finally, we note that externally-boune IPA is extremely powerful, reducing the accuracy of PAT-trained classifier on ImageNet- 100 to just 2.4%.
Besides these experiments, we always use externally -bounded attacks with AlexNet for evaluation. AlexNet correlates with human perception of adversarial examples (Figure and provides a standard measure of LPIPS distance; in contrast, self-bounded attacks by definition have varying bounds across evaluated models.
Table 5: Accuracy of PAT-trained ResNer-50 ImageNet- 100 against various perceptual adversarial attacks. PPGD and LPA attacks are shown self- bounded and externaly--ounded with AlexNet. We also experimented with different perceptual projection methods (see Appendix A.4). Bounds are c 0.25 for self self-bounded attacks and = 0.5 for externally-oounde attacks, since the LPIPS distance from AlexNet tends be about twice as high as that from ResNet-50.
Attack
LPIPS model
Projection
Accuracy against PAT
PPGD
self
bisection method Newton's 's method
43.3
PPGD
self
49.7
PPGD
AlexNet
bisection method
45.2
PPGD
AlexNet
Newton's 's method
28.7
LPA
self
bisection method
39.6
LPA
self
Newton's 's method
53,8
LPA
AlexNet
bisection method
4.2
LPA
AlexNet
Newton's 's method
2.4
20

---

Published as a conference paper at ICLR 2021
F PAT EXPERIMENTS
F.1 ABLATION STUDY
We perform an ablation study of Perceptual Adversarial Training (PAT). First, we examine Fast-LPA. the training attack. We attempt training without step size (1) decay and/or without increasing during Fast-LPA, and find that PAT performs best with both 17 decay and increase.
Training classifier with PAT gives robustness against a wide range of adversarial threat models (see Section 7). However, tends to give low accuracy against natural, unperturbed inputs. Thus, we use technique from Balaji et al. (2019) to improve natural accuracy in PAT-trained models: at each training step, only inputs which are classified correctly without any perturbation are attacked. In addition to increasing natural accuracy, this also improves the speed of PAT since only some inputs from each batch must be attacked. this ablation study, we compare attacking every input with Fast-LPA during training to only attacking the natural inputs which already classified correctly. We find that the latter method achieves higher natural accuracy at the cost of some robust accuracy.
Table 6: Accuracies against various attacks for models in the PAT ablation study. Attack bounds 8/255 for Loo, for Lg, 0.5 for PPGD/LPA, and the original bounds for StAdv/RcColorAdv.
Union
Unseen mean
Narrow threat models
NPTM
Ablation
Clean
Lec
L2
StAdv
ReColor
PPGD
LPA
None
21.9
45.6
82.4
30.2
34.9
46.4
71.0
13.1
2.1
No decay
17.2
48.1
82.7
37.5
43.3
39.7
72.1
10.5
1.1
No increase
8.1
42.1
85.1
33.7
35.9
27.8
71.1
11.6
1.0
No decay or À increase
19.6
49.2
82.1
36.7
41.9
44.8
73.4
10.9
0.9
Attack inputs
26.8
46.6
74.5
29.8
33.5
56.6
66.4
24.5
6.5
F.2 PROJECTION DURING TRAINING
We choose not add projection step to the end of Fast-LPA during training because it slows down the attack, requiring many more passes through the network per training step. However, we tested seffbboundd PAT with a projection step and found that increased clean accuracy slightly but decreased robust accuracy significantly. We believe this is because not projecting increases the effective bound on the training attacks, leading to better robustness. To test this, we tried training without projection using smaller bound (e 0.4 instead of 0.5) and found the results closely matched the results when using projection at the larger bound. That is, PAT with projection at 0.5 similar to PAT without projection at 0.4. These results shown in
Table 7: Accuracies against various attacks for PAT-trained models on CIFAR-1O, with and without projection step during training.
Union
Unseen mean
Narrow threat models
NPTM
Projection
Bound (c)
Clean
Loc
L2
StAdy
ReColor
PPGD
LPA
No
0.5
21.9
45.6
82.4
30.2
34.9
46.4
71.0
13.1
2.1
Yes
0.5
5.8
41.4
83.9
35.7
38.5
17.9
73.3
10.7
1.7
No
0.4
4.9
36.3
84.1
31.1
36.3
10.3
67.4
10.9
3.2
F.3
SELF-BOUNDED VS. ALEXNET-BOUNDDA PAT
Performing externally -oundeed PAT with AlexNet produces more robust models on CIFAR-10 than self-bounded PAT. This not the on ImageNet-100, where self- and AlexNet-bounded PAT perform similarly.
There is simple explanation for this: the effective training bound CIFAR-10 is greater for AlexNet-bounder PAT than for self-bounded PAT. To measure this, we generate adversarial examples
21

---

Published as a conference paper at ICLR 2021
CIFAR-10
ImageNet-100
Union robust accuracy
PAT (ours)
20
20
Adv. training
(narrow threat models)
Normal training
80
90
80
90
Clean accuracy
Clean accuracy
Figure 8: Several classifiers trained with PAT, adversarial training, and normal training on CIFAR-10 and ImageNet-100 are plotted with their clean accuracy and accuracy against the union of narrow threat models (see Section 7 for robustness evaluation methodology). PAT models on both datasets outperform adversarial trained models in both clean and robust accuracy.
for all the test threat models on CIFAR-10 (L2, Lox, StAdv, and ReColorAdv). We find that the average LPIPS distance for all adversarial examples using AlexNet is 1.13; for PAT-trained ResNet- 50, it is 0.88. Because this disparity, we use lower training bound for self-bounded PAT (e = 0.5) than for lexuneebounded PAT = 1). However, this means that the average test attack has 76% greater LPIPS distance than the training attacks for self-boue uuffdbouddd PAT, whereas the average test attack has only 13% greater LPIPS distance for AlexNet-bounde bounded PAT. This explains why lex-eetbounded PAT gives better robustness; it only has to generalize to slightly larger attacks on average.
We tried performing AlexNet-bounded PAT with more comparable bound (e 0.7) to self-bounded PAT. This gives the average test attack about 80% greater I.IIS distance than the training attacks, similar celf-bounded PAT. Table shows that the results are more similar for self-bounded and AlexNet-bounded PAT with 0.7.
Table 8: Accuracies against various attacks for classifiers CIFAR-10 trained with self- and AlexNet-bounded PAT using various bounds.
Union
Unseen mean
Narrow threat models
NPTM
LPIPS model Bound (c)
Clean
Too
T
StAdv
ReColor
PPGD
IPA
Self
0.5
21.9
45.6
82.4
30.2
34.9
46.4
71.0
13.1
2.1
AlexNet
0.7
16.7
45.3
80.0
35.5
41.3
33.2
71.5
17.8
4.9
AlexNet
1.0
27.8
48.5
71.6
28.7
33.3
64.5
67.5
26.6
9.8
F.4 ACCURACY-RORRISTNES TRADEOFF
Tsipras et al. 2019 have noted that there is often tradcoff the between adversarial robustness of classifier and its accuracy. That is, models which have higher accuracy under adversarial attack may have lower accuracy against clean images. We observe this phenomenon with adversarial training and PAT. Since PAT gives greater robustness against several narrow threat models, models trained with tend have lower accuracy on clean images than models trained with narrow adversarial training. In Figure g we show the robust and clean accuracies of several models trained on CIFAR- 10 and ImageNet- 100 with PAT and adversarial training. While some PAT models have lower clean accuracy than versariall trained models, at least one PAT model on each dataset surpasses the Pareto frontier of the accuracy-robustree both datasets with both higher robust accuracy and higher clean accuracy than adversarial training.
tradeoff for adversarial training. That is, there are PAT-trained models on
F.5 PERFORMANCE AGAINST STADV AND RECOLORADY
was surprising to find that PAT outperformed threat-specific adversarial training (AT) against the StAdv and ReColorAdy attacks CIFAR-10 (it does not do ImageNet-100), In Table (partially reproduced in
PAT-AlexNet improves robustness over AT against StAdv from 54% 65%; PAT-self improves robustness AT against ReColorAdy from 65% to 71%.
22

---

Published as a conference paper at ICLR 2021
We conjecture that, for these threat models, this is because training against a wider set of perturbations at training time helps generalize robustness to new imputs at test time, even within the same threat model. To test this, additionally train classifiers using adversarial training against the StAdv and ReColorAdy attacks with double the default bound. The results are shown in Table Table helow. We find that, because these classifiers during training, they perform better than PAT against those attacks at test time (76% vs 65% for StAdv and 81 % vs 71% for ReColorAdv).
exposed wider range of spatial and recoloring perturbations
This suggests that PAT not only improves robustness against a wide range of adversarial threat models, it can actually improve robustness over threat-specific adversarial training by incorporating wider range of attacks during training.
Table 9: Results of experiments training against StAdv and ReColorAdv on CIFAR-10 with double the default bounds. Columns are identical to Table
Union
Unseen mean
Narrow threat models
NPTM
Training
Clean
Ligo
L
StAdv
ReColor
PPGD
LPA
AT StAdv
0.0
1.4
86.2
0.1
0.2
53,9
5.1
0.0
0.0
AT StAdv (double e)
0.2
3.7
83.9
0.2
0.5
76.1
13,9
0.0
0.0
AT ReColorAdy
0.0
3.1
93.4
8.5
3.9
0.0
65.0
0.1
0.0
AT ReColorAdy (double c)
0.0
5.3
92.0
12.5
8.6
0.3
81.2
0.4
0.0
PAT-self
21.9
45.6
82,4
30.2
34.9
46.4
71.0
13.1
2.1
PAT-AlexNer
27.8
48.5
71.6
28.7
33.3
64.5
67.5
26.6
9.8
G COMMON CORRUPTIONS EVALUATION
We evaluate the robustness of PAT-trained models to common corruptions addition to adversarial examples on CIFAR-10 and ImageNet-1001 100. In particular, we test PAT-trained classifiers CIFAR 10-C and ImageNet-1000 100-C, where ImageNet-100- 100-C the 100-class subset of ImageNet-C formed by taking every tenth class (Hendrycks and Dietterich 2019). These datasets based on random corruptions of CIFAR-10 and ImageNet, respectively, using 15 perturbation types with levels of severity. The perturbation types split into four general categories: "noise," "blur," "weather," and "digital."
The metric to evaluate PAT against common corruptions is relative corruption error (relative mCE). The relative corruption error is defined by Hendrycks and Dietterich (2019) for classifier and corruption type e as
Σ
sml Ei_- ES
Relative CE
Σ.
AleeNNer AlexNet 5,5
clean
where Ef is the error of classifier f against corruption type at severity level s, and E is the error of classifier f on unperturbed inputs. The relative mCE is defined as the mean relative CE over all perturbation types.
The relative mCE for classifiers trained with normal training, adversarial training, and PAT is shown in Tables 10 and 11 PAT gives better robustness (lower relative mCE) against common corruptions on both CIFAR- I0-C and ImageNet-100-C. The only category of perturbations where L2 adversarial training outperforms PAT is "noise" on CIFAR- 10-C, which makes sense because Gaussian and other types noise are symmetrically distributed in an L2 ball. For the other perturbation types and on ImageNet- 100-- PAT outperforms L2 and Loc adversarial training, indicating that robustness against a wider range of worst-case perturbations also gives robustness against a wider range of random perturbations.
23

---

Published as a conference paper at ICLR 2021
Table 10: Robustness of classifiers trained with adversarial training and PAT against common corruptions in the CIFAR-10-C 10-C dataset. Results are reported relative mCE (lower is better). PAT improves robustness against common corruptions overall and for three of the specific perturbation categories.
Perturbation Type
Training
Noise
Blur
Weather
Digital
All
Normal
1.72
1.57
1.03
1.87
1.59
Loc
0.24
0.46
0.96
0.61
0.57
La
0.16
0.39
1.02
0.61
0.54
PAT-self
0.22
0.30
0.90
0.58
0.50
PAT-AlexNet
0.22
0.34
0.88
0.56
0.49
Table 11: Robustness classifiers trained with adversarial training and PAT against common corruptions in the ImageNet-1066 100-- dataset. Results reported as relative mCE (lower is better).
Perturbation Type
Training
Noise
Blur
Weather
Digital
All
Normal
0.85
0.57
0.56
0.35
0.55
Loc
0.54
0.41
0.62
0.25
0.42
L2
0.41
0.36
0.71
0.27
0.41
PAT-self
0.39
0.36
0.60
0.24
0.37
PAT-AlexNet
0.49
0.35
0.61
0.24
0.39
H EXPERIMENT DETAILS
For all experiments, train ResNet-50 (He al. 2016) with SGD for 100 epochs. We use 10 attack iterations for training and 200 for testing, except for PPGD and LPA, where we use 40 for testing since they are more expensive. Self-bounded PAT takes about 12 hours train for CIFAR-10 on an Nvidia RTX 2080 Ti GPU, and about 5 days to train for ImageNet-100 on 4 GPUs. We implement PPGD, LPA, and PAT using PyTorch (Paszke al. 2017).
We preprocess images after adversarial perturbation, but before classification, by standardizing them based on the mean and standard deviation of each channel for all images the dataset. We use default data augmentation techniques from the robustness library (Engstrom et al.) 2019). The CIFAR-10 10 dataset can be obtained from https: / /www. cs toronto. edu/~krr z. c cifarr
html The ImageNet-100 dataset is subset of the ImageNet Large Scale Visual Recognition Challenge (2012) (Russakovsky et al. 2015) including only every tenth class by WordNet ID order. It can be obtained from pttt::/. wwww://wwww///. Image net. wog..rr///...................... www./oadwwddaoddoooooed images.
24

---

Published as a conference paper at ICLR 2021
Table 12: Hyperparameters for the adversarial training experiments on CIFAR-10 and ImageNet-100.
For CIFAR-10, 10, hyperparameters are similar to those used by Zhang et al. (019a). For ImageNet-100, hyperparameters are similar those used by Kang al. (2019).
Parameter
CIFAR-10
ImageNet-100
Architecture
ResNet-50 23,520,842
ResNet-50
Number of parameters
23,712,932
Optimizer
SGD
SGD
Momentum
0.9
0,9
Weight decay
x 10
10 10-4
Batch size
50
128
Training epochs
100
90
Initial learning rate
0.1
0.1
Learning rate drop epochs (x0.1 drop)
75, 90
30, 60, 80
Attack iterations (train) Attack iterations (test)
10
10
200
200
H.1 LAYERS FOR LPIPS CALCULATION
Calculating the LPIPS distance using a neural network classifier g(.) requires choosing layers whose normalized, flattened activations <(.) should be compared between images. For AlexNet and VGG-16, we use the same layers to calculate LPIPS distance do Zhang al. (2018). For AlexNet (Krizhevsky et al. 2012). we use the activations after each of the first five ReLU functions. For VGG-T6 (Simonyan and Zisserman 2014), we use the activations directly before the five max pooling layers. In ResNet-50, we use the outputs of the conv2 x, conv3 x, cony4 and conv5 layers, as listed in Table 1 of of et al. (2016).
25


Reviews:
[{"review_id": "fku7ZrsAc3q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a threat model called Neural Perceptual Threat Model (NPTM) and under NPTM, they develop novel perceptual adversarial attacks: perceptual Projected Gradient Descent (PPGD) and Lagrangian Perceptual Attack (LPA). Also, they propose Perceptual Adversarial Training (PAT) which achieves good robustness against various types of adversarial attacks and even could generalize well to unforeseen perturbation types.  \n\nI think this is an interesting paper and solves an important problem. The writing is very clear and easy to follow. The main contributions are:\n\n1. Introduce a new threat model called the Neural Perceptual Threat Model (NPTM) based on Learned Perceptual Image Patch Similarity (LPIPS) and propose two perceptual attack methods PPGD and LPA. They perform a study on Amazon Mechanical Turk (AMT) to show that LPIPS correlates well with human judgments across 7 different types of adversarial perturbations and adversarial examples generated by their attacks are imperceptible to humans. Such a study can benefit future research and they promise to release their dataset with annotations. They also perform experiments to show that LPA is by far the strongest adversarial attack at a given level of perceptibility, which could be used to evaluate the robustness of the defenses in the future. \n\n2. Propose Perceptual Adversarial Training (PAT) which achieves state-of-the-art robustness against the union of the five attacks $L_2$, $L_\\infty$, spatial, recoloring, and JPEG, without training against any of them. Being able to generalize well to unforeseen perturbation types is a desirable property for adversarial robustness and previous adversarial defenses don’t have such a property. \n\nHowever, I have some concerns: \n\n1. I notice that the clean accuracy of PAT-self and PAT-AlexNet (in Table 2&3) is lower than other training methods. I think it is due to removing the projection step at the end of the Fast-LPA attack. Could the authors explain why they need to remove the projection step? What are the results without removing the projection step? I think achieving good clean accuracy is also important for an adversarial trained model and PAT should not hurt the clean accuracy much; \n\n2. Could the authors explain why AlexNet is the best proxy for human judgments of perceptual distance? Are there any insights? Also, could the authors provide detailed descriptions about how the AlexNet model is trained? \n\n3. The authors mention that training against multiple threat models simultaneously will result in lower robustness against any one of the threat models when compared to hardening against that threat model alone. But from the experimental results, PAT is also less robust against a threat model compared to the method that trains against that threat model. Could the authors compare PAT to those methods that train against multiple threat models (e.g. [1] and [2])? \n\n4. It would be good that the authors could evaluate PAT on images with common corruptions (e.g. CIFAR-10-C and ImageNet-C proposed in [3]). It can demonstrate that PAT could generalize well to unforeseen perturbation types further.  \n\nIf the authors could address these concerns, I am willing to raise my scores. \n\nMinor things: I find that the clean accuracy of the Normal model is 0 in Table 2. I think it is a typo. Please correct it. \n\n[1] Maini, Pratyush, Eric Wong, and J. Zico Kolter. \"Adversarial robustness against the union of multiple perturbation models.\" arXiv preprint arXiv:1909.04068 (2019).\n\n[2] Tramèr, Florian, and Dan Boneh. \"Adversarial training and robustness for multiple perturbations.\" Advances in Neural Information Processing Systems. 2019.\n\n[3] Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking neural network robustness to common corruptions and perturbations.\" arXiv preprint arXiv:1903.12261 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ufe-gWvDSeh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the adversarial robustness of deep neural networks against multiple and unforeseen threat models. Since there lacks a precise formalization of human perception, this paper adopts LPIPS, a metric that correlates well with human perception based on neural network activations. Then, two adversarial attack methods are proposed to generate adversarial examples under the metric. And an adversarial training method is also proposed. The experiments on various threat models validate the effectiveness of the proposed method.\n\nThe writing of this paper is clear. The generalizability of robust DNN against multiple threat models is an important problem. This paper is a good attempt to solve this problem. Based on a perception similarity metric, new adversarial attacks and defenses are studied. Thus the paper is comprehensive.\n\nThis biggest problem of this paper, in my opinion, is that the adopted metric is defined on neural networks. Although the authors have conducted human evaluations to prove that this metric correlates well with humans, I still doubt whether it can reflect all potential threats. For example, if the metric is perfect, any (adversarial) example, that is perceptually the same as a clean example to humans, could have the similar representation with the clean example when using the network \\phi. If this is true, \\phi itself is a perfectly robust network, and we do not need any adversarial training to robustify another network (i.e., f(x) in this paper). \n\nHowever, defining a perfect metric for adversarial training may be very difficult. And this paper makes a step towards this goal. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8I-FNv3DrZH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a new form of adversarial training, supported by two proposed adversarial attacks based off a perceptual distance. The choice of perceptual distance (LPIPS), is computed by comparing the activations of (possibly different) two neural networks with respect to a pair of inputs. The authors propose two new attacks based off this perceptual distance: PPGD and LPA, as it is distinct from the common choice of L_2 or L_inf. This work claims that performing adversarial training against adversarial examples crafted by the proposed attacks, induces robustness to a wide range of \"narrow\" threat models e.g. L_2, JPEG, L_inf.\nTo show this, the authors perform experiments on CIFAR-10 and ImageNet-100 — with the main results being that adversarial training with PPGD or LPA produces a model with some robustness to all other threat models. This is in contrast to adversarial training with \"narrow\" threat models, which fail to be robust to at least one other threat model in the set. The exception possibly being L_2, which retains some transferability.\n\nThe paper is written well, and is somewhat easy to follow.\n\nOverall I believe this to be a good paper. I appreciate the study of a union of threat models, as in practice, we of course have no guarantee that an adversary will choose to restrict itself to a single threat model. The experimental results are compelling, and match my intuition — namely that adversarial training with the L_2 threat model would produce the most comparable results to a perceptual threat model, when comparing against the union. Moreover, I find it very interesting that while L_2 looks to transfer well to other threat models, it does not transfer comparably well to the perceptual threat model. The authors explain that this is due to the \"broadness\" of the threat model, but this could use some further exposition.\n\nQualitatively, the LPA and PPGD adversarial examples appear distinct from other threat models in terms of the modified features. The LPA adversarial examples in particular to not seem to target local texture patches, which is nice to see.\n\nI also appreciate the addition of less computationally intensive algorithms to support the proposed attacks. PPGD in particular would be very difficult to run, as the jacobian of \\phi(x) can be very large.\n\nI have some questions, in order of importance.\n\n1) Why is AlexNet (externally-bounded) more effective than using the same network to compute LPIPS? This may be because the metric changes as the network trains, but then I would expect the self-bounded training to achieve worse clean accuracy if the metric degraded.\n\n2) What is the reason for believing that the neural perceptual threat model encompasses the L_p and spatial threat models? I see that there is some overlap in terms of how each threat model can transfer to other threat models. But it is not obvious that the \"narrow\" threat models are contained with the perceptual threat model.\n\n3) I think its interesting that PAT-* performs worse on PPGD and LPA, than on other threat models (that presumably the PAT-* model has not seen). This could mean that PPGD and LPA adversarial examples are \"harder\" in some way. Do the authors have more explanation/intuition about this?\n\nMore general comments:\n\nI found parts of the paper somewhat difficult to read due to having to check the appendix often. It would be easier to read if some details like the attack algorithms, and a figure showing the attack process were moved to the main paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "70O0E9B3JsN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n* This paper proposes perceptual adversarial robustness, an adversarial truing against the set of all imperceptible adversarial examples. Through a perceptual study, they approximate human perception with a neural neural - “neural perceptual distance”, and test the robustness against 5 threat models including L2 and Loo showing state-of-the-art robustness even without training on them, showing generality of the model considered. \n\nStrengths: \n* Relevant, challenging problem of interest to the CV and adversarial learning communities\n* Clearly define scope and contributions \n* Reasoning about both new robustness metric and new attacks to this metric\n* Authors plan to release adversarial examples with annotations \n\nWeaknesses:\n* Since it mostly relies on LPIPS, it is unclear how much novelty there is in the approach in the NPTM. \n* The PPGD and LPA attacks completely break the PAT (Perceptual Adversarial Training), so the reader is left with a bit of mixed feelings about lessons learned. I think the authors could have elaborated more on this. \n* Fast-LPA seems to be a relevant contribution but is relegated to the Appendix. \n* Related work is a bit succinct. Since the major proposal is about neural perceptual distance, but they rely on something from the state of the art, although extensively validation, it would be interesting to know more about what are the actual contributions with respect to this part as well. The new attacks against this NPTM are a clear contribution, in addition to validation with mechanical turk of the perceptual similarity. \n\nComments: \n* As a general comment, I believe it would be clearer to report \"attack success rate\" instead of decreased accuracies, although I understand the work is a bit on attacks and a bit on the robustness side. So, to show relevance of adversarial training, eventual model accuracy is okay. \n* It is a bit confusing that in Table 2 and 3, the “Normally” trained model has an Unseen Mean of 1.8% instead of 0%. I eventually understood why, but maybe you should consider commenting on this, as I initially felt like there was something wrong with the Table. \n* The authors emphasize a lot on the NPTM, which is definitely relevant. Nevertheless, the PAT (Perceptual Adversarial Training) seems mostly unsuccessful against the PPGD and LPA (and Fast-LPA) attacks, and this may \n* I am a bit confused by the overall narrative of the work. You first propose the NPTM, and develop the PAT mechanism, and show it generalizes well across different threat models. But then you develop an attack that “knows” the defense - which is great - but then show that it basically defeats PAT (mostly). I think this part of the story is left a bit under-explored, and the reader is missing actionable points from the work, as there is a lot of discussion on robustness but also on attacking this new perceptual models. Do not get me wrong: I think this is highly relevant, I am mostly commenting on the presentation of the results and the overall story and lessons learned. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on the adversarial robustness of deep neural networks against multiple and unforeseen threat models, which proposes a threat model called Neural Perceptual Threat Model (NPTM). The philosophy behind sounds quite interesting to me, namely, approximating human perception with a neural neural "neural perceptual distance". This philosophy leads to a novel algorithm design I have never seen, i.e., Perceptual Adversarial Training (PAT) which achieves good robustness against various types of adversarial attacks and even could generalize well to unforeseen perturbation types. 

The clarity and novelty are clearly above the bar of ICLR. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please carefully address all 
comments in the final version.
Decision: Accept (Poster)