Defending Against Physically Realizable Attacks on Image Classification
Authors: Tong Wu
Liang Tong
Yevgeniy Vorobeychik
Venue:  (2020)
Keywords: defense against physical attacks
adversarial machine learning

Abstract: We study the problem of defending deep neural network approaches for image classification from physically realizable attacks. First, we demonstrate that the two most scalable and effective methods for learning robust models, adversarial training with PGD attacks and randomized smoothing, exhibit very limited effectiveness against three of the highest profile physical attacks. Next, we propose a new abstract adversarial model, rectangular occlusion attacks, in which an adversary places a small adversarially crafted rectangle in an image, and develop two approaches for efficiently computing the resulting adversarial examples. Finally, we demonstrate that adversarial training using our new attack yields image classification models that exhibit high robustness against the physically realizable attacks we study, offering the first effective generic defense against such attacks.

Published as a conference paper at ICLR 2020
DEFENDING AGAINST PHYSICALLY REALIZABLE AT- TACKS ON IMAGE CLASSIFICATION
Tong Wu, Liang Tong & Yevgeniy Vorobeychik Department of Computer Science and Engineering Washington University in St. Louis
{tongwu, liangtong, yvorobaaychikeewustt..coo
ABSTRACT
We study the problem of defending deep neural network approaches for image classification from physically realizable attacks. First, we demonstrate that the most scalable and effective methods for learning robust models, adversarial training with PGD attacks and randomized smoothing, exhibit limited effective- ness against three of the highest profile physical attacks. Next, we propose a new abstract adversarial model, rectangular occlusion attacks, in which an adversary places a small adversarially crafted rectangle an image, and develop two ap- proaches for efficiently computing the resulting adversarial examples. Finally, we demonstrate that adversarial training using our new attack yields image classifica- tion models that exhibit high robustness against the physically realizable attacks we study, offering the first effective generic defense against such attacks.
INTRODUCTION
State-of-the trr effectiveness of deep neural networks has made it the technique of choice in riety of fields, including computer vision (He et al. 2016), natural language processing (Sutskever et al. 2014), and speech recognition (Hinton al. demonstrations showing that deep neural networks can be easily fooled by carefully perturbing pix- els in an image through what have become known adversarial example attacks (Szegedy al.) 2014 Goodfellow et al. 2015 Carlini & Wagner 2017b] Vorobeychik & Kantarcioglu, 2018). 2011)) In response, a large literature has emerged on defending deep neural networks against adversarial exam- ples, typically either proposing techniques for learning more robust neural network models (Wong Kolter 2018, WWon al.) 2018 Raghunathan al. 2018b Cohen al. 2019: Madry et al.,
2012). However, there have been myriad of
2018), or by detecting adversarial inputs (Metzen et al. 2017 Xu et al. 2018).
Particularly concerning. however, have been number of demonstrations that implement adversarial perturbations directly in physical objects that are subsequently captured by camera, and then fed
through the deep neural network classifier (Boloor et al. 2019 Eykholt et al.) 2018 Athalye et al. 2018b Brown al. 2018 Bhattad et al. 2020). Among the most significant of such physical attacks on deep neural networks are three that we specifically consider here: 1) the attack which fools face recognition by using adversarially designed eyeglass frames (Sharif et al. 2016), 2) the attack which fools stop sign classification by adding adversarially crafted stickers (Eykholt et al. (001)) and 3) the universal adversarial patch attack, which causes targeted misclassification of any object with the adversarially designed sticker (patch) (Brown et al. 2018). Oddly, while considerable attention has been devoted defending against adversarial perturbation attacks in the digital space, there no effective methods specifically to defend against such physical attacks.
Our first contribution is an empirical evaluation of the effectiveness of conventional approaches to robust ML against two physically realizable attacks: the eyeglass frame attack on face recogni- tion (Sharif et al. (016) and the sticker attack on stop signs (Eykholt et al.) (018). Specifically, we study the performance on adversarial training and randomized smoothing against these attacks, and show that both have limited effectiveness in this context (quite ineffective some settings, and somewhat more effective, but still not highly robust, in others), despite showing moderate effective- ness against bg and l2 attacks, respectively.
'The code can be found http://////ttttb.coo///o////////

---

Published as a conference paper at ICLR 2020
Our second contribution is novel abstract attack model which more directly captures the nature of common physically realizable attacks than the conventional 1p-based models. Specifically, we consider a simple class of rectangular occlusion attacks in which the attacker places rectangular sticker onto an image, with both the location and the content of the sticker adversarialyy chosen. We develop several algorithms for computing such adversarial occlusions, and use adversarial training to obtain neural network models that are robust to these. We then experimentally demonstrate that our proposed approach is significantly more robust against physical attacks on deep neural networks than adversarial training and randomized smoothing methods that leverage fy-based attack models.
Related Work While many approaches for defending deep learning in vision upplications have been proposed, robust learning methods have been particularly promising, since alternatives are often de- feated soon after being proposed (Madry et al.) 2018, |Rahhunthhnn et al., 2018a| Wong & Kolter 2018 Vorobeychik & Kantarciogluj (018). The standard solution approach for this problem is an adaptation of Stochastic Gradient Descent (SGD) where gradients are either with respect to the loss at the optimal adversarial perturbation for each (or approximation thereof, such as using heuristic local search (Goodiellow et al. [2015] Madry et al., 2018) or a convex over-approximation (Raghu- nathan al. [2018b] Wang al.) 2018)), or with respect the dual of the convex relaxation of the attacker maximization problem (Raghunathan et al. 2018a] Wong & Kolter 2018) Wong et al.) 2018). Despite these advances, adversarial training a la Madry al. (2018) remains the most prac- tically effective method for hardening neural networks against adversarial examples with -norm perturbation constraints. Recently, randomized smoothing emerged as another class of techniques for obtaining robustness (Lecuyer et al. 2019 Cohen et al. (011)), with the strongest results in the context of (2-norm attacks. In addition to training neural networks that are robust by construction, a number of methods study the problem of detecting adversarial examples (Metzen et al. 2017 Xu
al. 2018), with mixed results (Carlini & Wagner 2017a). Of particular interest is recent work on detecting physical adversarial examples (Chou et al. 2018). However, detection is inherently weaker than robustness, which is our goal, as even perfect detection does not resolve the question of how to make decisions adversarial examples. Finally, our work is in the spirit of other recent efforts that characterize robustness of neural networks to physically realistic perturbations, such as translations,
rotations, blurring, and contrast (Engstrom al. 2019 [Hendrycks & Dietterich 2019)
2
BACKGROUND
2.1
ADVERSARIAL EXAMPLES IN THE DIGITAL AND PHYSICAL WORLD
Adversarial examples involve modifications of input images that are either invisible to hu- mans, unsuspicious, and that cause systematic misclassification by state-of-the-art neural net- works (Szegedy et al.] [014] Goodfellow et al. 2013) Vorobeychik & Kantarcioglu 2018). Com- monly, approaches for generating adversarial examples aim to solve an optimization problem of the following form:
arg max L[f(xc + 8) 8.l. ||0||p Se
where is the original input image, 5 is the adversarial perturbation, I-(-) is the adversary's 's utility function (for example, the adversary may wish to maximize the cross-entropy loss), and || Ip is some 1p norm. While a host of such digital attacks have been proposed, two have come be viewed as state of the art: the attack developed by Carlini & Wagner (2017b). and the projected gradient descent attack (PGD) by Madry et al. (2018).
While most of the work to date has been on attacks which modify the digital image directly, we focus on class of physical attacks which entail modifying the actual object being photographed in order to fool the neural network that subsequently takes its digital representation as input. The attacks we will focus on will have three characteristics:
1. The attack can be implemented in the physical space (c.g., modifying the stop sign);
2. the attack has low suspiciousness; this is operationalized by modifying only small part of the object, with the modification similar to common "noise" that obtains in the real world; for example, stickers on a stop sign would appear to most people as vandalism, but covering the stop sign with a printed poster would look highly suspicious; and
3. the attack causes misclassification by state-of-the-art deep neural network.
2

---

Published as a conference paper at ICLR 2020
Since our ultimate purpose is defense, we will not concern ourselves with the issue of actually implementing the physical attacks. Instead, we will consider the digital representation of these attacks, ignoring other important issues, such robustness to many viewpoints and printability. For example, in the case where the attack involves posting stickers on a stop sign, we will only be concerned with simulating such stickers on digital images of stop signs. For this reason, refer to such attacks physically realizable attacks, to allude to the fact that it is possible to realize them in practice. It evident that physically realizable attacks represent somewhat stronger adversarial model than their actual implementation in the physical space. Henceforth, for simplicity, we will use the terms physical attacks and physically realizable attacks interchangeably.
STOP
STOP
(a)
(b)
Figure I: (a) An example of the eyeglass frame attack. Left: original face input image. Middle: modified input image (adversarial eyeglasses superimposed on the face). Right: an image of the predicted individual with the adversarial input in the middle image. (b) An example of the stop sign attack. Left: original stop sign input image. Middle: adversarial mask. Right: stop sign image with adversarial stickers, classified as speed limit sign.
We consider three physically realizable attacks. The first the attack on face recognition by Sharif iet al. (2016), in which the attacker adds ad adversarial noise inside printed eyeglass frames that can subsequently be put on to fool the deep neural network (Figure The second attack posts adver- sarially crafted stickers on stop sign to cause it to be misclassified as another road sign, such as the speed limit sign (Figure [b) (Eykholt et al. 2018). The third, adversarial patch, attack designs patch (a sticker) with adversarial noise that can be placed onto an arbitrary object, causing that object to be misclassified by deep neural network (Brown et al. 2018).
2.2 ADVERSARIALLY ROBUST DEEP LEARNING
While numerous approaches have been proposed for making deep learning robust, many are heuristic and have soon after been defeated by more sophisticated attacks (Carlini & Wagner) 2017b He et al. [017]] Carlini & Wagner 2017a Athalye al. 2018a). Consequently, we focus on principled approaches for defense that have not been broken. These fall broadly into two categories: robust learning and randomized smoothing. We focus on state-of-the-art representative from each class.
Robust Learning The goal of robust learning is to minimize a robust loss, defined as follows:
0" = arg min
max T(f((x +000,
(m.g)~~ S1
where 'D denotes the training data set. In itself this is highly intractable problem. Several tech- niques have been developed to obtain approximate solutions. Among the most effective in practice the adversarial training approach by |Madry et al. (2018), who use the PGD attack as an approx- imation to the inner optimization problem, and then take gradient descent steps with respect to the associated adversarial inputs. In addition. we consider a modified version of this approach termed curriculum adversarial training (Cai et al.] 2018). Our implementation of this approach proceeds as follows: first, apply adversarial training for a small e, then increase and repeat adversarial training, and so on, increasing until we reach the desired level of adversarial noise we wish to be robust to.
Randomized Smoothing The second class of techniques we consider works by adding noise to inputs at both training and prediction time. The key idea is to construct smoothed classifier g(-) from a base classifier f()) by perturbing the input æ with isotropic Gaussian noise with variance or. The prediction is then made by choosing class with the highest probability measure with respect to the induced distribution f(-) decisions:
g (x) = arg max P( (x+ + o)) = =),, a a~ (0,000,
(3)
3

---

Published as a conference paper at ICLR 2020
To achieve provably robust classification in this manner typically trains the classifier f()) by
adding Gaussian noise to inputs at training time (Lecuyer et al. 2019 Cohen et al. 2019).
ROBUSTNESS OF CONVENTIONAL ROBUST ML METHODS AGAINST PHYSICAL ATTACKS
Most of the approaches for endowing deep learning with adversarial robustness focus on adversarial models in which the attacker introduces 1y-bounded adversarial perturbations over the entire input. Earlier we described two representative approaches in this vein: adversarial training, commonly focused on robustness against (x attacks, and randomized smoothing, which is most effective against attacks (although certification bounds can be extended other ip norms as well). We call these methods conventional robust MI.
In this section, we ask the following question:
Are conventional robust ML methods robust against physically realizable attacks?
This is similar to the question was asked in the context of malware classifier evasion by Tong et al. 2019), who found that l 1p-based robust ML methods can indeed be successful in achieving robust- ness against realizable evasion attacks. Ours is the first investigation of this issue in computer vision applications and for deep neural networks, where attacks involve adversarial masking of objects P
We study this issue experimentally by considering two state-of-the-ar approaches for robust MI:: adversarial training a-la-IMadry al. (2018), along with its curriculum learning variation (Cai et al.) 2018), and randomized smoothing, using the implementation by Cohen et al. (2019). These ap- proaches are applied to defend against two physically realizable attacks described in Section 2. an attack on face recognition which adds adversarial eyeglass frames to faces (Sharif et al.] 2016). and attack on stop sign classification which adds adversarial stickers to stop sign to cause misclassification (Eykholt et al., 2018).
We consider several variations of adversarial training. function of the 1x bound, e, imposed on the adversary. Just as Madry et al. (2018), adversarial instances in adversarial training were generated using PGD. We consider attacks with (4,8) (adversarial training failed to make progress when we used e = 16). For curriculum adversarial training, we first performed adversarial training with 4, then doubled
and repeated adversarial training with the model robust to = 4, then doubled again, and so on. In the end, we learned models for E {4, 8, 16, 32}. For all versions adversarial training, we consider and 50 iterations of the PGD attack. We used the learning rate of e/4 for the former and 1 for the latter. In all cases, pixels are in 0 ~ 255 range and retraining was performed for 30 epochs using the ADAM optimizer.
For randomized smoothing, we consider noise levels a {0.25, 0.5, 1) in Cohen et al. 2019
and take 1,000 Monte Carlo samples test time.
3.1 ADVERSARIAL EYEGLASSES IN FACE RECOGNITION
We applied white-box dodging (untargeted) attacks the face recognition systems (FRS) from Sharif et al. 2016)). We used both the VGGFace data and transferred VGGFace CNN model for the face recognition task, subselecting 10 individuals, with 300-500 face images for each. Further details about the dataset, CNN architecture, and training procedure are in Appendix [A] For the attack, we used identical frames in Sharif al. (2016) occupying 6.5% of the pixels. Just as Sharif et al. (0066). we compute attacks (that is, adversarial perturbations inside the eyeglass frame area) by using the learning rate 20 well momentum value 0.4, and vary the number of attack iterations between 0 (no attack) and 300.
Figure FigureZ presents the results of classifiers obtained from adversarial training (left) as well as curricu- lum adversarial training (middle). in terms of accuracy (after the attack) function of the number of iterations of the (hhaii et al.) (2016) eyeglass frame attack. First, it is clear that none of the varia- tions of adversarial training are particularly effective once the number of physical attack iterations is
SSeerall related efforts study robustness of deep neural networks to other variants of physically realistic perturbations (Engstrom al. 2019 Hendrycks & Dietterich 2019).
4

---

Published as a conference paper at ICLR 2020
100
Odigina Mode!
Origiz
Accurscy(t)
80
Adw. Traniing
Accuracy(9)
80
CAdv. Trairing:
Traliing c
Accuracyth))
80
56 tare ndv. 2nd
CAdv. Trafiing am
Bardenited Smoothing
CAdv. Trairing:
40
20
20
0
Iterationn
Figure 2: Performance of adversarial training (left), curriculum adversarial training (with 7 PGD iterations) (middle), and randomized smoothing (right) against the eyeglass frame attack.
above 20. The best performance in terms of adversarial robustness is achieved by adversarial train- ing with 8, for approaches using either 7 or 50 PGD iterations (the difference between these appears negligible). However, non-adversarial accuracy for these models is below 70%, a ~20% drop in accuracy compared the original model. Moreover, adversarial accuracy is under 40% for sufficiently strong physical attacks, Curriculum adversarial training generally achieves significantly higher non-adversarial accuracy, but is far less robust, even when trained with PGD attacks that use c 32.
Figure 2 (right) shows the performance of randomized smoothing when faced with the eyeglass frames attack. It is readily apparent that randomized smoothing is ineffective at deflecting this physical attack: even as we vary the amount of noise we add, accuracy after attacks is below 20% even for relatively weak attacks, and often drops to nearly 0 for sufficiently strong attacks.
3.2 ADVERSARIAL STICKERS ON STOP SIGNS
Following Eykholt et al. 2018), we use the LISA traffic sign dataset for our experiments, and 40 stop signs from this dataset as our test data and perform untargeted attacks (this is in contrast the original work, which is focused on targeted attacks). For the detailed description of the data and the CNN used for traffic sign prediction, see Appendix[A] We apply the same settings as the original attacks and use ADAM optimizer with the same parameters. Since we observed few differences in performance between running PGD for 7 vs. iterations, adversarial training methods this section all use 7 iterations of PGD.
Model Adv. Traising: and Taaling: caB Talking:
Clean Mode
GZ7 Clean Mode omo
Cidv. Training:
Accuracy(%)
80
Accuracy(58)
80
Cydv Triiing:
Accuracy(%%) 40
80
40
40
20
Allack Iterations
Attack Iterations
Atlack Iterations
Figure 3: Performance of adversarial training (left), curriculum adversarial training (with 7 PGD iterations) (middle), and randomized smoothing (right) against the stop sign attack.
Again, we begin by considering adversarial training (Figure 3 left and middle). In this case, both the original and curriculum versions of adversarial training with PGD are ineffective when 32 (error rates on clean data are above 90%): these are consequently omitted from the plots. Curriculum adversarial training with = 16 has the best performance adversarial data, and works well on clean data. Surprisingly, most variants of adversarial training perform at best marginally better than the original model against the stop sign attack. Even the best variant has relatively poor performance, with robust accuracy under 50% for stronger attacks.
Figure (right) presents the results for randomized smoothing. In this set of experiments, we found that randomized smoothing performs inconsistently. To address this, we used 5 random seeds to repeat the experiments, and use the resulting mean values in the final results. Here, the best variant
5

---

Published as a conference paper at ICLR 2020
uses 0.25, and, unlike experiments with the eyeglass frame attack, significantly outperforms adversarial training, reaching accuracy slightly above 60% even for the stronger attacks. Neverthe- less, even randomized smoothing results in significant degradation of effectiveness adversarial instances (nearly 40%, compared to clean data).
3.3 DISCUSSION
There are two possible reasons why conventional robust ML perform poorly against physical at- tacks: 1) adversarial models involving lp-bounded perturbations are too hard to enable effective robust learning, and 2) the conventional attack model too much of mismatch for realistic phys- ical attacks. In Appendix B we present evidence supporting the latter. Specifically, we find that conventional robust ML models exhibit much higher robustness when faced with the bp-bounded attacks they are trained to be robust to.
4 PROPOSED APPROACH: DEFENSE AGAINST OCCLUSION ATTACKS (DOA)
As we observed in Section conventional models for making deep learning robust to attack can per- form quite poorly when confronted with physically realizable attacks. In other words, the evidence strongly suggests that the conventional models of attacks in which attackers perturbations to input images are not particularly useful one is concerned with the main physical threats that are likely to be faced in practice. However, given the diversity of possible physical at- tacks one may perpetrate, is it even possible to have a meaningful approach for ensuring robustness against a broad range of physical attacks? For example, the attacks we considered far couldn't be more dissimilar: in one, we engineer eyeglass frames; in another, stickers on a stop sign. We ob- serve that the key common element in these attacks, and many other physical attacks we may expect to encounter, is that they involve the introduction of adversarial occlusions to part of the input. The common constraint faced in such attacks is avoid being suspicious, which effectively limits the size of the adversarial occlusion, but not necessarily its shape or location. Next, we introduce simple abstract model of occlusion attacks, and then discuss how such attacks can be computed and how make classifiers robust to them.
make 1p lp-bounded
4.1 ABSTRACT ATTACK MODEL: RECTANGULAR OCCLUSION ATTACKS (ROA)
We propose the following simple abstract model of adversarial occlusions of input images. The attacker introduces fixed-dimension rectangle. This rectangle be placed by the adversary any- where in the image, and the attacker can furthermore introduce Loo noise inside the rectangle with an exogenously specified high bound e (for example, e 255, which effectively allows addition of arbitrary adversarial noise). This model bears some similarity to lo attacks, but the rectangle imposes contiguity constraint, which reflects common physical limitations. The model clearly abstract: in practice, for example, adversarial occlusions need not be rectangular or have fixed di- mensions (for example, the eyeglass frame attack clearly not rectangular), but at the same time cannot usually be arbitrarily superimposed on image, as they are implemented in the physical environment. Nevertheless, the model reflects some of the most important aspects common to many physical attacks, such stickers placed to identify. We call our attack model a rectangular occlusion attack (ROA). An important feature of this attack is that it is untargeted: since ultimate goal is defend against physical attacks whatever their target, considering untargeted attacks obviates the need to have precise knowledge about the attacker's goals. For illustrations of the ROA attack, see Appendix C
adversarially chosen portion of the object
wish
4.2 COMPUTING ATTACKS
The computation of ROA attacks involves 1) identifying region to place the rectangle in the image, and 2) generating fine-grained adversarial perturbations restricted to this region. The former task can be done by exhaustive search: consider all possible locations for the upper left-hand corner of the rectangle, compute adversarial noise inside the rectangle using PGD for each of these, and choose the worst-case attack (i.e., the attack which maximizes loss computed the resulting image). However, this approach would be quite slow, since we need to perform PGD inside the rectangle for every possible position. Our approach, consequently, decouples these two tasks. Specifically, first

---

Published as a conference paper at ICLR 2020
perform an exhaustive search using grey rectangle to find position for it that maximizes loss, and then fix the position and apply PGD inside the rectangle.
An important limitation of the exhaustive search approach for ROA location is that it necessitates computations the loss function for every possible location, which itself requires full forward propagation each time. Thus, the search itself is still relatively slow. To speed the process up further, we use the gradient of the input image to identify candidate locations. Specifically, we select subset of C' locations for the sticker with the highest magnitude of the gradient, and only exhaustively search among these C' locations. C' is exogenously specified to be small relative to the number of pixels in the image, which significantly limits the number of loss function evaluations. Full details of our algorithms for computing ROA are provided in Appendix
4.3 DEFENDING AGAINST ROA
Once we are able to compute the ROA attack, we apply the standard adversarial training approach for defense. We term the resulting classifiers robust to our abstract adversarial occlusion attacks Defense against Occlusion Attacks (DOA), and propose these as an alternative to conventional robust ML for defending against physical attacks. As we will see presently, this defense against ROA is quite adequate for purposes.
EFFECTIVENESS OF DOA AGAINST PHYSICALLY REALIZABLE ATTACKS
We now evaluate the effectiveness of DOA that is, adversarial training using the ROA threat model we introduced-againstt physically realizable attacks (see Appendix G for some examples that defeat conventional methods but not DOA). Recall that we consider only digital representations of the cor- responding physical attacks. Consequently, we can view our results in this section as lower bound on robustness to actual physical attacks, which have to deal with additional practical constraints, such being robust to multiple viewpoints. In addition to the two physical attacks we previously considered, we also evaluate DOA against the adversarial patch attack, implemented on both face recognition and traffic sign data.
5.1 DOA AGAINST ADVERSARIAL EYEGLASSES
We consider two rectangle dimensions resulting in comparable area: 100 x 50 and 70 70, both in pixels. Thus, the rectangles occupy approximately 10% of the 224 x 224 face images. We used (30,50) iterations of PGD with e 255/2 generate adversarial noise inside the rectangle, and with learning rate {8, 4] correspondingly. For the gradient version of ROA. we choose C' = 30. DOA adversarial training is performed for epochs with a learning rate of 0.0001.
100
100
100
80
80
60
AccuracyEMi
Driginal Motel
DOASUN!
AccuracyENN
Accuracy(Ni
60
60
60
40
40
40
105450 find. Boond Searc 635RGh
20
20
632PSD
20
70x70 Grad. Basss Search 630260
105+50 Gredi Seaed Searcy 623FC0
10
Search 450103
0
100
150 200 Iterations
250
300
50
100
150
200
250
300
50
100
150 200 250 300 # Iterations
Iterations
Figure 4: Performance of DOA (using the 100 x 50 rectangle) against the eyeglass frame attack in comparison with conventional methods. Left: comparison between DOA, adversarial training. and randomized smoothing (using the most robust variants of these). Middle/Right: Comparing DOA performance for different rectangle dimensions and numbers of PGD iterations inside the rectangle. Middle: using exhaustive search for ROA; right: using the gradient-based heuristic for ROA.
Figure (left) presents the results comparing the effectiveness of DOA against the eyeglass frame attack on face recognition adversarial training and randomized smoothing (we took the most robust variants of both of these). We can see that DOA yields significantly more robust classifiers for this domain. The gradient-based heuristic does come some cost, with performance slightly worse

---

Published as a conference paper at ICLR 2020
than when exhaustive search, but this performance drop is relatively small, and the result is still far better than conventional robust ML approaches. Figure 4 (middle and right) compares the performance of DOA between two rectangle variants with different dimensions. The key observation that as long as we use enough iterations of PGD inside the rectangle, changing its dimensions (keeping the area roughly constant) appears to have minimal impact.
5.2 DOA AGAINST THE STOP SIGN ATTACK
We now repeat the evaluation with the traffic sign data and the stop sign attack. In this case, we used 10 5 and 7 rectangles covering ~5 % of the 32 32 images. We set C 10 for the gradient- based ROA. Implementation of DOA is otherwise identical as in the face recognition experiments above.
We present our results using the square rectangle, which in this case was significantly more effec- tive; the results for the the effectiveness of DOA against the stop sign attack on traffic sign data with the best variants of adversarial training and randomized smoothing. Our results here are for 30 iterations of PGD; in Ap- pendix [P. we study the impact of varying the number of PGD iterations. We can observe that DOA is again significantly more robust, with robust accuracy over 90% for the exhaustive search variant, and ~85% for the gradient-baid based variant, even for stronger attacks. Moreover, DOA remains 100% effective at classifying stop signs on clean data, and exhibits ~95% accuracy on the full traffic sign classification task.
5 rectangle DOA attacks are in Appendix F Figure
compares
DGS Oean ReEd
DORTSN SERD
2G9 Ceaa
CAN loking p-is
Figure 5: Performance of the best case of adversarial training, randomized smoothing, and DOA against the stop sign attack (left) and adversarial patch attack (right).
5.3 DOA AGAINST ADVERSARIAL PATCH ATTACKS
Finally, we evaluate DOA against the adversarial patch attacks. In these attacks, adversarial patch (e.g., sticker) is designed to be placed on an object with the goal of inducing a target prediction. We study this in both face recognition and traffic sign classification tasks. Here, we present the results for face recognition; further detailed results on both datasets are provided in Appendix F
As we can see from Figure 5 (right), adversarial patch attacks are quite effective once the attack region (fraction of the image) is 10% or higher, with adversarial training and randomized smoothing both performing rather poorly. In contrast, DOA remains highly robust even when the adversarial patch covers 20% of the image.
CONCLUSION
As we have shown, conventional methods for making deep learning approaches for image classi- fication robust to physically realizable attacks tend to be relatively ineffective. In contrast, a new threat model we proposed, rectangular occlusion attacks (ROA), coupled with adversarial training, achieves high robustness against several prominent examples of physical attacks. While we explored number of variations of ROA attacks as means to achieve robustness against physical attacks, numerous questions remain. For example, can we develop effective methods to certify robustness against ROA, and are the resulting approaches effective in practice combination of heuristically computed attacks and adversarial training? Are there other types of occlusions that more effective? Answers these and related questions may prove promising
method based
8

---

Published as a conference paper at ICLR 2020
path towards practical robustness of deep learning when deployed for downstream applications of computer vision such as autonomous driving and face recognition.
ACKNOWLEDGMENTS
This work was partially supported by the NSF (1S5-195558, IIS- 190-2003,77) ARO INF-I-9- 0241), and NVIDIA.
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of se- curity: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In ICML, 2018b.
Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and David Forsyth. Unrestricted adversarial examples via semantic manipulation. In International Conference on Learning Representations, 2020.
Segee=ySyeeeeeWWeeeeee WH
Adith Boloor, Xin He, Christopher Gill, Yevgeniy Vorobeychik, and Xuan Zhang. Simple physi- cal adversarial examples against end-to-end autonomous driving models. In IEEE International Conference Embedded Software and Systems, 2019.
Tom Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. CoRR, abs/1712.096655, 2018.
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In International Joint Conference on Artificial Intelligence, pp. 3740-3747, 2018.
N. Carlini and D. Wagner. Adversarial examples not casily detected: Bypassing ten detection methods. In International Workshop on Al and Security, 2017a.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. IEEE Symposium on Security and Privacy, pp. 39-57, 2017b.
Edward Chou, Florian Tramer, and Giancarlo Pellegrino. Sentinet: Detecting physical attacks against deep learning systems. 2018. arxiv preprint.
Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via random- ized smoothing. In International Conference on Machine Learning, 2019.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. In International Conference on Machine Learning, pp. 1802-1811. 2019.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Xiaodong Song. Robust physical-worid attacks on deep learning visual classification. IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 1625-1634, 2018
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference Computer Vision and Patlern Recognition (CVPR), pp. 770- 778. 2016.
Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defenses: ensembles of weak defenses are not strong. In USENIX Workshop on Offensive Tech- nologies, 2017.

---

Published as a conference paper at ICLR 2020
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2019.
Geoffrey Hinton, Li Deng, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Vincent Van- houcke, and Tara N. Sainath. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):22977, 2012.
Zico Kolter and Aleksander Madry. https://ddverraaaaall--tutttoold
Adversarial robustness theory and practice 2019.
M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In International Conference on Learning Representations, 2017.
Andreas Møgelmose, Mohan Manubhai Trivedi, and Thomas B. Moeslund. Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey. IEEE Transactions on Intelligent Transportation Systems, 13:1484-1497, 2012.
Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, 7.. Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adversarial settings. 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372-387, 2015.
Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC, 2015.
Aditi Raghumathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial exam- ples. In International Conference Learning Representations, 2018a.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Semidefinite relaxations for certifying bustness to adversarial examples. In Neural Information Processing Systems, 2018b.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to crime: Real and stealthy attacks state-of-the-art face recognition. In ACM SIGSAC Conference Computer and Communications Security, pp. 1528-1540, 2016.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS. 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Liang Tong, Bo Li, Chen Hajaj. Chaowei Xiao, Ning Zhang, and Yevgeniy Vorobeychik. Improv- ing robustness of ml classifiers against realizable evasion attacks using conserved features. In USENIX Security Symposium, pp. 285-302, 2019.
Yevgeniy Vorobeychik and Murat Kantarcioglu. Adversarial Machine Learning. Morgan Claypool, 2018.
S. Wang, K. Pei, J. Whitehouse, J. Yang. and S. Jana. Efficient formal safety analysis of neural networks. In Neural Information Processing Systems, 2018
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference Machine Learning, 2018.
Eric Wong, Frank Schmidt. Jan Hendrik Metzen, and J. Zico Kolter. Scaling provable adversarial defenses. In Newral Information Processing Systems, 2018.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep
neural networks. In Network and Distributed Systems Security Symposium, 2018.
10

---

Published as a conference paper at ICLR 2020
A DESCRIPTION OF DATASETS AND DEEP LEARNING CLASSIHIERS
A.1 FACE RECOGNITION
The VGGFace datase Parkhi al. 2015) is a benchmark for face recognition, containing 2622 subjusts with 2.6 million images in total. We chose ten subjects: A. J. Buckley, A. R. Rahman, Aamir Khan, Aaron Staton, Aaron Tveit, Aaron Yoo, Abbie Cornish, Abel Ferrara, Abigail Bres- lin, and Abigail Spencer, and subselected face images pertaining only to these individuals. Since approximately half of the images cannot be downloaded, our final dataset contains 300-500 images for each subject.
We used the standard corp-and-resize method to process the data to be 224 x 224 pixels, and split the dataset into training, validation, and test according to 7:2:1 ratio for each subject. In total, the data set has 3178 images in the training set, 922 images in the validation set, and 470 images in the test set.
We use the VGGFace convolutional neural network (Parkhi et al. 2013) model, a variant of the VGG16 model containing 5 convolutional layer blocks and fully connected layers. We make use of standard transfer learning as we only classify 10 subjects, keeping the convolutional layers same as VGGFace structure but changing the fully connected layer be 1024 - 1024 -,10 instead of 4096 - 4096 -2622. Specifically, in our Pytorch implementation, we convert the images from RGB BGR channel orders and subtract the mean value [129. 1863, 104.7624, 93.5940] in order to use the pretrained weights from VGG-Face on convolutional layers. We set the batch size to be 64 and use Pytorch built-in Adam Optimizer with an initial learning rate of 10-4 and default parameters in Pytorch We drop the learning rate by 0.1 every 10 epochs. Additionally, we used validation set accuracy to keep track of model performance and choose model in case of overfitting. After 30 epochs of training, the model successfully obtains 98.94 % on test data.
A.2 TRAFFIC SIGN CLASSIFICATION
To be consistent with (Eykholt et al. 2018). we select the subset of LISA which contains 47 different U.S. traffic signs MMgellose et 2012). To alleviate the problem of imbalance and extremely blurry data, we picked 16 best quality signs with 3509 training and 1 148 validation data points. From the validation data, we obtain the test data that includes only 40 stop signs evaluate performance with respect to the stop sign attack, as done by EEkhhlt et al. (2018). In the main body of the paper, we present results only on this test data evaluate robustness to stop sign attacks. the appendix below, we also include performance on the full validation set without adversarial manipulation.
All the data was processed by standard crop-and-resize to 32 32 pixels. We use the LISA-CNN ar- chitecture defined in (Eykholt et al. 2018), and construct a convolutional neural network containing three convolutional layers and one tully connected layer. We use the Adam Optimizer with initial learning rate of 10 and default parameters { dropping the learning rate by 0.1 every 10 epochs. We set the batch size to be 128. After 30 epochs, we achieve the 98.69 % accuracy on the validation set, and 100% accuracy in identifying the stop signs in our test data.
hhtpp://ww/ htttp://ww.robbts... robots. ox..:oo..cc...oo....... wwwpossooo......kkug//dddde.edd.
face/ Fxternal code that we use for transfering VGG-Face Pytorch Framework is available at https: //
git iithub.
conub.com//rr hoom/prlae pttp://1177777/ 277/
Default Pytorch Adam parameters stand for 33 =0.9, 91=0.999 and e=10-

---

Published as a conference paper at ICLR 2020
B EFFECTIVENESS OF CONVENTIONAL ROBUST ML METHODS AGAINST 7, AND l2 ATTACKS
In this appendix, we show that adversarial training and randomized smoothing degrade more grace- fully when faced with attacks that they designed for. In particular, consider here variants of projected gradient descent (PGD) for both the loc and l2 attacks Madry et al. (018). In partieular, the form of PGD for the lo attack is
1++1 (roj(#:+ osgnn (L(,;; 0)))
where Proj is projection operator which clips the result be feasible, 2; the adversarial example in iteration t, a the learning rate, and L(-) the loss function. In the case of an la attack, PGD becomes
2++1 = Proj
Iy +
[V
where the projection operator normalizes the perturbation 5 21+1 2t to have ||ō||2 if doesn't already Kolter & Madry (2019).
The experiments were done the face recognition and traffic sign datasets, but unlike physical attacks on stop signs, we now consider adversarial perturbations to all sign images.
B. FACE RECOGNITION
Table 1: Curriculum Adversarial Training against 7 Iterations Lox Attacks on Face Recognition
Attack Strength
0
= 2
6 8 16
Clean Model
98.94 %
57.87%
13.62%
0%
0%
CAdv. Training: 97.45%
94.68% %
87.02%
65.11%
17.23%
CAdv. Training: == 96.17%
93.40%
89.36%
75.53%
31.49%
CAdv. Training: e 90.64%
88.09%
84.04%
75.96%
45.74%
CAdv. Training: == 32
80.85%
76.60%
74.04%
65.10%
47.87%
Table 2: Curriculum Adversarial Training against 20 Iterations Loc Attacks on Face Recognition
Attack Strength
0
6
8
16
Clean Model
98.94% %
44.04%
1.70%
0%
0%
CAdv. Training: 97.45%
94.68%
85.74%
46.60%
5.11%
CAdv. Training: c 06.17%
93.19%
88.94%
69.36%
9.57%
CAdv. Training: 16
90.64%
88.08%
83.83%
73.62%
35.96%
CAdv. Training: c 32
80.85%
76.17%
74.04%
63.82%
43.62%
We begin with our results the face recognition dataset. Tables and results for (curricu- lum) adversarial training for varying of the loc attacks, separately for training and evaluation. As see, curriculum adversarial training with = 16 is generally the most robust, and remains reasonably effective for relatively large perturbations. However, we do observe a clear tradeoff between accuracy non-adversarial data and robustness, as would expect.
12

---

Published as a conference paper at ICLR 2020
Table 3: Randomized Smoothing against 20 Iterations I/
Attacks on Face Recognition
Attack Strength
0
0.5
1.5
2
2.5
= 3
Clean Model
98.94% %
93.19%
70.85%
44.68%
22.13%
8.29%
3.19%
RS: 0 0.25
%
97.23% %
95.53% %
91.70%
81.06%
67.87%
52.97%
RS: 0.5
97.65%
94.25%
93.61%
91.70%
87.87%
82.55%
71.70%
RS: a 1
92.97%
93.19%
91.70%
91.06%
88.51 %
85.53%
82.98%
Table 3 presents the results of using randomized smoothing on face recognition data, when facing the la attacks. Again, we observe high level of robustness and, in most cases, relatively limited drop in performance, with o = 0.5 perhaps striking the best balance.
B.2 TRAFFIC SIGN CLASSIFICATION
Table 4: Curriculum Adversarial Training against 7 Iterations Lo Attacks on Traffic Signs
Attack Strength
0
c
c= 8
16
Clean Model
98. 08.69%
90.24%
65.24%
33.80%
10.10%
CAdv. Training: 99.13%
97.13%
93.47%
63.85%
20.73%
CAdv. Training: c == 98.72%
96.86%
93.90%
81.70%
38.24%
CAdv. Training: 16
96.95%
95.03%
92.77%
87.63%
64.02%
CAdv. Training: 6 32
65.63%
53.83%
50.87%
46.69%
38.07%
Table 5: Curriculum Adversarial Training against 20 Iterations Lxp Attacks on Traffic Signs
Attack Strength
c= 0
c=
2
c=
c=8 8
c = 16
Clean Model
98.69%
89.54%
61.58%
24.65%
5.14%
CAdv. Training: 99.13%
96.95%
91.90%
56.53%
12.02%
CAdv. Training: c 98.72%
96.68%
93.64%
76.22%
28.13%
CAdv. Training: 6 96.95%
95.03%
92.51%
86.76%
54.01 %
CAdv. Training: 32
65.63%
53.83%
50.78%
46.08%
36.49%
Tables 4 and plresent evaluation on traffic sign data for curriculum adversarial training against the Too attack for varying As with face recognition data, can observe that the approaches tend to be relatively robust, and effective on non-adversarial data for adversarial training methods using 32.
Table 6: Randomized Smoothing against 20 Iterations L2 Attacks on Traffic Signs
Attack Strength
0
0.5
6=1
1.5
2
2.5
3
Clean Model
98.69 %
61.67%
25.78%
11.50%
7.84%
4.97%
3.57%
RS: a 0.25 98.22%
89.08%
55.69%
34.06%
23.46%
18.61%
14.75%
RS: = 0.5
96.28%
90.80% %
76.13%
52.64%
35.31%
23.43%
16.52%
RS: a =1 88.21%
83.68%
75.49%
64.90%
50.03%
36.53%
26.22%
The results randomized smoothing on traffic sign data are given in Table 6 Since images are smaller here than in VGGFace, lower values of for the la attacks meaningful, and for < 1
13

---

Published as a conference paper at ICLR 2020
generally robust performance randomized smoothing, with 0.5 providing good balance between non non-ad adversarial accuracy and robustness, just as before.
C EXAMPLES OF ROA
Figure 6: Examples of the ROA attack on face recognition, using a rectangle of size 100 50. (a) Left: the original A. J. Buckley's s image. Middle: modified input image (ROA superimposed on the face). Right: an image of the predicted individual who Aaron Tveit with the adversarial imput in the middle image. (b) Left: the original Abigail Spencer's image. Middle: modified imput image (ROA superimposed on the face). Right: an image of the predicted individual who is Aaron Yoo with the adversarial input in the middle image.
B several examples of the ROA attack in the context of face recognition. Note that in these examples, the adversaries choose to occlude the noise on upper lip and eye areas of the image, and, indeed, this makes the face more challenging recognize even to human observer.
SPEED LIMIT 30
STOP
STOP
45
41
(a)
(b)
Figure 7: Examples of the ROA attack on traffic sign, using rectangle of size 7 7. (a) Left: the
original Speedlimit45 sign. Middle: modified input image (ROA superimposed on the sign). Right: an image the predicted which is Speedlimit30 with the adversarial input Left: the original Stop sign. Middle: modified input image (ROA superimposed on the sign). Right: an image of the predicted Yield sign with the adversarial input in the middle image.
middle image. (b)
D DETAILED DESCRIPTION OF THE ALGORITHMS FOR COMPUTING THE RECTANGULAR OCCLUSION ATTACKS
Our basic algorithm for computing rectangular occlusion attacks (ROA) proceeds through the fol- lowing two steps:
1. Iterate through possible positions for the rectangle's upper left-hand corner point in the image. Find the position for grey rectangle (RGB value -1127.5,127.5.127.35) in the image that maximizes loss.
2. Generate high-e (o0 noise inside the rectangle the position computed in step
Algorithm presents the full algorithm for identifying the ROA position, which amounts to exhaus- tive search through the image pixel region. This algorithm has several parameters. First, we assume that images are squares with dimensions 22, Second, we introduce stride parameter S. The pur- pose of this parameter to make location computation faster by only considering every other Sth pixel during the search (in other words, skip S pixels each time). For implementation of ROA attacks, we choose the stride parameter = 5 for face recognition and S = 2 for traffic sign classification.
14

---

Published as conference paper at ICLR 2020
Algorithm Computation of ROA position using exhaustive search.
Input:
Data: X1,3; yi:
Test data shape: N x N; Target model parameters: 0: Stride: S
Output:
ROA Position: (J'
l.function SahaustivSeurchhnggMoleee, X Us. N.
2. for in (ange(N/S) do:
for k in range(N/S) do:
4. Generate the adversarial X:du image by:
place a grey rectangle onto the image with top-left corner at (j x S, x S);
6. if' L(X, :y:, 0) is higher than previous loss: Update (j',k') k) = (j, k)
7.
8.
end for end for
9.
10.
return k)),
Algorithm 2 Computation of ROA position using gradient-basee based search.
Input:
Data: X,, Mi
Test data shape: N x N;
Target Model: 8: Stride: S
Number of Potential Candidates: C:
Output:
Best Sticker Position:
(j', k')
N, N, 2. Calculate the gradient V L of Loss(X, Vi, 0) w.r.t. X;
I.function GradientBasedSeacchk..
J, K HelperSearching( 4. for j, in J, IK do:
N, C)
Generate the adversarial Xıdo image by: 6. put the sticker the image where top-left corner at (j 8, S); Loss(XX (asss(Xaav, ys, 0) is higher than previous loss:
8. Update (j'Kk) kj.k)
9. end for
10. return (j', k')
l.function HelperSearching LV,
2. for in ra))e(N/S)) do: for k in range(N/S)) do: 4. Calculate the Sensitivity value I. Σ Eigrectangle (VL,)² where top-left corner at (j x
S,k k: S);
the Sensitivity value is in top C of previous values: 7. Put (j, in J, K and discard (ja: k,) with lowest L
end for end for
10.
return J.
Despite introducing the tunable stride parameter, the search for the best location for ROA still entails a large number of loss function evaluations, which are somewhat costly (since each such evaluation means a full forward pass through the deep neural network), and these costs add up quickly. To speed things up, we consider using the magnitude of the gradient of the loss as a measure of sensitivity of particular regions to manipulation. Specifically, suppose that we compute gradient V L., and let L; be the gradient value for a particular pixel i the image. Now, we can iterate over the possible ROA locations, but for each location compute the gradient of the loss at that location corresponding to the rectangular region. We do this by adding squared gradient values (VL;)² over pixels in the rectangle. We this approach to find the top C' candidate locations for the rectangle. Finally, consider each of these, computing the actual loss for each location, to find the position of ROA. The full algorithm is provided Al Algorithm
15

---

Published as a conference paper at ICLR 2020
Once we' ve found the place for the rectangle, next step to introduce adversarial noise inside it. For this, we use the le version of the PGD attack, restricting perturbations to the rectangle. We used {7, 20, 30, 50} iterations of PGD generate adversarial noise inside the rectangle, and with learning rate = (32, 16, 8, 4) correspondingly.
Original image
Grad. plot
Grad. searching
Exh. searching
Figure 8: Examples of different search techniques. From left to right: 1) the original input image, 2) the plot input gradient, 3) face with ROA location identified using gradient-basee based search, 4) face with ROA location identified using exhaustive search. Each row is different example.
Figure offers visual illustration of how gradient- based search compares exhaustive search for computing ROA.
DETAILS OF PHYSICALLY REALIZABLE ATTACKS
Physically realizable attacks that we study have a common feature: first, they specify a mask, which typically precomputed, and subsequently introduce adversarial noise inside the mask Let M denote the mask matrix constraining the area of the perturbation ò; .M has the same dimensions as the input image and contains Os where no perturbation allowed, and in the area which can be perturbed. The physically realizable attacks we consider then solve optimization problem of the following form:
arg max L(s(x + M6;0),
(4)
Next, we describe the details of the three physical attacks we consider in the main paper.
E. EYEGLASS FRAME ATTACKS ON FACE RECOGNITION
Following Sharif et al. (2066), we first initialized the eyeglass frame with 5 different colors, and chose the best starting color by calculating the cross-entropy loss. For each update step, we divided the gradient value by its maximum value before multiplying by the learning rate which is 20. Then we only kept the gradient value of eyeglass frame area. Finally, we clipped and rounded the pixel value keep it in the valid range.
E.2 STICKER ATTACKS ON STOP SIGNS
Following Eykholt et al, (008)) we initialized the stickers on the stop signs with random noise. For each update step. used the Adam optimizer with 0.1 learning rate and with default parameters. Just as for other attacks, adversarial perturbations were restricted to the mask area exogenously spec- ified; in case, used the mask Eykholt et al. (2018 collection of small rectangles.
16

---

Published as a conference paper at ICLR 2020
E.3 ADVERSARIAL PATCH ATTACK
We used gradient ascent to maximize the log probability of the targeted class Plytarget [z]. as the original paper (Brown et al., 2018). When implementing the adversarial patch, used square patch rather than the circular patch the original paper; we don't t anticipate this choice to be prac- tically consequential. We randomly chose the position and direction of the patch, used the learning rate of 5, and fixed the number of attack iterations to 100 for each image. We varied the attack region (mask) {0%, 5%, 10%, 15%, 20%. 25%}.
For the face recognition dataset, we used 27 images (9 classes (without targeted class) X images each class) design the patch, and then ran the attack over 20 epochs. For the smaller traffic sign dataset, we used 15 images (15 classes (without targeted class) image in each class) to design the patch, and then ran the attack over epochs. Note that when evaluating the adversarial patch, we used the validation set without the targeted class images.
F ADDITIONAL EXPERIMENTS WITH DOA
F.1
FACE RECOGNITION AND EYEGLASS FRAME ATTACK
100
100
80
80
Accuracy(96)
Accuracy(%%
60
60
40
40
100x50 Exh. Search & 7PGD
100x50 Exh. Search &20PGD
100x50 Grad. Search 7PED
20
20
Search S20PGD
100x50 Search 630PGD
100x50 Grad. Search &30PGD
100x50 Search 6500GD
100x50 Grad. Search 650PGD
50 100 150 200
250
300
100 150 200
250 300
# Iterations
Iterations
Figure 9: Effectiveness of DOA using the gradient-based method and the 100 50 region against the eyeglass frame attack, varying the number of PGD iterations for adversarial perturbations inside the rectangle. Left: using exhaustive search. Right: using gradient-based search.
100
100
80
Accuracy(%)
Accuracy(%)
60
60
40
70 Exh. Search 7PGD
70 Grad.
Search 7PGD
20
70 Exh. Search 820PGD
20
70 Grad.
Search 620PGD
Exh. &30PGD
70x God..
Search 630PGD
70 Exh, Search 650PGD
70 Grad. Search 650PGD
50
100
150 200
250
300
0 50
100
150 200
250 300
# Iterations
# Iterations
Figure 10: Effectiveness of DOA using the gradient- based method and the 70 x 70 region against the eyeglass frame attack, varying the number of PGD iterations for adversarial perturbations inside the rectangle. Left: using exhaustive search. Right: using gradient-baat search.
17

---

Published as a conference paper at ICLR 2020
F.2 TRAFFIC SIGN CLASSIFICATION AND THE STOP SIGN ATTACK
Table 7: Comparison of effectiveness of different approaches against the stop sign attack. Best
parameter choices were made for each.
Attack Iterations
Validation Set
10a
102
10a
Clean Model
98.69 %
100%
42.5%
32.5%
25%
CAdv. Training: = 16
96.95%
97.5%
57.5%
45%
42.5%
Randomized Smoothing:
0.25
98.22%
100%
70%
64.5%
62.5%
DOA (exhaustive; 7 7; 30 PGD)
92.51%
100%
92%
92%
90.5%
DOA (gradient-based: x 7; 30 PGD)
95.82%
100 %
85%
85.5%
84%
Table 8: Effectiveness of 10 x 5 DOA using exhaustive search with different numbers of PGD iterations against the stop sign attack.
Attack Iterations
Validation Set
10a
102
103
Clean Model
98.69 %
100 %
42.5%
32.5%
25%
Exhaustive Search with 7 PGD
96.50%
100 %
73%
70.5%
71%
Exhaustive Search with 20 PGD
95.86%
%
86%
85%
85%
Exhaustive Search with 30 PGD
95.59%
100% %
86.5% %
84%
82.5%
Exhaustive Search with 50 PGD
95.87%
100%
77%
76.5%
73.5%
Table 9: Effectiveness of 10 gradient-based DOA with different numbers PGD iterations against the stop sign attack.
Attack Iterations
Validation
10a
102
10
Clean Model
98.69
100 %
42.5%
32.5%
25%
Gradient Based Search with 7 PGD
97.53%
100 %
83.5 %
78%
78%
Gradient Based Search with 20 PGD
97.11%
100 %
82.5%
81.5%
80.5%
Gradient Based Search with 30 PGD
96.83%
100 %
83.5%
79.5%
81%
Gradient Based Scarch with 50 PGD
96.46%
100% %
82.5%
82.5%
81.5%
Table 10: Effectiveness of 7 X 7 DOA using exhaustive search with different numbers of PGD iterations against the stop sign attack.
Attack Iterations
Validation Set
0
10A
102
10a
Clean Model
98.69 %
100 %
42.5%
32.5%
25%
Exhaustive Search with 7 PGD
94.16%
100 %
86.5%
85%
85%
Exhaustive Search with 20 PGD
92.74%
100 %
83%
80%
79%
Exhaustive Search with 30 PGD
92.51%
100 %
92%
92%
90.5%
Exhaustive Search with 50 PGD
92.94%
100
89.5%
89%
88.5%
18

---

Published as a conference paper at ICLR 2020
Table 11: Effectiveness of gradient-based DOA with different numbers of PGD iterations against the stop sign attack.
Atlack Iterations
Validation Set
10'
10a
103
Clean Model
98.69 %
100 %
42.5%
32.5%
25%
Gradient Based Search with 7 PGD
96.52%
100 %
81.5%
80%
79.5%
Gradient Based Search with 20 PGD
95.51 %
100% %
83.5%
82.5%
83%
Gradient Based Search with 30 PGD
95.82%
100%
85%
85.5%
84%
Gradient Based Search with 50 PGD
95.59%
100 %
81%
81%
81.5%
F.3 EVALUATION WITH THE ADVERSARIAL PATCH ATTACK
F.3.1 FACE RECOGNITION
Table 12:
Comparison of effectiveness of different approaches against the adversarial patch attack on the face recognition data. Best parameter choices were made for each.
Size of Attacking Region
0%
5%
10%
15%
20%
25%
Clean Model
99.30%
80.69%
41.21% %
19.41%
9.76%
5.75%
CAdv. Training: = 16
94.15%
48.26%
34.38%
28.85%
23.54%
19.02%
Randomized Smoothing: DOA (exh. 100 50; 50 PGD)
98.83%
61.99%
36.26%
33.80%
24.68%
16.02%
99.30%
98.72%
97.83%
97.18%
81.24%
24.19%
DOA (exh. 70 x 70; 50 PGD)
99.41 %
98.37%
97.29%
96.53%
63.23%
44.14%
Table 13: Effectiveness of adversarial training against the adversarial patch attack on face recogni- tion data.
Size of Attacking Region
0%
5%
10%
15%
20%
25%
Clean Model
99.30 %
80.69 %
41.21% %
19.41% %
9.76%
5.75%
Iterations Adv. Training: c = 92.28%
57.05%
39.15%
34.71% %
10.52%
7.81 1%
Iterations Adv. Training: =
66.54%
27.98%
23.54%
21.26%
18.55%
17.03% %
50 Iterations Adv. Training: 4
90.29%
57.38%
53.58
36.01 % 17.79%
28.85 %
16.05%
Iterations Adv. Training: 53.68%
21.80%
21.15%
17.79%
16. 16.70%
Table 14: Effectiveness of curriculum adversarial training against the adversarial patch attack face recognition data.
Size of Atlacking Region
0%
5%
10%
15%
20%
25%
Clean Model
99.30 %
80.69%
41.21%
19.41%
9.76%
5.75%
CAdv. Training: 97.89%
52.49%
45.44%
30.91%
16.05%
10.85%
CAdv. Training: = 8
96.84%
44.79%
29.18%
23.86%
12.47%
10.20%
CAdv. Training: 16
94.15%
48.26%
34.38%
28.85%
23.54%
19.20%
CAdy. Training: 81.05%
42.95%
40.46%
32.00%
20.93%
17.14%
19

---

Published as a conference paper at ICLR 2020
Table 15: Effectiveness of randomized smoothing against the adversarial patch attack face recog- nition data.
Size of Attacking Region
0%
5%
10%
15%%
20%
25%
Clean Model
99.30%
80.69%
41.21%
19.41%
9.76%
5.75%
Randomized Smoothing: 0 0.25
98. 98.95%
60.35%
26.67%
13.80%
6.67%
5.15%
Randomized Smoothing: 0.5
97. 66.
81.05 %
58.83%
33.92%
26.55%
14.74%
Randomized Smoothing: a
98. 98.83%
61.99%
36.26%
33.80%
24.68%
16.02
Table 16:
Effectiveness of DOA (50 PGD iterations) against the adversarial patch attack face
recognition data.
Size of Attacking Region
0%
5%
10%
15%
20%
25%
Clean Model
99.30 99.30%
80.69%
41.2 41.21%
19.41%
9.76%
5.75%
(100 x 50)Exhaustive Search
99.30%
98.70% %
97.83%
97.18%
81.24%
24.19%
(100 x 50)Gradient Based Search
99.53 %
98.41%
96.75%
87.42%
57.48%
24.62%
(70 x 70)Exhaustive Search
99.41% %
98.37%
97.29%
96.53%
63.23%
44.14%
(70 70)Gradient Based Search
98.83%
97.51%
96.31%
93.82%
92.19%
31.34%
F.3.2 TRAFFIC SIGN CLASSIFICATION
Table 17: Comparison of effectiveness of different approaches against the adversarial patch attack on the traffic sign data. Best parameter choices were made for each.
Size of Allacking Region
0%
5%
10%
15%
20%
25%
Clean Model
98.38
75.87%
58.62%
40.68%
33.10%
22.47%
CAdv. Training: 16
96.77%
86.93%
70.21 %
66. 11%
51.13%
40.77%
Randomized Smoothing: DOA (grad. : x 7; 30 PGD)
0.5
95.39%
83.28%
68.17%
53.86%
48.79%
33.10%
94.69%
90.68%
88.41 %
81.79%
72.30 %
58.71 %
DOA (grad. x 7; 50 PGD)
94.69%
90.33%
89.29 %
78.92%
70.21%
58.54%
Table 18: Effectiveness of adversarial training against the adversarial patch attack on traffic sign data.
Size of Attacking Region
0%
5%
10%
15%
20%
25%
Clean Model
98.38%
75.87%
58.62%
40.68%
33.10%
22.47%
Adv. Training: 4
98.96 %
82.14%
64.46 %
45.73%
23.78% 37.37%
31.36%
Adv. Training: e
95.62%
76.74%
62.89%
46.60 %
25.09 %
20

---

Published as a conference paper at ICLR 2020
Table 19:
Effectiveness of curriculum adversarial training against the adversarial patch attack
traffic sign data.
Size of Atlacking Region
0%
5%
10%
15%
20%
25%
Clean Model
98.38%
75.87%
58.62%
40.68%
33.10%
22.47%
CAdv. Training:
98.96 %
77.18%
57.32%
39.02%
38.94%
31.10%
CAdv. Training: e 8
98.96%
86.24%
68 38.64%
60.19%
42.42%
35.10%
CAdv. Training: 16
96.77%
86.93%
70.21%
66.11%
51.13%
40.77%
CAdv. Training: e = 32
63.78%
59.93%
51.57%
45.38%
37.37%
27.79%
Table 20: Effectiveness of randomized smoothing against the adversarial patch attack on traffic sign data.
Size of Attacking Region
0%
5%
10%
15%
20%
25%
Clean Model
98.38 %
75.87%
58.62%
40.68%
33.10%
22.47%
Randomized Smoothing: 0.25
98.27%
82.24%
66.78%
54.67 %
40.37%
33.79%
Randomized Smoothing: 0.5
95.39%
83.28%
68.17%
53.86%
48.79%
33.10%
Randomized Smoothing: a =
85.47%
74.39%
54.44%
45.44%
40.48%
29.06%
Table 21: Effectiveness of DOA (30 PGD iterations) against the adversarial patch attack on traffic
sign data.
Size of Attacking Region
0%
5%
10%
15%
20%
25%
Clean Model
98.38 %
75.87%
58.62%
40.68%
33.10%
22.47%
(10 x 5)Exxaustive Search
94.00%
91.46%
86.67%
76.39%
68.12%
55.57%
5)Gradient Based Search
93.77%
86.67%
80.57%
75.00%
64.90%
54.44%
(7 x Exhhuustivv Search
92.27%
89.02%
82.93%
78.40%
66.90%
55.92%
(7 G)ardiet Based Search
94.69%
90.68%
88.41 %
81.79%
72.30 %
58.71%
Table 22: Effectiveness of DOA (50 PGD iterations) against the adversarial patch attack traffic
sign data.
Size of Attacking Region
0%
5%
10%
15%
20%
25%
Clean Model
98.38 %
75.87%
58.62%
40.68%
33.10%
22.47%
(10 x 5)Exhaustive Search
96.42%
93.90%
90.42%
80.66%
67.77%
53.92%
(10 x 5)Gradient Based Search
93.54%
90.33%
85.80%
79.79%
69.86%
49.74%
(7 E)Exhaustive Search
87.08%
82.49%
75.96%
70.99%
60.10%
52.53%
(7 Graddient Based Search
94.69%
90.33%
89.29%
78.92%
70.21
58.54%
21

---

Published as a conference paper at ICLR 2020
G EXAMPLES OF PHYSICALLY REALIZABLE ATTACK AGAINST ALL DEFENSE MODELS
G.1 FACE RECOGNITION
Original image
Adv. image
Adv. training
Rand. smoothing
DOA
Aaron Tveit
Aaron Tveit
A.R. Rahman
Aaron Yoo
Aaron Tveit
Abbie Cornish
Abbic Cornish
Abigail Breslin
Aaron Tveit
Abigail Cornish
Examples of the eyeglass attack on face recognition. From left to right: 1) the original input image, 2) image with adversarial eyeglass frames, 3) face predicted by a model generated through adversarial training, 4) face predicted by a model generated through randomized smoothing, 5) face predicted (correctly) by a model generated through DOA. Each row is a separate example.
Figure 11:
G.2 TRAFFIC SIGN CLASSIFICATION
Original image
Ady. image
Ady. training
Rand. smoothing
DOA
SPEED LIMIT 35
STOP
STOP
STOP
Stop sign
Stop sign
Added lane
Speed limit 35
Stop sign
SPEED LIMIT 35
SPEED LIMIT 35
STOP
STOP
STOP
Stop sign
Stop sign
Speed limit 35
Speed limit 35
Stop sign
Figure 12: Examples of the stop sign attack. From left right: 1) the original input image, 2) image with adversarial eyeglass frames, 3) face predicted by model generated through adversarial training. face predicted by a model generated through randomized smoothing, 5) face predicted (correctly) by model generated through DOA. Each row is separate example.
22

---

Published as a conference paper at ICLR 2020
H EFFECTIVENESS OF DOA METHODS AGAINST Lo0 ATTACKS
For completeness, this section includes evaluation of DOA in the context of -bounded attacks implemented using PGD, though these are outside the scope of threat model.
Table 23: Effectiveness of DOA (50 PGD iterations) against 20 Iterations Loo Attacks Face Recognition
Attack Strength
0
2
4
c
Clean Model
98.94%
44.04%
1.70%
0%
0%
(100 50)Exhaustive Search
98.72%
39.79%
1.06%
0%
0%
(100 x 50)Gradient Based Search
98.51%
40.64%
3.40%
0%
0%
(70 70)Exhaustive Search
98.51%
35.74%
0.43%
0%
0%
(70 x 70)Gradient Based Search
97.45%
33.40%
0.43%
0%
0%
Table 24: Effectiveness of DOA (50 PGD iterations) against 20 Iterations L, Attacks on Traffic Sign Classification
Attack Strength
2
4
16
Clean Model
98.69%
89.54%
61.58%
24.65%
5.14%
(10 x 5)Exhaustive Search
95.87%
91.55%
76.57%
39.02%
7.75%
(10 5)arddeen Based Search
96.46%
91.81%
78.83%
46.86%
7.84%
(7 7)Exhaustive Search
92.94%
89.54%
77.09%
42.77%
5.83%
(7 x 7)Gradient Based Search
95.59%
91.20%
79.52%
46.86%
6.70%
presents results of several variants of DOA in the context of PGD attacks in the context of face recognition, while Table 24 considers these traffic sign classification. The results are quite consistent with intuition: DOA is largely unhelpful against these attacks. The reason is that DOA fundamentally assumes that the attacker only modifies a relatively small proportion (~5%) of the scene (and the resulting image), otherwise the physical attack would be highly suspicious. loc. bounded attacks, on the other hand, modify all pixels.
Table 23
23

---

Published as a conference paper at ICLR 2020
I EFFECTIVENESS OF DOA METHODS AGAINST to ATTACKS
In addition to considering physical attacks, we evaluate the effectiveness of DOA against Jacobian- based saliency map attacks (JSMA) (Papernot al. 2015) for implementing lo-constraint adversar- ial examples. As Figure 13 shows, in both the face recognition and traffic sign classification, DOA able to improve classificatio robustness compared the original model.
Face Recognition
Traffic Sign Classification
100
Original Model
100
Original Model
100x50 Exh, Search S50PGD
10x5 Exh. Search 650PGD
80
100x50 Search
80
550PGD
Accuracy
70x70 Grad. Search 650PGID
70x70 Exh, Search 850PGD
Accuracy
7x7 Grad. Search &50PGD
7x7 Exh, Search 630PGD
60
40
40
20
0
0
2
percent of image
percent of image
Figure 13: Lo attacks on face recognition and traffic sign classification.
24

---

Published as a conference paper at ICLR 2020
J EFFECTIVENESS OF DOA AGAINST OTHER MASK-BASED ATTACKS
To further illustrate the ability of DOA to generalize, we evaluate its effectiveness the context of three additional occlusion patterns: union of triangles and circle, single larger triangle, and heart pattern.
As the results in Figures 14 and 15 suggest, DOA able to generalize successfully variety of physical attack patterns. is particularly noteworthy that the larger patterns (large triangle-midde middle of the figure, and large heart-right right of the figure) are actually quite suspicious (particularly the heart pattern), as they occupy significant fraction of the image (the heart mask, for example, accounts for 8% the face).
Mask1
Mask2
Mask3
Abbie Cornish
Abbie Cornish
Abbie Cornish
Accuracy(S)
100x50 Eth. Search 450PCD
Orignnl Madel
Accuraset
Driginal Model
100x Exh. Search 550PGD
Accoraayy(5)
100x50 Grad. Search 6307GD
100x Grad. Search 653P5D
70x Eaih Search S33PGD
70 10 Grad. Search E5OPGD
703 72 Een. Search 630PGD
70x Grdd. Search 6307CD
100x50 Eah. Searth 610PGD
100 Search 650PCD
20
70x70 Esh 5SPGD
20x30 Search 650eGD
0-
100 150
1000
200
100
150
Iterations
Reertions
Iterations
Figure 14: Additional mask-bas based attacks on face recognition.
25

---

Published as a conference paper at ICLR 2020
Maski
Mask2
Mask3
Keep Right
Keep Right
Keep Right
10ak DSSIO LiNDo
Atlack baraiona
Albacc Rarat
Atlack haratona
Figure 15: Additional mask-based attacks on traffic sign classification.
26


Reviews:
[{"review_id": "SkgpWdyEqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all, the two physical attacks evaluated in this paper have similar attacking patterns, i.e., mask-based pixel attacks. So it is not surprising that DOA is more robust in these cases since DOA is trained on this attacking pattern.\n\nActually it has been shown that the framework of adversarial training (AT) will overfit to the attacking patterns used in training. For example, PGD-AT models are less robust to simple non-pixel transformation, like rotation, than the standard models [1]. So what DOA does is just substituting the PGD module in AT to overfit the new attacking patterns, which is of limited contribution and novelty. \n\nBesides, AT is not really scalable compared to other simpler defense strategies like input transformation. [2] proposes a simple and effective defense based on different combinations of input transformation and its performance even surpasses some SOTA AT models with less computation. \n\nAnother advantage of these off-the-shelf defenses like input transformation is that they do not depend on the specific details of attacks, so they are more reliable when you are unknown about the potential attacking patterns in practice. In comparison, there is an implicit assumption in AT methods that the attacking patterns in training and test are similar. This is the reason why PGD-AT is not robust facing mask-based physical attacks or simple rotations.\n\nSo under the more completed and flexible physical attacks, a defense based on the AT framework like DOA may not be a good choice. Although AT methods are quite effective and widely studied under the l_p attacks, the authors are expected to consider more factors if they really want to design a robust system in the physical world, rather than just follow or apply the most popular pipeline like AT.\n\nReference:\n[1] Engstrom et al. A rotation and a translation suffice: Fooling cnns with simple transformations. ICML 2019\n[2] Raff et al. Barrage of Random Transforms for Adversarially Robust Defense. CVPR 2019", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skg9iIyx9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Note: I gave a 3 for thoroughness even though I only read the paper once, because I believe that I carefully considered the paper while reading it.]\n\nThis paper argues that threat models such as L-inf are limited when considering physically realizable attacks, and provides evidence for this by showing that L-inf adversarial training is insufficient to fully confer robustness against physically realizable attacks in the literature such as the adversarial glasses attack. The paper then proposes an alternate threat model based on contiguous rectangular regions, and shows that adversarial training against this model does far better.\n\nOverall this is a strong paper with thorough experiments, and was for the most part carefully written (although see some quibbles below). I particular liked the paper's carefully distinction between physically *realizable* and physically *realized* attacks, and the admission that not all realizable attacks would fall under the threat model (while still justifying why the model is interesting).\n\nI am on the border of weak accept and strong accept. The main two points keeping me from strong accept are discussed below (and seem perhaps addressable by the authors):\n\n1. The assertion that rectangular occlusions might be a fruitful model for realizable attacks is only lightly tested, since the attacks considered in this paper all fall into the rectangular occlusions threat model. Since one of the key points in the paper is that training in the L-inf threat model could leave vulnerabilities to non-Linf attacks, we might expect the same with rectangular occlusions. A more convincing demonstration would be to test robustness to physically realizable attacks that fall outside the rectangular model (perhaps even skewed or rotated rectangles, although a less synthetic example would be even better). A more minor point is that it would be nice to test the robustness of models trained on rectangular occlusions against L-inf adversarial attacks. This is relevant to knowing whether training on occlusions is giving up on adversarial robustness or actually helps against both (I actually think it's plausible that you would do well on L-inf, but it seems worth testing either way).\n\n2. The high-level claimed take-away that adversarial training does not help against physically realizable attacks seems false in light of Figure 2, which shows that L-inf adversarial training substantially improves robustness relative to the baseline. A more accurate take-away would be that on some datasets adversarial training helps but still leaves a gap, while on others it does not help at all and perhaps hurts. I would prefer more careful wording as a reader who only goes through the introduction might not see Figure 2.\n\nOn #1, I should stress that the paper would still be interesting regardless of the outcome of the two experiments in #1; I just think that for thoroughness it would be nice to include them. The existing experiments are already thorough so this would be going above and beyond, but that is why it might help raise my score from weak to strong accept.\n\nEDITED TO ADD: For #2, I would also be happy with an argument from the authors as to why the current language appropriately describes the experiments. I do not wish to dictate to the authors what their take-aways are, but more to open up for discussion a point that seemed slightly sloppy to me.\n\nMinor comments:\n\nPlease avoid subjective intensifiers: \"We then use an extensive experimental evaluation to demonstrate that our proposed approach is far more robust against physical attacks on deep neural networks than adversarial training and randomized smoothing methods that leverage lp-based attack models.\" Both \"extensive\" and \"far\" are unnecessary.\n\nMake sure to use \\citep vs \\citet correctly.\n\nFirst sentence of 2.1 is too verbose. Overall the prose in that paragraph is turgid, due to too many action phrases being turned into nouns. E.g. \"The focus is on\", \"The typical goal is\" both indicate *action* and could be profitably turned into verbs. See Williams and Bizup's book on Style.\n\n\"since our ultimate goal is to defend against physical attacks, untargeted attacks that aim to maximize error are the most useful\" This seems weak; I don't understand why an attack being physical should go in line with it being untargeted; oftentimes an attacker will have a specific targeted goal.\n\n\"another advantage of the ROA attack is that it is, in principle,easier to compute than,  say,l∞-bounded attacks\" This seems incongruous with the subsequent text, which admits that ROA if implemented naively would be slower than L-inf attacks.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byglyo0otB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to provide a defense to physically realizable attacks for image classifiers. First, the paper demonstrates that Lp-ball robustness (obtained via adversarial training or randomized smoothing) do not necessarily result in robustness to physical attacks such as adversarial stickers. Next, the paper proposes a variant of adversarial training (using adversarial rectangles) and shows empirically that such a method results in much improved robustness to the aforementioned physical attacks.\n\nI vote to accept this paper. It has clear motivation, is very clearly written, evaluates proper benchmarks from recent literature, and proposes a new method that shows a clear improvement over benchmarks in literature. The goal is clearly laid out in Section 3, and the paper justifies its claims via various experiments. Overall, I think it is a good contribution to the adversarial examples literature, as it provides robustness against more “real-world” attacks.\n\nIn particular, the results from Fig. 2 and Fig. 3, while not surprising, appear to be done thoroughly, as the authors evaluate against various forms of adversarial training and various degrees of randomized smoothing. Later, the results in Fig. 4 and Fig. 5 show a clear benefit from their method.\n\nI do have a few pieces of feedback that I think could improve the work.\n\nIn section 5.1, do you really only train for 5 epochs? Is this just a fine-tuning procedure after standard training is performed, or is the entire training procedure 5 epochs? That seems especially short for getting any amount of accuracy. I think it is worth clarifying.\n\nIn Section 3 (or in the Related Works), I would suggest that the authors do mention related works about other types of realistic adversarial examples or perturbations, such as those generated via physical transformations like translations and rotations [1] or even common corruption robustness [2]. This is especially relevant since the authors claim to be the “first investigation of this issue in computer vision applications,” so it’s worth clarifying that the authors do not claim to be the first work about robustness to all realistic perturbations.\n\nOne thing I would be curious to see is if the DOA training method provides any robustness to standard Lp-ball adversarial examples or to perturbations like rotations.\n\nAdditional Feedback:\n\n- In Related Works, I think there are some places where it would look better to use parentheses around the citations.\n\n[1] https://arxiv.org/abs/1712.02779\n[2] https://arxiv.org/abs/1903.12261", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of defending deep neural network approaches for image classification from physically realizable attacks. It first demonstrates that adversarial training with PGD attacks and randomized smoothing exhibit limited effectiveness against three of the highest profile physical attacks. Then, it proposes a new abstract adversarial model, where an adversary places a small adversarially crafted rectangle in an image, and develops two approaches for efficiently computing the resulting adversarial examples. Empirical results show the effectiveness. Overall, a good paper. The rebuttal is convincing.
Decision: Accept (Spotlight)