Testing Robustness Against Unforeseen Adversaries

1 INTRODUCTION . Neural networks perform well on many datasets ( He et al. , 2016 ) yet can be consistently fooled by minor adversarial distortions ( Goodfellow et al. , 2014 ) . The research community has responded by quantifying and developing adversarial defenses against such attacks ( Madry et al. , 2017 ) , but these defenses and metrics have two key limitations . First , the vast majority of existing defenses exclusively defend against and quantify robustness to Lp-constrained attacks ( Madry et al. , 2017 ; Cohen et al. , 2019 ; Raff et al. , 2019 ; Xie et al. , 2018 ) . Though real-world adversaries are not Lp constrained ( Gilmer et al. , 2018 ) and can attack with diverse distortions ( Brown et al. , 2017 ; Sharif et al. , 2019 ) , the literature largely ignores this and evaluates against the Lp adversaries already seen during training ( Madry et al. , 2017 ; Xie et al. , 2018 ) , resulting in optimistic robustness assessments . The attacks outside the Lp threat model that have been proposed ( Song et al. , 2018 ; Qiu et al. , 2019 ; Engstrom et al. , 2017 ; Evtimov et al. , 2017 ; Sharif et al. , 2016 ) are not intended for general defense evaluation and suffer from narrow dataset applicability , difficulty of optimization , or fragility of auxiliary generative models . Second , existing defenses assume that attacks are known in advance ( Goodfellow , 2019 ) and use knowledge of their explicit form during training ( Madry et al. , 2017 ) . In practice , adversaries can deploy unforeseen attacks not known to the defense creator . For example , online advertisers use attacks such as perturbed pixels in ads to defeat ad blockers trained only on the previous generation of ads in an ever-escalating arms race ( Tramèr et al. , 2018 ) . However , current evaluation setups implicitly assume that attacks encountered at test-time are the same as those seen at train-time , which is unrealistic . The reality that future attacks are unlike those encountered during training is akin to a train-test distribution mismatch—a problem studied outside of adversarial robustness ( Recht et al. , 2019 ; Hendrycks & Dietterich , 2019 ) —but now brought to the adversarial setting . The present work addresses these limitations by proposing an evaluation framework ImageNet-UA to measure robustness against unforeseen attacks . ImageNet-UA assesses a defense which may have been created with knowledge of the commonly used L∞ or L2 attacks with six diverse attacks ( four of which are novel ) distinct from L∞ or L2 . We intend these attacks to be used at test-time only and not during training . Performing well on ImageNet-UA thus demonstrates generalization to a diverse set of distortions not seen during defense creation . While ImageNet-UA does not provide an exhaustive guarantee over all conceivable attacks , it evaluates over a diverse unforeseen test distribution similar to those used successfully in other studies of distributional shift ( Rajpurkar et al. , 2018 ; Hendrycks & Dietterich , 2019 ; Recht et al. , 2019 ) . ImageNet-UA works for ImageNet models and can be easily used with our code available at https : //github.com/ anon-submission-2020/anon-submission-2020 . Designing ImageNet-UA requires new attacks that are strong and varied , since real-world attacks are diverse in structure . To meet this challenge , we contribute four novel and diverse adversarial attacks which are easily optimized . Our new attacks produce distortions with occlusions , spatial similarity , and simulated weather , all of which are absent in previous attacks . Performing well on ImageNet-UA thus demonstrates that a defense generalizes to a diverse set of distortions distinct from the commonly used L∞ or L2 . With ImageNet-UA , we show weaknesses in existing evaluation practices and defenses through a study of 8 attacks against 48 models adversarially trained on ImageNet-100 , a 100-class subset of ImageNet . While most adversarial robustness evaluations use only L∞ attacks , ImageNet-UA reveals that models with high L∞ attack robustness can remain susceptible to other attacks . Thus , L∞ evaluations are a narrow measure of robustness , even though much of the literature treats this evaluation as comprehensive ( Madry et al. , 2017 ; Qian & Wegman , 2019 ; Schott et al. , 2019 ; Zhang et al. , 2019 ) . We address this deficiency by using the novel attacks in ImageNet-UA to evaluate robustness to a more diverse set of unforeseen attacks . Our results demonstrate that L∞ adversarial training , the current state-of-the-art defense , has limited generalization to unforeseen adversaries , and is not easily improved by training against more attacks . This adds to the evidence that achieving robustness against a few train-time attacks is insufficient to impart robustness to unforeseen test-time attacks ( Jacobsen et al. , 2019 ; Jordan et al. , 2019 ; Tramèr & Boneh , 2019 ) . In summary , we propose the framework ImageNet-UA to measure robustness to a diverse set of attacks , made possible by our four new adversarial attacks . Since existing defenses scale poorly to multiple attacks ( Jordan et al. , 2019 ; Tramèr & Boneh , 2019 ) , finding defense techniques which generalize to unforeseen attacks is crucial to create robust models . We suggest ImageNet-UA as a way to measure progress towards this goal . 2 RELATED WORK . Adversarial robustness is notoriously difficult to correctly evaluate ( Papernot et al. , 2017 ; Athalye et al. , 2018a ) . To that end , Carlini et al . ( 2019a ) provide extensive guidance for sound adversarial robustness evaluation . By measuring attack success rates across several distortion sizes and using a broader threat model with diverse differentiable attacks , ImageNet-UA has several of their recommendations built-in , while greatly expanding the set of attacks over previous work on evaluation . We are only aware of a few prior works which evaluate on unforeseen attacks in specific limited circumstances . Wu et al . ( 2020 ) evaluate against physically-realizable attacks from Evtimov et al . ( 2017 ) and Sharif et al . ( 2016 ) , though this limits the threat model to occlusion attacks on narrow datasets . Outside of vision , Pierazzi et al . ( 2020 ) proposes constraining attacks by a more diverse set of problem-space constraints in diverse domains such as text and malware or source code generation ; however , even in this framework , analytically enumerating all such constraints is impossible . Within vision , prior attacks outside the Lp threat model exist , but they lack the general applicability and fast optimization of ours . Song et al . ( 2018 ) and Qiu et al . ( 2019 ) attack using variational autoencoders and StarGANs , respectively , resulting in weaker attacks which require simple image distributions suitable for VAEs and GANs . Engstrom et al . ( 2017 ) apply Euclidean transformations determined by brute-force search . Zhao et al . ( 2019 ) use perceptual color distances to align human perception and L2 perturbations . Evtimov et al . ( 2017 ) and Sharif et al . ( 2016 ) attack stop signs and face-recognition systems with carefully placed patches or modified eyeglass frames , requiring physical object creation and applying only to specific image types . 3 NEW ATTACKS FOR A BROADER THREAT MODEL . There are few diverse , easily optimizable , plug-and-play adversarial attacks in the current literature ; outside of Elastic ( Xiao et al. , 2018 ) , most are Lp attacks such as L∞ ( Goodfellow et al. , 2014 ) , L2 ( Szegedy et al. , 2013 ; Carlini & Wagner , 2017 ) , L1 ( Chen et al. , 2018 ) . We rectify this deficiency with four novel adversarial attacks : JPEG , Fog , Snow , and Gabor . Our attacks are differentiable and fast , while optimizing over enough parameters to be strong . We show example adversarial images in Figure 1 and compare stochastic and adversarial distortions in Figure 2 . Our novel attacks provide a range of test-time adversaries visually and semantically distinct from L∞ and L2 attacks . Namely , they cause distortions with large L∞ and L2 norm , but result in images that are perceptually close to the original . These attacks are intended as unforeseen attacks not used during training , allowing them to evaluate whether a defense can generalize from L∞ or L2 to a more varied set of distortions than current evaluations . Though our attacks are not exhaustive , performing well against them already demonstrates robustness to occlusion , spatial similarity , and simulated weather , which are absent from previous evaluations . Our attacks create an adversarial image x′ from a clean image x with true label y . Let model f map images to a softmax distribution , and let ` ( f ( x ) , y ) be the cross-entropy loss . Given a target class y′ 6= y , our attacks attempt to find a valid image x′ such that ( 1 ) the attacked image x′ is obtained by applying a distortion ( of size controlled by a parameter ε ) to x , and ( 2 ) the loss ` ( f ( x′ ) , y′ ) is minimized . An unforeseen adversarial attack is a white- or black-box adversarial attack unknown to the defense designer which does not change the true label of x according to an oracle or human . 3.1 FOUR NEW UNFORESEEN ATTACKS . JPEG . JPEG applies perturbations in a JPEG-encoded space of compressed images rather than raw pixel space . More precisely , JPEG compression is a linear transform JPEG which applies colorspace conversion , the discrete cosine transform , and then quantization . Our JPEG attack imposes the L∞-constraint ‖JPEG ( x ) − JPEG ( x′ ) ‖∞ ≤ ε on the attacked image x′ . We optimize z = JPEG ( x′ ) under this constraint to find an adversarial perturbation in the resulting frequency space . The perturbed frequency coefficients are quantized , and we then apply a right-inverse of JPEG to obtain the attacked image x′ in pixel space . We use ideas from Shin & Song ( 2017 ) to make this differentiable . The resulting attack is conspicuously distinct from Lp attacks . Fog . Fog simulates worst-case weather conditions . Robustness to adverse weather is a safety critical priority for autonomous vehicles , and Figure 2 shows Fog provides a more rigorous stress-test than stochastic fog ( Hendrycks & Dietterich , 2019 ) . Fog creates adversarial fog-like occlusions by adversarially optimizing parameters in the diamond-square algorithm ( Fournier et al. , 1982 ) typically used to render stochastic fog effects . This algorithm starts with random perturbations to the four corner pixels of the image . At step t , it iteratively perturbs pixels at the centers of squares and diamonds formed by those pixels perturbed at step t−1 . The perturbation of a step t pixel is the average of the neighboring step t−1 perturbations plus a parameter value which we adversarially optimize . We continue this process until all pixels have been perturbed ; the outcome is a fog-like distortion to the original image . Snow . Snow simulates snowfall with occlusions of randomly located small image regions representing snowflakes . Because the distortions caused by snowflakes are not differentiable in their locations , we instead place occlusions representing snowflakes at randomly chosen locations and orientations and adversarially optimize their intensities . This choice results in a fast , differentiable , and strong attack . Compared to synthetic stochastic snow ( Hendrycks & Dietterich , 2019 ) , our adversarial snow is faster and includes snowflakes at differing angles . Figure 2 shows adversarial snow exposes model weaknesses more effectively than the easier stochastic , average-case snow . Gabor . Gabor spatially occludes the image with visually diverse Gabor noise Lagae et al . ( 2009 ) . Gabor noise is a form of band-limited anisotropic procedural noise which convolves a parameter mask with a Gabor kernel which is a product of a Gaussian kernel and a harmonic kernel . We choose the Gabor kernel randomly and adversarially optimize the parameters of the mask starting from a sparse initialization . We apply spectral variance normalization ( Co et al. , 2019 ) to the resulting distortion and add it to the input image to create the attack .

Summary: This paper proposes four novel efficient adversarial attack methods beyond Lp threat models. Together with other two existing attack methods, these six attack methods combine as a framework to evaluate robustness of defenses against unforeseen attacks. In this framework, the novel measure is normalized with the performance of adversarial training. The experiments show that the Linf adversarially trained model may not lead to improvement of robustness against other threat models. It is expected that the framework could help test model robustness.