Learning perturbation sets for robust machine learning

1 INTRODUCTION . Within the last decade , adversarial learning has become a core research area for studying robustness and machine learning . Adversarial attacks have expanded well beyond the original setting of imperceptible noise to more general notions of robustness , and can broadly be described as capturing sets of perturbations that humans are naturally invariant to . These invariants , such as facial recognition should be robust to adversarial glasses ( Sharif et al. , 2019 ) or traffic sign classification should be robust to adversarial graffiti ( Eykholt et al. , 2018 ) , form the motivation behind many real world adversarial attacks . However , human invariants can also include notions which are not inherently adversarial , for example image classifiers should be robust to common image corruptions ( Hendrycks & Dietterich , 2019 ) as well as changes in weather patterns ( Michaelis et al. , 2019 ) . On the other hand , although there has been much success in defending against small adversarial perturbations , most successful and principled methods for learning robust models are limited to human invariants that can be characterized using mathematically defined perturbation sets , for example perturbations bounded in ` p norm . After all , established guidelines for evaluating adversarial robustness ( Carlini et al. , 2019 ) have emphasized the importance of the perturbation set ( or the threat model ) as a necessary component for performing proper , scientific evaluations of adversarial defense proposals . However , this requirement makes it difficult to learn models which are robust to human invariants beyond these mathematical sets , where real world attacks and general notions of robustness can often be virtually impossible to write down as a formal set of equations . This incompatibility between existing methods for learning robust models and real-world , human invariants raises a fundamental question for the field of robust machine learning : How can we learn models that are robust to perturbations without a predefined perturbation set ? In the absence of a mathematical definition , in this work we present a general framework for learning perturbation sets from perturbed data . More concretely , given pairs of examples where one is a perturbed version of the other , we propose learning generative models that can “ perturb ” an example by varying a fixed region of the underlying latent space . The resulting perturbation sets are welldefined and can naturally be used in robust training and evaluation tasks . The approach is widely applicable to a range of robustness settings , as we make no assumptions on the type of perturbation being learned : the only requirement is to collect pairs of perturbed examples . Given the susceptibility of deep learning to adversarial examples , such a perturbation set will undoubtedly come under intense scrutiny , especially if it is to be used as a threat model for adversarial attacks . In this paper , we begin our theoretical contributions with a broad discussion of perturbation sets and formulate deterministic and probabilistic properties that a learned perturbation set should have in order to be a meaningful proxy for the true underlying perturbation set . The necessary subset property ensures that the set captures real perturbations , properly motivating its usage as an adversarial threat model . The sufficient likelihood property ensures that real perturbations have high probability , which motivates sampling from a perturbation set as a form of data augmentation . We then prove the main theoretical result , that a learned perturbation set defined by the decoder and prior of a conditional variational autoencoder ( CVAE ) ( Sohn et al. , 2015 ) implies both of these properties , providing a theoretically grounded framework for learning perturbation sets . The resulting CVAE perturbation sets are well motivated , can leverage standard architectures , and are computationally efficient with little tuning required . We highlight the versatility of our approach using CVAEs with an array of experiments , where we vary the complexity and scale of the datasets , perturbations , and downstream tasks . We first demonstrate how the approach can learn basic ` ∞ and rotation-translation-skew ( RTS ) perturbations ( Jaderberg et al. , 2015 ) in the MNIST setting . Since these sets can be mathematically defined , our goal is simply to measure exactly how well the learned perturbation set captures the target perturbation set on baseline tasks where the ground truth is known . We next look at a more difficult setting which can not be mathematically defined , and learn a perturbation set for common image corruptions on CIFAR10 ( Hendrycks & Dietterich , 2019 ) . The resulting perturbation set can interpolate between common corruptions , produce diverse samples , and be used in adversarial training and randomized smoothing frameworks . The adversarially trained models have improved generalization performance to both in- and out-of-distribution corruptions and better robustness to adversarial corruptions . In our final setting , we learn a perturbation set that captures real-world variations in lighting using a multi-illumination dataset of scenes captured “ in the wild ” ( Murmann et al. , 2019 ) . The perturbation set generates meaningful lighting samples and interpolations while generalizing to unseen scenes , and can be used to learn image segmentation models that are empirically and certifiably robust to lighting changes . All code and configuration files for reproducing the experiments as well as pretrained model weights for both the learned perturbation sets as well as the downstream robust classifiers are at https : //github.com/locuslab/perturbation_learning . 2 BACKGROUND AND RELATED WORK . Perturbation sets for adversarial threat models Adversarial examples were initially defined as imperceptible examples with small ` 1 , ` 2 and ` ∞ norm ( Biggio et al. , 2013 ; Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) , forming the earliest known , well-defined perturbation sets that were eventually generalized to the union of multiple ` p perturbations ( Tramèr & Boneh , 2019 ; Maini et al. , 2019 ; Croce & Hein , 2019 ; Stutz et al. , 2019 ) . Alternative perturbation sets to the ` p setting that remain well-defined incorporate more structure and semantic meaning , such as rotations and translations ( Engstrom et al. , 2017 ) , Wasserstein balls ( Wong et al. , 2019 ) , functional perturbations ( Laidlaw & Feizi , 2019 ) , distributional shifts ( Sinha et al. , 2017 ; Sagawa et al. , 2019 ) , word embeddings ( Miyato et al. , 2016 ) , and word substitutions ( Alzantot et al. , 2018 ; Jia et al. , 2019 ) . Other work has studied perturbation sets that are not necessarily mathematically formulated but welldefined from a human perspective such as spatial transformations ( Xiao et al. , 2018b ) . Real-world adversarial attacks tend to try to remain either inconspicuous to the viewer or meddle with features that humans would naturally ignore , such as textures on 3D printed objects ( Athalye et al. , 2017 ) , graffiti on traffic signs ( Eykholt et al. , 2018 ) , shapes of objects to avoid LiDAR detection ( Cao et al. , 2019 ) , irrelevant background noise for audio ( Li et al. , 2019a ) , or barely noticeable films on cameras ( Li et al. , 2019b ) . Although not necessarily adversarial , Hendrycks & Dietterich ( 2019 ) propose the set of common image corruptions as a measure of robustness to informal shifts in distribution . Generative modeling and adversarial robustness Relevant to our work is that which combines aspects of generative modeling with adversarial examples . While our work aims to learn real-world perturbation sets from data , most work in this space differs in that they either aim to generate synthetic adversarial ` p perturbations ( Xiao et al. , 2018a ) , run user studies to define the perturbation set ( Sharif et al. , 2019 ) , or simply do not restrict the adversary at all ( Song et al. , 2018 ; Bhattad et al. , 2020 ) . Gowal et al . ( 2019 ) trained a StyleGAN to disentangle real-world perturbations when no perturbation information is known in advance . However the resulting perturbation set relies on a stochastic approximation , and it is not immediately obvious what this set will ultimately capture . Most similar is the concurrent work of Robey et al . ( 2020 ) , which uses a GAN architecture from image-to-image translation to model simple perturbations between datasets . In contrast to both of these works , our setting requires the collection of paired data to directly learn how to perturb from perturbed pairs without needing to disentangle any features or translate datasets , allowing us to learn more targeted and complex perturbation sets . Furthermore , we formulate desirable properties of perturbation sets for downstream robustness tasks , and formally prove that a conditional variational autoencoder approach satisfies these properties . This results in a principled framework for learning perturbation sets that is quite distinct from these GAN-based approaches in both setting and motivation . Adversarial defenses and data augmentation Successful approaches for learning adversarially robust networks include methods which are both empirically robust via adversarial training ( Goodfellow et al. , 2014 ; Kurakin et al. , 2016 ; Madry et al. , 2017 ) and also certifiably robust via provable bounds ( Wong & Kolter , 2017 ; Wong et al. , 2018 ; Raghunathan et al. , 2018 ; Gowal et al. , 2018 ; Zhang et al. , 2019 ) and randomized smoothing ( Cohen et al. , 2019 ; Yang et al. , 2020 ) . Critically , these defenses require mathematically-defined perturbation sets , which has limited these approaches from learning robustness to more general , real-world perturbations . We directly build upon these approaches by learning perturbation sets that can be naturally and directly incorporated into robust training , greatly expanding the scope of adversarial defenses to new contexts . Our work also relates to using non-adversarial perturbations via data augmentation to reduce generalization error ( Zhang et al. , 2017 ; DeVries & Taylor , 2017 ; Cubuk et al. , 2019 ) , which can occasionally also improve robustness to unrelated image corruptions ( Geirhos et al. , 2018 ; Hendrycks et al. , 2019 ; Rusak et al. , 2020 ) . Our work differs in that rather than aggregating or proposing generic data augmentations , our perturbation sets can provide data augmentation that is targeted for a particular robustness setting . 3 PERTURBATION SETS LEARNED FROM DATA . For an example x ∈ Rm , a perturbation set S ( x ) ⊆ Rm is defined informally as the set of examples which are considered to be equivalent to x , and hence can be viewed as “ perturbations ” of x . This set is often used when finding an adversarial example , which is typically cast as an optimization problem to maximize the loss of a model over the perturbation set in order to break the model . For example , for a classifier h , loss function ` , and label y , an adversarial attack tries to solve the following : maximize x′∈S ( x ) ` ( h ( x′ ) , y ) . ( 1 ) A common choice for S ( x ) is an ` p ball around the unperturbed example , defined as S ( x ) = { x+ δ : ‖δ‖p ≤ } for some norm p and radius . This type of perturbation captures unstructured random noise , and is typically taken with respect to ` p norms for p ∈ { 0 , 1 , 2 , ∞ } , though more general distance metrics can also be used . Although defining the perturbation set is critical for developing adversarial defenses , in some scenarios , the true perturbation set may be difficult to mathematically describe . In these settings , it may still be possible to collect observations of ( non-adversarial ) perturbations , e.g . pairs of examples ( x , x̃ ) where x̃ is the perturbed data . In other words , x̃ is a perturbed version of x , from which we can learn an approximation of the true perturbation set . While there are numerous possible approaches one can take to learn S ( x ) from examples ( x , x̃ ) , in this work we take a generative modeling perspective , where examples are perturbed via an underlying latent space . Specifically , let g : Rk × Rm → Rm be a generator that takes a k-dimensional latent vector and an input , and outputs a perturbed version of the input . Then , we can define a learned perturbation set as follows : S ( x ) = { g ( z , x ) : ‖z‖ ≤ } ( 2 ) In other words , we have taken a well-defined norm-bounded ball in the latent space and mapped it to a set of perturbations with a generator g , which perturbs x into x̃ via a latent code z. Alternatively , we can define a perturbation set from a probabilistic modeling perspective , and use a distribution over the latent space to parameterize a distribution over examples . Then , S ( x ) is now a random variable defined by a probability distribution p ( z ) over the latent space as follows : S ( x ) ∼ pθ such that θ = g ( z , x ) , z ∼ p ( 3 ) where p has support { z : ‖z‖ ≤ } and pθ is a distribution parameterized by θ = g ( z , x ) .

Summary: This paper addresses the problem of constraining adversarial image perturbations to be similar to some natural class of perturbations. This would allow natural perturbations to be treated with the same rigor (e.g. quantifiable attack and defense strengths) as standard adversarial perturbations. Standard adversarial perturbations are arbitrary perturbations of an image that are within a certain $\mathcal{l}_p$-ball around the datapoint in pixel space. They are otherwise unconstrained and therefore appear as unstructured noise. Instead, the paper proposes to train a conditional autoencoder to generate perturbed versions of clean images from pairs of clean and perturbed images. The autoencoder can then be used to generate new perturbations that are similar to the training data. Adversarial versions of these perturbations can then be defined in the latent space of the autoencoder, rather than in pixel space. The paper provides theoretical arguments that perturbations generated in this way are close to the perturbations used for training. Using these learned adversarial perturbations, models can then be trained that are robust to them.