Learning perturbation sets for robust machine learning
Authors: Eric Wong
J Zico Kolter
Venue: ICLR 2021 Poster (2021)
Keywords: adversarial examples
perturbation sets
robust machine learning
conditional variational autoencoder

Abstract: Although much progress has been made towards robust deep learning, a significant gap in robustness remains between real-world perturbations and more narrowly defined sets typically studied in adversarial defenses. In this paper, we aim to bridge this gap by learning perturbation sets from data, in order to characterize real-world effects for robust training and evaluation. Specifically, we use a conditional generator that defines the perturbation set over a constrained region of the latent space. We formulate desirable properties that measure the quality of a learned perturbation set, and theoretically prove that a conditional variational autoencoder naturally satisfies these criteria. Using this framework, our approach can generate a variety of perturbations at different complexities and scales, ranging from baseline spatial transformations, through common image corruptions, to lighting variations. We measure the quality of our learned perturbation sets both quantitatively and qualitatively, finding that our models are capable of producing a diverse set of meaningful perturbations beyond the limited data seen during training. Finally, we leverage our learned perturbation sets to train models which are empirically and certifiably robust to adversarial image corruptions and adversarial lighting variations, while improving generalization on non-adversarial data. All code and configuration files for reproducing the experiments as well as pretrained model weights can be found at https://github.com/locuslab/perturbation_learning. 

Published as a conference paper at ICLR 2021
LEARNING PERTURBATION SETS FOR ROBUST MA- CHINE LEARNING
Eric Wong
J. Zico Kolter
Computer Science and Artificial
Computer Science Department Carnegic Mellon University and Bosch Center for Artificial Intelligence Pittsburgh, PA 15213, USA zkolterêcs, cmu. edu
Intelligence Laboratory Massachusetts Institute Technology Cambridge, MA 02139, USA wongeri Snngeric@miteede
ABSTRACT
Although much progress has been made towards robust deep learning, a significant gap in robustness remains between real-world perturbations and more narrowly defined sets typically studied in adversarial defenses. this paper, we aim to bridge this gap by learning perturbation sets from data, in order to characterize real-wor world effects for robust training and evaluation. Specifically, we use a conditional genera- tor that defines the perturbation set We formulate desirable properties that measure the quality of a learned perturbation set, and theoretically prove that conditional variational autoencoder naturally satisfies these criteria. Using this framework, our approach can generate a variety of perturbations at different complexities and scales, ranging from baseline spatial transformations, through common image corruptions, to lighting variations. We measure the quality of our learned perturbation sets both quantitatively and qualita- tively, finding that our models are capable of producing diverse set of meaningful perturbations beyond the limited data seen during training. Finally, we leverage our learned perturbation sets to train models which are empirically and certifiably robust to adversarial image corruptions and adversarial lighting variations, while improving generalization for reproducing the experiments as well as pretrained model weights can be found at wttttts::/// hetttp::/githubb com/ locus wwcuusaabpppee
constrained region of the latent space.
non-adversarial data. All code and configuration files
on ea an ng
INTRODUCTION
Within the last decade, adversarial learning has become a core research area for studying robustness and machine learning. Adversarial attacks have expanded well beyond the original setting of imperceptible noise
general notions of robustness, and can broadly be described capturing sets of perturbations that humans are naturally invariant to. These invariants, such as facial recognition should be robust adversarial glasses (Sharif al., 2019) traffic sign classification should be robust to adversarial graffiti (Eykholt et al.) 2018), form the motivation behind many real world adversarial attacks. However, human invariants can also include notions which are not inherently adversarial, for example image classifiers should be robust to common image corruptions (Hendrycks Dietterich 2019) as well changes in weather patterns (Michaelis et 2019).
On the other hand, although there has been much success in defending against small adversarial perturbations, most successful and principled methods for learning robust models are limited to human invariants that can be characterized using mathematically defined perturbation sets. for example perturbations bounded in lp norm. After all, established guidelines for evaluating adversarial robustness (Carlini et al.) (019) have emphasized the importance of the perturbation set (or the threat model) as necessary component for performing proper, scientific evaluations of adversarial defense proposals. However, this requirement makes it difficult to learn models which are robust to human invariants beyond these mathematical sets, where real world attacks and general notions of robustness often be virtually impossible to write down formal set of equations. This incompatibility between existing methods for learning robust models and real-world, human invariants raises a fundamental question for the field of robust machine learning:

---

Published as a conference paper at ICLR 2021
How can we learn models that are robust to perturbations without a predefined perturbation set?
In the absence of mathematical definition, in this work we present general framework for learning perturbation sets from perturbed data. More concretely, given pairs of examples where one is perturbed version of the other, we propose learning generative models that can "perturb" an example by varying fixed region of the underlying latent space. The resulting perturbation sets are well- defined and can naturally be used in robust training and evaluation tasks. The approach is widely applicable to range of robustness settings, as we make no assumptions on the type of perturbation being learned: the only requirement is to collect pairs of perturbed examples.
Given the susceptibility of deep learning to adversarial examples, such a perturbation set will undoubtedly come under intense scrutiny, especially if it is to be used threat model for adversarial attacks. this paper, we begin our theoretical contributions with broad discussion of perturbation sets and formulate deterministic and probabilistic properties that learned perturbation set should have in order to be a meaningful proxy for the true underlying perturbation set. The necessary subset property ensures that the set captures real perturbations, properly motivating its usage as an adversarial threat model. The sufficient likelihood property ensures that real perturbations have high probability, which motivates sampling from a perturbation set as a form of data augmentation. We then prove the main theoretical result, that a learned perturbation set defined by the decoder and prior of conditional variational autoencoder (CVAE) (Sohn et al.) 2015) implies both of these properties, providing a theoretically grounded framework for learning perturbation sets. The resulting CVAE perturbation sets are well motivated, can leverage standard architectures, and are computationally efficient with little tuning required.
We highlight the versatility of our approach using CVAEs with an array of experiments, where we vary the complexity and scale of the datasets, perturbations, and downstream tasks. We first demonstrate how the approach can learn basic fo and srriiiinttranslation---see (RTS) perturbations (Jaderberg et al. 2015) in the MNIST setting. Since these sets can be mathematically defined, our goal is simply to measure exactly how well the learned perturbation set captures the target perturbation set on baseline tasks where the ground truth is known. We next look at more difficult setting which can not be mathematically defined, and learn a perturbation set for common image corruptions on CIFARIO 10 (Hendrycks & Dietterich] 2019). The resulting perturbation set interpolate between common corruptions, produce diverse samples, and be used in adversarial training and randomized smoothing frameworks. The adversarially trained models have improved generalization performance to both in- and nut-of-distribution corruptions and better robustness to adversarial corruptions. In our final setting, we learn perturbation set that captures real-world variations in lighting using a multi- cuttiilllumintion dataset of scenes captured "in the wild" (Murmann et al.] 2019). The perturbation set generates meaningful lighting samples and interpolations while generalizing unseen scenes, and can be used to learn image segmentation models that are empirically and certifiably robust to lighting changes. All code and configuration files for reproducing the experiments as well as pretrained model weights for both the learned perturbation sets as well the downstream robust classifiers are at com/ www.ssaab/pertuuuue
BACKGROUND AND RELATED WORK
Perturbation sets for adversarial threat models Adversarial examples were initially defined as imperceptible examples with small L, l2 and foo norm (Biggio et al. 2013 Szegedy et al. 2013 Goodfellow et al. 2014), forming the earliest known, well-detined perturbation sets that were eventually generalized to the union of multiple fp perturbations (Tramer & Boneh 2019 Maini al. 2019 Croce& Hein 2019: Stutz al., 2019). Alternative perturbation sets to the ?p setting that remain well-defined incorporate more structure and semantic meaning, such as rotations and translations (Engstrom al. 2017). Wasserstein balls (Wong et al. 2019), functional perturbations (Laidlaw & Feizi, 2019), distributional shifts (Sinha et al.) 2017 Sagawa et al. 2019). word embeddings (Miyato et al. 2016), and word substitutions (Alzantot et al. 2018 Jia et al. 2019).
Other work has studied perturbation sets that are not necessarily mathematically formulated but well- defined from a human perspective such as spatial transformations (Xiao et 2018b).) Real-world adversarial attacks tend to try to remain either inconspicuous to the viewer or meddle with features that humans would naturally ignore, such as textures on 3D printed objects (Athalye et al., 2017). graffiti traffic signs (Bykholt et al. 22018), shapes of objects to avoid LiDAR detection (Cao et al.
2

---

Published as a conference paper at ICLR 2021
2019). irrelevant background noise for audio (Li et al. 2019a). barely noticeable films cameras (Li et al. 2019b). Although not necessarily adversarial, Hendryccs & Dietterich (2019) propose the set of common image corruptions as measure of robustness to informal shifts in distribution.
Generative modeling and adversarial robustness Relevant our work is that which combines aspects of generative modeling with adversarial examples. While our work aims to learn real-world perturbation sets from data, most work in this space differs in that they either aim to generate synthetic adversarial Ep perturbations (Xiao et al. (2018).. run user studies to define the perturbation set (Sharif
al. 2.19), or simply do not restrict the adversary at all (Song et al. 2018 Bhattad et al. 2020). Gowal et al. (019)) trained a StyleGAN to disentangle real-world perturbations when no perturbation information is known in advance. However the resulting perturbation set relies on stochastic approximation, and it is not immediately obvious what this set will ultimately capture. Most similar is the concurrent work of Robey et al. (2020), which uses a GAN architecture from image-to-image translation to model simple perturbations between datasets. In contrast to both of these works, our setting requires the collection of paired data to directly learn how to perturb from perturbed pairs without needing disentangle any features translate datasets, allowing us to learn more targeted and complex perturbation sets. Furthermore, we formulate desirable properties of perturbation sets for downstream robustness tasks, and formally prove that a conditional variational autoencoder approach satisfies these properties. This results in a principled framework for learning perturbation sets that is quite distinct from these GAN-based approaches in both setting and motivation.
Ad Adversarial defenses and data augmentation Successful approaches for learning adversarially robust networks include methods which are both empirically robust via adversarial training (Goodfel
low et al.] 2014] 14 Kurakin et al. 2016,] Madry et al., 2017) and also certifiably robust via provable bounds (Wong& & Kolter 2017 Wong et al. 2018 Raghunathan et al. 2018 Gowal et al. 2018
Zhang et al. 2019) and randomized smoothing (Cohen et al.) 2019 Yang et al. 2020). Critically,
these defenses require seathemmtiiclly-dffiee perturbation sets, which has limited these approaches from learning robustness to more general, real-world perturbations. We directly build upon these approaches by learning perturbation sets that can be naturally and directly incorporated into robust training, greatly expanding the scope of adversarial defenses to new contexts. Our work also relates to using non-adversarial perturbations via data augmentation to reduce generalization error (Zhang) et al. 2017 DeVries & Taylor 2017 Cubuk et al.] (219)), which can occasionally also improve robustness to unrelated image corruptions (Geirhos et al., [2018, Hendrycks et al. 2019 Rusak et al., 2020). Our work differs in that rather than aggregating or proposing generic data augmentations, our perturbation sets can provide data augmentation that is targeted for a particular robustness setting.
3 PERTURBATION SETS LEARNED FROM DATA
For an example x E RT a perturbation set S(x) C Rm is defined informally the set of examples which are considered to be equivalent to x, and hence can be viewed as "perturbations" of x. This set is often used when finding an adversarial example, which is typically cast an optimization problem to maximize the loss of a model over the perturbation set in order to break the model. For example, for classifier h, loss function f, and label y, adversarial attack tries to solve the following:
maximize (h(a))) z'ES(z)
A common choice for S(z) is an Ep ball around the unperturbed example, defined as S(x) far + 5 [|b||, S <} for some norm p and radius e. This type of perturbation captures unstructured random noise, and is typically taken with respect to lp norms for p E (0, 2,0,, though more general distance metrics can also be used.
Although defining the perturbation set is critical for developing adversarial defenses, in some scenar- ios, the true perturbation set may be difficult to mathematically describe. In these settings, it may still be possible to collect observations of (non-adversarial) perturbations, e.g. pairs of examples (ar Ø) where ş is the perturbed data. In other words, is a perturbed version of x, from which we can learn an approximation of the true perturbation set. While there are numerous possible approaches one can take to learn S(z) from examples (I, in this work we take a generative modeling perspective, where examples are perturbed via underlying latent space. Specifically, let g IR x Ihm - R
3

---

Published as a conference paper at ICLR 2021
be a generator that takes a k-dimensional latent vector and input, and outputs perturbed version of the input. Then, we can define a learned perturbation set as follows:
S(x) = (0(2, :)): ||=|| S c)
In other words. we have taken a well-defined bounddddded ball in the latent space and mapped it to a set of perturbations with a generator 9. which perturbs x into E via a latent code 2. Alternatively, we can define a perturbation set from probabilistic modeling perspective, and use a distribution over the latent space to parameterize a distribution over examples. Then, S(x) is now random variable defined by probability distribution (=(=) over the latent space as follows:
S(z) ) ~ pe such that 9 = 2),
Pe
(3)
where Pe has support (= ||=|| S c) and pa is distribution parameterized by 0 = 2(2,x))
3.1 GENERAL MEASURES OF QUALITY FOR PERTURBATION SETS
A perturbation set defined by generative model that is learned from data lacks the mathematical rigor of previous sets, so care must be taken to properly evaluate how well the model captures real perturbations. In this section we formally define two properties relating perturbation set data, which capture natural qualities of a perturbation set that are useful for adversarial robustness and data augmentation. We note that all quantities discussed in this paper can be calculated on both the training and testing sets, which allow us to concretely measure how well the perturbation set generalizes to unseen datapoints. For this section, let d IR' I IR be an distance metric (e.g. mean squared error) and let 2, 2 E R. be a perturbed pair, where ã is a perturbed version of x.
To be a reasonable threat model for adversarial examples, one desirable expectation is that a perturba- tion set should at least contain close approximations of the perturbed data. In other words, the set of perturbed data should be (approximately) necessary subset of the perturbation set. This notion of containment can be described more formally as follows:
Definition 1. A perturbation set S(x) satisfies the necessary subset property at approximation error at most for a perturbed pair (z, ž) if there exists an x' E S(x) such that d(x', , S 8.
For perturbation set defined by the generative model from Equation (2. this amounts to finding latent vector z which best approximates the perturbed example ă by solving the following problem:
min d(g(z, 2), Se
(4)
This approximation error can be upper bounded with point estimates or can be solved more accurately with projected gradient descent. Note that nathematicalll--dfinne perturbation sets such lp balls around clean datapoints contain all possible observations and naturally have zero approximation error
Our second desirable property is specific to the probabilistic view from Equation (3), where we would expect perturbed data to have a high probability of occurring under a probabilistic perturbation set. In other words, a perturbation set should assign sufficient likelihood to perturbed data, described more formally the following definition:
Definition 2. A probubilistic perturbation set S(x) satisfies the sufficient likelihood property at likelihood least ê for a perturbed pair (x. 5) if Ep.(:))pp (ā) > 8 where 0 g(2,z).
A model that assigns high likelihood to perturbed observations is likely to generate meaningful samples, which can then be used as a form of data augmentation in settings that care more about average-case over worst-case robustness. To measure this property, the likelihood can be approximated with standard Monte Carlo estimate by sampling from the prior Pe-
VARIATIONAL AUTOENCODERS FOR LEARNING PERTURBATIONS SETS
In this section we will focus on one possible approach using conditional variational autoencoders (CVAEs) to learn the perturbation set (Sohn et al. 20155). We shift notation here be consistent with the CVAE literature and consider a standard CVAE trained to generate x E RM from a latent space E R conditioned on some auxiliary variable y, which is traditionally taken to be label. In our
4

---

Published as a conference paper at ICLR 2021
setting, the auxiliary variable y is instead another datapoint such that æ is perturbed version of y- but the theory we present is agnostic to the choice in auxiliary variable. Let the posterior distribution ((2/2, y). prior distribution p(=ly), and likelihood function p(z|2, y) be the following multivariate normal distributions with diagonal variance:
(2 y) (N,((, y), o² v)), p(zly) ~ ((((v),
p(z|2, v) ~ (()(2,)). (5)
where u(x,y). of (2, y). ((v), o² (v), and g(2, v) are arbitrary functions representing the respective encoder, prior, and decoder networks. CVAEs are trained by maximizing a likelihood lower bound
logp(r y) 2 Ee(=lss.y p(z| y)] - K (a(])z ()(lp()))))
(6)
also known the SGVB estimator, where (-II-))) is the KL divergence. The CVAE framework lends to a natural perturbation set by simply restricting the latent space to an & ball that scaled and shifted by the prior network. For convenience,
will define the perturbation set in the latent space before the reparameterization trick, so the latent perturbation set for all examples a standard & ball be defined
(ty) + v((y). Similarly, probabilistic perturbation by simply truncating the prior distribution at radius é (also before the reparameterization trick).
{u: ||u||2 S <) where
4.1 THEORETICAL MOTIVATION OF USING CVAES TO LEARN PERTURBATION SETS
Our theoretical results prove that optimizing the CVAE objective naturally results in both the necessary subset and sufficient likelihood properties outlined in Section 3.1 which motivates why the CVAE is reasonable framework for learning perturbation sets. Note that these results are not immediately obvious, since the likelihood of the CVAE objective is taken over the full posterior while the perturbation set is defined over a constrained latent subspace determined by the prior. The proofs rely heavily on the multivariate normal parameterizations, with requiring several supporting results which relate the posterior and prior distributions. We give concise, informal presentation of the main theoretical results in this section, deferring the full details, proofs, and supporting results to Appendix A Our results are based on the minimal assumption that the CVAE objective has been trained to some threshold as described in Assumption !
Assumption 1. The CVAE objective has been trained to some thresholds R, Ki as follows
Ea(t) Blo Bogg p00)2, v)) R, KL(g(ele.w)llp(ll))
ΣE
K;
where each K, bounds the KL-divergence of the ith dimension.
Our first theorem, Theorem I states that the approximation error of a perturbed example is bounded by the components of the CVAE objective. The implication here is that with enough representational capacity to optimize the objective, one can satisfy the necessary subset property by training CVAE, effectively capturing perturbed data at low approximation error the resulting perturbation set.
Theorem 1. Let be the Mahalanobis distance which captures 1 - of the probability mass for k-dimensional standard multivariate normal for some O < a < Then, there exists a z such that =-m(y) (5) 2
< and (19(2,)) - xx12 < 8 for
c= Br
Σ Ki, EKe
8
(2R+ m-mmmn mogg(27)
where B is constant dependent on K,, Moreover, as R + m log(2.r) and K; + 0 (the theoretical limits of these bound then - and 8 - O.
Our second theorem, Theorem 2 states that the expected approximation error over the truncated prior can also be bounded by components of the CVAE objective. Since the generator parameterizes multivariate normal with identity covariance, an upper bound on the expected reconstruction error implies a lower bound on the likelihood. This implies that one can also satisfy the sufficient likelihood property by training a CVAE, effectively learning a probabilistic perturbation set that assigns high likelihood to perturbed data.
In practice VAF architectures in general have non-trivial gap from the approximating posterior which may make these theoretical limits unattainable.
5

---

Published as a conference paper at ICLR 2021
Table 1: Condensed evaluation of CVAE perturbation sets trained produce rotation, translation, ans skew transformations on MNIST (MNIST-RTS), CIFARIO common corruptions (CIFARIO-C) and multi-illumination perturbations (MI). The approximation error measures the necessary subset property, and the expected approximation error measures the sufficient likelihood property.
Setting MNIST-RTS
Approx. error
Expected approx. error
CVAE Recon. error
KL
0.11
0.54
0.04
22.2
CIFARIO-C MI
0.005
0.029
0.001
69.3
0.006
0.049
0.004
65.8
Theorem 2. Let be the Mahalanobis distance which captures - a of the probability mass for k-dimensional standard multivariate normal for some < 1. Then, the truncated expected approximation error can be bounded with
1 S -5---- (2R + mlog(2n))
Bp)( [llg(u (oo(u-o(u)--0
where pu) is a multivariate normal that has been truncated to radius r and H is a constant that depends exponentially K, and r.
The main takeaway from these two theorems that optimizing the CVAE objective naturally results in learned perturbation set which satisfies the necessary subset and sufficient likelihood properties. The learned perturbation set consequently useful for adversarial robustness since the necessary subset property implies that the perturbation set does not "miss" perturbed data. It is also useful for data augmentation since the sufficient likelihood property ensures that perturbed data occurs with high probability. We leave further discussion of these two theorems Appendix A.3
EXPERIMENTS
Finally, we present variety of experiments to showcase the generality and effectiveness of our perturbation sets learned with CVAE. Our experiments for each dataset can be broken into two distinct types: the generative modeling problem of learning and evaluating perturbation set, and the robust optimization problem of learning an adversarially robust classifier to this perturbation. We note that our approach is broadly applicable, has no specific requirements for the encoder, decoder, and prior networks, and avoids the unstable training dynamics found in GANs. Furthermore, we do not have the blurriness typically associated with VAEs since we are modeling perturbations and not the underlying image.
In all settings, first train perturbation sets and evaluate them with number of metrics averaged over the test set. We present a condensed version of these results in Table []] which establishes a quantitative baseline for learning real-world perturbation sets in three benchmark settings that future work can improve upon. Specifically, the approximation error measures the necessary subset property, the expected approximation error measures the sufficient likelihood property, and the reconstruction error and KL divergence are standard CVAE metrics. The full evaluation is described in Appendix B and complete tabulation of our results on evaluating perturbation sets can be found in Tables and 10 in the appendix for each setting.
We then leverage our learned perturbation sets into new downstream robustness tasks by simply using standard, well-vetted techniques for &, robustness directly on the latent space, namely adversarial training with an Eg PGD adversary (Madry et al.) 2017), certified defense with le randomized smoothing (Cohen et al., 2019), as well an additional augaettugmntttionn baseline via sampling from the CVAE truncated prior. Further discussion on the meaning of the randomized smoothing certification can be found in Appendix B and the corresponding pseudo-code for all of these
C We defer the experiments on MNIST to Appendix D and spend the remainder of this section highlighting the main empirical results for the CIFARIO and
approaches can be found in Appendix
settings. Additional details and supplementary experiments can be found in Appendix E for CIFAR 10 common corruptions, and Appendix for the multi-illmmination dataset.

---

Published as a conference paper at ICLR 2021
Figure I: Visualization of leamned perturbation trained on CIFAR10 common corruptions. (top row) Interpolations from fog, through defocus blur, to pixelate corruptions. (middle row) Random corruption samples for three examples. (bottom row) Adversarial corruptions that misclassify an adversarially trained classifier at 10.2.
Table 2: Adversarial robustness to CIFARIO common corruptions with a CVAE perturbation set.
Test set accuracy (%)
Test set robust accuracy (%)
Method
Clean
Perturbed
OOD
2.7
3.9
10.2
CIFAR1O-C data augmentation
90.6
87.7
85.0
42.4
37.2
17.8
CVAE data augmentation
94.5
90.5
89.6
68.6
63.3
43.4
CVAE adversarial training
94.6
90.3
89.9
72.1
66.1
55.6
Standard training
95.2 2
67.0
68.1
20.1
17.8
10.1
Fast AutoAugment (Lim et al.
2019
93.9
63.7
79.3
22.6
20.8
10.9
AugMix (Hendrycks et
2019
92.0
68.8
82.9
39.5
34.4
16.8
E2 robust (Engstromm et al.
2019
90.8
74.4
82.8
58.4
48.1
20.6
Eoo robust (Carmon et al.
2019
89.7
71.2
80.3
60.2
50.7
23.6
Computational cost The cost of using learned perturbation sets in downstream tasks is pro- portional to the cost of a forward pass through the generator. For example, training with data augmentation increases the cost by single pass through the generator per example, while adversarial training with I' steps increases the cost by T' passes through the generator. Although generative models can often be quite large in size, find that only modestly sized networks are sufficient for modeling perturbations. In the following experiments, we use small residual (He et al., 2016 and UNet (Ronneberger al. 2013) architectures for the generator. These models are only fraction of the size of the downstream classifier, which dominates the computational cost of training. An experiment exploring the scalability of the CVAE perturbation set higher resolutions (e.g. up to 500 x 750) is discussed in Appendix F.I
5.1 CIFARIO COMMON CORRUPTIONS
In this section, we first learn perturbation which captures common image corruptions for CIFAR1O (Hendrycks & Dietterich, 2 We focus the highest severity level of the blur, weather, and digital categories, resulting corruptions that are unlike random f, noise (a complete description of the setting is in Appendix We find that the resulting perturbation set accurately captures common corruptions with a mean approximation error of 0.005 as seen in Table [ A more in-depth quantitative evaluation as well as architecture and training details are in Appendix
different corruptions which capture more "natural"
We qualitatively evaluate perturbation set in Figure I which depicts interpolations, random samples, and adversarial examples from the perturbation set. For additional analysis of the perturba- tion refer the reader to Appendix F.2 for study using different pairing strategies during training and Appendix E.3 which finds semantic latent structure and visualizes additional examples.
We note that this is not the original intended use of the dataset, which was proposed general measure for evaluating robustness. Instead, using the dataset for different setting of learning perturbation sets.
7

---

Published as a conference paper at ICLR 2021
Figure Visualizations from a learned perturbation trained on the multi-llumination dataset. (top row) Interpolations between different lighting angles. (bottom row) Random lighting perturbations for two scenes.
Figure 3: Pairs of MI scenes (left) and their adversarial lighting perturbations (right).
Robustness to corruptions We next employ the perturbation set in adversarial training and random- ized smoothing to learn models which are robust against worst-case CIFARIO common corruptions. We report results at three radius thresholds (2.7,3.9, 10.2) which correspond to the 25th, 50th, and 75th percentiles of latent encodings as described in Appendix E.3 We compare to two data augmen- tation baselines of training on perturbed data or samples drawn from the learned perturbation set, and also evaluate performance three extra out-of-distribution corruptions (one for each weather, blur, and digital category denoted OOD) that are not present during training.
We highlight some empirical results in Table 2 where we first find that training with the CVAE perturbation set can improve generalization. Specifically, using the CVAE perturbation set during training achieves 3 5% improved accuracy training directly the common corruptions (data augmentation) across all non-adversarial metrics. These gains motivate learning perturbation sets beyond the setting of worst-case robustness as way improve standard generalization. Additionally, the CVAE perturbation set improves worst-case performance, with the adversarially trained model being the most robust 66% robust accuracy for e 3.9 whereas pure data augmentation only achieves 17.8% robust accuracy. Finally, we include a comparison to models trained with standard training. AugMix data augmentation. foc adversarial training, and lg adversarial training, none which can perform as well as our CVAE approach. We note that this is not too surprising, since these approaches have different goals and data assumptions. Nonetheless, we include these results for the curious reader, with additional details and discussion in Appendix [E.5 For certifiably robust models with randomized smoothing, we defer the results and discussion to Appendix E.6
5.2 MULTI-ILLUMINATION
Our last set of experiments looks at learning perturbation set that captures multiple lighting conditions using the Multi-llumination (MI) dataset (Murmann et al.) 2019). These consist of thousand scenes captured in the wild under 25 different lighting variations, and goal is to learn perturbation set which captures real-world lighting conditions. Architecture and training specifics can be found in Appendix [.. and our perturbation set accurately captures real-world changes in lighting that generalizes to new, unseen scenes with a low approximation error 0.006 as seen in Table [ Qualitatively, the perturbation set can produce interpolations between lighting angles well as new lighting variations from random samples as shown in Figure 2 This allows us to capture a continuous range of lighting changes, a significantly larger set than the discrete angles manually collected in the dataset. Additional samples and interpolations can be found in Appendix F2
Robustness to lighting perturbations The MI dataset contains material map annotations for each scene such as wood, plastic, and fabric, and so natural downstream task is to predict materials in each scene. Specifically, for each pixel, predict one of 40 different material types generate material segmentation map. Our goal is to generate material segmentation maps which are adversarially robust to lighting perturbations, using CVAR perturbation set. Several adversarial examples can be
8

---

Published as a conference paper at ICLR 2021
Table 3: Learning image segmentation models that are robust to real-world changes in lighting with
CVAE perturbation set.
Test set accuracy (%)
Test set robust accuracy (%)
Method
Perturbed
7.35
8.81
17
Fixed lighting angle MI data augmentation
37.2
26.3
24.2
14.9
45.2
38.0
36.5
27.1
CVAE data augmentation
41.5
35.5
33.9
24.7
CVAE adversarial training
41.7
39.4
38.8
35.4
found in Figure +]
which remain reasonable lighting perturbations despite being adversarial, and additional examples as well as their corresponding segmentation maps can be found in Appendix F3
Table summarizes the material segmentation results for adversarial lighting perturbations. We report robustness at thresholds E {7.35, 8.81, 17) which correspond to the 50th, 75th, and 100th percentiles of the latent encodings for real data described in Appendix [F.T] We find that adversarial training can improve robustness to worst-case lighting perturbations over directly training on the perturbed examples, increasing robust accuracy from 27.1% to 35.4% at the maximum radius e = 17. However, training with the learned perturbation set does reduce performance this setting. This can be seen as the cost adversarial robustness, which is a common tradeoff when training adversarially robust models. However, this reduction in perturbed accuracy can also be as reduced overfitting to the discrete camera angles in the perturbed dataset, since the perturbation set captures continuous spectrum of lighting changes that are not explicitly collected. Additional results on certifiably robust models with randomized smoothing can be found in Appendix F3
the perturbed data in
6 CONCLUSION
In this paper, presented general framework for learning perturbation sets from data when the perturbation cannot be mathematicalll--deiin We outlined deterministic and probabilistic properties that measure how well perturbation fits perturbed data, and formally proved that a perturbation set based upon the CVAE framework satisfies these properties. This work establishes a principled baseline for learning perturbation improve upon. Different approaches may result in better perturbation sets, for example by using more powerful generative models such as autoregressive models and generative adversarial networks, or by improving the training procedure.
with quantitative metrics which future work can potentially
Our framework of learning perturbation sets opens up a diverse range of new downstream robustness tasks that more accurately reflect real-world changes. Although in this work we focused primarily on adversarial and certifiable robustness to common image corruptions and lighting perturbations for vision tasks, our framework is not specific to these tasks and so new perturbation sets could potentially be Iearned and leveraged in robustness problems found in other domains. Our work may also have implications beyond the adversarial regime, can sometimes improve non-adverss aonvarrrsaaiall performance to natural perturbations, though this is not necessarily guaranteed. Overall, our work opens pathway for practitioners to learn machine learning models that are robust to targeted, real-world perturbations that can be collected as data.
found that Icarned perturbation set
REFERENCES
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho. Mani Srivastava, and Kai- Wei Chang. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07999 2018.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint arXiv:1707.07333 2017.
Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and David Forsyth. Unrestricted adversarial examples via semantic manipulation. International Conference on Learning Representations, 2020.

---

Published as a conference paper at ICLR 2021
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Yulong Cao, Chaowei Xiao, Dawei Yang. Jing Fang, Ruigang Yang, Mingyan Liu, and Bo Li. Adver- sarial objects against lidar-based autonomous driving systems. arXiv preprint arXiv: 1907. 054 2019.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 2019.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang. Unlabeled data improves adversarial robustness, In Advances in Neural Information Processing Systems, pp. 11192-1123,, 2019.
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint 1902.02990 2019.
Francesco Croce and Matthias Hein. Provable robustness against all adversarial (-p-perturbations for p 1. arXiv preprint arXiv:195..11213, 2019.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 113--23, 2019.
Terrance Deesies and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1700:04582.. 2017.
Logan Engstrom, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. rotation and translation suffice: Fooling cnns with simple transformations. arXiv preprint 1rxivv17/2.022799, 1 (2):3, 2017.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL http:tttt///ttttiiee com/ cat.///aaary.abbrouuseeeee
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-worles attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1625-1634, 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint 1rXiv:1811.1222 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint
2018.
Sven Gowal, Chongli Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Timothy Mann, and Pushmeet Kohli. Achieving robustness in the wild via adversarial mixing with disentangled representations. arXiv preprint arXiv: 1912.03 092,0 2019.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: /2261, 2019.
10

---

Published as a conference paper at ICLR 2021
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 1711::111..02811,00 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adversarial word substitutions. arXiv preprint
2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint 161vv:661..022566 2016.
Cassidy Laidlaw and Soheil Feizi. Functional adversarial attacks. In Advances in Neural Information Processing Systems, pp. 10408-10418, 2019.
Juncheng Li, Shuhui Qu, Xinjian Li, Joseph Szurley, J Zico Kolter, and Florian Metze. Adversarial music: Real world audio adversary against wake- word detection system. Advances Neural Information Processing Systems, pp. 11908-1181 11918, 2019a.
Juncheng B Li, Frank R Schmidt, and J Zico Kolter. Adversarial camera stickers: A physical camera attack on deep learning classifier. arXiv preprint arXiv:1904.00759 2019b.
Sunghin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706. 06083, 2017.
Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. arXiv preprint arXiv: 140..00099, 2019.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak. Oliver Bringmann. Alexan- der S Eckerr Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv: (eXXiv:907.074500 /907.07484, 2019.
Takeru Miyato, Andrew M Dai. and Ian Goodfellow. Adversarial training methods for semi semi-supervised text classification. arXiv preprint 1rXiv:6605.07725. 2016.
Lukas Murmann, Michael Gharbi, Miika Aittala, and Fredo Durand. A dataset of multi-illume illuminatiinnn images in the wild. In Proceedings the IEEE International Conference Computer Vision, pp- 4089-4089, 2019.
Aditi Raghunathan, Jacob Steinhardt, and Perey Liang. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344 2018.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. arXiv preprint arXiv:2002. 1569, 2020.
Alexander Robey, Hamed Hassani, and George Pappas. Model-based robust deep learning, arXiv preprint arXiv:2005.100 10247, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against image corruptions by playing the game of noise. arXiv preprint arXiv:2001.0605) 2020.

---

Published as a conference paper at ICLR 2021
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.087330 2019.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. A general framework for adversarial examples with objectives. ACM Transactions on Privacy and Security (TOPS), 22(3): 1-30, 2019.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprini arXiv: 2017.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464-472. IEEE. 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp. 3483-3491, 2015.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems, pp. 8312-8323, 2018
David Stutz, Matthias Hein, and Bernt Schicle. Confidence--alibratted adversarial training and detection: More robust models generalizing beyond the attack used during training. arXiv preprint arXiv:1910.06259, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312..699, 2013.
Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. In Advances in Neural Information Processing Systems, pp. 5858-5868, 2019.
Eric Wong and J Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv preprint arXiv:1711.00855 2017.
Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and Zico Kolter. Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems, pp.
2018.
Eric Wong, Frank R Schmidt, and J Zico Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. arXiv preprint 1rXiv:1902.07990 2019.
Chaowei Xiao, Bo Li, Jun- Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. arXiv preprint 1r01vv/801.0200 2018a.
Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. arXiv preprint arXiv: 020122. 2018b.
Greg Yang, Tony Duan, Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li. Randomized smoothing of all shapes and sizes. arXiv preprint arXiv:2002.081188 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 4rX1v:1605.07146. 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.094122 2017.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint 2019.
12

---

Published as a conference paper at ICLR 2021
A THEORETICAL RESULTS
In this section, we present the theoretical results in their full detail and exposition. Both of the main theorems presented in this work require number of preceding results in order formally link the prior and the posterior distribution based on their KL divergence. We will present and prove these supporting results before proving each main theorem.
A.1 PROOF OF THEOREM 1
Theorem 1 connects the CVAE objective to the necessary subset property. In order to prove this, we first prove three supporting lemmas. Lemma] states that if the expected value of function over normal distribution is low, then for any fixed radius there must exist point within the radius with proportionally low function value. This leverages the fact that the majority the probability mass concentrated around the mean and can be characterized by the Mahalanobis distance from the mean of the normal distribution. Specifically, for a given 0 I a S 1, r is the Mahalanobis distance which captures of the probability mass of (0, T) if Jiwllass p(u)du = a.
Lemma 1. Let f(u) R* + R4 be non-negative integrable function, and let (0, be k-dimensional standard multivariate normal random variable with zero mean and identity covariance. Suppose (o, (f [f)) for some 8 O. Then, for any pair (r, a) where is the Mahalanobis distance which captures 1 of the probability mass of (0, f), there exists such that ||u||2 S and f(u)= I-aa
Proof. We will prove this by contradiction. Assume for sake of contradiction that for all such that [|ul|2 S r, we have f())= > T We divide the expectation into two integrals over the inside and outside of the Mahalanobis ball:
f(u)p(u)du + I
Eu=N((,n)) [f(w)]= !ullase
(())
Using the assumption the first integral and non-negativity of f in the second integrand, we can conclude
Jullarr 1 a p(u)du= 
(o,, [f(u)]
where the equality holds by definition of the Mahalanobis distance, which contradicts the initial assumption that (o [f( < 6. Thus, we have proven by contradiction that there exists a such that ||u]|a and f(u)= S
Our second lemma, Lemma is an important result that comes from the algebraic form of the KL divergence. It is needed to connect the bound on the KI. divergence to the actual variances of the prior and the posterior distribution, and uses the LambertW function do so.
Lemma 2. Let Wy be the LambertW function with branch k. Let x. > O, and suppose loog S y. Then, I E [a, b) where a = -Wo Wo(-e-w) and b = -W-1(-e-))}. Additionally, these bounds coincide
when y
Proof. The LambertW function W₂ (v) is defined as the inverse function of y = ze", where the path multiple solutions is determined by the branch We can then write the inverse of y = x log as one of the following two solutions:
x E (-Wo( -
Since - log is convex with minimum at = 1, and since the two solutions surround = 1.
the set of points which satisfy x log S y precisely the interval Wolee "). -W- Evaluating the bound at = 1 completes the proof.
Lemma is the last lemma needed to prove Theorem I which explicitly bounds terms involving the mean and variance of the prior and posterior distributions by their KL distance, leveraging Lemmal to bound the ratio of the variances. The two quantities bounded in this lemma will used the main theorem to bound the distance of point with low reconstruction error from the prior distribution.
13

---

Published as a conference paper at ICLR 2021
Lemma 3. Suppose the KL distance between two normals is bounded, KL(N(H, of)ll oB)) & K for some constant K. Then,
(#1 - u2)² 1 - K
02
and also of E (a, where
dg
a a= -Woo-e-
b b= b--W-i(-e
Proof. By definition of KL divergence, have
1 log o K
(1) - )22)² +
(10)
02
0
Since log 2 for x 2 0, we apply this for x prove the first bound on the squared distance as
(#1- - n2) of k
(11)
Next, since (M1 u222 02 2 0, can bound the remainder as the following
of
log
2 S +1
(12)
2
of and using Lemma bound
(a, b] where
of
a = -Wol -Wo(-e-(ktl)) b= b (KK+))
With these three results, we can now prove the first main theorem, which we is presented below in its complete form, allowing us to formally tie the CVAE objective to the existence of nearby point with low reconstruction error.
Theorem 1. Consider the likelihood lower bound for x RD from the conditional VAE k- dimensional latent spuce conditioned on some other inpul y. Let the posterior, prior, and decoder distribations be standard multivariate normals with diagonal covariance as follows
q(2|x1 v) N((xx, y), o² ((2,y)) p(zlu) ~ N(u(y), (y)), p(x|z, ~ N(ll,,
resulting in the following likelihood lower bound:
log p(x y) 2 Suppose we have trained the lower bound to some thresholds R, K;
p(1|2, v)] - (L(g(=/ w)ll=()l)
(13)
(ogl)e.))oogprrt 2 R. vllp(ee))
K
where K₃ bounds the KL-divergence of the ith dimension. Let be the Mahalanobis distance which captures! 1 a of the probability mass for a k-dimensional standard multivariate normal for some
0 0<ac <<ck 1. Then, there exists a such that || [|==(()] ||2 & and [|g(2, 3) - 2113 < f
Br ΣKi Ki, 8
(2R m (og()
where B constant dependent on K,, Moreover, as + log(2)) and K; + (the theoretical limits of these bounds, e.g. by training), then and 0.
14

---

Published as a conference paper at ICLR 2021
Proof. The high level strategy for this proof will consist of two main steps. First, will show that there exists a point near the encoding distribution which has low reconstruction error, where we leverage the Mahalanobis distance to capture nearby points with Lemma[] Then, we apply the triangle inequality to bound its distance under the prior encoding. Finally, we will use Lemma bound remaining quantities relating the distances between the prior and encoding distributions to complete the proof.
Let z(u) u-o (xt p y) be the parameterization trick of the encoding distribution. Rearranging the log probability in the assumption and using the reparameterization trick, we get
Eu -| - g(((u), ()1 I -2 - m logg(22))) (2x(()x))))) = 6(1 - a)
(14)
Applying Lemmal 1 there must exist a such that ||w|| S and || - g (u(), w S ó, and so =(u) satisfies the reconstruction criteria for We will now show that z(u) fulfills the remaining criteria for e: calculating its & norm under the prior distribution and applying the triangle inequality, we get
=(u)= (vv (1)
u o oz, v) + plx.v) mv))
u u-d N..0(x.)) v)
pl y) -m()
+
(v)
(lv)
2 (15)
We use Lemma on the KL assumption bound the following quantities
(x
(14(v) (1 "))² o7(w) S Ki,
[ac, bi]
(16)
of
where (a,, by] are as defined from Lemma
3
Let B = max, Vbg, so dy(y)
< B for all i. Plugging this in along with the previous bounds we get
the following bound on the norm of z(u) before the prior reparameterization:
a(u) u(w)
IKi=
Br +
(17)
(v)
Thus, the norm (before the prior reparameterizzation) and reconstruction error of 2(u) can be bounded by e and 6.
To conclude the proof, we note that from Lemma 2 K; 0 for all implies B + 1, and so e ++ r.
Similarly by inspection, R + - -m (2x) implies that 5 -> 0, which concludes the proof. Li
A.2 PROOF OF THEOREM 2
In this section, we move on to prove the second main result of this paper, Theorem 2 which connects the CVAE objective the sufficient likelihood property. The proof for this theorem is not immediate, because since the generator is an arbitrary function, two normal distributions which have a difference in means have an exponentially growing ratio in their tail distributions, and so truncating the normal distributions is crucial. This truncation is leveraged in Lemma 4 which bounds the ratio of two normal distributions constrained within an & ball, and is what allows us to connect the expectation over the prior with the expectation over the posterior.
Lemma 4. Let (O, 1) and ~ NN(=u/0², 1/0²). Then,
q(2)
1[ + wl < < h(r, p, O,)).
(18)
p(2)
Furthermore if n²2 < K and a E |a, bj. then h(r, μ, σ) < be max where
G = (b² - 1)- K. C₂ बه
(1 (1- a² + B
Proof. The proof here is almost purely algebraic in nature. By definition of q and p. we have
(2) = oe -f(((=z)))-=3 = ((1 (((0-oo)=--2pp=-00
(19)
p(2)
15

---

Published as a conference paper at ICLR 2021
We will focus on bounding the exponent, (1 - p². We can bound this by considering
two cases. First, suppose that o² < O and so the exponent a concave quadratic. Then, the maximum value of the quadratie is at its root:
Tp 2(1 o²)== 20y 0 = 1 o2
(20)
Further assume that this z is within the interval the indicator function, so 1- = Then, plugging in the maximum value into the quadratic results in the following bound fi for this case:
1- S r.
o²p² 02- - (o² 1)r² 1²² = fr, (21)
(1-00)2²- 201z - -1 -1 0²)
Consider the other case, so either o² 2 0. the optimal value for z" the previous case when 1 1-0² < 0 is not within the interval oz + Sr. r. Then, the maximum value of this quadratic must
occur at [oz + μ = T, S0 - Plugging this into the quadratic for positive r. this results in
(1 -²)² - 20nz - 1² ostu=r o² ((1 - -o²))² - 2r + 12)
(22)
and for negative r, this
(1 - )ー 20p2
σ² ((1 02) + 2r + 122
(23)
lasty
and bound this with the following function fz:
1
(1- o2 20μz 2² s 02 - - + 2/mp| + n²² = fa(r, u.,
(24)
Plugging in the maximum over both cases forms our final bound on the ratio of distributions.
((2) 1(loz + ml I
h(r, p)
(25)
p(2)
To finish the proof, assume we have the corresponding bounds μ and Then, the first case can be bounded with C₁ defined as
f(),p,)) S (b² 1)r² K K C1
The second case can be bounded with C2 defined as
fa(r.1.)) S az ((1 a² 2r + k)) C2
And thus we can bound hr,, (1, as
h(r,4,0) < be max( H
Lemma can be directly applied to the setting of Theorem 2 for the case of non-conditional VAE with a standard normal distribution for the prior. However, since we are using conditional VAEs instead, the prior distribution has its own mean and variance and so the posterior distribution needs to be unparameterized from the posterior and reparameterized with the prior. Corollary [ establishes this formally, extending Lemma to the conditional setting.
Corollary 1. Let p ~ ((,0 1) and q ~ N
P2-m #1
ot
Then,
q(2)
012 + 11
P2
Sr) < h((.u,))
(26)
p(=)
where μ ( 12) and L. Furthermore, KLNN( INW oï)) < K for some constant K, then h(r, 0) < H where H is constant which depends K.
as
16

---

Published as a conference paper at ICLR 2021
Proof. First, note that Lemma iimlies that μ² v K and a E [a, Observe that q N (-# ديي
and -1 o2 + 14, and so the proof reduces an application of Lemma to this particular j and a.
2
These results allow us to bound the ratio of the prior and posterior distributions within fixed radius, which will allow us to bound the expectation over the prior with the expectation over the posterior. We can now finally prove Theorem 2, presented in its complete form below.
Theorem 2. Consider the likelihood lower bound for x E R from the conditional VAE in k- dimensional latent space conditioned on some other input y. Let the posterior, prior, and decoder distributions be standard multivariate normals with diagonal covariance as follows
q(2|0, v) N(plæ y), o² (ar, w)), p(zly) ~N(uu(), p(z|z,)) N(g(z,y),
resulting in the following likelihood lower bound:
log p( y) 2 Ee(aaa.,y p(æ) 2, 2) K (L(g(= (llp(()))) Suppose we have trained the lower bound to some thresholds K;
(27)
Bog R, KL(g(=/2, (I(a()lo.@lll<<e)))) S Σ K;
where K₂ bounds the KL-divergence of the ith dimension. Let r be the Mahalanobis distance which captures 1 a of the probability mass for k-dimensional standard multivariate normal for some 0 < a < 1. Then, the truncated expected reconstruction error can be bounded with
1 Bp-(u) [[g(u -y)) + +(v).)) - =122 1-a
(2R mlg((22)
where (u( is multivariate normal that has been truncated to radius and H is constant that depends exponentially on K,.
Proof. The overall strategy for this proof will be follows. First, will rewrite the expectation over a truncated normal into an expectation over standard normal, using an indicator function to control the radius. Second, we will do an "unparameterrizatoon" of the standard normal match the parameterized objective in the assumption. Finally, we will bound the ratio of the unparameterized density over the normal prior, which allows assumption. This last step to bound the ratio of densities is made possible by the truncation, which would otherwise grow exponentially with the tails of the distribution.
bound the expectation over the prior with the
For notational simplicity, let f(-) = [1 v) x=122- The quantity we wish to bound can be rewritten
using the Mahalanobis distance as
Ep, [[(u - (1)) + (())) T 1 I. s(u o(y) + +()) S <))(dddd
(28)
where we used the fact that p. (u) = 1 S r)p(u), which simply rewrites the density of truncated normal using scaled standard normal density and an indicator function. We can do parameterization trick ooy) + v(v) to rewrite this
- / f(2)1 (1
m(v) o(v) (1)))))
Ep. [f(u (g) + (())) = 1
sr)
(2)y
(29)
followed by reverse parameterization trick with =-u(z.0) o(a.y)
to get the following equivalent
expression
1 1- a - f(v-0(2,0)) ((x,
(|0-o a(x, y) + m(x, m(v) sr) po(w)dd
(30)
(ly)
where ~ ~N( (M(a)-p(a.n) ~V(Mee)=nta))
(((..)))
For convenience, we can let
pv(v)= =1 (l)
m(w) v ((x v) + plx, v) -
sr) po(v))
o(v)
17

---

Published as a conference paper at ICLR 2021
which can be interpreted truncated version of pe, and the expectation can be represented more succinctly as
ER (f oly) p(v)) - T- f f(v o(x, y) + p(a,,))).
1
(31)
We will now bound (v) with the standard normal distribution p(v) that can apply assumption. Because f is non-negative, the & constraint in (x) can be relaxed to an clement-wise loo constraint to get
v 0.0((z 3) + plx, y) p(v)
Po((u)) p(u:)
Po( S p(u)
II 1=1
2r)
(32)
(2y
where we also used the fact that p a diagonal normal, so II p(ve) p(u). Each term this product can be bounded by Lemma to get
p.(1) S p(u) H H
(33)
where H, is defined in Corollary for each using the corresponding K,, and we let H Thus we can now bound the truncated expected value by plugging in our bound for Pa (v) into Equation
I
- H₃,
to get
Ep,
[f(u (v) + p(y)) - -
Ex())) [f())]
(34)
Using our lower bound on the expected log likelihood from the assumption, can bound the remaining integral with
Ba(((a [f(=)] < -2R m log(2x)
(35)
and so combining this into our previous bound from Equation 34 results in the final bound
Ep, [.([(w)]] S 1 (2R + m log(2x))
(36)
Finally, while Theorem implies a bound on the the expected reconstruction error, the sufficient likelihood property is defined with respect the likelihood. For multivariate Gaussian, reconstruc- tion error and likelihood are closely linked, and so the upper bound on reconstruction error directly implies lower bound the likelihood as described in Corollary 2
Corollary 2. Suppose the truncated reconstruction error of perturbed pair (Ir,)) is upper bounded by U:
Epp((w) [llg(u a(x) + p(x), x) 0 SU
(37)
Then, the likelihood of ă conditioned of x is lower bounded by L:
Ep.(() PP (8) L
(38)
where L. exp (- m (2x) 4)
Proof. The results follows by first applying Jensen's inequality and expanding the density function
Ep.(u) (f)] 2 exp (E).(u) [log Pu. (i)))
(39)
exp
log (2T))
(m) [llg(u (()) x)x), -
(40)
U
exp
log (2Tr)
(41)
(42)
18

---

Published as a conference paper at ICLR 2021
62(2)
a
-0
Figure 4: A simple example demonstrating how the expected value of a function can tend to zero while the maximum tends to infinity as a + cc.
A.3 DISCUSSION THEORETICAL RESULTS
We first note that the bound on the expected reconstruction error in Theorem 2 is larger than the corresponding bound from Theorem by an exponential factor. This gap is to some extent desired, since Theorem I characterizes the existence of highly accurate approximation whereas Theorem characterizes the average case, and so if this gap were too small, the average case allowable perturbations would be constrained in reconstruction error, which is not necessarily desired in all settings.
2
This relates to the overapproximation error, the maximum reconstruction error within the per- turbation set. Initially one may think that having low overapproximation error to a desirable property of perturbation sets, in order constrain the perturbation from deviating by "too much." " However, perturbation sets such as the olaaiin-translationnskew perturbations for MNIST will always have high overapproximation error, and so reducing this is not necessary desired. Furthermore, not theoretically guaranteed for the CVAE to minimize the overapproximation error without further assumptions. This because is possible for function to be arbitrarily small in expectation but be arbitrarily high at some nearby point, so optimizing the CVAE objective does not imply low overapproximation error. A simple concrete example demonstrating this is the following piecewise linear function
2 +
for
<
(x(2)
a t
+
for 0 S z +
0
otherwise
for a > seen in Figure Note that max | Se (2)
which unbounded as a -+ co
for all 2 0, while at the same time 1) overapproximation error is not necessarily desired nor it guaranteed by the CVAE. Nonetheless, report it anyways to give the reader a sense of how much the CVAE can deviate within the learned perturbation set.
I 1 + Thus, explicitly minimizing
B EVALUATING A PERTURBATION SET
In this section we describe more detail the process selecting radius for a learned perturbation and formally describe all the details regarding the evaluation metrics for a perturbation with CVAE. As a reminder, let μ(æ, ž), a (xx T be the posterior encoding, ((x)), a (z) be the prior encoding, and g(x, 2) be the decoder for the CVAE. We will use u to represent the latent space before the parameterization trick. Then the perturbation set defined by our generative process is
learned
S(x) = {g(2, x) : 2 u (x) - u(x), ||w|| S <}
and the probabilistic perturbation set is defined by the truncated normal distribution No before the parameterization trick as follows,
S(x) x)) z u.)()))) m(x), a~ No,
19

---

Published as a conference paper at ICLR 2021
Selecting radius To select a radius, we take the following conservative estimate calculated held-out validation set, which computes the smallest Ey ball under the prior which contains all the mean encodings under the posterior:
pa 5.) p(x)
max
(xx)
The benefits of such a selection is that by taking the maximum, we are selecting learned perturbation set that includes every point with approximation error as low as the posterior encoder. To some extent, this will also capture additional types of perturbations beyond the perturbed data, which can be both beneficial and unwanted depending on what is captured. However, in the context of adversarial attacks, using perturbation set which is too large generally more desirable than one which is too small, in order to not underspecify the threat model.
Evaluation metries We present each evaluation metric in detail for single perturbed pair (a, 2). These evaluation metrics can then be averaged over the test dataset to produce the evaluations presented in this paper.
1. Encoder approximation error (Enc. AE) We can get fast upper bound of approximation error by taking the posterior mean. unparameterizing with respect the prior, and projecting it to the ball as follows:
Enc. AE(æ, ū) || g(2,( 21
where 2 = u ozz) + plx))
u = =Proj, p 5) m(x) o (④x)
2. PGD approximation error (PGD AE) We can refine the upper bound by solving the following problem with projected gradient descent:
PGD AE( .) min || g(2,2)112
|
where
o (z) u(x)
In practice, implement this by warm starting the procedure with the solution from the encoder approximation error, and run 50 iterations of PGD at step size e/ e/20.
3. Expected approximation error (EAE) We can compute this by drawing samples u, ~ (0, and calculating the following Monte Carlo estimate:
N
EAB(a, 3) =
Σ - g(2,,,))) x)
N
where 2; = u4 (x) + (a())
In practice, we find that N = is sufficient for reporting means over the dataset with near-zero standard deviation.
4. Over approximation error (OAE) We can compute this by performing typical 6g PGD adversarial attack: OAE(z, (z, Ø) max _ - g(2,z)|1 [|u
where = w (I) + μ(zr
In practice, we implement this by doing random initialization and run 50 iterations of PGD step size e/20.
5. Reconstruction error (Recon. err) This is the typical reconstruction error of variational autoencoder, which is Monte Carlo estimate over the full posterior with one sample:
Recon. err(x, .) =
m | 2(2,0r)11
where z = u oxx x) + p(x.
I0,,
20

---

Published as a conference paper at ICLR 2021
Note that report the average all pixels be consistent with the other metrics in this paper, however is typical to implement this during training as a sum of squared error instead of a mean.
6. KL divergence (KL) This is the standard KL divergence between the posterior and the prior distributions
KL(x,5) od
C ADVERSARIAL TRAINING, RANDOMIZED SMOOTHING, AND DATA AUGMENTATION WITH LEARNED PERTURBATION SETS
this section, we describe how various standard, successful techniques in robust training can applied to learned perturbation sets. We note that each method is virtually unchanged, with the only difference being the application of the method to the latent space of the generator instead directly in the input space.
Algorithm 1 Given a dataset D, perform an epoch of adversarial training with learned perturbation set given by generator 9 and a radius c with step size using PGD adversary with T steps and step size a
for (x, y) E do
δ 0 // Initialize perturbation
for / E T do
g VgL(h(g(z + 6, x)). Calculate gradient
6 = 8 + a g/ //ll9 // Take gradient step
if ||00||2 then
8 = 5/ // Project onto ball
0 0 7 (h 5,x)), v) // Optimize model weights
C.I ADVERSARIAL TRAINING
Adversarial training, or training on adversarial examples generated by an adversary, is a leading empirical defenses for learning models which are robust to adversarial attacks. The adversarial attack is typically generated with PGD adversary and is used in the context of l, adversarial examples. We follow closely the adversarial training formulation from (Madry et al., 2017). In order to do adversarial training on learned perturbation sets, it suffices to simply run the PGD adversarial attack in the latent space of the generator, as shown in Algorithm
C.2 RANDOMIZED SMOOTHING
Randomized smoothing is method for learning models robust adversarial examples which come with certificates that can prove (with high probability) the non-existance of adversarial examples. We follow closely the randomized smoothing procedure from (Cohen al. 2019), which trains and certifies network by augmented the inputs with large Gaussian noise. In order to do randomized smoothing on the learned perturbation sets, it suffices to simply train, predict, and certify with augmented noise in the latent space of the generator, as shown in Algorithm 2 for prediction and certification.
The algorithms are almost identical to that propose by Cohen et al. 2019). with the exception that the noise passed to the generator before going through the classifier. Note that LOwER CONFI- DENCEBOUND(K)
- a)) returns a one-sided 1 - confidence interval for the Binomial parameter BINOMIAL(n.pp) p). and BINOMPVALUE(a.. + b,p) returns the value of
p given a sample two-sided hypothesis test that (I + b.p)
On the meaning of "certified" robustness to a learned perturbation set Note that the training and certification procedure with randomized smoothing operates solely on the learned perturbation set. Strictly speaking, this implies that the meaning of this certification only as good as the quality of the learned perturbation set.
21

---

Published as a conference paper at ICLR 2021
Algorithm 2 Given a datapoint z, pseudocode for certification and prediction for classifier which has been smoothed over learned perturbation set given by generator g and radius using noise level a with probability at least
a.
function SAMPLEUNDERNosSEth g, x. O) 0 for E (k) // Initialize counts
for / E [mi do
// Draw random samples from the generator and count classes for prediction
2 = (0, o²)
j arg max; hl)(2,x))
Cj Cj
return
function PREDICT(h, 9. n, a,)
= SAMPLEUNDERNOISIO.... 2,
1 == top two indices in e
if BINOMPVALUE(C,, C + Cy, 0.5) then
return prediction
else
return ABSTAIN
function CERTIFY(h, I, no. n. 0,
Cu SAMPLEUNDERNOISEEH..o v, n))
k: = arg max, (co);
6 SAMPLLEUNDDENNOSSSC....... σ)
Pa := LOVERCCONTDENNEBBUNNCCC
())
Pa. 0.5 then
return prediction & with radius a (pa)
else
return ABSTAIN
22

---

Published as a conference paper at ICLR 2021
For example, if a learned perturbation set contains exactly the real perturbed examples, then certified accuracy according to this perturbation set is precisely an upper bound on the worst case adversarially perturbed example. In practice, the generative model produces inexact approximations, and so this is an upper bound on the performance to the synthetically generated perturbations. As the generative model becomes more accurate, the certified aceuracy becomes more reflective of real world performance. Careful evaluation of these generative models is necessary if learned perturbation sets are to be used as proxies for measuring performance in the real world.
D MNIST
In this section, we discuss some benchmark tasks on MNIST, where the perturbation set is mathematically-defnnel Since exact perturbation set will always be better than learned ap- proximation, the goal here is simply to explore the data and architectural requirements for learning accurate perturbations sets with CVAEs, and establish baseline where the ground truth is known. We consider learning the classic for perturbation and rotation-translation-sso (RTS) perturbations (Jaderberg et al., 2015).
We use the standard MNIST dataset consisting of 60,000 training examples with 1,000 examples randomly aside for validation purposes, and 10,000 examples the test set. These experiments were run on single GeForce RTX 2080 Ti graphics card, with the longest perturbation set taking hour train.
D.1 EXPERIMENTAL DETAILS
to details For the foe setting, perturbations with radius 0.3. Our fully connected network encoders have one hidden layer with 784 hidden units, then two linear modules with 784 outputs each for computing the mean and log variance of the latent space distributions. The generator network has the same structure, one hidden layer with 784 hidden units. The posterior and generator networks simply concatenate their inputs into a single vector.
The encoders for the convolutional network for the foc setting uses two hidden convolutional layers with 32 and 64 channels, each followed by ReLU and max pooling. To generate the mean and log variance, there are two final convolution with 128 channels. The posterior network concatenates its inputs via the channel dimension. The generator has two analogous hidden layers using convolutions with 64 and 32 channels followed by ReLU and nearest neighbor upsampling. One final convolution reduces the number channels to 1, and the conditioned input x is downsampled with 1 x 1 convolutions and concatenated with the feature map before each convolution. All convolutions other than the downsampling convolutions use a 3 x 3 kernel with one padding.
Both networks trained for 20 epochs, with step size following piece-wise linear schedule of [o, 0.001, 0.0005, 0.0001] over epochs [o, 10, 15, 20) with the Adam optimizer using batch size 128. The KI. divergence weighted by which follows piece-wise linear schedule from (0, 0.001, 0.01] over epochs [0, 5, 20).
RTS details For the RTS setting, we follow the formulation studied by Jaderberg et (0155). which consists of a random rotation between [-45, 45) degrees, random scaling factor between [037, and random placement in a 42 42 image. The architecture uses typical pre-activation convolution block, which consists of 2 convolutions with reflection padding which are pre-activated with batchnorm and ReLU. The encoders uses single block followed by two linear modules 128 units to generate the mean and log variance, where the posterior network concatenates its inputs via the channel dimension. The generator has a linear layer to map the latent vector back to 42 x 42 feature map, followed by five parallel spatial transformers which the latent vector transform the conditioned input. Multiple spatial transformers here are used simply because training just one can be inconsistent. The outputs of the spatial transformers are concatenated via the channel dimension, followed by a convolutional block and one final convolution to reduce it back to one channel.
All networks trained for 100 epochs with eyclic learning rate peaking 0.0008 the 40th epoch with the Adam optimizer using batch size 128. The KL divergence is weighted by 3 which follows a piece-wise linear schedule from [0, 0.01, 1] over epochs [0, 10, 50, 100).
23

---

Published as a conference paper at ICLR 2021
4
1
4
9
5
9
0
6
9
0
1
5
9
7
3
4
6
u
4
9
6
9
O
9
0
/
5
9
>
3
Figure 5: Samples of co (top) and RTS (bottom) perturbations for MNIST from the convolutional too and RTS-1 models.
Table 4: Evaluation of perturbation set learned from MNIST examples perturbed by foo noise with e = 0.3 and RTS perturbations. The standard deviation less than 0.01 for all metries except KL divergence.
Test set quality metrics
Test set CVAE metrics
Model
Enc. AE
PGD AE
EAE
OAE
Recon.
KL
Fully connected 60
28
0.31
0.25
0.32
0.65
0.27
585.5 + 0.29
Convolutional loo
29
0.30
0.27
0.32
0.35
0.27
610.3 + 0.10
RTS
14
0.29
0.11
0.54
1.03
0.04
22.2 0.02
RTS-1
14
0.28
0.10
0.61
1.34
0.05
22.2
RTS-5
0.28
0.10
0.53
0.86
0.05
23.7 0.02
D.2 SAMPLES AND VISUALIZATIONS
Figure 5 plots samples from the convolutional model producing Eoo perturbations and the RTS model the perturbation was tuned to produce reasonable samples in order to balance the KL divergence and reconstruction error, likely due to the inherent difficulty of this setting.
trained with only one perturbed datapoint per example. We note that final 3 value for the foo
D.3 EVALUATING THE PERTURBATION SET
the full tabulated results evaluating the MNIST perturbation sets. The KL divergence for the fox. perturbation sets are much higher due to the KL divergence being weighted by 0.01 in order to produce reasonable samples. We note that the convolutional network and fully connected network evaluate to about the same, except that the convolutional network has much lower overapproximation error and is likely restraining the size the learned perturbation set.
The RTS-1 1 results demonstrate how fixing the number of perturbations seen during training to per datapoint is still enough to learn a perturbation set with as much approximation error as a perturbation set with an infinite number of samples, denoted RTS. Increasing the number of perturbations five per datapoint allows the remaining metrics to match the RTS model, and so this suggests that not many samples needed learn perturbation set in this setting.
E CIFAR10 1O COMMON CORRUPTIONS
In this section, we describe the CIFARIO common corruptions setting and experiments in greater detail. The dataset comes with 15 common computions covering noise, blurs, weather, and digital
24

---

Published as a conference paper at ICLR 2021
Table 5: Prior and encoder architecture for learning CIFARIO common corruptions
Table Decoder architecture for learning CIFAR10 common corruptions
Input(3 x 32 x 32)
Input(k x 32 x 32)
(yput(500)
Concat(3, k)
FC(1 x 32 32)
Input(3 32 32)
Conv(64)
(1,cat(1,3)
4 (6sidual(64)
Conv(64)
Conv
4 Residual(64)
FC(512)
FC(512)
Conv(3)
corruptions. We omit the three noise corruptions due to similarity to unstructured fp noise, leaving us with the following 12 corruptions for each example:
1. Blurs: defocus blur, glass blur, motion blur, zoom blur
2. Weather: snow, frost, fog
3. Digital:
brightness, contrast, elastic, mixelate,
The dataset comes with different corruption levels, so we focus on the highest severity, corruption level five, the total training has 600,000 perturbations (12 for each of 50,000 examples) and the test set has 120,000 perturbations. The dataset also comes with several additional corruptions meant to be used validation set, however for our purposes will serve as way measure performance on out-of--istribun the blur, weather, and digital categories respectively. We generate a validation by randomly setting aside 1 1/50 of the CIFAR10 10 training set and all of their corresponding corrupted variants. These experiments were run on single GeForce RTX 2080 Ti graphics card, taking 22 hours to train the CVAE and 28 hours to run adversarial training.
corruptions. These are Gaussian blur, spatter, and saturate which correspond to
from the training set
E.I PERTURBATION MODEL ARCHITECTURE AND TRAINING SPECIFICS
We use standard preactivation residual blocks (He et al. 2016), with convolutional bottleneck which reduces the number of channels rather than downsampling the feature space. This aids in learning to produce CIFAR1O common corruptions since the corruptions are reflective of local rather than global changes, and there is not much benefit from compressing the perturbation information into smaller feature map. Specifically, our residual blocks uses convolutions that go from - 16 - 64 channels denoted (esidual(64). Then, our encoder and prior networks are shown in Table 5 where t: = 0 for the prior network and k = 3 for the encoder network, and the decoder network is as shown in Table
Stabilizing the exponential function Note that the CVAE encoders output the log variance of the prior and posterior distributions, which need be exponentiated in order calculate the KL distance. This runs the risk of of numerical overflow and exploding gradients, which can suddenly cause normal training to fail. To stabilize the training procedure, function before predicting the log variance, which is a Tanh activation which has been scaled to output a log variance betwen [ln(10 In0)) This has the effect of preventing the variance from being outside the range of [10-3 10). and stops overflow from happening. In practice, the prior and posterior converge to a variance within this range, and so this doesn't seem to adversely effect the resulting representative power of the CVAE while stabilizing the exponential caleulation.
sufficient to use scaled Tanh activation
Training is done for 1000 epochs using a cyclic learning rate (Smith 2017), peaking at 0.001 on the 400th epoch using the Adam optimizer (Kingma & Ba 2014) with momentum 0.9 and batch size 128. The hyperparameter 3 is also scheduled to increase linearly from 0 to 10 over the first 400 epochs. We standard CIFAR 10 data augmentation with random cropping and flipping.
25

---

Published as a conference paper at ICLR 2021
Table Mcasuring and comparing quality metrics for a CVAE perturbation set trained on CIFARIO common corruptions with different pairing strategies depending on the available information.
Test set quality metrics
Test set CVAE metrics
Method
Enc. AE
PGD AE
EAE
OAE
Recon. err
KI.
Centered original
28
0.005
0.006
0.029
0.17
5.73
10-4
69.31
Original + perturbed
34
0.019
0.010
0.035
0.20
8.33
10-4
160.60
Perturbed only
28
0.019
0.007
0.038
0.17
8.53
-0-
182.85
Table 8: CIFARIO common corruptions validation set statistics for the Eg norm of the latent space encodings.
Method
Mean
Std
25%
50%
75%
Max
Centered at original
0.01
6.53
5.22
2.69
3.85
10.16
26.61
Original perturbed
0.01
11.50
4.52
7.81
10.95
14.48
31.30
Perturbed only
0.01
11.04
3.81
7.93
10.82
13.46
29.57
E.2 PAIRING STRATEGIES
Here, we quantitatively evaluate the effect of the different pairing strategies for the CIFARIO common corruptions setting when training perturbation set with CVAE in Table We first note that the approach using random pairs of both original and perturbed data has larger radius than the others, likely since the generator needs to produce not just the perturbed data but also the original as well. We next see that all metrics across the board are substantially lower for the approach which centers the prior by always conditioning on the unperturbed example, and so if such an unperturbed example known, we recommend conditioning on said example learn better perturbation set.
E.3 PROPERTIES OF THE CVAE LATENT SPACE
We next study closely the properties of the underlying latent space which has been trained generate CIFAR10 1 common corruptions. We summarize number of statistics in Table 8 namely the mean, standard deviation, and some percentile thresholds where we can clearly see that the latent vectors for the CVAE centered around the original are much smaller in &g norm and thus learns the most compact perturbation set. We remind the reader that maximum l2 norm reported here is calculated on the test set and is not necessarily the same the radius reported in Table 7 since the radius is selected based on a validation set.
Distribution of l distances in latent space We plot the distribution of the & norm of the latent space encodings Figure and further find the centered CVAE has semantic latent structure. Specifically, corruptions can be ordered by their & norm in the latent space where each corruption inhabits a particular range of distance, so reducing the radius directly constrains the type of corruptions in the perturbation set. For example, restricting the perturbation set to radius leaving out frost fog, snow, elastic, and glass blur corruptions, while containing all defocus blur, jpeg compression, and motion blur corruptions. A larger version of Figure 6a with complete legend listing all the corruptions in Figure
corresponds to
Additional samples and interpolations from the CVAE To further illustrate what corruptions are represented the CVAE latent space, we plot additional samples and interpolations from the CVAE. In Figure we draw four random samples for eight different examples, which show range of corruptions and provide qualitative evidence that the perturbation set generates reasonable samples. In Figure 9 we number of examples being interpolated between weather, blur, and digital corruptions, which also demonstrate how the perturbation set not only includes the corrupted datapoints, but also variations and interpolations in between corruptions.
26

---

Published as a conference paper at ICLR 2021
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
20
0
20
20
(a) CVAE centered at original
(b) CVAE original perturbed
(c) CVAE perturbed only
Figure 6: Distribution of &2 norms for latent encodings of CIFARIO common corruptions on the test set for each pairing strategy.
2.5
defocus blur
jpeg_ compression
motion_blut
2.0
brightness
pixelate
contrast
1.5
blur
frost
1.0
fog
snow
transform clastic_transfomm
0.5
glass blur
0.0
10
15
20
25
30
Figure 7: A larger version of the density of the & norms of latent encodings of CIFAR10 common corruptions, broken down by type of corruption.
E.4 ADVERSARIAL TRAINING
standard wide residual network (Zagoruyko & Komodakis 2016) with depth 28 and width factor 10. All models are trained with the Adam optimizer with momentum 0.9 for 100 epochs with batch size 128 and cyclic learning rate schedule which peaks at 0.2 at epoch 40. This cyclic learning rate was tuned to optimize the validation performance for the data augmentation baseline, and kept fixed as-is for all other approaches. We use validation-bawee early stopping for all approaches to combat robust overfitting (Rice al. 2020). However, we find that in this setting and also likely due to the cyclic learning rate, we do not observe much overfitting, Consequently, the final model at the end of training ends up being selected for all methods regardless. Adversarial training is done with a PGD adversary at the full radius c = 28 with step size c/5 = 5.6 and 7 iterations. At evaluation time, for given radius size e/20. Additional examples of adversarial attacks under this adversary are in Figure 10 which still appear be reasonable corruptions despite being adversarial.
For learning robust CIFARIO classifier,
a PGD adversary with 50 steps of
27

---

Published as a conference paper at ICLR 2021
Figure 8: Random perturbations from the CVAE prior, showing four random samples for each example.
Figure 9: Interpolations between fog (left), defocus blur (middle), and pixelate (right) corruptions as representative distinct types of corruptions from the weather, blur, and digital corruption categories.
E.5 COMPARISONS OTHER BASELINES
One may be curious to how these results compare to other methods for learning models which are robust to different threat models. In this section, we elaborate more on the comparison to other baselines. We preface this discussion with the following disclaimers:
1. It not necessarily expected for robustness to one threat model to generalize to another threat model.
2. It is expected that training against given threat model gives the best results to that threat model, in comparison to training against different threat model.
3. These alternative baselines have different data assumptions (no access corrupted data) or solve for different type of robustness than what is being considered in this paper (lp robustness)
Consequently, the fact that these baselines perform universally worse than the CVAE approaches is unsurprising and confirms our expectations. Nonetheless, we provide this comparison to satiate the readers curiosity.
As alternative baselines, we compared to standard training, methods in general data augmentation (AugMix), & adversarial robustness, and fo adversarial robustness for CIFAR10. For la and foo adversarially robust models, we download pretrained ofate-of-the-aat models ranked by the Auto AutoAAtacyy leaderboard. For AugMix, there no released CIFAR10 model, retrained an https:://github..com//ttt atuta
28

---

Published as a conference paper at ICLR 2021
Figure 10: Adversarial examples that cause misclassification for an adversarially trained classifier.
Table 9: Certified robustness to CIFAR1O common corruptions with a CVAE perturbation set.
Test set accuracy (%)
Test set certified accuracy (%)
Noise level
Clean
Perturbed
OOD
2.7
3.9
10.2
a 0.84
93.8
92.5
84.2
34.7
00.0
00.0
o 1.22
94.0
92.6
83.9
66.1
15.3
00.0
3.19
86.7
84.7
65.0
86.7
39.4
00.0
AugMix model using the official AugMix implementation using the parameters recommended by the authors. We note that the AugMix results for CIFARIO in this paper worse than what is reported by Hendrycks et al. (2019), which is a known issue the repository that others have encountered (see https:://ssg/ /tttps//githubs com gwwp://oogleeeoooo rese www.arrch/ugmm///asssueeee http://////uuumii/////ss////////
E.6 RANDOMIZED SMOOTHING
Randomized smoothing is done in its most basic form, with normal Gaussian data augmentation the specified noise level a. We predict labels with no 100 samples and do certification with 0.001, so each example has a 0.1% chance of being
10, 000 samples at confidence incorrectly certified. Unlike in the typical setting, we have a known maximum radius for the perturbation set. This allows us to backwards engineer noise level to have specific maximum certified radius as follows: let be the maximum possible lower bound obtained from a confidence interval at level 1 with classifications that in unanimous agreement. Then, the maximum certifiable radius r = o (l) where x the inverse cumulative distribution function of standard normal distribution. By setting 6, we can solve for the desired noise level a --Ι(Ø naturally select a noise level with maximum certified radius e. We do this for the e E 12.7.3.9 10.2) which are the 25th, 50th, and 75th percentiles of the perturbation set, and get E (0.84, 1.22, 3.19) respectively.
The results of randomized smoothing these thresholds are presented in Table Note that by construction, each noise level cannot certify radius larger than what calculated for. We find that clean accuracy isn't significantly harmed when smoothing the lower noise levels, and actually improves perturbed accuracy over the adversarial training approach from Table 2 but with lower out-of-distribution accuracy than adversarial training. For example, smoothing with noise level of a = 0.84 can achieve 92.5% perturbed accuracy, which is 2% higher than the best empirical approach from Table [2] Furthermore, all noise levels are able achieve non-trivial degrees of certified robustness, with the highest noise level being the most robust but also taking a hit to standard test set accuracy metrics. Most notably, due to the latent structure present in the perturbation set as discussed in Appendix E.3 even the ability to certify small radii translates to meaningful provable
https: wwt.ggttthuuh... wttp:htwww.oooooooh//tco resea rch www.c/hh//aaa/.
29

---

Published as a conference paper at ICLR 2021
1.0
# 0.84
0= 1.22
d=
0.4
0.2
0.0
1
3
5
7
Figure 11: Certified accuracy against CIFAR10 10 common corruptions using randomized smoothing at various noise levels. The horizontal axis plots the certified radius, and the vertical axis plots the fraction of examples certified at that radius.
guarantees against certain types of corruptions, like defocus blur and jpeg compression which largely captured at radius = 3.9.
We note that although the largest noise level theoretically reach the 75th percentile of the perturbation set at radius possibly a limitation of the certification procedure, since although noise level of a 3.19 can theoretically certify radius of up to 10.2, the &g norm of the random samples needed to perform randomized smoothing approximately 23, which is well beyond the 75th percentile and towards the boundary of the perturbation set. The complete robustness curves for all three noise levels are plotted in Figure
10.2, none of the examples can be certified to that extent. This is
F MUL.TI-ILLUMINATIOON
In this final section present the multi- iltimiilliinnnoonn experiments in greater detail with expanded discussion. We use the test set provided by Murmann et al. (2019) which consists of 30 held out scenes and hold out 25 additional "drylab" scenes for validation. Unlike in the CIFARI0 10 common corruptions setting, there is no such thing as an "unperturbed" example in this dataset so we train on random pairs selected from the 25 different lighting variations for each scene. These experiments were run on a single Quadro RTX 8000 graphics card, taking 16 hours to train the CVAE and 12 hours run adversarial training.
F.1 ARCHITECTURE AND TRAINING SPECIFICS
We convert generic UNet architecture |Ronneberger et al. 2015) to use as a CVAE, by inserting the variational latent space in between the skip connections of similar resolutions. At high level, the encoder and prior networks will be based on the downsampling half of a UNet architecture, the conditional generator will be based on the full UNet architecture, and the components will be linked via the latent space of the VAE. Specifically, our networks have [64, 128, 256, 512, 512] channels when downsampling, and twice the number of channels when upsampling due the concatenation from the skip connection. Each skip connection passes through a 1 x convolution that reduces the number of channels to 16, and an adaptive average pooling layer that reduces the feature maps to height and width [(127, 187), (62, 93), (31, 46), (15, 23), (7, 11)] respectively. The adaptive average null operator for the MIP5 resolution, but allows higher resolutions then passed through fully connected
pooling layer has the effect use the same architecture. These fixed-size feature maps layer to output a mean and log variance for the latent space with dimension [128, 64, 32, 16, 16) for each respective feature map. The concatenation of all the latent vectors from each skip connection
30

---

Published as a conference paper at ICLR 2021
Table 10: Measuring and comparing quality metrics for a muli-illumination CVAE at different resolutions.
Test set quality metrics
Test set CVAE metrics
Resolution
Encoder AE
PGD AE
OAE
EAE
Recon. err
KI.
MIP5 (125 x 187)
17
0.019
0,006
0.049
0,13
0.0040
65.8
MIP4 (250 x 375)
25
0.034
0.008
0.049
0.27
0.0042
146.6
MIP3 (500 x 750)
21
0.060
0.009
0.048
0.33
0.0055
106.1
Table 11:
validation set statistics for the &g norm of the latent space encodings.
Resolution
8
Mean
Std
25%
50%
75%
Max
MIP5 (125 x 187)
7.42
2.14
5.93
7.35
8.81
16.63
MIP4 (250 x 375)
10.88
3.18
8.69
10.75
12.95
24.69
MIP3 (500 x 750)
10
9.11
2.72
7.14
9.00
10.97
20.65
forms the full, 256 dimensional latent space vector for the CVAE. Similar to the CIFARIO setting, we use scaled Tanh activation to stabilize the log variance calculation.
The generator of the CVAE UNet is implemented a typical UNet that takes as input the conditioned example, where intermediate feature maps are concatenated with extra feature maps from the latent space of the CVAE. Specifically, each latent space sub-vector is mapped with a fully connected layer back to a feature map with size [117,, 187). (62, 93), (31, 46), (15, 23), (7, 11)] with one channel. then interpolated to the actual feature map size of the UNet (which is a no-op for the MIP5 resolution) and concatenated the standard UNet feature map before upsampling.
We train the model for 1000 epochs with batch size 64, using the Adam optimizer with 0.9 momentum, weight decay 5 10-7 and cyclic learning rate from (0.0001, 0.001, 0] over [0, 400, 1000] epochs. Same as the CIFARIO 10 setting, the learning rate and weight decay were chosen to optimize the validation performance of the baseline data augmentation approach and kept fixed as-is for all other methods. We weight the KL divergence with 33 hyperparameter which is scheduled from [0, 1, over [0, 400, 1000], using random flip and crop with padding of 10 for data augmentation. We then fine tune the model for higher resolutions: for MIP4 we fine tune for 100 epochs with the same learning rate schedule scaled down by a factor of 4, and for MIP3 we fine tune for 25 epochs with the learning rate schedule scaled down by 40. In order to keep the samples looking reasonable, for MIP3 we also increase the 3 weight on the KL divergence to 10. Random cropping augmentation is proportionately increased with the size of the image, using padding of 20 and 40 for MIP4 and MIP3 respectively.
Learning and evaluating an illumination perturbation set at multiple scales The results of this fine tuning procedure to learn a perturbation at higher resolutions are summarized in Tables and and [I] As expected, the quality metrics of the perturbation set get slightly worse at higher resolutions which counteracted to some degree by the increase in weighting for the KI divergence MIP3. Despite using the same architecture size, the perturbation set is able to reasonably scale to images with 16 times more pixels and generate reasonable samples while keeping relatively similar quality metrics. However, to keep computation requirements at reasonable threshold, we focus our experiments at the MIPS resolution, which sufficient for our robustness tasks.
F.2 ADDITIONAL SAMPLES AND INTERPOLATIONS FROM THE CVAE
We present additional samples and interpolations of lighting changes learned by the CVAE perturba- tion set. Figure 12 shows interpolations between three randomly chosen lighting perturbations for four different scenes, while Figure 13 shows 24 additional random samples from the perturbation set, showing a variety of lighting conditions. These demonstrate qualitatively that the perturbation set contains reasonable set of lighting changes.
31

---

Published as a conference paper at ICLR 2021
Figure 12: Additional interpolations between three (left, middle, right) randomly chosen lighting perturbations in each row.
Figure 13: Additional random samples from the CVAE prior showing variety in lighting perturbations various different scenes.
F.3 ADVERSARIAL TRAINING AND RANDOMIZED SMOOTHING
In this final task, we leverage our learned perturbation set to learn model which is robust to lighting perturbations. Specifically the multi-illmiintion dataset comes with annotated material maps that label each pixel with the type of material (e.g. fabric, plastic, and paper), so a natural task to perform in this setting is to learn material segmentation maps. We report robustness results at E (7.35, 8.81, 17) which correspond to the the 50th, 75th, and 100th percentiles. We select fixed lighting angle which appears to be neutral and train a material segmentation model to generate a baseline, which achieves 37. 7.2% accuracy on the test set but only 14.9% robust accuracy under the full perturbation model.
We evaluate several methods for improving robustness of the segmentation maps to lighting pertur- bations, namely data augmentation with the perturbed data, data augmentation with the CVAE, and adversarial training with the CVAE. The results are tabulated in Table The CVAE data augmentation approach is not as effective in this setting at improving robust accuracy, as the pure data augmentation approach does reasonably well. However, the adversarial training approach unsurprisingly has the most robust accuracy, maintaining 35.4% robust accuracy under the full perturbation set at 17 and outperforming the data augmentation approaches. We plot adversarial perturbations at e = 7.35 and the resulting changed segmentation maps in Figure 14 for a model trained with pure data aug- mentation and in Figure [5] for a model trained to be adversarially robust, the latter of which has, on average, less pixels that are effected by adversarial lighting perturbation. Adversarial examples at the full radius = 17 are shown in Figure [6] where we see that the perturbation set is beginning to cast dark shadows over regions of the image to force the model to fail.
32

---

Published as a conference paper at ICLR 2021
Wood
Plastic opaque
Fabric/cloth
Painted
Metal
Plastic clear
Other
Tile
Figure 14: Adversarial examples that can cause on average 9.9% more pixels in the shown segmenta- tion maps be incorrect for model trained with data augmentation. The first two rows are benign lighting perturbation and its corresponding predicted material segmentation, and the next two rows are an adversarial lighting perturbation and its corresponding predicted material segmentation. For reference, the final row contains the true material segmentation.
Plastic opaque
Painted
Fabric/cloth
Wood
Plastic clear
Other
Metal
Glass
Figure 15: Adversarial examples that can cause on average 3% more pixels in the shown segmentation maps to be incorrect for adversarially trained model.
Finally, we train a certifiably robust material segmentation model using the perturbation set. We train using a noise level of 6.90 which can certify radius of most 17, or the limit of the perturbation set. The resulting robustness curve is plotted in Figure 17 The model achieves 30.7% perturbed accuracy and is able to get 12.4% certified accuracy at the 50th percentile radius 7.35. The key takeaway is that this case certifying robustness to 50% of lighting perturbations with non-trivial guarantees.
now certify real-world perturbations to some degree, in
33

---

Published as a conference paper at ICLR 2021
Plastic opaque
Fabric/cloth
Painted
Wood
Plastic clear
Other
Metal
Glass
Figure 16: Adversarial examples at the full radius of 17 that for an adversarially trained model, which are starting to cast dark shadows to obscure the objects in the image.
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
6
10
12
14
16
Figure 17: Certified accuracy for material segmentation model using randomized smoothing. The horizontal axis denotes the certified radius, and the vertical axis denotes the fraction of pixels that are certifiably correct at that radius. Note that radius of e 17 is the maximum radius of the perturbation set.
34


Reviews:
[{"review_id": "ylfTwYbIIi0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary.** In this work, the author(s) have presented an approach to identify valid perturbation operations that can be applied to the model inputs, which can be exploited to boost model robustness via purposefully corrupting the inputs during training. A weakly supervised setting has been assumed, such that pairs of valid perturbations (x, x') are available to support the identification. Technically, a conditional variational auto-encoder is trained to capture possible variations, where the perturbated inputs are used for reconstruction. The latent codes are treated as source space disruptions, concatenated with the original input x to reproduce x'. The author(s) provided empirical evidence to support their claim, along with some theoretical justifications. \n\n**Quality.** While the presentation is okay, I have concerns wrt the technical novelty and significance of this work. My detailed arguments are provided below. \n\n**Clarity.** The first three pages of this paper are very well written. The author(s) have clearly defined the problem setup, adequately reviewed relevant literature, and heuristically motivated the technical route. However, confusions arise when the author(s) try to provide some theoretical underpinnings. While I do not particularly mind if a paper has incredible empirical performance but offers very limited theoretical insights, I do find it annoying if some not-so-well-developed \"theories\" have been imposed upon the paper to boost technical significance. Notations of perturbated distributions are not clearly defined (Eqn (3)). Also, the necessary subset property (Defn. 1) and sufficient likelihood (Defn. 2) are heuristically defined, and subsequent development seems to inconsistently use these notations. For example, the reconstruction error in Thm 2 is actually the expected log-likelihood, rather than expected likelihood as defined in Defn. 2. It is also not clear what exactly does \"Let r be the Mahalanobis distance which captures 1 − α of the probability mass for a k-dimensional standard multivariate normal for some 0 < α < 1. \" mean. \n\n**Originality.** While the idea of applying CVAE to learn valid perturbations is new, the technical contributions seem very incremental. It feels like a direct application of CVAE to this problem, rather than develop specialized treatments to tailor the solution. \n\n**Significance.** This is an interesting proposal. But I encourage the author(s) to include additional comparisons wrt related techniques in order for the reviewers to better evaluate significance. For example, baselines such as AutoAug should be compared, as these methods also try to identify valid perturbations that do not affect the label. I would also love to see an adversarial variant of the proposed approach is compared as well. \n\n**Correctness.** The main reason I am leaning towards a rejection is that I have serious concerns about the correctness of the theories presented. The author(s) seem to imply the KL terms should go away in order to bound the approximation error, this is definitely not the case for VI. Such a phenomenon is more commonly known as KL-vanishing in VAE literature, which is typically associated with uninformative latents -- where VAE learning has failed. The limit of R also doe not make much sense. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dUAVNxMhxLu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to learn generative models, CVAE in particular, such that a fixed region of the latent space in CVAE corresponds to a possible perturbation set in adversarial robustness. The CVAE is learned from collected perturbation pairs, and it is expected to contain close approximations of the perturbed data and assign sufficient likelihood to perturbed data. \n\nPros:\n\n* This paper novelly proposes to learn perturbation sets with CVAE and addresses the problem of learning robust models without a predefined perturbation set.\n* The paper defines measures for evaluating the quality of learned perturbation sets and theoretically proves learning a CVAE matches the quality measures. \n* The paper demonstrates a robustness improvement on OOD corruptions and learned perturbations, and also some collected perturbed data, with CVAE data augmentation or adversarial training. It is also shown that the approximation error of the learned CVAE appears to be small. \n\nCons:\n\n* In terms of robustness, the CVAE generator itself may not be adversarially robust and bring additional robustness concerns. For instance, is it possible to somehow attack the CVAE to find some small perturbations that can be missed by the learned CVAE? Or maybe it is possible to find a perturbed input whose perturbation set by CVAE is broken? I think it may also be important to take the robustness of CVAE itself into consideration when trying to use CVAE to model perturbation sets.\n* In Table 3, CVAE training achieves much lower test set accuracy on perturbed data (41.7 v.s. 45.2), and it raises a concern whether the learned perturbation set is really good enough to match collected perturbed data. \n\n=================================================================\n\nPost-rebuttal update: Thanks to the authors for their clarifications. I have read author response and understand that the cons I mentioned above might be orthogonal to the focus of this paper. Thus I have revised my rating to 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MwDdosA8K0A", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "===============================================================================================\n\nPost-author response: I have read the response and am satisfied with the answer. I am leaning more towards accepting this paper.\n\n====================================================================================================\nThis work approaches adversarial training (robustness) from a different approach by learning the uncertainty set, as opposed to studying a prespecified one such as those using divergences. This problem is largely unsolved and quite an important one since in most adversarial training settings, it is typically difficult to understand what uncertainty is (when using other forms of perturbations). The proposed method *learns* an uncertainty set using a combination of neural networks and data and principally motivated. Theoretical results are then given and particularly shown how it can be solved using Conditional Variational Autoencoders (CVAE) and the properties are then empirically studied.\n\nPros:\n- A novel idea of learning uncertainty sets, which gives more interpretation than standard uncertainty sets.\n- Derivation of CVAEs as useful for uncertainty sets.\n\nCons:\n- Unclear how useful this would be for future developments.\n- Theoretical results are slightly incomplete (minor).\n\nI think this work makes a step in an important direction however I am somewhat reserved regarding future developments of such a method. In particular, I have two questions\n\n(1) What relation, if any, does this method have to distributional robustness? I would imagine there is some link since the distribution sets become conditional distributions. There have been recent developments in this area such as in [1,2,3] which are not discussed, which are possibly highly relevant if there is a link to distributional robustness.\n\n(2) while it is nice that uncertainty sets can be constructed with CVAEs, how can one satisfy Assumption (1)? Is there any particular structure in Assumption (1) that can be rewritten into a single objective? I believe this would strengthen at least the theoretical foundations of the results in this paper.\n\nWhile I have these concerns, they may be answered in a more theoretically focused paper. I believe the problem is important and the proposed method is novel and therefore I am in favour of accepting this paper.\n\n[1] Staib, M., & Jegelka, S. (2019). Distributionally robust optimization and generalization in kernel methods. In Advances in Neural Information Processing Systems (pp. 9134–9144).\n\n[2] Blanchet, J., & Murthy, K. (2019). Quantifying distributional model risk via optimal transportMathematics of Operations Research, 44(2), 565–600.\n\n[3] Husain, H. (2020). Distributional Robustness with IPMs and links to Regularization and GANsarXiv preprint arXiv:2006.04349.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FzC5lJp_pG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Paper summary\nThis paper addresses the problem of constraining adversarial image perturbations to be similar to some natural class of perturbations. This would allow natural perturbations to be treated with the same rigor (e.g. quantifiable attack and defense strengths) as standard adversarial perturbations. Standard adversarial perturbations are arbitrary perturbations of an image that are within a certain $\\mathcal{l}_p$-ball around the datapoint in pixel space. They are otherwise unconstrained and therefore appear as unstructured noise. Instead, the paper proposes to train a conditional autoencoder to generate perturbed versions of clean images from pairs of clean and perturbed images. The autoencoder can then be used to generate new perturbations that are similar to the training data. Adversarial versions of these perturbations can then be defined in the latent space of the autoencoder, rather than in pixel space. The paper provides theoretical arguments that perturbations generated in this way are close to the perturbations used for training. Using these learned adversarial perturbations, models can then be trained that are robust to them.\n\n## Arguments for acceptance\n* The paper provides a method for generating adversarial perturbations that are similar to natural perturbations, such as lighting changes. This has great promise for studying and improving the robustness of vision models to natural perturbations.\n* The paper provides a convincing motivation and thorough theoretical justification for the proposed method.\n* The theoretical section of the paper is clear and well written.\n\n## Arguments against acceptance\n* The experimental section only mentions the bare minimum of methodological details and results, and instead defers most experiments to the extremely extensive appendix. For example, the experiments on the multi-illumination dataset, while very intriguing, are barely intelligible without reading the appendix. Perhaps a venue allowing longer articles (e.g. a journal) would have been more appropriate.\n* The method is evaluated only on CIFAR and the multi-illumination dataset. It is unclear if the method scales to larger datasets such as ImageNet-C.\n\n## Conclusion and suggestions\nThis paper provides a well motivated and generally applicable method for generating adversarial examples that are similar to natural image perturbations. This work will be of interest to many ICLR attendees, so I recommend acceptance.\n\nSuggestions for improvement:\n* Move material from the appendix to the experimental section, e.g. more detailed descriptions of the experimental methods.\n* Repeat experiments on ImageNet-C.\n* Discuss computational cost/scalability of the method in the main paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an approach to learn perturbation sets from data and go beyond the mathematically sound L_p adversarial perturbations towards more realistic real-world perturbations. To measure the quality of the learned perturbation set the authors put forward two specific criteria and prove that an approach based on conditional variational autoencoders (cVAE) can satisfy these criteria. In particular, given access to paired data (instance and its perturbation), the authors train a cVAE which can then be used to generate novel perturbations similar to the ones observed during training. Leveraging this generative model the authors train models which are robust to such perturbations while improving the generalisation performance on clean data.

The studied problem is of high significance and the proposed solution is sufficiently novel. The reviewers agree that the paper presents a significant step in the right direction and will be of interest to the ICLR community. The authors addressed all major concerns raised by the reviewers. In my opinion, given the inherent tradeoff between the two terms in Assumption 1, and the approximation gap due to the design choices of the particular cVAE, I feel that a hard problem was reduced to an almost equally hard problem. Nevertheless, the principled approach coupled with promising empirical results are sufficient to recommend acceptance. I strongly advise the authors to incorporate the remaining reviewer feedback and try to tone down claims such as “certifiably robust” given the issues pointed out above. 

Decision: Accept (Poster)