Learning perturbation sets for robust machine learning

Abstract: Although much progress has been made towards robust deep learning, a significant gap in robustness remains between real-world perturbations and more narrowly defined sets typically studied in adversarial defenses. In this paper, we aim to bridge this gap by learning perturbation sets from data, in order to characterize real-world effects for robust training and evaluation. Specifically, we use a conditional generator that defines the perturbation set over a constrained region of the latent space. We formulate desirable properties that measure the quality of a learned perturbation set, and theoretically prove that a conditional variational autoencoder naturally satisfies these criteria. Using this framework, our approach can generate a variety of perturbations at different complexities and scales, ranging from baseline spatial transformations, through common image corruptions, to lighting variations. We measure the quality of our learned perturbation sets both quantitatively and qualitatively, finding that our models are capable of producing a diverse set of meaningful perturbations beyond the limited data seen during training. Finally, we leverage our learned perturbation sets to train models which are empirically and certifiably robust to adversarial image corruptions and adversarial lighting variations, while improving generalization on non-adversarial data. All code and configuration files for reproducing the experiments as well as pretrained model weights can be found at https://github.com/locuslab/perturbation_learning. 

Reviews and author responses:

===============================================================================================

Post-author response: I have read the response and am satisfied with the answer. I am leaning more towards accepting this paper.

====================================================================================================
This work approaches adversarial training (robustness) from a different approach by learning the uncertainty set, as opposed to studying a prespecified one such as those using divergences. This problem is largely unsolved and quite an important one since in most adversarial training settings, it is typically difficult to understand what uncertainty is (when using other forms of perturbations). The proposed method *learns* an uncertainty set using a combination of neural networks and data and principally motivated. Theoretical results are then given and particularly shown how it can be solved using Conditional Variational Autoencoders (CVAE) and the properties are then empirically studied.

Pros:
- A novel idea of learning uncertainty sets, which gives more interpretation than standard uncertainty sets.
- Derivation of CVAEs as useful for uncertainty sets.

Cons:
- Unclear how useful this would be for future developments.
- Theoretical results are slightly incomplete (minor).

I think this work makes a step in an important direction however I am somewhat reserved regarding future developments of such a method. In particular, I have two questions

(1) What relation, if any, does this method have to distributional robustness? I would imagine there is some link since the distribution sets become conditional distributions. There have been recent developments in this area such as in [1,2,3] which are not discussed, which are possibly highly relevant if there is a link to distributional robustness.

(2) while it is nice that uncertainty sets can be constructed with CVAEs, how can one satisfy Assumption (1)? Is there any particular structure in Assumption (1) that can be rewritten into a single objective? I believe this would strengthen at least the theoretical foundations of the results in this paper.

While I have these concerns, they may be answered in a more theoretically focused paper. I believe the problem is important and the proposed method is novel and therefore I am in favour of accepting this paper.

[1] Staib, M., & Jegelka, S. (2019). Distributionally robust optimization and generalization in kernel methods. In Advances in Neural Information Processing Systems (pp. 9134–9144).

[2] Blanchet, J., & Murthy, K. (2019). Quantifying distributional model risk via optimal transportMathematics of Operations Research, 44(2), 565–600.

[3] Husain, H. (2020). Distributional Robustness with IPMs and links to Regularization and GANsarXiv preprint arXiv:2006.04349.

**Summary.** In this work, the author(s) have presented an approach to identify valid perturbation operations that can be applied to the model inputs, which can be exploited to boost model robustness via purposefully corrupting the inputs during training. A weakly supervised setting has been assumed, such that pairs of valid perturbations (x, x') are available to support the identification. Technically, a conditional variational auto-encoder is trained to capture possible variations, where the perturbated inputs are used for reconstruction. The latent codes are treated as source space disruptions, concatenated with the original input x to reproduce x'. The author(s) provided empirical evidence to support their claim, along with some theoretical justifications. 

**Quality.** While the presentation is okay, I have concerns wrt the technical novelty and significance of this work. My detailed arguments are provided below. 

**Clarity.** The first three pages of this paper are very well written. The author(s) have clearly defined the problem setup, adequately reviewed relevant literature, and heuristically motivated the technical route. However, confusions arise when the author(s) try to provide some theoretical underpinnings. While I do not particularly mind if a paper has incredible empirical performance but offers very limited theoretical insights, I do find it annoying if some not-so-well-developed "theories" have been imposed upon the paper to boost technical significance. Notations of perturbated distributions are not clearly defined (Eqn (3)). Also, the necessary subset property (Defn. 1) and sufficient likelihood (Defn. 2) are heuristically defined, and subsequent development seems to inconsistently use these notations. For example, the reconstruction error in Thm 2 is actually the expected log-likelihood, rather than expected likelihood as defined in Defn. 2. It is also not clear what exactly does "Let r be the Mahalanobis distance which captures 1 − α of the probability mass for a k-dimensional standard multivariate normal for some 0 < α < 1. " mean. 

**Originality.** While the idea of applying CVAE to learn valid perturbations is new, the technical contributions seem very incremental. It feels like a direct application of CVAE to this problem, rather than develop specialized treatments to tailor the solution. 

**Significance.** This is an interesting proposal. But I encourage the author(s) to include additional comparisons wrt related techniques in order for the reviewers to better evaluate significance. For example, baselines such as AutoAug should be compared, as these methods also try to identify valid perturbations that do not affect the label. I would also love to see an adversarial variant of the proposed approach is compared as well. 

**Correctness.** The main reason I am leaning towards a rejection is that I have serious concerns about the correctness of the theories presented. The author(s) seem to imply the KL terms should go away in order to bound the approximation error, this is definitely not the case for VI. Such a phenomenon is more commonly known as KL-vanishing in VAE literature, which is typically associated with uninformative latents -- where VAE learning has failed. The limit of R also doe not make much sense. 


This paper aims to learn generative models, CVAE in particular, such that a fixed region of the latent space in CVAE corresponds to a possible perturbation set in adversarial robustness. The CVAE is learned from collected perturbation pairs, and it is expected to contain close approximations of the perturbed data and assign sufficient likelihood to perturbed data. 

Pros:

* This paper novelly proposes to learn perturbation sets with CVAE and addresses the problem of learning robust models without a predefined perturbation set.
* The paper defines measures for evaluating the quality of learned perturbation sets and theoretically proves learning a CVAE matches the quality measures. 
* The paper demonstrates a robustness improvement on OOD corruptions and learned perturbations, and also some collected perturbed data, with CVAE data augmentation or adversarial training. It is also shown that the approximation error of the learned CVAE appears to be small. 

Cons:

* In terms of robustness, the CVAE generator itself may not be adversarially robust and bring additional robustness concerns. For instance, is it possible to somehow attack the CVAE to find some small perturbations that can be missed by the learned CVAE? Or maybe it is possible to find a perturbed input whose perturbation set by CVAE is broken? I think it may also be important to take the robustness of CVAE itself into consideration when trying to use CVAE to model perturbation sets.
* In Table 3, CVAE training achieves much lower test set accuracy on perturbed data (41.7 v.s. 45.2), and it raises a concern whether the learned perturbation set is really good enough to match collected perturbed data. 

=================================================================

Post-rebuttal update: Thanks to the authors for their clarifications. I have read author response and understand that the cons I mentioned above might be orthogonal to the focus of this paper. Thus I have revised my rating to 6.


Thank you for your feedback, we hope to clarify your understanding of our theoretical results in the following response. If parts of the theory are still unclear or believed to be incorrect, let us know and we are happy to follow up. 

### On correctness: 
We would like to first clear up the main concern which arises from a misunderstanding of the theorems: we do *not* need the KL terms to disappear in order to bound the approximation error, and in fact we recognize that this typically does not occur in practice in a footnote on Page 5. Rather, our Theorem 1 says that (1) any bound on the KL terms implies a bound on the approximation error, and that (2) the limits show that the bound is *consistent*: any improvement in the VAE objective directly improves the bound on the approximation error. The KL terms do *not* need to vanish for our bounds to be valid. Our results are in fact consistent *even if KL-vanishing occurs*: a VAE that has failed due to KL-vanishing will have poor likelihoods, and so the bound on approximation error will still be large. Thus, the presence or lack of KL-vanishing does not make our bounds in our theorems incorrect.  

### On significance: 
We would like to point out that we *do* include additional comparisons to top-performing methods that also try to identify label-preserving perturbations. These are in the lower half of Table 2. Note that we have already compared to AugMix (Hendrycks et al. 2019), which was previously found to be better than the AutoAug baseline that was suggested. Nonetheless, we have added the requested comparison to an AutoAug baseline to Table 2 in our revision, which is consistent with what was reported by Hendrycks et al 2019 (it performs worse than AugMix). The requested adversarial variant of our proposed approach is also already included in our results (e.g. "CVAE adversarial training" in Table 2). On top of all this, we also have a randomized smoothing variant which is presented in the appendix, and so our comparison is quite extensive with respect to related techniques. 

### On the "theory": 
We respectfully disagree with the framing of our proposed properties and theoretical results. Our properties are not heuristics, they are criteria that *directly* quantifies how well real world data is represented within the perturbation set, irrespective of how the set was constructed. This is particularly critical for adversarial robustness: for example, the necessary subset property is sufficient in order to conclude that robustness to the learned perturbation set implies robustness to real world data. Since adversarial robustness has a long history of having heuristic defenses which are ultimately shown to be ineffective, one will naturally be skeptical of a perturbation set for adversarial robustness with no guarantees on how well it models real data. Our theorems serve exactly this purpose: they prove that a perturbation set using a trained CVAE-based generator will actually contain real data by satisfying these properties, which may not be necessarily true for other perturbation sets using different generative modeling frameworks. On the contrary, without our theorems the CVAE approach becomes a heuristic as there is no clear reason to believe that the approach results in reasonable perturbation sets. 

We address the remaining issues of clarity below. 

+ The limit of R is obtained by writing out the log likelihood in Assumption 1 with the multivariate distribution and taking the reconstruction error down to zero. This is done in the proof of Theorem 1 in the Appendix A.1. 

+ The Mahalanobis distance is explained in more detail at the start of our supplementary material in Appendix A.1. It is the radius from the mean which captures (1-\alpha) of the total probability mass of a multivariate Normal. The wording that we use matches that which is used in Section 5.7.2 of the textbook "Statistics for Imaging, Optics, and Photonics" by Peter Bajorski, but we can certainly adjust it. 

+ Theorem 2 does bound the expected reconstruction error, which indeed is not exactly the likelihood. The link between these two is in the main paper in Section 4.1, where we mention (albeit briefly) that this bound on reconstruction error implies a lower bound on the likelihood. This follows directly by applying Jensen's inequality. To make the link from reconstruction error to the sufficient likelihood property more clear, in the revision of the paper we have added Corollary 2 at the end of Appendix A.2, which spells this out explicitly. 

+ We are not sure what was unclear about Equation (3). All the distributions and variables are described in the surrounding text. To our knowledge, the only part not explicitly described is the tilde notation (i.e. z ~ p, which means z is a random variable with distribution p), but this is the recommended default notation from the ICLR 2021 template. If the reviewer still finds this confusing, please let us know so we can correct it.

Thank you for your feedback and questions. We summarize the relevant changes in our revision and answer your questions below: 

To clarify the usefulness of this work for future developments, in the revision we have expanded the conclusion to discuss several remaining open problems for future developments. These include (a) improving the quality of the learned perturbation sets with different generative modeling frameworks or training schemes, (b) learning new perturbation sets to model natural variations found in other domains for use in new robustness tasks, and (c) exploring when learned perturbation sets can be exploited to improve non-adversarial generalization as we saw in the CIFAR10-C setting. 

### Answer to Question 1: 
Although we do learn an uncertainty set, the key distinguishing factor from our work (and adversarial examples in general) to DRO is that our uncertainty sets are independent between datapoints. More specifically, the uncertainty sets that we learn for an example x1 does not depend on the uncertainty set that we learn for a different example x2, whereas distributional robustness typically defines an uncertainty set over the entire population density which is constrained jointly over all examples. Because of this, our work is much more related to the adversarial examples literature (which perturbs each individual example) and not so much the DRO literature (we believe the link from our learned perturbation sets to DRO is about the same link from Lp-adversarial robustness to DRO). 

### Answer to Question 2: 
Assumption 1 simply states that the terms found in the CVAE objective are small, where smaller terms result in a better bound on approximation error. This is achievable by simply training the CVAE, which intuitively results in a better generator and explicitly maximizes the CVAE objective, satisfying the assumptions. With respect to rewriting the bounds on the objective, for the KL term specifically, it is possible to condense the sum of KL bounds into a single bound K, which then trivially satisfies the previous assumption by taking K_i = K, but this results in a looser result. However, it is not so straightforward to rewrite the entirety of Assumption 1 as a single bound on the CVAE objective, because this merely bounds the gap between the KL divergence and the expected log likelihood without constraining either quantity. Constraining these separate quantities is crucial in how we proved Theorems 1 & 2, and also gives a more fine-grained understanding of how each of these components affects the resulting bound on approximation error. In practice, these quantities can be easily computed individually and are both reduced during successful VAE training, so we believe this assumption is realistic and attainable. 

Thank you for your feedback, which brings up interesting points. The first critique, while potentially interesting, is out of scope of the problem that we study and is better suited as a future direction. The second critique is reflective of a typical trade-off in adversarial robustness and isn't so specific to our work. Both of these points are orthogonal to the problem studied in this paper, which is learning perturbation sets for training adversarially robust classifiers to real-world changes. We elaborate more below: 

### On robustness of the generator: 
Being a deep network, the generator itself may not be adversarially robust, and so it may be an interesting direction to investigate the adversarial robustness of the generator itself. However, this is a nontrivial question that is not so straightforward. 

First of all, the question itself needs to be carefully posed: what should the perturbation generator be robust to, what does it mean for a perturbation set to be broken, and why do we want the generator to be adversarially robust? We could, for example, attempt to evaluate the Lp-robustness of the generator and define "broken" as having poor approximation error, but this may not have much motivation when studying natural variations as we do in our paper. 

Furthermore, even if we could justify this particular framing of the question, it will require the development of new techniques that go well beyond the work in this paper. Specifically, this involves solving a bi-level optimization problem, where the inner loops calculates the approximation error and the outer loop searches for an adversarial example. Adversarial training would then require a third layer of optimization on top of this, and will likely require new techniques to be handled efficiently. In short, studying the adversarial robustness of the generator would require setting up a new problem and developing new solutions which are beyond the scope of this paper, which focuses on capturing natural variations within a well-defined set. 

### On the drop in standard accuracy: 
Our goal in this paper is to learn perturbation sets to learn adversarially robust models to natural perturabtions. Although the CVAE training on the MI dataset does result in lower accuracy on the fixed lighting angles present on the dataset, it does achieve our goal of learning an adversarially robust model. This is a typical tradeoff quite frequently observed in the adversarial robustness literature: standard accuracy on a finite dataset is lower in exchange for better adversarial accuracy over a continuous set of Lp perturbations. 

Furthermore we emphasize that the learned perturbation set has value beyond perturbed accuracy and adversarial accuracy, which are just two axes of evaluation. The set encompasses a substantially more diverse set of lighting changes than the discrete set of lighting angles in the perturbation dataset (i.e. see the random samples, interpolations, and adversarial examples in Figures 2-3 & 12-15). Consequently, it is a more accurate representation of performance under *continuously* varying lighting conditions, and is less prone to overfitting to the specific lighting angles in the data. Additionally, the lighting perturbation set generalizes to new scenes (Table 10). This means we can augment new scenes without having to keep collecting lighting variations for new data, which required a special camera setup in Murmann et al. (2019). 

In our revision of the paper, we have added this discussion to section 5.2 of the main paper. 

Thank you for your review and suggestions. We have taken them into account as follows: 

1. We have revised the experimental section as recommended. In particular, we have added a paragraph discussing the cost of the method at the start of Section 5 as well as expanded Section 5.2 on the multi-illumination experiments, improving the presentation of the experiments as well as including more material from the appendix. The cost of the method is proportional to the cost of a forward pass through the generator, which in practice is quite small (e.g. 4 residual blocks is sufficient for modeling common image corruptions, which can be calculated in a fraction of the cost of the downstream classifier). 

2. We have several experiments for the multi-illumination setting at the end of Appendix F.1 that explore the scalability of learning perturbation sets to larger resolutions. Specifically, we trained and evaluated generators for the multi-illumination dataset at the 125x187, 250x375, and 500x750 resolutions, which are all well above the CIFAR10 resolution (and also ImageNet). In short, learning perturbations at higher resolutions is slightly harder with worse approximation error but the performance does not degrade substantially (e.g. approximation error increases from 0.006 for 125x187 to 0.009 for 500x750). As part of our revision, this discussion is now referenced in the main paper. 

3. In principle we believe that the approach would work on ImageNet-C, since 
(a) the common corruptions from CIFAR10-C are identical to that of ImageNet-C
(b) the common corruptions are easily modeled by a small generator in the CIFAR10 setting
(c) the approach can scale to larger resolutions (e.g. up to 500x750) as seen for the multi-illumination dataset
Although we would like to train an ImageNet-C generator, the main roadbloack here is not in the approach, but actually in generating the data. Specifically, although most papers that measure ImageNet-C performance evaluate on the corrupted *test-set*, in order to train our generator we would need to generate the ImageNet-C *train-set*. With 15 different corruptions, this entails generating (and training) on 15 copies of ImageNet (>15 million images), which would take over 2 terabytes of space. On top of that, generating the CIFAR10-C train-set took several days to create, which only has 50k+10k images that are of significantly smaller resolution, and so even creating the ImageNet-C train-set would take a substantial amount of both time and space, let alone training. Consequently, the ImageNet-C dataset is unfortunately too prohibitively expensive for us to create at this point. 

## Paper summary
This paper addresses the problem of constraining adversarial image perturbations to be similar to some natural class of perturbations. This would allow natural perturbations to be treated with the same rigor (e.g. quantifiable attack and defense strengths) as standard adversarial perturbations. Standard adversarial perturbations are arbitrary perturbations of an image that are within a certain $\mathcal{l}_p$-ball around the datapoint in pixel space. They are otherwise unconstrained and therefore appear as unstructured noise. Instead, the paper proposes to train a conditional autoencoder to generate perturbed versions of clean images from pairs of clean and perturbed images. The autoencoder can then be used to generate new perturbations that are similar to the training data. Adversarial versions of these perturbations can then be defined in the latent space of the autoencoder, rather than in pixel space. The paper provides theoretical arguments that perturbations generated in this way are close to the perturbations used for training. Using these learned adversarial perturbations, models can then be trained that are robust to them.

## Arguments for acceptance
* The paper provides a method for generating adversarial perturbations that are similar to natural perturbations, such as lighting changes. This has great promise for studying and improving the robustness of vision models to natural perturbations.
* The paper provides a convincing motivation and thorough theoretical justification for the proposed method.
* The theoretical section of the paper is clear and well written.

## Arguments against acceptance
* The experimental section only mentions the bare minimum of methodological details and results, and instead defers most experiments to the extremely extensive appendix. For example, the experiments on the multi-illumination dataset, while very intriguing, are barely intelligible without reading the appendix. Perhaps a venue allowing longer articles (e.g. a journal) would have been more appropriate.
* The method is evaluated only on CIFAR and the multi-illumination dataset. It is unclear if the method scales to larger datasets such as ImageNet-C.

## Conclusion and suggestions
This paper provides a well motivated and generally applicable method for generating adversarial examples that are similar to natural image perturbations. This work will be of interest to many ICLR attendees, so I recommend acceptance.

Suggestions for improvement:
* Move material from the appendix to the experimental section, e.g. more detailed descriptions of the experimental methods.
* Repeat experiments on ImageNet-C.
* Discuss computational cost/scalability of the method in the main paper.

Meta-review: The authors propose an approach to learn perturbation sets from data and go beyond the mathematically sound L_p adversarial perturbations towards more realistic real-world perturbations. To measure the quality of the learned perturbation set the authors put forward two specific criteria and prove that an approach based on conditional variational autoencoders (cVAE) can satisfy these criteria. In particular, given access to paired data (instance and its perturbation), the authors train a cVAE which can then be used to generate novel perturbations similar to the ones observed during training. Leveraging this generative model the authors train models which are robust to such perturbations while improving the generalisation performance on clean data.

The studied problem is of high significance and the proposed solution is sufficiently novel. The reviewers agree that the paper presents a significant step in the right direction and will be of interest to the ICLR community. The authors addressed all major concerns raised by the reviewers. In my opinion, given the inherent tradeoff between the two terms in Assumption 1, and the approximation gap due to the design choices of the particular cVAE, I feel that a hard problem was reduced to an almost equally hard problem. Nevertheless, the principled approach coupled with promising empirical results are sufficient to recommend acceptance. I strongly advise the authors to incorporate the remaining reviewer feedback and try to tone down claims such as “certifiably robust” given the issues pointed out above. 


Decision: poster-presentations