A PAC-Bayes Analysis of Adversarial Robustness

Abstract: We propose the first general PAC-Bayesian generalization bounds for adversarial robustness, that estimate, at test time, how much a model will be invariant to imperceptible perturbations in the input. Instead of deriving a worst-case analysis of the risk of a hypothesis over all the possible perturbations, we leverage the PAC-Bayesian framework to bound the averaged risk on the perturbations for majority votes (over the whole class of hypotheses). Our theoretically founded analysis has the advantage to provide general bounds (i) that are valid for any kind of attacks (i.e., the adversarial attacks), (ii) that are tight thanks to the PAC-Bayesian framework, (iii) that can be directly minimized during the learning phase to obtain a robust model on different attacks at test time.


Reviews and author responses:

 Dear Reviewer,

Many thanks for your answer and your support.

About your last concern related to Comment 6, we are not really sure to exactly understand what you mean by “optimizing the distribution of perturbations”.

What we can say is that from the theoretical point of view, the bound stands for any distributions (and thus for an arbitrary distribution). And you are right: the study of more specific distributions that depend on the perturbation may improve the connection to the worst-case perturbation. An additional perspective would be to embed $n$ in the bound, a possible direction could be to connect this $n$ with the confidence parameter $\delta$, which would correspond to integrate the probability to obtain an adversarial example. This seems technical and we have left this study as future work.  
From a practical point of view, to get an attacked example $(x+\epsilon, y)$ with the attack function in Algorithm 1, we attack the majority vote and then apply a uniform noise on the example. This approach is correct according to our PAC-Bayesian bound, and allows us to consider a distribution of perturbations related to the worst-case adversarial risk in practice. We will clarify this point in the paper.

We hope that we answered your concern. We will appreciate your feedback and do not hesitate to ask for more details if needed.

Adversarial examples are test time attacks on machine learning models that add small imperceptible noise to examples with the goal of having them misclassified. This paper studies the problem in the "PAC-Bayes" framework where the learner produces a distribution Q over the models (and some way of combining their decisions) and generalization bounds are produced based on picking some priori distribution P (perhaps based on the instance distribution D) to derive generalization bounds. The novel part is the PAC-Bayes form, as the final proved results are either for random noise or are “conditional” and depend on some parameters that are not quite clear how to bound (see my comments below).  The adversarial examples is a worst-case phenomenon where the noise can completely depend on the instance x (or even its true label y) and the produced model h. This paper’s approach is to first consider an average case noise where eps (the onise) is a random variable only depending on the instance x. Then, it considers a “strengthening” of this noise that allows sampling it n times and picking the one that gives the worst loss (based on hte model h). It is easy to see that when n goes to infinity, the n-sampled version converges to worst-case adversarial noise (assuming they have the same support set). So the paper uses this connection and provides some generalization bounds in the form of PAC Bayes for the random noise and n-sample noise. The bounds relate the generalization error of the “Q majority based” classifier to its KL divergence from an a priori distribution P -- that can in turn be selected based on the distribution D to get a good bound on the error.

One issue is that the generalization bounds are either for random noise (that does not depend on the “model” or “model ensemble”) or have extra terms that are hard to interpret (see my comments below). Moreover, since the results of the paper use an “ensemble model” where majority is taken, it is in spirit similar to recent (not cited) results that use Boosting style techniques for getting generalization bounds for adversarially robust learning. See:
https://arxiv.org/abs/2010.12039
https://arxiv.org/abs/1902.04217

The paper also presents some experiments for particular models, and it shows robustness achieved against specific attacks. Note that this is not really a proof that the employed methods are robust, as that can only happen using a “complete” attack or proofs, and the experiments are more like weak evidence that the methods of the paper can potentially be used to obtain generalization for adversarial setting.

Evaluation: On the down side, the generalization bounds still do not resolve the main question (as explained above) but they are a step in a new interesting direction. Assuming that the authors can provide a good comparison of their results with the two papers above I can see it being accepted.

Comments:

Beginning of Section 4, it says “we illustrate the soundness of our framework”. But I am not sure exactly what it means. In addition to giving the details of the experiment and the numbers, the paper lacks a high level discussion of how the experiments relate to the theory section and how their results should be interpreted.

Line 114: “of existing works comes either without theoretical guarantee or with very specific theoretical justifications. ” 
As mentioned above, there is actually some progress on the theory side that needs to be cited and compared with.

Line 15: “This phenomenon known as adversarial robustness” maybe here it is more suitable to be called adversarial vulnerability. Robustness is the opposite.

Line 38: “adversarial robust PAC-Bayes.” it seems “adversarially robust” is more accurate.

Line 73: it seems the definition of B is not quite accurate, because what is important is that x+\eps belongs to X (not eps).

Line 130: the sentence needs to be rewritten.

Line 137: B^n at the end of the line needs correction: the whole thing (including B^n) needs to be raised to power m.

Line 138: notation D^n was already defined for a different thing (product distribution) and hence should be avoided!

Many definitions, claims and results are written for H_Q while they hold for any randomized classifier. This is a matter of taste, but maybe it is better to state them in the simpler sense for any randomized verifier h. In particular, Theorem 5 looks like a new theorem while the relation between this surrogate loss and its original variant is a simple (well known) fact that does not have to do with H_Q being a majority-based model.

Why is A also defined for risk? Why not use the letter R again?

Line 193 and 194: the sentences are not clear and might need to be clarified.

Theorem 7: how should we interpret the term TV(rho_i,pi_i)? How trivial/nontrivial does it make the inequality? This seems to be the term that needs to go away to get a meaningful generalization result?
 No direct societal impact, and the long term impact is positive.

 In light of clarifications, I can increase my score.

 Dear authors,

Thank you for your response. 

Comments 1 and 5: I understand your point.  Happy to hear that you will add this remark to the paper.

Comment 4: I like this form and look forward to seeing it in the statement of theorem.

Comment 6: I was worried that you are optimizing the distribution of perturbations. Is this correct? If not, it would be better to clarify the choice of that distribution and connect it to the remark that you proposed in response to my comment 1. For example, if the distribution is uniform over l_2 ball, then you can perhaps connect the bound to worst-case l_2 perturbations. But if the distribution is selected in an arbitrary way, then the connection to adversarial examples is lost.

To conclude, I am happy with your response and am increasing my score.





This paper studies the generalization of adversarial error in the context of PAC-Bayes. Interestingly, the authors obtain generalization bounds that are independent of the type and budget of perturbations. These bounds are obtained for majority-based classifiers in the PAC-Bayes framework. Further, the authors use the results to optimize the obtain generalization bound for an ensemble of decision trees and they show that their method is actually effective and can provide non-vacuous bounds.


  The paper studies the generalization gap of adversarial risk for majority-based classifiers. Namely, we have hypothesis class H, and a prior distribution P, and a posterior distribution Q. Then, the goal is to find ou the maximum difference between empirical, adversarial risk, and population adversarial risk for such a classifier based on the deviation of Q from P.


The authors take the following steps:

1- [Definition 2] They first define a new notion of advRisk_n that is something between random noise and adversarial noise. This definition samples n perturbations from a noise distribution (that could depend on the original point, but not the classifier in hand) and then takes the maximum of loss over all those perturbations. 

2- [Theorem 7] They provide an upper bound on the difference between empirical adversarial risk and the population adversarial risk using the PAC-Bayes framework. Their generalization gap depends on the KL divergence of P and Q, the sample complexity, and another non-standard term which deals with the entropy of the worst-case adversarial examples on different classifiers. Their theorem is interesting in that it does not depend on n, which means you can get the limit in the infinity without the theorem changing much. Their bound is also independent of the type and degree of perturbation which is important.

3- They study their bound in Theorem 7 and observe that they can optimize the bound for simple random forest classifiers and obtain non-trivial robustness and generalization gaps.


Evaluation: I find this paper interesting. It introduces a new way to study generalization in adversarial settings. I doubt their result will be of any help for a state-of-the-art classifier but it's a nice step in the understanding of the adversarial generalization. The paper is written well with a good amount of explanations. The work is also novel in that it uses the PAC-Bayes framework for the analysis of adversarial risk.  However, I have some concerns/questions mentioned below. I will update the score after hearing back from the authors.


Comments to the authors: 

1- Proposition 3: Does A_{D_n} converge to adversarial risk as n grows, at least for some specific settings? I think that is a crucial thing to argue, otherwise the obtained bounds will not have any connection to adversarial risk.

2-Proposition 4: the distribution \Pi is dependent on Q, right? It will be better to clarify that as it looks confusing now.


3- After reading Theorem 6, I did not expect the bound to depend on n, as nothing in the statement of theorem depends on n. But after reading the text after the theorem and checking the proof I realized that set S is still defined based on n samples from the perturbation distribution. This is not clear from the statement of theorem at all. I think defining the notion of A_S^n would help the readability of the theorems. 

4- Theorem 7: Is there any intuitive way of thinking about the annoying middle part of the generalization gap? e.g. entropy of adversarial examples? Also, is there any hope to get rid of that part?

5- It seems the bounds in Theorem 7 are independent of n. So one can potentially push n to infinity. It would be great if authors can provide such a corollary. Also, what are the implications of such a corollary on adversarial risk? Can you prove that the notion of average-max-risk will converge to adversarial risk, at least for certain spaces (e.g., when the instance space is discrete.)? 

6- In algorithm 1, the distribution \pi_i is completely replaced by adversarial training. It seems that \pi_i is chosen in a way that the bounds are valid. But unfortunately, this approach could lead the definition of average-max adversarial risk to be completely irrelevant to adversarial risk. For instance, if \pi_i is defined to be a Dirac distribution, then increasing n would not have any effect and making it closer to adversarial risk.  This must be clarified in Table 1, Figure 1 and Figure 2. 
 See above

 Thank you for your review and useful comments. Thank you also to point out some ambiguities (lines 15, 38, 73, 137); it will surely clarify the paper.  

Concerning the other typos, we will modify the lines as follows.   
Line 130: The adversarial attack of the literature (like PGD or IFGSM) aims at finding the optimal perturbation $\epsilon^*(x,y)$, however, they only give an approximation.  
Line 193-194: Surprisingly, this theorem states bounds that do not depend on the number of perturbed examples $n$ but only on the number of original example $m$. This is due to the fact that the $n$ perturbed examples are actually inter-dependent (see the proof for more details).
Note that these bounds involve the usual trade-off between the empirical risk $\overline{R_{\bf S }(H_{\cal Q})}$ and the KL divergence $\text{KL}({\cal Q}\|{\cal P})$.

About line 138, this is the first time we introduce the notation ${\bf D}^n$, the distribution of perturbed examples. However, at line 60 we introduced $D^m$, the distribution of “original” (non perturbed) examples which represent the distribution of the training set composed of $m$ examples.

Thank you for pointing out the references [1,2], we will correctly cite them in the paper. They study robust PAC-Learning for PAC-learnable classes with finite VC dimension. They propose an ensemble strategy based on a boosting algorithm that can be interpreted as a way to robustify majority votes. They give a generalization bound on the adversarial risk of the majority vote output by their algorithm. Their bound depends on the VC-dimension and the dual VC-dimension of the hypotheses considered in the voting. The dual VC-dimension may induce an exponential dependence on the VC dimension which can imply loose bounds. In addition, their bound suffers also from a large constant as indicated in the end of the proof of Theorem 3.1 in [1]. Their bounds are then a priori loose, their goal anyway was not to provide tight bounds but rather to study conditions for having robust guarantees and in particular to make some links between PAC-learning and robust learning which is not our focus. Additionally, their algorithm requires at some point to consider all the possible adversarial perturbations for each learning example which is intractable in practice, even though they proposed some discretized approaches to tackle this, but without practical evaluation. Lastly their result is valid for non weighted majority votes over a finite set of voters.   
In our approach, we learn the weights associated with the hypotheses considered in the vote. Our bounds are adapted for the averaged adversarial risks and appear to be more informative. Our approach seems also more efficient algorithmically since we consider a limited number of perturbations. Thanks to the tools provided in the PAC-Bayesian framework, it is possible to learn majority votes over an infinite set of hypotheses.   
Overall, these are complementary work for the understanding of adversarial training.  About the statement  “we illustrate the soundness of our framework”. By “soundness”, we meant that our theory can be applied to use-cases and provide informative robust bounds on the algorithm considering this specific setting.   


About giving “high level discussion of how the experiments relate to the theory section”. In Section 4.1, we discuss how the experiments relate to the theory. We explain how to derive an algorithm from the theoretical bounds. Indeed, our bounds are composed (roughly speaking) of two main quantities: (1) an empirical risk (the average risk or average-max risk) and (2) a regularization term (the KL divergence). Thus, as commonly done in machine learning, we use a loss that is composed of these two quantities: we are optimizing a generalization bound. Then, we propose Algorithm 1 that optimizes these “particular” losses (i.e., the bounds), in an adversarial training fashion.  


Concerning the interpretation of the results, our primary goal was to highlight that our theoretical setting can provide informative bounds and despite our relaxation we may be close to the (worst case) adversarial risk (Figure 1).   


About the introduction of the notation $A$ instead of using $R$ again.   
It was a choice to choose $A$. Indeed if we had used $R$ instead of $A$ it would have been necessary to add subscript/superscript to make the difference between both notations.  Considering that we already need subscripts, we prefered to introduce a new notation ($A$) for the averaged-max adversarial risk in order to lighten the notation.  


About your comment on the interpretation of the term $\text{TV}(\rho_i, \pi_i)$, it is better to think about Theorem 7 in terms of a trade-off between the empirical risk, which reflects the robustness of the majority vote, and two “complexity” terms: the KL term and the TV term. The KL term will control how much the posterior weights of the majority vote can differ from the prior ones. The TV term controls, in some sense, the diversity of the voters, i.e., the ability of the voters to be fooled on the same adversarial example. We will specify it in the text to avoid confusion. Indeed, the smaller the TV distance will be close to 0 the more informative the bound will be.

References:  
[1] Omar Montasser, Steve Hanneke, Nathan Srebro. Reducing Adversarially Robust Learning to Non-Robust PAC Learning. NeurIPS 2020.  
[2] Omar Montasser, Steve Hanneke, Nathan Srebro. VC Classes are Adversarially Robustly Learnable, but Only Improperly. COLT 2019.


 Thank you for your review and your detailed comments.

We make a joint answer to your comments 1 and 5 because from our point of view they are related. Actually, by construction and under the mild hypothesis that the event of drawing an adversarial example is not null, then $A_{D_n}$ will correspond to the adversarial risk when $n$ goes to infinity. We will precise this important point in the paper, thanks for the remark. Note that the convergence to the adversarial risk will highly depend on the structure and the density of the adversarial examples.

About your comment 2, yes, you are right. Since $\Pi$ corresponds to the probability to draw an adversarial example $x+\epsilon^*(x,y)$ and that $\epsilon^*(x,y)$ is defined according to $H_{\cal Q}$, then $\Pi$ is dependent on ${\cal Q}$. Note that Proposition 4 holds for any distribution ${\cal Q}$. We will clarify it in the paper.

About your comment 3, we show that $n$ does not have an influence on the bound of Theorem 6 (ie the right-hand side of Equation (9)). However, $n$ is nevertheless present in the empirical risk $\overline{R_{\bf S}(H_{\cal Q})}$. Note that $A$ refers to Theorem 7 and not to Theorem 6. We will clarify this point in the paper.

About your comment 4, it is better to think about Theorem 7 in terms of a trade-off between the empirical risk, which reflects the robustness of the majority vote, and two “complexity” terms: the KL term and the TV term. The KL term will control how much the posterior weights of the majority vote can differ from the prior ones. The TV term controls, in some sense, the diversity of the voters, i.e., the ability of the voters to be fooled on the same adversarial example. We will precise it in the text to avoid confusion. 
Concerning the “hope to get rid of the TV”, it is actually possible to obtain the inequality $\overline{A\_{{\bf D}^n}}(H\_{\cal Q}) \le \mathbb{E}\_{h\sim{\cal Q}}\frac{1}{m}\sum\_{i=1}^{m}\max\_{\epsilon\in\boldsymbol{\cal E}\_i}\frac{1}{2}(1{-}y\_ih(x\_i{+}\epsilon)) + \sqrt{\frac{\text{KL}({\cal Q}\|{\cal P})+\ln\frac{2\sqrt{m}}{\delta}}{2m}}$ that holds with probability at least $1-\delta$ over ${\bf S}$ (see the proof of Theorem 7). 
Hence, in order to minimize the true average-max risk, a trade-off between the complexity of the majority vote (represented by the KL divergence) and the empirical risk. Moreover, the empirical risk will be minimized when the loss for each perturbation and for every hypothesis (that are weighted by the distribution ${\cal Q}$) is minimized.
This version of the bound will appear in the statement of Theorem 7 because it gives another view of the bound.


About your comment 6, if we understand your comment correctly, you talk about the optimization of the TV distance.
We recall that Algorithm 1 was introduced to optimize both bounds, i.e., the bound of Theorems 6 and 7. Moreover, we make the algorithmic choice to get rid of the TV through an algorithmic procedure that leads to a TV equals zero: we draw only one perturbed example by iteration. Note that this algorithmic choice reduces the time complexity since we sample one perturbed example instead of several one for each iteration. 


 We are grateful to all the reviewers for their feedback: thank you for the time spent on the review of our paper, for the relevant comments, for all the typos and for the ambiguities left on the paper. Your feedback confirms us in our idea that this first PAC-Bayesian analysis for the adversarial robust is novel and original and that this line of research is relevant. Furthermore, we believe that this work can be useful for the community by bringing a novel point of view on the problem. We provide an answer to the few concerns raised and specific answers to each review.

 Thank you for your review and positive feedback. Indeed, a more exhaustive study of other adversarial learning approaches could have been made but as you said, our primary goal was to compare bounds and show that they can be informative. However, this can be considered in the future to assess the behavior of a model on the different adversarial learning approaches. We will surely correct all the typos pointed out in our paper. 

The authors provide a PAC-Bayes analysis of two relaxations of the adversarial robust risk, the averaged adversarial robustness risk, and the averaged-max adversarial robustness risk.  They derive a series of relationships between these adversarial risks, in particular showing that the difference from the true risk and adversarial robust risk can be bounded by a total variation term.  This analysis is used to derive PAC-Bayes bounds on the two relaxed forms, which in turn are used to motivate adversarially robust training algorithms via minimizing the proposed bounds.  These are shown to be effective empirically in adversarial learning on several datasets.  The paper provides an elegant analysis of adversarial learning in a PAC-Bayes setting, providing a principled approach to adversarial training with theoretical guarantees.  The relaxations proposed are novel, and lead to a novel form of bound including a total variation term.  The submission appears to be technically sound and is accompanied by an informative empirical study.  Additional comparisons could have been made to other adversarial learning approaches in the experimentation, although the primary goal was to empirically compare different versions of the bound.  The presentation is clear and the paper well written.  The analysis is significant for providing a PAC-Bayes viewpoint to complement previous approaches to generalization analysis in the adversarial setting using Radamacher complexity and margin-based analysis.

There are a few minor typos which should be addressed in the final version. Not relevant.

Meta-review: This paper studies the generalization of adversarial error using the PAC-Bayes framework. The authors provide generalization bounds that are independent of the type of perturbations. 

The reviewers see this direction as novel and the results of interest. The more important concerns initially raised by the reviewers were addressed in the rebuttal (some reviewers increased their scores as a result). Overall, I think this paper makes an interesting contribution and I, therefore, recommend acceptance. Some reviewers still pointed out some limitations of the results and I would thus encourage the authors to add a discussion in the revised version.



Decision: accept