Degradation Attacks on Certifiably Robust Neural Networks

Abstract: Certifiably robust neural networks employ provable run-time defenses against adversarial examples by checking if the model is locally robust at the input under evaluation. We show through examples and experiments that these defenses are inherently over-cautious. Specifically, they flag inputs for which local robustness checks fail, but yet that are not adversarial; i.e., they are classified consistently with all valid inputs within a distance of $\epsilon$. As a result, while a norm-bounded adversary cannot change the classification of an input, it can use norm-bounded changes to degrade the utility of certifiably robust networks by forcing them to reject otherwise correctly classifiable inputs. We empirically demonstrate the efficacy of such attacks against state-of-the-art certifiable defenses.

Reviews and author responses:

 We thank the reviewer for pointing out related work [A-G]. We will discuss all of them in a revised version.

With respect to SOTA techniques, we would like to point out that our proposed attack would work even for *"perfectly robust"* networks. By this we mean the following. Suppose we use [F] or [G] to train a network that is $\epsilon$-locally robust at *every* legitimate input (hence the network is "perfectly" robust). The network would still wrongly flag inputs $x$ that are close to decision boundaries but are not adversarial. This is because the network is not robust at those points, but this does not mean the points are adversarial (the network may assign the same label to all $\epsilon$-close points in $M$, but different labels to $\epsilon$-close points not in $M$). Following the same argument, suppose we use SOTA complete verification techniques such as [B] or [C] to check local robustness at $x$. These techniques will correctly say that the network is not robust at $x$ yet $x$ is still non-adversarial. Thus, unless the network is trained for $2\epsilon$ robustness, it will be vulnerable to such attacks.

We kindly ask the reviewer to comment. We will add this argument to the paper

This paper proposes a degradation attack on certifiable defenses GloRo and randomized smoothing aiming to find correct inputs within robustness region but hard to certify, causing rejections. The proposed attack is simple which is just a smoothed variance of PGD attack. Experimental results show that the two evaluated certification methods are susceptible to this attack. Strength

1. Experimental results show that randomized smoothing and GloRo have degraded verified rates under the proposed attacks.
2. A customized defense is proposed and discussed.

Weakness

1. I doubt the practical usefulness of this degradation attack. In practice, verification is a plus for the model robustness evaluation. It is safe to trust the results if a sample is verified but a sample that cannot be verified does not necessarily mean we have to reject it. There are many other ways to measure the robustness and give confidence scores of the predictions. GloRo proposes to do the rejection mainly because they want to have global verified robustness with global Lipschitz constant but eventually it still serves for local robustness verification.
2. I do not think the example in figure 1 makes sense. When the ground-truth robustness region of x' is already crossing the decision boundary, it is truly an adversarial example. Rejection is indeed the correct thing to do for deterministic verification methods like GloRo unless we can retrain the weights to reshape the decision boundary. Please note that over cautiousness is the intrinsic property for certifiable defenses and should be the correct thing to have. The local verified robustness should only focus on the samples belonging to the original input distribution (x belongs to M) while the verified robustness around other samples like perturbed ones that out of the distribution (x' not belongs to M) should not be encouraged. Otherwise, it will become an unnecessarily too difficult problem unless global verified robustness is required.
3. The smoothed PGD attack has very limited novelty. A similar method is already proposed and also used for training to achieve better results over randomized smoothing alone. The attacks should at least discuss and evaluate over [A].
4. The proposed attack should really be discussed and evaluated with other SOTA complete verifiers like alpha-beta-CROWN [B, C] (the winner of neural network verification competition this year) or incomplete ones like CROWN [D] and K&W [E], and certifiable training methods like CROWN-IBP [F] and IBP [G].

[A] Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers, NeurIPS 2019

[B] Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification, NeurIPS 2021

[C] Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers, ICLR 2021

[D] Efficient Neural Network Robustness Certification with General Activation Functions, NeurIPS 2018

[E] Provable defenses against adversarial examples via the convex outer adversarial polytope, ICML 2018

[F] Towards Stable and Efficient Training of Verifiably Robust Neural Networks, ICLR 2020

[G] On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Model, ICCV 2019 I recommend rejection due to lack of novelty and comprehensive evaluations over SOTA verification methods. Also, I am not very convinced by the motivation of the proposed methods.

=== Post Rebuttal ===
Thank the authors for the extensive discussions. Some of my concerns have been clarified in the discussion and the revised submission. Some insights of the paper are quite interesting. However, I am still concerned about the novelty and SOTA evaluation of the proposed methods. Overall, I decided to increase my score to 5.

The paper considers a different perspective of attacks to certifiably robust ML models at test time -- the attacker can add imperceptible perturbation to force false positive of adversarial example (AE) detection, and thus lower the utility of the model/detection mechanism. It finds an empirical lower and upper bound to the degradation in performance, and validates the efficacy of such attacks against popular detection mechanisms.    I like the idea of.degradation attack on models. The problem setting is fairly intuitive. Since the learner only gets the AE at test-time instead of the original input, some information about the original input is already lost. As a result, the detection mechanism will have a tendency to overprotect.

The theoretical part of the paper is straightforward to understand. However, I find the discussion for the double-radius defense confusing. From Algorithm 2.1, it seems that a certified run-time defense will abstain at a radius. However, in Section 4.1, the models are said to be "trained" with parameter $2\epsilon$. Does it mean that the models will abstain at radius $2\epsilon$ during test-time? If yes, then it just opens a larger feasible set for degradation attack check. Could you please explain this more formally with helping lemma stating the guarantee of the defense mechanisms?

My other main worries about the paper are on the experiment results and design. Since the degradation attack phenomenon is quite easy to understand, I hope the experiments can clearly show the severity of such new threats. On top of the existing results, I hope the authors can address the following questions.

1) What's the choice of your attack parameter $\epsilon$? The choice now looks quite arbitrary. Please show degradation AE created at each $\epsilon$ so that the readers know the visual similarity at the perturbation level.

2) What's the gap between the lower and upper bound of the attack? Is there any pattern/conclusion we can find from the gap?

3) How about the result for other $\ell_p$ distances like $\ell_{\infty}$? 

Also, is there a reason only showing the upperbound for GloRo in table and that for Randomized Smoothing in figure? The idea of degradation attacks on certifiably robust AE detection is novel. However, the experiment setting is a bit too simplified. I recommend the authors to strengthen the experiment part so that the severity of the threat is better motivated. I'm giving a marginally below acceptance for now, but I'm willing to increase the score once my concerns are addressed.

================================================================

After viewing the author response, I'm raising my score to 6. The technical aspect is sound, and the author explained a relatively intuitive concept in a reasonable manner. Thanks for the clarification.

This paper presents a new line of attack, so-called degradation attacks, targeting the certified robustness of a model rather than directly targeting the classifier. They argue that in the case of certified robustness, a suitable norm-bounded adversary cannot change the classification of an input at which the output is robust, but can force the rejection of that input as potentially adversarial, thus degrading the utility of the system. The presentation of the paper was generally clear and well laid out.

However, there are potentially some issues, though I may have misunderstood and would welcome the authors’ responses to the comments below.

There is a statement (section 5) “we focus on certified runtime defences that are deployed during model evaluation and are guaranteed to detect adversarial examples”. I would contend that certifiable robustness processes are usually such that the certified robustness (CR) check (for radius epsilon) passing at x implies the classifier is epsilon-robust at x. But the converse is not necessarily true – as evidenced by the fact that different certified robustness approaches perform differently. Failure of the CR check at radius epsilon I am not sure guarantees the presence of an adversarial point in the epsilon ball. It simply means that there wasn’t appropriate evidence to guarantee there is no such point.

Given this, in section 3.2.2, it is stated that if a point x is 2*epsilon-robust, then every point within the epsilon ball around x is epsilon robust. This is true. But it is then said that “and so (the classifier) cannot be forced to unnecessarily reject the input”.  I am not sure of the latter statement, since (as argued above) the CR check may fail at that point, but that does not mean it is not epsilon robust at that point.

Similarly, the statement in 3.2.2 that “if the 2*epsilon CR check fails at x, then even though the model is epsilon robust at x, there exists an x’ in the epsilon ball at x where the model is not epsilon-robust” (paraphrasing a bit). But I would contend that failure of the 2*epsilon CR check does NOT mean that there is an adversarial example in the larger ball. It could still be 2*epsilon robust (and fail the check), and hence the paper’s conclusion is not true. 

In the same vein, I have problems therefore in section 3.3.1 in say defining test_R. How do we determine the set of points on which the model is epsilon robust (according to Definition 2). We can find the set of points on which it is certified robust by our check, but this will presumably miss points at which the test fails even though it is robust. Is it this certified robust set instead which is meant here to be test_R? Also, if this is the case, this is not really establishing “A lower bound on attack efficacy” – it is actually evaluating the efficacy of the given attack. This establishes a lower bound on the fraction of test points which are vulnerable to degradation attacks. Other degradation attacks may fare worse. Hence I question the wording of the title of this section and the way it is presented.

Regardless of the above discussion re the correctness of the approach, I have an issue with the phrasing of the basic claims of the paper. It is claimed (rightly) that an input point may be perturbed within the epsilon ball and no longer pass the robustness check, even though the original point did. So the perturbed point is then flagged as not-certifiable. It might be arguing semantics, but the system is really I would contend not flagging adversarial points, but just indicating if a given point is certifiably robust. In this case (the attack), it no longer is certifiably robust, and the system indicates this. The statement (section 5) that the system can “flag even non-adversarial inputs as adversarial” I feel then is a bit of a misrepresentation of what it is doing (And to call these perturbed points non-adversarial inputs also is a bit disingenuous. They are constructed to mislead the system and so, in any reasonable sense, are adversarial).

Also, some discussion in the paper I believe is warranted to address the fact that certified robustness is typically (e.g. in the case of Cohen et al) probabilistic, i.e. up to a certain confidence level (ignored also in the above arguments).

In the experiments, I think some comparison to say PGD would be valuable, i.e. investigate how successful (accidentally!) PGD (in an epsilon ball NOT 2* epsilon) is at finding degradation attacks at robust points. PGD will fail to find an adversarial example at such points of course, but checking whether the (failed) attack point is still certified robust would be an interesting measure to see how more efficient at finding such points the algorithms proposed in the paper are. Further, one could consider for the other points in the test set, how effective the proposed algorithms are at finding adversarial examples compared to vanilla PGD. i.e. do the proposed algorithms degrade efficacy of an adversarial attack on non-robust points?

Some minor comments:
-	It is worth noting that Cohen et al, for example, evaluates the arg max (label) for each noise-perturbed input, and then the counts of different classes go into the evaluation. Instead, in Algorithm 3.2 (line 5), it takes the average of the logits for each input in evaluating the loss. This approach to constructing the smoothed classifier is more akin to the approach of Lecuyer et al. Not necessarily an issue – as I am not sure that there would be a direct analog of the Cohen approach in determining the gradient. But worth highlighting.
-	There is a slight abuse of notation in lines 4 and 5 of Algorithm 3.2. The decomposition of the noise vectors into n d dimensional vectors is obvious, but not explicitly stated in the definition
-	When it is stated in 3.3.1 that “assuming there exists no other point in M, except x, that is epsilon-close to x’” presumably this should say “.. point of a different class label ...” 
-	It would be good to show both lower and upper bounds on the same graphs (fig 2) so as to visually show how tight (or otherwise) they are
-	Is it not the case that the defense 4.2 is not really practical as membership of M is not really computable typically.
 Though the idea of the paper is interesting, and the basic attack techniques proposed seem effective, I had, besides some more semantic issues, some concerns with the correctness of the argument of the paper with respect to the upper bound determination. This means that I cannot accept the paper in its current form (unless of course the authors correct any misunderstanding on my behalf – which is very possible). 

Edit: The author revisions have addressed some of the main issues with the upper bound determination (including a discussion of monotonicity). Interestingly, the inclusion of the PGD attacks in the supplementary materials, as per my suggestion, seems to perform about as effectively as the proposed algorithms in the main paper. Given the improvements, I have increased my rating to reflect that and the fact that the underlying idea is interesting.

 We have added the results from evaluating our degradation attack against an $\ell_\infty$ defense (Appendix C). We attacked models that are defended using the convex outer adversarial polytope approach of Wong & Kolter(2018), a state-of-the-art certified $\ell_\infty$ defense. Our results show that $\ell_\infty$ defenses are even more susceptible to degradation attacks than $\ell_2$ certified defenses. For a model trained on the MNIST dataset that achieves 98.9% VRA against $\ell_\infty$ perturbations bounded by $\epsilon=0.1$, our simple degradation attack succeeds approximately 92% of the time.

While striking, these results are not entirely unexpected, given the geometry of $\ell_\infty$ space. Defenses based on robustness certification are susceptible to degradation attacks unless the underlying model is $(2\epsilon,\ell_p)$-robust. In $\ell_\infty$ space, the volume contained by a ball overlaps much more closely with $[0,1]^d$ (the typical the domain for, e.g., image data), which intuitively means that the domain is “used up” much more quickly
as the required robustness radius increases. By contrast, in high-dimensional Euclidean space, the radius of the ball can be as large as $\sqrt{d}$ before it necessarily envelops the entire domain.

This evaluation against the KW $\ell_\infty$ defense addresses the suggestion by reviewer **rHQj** to evaluate $\ell_\infty$ defenses and the suggestion by reviewer **akrY** to evaluate against the KW defense. We plan to incorporate these results into the main body of the paper for the camera-ready version.


This work investigates an observation that certifiably robust neural networks flag inputs for which local robustness checks fail, but are classified consistently with all valid inputs within a distance of $\epsilon$. The authors actuate this observation in an attack where norm-bounded changes degrade the utility of certifiably robust networks by forcing them to reject otherwise correctly classifiable inputs.  While a number of concerns on clarity, the overall observation made by the authors is simple, useful, and evaluated thoroughly. It is, however, unclear whether the observation is novel under the current manuscript. See below for more details:

- "One need not assume that the inputs are scaled to be in $[0,1]^{d}$ , and hence we can drop the second optimisation constraint." This constraint is there because neural networks typically operate on bounded inputs, where there are no guarantees on inputs which do not follow the scaling rule the network uses for preprocessing. I assume the argument for why this can be dropped is this constraint is replaced by ensuring the adversarial belongs in the support set. Please confirm, and update the text s.t. this is explained explicitly. 
- While Figure 1 is clear, it would be useful if the motivation could be based on a practical example that could come up in practice, s.t. the practical application of the method presented in the paper is clear to the reader. 
- Can the authors clarify why Algorithm 3.1 is effective? First, clarify why simply applying an off-the-shelf white-box attack with epsilon bound would fail, then explain why projecting the perturbation from an off-the-shelf white-box attack with double the epsilon bound projected onto the epsilon sphere works. Will the projection step project onto a sphere for any valid Lp norm?
- For attacking stochastic inputs, the authors should cite any attacks which have used a similar formulation. Many attacks against stochastic defences exist, most prominent is expectation over transformation (https://arxiv.org/abs/1707.07397). Evidence of a literature review should be provided. 
- Could the authors visualize random examples found by the attack algorithm which fool the local robustness check? 
-A much clearer, description integrating sections 3.3.1 and 3.3.2 is warranted. From what I can tell, $\text{test}\_{R}$ is the same in both sections. $\text{test}\_{A}$ is identified by the degradation attack algorithms presented in the earlier section in 3.3.1, and $\text{test}\_{A}$ is identified by modifying the local robustness check in 3.3.2? If this is incorrect, please clarify. Regardless, these sections should be integrated and presented more clearly. 
-The commentary on "overapproximation" requires an extended discussion, where the concepts are introduced in a less rushed manner. 
- "A plot for the randomised smoothed models as compared to a table for the GloRo Nets is simply a reflection of the slightly different ways in which these models are evaluated." While emphasising the different ways these models are evaluated is appreciated, that by itself isn't a reason to present results in different ways. If by "reflection" the authors are implying they have to present results in different ways, then this should be explicitly clarified. If not, if the authors can compute the table computed for GloRo Nets for randomised smoothing, they should do so. Likewise, if the authors can make the plot made for randomised smoothing for GloRo Nets, they should do so. Comparing and contrasting the tables and/or the plots would also be informative to the reader. While I understand the subsequent text after this quote was intended to clarify this, it was insufficient. 
-"However, we re-train all the models using twice the values that we want to defend against." Given the commentary prior to experiments, the authors should also evaluate without re-training all models using twice the values. 
-"None of these definitions capture the idea that, to be protected from adversarial examples, a model only needs to be locally robust at every input in the support set, M." This makes the claim on novelty unclear. The formalization of the support set was only present in the presentation, it was not leveraged in neither the proposed attacks nor the proposed defenses (as sec. 4.2 leaves it to future work). What was actuated upon is the notion of "$2\epsilon$", which, as the related work shows, a number of works have explored. The authors need to further clarify the contribution relative to the existing works from the perspective of the proposed algorithms evaluated in the work. 
 Since the novelty of the paper currently needs further clarification, I am providing a borderline rating, subject to the author's rebuttal. 

AFTER REBUTTAL: The author's rebuttal did not sufficiently convince me of the novelty of the contribution, and thus the paper remains borderline in my estimation. However, I will be raising my score. 

 Thanks for the extensive discussions, I have decided to raise my score. Overall, while I remain concerned regarding the novelty of the contribution, I find the paper to provide valuable insights. Please devote space to an extended discussion regarding the novelty of the claimed central contribution relative to existing work in the revision. 

 Dear reviewers,

We have uploaded a revised version of the paper. Following is the list of changes we made:
- Added (in the appendix) an evaluation of the efficacy of off-the-shelf PGD with $\epsilon$-bound as a degradation attack algorithm  (suggested by reviewers **TyKj** and **YcnJ**)
- Added (in the appendix) visualization of some successful degradation attack inputs demonstrating that the $\epsilon$ values we used lead to visually imperceptible perturbations (suggested by reviewers **TyKj** and **rHQj**)
- Replaced (in Section 3.3) the *table* showing the attack efficacy upper bound results for GloRo Nets (Table 3 in original version) with *plots* (Figure 2 in revised version) similar to the ones used for presenting the Randomized Smoothing results (suggested by reviewers **TyKj** and **rHQj**)
- Updated the text in the paper with the following clarifications that we promised in our previous response (based on questions and comments by all the reviewers):
    - (Section 2) Removed line “One need not assume that the inputs are scaled to be in $[0,1]^d$, and hence we can drop the second optimization constraint”, since whether the second constraint is dropped or not does not affect the results of this paper (reviewer **TyKj** )
    - (Section 3.2) Updated the algorithms to use a general $\texttt{Project}$ operation and specified the projection operation for $\ell_2$ and $\ell_\infty$ spheres (reviewer **TyKj**)
    - (Section 3.2) Clarified that the algorithms are only meant to serve as a proof-of-concept. This is to highlight that we do not claim the novelty of our attack algorithms as primary contributions (reviewers **TyKj** and **akrY**  )
    - (Section 3.3) Mild reorganization so that the details of how $test_R$ and $test_A$ are constructed appear before Section 3.3.1 (reviewer **TyKj**)
    - (Section 3.3.1) Explained our choice of $\epsilon$ (reviewers **TyKj** and **rHQj** )
    - (Section 3.3, 3.3.1, 3.3.2) Clarified the text describing the construction of $test_R$ and $test_A$ to justify the validity of our lower and upper bounds (reviewer **YcnJ**)
    - (Section 3.3) Clarified that the bounds for Randomized Smoothing are probabilistic (reviewer **YcnJ**)
    - (Section 3.3.2 and Section 4.1) Updated the text to describe GloRo Net upper bound results in terms of plots instead of the present table (reviewers **TyKj** and **rHQj**)
    - (Section 4.1) Minor modifications to the text to clarify the idea behind defense via radius doubling (reviewer **rHQj**)
    - (Section 5) Clarified the description of global robustness by Leino et al.(2021) (reviewer **TyKj**)
    - (Section 5) Clarified the sentence “we focus on certified runtime defences that are deployed during model evaluation and are guaranteed to detect adversarial examples” (reviewer **YcnJ**)
    - Added (in the appendix) a definition of monotonic local robustness certifiers and proofs showing that complete certifiers and Lipschitz-based certifiers are monotonic. This helps justify the claim that we are computing upper bounds (motivated by a question posed by reviewer **YcnJ**)
    
We are working on adding an evaluation of degradation attacks against certified $\ell_\infty$ defenses (as suggested by reviewer **rHQj**), and will add these results to the paper as soon as they are available.


 Many thanks to the authors for responding to my comments. The clarifications and proposed corrections help.

 > In the same vein, I have problems therefore in section 3.3.1 in say defining test_R. How do we determine the set of points on which the model is epsilon robust (according to Definition 2) …

A degradation attack is successful at a point $x$ if the certified defense procedure declares the model to be $\epsilon$-locally robust at $x$, but we can find a point $x’$ in the $\epsilon$-ball centered at $x$ such that the defense cannot certify local robustness at $x’$. In other words, a degradation attack is successful against a defense if it manages to degrade the performance of the defense by forcing unnecessary rejections.

Consequently, $test_R$ is the set of test inputs on which the model is *certified* $(\epsilon,\ell_p)$-locally robust. We will add this clarification in Section 3.3.1. To construct $test_R$, we only need to apply our $(\epsilon, \ell_p)$-local robustness check at every input in the test set. We do not need to determine the set of points on which the model is $\epsilon$ robust.

The numbers reported in 3.3.1 are indeed the attack efficacy results for the specific attacks we present in Section 3.2. We refer to these as lower bounds since our attacks help demonstrate a lower bound on the percentage of certified test inputs that are susceptible to degradation-style attacks.

> Regardless of the above discussion re the correctness of the approach, I have an issue with the phrasing of the basic claims of the paper. It is claimed (rightly) that an input point may be perturbed within the epsilon ball and no longer pass the robustness check, even though the original point did. So the perturbed point is then flagged as not-certifiable. It might be arguing semantics, but the system is really I would contend not flagging adversarial points, but just indicating if a given point is certifiably robust. In this case (the attack), it no longer is certifiably robust, and the system indicates this. 

Although it may be independently useful to certify a model as locally robust at a point, such checks seem to be used most frequently in the context of defending against adversarial examples, i.e., flagging adversarial examples. Our argument is simply to say that when used as adversarial example detectors, robustness checks expose the model to utility degradation attacks.

> The statement (section 5) that the system can “flag even non-adversarial inputs as adversarial” I feel then is a bit of a misrepresentation of what it is doing (And to call these perturbed points non-adversarial inputs also is a bit disingenuous. They are constructed to mislead the system and so, in any reasonable sense, are adversarial).

Although a perturbed input may be generated by an adversary, only perturbed inputs that lead to misclassification have been referred to as adversarial examples in the literature, and we find it fruitful and important to distinguish between these two concepts. Perhaps, in hindsight, the community could have used “adversarial examples” to refer to all inputs perturbed by an adversary, and “successful adversarial examples” for perturbed inputs that lead to misclassification, but given the constraints of existing nomenclature, we believe that our usage of non-adversarial inputs is justified.

> Also, some discussion in the paper I believe is warranted to address the fact that certified robustness is typically (e.g. in the case of Cohen et al) probabilistic, i.e. up to a certain confidence level (ignored also in the above arguments).

It is true that all the upper and lower bounds computed for the randomized smoothing defense are only probabilistically valid because the certified radius is calculated by Cohen et al using a statistical procedure, and we will add a clarification to this effect.

> In the experiments, I think some comparison to say PGD would be valuable, i.e. investigate how successful (accidentally!) PGD (in an epsilon ball NOT 2* epsilon) is at finding degradation attacks at robust points.

Thank you for the suggestion.  This would indeed be interesting, and we are working on adding an evaluation of the attack efficacy of PGD with an epsilon bound to the revised version of the paper.


 > There is a statement (section 5) “we focus on certified runtime defences that are deployed during model evaluation and are guaranteed to detect adversarial examples”. I would contend that certifiable robustness processes  …

Our apologies; we will clarify this statement; it was intended to be interpreted to mean that *if a point is an adversarial example then it is guaranteed to be flagged*, not that all flagged points are correctly identified as adversarial examples.

> Given this, in section 3.2.2, it is stated that if a point x is 2*epsilon-robust, then every point within the epsilon ball around x is epsilon robust. This is true. But it is then said that “and so (the classifier) cannot be forced to unnecessarily reject the input”. I am not sure of the latter statement, since (as argued above) the CR check may fail at that point, but that does not mean it is not epsilon robust at that point.

This is a good point, although it does not pose a problem in our context. The defenses we consider have the property that if a point is certifiable at a radius of $2\epsilon$ then all points within the $\epsilon$ ball of the point are certifiable at a radius of $\epsilon$, formalized in the Lemmas below. We will clarify our claims with the additional relevant context about the defenses being evaluated, and provide proofs of the following claims in the appendix.

**Definition (monotonicity of certification methods)**:
We say that a certification method $\texttt{cert} : \mathbb{R}^d\times \mathbb{R}^+ \to \textit{bool}$ is _monotonic_ if
$$
\forall x, x’ \in \mathbb{R}^d , \ \epsilon > 0 \quad.\quad   \texttt{cert}(x, \epsilon) \land ||x - x’|| = \delta < \epsilon \quad\Longrightarrow\quad  \texttt{cert}(x’, \epsilon - \delta)
$$

**Lemma 1**:
An _exact_ certification method is one for which $\texttt{cert}(x, \epsilon) \iff \forall x’\in\mathbb{R}^d . ||x - x’||_p \leq \epsilon \Longrightarrow F(x) = F(x’)$ (in other words, certification is equivalent to local robustness). All exact certification methods are monotonic.

This follows fairly easily from the triangle inequality.

**Lemma 2**:
A _Lipschitz-based_ certification method is one which, given an upper bound, $K_{ij}(x, \epsilon)$ on the local Lipschitz constant of the margin $f_j - f_i$ at $x$ (note that the global Lipschitz constant is such a bound), certifies a point $x$, where $F(x) = j$, exactly when
$$
\max_{i \neq j}\left[ f_i(x) + \epsilon K_{ij}(x, \epsilon) \right] \leq f_j(x).
$$
All Lipschitz-based certification methods are monotonic.


We note that monotonicity directly tells us that our upper bounds are valid. GloRo Nets are Lipschitz-based, therefore they are monotonic, and our upper bound is valid. If it were not for the nondetermism/uncertainty that arises because of the need to sample in Randomized Smoothing (RS), it would otherwise be considered a complete method. This means that our upper bounds for RS are _probabilistic_ upper bounds, similar to how RS gives probabilistic robustness guarantees. That is, our bounds would be exact if we had taken the infinite sample limit. We believe this is consistent with how other metrics are reported on RS in the literature, e.g., VRA, which is reported as if guarantees are taken at face value.

> Similarly, the statement in 3.2.2 that “if the 2epsilon CR check fails at x, then even though the model is epsilon robust at x, there exists an x’ in the epsilon ball at x where the model is not epsilon-robust” (paraphrasing a bit). But I would contend that failure of the 2epsilon CR check does NOT mean that there is an adversarial example in the larger ball. It could still be 2*epsilon robust (and fail the check), and hence the paper’s conclusion is not true.

Yes, in the context of incomplete defenses there may not be such an $x’$. We will clarify this statement; if the check is incomplete, it should say “there may exist” rather than “there exists.” However, this does not affect the conclusion that this method provides an upper bound on the success of the attack. In the best case for the attacker, such an $x’$ _does_ exist, and can be found by the attacker. If the defense is incomplete, our bound may not be tight, but it is an upper bound nonetheless. Therefore we don’t think this affects the paper’s conclusion.


 > I doubt the practical usefulness of this degradation attack. In practice, verification is a plus for the model robustness evaluation. It is safe to trust the results if a sample is verified but a sample that cannot be verified does not necessarily mean we have to reject it. There are many other ways to measure the robustness and give confidence scores of the predictions. GloRo proposes to do the rejection mainly because they want to have global verified robustness with global Lipschitz constant but eventually it still serves for local robustness verification.

Yes, for some applications, if a sample is not verified, it does not necessarily have to be rejected. However, this choice is application-dependent. For safety-critical applications where mislabeling a perturbed input can harm life and property, unless a sample can be verified, rejection may be the appropriate choice.

Moreover, even if unverified points are not rejected, one has to pay the computational cost of  executing the alternative checks. By frequently causing the robustness verification methods to fail unnecessarily, an adversary can increase the computational costs.

> I do not think the example in figure 1 makes sense. When the ground-truth robustness region of x' is already crossing the decision boundary, it is truly an adversarial example. Rejection is indeed the correct thing to do for deterministic verification methods like GloRo unless we can retrain the weights to reshape the decision boundary. Please note that over cautiousness is the intrinsic property for certifiable defenses and should be the correct thing to have. 

A local robustness check is only a proxy for detecting adversarial examples. Figure 1 demonstrates the relationship between the concepts of adversarial examples and local robustness. Even though the model is not locally robust at $x’$ in Figure 1, $x’$ itself is NOT an adversarial example, since it shares its label with all in-distribution points that are less than $\epsilon$ apart, i.e., points in the white region including $x$.  Put another way, local robustness at a point implies the point is not adversarial, but the lack of local robustness at a point does not imply that it is adversarial.


> The local verified robustness should only focus on the samples belonging to the original input distribution (x belongs to M) while the verified robustness around other samples like perturbed ones that out of the distribution (x' not belongs to M) should not be encouraged. Otherwise, it will become an unnecessarily too difficult problem unless global verified robustness is required.

We fully agree with the reviewer’s statement that “local verified robustness should only focus on the samples belonging to the original input distribution (x belongs to M) while the verified robustness around other samples like perturbed ones that are out of the distribution (x' not belongs to M) should not be encouraged.” This is exactly our insight. 

Our point is that at test-time it is *unknown* if the test input $x$ belongs to $M$ or not. As noted by reviewer rHQj,  the learner only gets the perturbed input at test-time instead of the original input, so some information about the original input is already lost. Therefore, by checking for verified robustness at test-time, existing defense techniques are encouraging robustness not only at points in the original input distribution, but also at points that are out of the distribution. We show how this can be exploited by an attacker to cause degradation attacks.

> The smoothed PGD attack has very limited novelty. A similar method is already proposed and also used for training to achieve better results over randomized smoothing alone. The attacks should at least discuss and evaluate over [A].

Algorithm 3.2 (smoothed PGD attack) is not meant to be novel, but rather to serve as a proof-of-concept to demonstrate the existence of degradation attacks against stochastic defenses. It is possible that an even simpler algorithm suffices, but this would only serve as even stronger evidence for the ubiquity of degradation attacks. Alternatively, it is possible that a more sophisticated algorithm may be more successful at finding degradation attacks, but this would again strengthen our point that degradation attacks are ubiquitous.


> The proposed attack should really be discussed and evaluated with other SOTA complete verifiers like alpha-beta-CROWN ...

Note that (1) using a complete verifier does not necessarily prevent the attack from being successful, since the threat of degradation attacks is independent of whether the defense is complete or only conservative when flagging potential adversarial examples; and (2) GloRo Nets are SOTA when it comes to deterministic L2 certification; e.g., they outperform K&W.


 > I find the discussion for the double-radius defense confusing. From Algorithm 2.1, it seems that a certified run-time defense will abstain at a radius. However, in Section 4.1, the models are said to be "trained" with parameter $2\epsilon$ . Does it mean that the models will abstain at radius $2\epsilon$ during test-time? If yes, then it just opens a larger feasible set for degradation attack check. Could you please explain this more formally with helping lemma stating the guarantee of the defense mechanisms?

The double-radius defense trains the model with radius $2\epsilon$, i.e., twice the required robustness radius, but uses only radius $\epsilon$ during test-time for checking robustness. In other words, during test time the model will abstain from prediction only if it cannot be shown to be robust with radius $\epsilon$.

When we say “trains the model with radius $2\epsilon$” we simply mean that the robust training procedure is modified to encourage the ability to certify twice the radius that will be ultimately used at test time.

This defense strategy is motivated by the geometric intuition that if the model is locally robust at a point $x$ with radius $2\epsilon$, then for every point $x’$ such that $||x-x’||_p \leq \epsilon$, the model is locally robust at $x’$ with radius $\epsilon$.

Formally, if a classifier $f \in \mathbb{R}^d \to L$ is $(2\epsilon, \ell_p)$-locally robust at $x \in \mathbb{R}^d$, then
$$
\forall x’ \in \mathbb{R}^d. ||x-x’||_p \leq \epsilon \implies f \textit{ is } (\epsilon,\ell_p) \textit{-locally robust at } x’
$$

By training the model to be locally robust with radius $2\epsilon$, we hope that the trained model is robust with radius $2\epsilon$ at the valid in-distribution inputs, and therefore protected from degradation attacks with respect to radius $\epsilon$. Note that the double-radius defense is *not* guaranteed to offer protection against degradation attacks, and only serves as a heuristic defense.

> What's the choice of your attack parameter $\epsilon$? The choice now looks quite arbitrary. Please show degradation AE created at each so that the readers know the visual similarity at the perturbation level.

We are working on generating these visualizations, and will add them to the revised version of the paper. Note that because the examples found are within a distance of $\epsilon$ from the point being attacked, they should be interpreted as “imperceptible” perturbations, so we expect visualizations will be hard to distinguish from the points being attacked.

The $\epsilon$ values chosen for the attacks on the GloRo Net models are the same as the $\epsilon$ values used by Leino et al.(2021) in their experiments for evaluating the GloRo Net models. The $\epsilon$ values for attacking randomized smoothed models were chosen so that the perturbations are imperceptible, and these are commonly used values for $\epsilon$.

> What's the gap between the lower and upper bound of the attack? Is there any pattern/conclusion we can find from the gap?

The lower bounds represent the percentage of certified robust test points that could be subjected to a degradation attack using the algorithms from Section 3.2. The upper bounds represent the percentage of certified robust test points that are susceptible to a degradation attack.

The lower bound numbers tell us that even with the simple algorithms from Section 3.2, it is easy to subject state-of-the-art models and certified defenses to degradation attacks. The gap between the lower and upper bounds indicates that with more powerful and sophisticated attack algorithms, there is the possibility that a much larger percentage of test points could be subjected to degradation attacks, conveying the severity of the problem.

> How about the result for other $\ell_p$ distances like $\ell_\infty$?

We are working on implementing and evaluating the degradation attack on models with $\ell_\infty$ certified defenses. We will add these results to the revised version of the paper.

> Also, is there a reason only showing the upperbound for GloRo in table and that for Randomized Smoothing in figure?
 
We are working on revising the paper to replace the tables for GloRo Net results with plots.

The original presentation was driven by the fact that Gloro Nets either output the predicted label if the model is locally robust or output $\bot$ if the model is not locally robust. In contrast,  randomized smoothed models output their prediction along with the radius of the region that can be certified robust. This difference prevented us from presenting the GloRo Net results in figures similar to the randomized smoothing results. However, we are working on modifying the GloRo Net implementation so that we get the certified radius along with the prediction, and can generate plots similar to randomized smoothing.


 > "However, we re-train all the models using twice the values that we want to defend against." Given the commentary prior to experiments, the authors should also evaluate without re-training all models using twice the values. 

In Table 3 and Section 3.3.2, we do evaluate the upper bounds on the false positive rates for GloRo models trained without twice the $\epsilon$ values. In Table 3, these results are reported in the columns labeled **S**. Columns labeled **D** report the results when the models are trained using twice the required $\epsilon$ value, i.e., defense via radius doubling.


> "None of these definitions capture the idea that, to be protected from adversarial examples, a model only needs to be locally robust at every input in the support set, M." This makes the claim on novelty unclear. The formalization of the support set was only present in the presentation, it was not leveraged in neither the proposed attacks nor the proposed defenses (as sec. 4.2 leaves it to future work). What was actuated upon is the notion of $2\epsilon$, which, as the related work shows, a number of works have explored. The authors need to further clarify the contribution relative to the existing works from the perspective of the proposed algorithms evaluated in the work. 

Though the notion of the support set is not explicitly used in our attacks, it has directly inspired the observation that an input is adversarial *only* if it does not share its label with $\epsilon$-close points in the *support set*. Moreover, the failure to account for the role of the support set in previous work leads to test-time defenses that are susceptible to degradation attacks.  Thus, we believe that the concept of a degradation attack is the primary contribution of our work, and it depends directly on the notion of accounting for the support set. The attack algorithms we present in Section 3.2 are only proofs-of-concept to convey the ease of subjecting state-of-the-art models to degradation attacks.

Regarding the notion of $2\epsilon$, we believe you are referring to the description of GloRo Nets and global robustness from Leino et al.? Our description was less than clear, but Leino et al. use $2\epsilon$ in a different manner. They simply want to define “global” robustness as having a margin (predicted as $\bot$) of sufficient width between classes. Their observation is that checking points for $\epsilon$-local-robustness corresponds to a margin of width $2\epsilon$; i.e., there is a factor of two difference when referring to global vs. local robustness. This is independent of the issue we raise. Our point can be thought of in the context of Leino et al. as the following: if a point is at distance $\epsilon$ from the margin, it has a point in its $\epsilon$-neighborhood that lies in the margin. Thus, an attacker can target points within $\epsilon$ from either side of the margin boundary, meaning that there will be a margin of $4\epsilon$ separating points that are robust and invulnerable to degradation attacks. i.e., the $2\epsilon$ in question in Leino et al. is simply a conversion between local and global robustness, not related to defending against degradation attacks.


 > "One need not assume that the inputs are scaled to be in $[0,1]^d$ , and hence we can drop the second optimisation constraint." This constraint is there because neural networks typically operate on bounded inputs, where there are no guarantees on inputs which do not follow the scaling rule the network uses for preprocessing. I assume the argument for why this can be dropped is this constraint is replaced by ensuring the adversarial belongs in the support set. Please confirm, and update the text s.t. this is explained explicitly.

One can assume that support set $M$ only has appropriately scaled inputs. An adversarial example does not necessarily belong to $M$ but is, at most, $\epsilon$ apart from a point in $M$. We say that the second optimization constraint can be dropped because one may use a different scaling, so dropping the second constraint gives a more general, scale-independent definition of adversarial examples. However, all the results presented in this paper hold, irrespective of whether this constraint is included or not.
We will make this clear in the text.

> Can the authors clarify why Algorithm 3.1 is effective? First, clarify why simply applying an off-the-shelf white-box attack with epsilon bound would fail, then explain why projecting the perturbation from an off-the-shelf white-box attack with double the epsilon bound projected onto the epsilon sphere works. 

Applying an off-the-shelf white-box attack with an epsilon bound would not necessarily fail. We are working on adding an evaluation of the efficacy of such an attack to the revised version of the paper.

Note that Algorithm 3.1 is only meant to serve as a proof-of-concept to demonstrate the existence of degradation attacks. It is possible that an even simpler algorithm suffices, but this would only serve as even stronger evidence for the ubiquity of degradation attacks.

Algorithm 3.1 works because whenever an off-the-shelf white-box attack with double the epsilon bound succeeds, i.e., the classifier label at $x''$ is different than the label at $x$, it is guaranteed that the classifier is not $(\epsilon,\ell_p)$-locally robust at $x'$, the projection of $x''$ onto the $(\epsilon, \ell_p)$-sphere centered at $x$ (this is because $x'$ is at distance  $\leq \epsilon$ from $x''$). Moreover, if the classifier is $(\epsilon,\ell_p)$-locally robust at $x$, then $x'$ is a degradation attack input. Note that the success of this degradation attack does not depend on the method used for certification in the sense that if the white-box attack at $2\epsilon$ succeeds, the degradation attack will also succeed regardless of the details of certification.


> Will the projection step project onto a sphere for any valid Lp norm?

The projection step in Algorithm 3.1 works for $\ell_2$ norm. This suffices for degradation attacks on GloRo Net models and Randomized Smoothed models as they are both defenses against $\ell_2$ norm adversaries. We will clarify this in the paper.

It is also easy to modify Algorithm 3.1 to work with $\ell_\infty$ norm by simply clipping $x''$ in line 3.

> Could the authors visualize random examples found by the attack algorithm which fool the local robustness check? 

We are working on generating these visualizations, and will add them to the revised version of the paper. Note that because the examples found are within a distance of $\epsilon$ from the point being attacked, they should be interpreted as “imperceptible” perturbations, so we expect visualizations will be hard to distinguish from the points being attacked.

The $\epsilon$ values chosen for the attacks on the GloRo Net models are the same as the $\epsilon$ values used by Leino et al.(2021) in their experiments for evaluating the GloRo Net models. The $\epsilon$ values for attacking randomized smoothed models were chosen so that the perturbations are imperceptible, and these are commonly used values for $\epsilon$.

> From what I can tell, $test_R$ is the same in both sections. $test_A$ is identified by the degradation attack algorithms presented in the earlier section in 3.3.1, and $test_A$ is identified by modifying the local robustness check in 3.3.2? If this is incorrect, please clarify.

This is correct. In Section 3.3.1, $test_A$ represents the set of points that could be successfully attacked using the algorithms from Section 3.2, whereas $test_A$ in Section 3.3.2 represents the set of points that could be susceptible to a degradation attack.  We are revising these sections to make the presentation clearer.

> A plot for randomized smoothing vs table for Gloro Nets

We are working on revising the paper to replace the tables for GloRo Net results with plots.


 We thank the reviewers for their thoughtful comments. Based on the review feedback, we are working on revising the paper with updated text and additional experiments, and will be able to share the revised paper next week. We address the reviewers' questions in separate comments below.

Meta-review: The work focuses on the observation that, given a certified epsilon-robust model and a certified clean input x, many inputs within the epsilon ball around x are themselves not epsilon-certifiable although they are correctly classified. The authors argue that an adversary can exploit this property to produce inputs which are correctly classified by the model yet are not certifiably robust. Reviewers agreed that the paper was overall well written, the methods were clear and overall evaluated thoroughly, and many felt that the main idea was interesting. There were some concerns regarding the significance of the contribution, the primary observation itself is arguably novel but somewhat obvious, and the proposed algorithm for finding non-certifiable points isn't a significant contribution when standard techniques like PGD are sufficient. Much of the reviewer discussion concerned whether or not the proposed attack made sense as a threat model. It is the AC's opinion that this discussion did not reach any meaningful conclusions. It is important to remember that the lp threat model is intended as an abstract toy game so that a formal theory of neural network certification can be developed under idealized settings. It is not intended to model any realistic security scenarios, and even more generalized notions of "imperceptible" or "subtle" attacks aren't realistic when for the bulk of applied settings real adversaries are not restricted to small perturbations in the first place [1]. The example provided by the authors regarding small perturbations of a stop sign isn't a relevant example when the adversary has more effective options, e.g. knocking over stop signs [1, Figure 3]. 

For the sake of discussion, one could consider whether or not a degradation attack would make sense under more general threat models such as content-preserving perturbations. An example discussed in [1] concerns adversaries uploading copyrighted content to public streaming services—this attacker defender game is being actively played in the wild where defenders produce statistical models which attempt to flag content as semantically matching existing copyrighted content in a private database, while attackers make large semantically-preserving modifications in order to evade statistical detection. An example attack would be cropping 20% of the boundary pixels of a movie and replacing the cropped portion with arbitrary adversarially constructed backgrounds. Epsilon perturbations are possible, but are almost a measure 0 subset of the full attacker action space. Suppose in the far future neural network certification advanced to the point where we could certify that a classifier was robust to all possible content-preserving perturbations of a specific movie. In this case the defender would be using the certification method on their private database of copyrighted content, they would not be running the certifier on any content uploaded by users. If a movie in the private database is certified, then we already know that an attacker cannot successfully upload an adversarial version of it, it would be unnecessary to certify whether or not user uploaded content could be further perturbed in a way to become adversarial. Perhaps degradation attacks could be possible as a training poisoning attack, but this seems a bit far-fetched when more traditional training poisoning attacks would be preferred. Given this, at least in this example the AC does not see how a degradation attack would make sense as a threat model. 

Given that the primary contribution of this work is a novel threat model for ML security, it is crucial that the authors rewrite their work to make more realistic assumptions of the capabilities of realistic adversaries. Starting with some of the examples discussed in [1] may be useful to the authors. Although the example of adversarial attacks on copyright detection classifiers doesn't seem to fit the degradation attack threat model, perhaps other scenarios would.

1. Gilmer et. al, Motivating the Rules of the Game for Adversarial Example Research, https://arxiv.org/pdf/1807.06732.pdf

Decision: Reject