Degradation Attacks on Certifiably Robust Neural Networks
Authors: Klas Leino
Chi Zhang
Ravi Mangal
Matt Fredrikson
Bryan Parno
Corina Pasareanu
Venue: Accepted by TMLR ()

Abstract: Certifiably robust neural networks protect against adversarial examples by employing run-time defenses that check if the model is certifiably locally robust at the input under evaluation. We show through examples and experiments that any defense (whether complete or incomplete) based on checking local robustness is inherently over-cautious. Specifically, such defenses flag inputs for which local robustness checks fail, but yet that are not adversarial; i.e., they are classified consistently with all valid inputs within a distance of $\epsilon$. As a result, while a norm-bounded adversary cannot change the classification of an input, it can use norm-bounded changes to degrade the utility of certifiably robust networks by forcing them to reject otherwise correctly classifiable inputs. We empirically demonstrate the efficacy of such attacks against state-of-the-art certifiable defenses. Our code is available at https://github.com/ravimangal/degradation-attacks.



Published in Transactions on Machine Learning Research (11/2022)
Degradation Attacks on Certifiably Robust Neural Networks
Klas Leino"
kleino@css.
rrsinoo@csccmmu.coom crna. edu
Cornegie Mellon University
Chi Zhang
chi:5@andrew.cmme crv edu
Cornegie Mellon University
Ravi Mangal* Cornegie Mellon University
Suunggl/andrew..cme craa, edu
Matt Fredrikson
mfvedrik@eema.cmmm edu
Carnegie Mellon University
Bryan Parno
parno@ema. edu
Carnegie Mellon University
Corina Päsäreanu
pendina
www.: cmau. edu
Cornegie Mellon University
Reviewed OpenReview: https: httpe:://penreeviel net/ forum? id= POX05ZE98
openreview.
Abstract
Certifiably robust neural networks protect against adversarial examples by employing run-time defenses that check the model is certifiably locally robust at the imput under evaluation. We show through examples and experiments that any defense (whether complete or incomplete) based on checking local robustness inherently over-cautious. Specifically, such defenses flag inputs for which local robustness checks fail, but yet that are not adversarial; i.e., they are classified consistently with all valid inputs within distance of As result, while norm-bounded adversary cannot change the classification of input, it can use norm-bounded changes to degrade the utility of certifiably robust networks by forcing them to reject otherwise correctly classifiable inputs. We empirically demonstrate the efficacy of such attacks against state-of certifiable defenses. Our code is available at https://gtt httpa:://gttuubcoooto. httphhhwwww......aaagaaagaaaaaaa
1 Introduction
An adversarial example for a neural classifier is the result of applying small modifications to correctly classified valid input such that the modified input is classified incorrectly. For neural classifiers trained in a standard manner, has been shown that adversarial examples are rampant (Szegedy el al. 2014 Carlini Wagner 2017). While training models in an adversarially-awe 2018 Leino al., 2021) can help mitigate the issue an extent, does not solve the problem since these models continue to be susceptible to adversarial examples. In other words, adversarially-aware training, even with certified methods, does not yield models that are guaranteed to be free from adversarial examples.
manner (Madry al. 2018 Wong & Kolter
For neural classifiers deployed in safety-critical applications like autonomous vehicles and healthcare diagnostics, the existence single adversarial example can lead to disastrous consequences. In light of the limitations of training time interventions, how can we ensure that our models are guaranteed protection from adversarial examples? One naive approach to assume that all the valid inputs model will ever encounter are known
"Equal Contribution

---

Published in Transactions on Machine Learning Research (11/2022)
in advance. If we can certify that the model is c-locally robust at each of these inputs, then the model is guaranteed to be free from adversarial examples. However, such an assumption is too strong and is unlikely to be true in practice.
Certifiably robust classifiers (Cohen et al. 2019 Leino et al., 2021 offer the most rigorous solution to the problem and provably protect models against adversarial attacks. They work by providing a (provable) robustness certificate for each examined imput. These papers only discuss and evaluate the certification procedure on finite set of (valid) imputs with the implicit assumption that either the whole set of valid inputs is contained in the evaluated set or that the certification obtained on subset of (valid) inputs at test time generalizes also to unseen inputs at run-time. Both these assumptions may not hold in practice. In order to use such methods in safety-critical situations, we therefore assume that these classifiers use the robustness certificate to construct a certified run-time defense. In fact, we confirmed with the authors that these certified defenses are indeed meant to be used at run-time. The defenses work by by chocking the model e-locally robust at the evaluated input. If the check fails, the model rejects the input as the input can be polentially adversarial. This ensures that the model is only ever used for prediction at inputs where the adversary cannot cause misclassification. However, a rejection is not free of cost. Every time the model rejects an input, user of the model has to resort to some other process other than their presumed first choice (the defended model) to make prediction, reducing model utility. The rate of input rejection is then a natural measure of quality for certifiably robust models but, surprisingly, existing evaluations of such models fail to measure this quantity.
We show that existing certified run-time defenses are overly cautious and susceptible to rejecting non- adversarial inputs. This over-cautiousness is inherent in the design of such defenses, and it is manifested even when the e-local robustness check is exact (i.e., the defense is complete). Not only does this lead to a degradation in model utility because of unnecessary rejections, but it also exposes models to a new line of attack that we refer to as degrudation attacks. We develop new attacks that are aimed at causing certifiably robust classifiers to reject inputs frequently, and we show that this is a significant problem in practice. On MNIST and CI for stale-of-the-art certifiably & robust models, such as GloRo Nets (Leino et al., 2021) and Randomized Smoothed models (Cohen et al., 2019), our attacks successfully find perturbations that lead to unnecessary rejection on many as 57% of test inputs where the model is already known to be robust. On certified &o robust models, namely KW (Wong & Kolter 2018 models, the attack success rates are even higher. This is particularly distressing given the already considerable computational costs of training and defending certifiably robust models. Although we present our ideas in the context of lp norm-bounded perturbation attacks and current defenses, degradation attacks are a concern for any run-time defense that relies on local robustness checks to thwarl perturbation altacks.
As a concrete scenario, consider an autonomous driving system that uses a neural classifier for labeling road signs. It is unsafe for the classifier to misclassify adversarially perturbed road signs, and we want to prevent this at all costs. One simple strategy is for the classifier to always reject inputs and either hand over decision-making to the human driver or bring the vehicle to a standstill (UNECE, 2020). This is perfectly safe behavior, but this model has zero utility. Ideally, we want the model to hand over control (i.e. the run-time defense should raise a flag) only when the perturbed example is actually going to cause misclassification. Our result implies that an adversary can cause the model to hand over control to the human driver even when a perturbed input would not have been misclassifidd. Although this does not happen for all the inputs (as in our simple example), it happens often enough that the adversary can cause drastic and unnecessary reduction in model utility.
Degradation attacks are successful because certified run-time defenses do not account for the data manifold. We formalize valid inputs by means of a set M which is the support set of the underlying input distribution that characterizes the learning problem. Although the exact description of M is unknown, we know that both the training and test data are in M. An incorrectly flagged non-adversarial input is an input at which the local robustness check fails but if the model were not to reject the input, it would be classified consistently with all valid inputs within an e distance. While norm-bounded adversary cannot change the classification of an input in the presence of certified run-time defense, it can apply norm-bounded modifications to existing valid inputs (i.e., inputs in M) and force the model to reject otherwise correctly and consistently classifiable inputs, thereby degrading the model's utility.
2

---

Published in Transactions on Machine Learning Research (11/2022)
As we discuss in Section 5 defenders against degradation attacks have two options: (1) they can train models using existing adversarial training methods with radius 2e when the adversary is only allowed e-perturbations and then compose these models with an existing e-certified run-time defense; (2) they can develop new certified run-time defenses that account for the fact that an input needs to be consistently labeled only with points in M that are e-close. We evaluate the ramifications of the former option that does not. require knowledge of M. However, the latter option may not be actionable since the certifiable method would need access to a mathematically precise definition of M.
To summarize, the main contributions of our work are as follows: (1) we describe new attacks, referred to degradation aftacks, that can force cortifiably robust models (with either deterministic or stochastic defense mechanisms) to frequently and unnecessarily reject inputs; (2) we empirically demonstrate the severity of the problem for models using ooate--f-thhearr certified defense mechanisms like GloRo Nets and Randomized Smoothing; (3) we make explicit the set, M, from which valid inputs are drawn, and this helps us explain that cortified run-time defenses, basod on checking c-local robustness, are susceptible to degradation attacks, they enforce local robustness (i.e., consistency of the predicted label with all e-close inputs) instead of consistency just with inputs in M that are e-close; and (4) we discuss two possible defenses against degradation attacks, and evaluate the one based on training models with double the radius to be enforced by the certified run-time defense.
The rest of this paper is organized as follows. In Section 2 we revisit definitions of adversarial robustness and sketch a general approach for certified defenses. In Section B we demonstrate how a certified defense can unnecessarlly reject inputs; we present degradation attack algorithms to this end, and evaluate their efficacy in Section In Section 5 we sketch two defenses against degradation attacks, and concretely evaluate one of
these proposals. In Section 0
we describe the related work. Finally, conclude in Section 7
2 Adversarial examples and local robustness
A neural classifier f E Rd L, is function from d-dimensional real vector space, Red to a finite set of labels, L. The inputs to the classifier are drawn from some distribution with support set M C RRd In other words, the training and test sets are comprised of clements from M and their corresponding labels. Though the description of M is not available to us, it plays a key role in formalizing the notion of adversarial examples.
An adversarial example for a neural classifier is the result of applying small modifications to a correctly classified valid input such that the modified input is classified incorrectly. Formally, an input is valid if it belongs to M. Definition 2.1 below, adapted from Szegedy et al. (2014), formalizes the notion of targeted adversarial example.
Definition 2.1 (Targeted adversarial example). Given a neural classifier f E Rd + L and a valid input x E M C Id, an imput x' is a targeted adversarial example with respect to target label I E L, an Ep metric, and a fixed constant E IR the solution x' to the following optimization problem is such that || - =11p S c
Minimize ||æl - =zlpp subject to f(x') =
Untargeted adversarial eramples, simply adversarial cramples, are defined similarly except that the constraint [(x') = is replaced with [(x') + f(x)).
The fact that x needs to be a valid input, i.e., x E M, for x' to be an adversarial example is only stated informally in [Carlini & Wagner 2017 Szegody et al. 2014). We make this requirement explicit and formal, via set M; as we shall see, this explicit formulation of M will lead us to the design of our degradation al tacks.
Note that we can formulate the following alternate definition of an adversarial example (Definition 2.2).
Definition 2.2 (Adversarial example (alternate)). Given neural classifier f E IRd - L, an input x' E Rd is
adversarial example with respect to an lp metric and a fixed constant E R if
Fr M. || x lp S E and f(x)) # f(x)
3

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm 2.1: Prediction with a certified run-time defense
Inputs: A model f Rd + I,, an input x' R", an attack bound IR, and distance metric l Output: A prediction I E L. U {1}
Prodictt@tthDeeense(f z', kp): if f is (c,4p)-locall) robust x' then
return f(x')
else
return 1
The two definitions agree on the set of inputs that are identified as adversarial examples. Clearly, if x' is an adversarial example as per Definition 2.2 then it is also adversarial example with respect to x per the untargeted version of Definition 2.1 and vice versa.
A classifier is protected from adversarial examples with respect to a valid input x locally robust at æ.
As stated in Definition 2.3 a classifier is locally robust at x, if its prediction does not change in an e-ball
centered at æ.
Definition 2.3 (Local robustness). A neural classifier f E IRd + L is (e, (poccllyy robust at x E Rd if,
Va' E Rd. ||= - x c = +(xx)= = f(x)
Using the notion of local robustness allows us to formally state when a model is free of adversarial examples (we say that the model is robust).
Theorem 2.4. A classifier is free from adversarial ezamples if it is locally robust at all points in M.
Proof. Let A = Væ E M. E Rd ||x - x Se f(x') f(x). A formally states that classifier f is locally robust at all points in M. Let B == Ex E M.Er' E RR.12 ællp S A f(x') # f(x). Then, BB formally states that the classifier f is free from adversarial examples. Let us assume B. Given A, assuming B leads to a contradiction. Hcnco, A = BB
Note that without M, we could not formally state the requirement of freedom from adversarial examples. For instance, one could ignore M and define classifier to be locally robust at all inputs (both valid and invalid); however such a requirement could only be satisfied by the constant function, rendering the classifier useless. Thus, when studying adversarial examples and corresponding defenses, it appears be essential consider M explicitly.
Practically, since M is not known priori (and may be infinite), one not certify classifier as free from adversarial examples at test time; run-time check is needed. In this paper, we observe that cortified run-time defenses can erroneously flag non-adversarial inputs because they enforce local robustness, i.c., consistency of the predicted label with all e-close inputs, instead of consistency just with inputs in M that are e-close. other words, run-time checks should be based on Definition 2.2 instead of simply checking it the model is locally robust the evaluated input.
Dichotomy between defending against adversarial ezamples and certifying local robustness. To defend against adversarial examples, researchers have proposed methods based on certifying local robustness. guaranteed protection from adversarial examples. These defenses, briefly surveyed in Section G check for local robustness at given input, regardless of whether the input is valid or not, and reject inputs for which the check fails. A Aggorithm 2.1 describes this manner in which certified run-time defenses are deployed for protection.
Classifiers, when composed with certified run-time defenses based on local robustness checks,
At a first glance, the certifiably robust models may seem be working as intended; they reject inputs on which the model's prodiction is not robust. However, the goal of the certified defenses is to protect against adversarial examples (rather than simply ensuring local robustness), we argue that such defense is too
4

---

Published in Transactions on Machine Learning Research (11/2022)
x
Figure 1: A flagged non-adversarial example
Algorithm 3.1: Degradation attack algorithm
Inputs: A model F F + L, an input x D, an attack bound E RR and a distance metric &p Output: An attack imput a' Ra
DegradationAttack((F 2 c, Ey):
x' := Attack (Bzack(F,z,,,,, e,
return x!
strong, the rejected inputs may or may not be adversarial, leading cases when the defense rejects non-adversarial inputs unnecessarily.
Note that even exact (i.e., sound and complete) local robustness certifiers can be overly cautious if deployed at run-time since the model predictions only need to be consistent with valid (i.e., in-distribution) inputs that are 6-close and not with all 6-close inputs. On the other hand, the argument that classifiers abstain unnecessarily because they ignore the data manifold (i.e., M) may appear as non-actionable. For a certifiable method to take ad advattage of the domain valid inputs, would need access to mathematically precise definition of what constitutes valid input (i.e., the data manifold). It appears that even methods that aim to approximate the data manifold cannot be used to actually certify that certain inputs the manifold in any meaningful sense. However, will show in Section 5 that our work does lead to concrete actionable interventions.
3 Degradation attacks
In this section, we describe our degradation attack algorithms.First by means of an example, we demonstrate how certified defenses can lead to unnecessary rejections. Next, we describe our degradation attack algorithms against detorministic and stochastic cortified defensos.
3.1 An example of certified defense failure
Certified run-time defenses can erroneously flag non-adversarial inputs because they enforce local robustness, ie., consistency of the predicted label with all e-close inputs, instead of consistency just with inputs in M that are e-close. Consider the example in Figure | is valid input from M. Let us assume that none of the points in the gray region belong to M. A binary classifier f E IR² > {W, G} assigns label W to all points in the white region (which includes æ and x²) and label G to all points in the gray region. I and x' are apart and the radius of both the circles e. The (e, (2)-ccalll robustness check at x' will fail since the e-ball centered at x' includes white and gray points. However, x' not an adversarial input since it shares its label with all the e-close inputs in M. Consequently, the model rejection of x' is unnecessary and causes a degradation in model utility. This happens if the local check is precise and the bound is tight.
5

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm 3.2: Smoothed projected gradient descent attack (SPGD)
Inputs: A model f E R - | mapping inputs to logit outputs, a loss function C, an input x E M, an attack bound E R,, distance metric lp. step size 7, number of steps N, number of samples n, and a noise parameter a Output: An attack input x' E Rd
e, Eo 71 N, n, 0): SmoothedPgdhttack( ,, e,,
x' == x for Σ sley N do ~ o)
x' == r' + Vo (,E,
a' Project(r,
return r'
3.2 Algorithms
Attacking deterministic defenses. Algorithm 3.1 describes a simple degradation attack against a deterministic defense. Intuitively, given valid input (on which the classifier f is (e, (p)-ocallly robust, for some fixed and distance metric lp), we want to find an imput x' such that || - -''lp S and the classifier f is not (c, eoocaally robust at x'. The attack directly re-purposes existing white-box attack algorithms (like Carlini-Wagnu (Carlini& & Wagner 2017) and PGD (Madry et al., 2018)) that search for adversarial examples. In particular, the attack invokes an off-the-shelf white-box attack algorithm, indicated by Attack, to find an adversarial example x' in a ball of radius e centered at x (line 2). Then, x' returned as the candidate degradation attack input.
Attacking stochastic defenses. Stochastic defenses, such Randomized Smoothing, transform base classifier into a smoothed classifier by convolving the base classifier with Gaussian distribution. The smoothed classifier does not have an explicit representation, and to evaluate it on single input, the base classifier needs to evaluated on as many as 100,000 samples (Cohen et al.] 2019). Moreover, the lack of explicit function representation precludes the use of off-tho-shelf whitc-bon box attack algorithms that perform gradient descent in the input space. While one could simply perform standard white-box altack on the base classifier, we found it was more effective to implicitly attack the smoothed model by taking the average gradient over a set of randomly generated samples at each gradient descent step. This gives us "smoothed PGD" (SPGD) attack tailored for use against stochastic defenses. Our SPGD procedure is given in Algorithm 3.2 and it bears close similarity to the SMOOTHADVren procedure for attacking smoothed classifiers presented by Salman et al. 2019))
Note that Algorithms 3.1 and 3.2 are only meant to serve as proofs-of-concept for degradation attacks. As our empirical results in Section demonstrate, even these simple algorithms can be highly effective degrading model utility.
3.3
Threat model
Both the adversary and the defender can draw samples from M (but they may not have full knowledge of M). The only assumption in our work is that the adversary can see unperturbed inputs generated by Nature (e.g. images taken with camera) and these unperturbed inputs are in-distribution samples, ie., x e M. These are very standard assumptions in machine learning.
The operational details of our attack algorithm are similar to an algorithm for finding an adversarial example in the 2e ball. Technically, our attack is not just finding an adversarial example in the 2e ball. Instead, it tries to find an input z' in the e ball such that the model cannot be certified c-locally robust at x'. However, this similarity a 2e attack actually reinforces the key point that we are trying convey. The use of a run-time local robustness check defense against adversarial examples allows e-oonded adversary to

---

Published in Transactions on Machine Learning Research (11/2022)
exploit adversarial examples in a 2e ball. To be protected from such degradation attacks, the model is then forced to exhibit 26-local robustness at all points in M.
We acknowledge that given the myriad ways, beyond p-norm bounded perturbations, in which Nature or an adversary can causse a model to misbehave, our proposed degradation attacks may seem to have a very narrow scope. However, in line with prior works that question existing definitions of adversarial robust Gilmer let al. (2018); Hendrycks Hendyyyesss
al. (021], our work further highlights the issues with p-norm based definitions of adversarial robustness as security property and point to the incompatibility of such definitions with the semantics of the tasks to which machine learning is applied.
For the evaluation (described in the following section), we further make the empirically justified assump- tion (Yang el al. 2020b) that the data is well-separated, i.e., every pair of valid inpuls in M with different. ground truth labels are at least 2e apart, Please note that degradation attacks can be successful even when the data is not generally well-separated. We only use this assumption calculations and measurements. The adversary does not know or make any well-separaleness assumptions (and neither do our attack algorithms). Indeed, since the definition of M is unknown, the adversary does not know if the constructed attack input x' leads to a successful degradation attack or not. It is only during empirical evaluation, where we actually need to know whether the attack was successful or not, that we have to make assumptions about M.
our experiments for simplifying the
4 Empirical evaluation of attack efficacy
Our empirical evaluation is designed to measure the susceptibility of state-of-the-art robust models and certified run-time defenses to utility degradation attacks. For our experiments, we consider GloRo Nots (Leino) ) and Randomized Smoothed models (Cohen et al. 2019). These approaches lead to models with the best known verified robust accuracies (VRA), with respect to the 62 metric, on variety of popular image classification datasets like MNIST (LeCun et al. 2010). CIFAR-10 (Krizhevsky 2009). and ImageNet (Deng ct al. 2009). In Appendix C we also present an evaluation of degradation attacks on models with certified run-time defenses against &o adversaries (in particular, models trained and defended using the convex outer
ee al.
adversarial polytope approach of Wong & Kolter (2018)).
GloRo Nets incorporate global Lipschitz bound computation into the model architecture. The global Lipschitz bound is used to implement the local robustness check at input under evaluation. Moreover, these local robustness chocks are backpropagated over during training to encourage lcarning of models with high degree of robustness. In contrast, Randomized Smoothing transforms a base classifier into a smoothed classifier by convolving the base classifier with Gaussian distribution. The smoothed classifier typically demonstrates a much higher degree of robustness than the basc classifier as long as the base classifier has been trained to classify well under Gaussian noise. The method uses statistical techniques to compute the local (certified) robustness radius R of the smoothed classifier at a given input, so an (c, f2)-local robustness check simply amounts checking if the radius R at the evaluated input is greater than or not. Radius R is only probabilistcalll valid, so all the results for Randomized Smoothing in this section are also probabilistic statements. However, this probabilistic nature of R is typically ignored when metrics like VRA are reported on Randomized Smoothing in the literature, and we do the same here.
We conducted two sets of experiments. In our first set of experiments, described in Section 4.1 we attack
certifiably robust models at inputs the test set using the approaches described Section 3.2 This demonstrates the efficacy of our attack algorithms, and gives a lower bound on the efficacy of degradation- style attacks in general. Our second set of experiments, described in Section 1.2, measure the susceptibility of models to degradation attacks assuming all-powerful adversary, ie., adversary that succeeds whenever the model is susceptible to a degradation attack at inputs in the test set. This provides an upper bound on the efficacy of degradation attacks.
To measure the efficacy of degradation attacks on a particular model with a certified run-time defense and a datasct, we construct two subsets of the test set for the given datasot, that we refer to as testR and testa- The former, testR, is the set of test inputs on which the model accurate and certified to be (c, llocally robust.. To construct testn, we simply apply the (c, Eoocal robustness check at every input in the test set.
7

---

Published in Transactions on Machine Learning Research (11/2022)
where the model is accurate. If the check passes at an input æ, it is added to testg. The latter, testa, is a subset of tesla, constructed using degradation attack, such that for each input x E lesta the model is certified (e, Hooclly robust at æ but there exists an input x' in the e-ball centered at I such that the model cannot be certified (c, fp)-locllly robust at x'. Since the model is cortified locally robust at 2, we know that x and x' share the same label, so x' is not an adversarial input with respect to x (Definition 2. Further, under the assumptions that there exists no other point of a different class label in M that is 26-close to x (Yang al. 2020b) and for test inputs x E lesta the model accurate on all valid inputs (if any) in the 2e-ball centered at I, x' is not an adversarial example at all (Definition 2.2 Yet, the model would reject x' since the local robustness check fails at x'. Thus, under these assumptions, testa represents the set of test inputs where the model is susceptible to degradation attacks. The ratio | represents the folse posilive rale of the certified run-time defense, i.e., the rate at which the certified defense erroneously causes the model to reject inputs. We measure model susceptibility and attack efficacy in terms of the false positive rate. A value of one indicates that the model is susceptible a degradation attack on all test inputs where it is certified locally robust and accurate, while a value of zero indicates that model is safe from degradation attacks on all certifiably robust and accurate test imputs.
We train GloRo Nets using the publicly available code] For Randomized Smoothed models, we use the pre-trained models made available by the authors [ The GloRo Nets used in the evaluation are composed of convolution and fully-connected layers. For MNIST (e = 0.3), the model has two convolutio layers and two fully-connected layers (2C2F), for MNIST (e = 1.58), the model is 4C3F, and for CIFAR-10 (e = 0.141), the model is 6C2F. For CIFAR-10, the Randomized Smoothed model uses a 110-layer residual network as the base classifier, and for Image Net, a ResNet-50 model is used as the base classifier We implemented our attacks in Python, using TensorFlow and PyTorch. All our experiments were run an NVIDIA TITAN RTX GPU with 24 GB of RAM, and a 4.2GHz Intel Core 17-7700K with 32 GB of RAM.
4.1 Lower bounds on attack efficacy
To evaluate the efficacy of the altack algorithms presented Section 3.2 and compute lower bounds on the efficacy of degradation attacks, we construct an under-approximation of testa (denoted as festa) as follows. An input x E testa is added to testa if the attack algorithm succeeds in finding an attack input x in the c-ball at such that the model cannol be certified robust at w'. Table I presents lower bounds on degradation attack efficacy against GloRo Nets, measured by performing our degradation attack described in Algorithm 3.1 with PGD used for Attack. Table 2 Smoothing, measured by performing our SPGD degradation attack described in Algorithm 3.2 We employ the same values for GloRo Nets as used by Leino et al.] (202T), , and employ commonly used values for Randomized Smoothing that result in visually imperceptible perturbations.
presents lower bounds on degradation attack efficacy against Randomized
We see that against both GloRo Nets and Randomized Smoothing our attacks decrease model utility to varying degrees. In some cases the reduction in utility was substantial, with false positive rates at 50% or
more on MNIST (GloRo Net) and CIFAR-10 (Randomized Smoothing). Overall, falsc positive rates were higher when the robustness radius was relatively higher; e.g., the false positive rate on MNIST GloRo Nets increases from 6% to 57% when the radius increased from 0.3 to 1.58. This may be in part because at smaller radii, many points may be likely to be far more robust than required.
Despite the fact that an explicit representation of smoothed models is not available even a white-box attacker, we find that our SPGD attack was able to successfully degrade the utility of models defended by Randomized Smoothing. On smoothed models we observe that models trained with a larger noisc parameter a are less affected by false positives; however, this comes at the cost of a notable decrease in baseline utility.
In Appendix B.2 models, demonstrating that these attack inputs are visually imperceptible from valid inputs.
we present a visualization of some successful degradation attack inputs against GloRo
http://gtthub.cm//lsstciooooo http:///gttbb.cmm/ouuuaaaaaaa/ap
8

---

Published in Transactions on Machine Learning Research (11/2022)
Table 1: Lower bounds for false positive rates induced by degradation attacks against GloRo Nets. Models are trained and evaluated using the same value.
baseline VRA
false positive rate
utility reduction
(%)
(%)
(%)
MNIST (E = 0.3)
95.0
6.1
5.9
MNIST (E (== 1.58)
61.8
57.0
31.3
CIFAR- 10 (c (0==0. 0.141)
60.0
11.2
6.6
Table Lower bounds for false positive rates induced by degradation attacks against Randomized Smoothing. Similarly the evaluation of Cohen et al. (2019) results are obtained on sample of 100 arbitrary test points, as Randomized Smoothing costly to evaluate.
boseline VRA
false positive mote
adility reduction
(%)
(%)
(%)
CI 0.5)
0.25
48.0
50.0
24.0
CIFAR-10 fe 0.5)
0.50
41.0
29.3
12.0
ImageNet (E L.0)
0.50
46.0
17.4
8.0
MNIST, vary
CIFAR-10, vary
MNIST, vary
CIFAR-10, vary
1.0
18
0.8
08
c 3.16
1.50
0.342
3:16
0282
83
68
06
0.8
02
02
68
1.5
04
(a)
(b)
(c)
(d)
Figure 2: Upper bounds for false positive rates on Gloro Nets
4.2 Upper bounds attack efficacy
To compute upper bounds the efficacy of degradation attacks, i.e., upper bounds the false positive rates, construct over-approximation of testa (denoted testa) follows. We apply (2),oo)loocal robustness check at every input in leslz. If this stronger check passes at input æ, it means that no matter how the adversary perturbs x within the e-ball centered at x, the model is always (e, loccally robust at the perturbed input, and the cortified defense (whether complete or incomplete) cannot be forced to unnecessarily reject the input as long as satisfies a monotonicity property y GloroNets and Randomized Smoothing are both monotonic in this sense (see Appendix A However, the (2e, fp Hl))l robust check fails at æ, then even though the model is certified (e, (,) lo)-llyl robust x, there may exist an input x' in the e-ball æ where the model cannot be certified (c, (p) -locllyy robust and an adversary can force unnecessary rejection. Therefore, if the stronger check fails at v, add it to lesta-
Figures 2a and 2b show the false positive rates at different radii (i.e., values) for GloRo Net models trained to be robust against the e values indicated in the legend. The e values of 0.3 and 1.58 for MNIST and 0.141 for CIFAR-10 are the same as the ones used for evaluating the lower bounds. Figure @ also reports results for models trained to be robust against twice these values. This is for the purpose of evaluating one of our proposed defenses against degradation attacks (see Section 5.1 We see that the false positive rates are quite high and consistently approach one the radius increases. The models trained with higher values are
We say that a certification method cert Rd R+ bool is monotonic Vr, /f Rd > cert(x,e) A | 2'1p < cert(x', e || - l/)]

---

Published in Transactions on Machine Learning Research (11/2022)
CIFARR100 vary
ImageNell vary
CIFAR-10 vary
Imagetec vary
14
18
6-4
0.8
0.50
08
0.050
1.00 1.00
0 1.00
0.8
08
021
022
0-050
6:50
2.00
p 1.00
3.0
2:5
053
375
1.30
125
153
575
130
530
023
038
312
130
125
1.58
175
220
5.00
025
1.50
875
1,09
1.2
1.50
1.5
203
043
025
130
675
140
1.25
1.30
1.5
206
(a)
(b)
(c)
(d)
Figure 3: Upper bounds for false positive rates on Randomized Smoothing
more resilient to degradation attacks, but they pay the price of a lower VRA, specially at lower radius values
(Figures 2e and 2d.. By comparing the upper bounds to the lower bounds obtained in Section 4.1 that our degradation attacks recover between 1/4 and 1/2 of the best possible degradation.
we see
Interestingly the susceptibility to degradation attacks does not appear to result from the GloRo Net's overapproximation of local robustness. Leino et al. (2021) report that the Lipschitz bounds on the MNIST models are far tighter than on the CIFAR-10 model, yet the CIFAR-10 model is not more susceptible to degradation. As explained in Section | this is because the threat of degradation attacks arises intrinsically from any defense that filters based on local robustness at run-time, not because of the overapproximate nature of practical defenses.
Figures 3a and 3b show the false positive rates at different radii for each of the Randomized Smoothed models. Again, the false positive rates are high and approach one as e increases. Models that are smoothed using larger a values unsurprisingly demonstrate more resilience to degradation attacks at the cost of a reduced VRA (Figures 3e and 3d]. The sudden jumps in the false positive rate to one indicate the e value where all points in lestR also end up in Iesla; Lee, the model susceptible to degradation attacks at all the points where it is certified locally robust.
5 Defending against degradation attacks
In this section, we sketch two possible ways of defending against degradation altacks, and empirically evaluate the effectiveness of our first proposal.
5.1 Defense via double radius training
A simple strategy to defend against degradation attacks to first train models using existing adversarial training methods but with radius 2c when the norm-bounded adversary is allowed e-perturbations for adversarial and degradation attacks, and then to compose these models with existing e-certified run-time defenses. This strategy is motivated by the intuition that if model is 2c robust at point x in M, then no matter how the adversary perturbs & within the e-ball centered at x, the model is always (e, Hoolllylly robust at the perturbed input, and an (c, lo)caaall robustness defense is likely (or even guaranteed, in the case of a complete defense) to not reject the input unnecessarily.
Training a model for 2e-robustness is not guaranteed to help because of the frequently observed inverse relationship between accuracy and robustness. The model may be less accurate than the one trained for erobustness, and may also reject more frequently, is certainly helpful. The 2c-VRA on validation data is a lower bound on the accuracy one can expect on test data, in the face an e-adversary capable of adversarial as well as degradation allacks. This also suggests that, when evaluating certified defenses, one ought to report not just e- e-R,,, but also VRA. Validating against 2c can also help in configuring the certified run-time defense. In case the rejection rate too high a particular 2e, the certified defense can be configured enforce local robustness a smaller value. This reduces vulnerability to degradation attacks the cost of increased vulnerability adversarial attacks.
points in M. But validating the model against 2e
10

---

Published in Transactions on Machine Learning Research (11/2022)
We evaluate this defense strategy on GloRo Net models using the same architectures and datasets in Section 4 However, we re-train all the models using twice the values that we want to defend against. Figure 2 summarizes the results. We see that training against 2e-robustness helps reduce the false positive rates compared to the models trained against e-robustnoss. Morcover, this improvement is achieved without significantly affecting the VRA of the models. These results suggest that training against 2e-robust can reduce the vulnerability to degradation attacks, but the false positive rates still remain concerning.
While smoothed models are not explicitly trained for specific degree of robustness, and therefore this approach does not directly apply, the noise parameter, 0, does allow one to control the trade-off between robustness and accuracy. Cohen et al. (2019) found that the best VRA is typically achieved when a R 1/2. Table 2 shows that by increasing a beyond this point, one can obtain better resistance to degradation. However, as Figures 3e and 3d show, increasing a also negatively impacts utility.
5.2 Defense via an M-membership oracle
Certified run-time defenses which rely on local robustness checks are susceptible to degradation attacks because these defenses are overly strong. They try to enforce local robustness, i.c., consistency of predicted label with all e-close inputs, instead consistency just with inputs in M that are e-close. To "fix' these certified defenses, instead of checking local robustness at the given imput z', one could instead check if x' shares its label with all the inputs in M that are e-close x'. If the check passes, then x' is guaranteed to be a non-adversarial input, but, more importantly, if the check fails then x' is guaranteed to be adversarial input.
This fix works because an input x' is an adversarial example if and only if there exists some input x E M that is e-close to x' and is labeled differently from x'. To implement our proposed fix, we would need an explicit representation for M, or at least an oracle for membership in M. A membership oracle may be used to search for members of M that are e-close to a given x'. While an exhaustive search for all such members is likely infeasible, we may allocate a fixed budget of computational resources to the racle-based check. If, within the allocated budget, we find an input then we have detected an adversarial example and can reject x'. However, if the check fails to find such an & within the budget, we can fall back to checking local robustness at x'. Practically, one could use an Out-of- of-ssstributiin (OOD) detector as an J--emmeeshhip oracle. Although OOD detectors can be imprecise and susceptible to adversarial attacks themselves (Sehwag et al. 2019 Bitterwolf et al., 2020 some recent. developments (Chen et al., 2020) look promising. We leave this for future work.
M such that is e-close to x' but it has different label,
6 Related work
The discovery of adversarial examples has led to an intense effort in recent years towards formalizing the problem, well as designing new attack and defense mechanisms.
Formalizations. The presence of adversarial examples has been popularly formalized as the absence of local robustness in a neural classifier Only constant functions are locally robust everywhere, so this definition fails to capture the notion of a classifier free from adversarial examples. Leino et al. (2021) propose the notion of global robustness for classifiers that can reject inputs. They first define an extended notion of local robustness at a point x as the properly that every point in the e-ball at æ either shares its label with x or is rejected. Then, classifier is globally robust if it satisfies this notion of local robustness at every input. A different notion of global robustness is proposed by Ruan et al. (2019). They define model to be globally robust if it is locally robust at every input in a finite test set. These definitions do not capture the idea that, to be protected from adversarial examples, model only needs to be locally robust at every input in the support set, M. For real-world datasets, the support set of the input distribution is much smaller than the entire input space and inputs belonging to different classes are separated by a distance larger than twice the perturbation radii used in adversarial-example experiments (Yang et. al. 2020b).
Adversarial attacks. Algorithms for constructing adversarial attacks can be divided in to two major categories. White-box attacks assume that they have access to the model internals, i.e., the model architecture
11

---

Published in Transactions on Machine Learning Research (11/2022)
and weights. Their primary strategy is to perform gradient descent in the imput space to find an imput that maximizes the loss while satislying the constraint of staying within the e-ball centered at the original input. Some popular and successful attack algorithms of this nature include the ones proposed by Goodfellow ct al. (0155) Carlini & Wagner (2017), and Madry et al. (2018). Black-box attacks only assume query access to the model (Papernol et al. 2017]. In other words, their threat model weaker than the one assumed by white-box attacks. The degradation attacks proposed in this paper use adversarial attack algorithms as a sub-procedure and are agnostic to whether these algorithms are white-box or black-box.
Heuristic and certified defenses. Heuristic defenses against adversarial examples do not provide any guarantees about their defensive capabilities. These include approaches that modify the training objectives, modify the neural classifier post-training, or embed run-time checks to flag adversarial examples during ovaluation. The lack of guarantocs suggests that these defenses can be broken and it has indeed been demonstrated (Athalye et al.) 2018 Tramer et al. 2020) that a number of published defenses are breakable. In this paper, we focus on certified run-time defenses that are deployed during model evaluation and, if an example is adversarial, are guaranteed to flag it (though incomplete defenses can also report false positives). Such defenses check if the classifier is locally robust at the evaluated input. Guaranteed local robustness checks are implemented using variely of approaches that include constraint. solving and formal methods
(Huang et al. 2017 Katz et al.) 2017 Gehr et al. 2018 Singh et al. 2019 optimization (Bastani et al. 2016
Dvijotham et al. 2018 Raghunathan et al.] 2018 Wong & Kolter 2018 Tjeng et al.) 2019), Lipschitz bounds
compulation (Weng el al. 2018] Leino et al. 2021), and stochastin ic smoothing (Lecuyer el. al.) 2019 Cohen
et al. 2019 Yang et al. 2020a). Of these, GloroNets (Leino et al. 2021) and stochastic smoothing approaches |Cohen et al. 2019) flag inputs at run-time; we show that run-time defenses based on local robustness checks
are overly cautious and often flag even non-adversarial inputs.
The algorithm PREDICT from (Cohen et al. 2019) can return ABSTAIN when the prediction can not be computed for the given input, due to the imprecision of the statistical analysis. This can be potentially exploited by an adversary who can force a certifiably robust classifier to unnecessarily abstain as has been noted in (Cohen et al. 2019). Our attack is different as is not due to imprecision of statistical tests; it exploits the fact that certified defenses check local robustness and are therefore too conservative, flagging non-adversarial inputs unnecessarily.
7 Conclusion
In this paper, we have showed that certified run-time defenses against adversarial examples, based on local robustness checks, are inherently overcautions and can reduce model utility due to unnecessary input rejections. This is consequence of these defenses enforcing local robustness, i.e., consistency of the predicted label with all e-close inputs, instead of consistency just with valid e-close inputs (where valid inputs are those in the support set M of the input distribution). An adversary can exploit this overcautiousness: though it camnot change the classification of a valid input, it can apply norm-bounded modifications to valid inputs and force the model to reject otherwise correctly and consistently classifiable inputs. We have presented concrete degradation attacks of this nature that can be implemented using off- the-the-shelf white- box attack algorithms, Our empirical evaluation demonstrates that even state-of-the-art certifiably robust models, like Randomized Smoothed models and GloRo Nets, are highly susceptible to utility degradation attacks.
Though the fact that classifiers can abstain unnecessarily because they ignore M may appear as non- actionable due to the lack of a precise definition of M, model developers can take the following steps to mitigate degradation attacks:
Model developers that use certified run-time defenses based on local robustness should evaluate their models with 2e-robustness even when the adversary is e-bounded, Verified robust accuracy (VRA) numbers calculated with respect to c-robustness are overly optimistic and do not reflect the performance (% of inputs where the model is both accurate and does not reject) one would see for the deployed model.
12

---

Published in Transactions on Machine Learning Research (11/2022)
If it turns out that the VRA is too low when evaluated at 2c, the developer can configure the certified defense to enforce local robustness at e' <<. 6. This reduces the vulnerability to degradation attacks at. the cost of an increased vulnerability to adversarial attacks.
As suggested in Section 4.1, the developers can train the model to be robust with respect to 2e perturbations when faced with an e-bounded adversary in order to defend against degradation attacks.
Acknowledgements
We would like to thank the reviewers for their detailed comments, which helped us improve this article significantly. This material based upon work supported by Soflware Engineering Institute under its FFRDC Contract No. FA8702-15-D-0002 with the U.S. Department of Defense, DARPA GARD Contract HR001 1R000020 Google Faculty Fellowship, and the Alfred P. Sloan Foundation.
References
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pp- 274 283. PMLR, 2018.
Osbert Bastani, Yani Toannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi. Measuring neural net robustness with constraints. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp. 2621-2629, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of out-of-distribution data. Advances in Newral Information Processing Systems, 33, 2020.
N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39 57, Los Alamitos, CA, USA, may 2017. IEFE Computer Society. doi: 10.1109/SP.2017.22 URL https: :/ttps://do..o htw.dddoeee.eccopputorooooeey 11tp://.............././/..////. 1t10//.....00//.P2111
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-distribution detection for neural networks. arXie preprint orXiv:2003.00 09711, 2020.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedinggs of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1310-1320. PMLR, 09-15 Jun 2019. URL https::/prooceddaeee mlr. press/eeer Titess/v97/chhenee html
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. 2009 IEEE conference on computer vision and pullern recogrition, pp. 248 255. leee, 2009.
Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, and Pushmeet Kohli. dual approach to scalable verification of deep networks. In Proceedings of the Thirty-Foue Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-18), pp. 162 171, Corvallis, Oregon, 2018, AUAI Press.
Timon Gehr, Matthew Mirman, Dana Drachsler-Coher Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEFE Symposium Security and Privacy (SP), pp. 3-18, 2018.
Justin Gilmer, Ryan P Adams, Ian Goodfellow, David Andersen, and George Dahl. Motivating the rules of the game for adversarial example research. arxiv preprint arXiv: 807.00633, 2018.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Represenlations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
Srxxlv.org/ab/1442...0
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXit:2109.13916 2021.
Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification deep neural networks. In International Conference on Compuler Aided Verification, pp. 3 29. Springer, 2017.
Guy Katz, Clark Barrett, David I. Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pp. 97 117. Springer, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Corinna Cortes, and Chris Burges. Mnist handwritten digit database, 2010.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 656 672. IEEE, 2019.
Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning (ICML), 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellww, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519, 2017.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. Tn International Conference on Learning Representations, 2018. URL httpps://pparrrrieee.neetftoooo Bya4ob-Rb
Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustness evaluation of deep neural networks with provable guarantees for the hamming distance. In Proceedings of the Frrent-FFighhhe International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5944 5952. International Joint Conferences on Artificial Intelligence Organization, 2019. doi: (0.44966/jjaii20111
URL https://doi....... ottpidooooogg/..oo 24963/4933/33 ijaiiicccaaiiiaa..i 2019/824
Hadi Salman, Greg Yang, Jerry Li, Pengchuan Zhang, Huan Zhang, Ilya Razenshteyn, and Sébastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 11292-11303, 2019.
Vikash Schwag, Arjun Nitin Bhagoji, Liwei Song, Chawin Sitawarin, Daniel Cullina, Mung Chiang, and Prateck Mittal. Analyzing the robustness of open-world machine learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, pp- 105-116, 2019.
Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract domain for certifying neural networks. Proc. ACM Program. Long., 3(POPL) January 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (ods.), 2nd International Conference on Learning Representations, ICLR 2014, Bonff AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http: a/wvvv wttpd/.orrg/ab///112//
14

---

Published in Transactions on Machine Learning Research (11/2022)
Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed
integer programming. In International Conference on Learning Represenlations, 2019. URL https
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33, 2020.
UNECE.
ECE/TRANS/WP.25 /2220/81 United Nations form provisions concerning the approval of vehicles with regard to Automated Lane Keeping Systems, June 2020.
Regulation
on Uni-
URL
un-regulation-auue wetooonnuutmttte-laaaaae -atpttedd-aaae-eeee--------- sysppennggssttsssss-e safe owt.:aaao---acoddddcoooooaaaaaaa
Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning, pp. 5276 5285. PMLR, 2018.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. -295. PMLR, 2018.
Greg Yang, Tony Duan, Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li. Randomized smoothing of all shapes and sizes. In International Conference on Machine Learning, pp. 10693- 10705. PMLR, 2020a.
Yao- Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R. Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. In NewrIPS, 2020b.
15

---

Published in Transactions on Machine Learning Research (11/2022)
A
Monotonicity of local robustness certifiers
Definition A.1 (Monotonicity of local robustncss cortifiers). We say that cortification method cert R x R bool is monotonic if
x' E TR" <> O. cert( A ||x x' = = <E = cert(x',
Definition A.2 (Exact local robustness certifier). An exact certification method is one for which cert(z,c) c) = Va E &R',/l r'Ipp c F(x) = F(x') (in other words, cortification is equivalent to local robustness).
Lemma A.3. All eract certification methods are monotonic.
Proof. This follows essentially from the triangle inequality. Let x E R2 and >> 0, and suppose that cert(a, e). Thus, since cert is exact, we have that Vx' E Rd || - 2'Ilp Se = F(x) F(r)).
Consider E Rd for which || - x|| I c, and let 5 = || - =x'l.. Now for x" E Rd assumc || -x"1 S c-6 8. Thus,
E 2 8 +81 + = || - I'll || - x"1l N ||---x" I"1l
by the triangle inequality.
Thus, because we have cer we may conclude that F(x') = F(x) = F(x"). This tells us that the classifier F (c - 6)-locally robust x', which, by completion gives us cert(z', c- 6).
Definition A.4 (Lipschitz-based based local robustness certifier). A Lipschitz- based certification method is one which, given an upper bound, K,y (z, c) on the local Lipschitz constant of the margin f, f, at certifies a point H, where (xx) j, exactly when
max (f((z) + I fz(2). [Fj
Lemma A.5. All Lipschitz-based certification methods are monotonic.
Proof. Let (2, <) be an upper bound on the Lipschitz constant of the function f5(x) - fi(x). That is, Ki,(z,e) e) is the maximum rate of change of fz( - f(xx) within an lp ball of radius centered at x.
Let x E Rd and e > 0, and suppose that cert(a,e). Consider x' Rd for which || xI| S e, and let
8= || - z'll-
We begin with the observation that the lp ball of radius centered al will denote this as B(a, e) contains B(x', - 6). This, too, follows essentially from the triangle inequality:
2 EE B(x', c- = || 2|1 S c 8= = 1|0 - =x1l
|| x' || + || 56
|| 2|| Se E
by the triangle inequality
Báx, 6)
From this, we conclude that Equation 1 holds.
Kif(I,e) Kiylæ', - 6)
(1)
With this in mind, we proceed as follows. Let j = F(x); iee. j is the class predicted by the classifier at 2. Since cert(x, c) implies that the classifier is r-locally-robus x, and x' B(x, c), we conclude that. F(x')=j j.
'naee that the global Lipechitz constant is such bound
16

---

whereas for Published in Transactions on Machine Learning Research (11/2022)
Algorithm B.1: Degradation attack algorithm
Inputs: A model f E Rd I, an imput z M, an attack bound E R, and distance metric 6p Output: An attack input x' E Rd
DegradattioAAttack(
, Ep):
x" = 2,, r' = Project(2, x
return
Table 3: False positive rates induced by degradation attack in Algorithm B.1
against GloRo Nets.
baseline VRA
false positive rate
uhility reduction
(%)
(%)
(%)
MNIST (E = 0.3)
95.0
6.0
5.7
MNIST (e 1.58)
61.8
56.1
34.7
CIFAR-10 (c 0.141)
60.0
11.4
6.6
Because we assume that cert is Lipschitz-based (Definition A
we have Equation 2
#i j. ff(x) f((x) 2 (x,)
(2)
Since || x'11 e, by the definition of Kú we have that
Vi f+(2) fi(x))
2 fı(2) f((x) (z,)
2 ek (x, (I, (E- ()
by Equation
2
2 (i,', -6
by Equation
This is sufficient to conclude that we have cert(x', e - 8).
GloRo Nets are Lipschitz- based, therefore they are monotonic by Lemma A.5 Aside from the fact that nonde- ariscs because of the need to sample to evaluate the smoothed function, Randomized Smoothing provides the exact robustness radius on the smoothed function. That is, in the infinite sample limit, Randomized Smoothing is an exact method, and therefore monotonic by Lemma A.33 though in practice this property only holds with high probability (i.c., the probability can be bounded from below). We note, however, that the robustness guarantecs provided by Randomized Smoothing are analogously probabilistic.
B Lower bounds on attack efficacy (more results)
B.1 Efficacy of a modified version of Algorithm 3.1
We evaluated the degradation attack efficacy of a modified version of Algorithm 3.1 This attack (Algo- rithm B.1 also invokes an off-the-shelf white-box attack algorithm (PGD), indicated by Attack, but to find an adversarial example x" in ball of radius 2e contered at (line 2). Next, x" is simply projected onto the e-sphere centered (line 3). For l2, Project is defined fxc we simply need to clip x"
minfc, ||>"" po"-aa ==lpp)
Table presents the results of using this modified algorithm for degradation attacks against GloRo Net models. The models, datasets, and e values are the same as used for evaluating the efficacy of Algorithm 3.1 The efficacy of this atkk is similar Algorithm 3.1 (presented in Table 1
17

---

Published in Transactions on Machine Learning Research (11/2022)
Figure 4: Visualizations successful degradation attacks on CIFAR-10 (c = 0.141).
Table 4: Lower bounds for false positive rates induced by degradation attack against KW.
baseline VRA
false positive rate
uhility reduction
(%)
(%)
(%)
MNIST (c 0.1)
95.6
95.2
91.0
CIFAR- 10 (e 2/236)
52.5
16.8
8.8
Table 5: Upper bounds for false positive rates induced by degradation attack against KW.
baseline VRA
false positive rate
utility reduction
(%)
(%)
(%)
MNIST (E 0.1)
95.6
100.0
95.6
CIFA R-10 (c = 2/255)
52.5
64.8
34.0
B.2 Visualization of successful degradation attack inputs
Figure provides samples of inputs that constitute successful degradation attacks. In each pair of images, the original image is shown on the left and the perturbed image (the attack) is shown the right. As expected, the pairs of images are visually indistinguishabl;; thus the adversary can be considered to have successfully delivered an imperceptible attack.
C Degradation attacks against lo defenses
In this section, we evaluate the efficacy of degradation attacks against models defended with a too certified defense. In particular, we consider models trained and defended using the convex outer adversarial polytope approach of Wong & Kolter (2018), that we refer as KW. We compute lower and upper bounds in a similar manner described in Section using KW for the (e, (lo)al robustness check. For the lower bounds, the set leSLA is constructed using Algorithm 3.1 where PGD is used for Attack. We use the pre-trained models made available by the authors of KW The values are the same used by Wong & Kolter (018
Table reports the lower bounds, i.e., the efficacy of Algorithm 3.1 against KW. Table 5 reports the upper bounds. We see that loo defenses like KW can be extremely susceptible to degradation attacks on MNIST, the simple attack given by Algorithm succeeds approximately 95% of the time, and the upper bounds indicate that it may possible for a more sophisticated adversary succeed 100% of the time. On CIFAR-10, the attack success rates are more modest,, partially because of the small value of
While striking, particularly for MNIST, these results are not entirely unexpected, given the geometry of tx space. Defenses based on robustness certification are susceptible to degradation attacks unless the underlying model is (2e, -robust. In Eo space, the volume contained by a ball overlaps much more closely with [o, 1]" (the typical the domain for, c.g., image data), which intuitively mcans that the domain is "used up" much more quickly as the required robustness radius increases. By contrast, in high-dimensional Euclidean space, the radius can be as large vd before it necessarily envelops the entire domain.
18

---

Published in Transactions on Machine Learning Research (11/2022)
D Statement of broader impact
Our work sheds light on existing vulnerabilities in statc-of-the-art ocrtifiably robust neural classifiers. These degradation attacks could be deployed by malicious entities to degrade the utility of deployed models. However, by putting this knowledge out in the public domain and making practitioners aware of the existence of degradation attacks, we hope that precautions can be taken to protect existing systems. Moreover, it highlights the need to harden future systems against such attacks.
19


Reviews:
[{"review_id": "lrGjwRDxaL", "reviewer": "Reviewer_SUEh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe problem is certainly interesting. A main failing of certified defenses are that they are too conservative. I think that fully characterizing what type of inputs certified defenses are too cautious towards is a worthwhile endeavor.\n\n## Weaknesses\nI don't think this paper is well-posed. Most of my concerns have to do with the set $M$, which is central to both their overall claim (that certified defenses are overly cautious) and their degradation attack.\n\n**Defining M**: $M$ is not well-defined. What does it mean to be a valid input? Do adversarial examples count as valid inputs? Depending on how you define $M$, these certified defenses are either overly cautious or appropriately rejecting risky examples. \n\n**Assumptions of the adversary**: Even if we could define $M$, it is unclear to me why the adversary would know $M$ but the certified defense would not. These assumptions are not justified in the paper. \n\n**Assumption in Section 3.3** Why can't $x'$ be an adversarial input? I realize that most image datasets are well-separated, but (again) this statement makes a strong assumption about what $M$ is. Indeed, it seems like you are making the assumption that anything within $2\\epsilon$ of a data point that has a different class label is not a valid input. If you make this assumption, robustness is already solved!\n\n**The attack**: At the end of the day, the degradation attack seems like it is just performing an adversarial attack that is $2\\epsilon$ away instead of $\\epsilon$ away, when up against a defense with $\\epsilon$ robustness. That this works is not that interesting. Also, what are you using as $M$ in your empirical experiments?\n\n**Significance** The fact that certified defenses are overly cautious is already pretty well known (indeed, certified accuracy is almost always much lower than clean accuracy for almost every defense). It would have been interesting to characterize this more fully, but I don't think this paper puts forward an actionable framework. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors find that certified defenses against adversarial attacks may be overly cautious in that they reject examples which are not adversarial. In particular, they define a set $M$ (of valid inputs) and show that certified defenses might reject an example which is not locally robust but is consistent with all $\\epsilon$-close inputs in $M$. They then develop a \"degradation\" attack which causes certified defenses to reject examples that could have been correctly classified. They apply their experiments on the MNIST, CIFAR-10, and ImageNet datasets.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I think to improve this paper, you need a thorough section explaining \n1. What is $M$, and how does the definition of $M$ relate to current notions of adversarial robustness?\n2. What does the adversary know about $M$?\n3. Why is your attack different than just finding an adversarial example that is $2\\epsilon$ away?\n4. How is $M$ defined for your experiments?\n\nIn terms of writing, the paper is a bit repetitive. Section 2, Section 3.1, and Section 3.3 all explain the degradation attack. I would condense these sections, and explain the attack clearly (once).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YimSvnfSPQ", "reviewer": "Reviewer_Br6u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. This paper points out a very interesting problem of certified defense especially with GloRo nets and Randomized smoothing when applied in run-time applications. The empirical results well support the claim. This encourages a more general (certifiable) robustness definition and more study on global robustness.\n2. Solid theoretical attack efficacy upper and lower bounds are analyzed.\n3. The paper is well presented and easy to follow.\n4. Two potential defenses are provided and discussed.\n\nWeakness\n1. The proposed methods themselves have very limited novelty though the angle for applying these attacks is new.\n2. In practice, verification is a plus for the model robustness evaluation. It is safe to trust the results if a sample is verified but a sample that cannot be verified does not necessarily mean we have to reject it. There are many other ways to measure the robustness and give confidence scores of the predictions. Also, over-cautiousness is the intrinsic property of certifiable defenses and should be the correct thing to have. Therefore, the degradation attack is useful but not such a severe problem as described in the paper.\n3. Limited experimental evaluation (see requested changes below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the over-cautiousness of existing certifiable defenses and proposes corresponding degradation attacks to cause frequent rejections in run time. Empirical experiments show good results of the proposed attacks on two types of certified defenses GloRo nets and randomized smoothing.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Changes:\n1. The proposed SPGD is similar to [A]. A discussion could be helpful.\n2. To demonstrate the usefulness and generalization of the proposed attack, the proposed attack should be discussed and evaluated with other SOTA complete verifiers like alpha-beta-CROWN [B, C] (the winner of the latest neural network verification competition) or incomplete ones like K&W [D], and certifiable training methods like CROWN-IBP [E].\n\n[A] Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers, NeurIPS 2019\n\n[B] Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification, NeurIPS 2021\n\n[C] Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers, ICLR 2021\n\n[D] Provable defenses against adversarial examples via the convex outer adversarial polytope, ICML 2018\n\n[E] Towards Stable and Efficient Training of Verifiably Robust Neural Networks, ICLR 2020", "broader_impact_concerns": "Since the paper proposes new attacks aiming to break existing certifiable defenses, it would be beneficial to add a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3jSVX9GRsl", "reviewer": "Reviewer_8nCi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors propose an attack that is, to the best of my knowledge, novel. The paper is well-written and the methodology sound.\n\nAt the same time, the threat model considered by the paper is not well motivated. \n\nOn one hand, the argument that classifiers abstain unnecessarily because they ignore the “data manifold” is non-actionable. For a certifiable method to take advantage of the domain of valid images, one would need access to a mathematically precise definition of what constitutes a valid image. Note that even methods that aim to approximate the “data manifold” cannot be used to actually certify that certain inputs are on the manifold in any meaningful sense.\n\nOn the other hand, the significance of the proposed degradation attacks is questionable. After all, the certifiable robust models are working exactly as intended: they reject inputs on which the base model’s prediction is not robust. From that perspective, I find a significant part of the paper narrative misleading. For instance, terminology such as “unnecessary rejections” is misleading: if these inputs are not robust then why should they be accepted by a robustness certification method?\n\nMoreover, from a deployment point of view, it is reasonable to expect the overall system to behave as follows: the system is predicting accurately on natural inputs and abstains on ones that have been adversarially perturbed. The proposed degradation attacks do not change this behavior at all as the model still rejects corrupted inputs. At an even higher level, it is seems hard to imagine settings where the model designer is ok abstaining on certain suspicious inputs and at the same time the adversary can cause a significant utility degradation by causing the model to abstain. It would be useful for the discussion if the authors could provide examples of real-world threats that are at least somewhat plausible.\n\nOverall, I do not find the arguments presented by the paper particularly convincing and thus I don’t believe that it moves the discussion around adversarial robustness forward.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies certifiably robust classifiers---i.e., classifiers that for a given inference-time input, attempt to verify if all small perturbations of that input are classified consistently and potentially abstain if not. \n\nThe authors argue that such classifiers suffer from a drawback: if a small perturbation of the given input is classified differently, but that perturbation is not part of the space of valid inputs (i.e., “the data manifold”), then the classifier might abstain unnecessarily. The authors argue that this reduces the utility of the classifier as it requires a human operator to intervene.\n\nIn order to support this argument, the authors consider existing certifiably robust classifiers and develop “degradation attacks”: perturbations that cause the classifier to abstain (since the classifier is not robust around the perturbed input), even though the classifier is correct and robust on the original input. In terms of methodology, these attacks rely on existing methods for crafting adversarial examples (e.g., PGD). They show that for standard datasets an adversary can cause the classifier to abstain on a significant fraction (e.g., ~30%) of test inputs that are normally correctly classified and certifiably robust.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "My concerns are not specific to any particular part of the paper but rather to the broader motivation and approach.", "broader_impact_concerns": "The paper discusses methods for degrading the performance of ML systems. Thus there is the potential that such methods could be used to cause harm. At the same time, in its current form, the attack proposed does not seem to pose a perceivable threat.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I would like to thank the authors for their submission to TMLR. I hope the following feedback, together with the reviews and discussion, will provide some insights in the outcome and a way forward for the paper.

This manuscript builds on the notion of valid inputs [Szegedy et al., 2014; Carlini & Wagner, 2017] to define the set of valid inputs $M$. It then also makes the assumption that any two inputs in $M$ with a different ground-truth label are separated by at least $2\varepsilon$. The paper observes that when a model exhibits $\varepsilon$ robustness, it is susceptible to a degradation (i.e., availability) attack if the model is not $2\varepsilon$ robust. Put altogether, this implies that “even exact (i.e., sound and complete) local robustness certifiers can be overly cautious if deployed at run-time since the model predictions only need to be consistent with valid (i.e., in-distribution) inputs that are $\varepsilon$-close and not with all $\varepsilon$-close inputs” as summarized by the authors.

## Criteria 

I assessed the submission and reviews based on the two criteria listed by TMLR.

> Are the claims made in the submission supported by accurate, convincing and clear evidence?

Reviewers did not find any issues with the accurateness of the submission's claims. That said, all reviewers ended up disagreeing with the authors about the significance of their claims and the practicality of the threat model considered in the manuscript. I would therefore request that the authors clearly state the limitations of their work in the abstract, introduction, section 4.1, and conclusion. This would include: 

* A detailed discussion and acknowledgment describing how the lack of knowledge of $M$ makes the findings of this manuscript less actionable. 
  * For instance, the authors write “the developers can train the model to be robust wrt $2\varepsilon$ perturbations when faced with an $\varepsilon$-bounded adversary in order to defend against degradation attacks”: what would be the impact on accuracy? 
  * Alternatively, how much lower the value of $\varepsilon’$ used for certification would have to be in order to avoid degradation attacks? 
  * There are numerous excellent points that were made during the discussion between authors and reviewers, I would ask that the authors carefully go through these points and integrate them in the revised manuscript.

* A clearly marked paragraph/subsection containing a threat model. I would pay particular attention to the discussion with Reviewer SUEh when writing the threat model.

* A discussion of the dichotomy between “defending against adversarial examples” and “certifying local robustness”. Please pay particular attention to reflecting the perspectives brought up by Reviewer 8nCi. I believe that one way forward is to state that the conclusions of the present manuscript point to the incompatibility between $p$-norm based definitions of adversarial robustness and the semantics of tasks which ML is applied to. The authors may find it useful to discuss prior work that identifies this incompatibility.

It would also be beneficial to add the experiment on alpha-beta-crown (https://arxiv.org/abs/2103.06624) requested by Reviewer Br6u.

> Would at least some individuals in TMLR's audience be interested in the findings of this paper?

All 3 reviewers agreed that the paper reports findings that would be interesting to the TMLR audience - despite the current lack of practical solutions to the problem identified by the authors.

## Overall decision

I would like to thank the authors for engaging with reviewers in a discussion following the reviews being posted. I believe that if this discussion is reflected honestly in the manuscript, the resulting paper would be of value to our community. Thus, I encourage the authors to further discuss with reviewers on OpenReview should they have any questions as they revise the manuscript.
Decision: Accept (poster)