SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Adversarily Robust Learning with Uncertain Perturbation Sets
Tosca Lechner
University of Waterloo, Waterloo, Ontario, Canada, tlechner@uwaterloo.ca
Vinayak Pathak
Layer 6 AI, Toronto, Ontario, Canada, vinayak@layer6.ai
Ruth Urner
York University, Toronto, Ontario, Canada, uruth@eecs.yorku.ca
In many real-world settings exact perturbation sets to be used by an adversary are not plausibly available to a learner. While prior literature has studied both scenarios with completely known and completely unknown perturbation sets, we propose an in-between setting of learning with respect to a class of perturbation sets. We show that in this setting we can improve on previous results with completely unknown perturbation sets, while still addressing the concerns of not having perfect knowledge of these sets in real life. In particular, we give the first positive results for the learnability of infinite Littlestone classes when having access to a perfect-attack oracle. We also consider a setting of learning with abstention, where predictions are considered robustness violations, only when the wrong label prediction is made within the perturbation set. We show there are classes for which perturbation-set unaware learning without query access is possible, but abstention is required.
Adversarial perturbations, imperceivably small manipulations to input instances that cause unexpected failures of learned predictors, pose significant safety concerns for their employment. The phenomenon, first exposed a decade ago in image classification tasks [Szegedy et al., 2014], has since received substantial attention, both in the practical and theoretical machine learning research communities. Studies in both lines of research often skip over the question of how to suitably model "imperceivable perturbations" and rather investigate how to defend against various (fixed) types of attacks. In theoretical studies, this is typically modeled by defining an adversarial loss that, in addition to misclassification, penalizes the existence of a perturbed instance on which a classifier assigns a different label. This type of loss definition then crucially depends on a given perturbation type, that is some function  $u: \mathcal{X} \to 2^{\mathcal{X}}$  that assigns every domain instance a set of points that would be viewed as an admissible perturbation (often balls of some radius with respect to some norm are considered).
However, in most realistic settings, exact perturbation sets to be used by an adversary are not plausibly available to a learner. Moreover, it has been shown that encouraging robustness with respect to a perturbation type that is not suitable for the task easily degrades classification accuracy [Tsipras et al., 2019, Zhang et al., 2019, Yang et al., 2020]. In response, some work has then explored the question of learnability when the perturbation type is not known at all in advance [Montasser et al., 2021], but information about possible perturbations can be accessed through queries. It is not difficult to be convinced though that without any restrictions or knowledge of the perturbations to be used, robust
learning from samples only is impossible. Even for a small collection of possible perturbation types, as soon as these induce inconsistencies between optimal predictors a sample based learner cannot be simultaneously successful with respect to all options.
In this work we explore a middle ground between fixed and unknown perturbation types, which we term learning with uncertain perturbation sets. We will assume that the true perturbation type is a member of a fixed class of perturbation types. One can think of this as knowing that we should encourage robustness with respect to some type of norm induced ball, but may not be certain about which type of norm or which radius is most suitable for the task. We then study which structures (on the interplay between perturbation type class and the class of predictors to be learned) allow for PAC-type learnability guarantees.
Given a class  $\mathcal{U}$  of perturbation types and hypothesis class  $\mathcal{H}$ , we define an  $\mathcal{H}$ -induced partial order on  $\mathcal{U}$ . We show that when  $\mathcal{U}$  is actually totally ordered in this sense, statistical (PAC) learning is possible in the realizable case as soon as  $\mathcal{H}$  has finite VC-dimension.  $\mathcal{U}$  being totally ordered applies to a wide variety of settings, for example the case where the various perturbation types are balls in some metric space while the radius varies over  $\mathcal{U}$  (and actually also does not need to be constant over domain  $\mathcal{X}$  for each  $u\in \mathcal{U}$ ). However, we also show that, without the realizability assumption learning from samples alone is not feasible, even when  $\mathcal{U}$  is totally ordered with respect to  $\mathcal{H}$ .
We thus explore two natural remedies: (1) We allow the learner to interact with a perfect attack oracle [Montasser et al., 2021]. Given an instance  $x$  and classifier  $f$ , such an oracle certifies robustness or provides an adversarial perturbation of  $x$  for  $f$ . Previous work on learning with unknown perturbation sets has employed such an oracle and shown learnability for classes with finite Littlestone dimension [Montasser et al., 2021]. In this work, we show that provided certain structures ( $\mathcal{U}$  being totally ordered or a finite union of totally ordered perturbation type classes) learning with a perfect attack oracle becomes feasible for all classes of finite VC-dimension (which is a much wider space of natural hypothesis classes than those of finite Littlestone dimension). (2) We allow the learner to output a classifier that sometimes abstains from prediction. For this, we define a modification of the adversarial loss, where a hypothesis does not suffer loss when it abstains on an instance that was actually manipulated. We show that this again yields learnability for VC-classes when the perturbation type class is a finite union of totally ordered classes. We then consider the case in which  $\mathcal{H}$  has finite disagreement-coefficient [Hanneke, 2007] and show that such  $\mathcal{H}$  can be learned with respect to every class of perturbation types. And finally, we define a notion of  $(\epsilon, \mathcal{H})$ -cover for a class of perturbation types  $\mathcal{U}$ , introduce learners for this kind of cover and show that a class  $\mathcal{H}$  is robustly learnable with respect to  $\mathcal{U}$  if there is a finite-disagreement-cover learner for  $(\mathcal{H}, \mathcal{U})$ .
Our results on learning with abstentions show that different strategies for guarantees for different kinds of robustness (for example different types of perturbation sets) can be combined into an abstention learner that only predicts in the agreement region. The guarantees for each learning strategy then yield a guarantee for the combined classifier that depends on the number of strategies used. This highlights a different aspect to the power of abstention in adversarially robust learning from what has been established in prior work [Balcan et al., 2020, 2023]. We show that abstentions can also be used to address uncertainty in perturbation types.
Providing PAC type learning guarantees under adversarial robustness requirements has received an enormous amount of research attention in the past decade [Feige et al., 2015, Cullina et al., 2018, Wang et al., 2018, Awasthi et al., 2019, Montasser et al., 2019, Attias et al., 2019, Montasser et al., 2020, Ashtiani et al., 2020, Montasser et al., 2021, Gourdeau et al., 2021, 2022, Montasser et al., 2022, Attias and Hanneke, 2022, Attias et al., 2022, Bhattacharjee et al., 2021, Awasthi et al., 2022b,a, 2023, Mao et al., 2023]. We will focus here on aspects within this literature that are most relevant to our settings.
Most prior works study the sample complexity of adversarial robustness with respect to a fixed type of adversarial perturbation (often a metric ball with a certain radius). However, recent research has developed frameworks of analysis that go beyond learning with respect to a fixed type of perturbations. It has been argued that the "correct" type of admissible adversarial perturbations (which originally were "imperceivable" modifications to input instances that lead to misclassification by a learned predictor) should depend on the underlying data-generating process [Bhattacharjee and Chaudhuri,
2021, Chowdhury and Urner, 2022]. These studies define a notion of "adaptive robustness" with respect to which a predictor should be evaluated. The drawback of these notions is that the correct perturbation type is not available to the learner and thus a predictor cannot be straightforwardly evaluated with respect to this loss. A different way of relaxing the assumption of a fixed, known perturbation type is provided in the framework of tolerance [Ashtiani et al., 2023, Bhattacharjee et al., 2023]. Here, a learner is evaluated with respect to some fixed perturbation type, while being compared to the optimal achievable loss with respect to a larger perturbation type, relaxing the requirement for the learner and reflecting that the exact relevant perturbation type is typically not known (or even uniquely existing).
Most relevant to our work is a recent study on PAC learning with respect to unknown perturbation sets [Montasser et al., 2021]. This work provides learning guarantees for the case that the robustness requirement will be with respect to an entirely unknown perturbation type without any prior knowledge about the nature of these perturbations. Those learning bounds assume access to a perfect attack oracle (as we do in some parts of this work), and require the class to have finite Littlestone dimension. In this work, we show that adding some structure to the class of perturbation types (as well as the promise that the true perturbation type will be a member of this class), allows for sample and query complexity bounds that are independent of the hypothesis class's Littlestone dimension.
Finally, there is a long line of work studying the benefits of abstentions for classification tasks [Bartlett and Wegkamp, 2008, El-Yaniv and Wiener, 2010, Wiener and El-Yaniv, 2015, Yuan and Wegkamp, 2010, Zhang and Chaudhuri, 2016]. Some recent studies have also shown that the ability to abstain from prediction for manipulated instances can be beneficial for adversarial robustness requirements [Balcan et al., 2020, 2023]. However, the former studies a different setting for the generation of adversarial perturbations, and the latter considers fixed perturbation types.
1.2 Overview of results
In Section 2, we define our setup, introduce the notions of learnability and loss we investigate and provide an overview of the different kinds of prior knowledge and oracle access we employ. In particular, we there define our partial order on classes of perturbation types.
In Section 3 we analyze perturbation type classes for which this order is a total order. Theorem 1 states that every hypothesis class with finite VC dimension can be robustly learned in the realizable case with respect to any totally ordered perturbation type class. We then prove that this result cannot be extended to the agnostic case (Observation 1). This motivates investigating the benefits of a perfect attack oracle. Theorem 2 shows that the impossibility can be overcome with access to a perfect attack oracle and provides a finite sample and oracle-query bound for the agnostic case.
In Section 4, we investigate classes of perturbation types  $\mathcal{U}$  that are not necessarily totally ordered. We first extend our previous results on learning with access to a perfect-attack-oracle to the case where  $\mathcal{U}$  is a union of finitely many totally ordered perturbation classes (Theorem 3). We then establish that there are classes of perturbation types and hypothesis classes which cannot be learned without abstention, even with access to a perfect attack oracle (Observation 3), but which can be learned with abstention without the need for access to a perfect-attack-oracle (Observation 4). This motivates our investigations into learning with respect to our adversarial abstention loss (Definition 5).
We show that in the realizable case, every hypothesis class with finite VC-dimension can be learned with respect to a finite union of totally ordered perturbation types in Theorem 4. We then consider the case in which  $\mathcal{H}$  has finite disagreement-coefficient and show that such  $\mathcal{H}$  can be learned with respect to every class of perturbation types (Theorem 5). Lastly, we define a notion of  $(\epsilon, \mathcal{H})$ -cover for a class of perturbation types  $\mathcal{U}$ , which states that for every  $u \in U$  there is a close-to-optimal hypothesis for  $u$  in the cover (Definition 7) and introduce the notion of learners for this kind of cover (Definition 8). We then generalize our previous results by showing that a class  $\mathcal{H}$  is robustly learnable with respect to  $\mathcal{U}$  if there is a finite-disagreement-cover learner for  $(\mathcal{H}, \mathcal{U})$  (Theorem 6). While we provide some proof sketches in the main body of the paper, the detailed proofs for all results are in the supplement.
2 Setup
For sets  $A$  and  $B$ , we let  $2^A$  denote the powerset of  $A$  and  $A^B$  the set of all functions from  $B$  to  $A$ . Let  $\mathcal{X}$  be a domain and  $\mathcal{Y} = \{0,1\}$  be a label space. We model the data generation as a distribution
$P$  over  $\mathcal{X} \times \mathcal{Y}$ , and let  $P_{\mathcal{X}}$  denote its marginal over  $\mathcal{X}$ . A hypothesis or predictor is a function  $h: \mathcal{X} \to \{0,1,\star\}$ . Thus, on input  $x \in \mathcal{X}$ , a hypothesis  $h$  either predicts label 0 or 1 or abstains by outputting  $\star$ . We call a hypothesis non-abstaining if  $h(x) \neq \star$  for all  $x \in \mathcal{X}$ . A hypothesis class  $\mathcal{H}$  is a set of non-abstaining predictors. We will let  $\mathcal{F} = \{0,1,\star\}^{\mathcal{X}}$  denote the set of all predictors over  $\mathcal{X}$ .
The performance of a predictor  $h$  is measured by a loss function  $\ell: \mathcal{F} \times \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ . For loss  $\ell$ , we use the notation  $\mathcal{L}_P(h) = \mathbb{E}_{(x,y) \sim P}[\ell(h,x,y)]$ , to denote the expected loss of predictor  $h$  with respect to distribution  $P$  and  $\mathcal{L}_S(h) = \frac{1}{|S|} \sum_{i=1}^{|S|} \ell(h,x_i,y_i)$  the empirical loss on dataset  $S = ((x_1,y_1),\ldots,(x_m,y_m))$ . Further, for a hypothesis class  $\mathcal{H}$  we will let  $\mathrm{opt}_P(\mathcal{H}) = \inf_{h \in \mathcal{H}} \mathcal{L}_P(h)$  denote the approximation error of class  $\mathcal{H}$  with respect to  $P$  and loss  $\ell$ .
The standard loss for classification tasks is the binary loss  $\ell^{0/1}(h, x, y) = \mathbb{1}[h(x) \neq y]$ , where  $\mathbb{1}[\cdot]$  denotes the indicator function. The standard loss for adversarial robustness additionally penalizes  $h$  on instance  $(x, y)$  for the existence of a perturbation on  $x$  that  $h$  does not label with  $y$ . This loss is defined for a specified perturbation type, which is a function  $u: \mathcal{X} \to 2^{\mathcal{X}}$  that assigns every instance  $x \in \mathcal{X}$  a set  $u(x) \subseteq \mathcal{X}$  with  $x \in u(x)$  of admissible perturbations. The adversarial loss with respect to  $u$  is then defined as
$$
\ell^ {u} (h, x, y) = \mathbb {1} [ \exists z \in u (x): h (z) \neq y ].
$$
We will use the notation  $\ell^{u,\perp}(h,x)$  to denote the adversarial component loss, which indicates whether a domain point  $x$  lies close to the decision boundary with respect to the perturbation type  $u$ :
$$
\ell^ {u, \perp} (h, x) = \mathbb {1} [ \exists z \in u (x): h (z) \neq h (x) ].
$$
For hypotheses that have the option to abstain from prediction, we propose a variation of the adversarial loss that allows for a predictor to abstain from prediction on perturbed instances (but not on unperturbed instances) without suffering loss. The following definition captures this modification:
Definition 1 (Adversarial abstention loss). For a perturbation type  $u: \mathcal{X} \to 2^{\mathcal{X}}$ , the adversarial abstention loss is defined by
$$
\ell^ {u, \mathrm {a b s t}} (h, x, y) = \left\{ \begin{array}{l} 1 i f h (x) \neq y \\ 1 i f \exists z \in u (x) s u c h t h a t h (z) \neq y a n d h (z) \neq \star \\ 0 o t h e r w i s e \end{array} \right.
$$
The main difference with adversarial loss is that if  $h(z) = \star$  for an adversarial point  $z \in u(x)$  with  $z \neq x$ , then there is no penalty and  $\ell^{u,\mathrm{abs}}(h,x,y) = 0$ , thus allowing the predictor to abstain on points that were perturbed. This definition models the scenario where if the predictor can correctly detect that an adversarial attack has happened and abstains, then it is not penalized. However, abstaining on non-perturbed points is still penalized. Note that in case a predictor  $h$  never abstains (that is  $h(x) \neq \star$  for all  $x \in \mathcal{X}$ ) the two definitions of adversarial loss coincide. We let  $\mathcal{L}_P^u(h)$ ,  $\mathcal{L}_P^{u,\mathrm{abs}}(h)$ ,  $\mathcal{L}_S^u(h)$ ,  $\mathcal{L}_S^{u,\mathrm{abs}}(h)$  denote the corresponding expected and empirical losses, and  $\operatorname{opt}_P^u(\mathcal{H})$ ,  $\operatorname{opt}_P^{u,\mathrm{abs}}(\mathcal{H})$  the corresponding approximation errors of a class  $\mathcal{H}$  and distribution  $P$ .
Uncertain perturbation type In the literature, adversarial robustness is mostly treated with respect to a fixed (known) perturbation type  $u$ , while learning with respect to an entirely unknown perturbation type has also been investigated. In this work, we introduce a setting that naturally interpolates between these two extremes. We assume that the true perturbation type  $u^{*}$  is unknown to the learner, but promised to lie within a fixed class  $\mathcal{U}$  of perturbation types. That is, the learner has prior knowledge of a class  $\mathcal{U}$  with  $u^{*} \in \mathcal{U}$ , where  $u^{*}$  is the perturbation type that the learner will be evaluated with (see Definition 3 below). We can think of  $u^{*}$  as the perturbation type that the true adversarial environment will employ to manipulate input instances and induce misclassifications. We let  $\mathcal{U}^{\mathrm{all}}$  denote the class that contains all possible perturbation types.
It is easy to see that without any restrictions on the class  $\mathcal{U}$  and any capability of the learner to (interactively) gather information about  $u^{*}$  and no option to abstain, a learner cannot succeed (see also Observations 1 and 2 below). To gain control over potentially infinite classes of perturbation types, we will now define a partial order on a class  $\mathcal{U}$  that is induced by a hypothesis class  $\mathcal{H}$ . This structure will prove to be conducive for learnability.
Definition 2. Let  $\mathcal{H} \subseteq \{0,1\}^{\mathcal{X}}$  be a hypothesis class. For perturbation types  $u_1, u_2$  we say that  $u_1$  is smaller than  $u_2$  with respect to class  $\mathcal{H}$ , and write  $u_1 \preceq_H u_2$ , if and only if for all  $h \in \mathcal{H}$  and all  $x \in \mathcal{X}$  we have
$$
\ell^ {u _ {1}, \perp} (h, x) \leq \ell^ {u _ {2}, \perp} (h, x).
$$
We say a set of perturbation types  $\mathcal{U}$  is totally ordered, with respect to  $\preceq_{\mathcal{H}}$ , if for every  $u_1, u_2 \in \mathcal{U}$ , we have either  $u_1 \preceq_{\mathcal{H}} u_2$  or  $u_2 \preceq_{\mathcal{H}} u_1$ .
We have  $u_{1} \preceq_{H} u_{2}$  if, for every function  $h \in \mathcal{H}$  the margin area (the points which receive positive adversarial component loss) with respect to  $u_{1}$  is included in the margin area with respect to  $u_{2}$ . This is, for example, the case if all perturbation sets with respect to  $u_{1}$  are included in those induced by  $u_{2}$ .
Resources to learners and learnability A learner is a function  $A:(\mathcal{X}\times \mathcal{Y})^{*}\to \mathcal{F}$  that takes in a finite sequence  $S = ((x_{1},y_{1}),\ldots (x_{m},y_{m}))$  of labeled examples and outputs a predictor  $A(S):\mathcal{X}\rightarrow \{0,1,\star \}$ . We will study the following PAC-type [Valiant, 1984] notion of learnability of hypothesis classes  $\mathcal{H}$  with perturbation type classes  $\mathcal{U}$ .
Definition 3 (Learnability without abstentions). Let  $\mathcal{H} \subseteq \{0,1\}^{\mathcal{X}}$  be a hypothesis class and  $\mathcal{U}$  a class of perturbation types. We say that  $\mathcal{H}$  is robustly learnable with respect to perturbation class  $\mathcal{U}$  if there exists a learner  $A$  and a function  $m:(0,1)^2 \to \mathbb{N}$  such that for every  $\epsilon, \delta > 0$ , every  $m \geq m(\epsilon, \delta)$ , every  $u^* \in \mathcal{U}$  and every data-generating distribution  $P$  we have
$$
\mathbb {P} _ {S \sim P ^ {m}} \left[ \mathcal {L} _ {P} ^ {u ^ {*}} (A (S)) \leq \operatorname {o p t} _ {P} ^ {u ^ {*}} (\mathcal {H}) + \epsilon \right] \geq 1 - \delta .
$$
We speak of learnability in the realizable case if the above requirement is relaxed to only hold for all distributions  $P$  with  $\mathrm{opt}_P^{u^*}(\mathcal{H}) = 0$ . Without this assumption, we also speak of learnability in the agnostic case.
If the above definition is fulfilled for a class  $\mathcal{H}$ , then the function  $m:(0,1)^2\to \mathbb{N}$  in the above definition is an upper bound on the sample complexity of the learning task. Note that, while we don't require the learner in the above definition to output a non-abstaining hypothesis, predicting  $\star$  will always cause loss with respect to  $\ell^u$ . We can thus assume all considered predictors are non-abstaining classifiers for this setting.
Note that the above definition generalizes and unifies previous notions of adversarially robust learnability. We obtain learnability with respect to a known perturbation type (the setting that is mostly considered in prior works) when  $\mathcal{U} = \{u^{*}\}$  is a singleton class. Learning with unknown perturbations [Montasser et al., 2021] on the other hand, is the setting where  $\mathcal{U}$  is the set of all possible perturbation types. In this work, we are interested in studying more fine-grained conditions on the structure of (the combination of)  $\mathcal{H}$  and  $\mathcal{U}$  that allow for learnability. We will call the case where  $\mathcal{U}$  is neither a singleton set nor the set of all possible perturbation types learning with uncertain perturbation sets.
We will show that, even when the class of perturbation types  $\mathcal{U}$  in consideration is totally ordered with respect to hypothesis class  $\mathcal{H}$ , agnostically learning  $\mathcal{H}$  with  $\mathcal{U}$  is impossible even in very simple cases (see Observation 1). We will thus, as has been done in earlier studies [Ashtiani et al., 2020, Montasser et al., 2021] in addition allow the learner access to a perfect attack oracle, which we model as follows:
Definition 4 (Perfect Attack Oracle). A perfect attack oracle for perturbation type  $u$  is a function  $\mathcal{O}_u: \mathcal{F} \times \mathcal{X} \to \mathcal{X} \cup \{\text{robust}\}$ , that takes as input a non-abstaining predictor  $f$  and a domain point  $x$  and either certifies that  $f$  is robust on  $x$  with respect to  $u$  by outputting robust or returns an admissible perturbation of  $x$  for  $f$ . That is
$$
\mathcal {O} _ {u} (f, x) = \left\{ \begin{array}{l} \text {r o b u s t i f} f (z) = f (x) \text {f o r a l l} z \in u (x) \\ z \in u (x) \text {w i t h} f (x) \neq f (z) \text {o t h e r w i s e} \end{array} \right.
$$
We say that a class  $\mathcal{H}$  and perturbation class  $\mathcal{U}$  are learnable with access to a perfect attack oracle if there exists a function  $n:(0,1)^2\to \mathbb{N}$  such that the conditions in Definition 3 are satisfied for a learner that additionally makes at most  $n(\epsilon ,\delta)$  queries to the perfect attack oracle. The function  $n(\cdot ,\cdot)$  then is an upper bound on the attack oracle complexity of the learning problem.