SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Surbhi Goel
University of Pennsylvania surbhig@cis.upenn.edu
Steve Hanneke
Purdue University steve.hanneke@gmail.com
Shay Moran
Technion and Google Research, Israel smoran@technion.ac.il
Abhishek Shetty
University of California, Berkeley shetty@berkeley.edu
We study the problem of sequential prediction in the stochastic setting with an adversary that is allowed to inject clean-label adversarial (or out-of-distribution) examples. Algorithms designed to handle purely stochastic data tend to fail in the presence of such adversarial examples, often leading to erroneous predictions. This is undesirable in many high-stakes applications such as medical recommendations, where abstaining from predictions on adversarial examples is preferable to misclassification. On the other hand, assuming fully adversarial data leads to very pessimistic bounds that are often vacuous in practice.
To move away from these pessimistic guarantees, we propose a new model of sequential prediction that sits between the purely stochastic and fully adversarial settings by allowing the learner to abstain from making a prediction at no cost on adversarial examples, thereby asking the learner to make predictions with certainty. Assuming access to the marginal distribution on the non-adversarial examples, we design a learner whose error scales with the VC dimension (mirroring the stochastic setting) of the hypothesis class, as opposed to the Littlestone dimension which characterizes the fully adversarial setting. Furthermore, we design learners for VC dimension 1 classes and the class of axis-aligned rectangles, which work even in the absence of access to the marginal distribution. Our key technical contribution is a novel measure for quantifying uncertainty for learning VC classes, which may be of independent interest.
Consider the problem of sequential prediction in the realizable setting, where labels are generated from an unknown  $f^{*}$  belonging to a hypothesis class  $\mathcal{F}$ . Sequential prediction is typically studied under two distributional assumptions on the input data: the stochastic setting where the data is assumed to be identically and independently distributed (i.i.d) according to some fixed (perhaps unknown) distribution, and the fully-adversarial setting where we make absolutely no assumptions on the data generation process. A simple empirical risk minimization (ERM) strategy works for the stochastic setting where the learner predicts according to the best hypothesis on the data seen so far. The number of mistakes of this strategy typically scales with the Vapnik-Chervonenkis (VC) dimension of the underlying hypothesis class  $\mathcal{F}$ . However, in the fully adversarial setting, this strategy can lead to infinite mistakes even for classes of VC dimension 1 even if the adversary is required to be consistent with labels from  $f^{*}$ . The Littlestone dimension, which characterizes the complexity of sequential prediction in fully-adversarial setting, can be very large and often unbounded compared to
the VC dimension [Lit87]. This mismatch has led to the exploration of beyond worst-case analysis for sequential prediction [RST11, HRS20, RS13a, BCKP20].
In this work, we propose a new framework that sits in between the stochastic and fully-adversarial setting. In particular, we consider sequential prediction with an adversary that injects adversarial (or out-of-distribution) examples in a stream of i.i.d. examples, and a learner that is allowed to abstain from predicting on adversarial examples. A natural motivation for our framework arises in medical diagnosis where the goal is to predict a patient's illness based on symptoms. In cases where the symptoms are not among the commonly indicative ones for the specific disease, or the symptoms may suggest a disease that is outside the scope of the doctor's knowledge, it is safer for the doctor to abstain from making a prediction rather than risk making an incorrect one. Similarly, for self-driving cars, in cases where the car encounters weather conditions outside of its training, or unknown information signs, it is better for the algorithm to hand over access to the driver instead of making a wrong decision which could end up being fatal.
In the proposed framework, the learner's goal is to minimize erroneous predictions on examples that the learner chooses to predict on (i.i.d. or adversarial) while refraining from abstaining on too many i.i.d. examples. If the learner was required to predict on every example, then the adversary could produce a fully-adversarial sequence of examples which would force the learner to make many erroneous predictions. The abstention option allows us to circumvent this challenge and handle any number of adversarial injections without incurring error proportional to the number of injections. In this framework, we can ask the following natural question:
Is there a statistical price for certainty in sequential prediction?
In particular, can we recover stochastic-like guarantees in the presence of an adversary if we are allowed to abstain from predicting on adversarial examples? A priori, it is not clear where on the spectrum between the fully-adversarial and stochastic models, the complexity of this problem lies. The main challenges arise from the fact that the adversary fully controls the injection levels and provides no feedback about which examples were adversarial, and the learner has to perform one-sample outlier detection, which is nearly impossible. Despite this, we show that it is possible to guarantee certainty in a statistically efficient manner.
1.1 Main Contributions
We summarize the main contributions of our work:
- We formalize a new framework of beyond-worst case learning which captures sequential prediction on a stochastic sequence with a clean-label injection-only adversary. With the option of abstention, our framework allows for any number of injections by the adversary without incurring error proportional to the number of injections. Our notion of error simultaneously guarantees few mistakes on classified data while ensuring low abstention rate on non-adversarial data. Our framework naturally connects to uncertainty quantification and testable learning.
- In our framework, we design an algorithm that achieves error  $O(d^{2}\log T)$  for classes with VC dimension  $d$  for time horizon  $T$ , given access to the marginal distribution over the i.i.d. examples. This allows us to get (up to a factor of  $d$ ) the guarantees of the stochastic setting while allowing for any number of adversarial injections.
- We further design an algorithm that achieves  $O(\sqrt{T \log T})$  error for the special (but important) case of VC dimension 1 classes without any access to the marginal distribution over the i.i.d. examples. Similar ideas also allow us to design an algorithm for the class of axis-aligned rectangles in any dimension  $d$  with error  $O(d \sqrt{T \log T})$ .
Our algorithms use a novel measure of uncertainty for VC classes to identify regions of high uncertainty (where the learner abstains) or high information gain (where the learner predicts and learn from their mistakes). The measure uses structural properties of VC classes, in particular, shattered sets of varying sizes. In the known distribution setting, our measure is easy to compute, however for the unknown distribution setting, we show how to design a proxy using only the examples we have seen so far using a leave-one-out type strategy.
Beyond-worst case sequential prediction. Due to pessimistic nature of bounds in adversarial online learning, there are several frameworks designed to address this issue. One approach is to consider
mild restrictions on the adversarial instances such as slight perturbation by noise. This has been formalized as the smoothed adversary model (see [RST11, HRS20, HRS22, HHSY22, BDGR22]) and has been used to get statistical and computationally efficient algorithms. Another approach has been to make the future sequences more predictable given the past instances. Examples of such settings are predictable sequence [RS13b], online learning with hints [BCKP20], and notions of adaptive regret bounds [FRS20].
Abstention-based learning. Abstention has been considered in several other works in classification, both in the online and offline settings. An early example of this is the Chow reject model [Cho70]. Various versions of this have been considered in offline learning (see e.g. [HW06, BW08, BZ20] and references therein) and online learning (see e.g. [ZC16,  $\mathrm{CDG}^{+}19$ , NZ20] and references therein). These results show that abstention can lead to algorithms with desired features, for example fast rates without margins assumptions. Another line of work that is closely related to our setting is the KWIK (knows what it knows) framework by [LLW08] which requires the learner to make predictions only when it is absolutely confident, and abstain otherwise. This requirement was relaxed to allow for mistakes by [SZB10]. The key difference from our work is that they assume a fully-adversarial stream thus the error bounds can be as large as the size of the domain, which is unbounded in the settings we consider. Perhaps, the work that is closest to our setting is the study of adversarial learning with clean-label injections by [GKKM20, KK21]. In their transductive adversarial model, the learner is given labeled training examples and unlabeled test examples on which it must predict, where the test examples may have been injected by an adversary. They show how to abstain with few test misclassifications and few false abstentions on non-adversarial examples. However, in many real-world scenarios, it is unrealistic to expect to have the entire test set in advance, which further motivates the fully online setting that we consider.
Adversarily robust learning. Highly related to our setting is the problem of inductive learning in the presence of adversarial corruptions. The literature on this is generally divided into two scenarios: test-time attacks and training-time attacks. In the case of test-time attacks, the learning algorithm is trained on an (uncorrupted) i.i.d. training set, but its test examples may be corrupted by an adversary whose intention is to change the classification by corrupting the test example  $\left[\mathrm{SZS}^{+}13, \mathrm{BCM}^{+}13, \mathrm{GSS}15, \mathrm{FMS}18, \mathrm{AKM}19, \mathrm{MHS}19, \mathrm{MHS}20, \mathrm{MHS}21, \mathrm{MHS}22, \mathrm{MGDS}20\right]$ . Often the goal in this setting is to learn a classifier that predicts correctly on all adversarial test examples, which is a very strong requirement. Empirical work in this space has focused on designing methods to make training adversarially robust  $\left[\mathrm{MMS}^{+}17, \mathrm{WK}18\right]$ , and also on detecting adversarial examples [PDDZ18, AHFD22]. Detecting adversarial examples is a very challenging task and proposed solutions are often brittle [CW17]. In fact, our framework does not explicitly require detection as long as we can predict correctly on these. On the other hand, in the case of training-time attacks, the training data the learning algorithm trains on is corrupted by an adversary (subject to some constraints on what fraction it may corrupt and what types of corruptions are possible), while the test examples are uncorrupted [Val85, KL93, BEK02, BNL12, ABL17, SKL17, SHN  $^{+}18$ , LF21, GKM21, HKM  $^{+}22$ , BBHS22]. In particular, within this literature, most relevant to the present work is the work on clean-label poisoning, where the adversary's corrupted examples are still labeled correctly by the target concept  $\left[\mathrm{SHN}^{+}18, \mathrm{BHQS}21\right]$ . Comparing to these works, it is interesting to note that the fact that our setting involves sequential prediction (i.e., online), our problem may be viewed simultaneously as both training-time and test-time corruption: that is, because on each round the point we are predicting (or abstaining) on may be inserted by an adversary, this could be viewed as a test-time attack; on the other hand, since the prefix of labeled examples we use to make this prediction may also contain adversarial examples, this can also be viewed as a training-time attack. Thus, our setting requires reasoning accounting for issues arising from both attack scenarios, representing a natural blending of the two types of scenarios.
2 Abstention Framework
In this section, we present the formal framework for sequential prediction with abstentions. For background on sequential prediction and other related concepts, please refer to Appendix A.
Notation. We will denote the domain with  $\mathcal{X}$  and the distribution over  $\mathcal{X}$  as  $\mathfrak{D}$ . We let  $\Delta(\mathcal{X})$  denote the set of all distributions over  $\mathcal{X}$ . We will work in the realizable setting where our label will be according to some function in  $\mathcal{F}$ . Given a class  $\mathcal{F}$  and a data set  $S = \{(x_i, y_i)\}$ , we will denote
by  $\mathcal{F}|_{S}$ , the class  $\mathcal{F}|_{S} = \left\{f\in \mathcal{F}:\forall i\quad f(x_{i}) = y_{i}\right\}$ . When the data set contains a single point  $S = \{(x,y)\}$ , it will be convenient to denote  $\mathcal{F}|_{S}$  as  $\mathcal{F}^{x\to y}$ .
Protocol. At the start, the adversary (or nature) picks a distribution  $\mathfrak{D}$  over the domain  $\mathcal{X}$  and the labelling function  $f^{\star} \in \mathcal{F}$ . We will be interested in both the setting where the learner knows the distribution  $\mathfrak{D}$  and the setting where the learner does not know the distribution  $\mathfrak{D}$ . In the traditional sequential prediction framework, the learner sees input  $x_{t}$  at time  $t$  and makes a prediction  $\hat{y}_{t}$  and observes the true label  $y_{t}$ . The main departure of our setting from this is that an adversary also decides before any round whether to inject an arbitrary element of  $\mathcal{X}$  (without seeing  $x_{t}$ ). We denote by  $\hat{x}_t$  the instance after the adversarial injection  $(\hat{x}_t = x_t$  or  $\hat{x}_t \neq x_t)$ . The learner then observes  $\hat{x}_t$  and makes a prediction  $\hat{y}_{t}$  and observes the true label  $y_{t}$ , as in the traditional sequential prediction framework. We present this formally below.
Protocol 1 Sequential Prediction with Adversarial Injections and Abstentions
Adversary (or nature) initially selects distribution  $\mathfrak{D} \in \Delta(\mathcal{X})$  and  $f^{\star} \in \mathcal{F}$ . The learner does not have access to  $f^{\star}$ . The learner may or may not have access to  $\mathfrak{D}$ .
for  $t = 1,\dots ,T$  do
Adversary decides whether to inject an adversarial input in this the round  $(c_{t} = 1)$  or not  $(c_{t} = 0)$ .
if  $c_{t} = 1$  then Adversary selects any  $\hat{x}_t\in \mathcal{X}$
else Nature selects  $x_{t}\sim \mathfrak{D}$  , and we set  $\hat{x}_t = x_t$
Learner receives  $\hat{x}_t$  and outputs  $\hat{y}_t \in \{0,1,\perp\}$  where  $\perp$  implies that the learner abstains.
Learner receives clean label  $y_{t} = f^{\star}(\hat{x}_{t})$ .
Remark. It is important to note we are in the realizable setting, even after the adversarial injections since the labels are always consistent with a hypothesis  $f^{\star} \in \mathcal{F}$ . This model can naturally be extended to the agnostic setting with adversarial labels. It is also possible to allow the adversary to adaptively choose  $f^{\star} \in \mathcal{F}$ , that is, the adversary has to make sure the labels are consistent with some  $f \in \mathcal{F}$  at all times but does not have to commit to one fixed  $f^{\star}$ . Another interesting variation would be change the feedback to not include the true label  $y_{t}$  on rounds that the learner abstains. As we will see, the known distribution case will be able to handle adaptive  $f^{\star}$  and limited label access.
Objective. In our framework, the goal of the learner is to have low error rate on the rounds it decides to predict (that is when  $\hat{y}_t \in \{0,1\}$ ) while also ensuring that it does not abstain ( $\hat{y}_t = \bot$ ) on too many non-adversarial rounds ( $c_t = 0$ ). More formally, the learner's objective is to minimize the following error (or regret),
$$
\text {E r r o r} := \underbrace {\sum_ {t = 1} ^ {T} \mathbb {1} \left[ \hat {y} _ {t} = 1 - f ^ {\star} \left(\hat {x} _ {t}\right) \right]} _ {\text {M i s c l a s s i f i c a t i o n E r r o r}} + \underbrace {\sum_ {t = 1} ^ {T} \mathbb {1} \left[ c _ {t} = 0 \wedge \hat {y} _ {t} = \bot \right]} _ {\text {A b s t e n t i o n E r r o r}}.
$$
It is important to note that we allow the learner to abstain on adversarial examples for free. This allows the learner to have arbitrarily many injections without paying linearly for them in the error.
Remark. There are many natural generalizations and modification of the model and objective that one could consider. For example, we would consider a cost-based version of this objective that would allow us to trade-off these errors.
Connections to testable learning. A further interesting connection can be made by viewing our model as an online version of testable learning framework of [RV22]. In order to see the analogy more direct, we will focus on the setting where the learner knows the distribution  $\mathfrak{D}$ . In the setting, a learning algorithm is seen as a tester-learner pair. The tester takes the data set as input and outputs whether the data set passes the test. The algorithm then takes as input any data set that passes the test and outputs a hypothesis. The soundness guarantee for the pair of algorithms is that the algorithm run on any data set that passes the test must output a hypothesis that is good on the distribution. The completeness requires that when the dataset is indeed from the "nice" distribution, then the tester passes with high probability. We can see our framework in this light by noting that the decision of whether to abstain or not serves as a test. Thus, in this light, completeness corresponds to the abstention error being small when the data is non-adversarial i.e. is from the true distribution, while the soundness corresponds to the misclassification error being small on points the algorithm decides not to abstain. While the testable learning literature primarily focuses on the computational aspects of learning algorithms, our focus is solely on the statistical aspects.
3 Warm-up: Disagreement-based Learners
As a first example to understand the framework, we consider the most natural learner for the problem. Given the data  $S$  of the examples seen thus far, the learner predicts on examples  $\hat{x}$  whose labels it is certain of. That is, if there is a unique label for  $\hat{x}$  consistent with  $\mathcal{F}|_{S}$ , the learner predicts that label. Else, it abstains from making a prediction. This region of uncertainty is known as the disagreement region.
Example: thresholds in one dimension. Consider learning a single-dimensional threshold in  $[0,1]$  (that is, concepts  $x\mapsto \mathbb{1}[x\geq t]$  for any  $t\in [0,1])$  . While it is well known that ERM achieves  $\log T$  misclassification error for i.i.d. data sequences, in the case of an adversarially chosen sequence, it is also well known that the adversary can select inputs in the disagreement region each time (closer and closer to the decision boundary) and thereby force any non-abstaining learner to make a linear number of mistakes (recall that the Littlestone dimension of thresholds is infinite [Lit87]). Indeed, it is known that the function classes  $\mathcal{F}$  for which non-abstaining predictors can be forced to have MisclassificationError  $= \Omega (T)$  are precisely those with embedded threshold problems of unbounded size [Lit87, She78, Hod97, ALMM19]. Let us now consider the learner that abstains in the disagreement region and predicts based on the consistent hypothesis outside of this region.
Proposition 3.1. Disagreement-based learner for one dimensional thresholds has
$$
\text {M i s c l a s s i c i a t i o n E r r o r} = 0 \text {a n d A b s t e n t i o n E r r o r} \leq 2 \log T.
$$
To see this note that our learner only predicts when the input is not in the disagreement region and thus it never predicts incorrectly (MisclassificationError = 0). As for the abstentions, a simple exchangeability, argument shows that when there are  $n$  i.i.d. examples in the sequence, the probability of the new non-adversarial example being in the disagreement region is  $1/n$ . Summing this over the time horizon gives us the above proposition.
Perfect selective classification and active learning. The learner for the above thresholds problem is a well-known strategy from the areas of perfect selective classification and active learning known as disagreement-based learning [RS88, EYW10, CAL94, BBL09, DHM07, HY15]. In the perfect selective classification setting [RS88, EYW10], the learner observes the examples sequentially, as in our framework, and may predict or abstain on each, and must satisfy the requirement that whenever it predicts its prediction is always correct. From this fact, it immediately follows that applying any perfect selective classification strategy in our setting, we always have MisclassificationError  $= 0$ , so that its performance is judged purely on its abstention rate on the iid examples. It was argued by [EYW10] that the optimal abstention rate among perfect selective classification strategies is obtained by the strategy that makes a prediction on the next example if and only if all classifiers in the hypothesis class that are correct on all examples observed so far, agree on the example. Note that this is precisely the same learner used above. This same strategy has also been studied in the related setting of stream-based active learning<sup>1</sup> [CAL94, BBL09, Han09, Han14, DHM07, HY21]. The abstention rate achievable by this strategy for general hypothesis classes is thoroughly understood [Han07, Han11, Han09, Han12, Han14, Han16, EYW10, EYW12, WHE15, HY15]. In particular, a complete characterization of the optimal distribution-free abstention rate of perfect selective classification is given by the star number of the hypothesis class [HY15, Han16]. The star number  $\mathfrak{s}$  is the size of the largest number  $s$  such that there are examples  $\{x_1,\ldots ,x_s\}$  and hypotheses  $h_0,h_1,\ldots ,h_s$  such that  $h_i$  and  $h_0$  disagree exactly on  $x_{i}$ . For instance,  $\mathfrak{s} = 2$  for threshold classifiers [HY15]. It was shown by [Han16] that the optimal distribution-free abstention rate for perfect selective classification is sublinear if and only if the star number is finite (in which case it is always at most  $\mathfrak{s}\log T$ ). One can show that  $\mathfrak{s}$  is always lower bounded by the VC dimension of the class. Unfortunately, the star number is infinite for most hypothesis classes of interest, even including simple VC classes such as interval classifiers [HY15].
Beyond disagreement-based learning. The learner that abstains whenever it sees an example that it is not certain of may be too conservative. Furthermore, it does not exploit the possibility of learning from mistakes. Let us consider another example to elucidate this failure. Consider the
<sup>1</sup>In this setting, instead of observing a sequence of labelled examples, the learner only observes the examples without their target labels, and at each time may query to observe the target label. The disagreement-based strategy chooses to query precisely on the points for which the classifiers in the hypothesis class correct on the observed labels so far do not all agree on the label [CAL94]. The rate of querying for this strategy is precisely the same as the abstention rate in the perfect selection classification setting [Han11, Han14, EYW12].