SlowGuess/NeurIPS_2023_OCR huggingface dataset:
S. Goel would like to sincerely thank Thodoris Lykouris and Adam Tauman Kalai for working with her on designing the model of adversarial learning with abstention. Several ideas presented in the Section 2 and 3 are based on discussions with T. Lykouris and A. Kalai. We would also like to thank our anonymous NeurIPS reviewer for helping us improve the clarity and accessibility of our paper.
[ABL17] Pranjal Awasthi, Maria Florina Balcan, and Philip M Long. The power of localization for efficiently learning linear separators with noise. Journal of the ACM (JACM), 63(6):1-27, 2017.
[AHFD22] Ahmed Aldahdooh, Wassim Hamidouche, Sid Ahmed Fezza, and Olivier Déforges. Adversarial example detection for dnn models: A review and experimental comparison. Artificial Intelligence Review, 55(6):4403-4462, 2022.
[AKM19] Idan Attias, Aryeh Kontorovich, and Yishay Mansour. Improved generalization bounds for robust learning. In Proceedings of the  $30^{\text{th}}$  International Conference on Algorithmic Learning Theory, 2019.
[ALMM19] Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private PAC learning implies finite littlestone dimension. In Proceedings of the 51st Annual ACM Symposium on Theory of Computing (STOC), page 852–860, 2019.
[BBHS22] M.-F. Balcan, A. Blum, S. Hanneke, and D. Sharma. Robustly-reliable learners under poisoning attacks. In Proceedings of the  $35^{\text{th}}$  Annual Conference on Learning Theory, 2022.
[BBL09] M.-F. Balcan, A. Beygelzimer, and J. Langford. Agnostic active learning. Journal of Computer and System Sciences, 75(1):78-89, 2009.
[BCKP20] Aditya Bhaskara, Ashok Cutkosky, Ravi Kumar, and Manish Purohit. Online learning with imperfect hints. In Proceedings of the International Conference on Machine Learning (ICML), pages 822-831. PMLR, 2020.
$\left[\mathrm{BCM}^{+}13\right]$  Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pages 387-402. Springer, 2013.
[BD15] Shai Ben-David. 2 notes on classes with vapnik-chervonenkis dimension 1, 2015.
[BDGR22] Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. In Proceedings of the Conference on Learning Theory (COLT), pages 1716-1786. PMLR, 2022.
[BEK02] Nader H Bshouty, Nadav Eiron, and Eyal Kushilevitz. PAC learning with nasty noise. Theoretical Computer Science, 288(2):255-275, 2002.
[BHQS21] Avrim Blum, Steve Hanneke, Jian Qian, and Han Shao. Robust learning under clean-label attack. In Conference on Learning Theory (COLT), 2021.
[BNL12] Battista Biggio, B Nelson, and P Laskov. Poisoning attacks against support vector machines. In 29th International Conference on Machine Learning, pages 1807-1814. ArXiv e-prints, 2012.
[BW08] Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(59):1823-1840, 2008.
[BZ20] Olivier Bousquet and Nikita Zhivotovsky. Fast classification rates without standard margin assumptions, 2020.
$\left[\mathbf{MMS}^{+}17\right]$  Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
[NZ20] Gergely Neu and Nikita Zhivotovskiy. Fast rates for online prediction with abstention, 2020.
[PDDZ18] Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. In Advances in Neural Information Processing Systems, pages 4579-4589, 2018.
[RS88] Ronald L. Rivest and Robert H. Sloan. Learning complicated concepts reliably and usefully (extended abstract). In Tom Mitchell and Reid Smith, editors, Proceedings AAAI-88, pages 635–640. AAAI, 1988.
[RS13a] Alexander Rakhlin and Karthik Sridharan. Online learning with predictable sequences. In Proceedings of the Conference on Learning Theory (COLT), pages 993-1019. PMLR, 2013.
[RS13b] Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems, 26, 2013.
[RST11] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Stochastic and constrained adversaries. arXiv preprint arXiv:1104.5070, 2011.
[RV22] Ronitt Rubinfeld and Arsen Vasilyan. Testing distributional assumptions of learning algorithms. arXiv preprint arXiv:2204.07196, 2022.
[She78] Saharon Shelah. Classification Theory and the Number of Non-isomorphic Models. North Holland, 1978.
[SHN+18] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems, pages 6103-6113, 2018.
[SKL17] Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses for data poisoning attacks. Advances in Neural Information Processing Systems, 30, 2017.
[SZB10] Amin Sayedi, Morteza Zadimoghaddam, and Avrim Blum. Trading off mistakes and don't-know predictions. In Advances in Neural Information Processing Systems, pages 2092-2100, 2010.
[SZS+13] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
[Val85] Leslie G Valiant. Learning disjunctions of conjunctions. In Proceedings of the 9th International Joint Conference on Artificial intelligence, pages 560-566, 1985.
[WHE15] Y. Wiener, S. Hanneke, and R. El-Yaniv. A compression technique for analyzing disagreement-based active learning. Journal of Machine Learning Research, 16(4):713-745, 2015.
[WK18] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pages 5286-5295. PMLR, 2018.
[ZC16] Chicheng Zhang and Kamalika Chaudhuri. The extended littlestone's dimension for learning with mistakes and abstentions. In Conference on Learning Theory, pages 1584-1616. PMLR, 2016.
We introduce the framework of sequential binary prediction. We focus on the realizable case for direct comparison to the model that we will subsequently introduce. In sequential binary prediction, at time step  $t$  an adversary picks a  $x_{t} \in \mathcal{X}$  (potentially from a distribution  $\mathfrak{D}_t$ ) and presents it to the learner. The learner then picks  $\hat{y}_{t} \in \{0,1\}$  and then receives as label  $y_{t} = f^{\star}(x_{t})$  where  $f \in \mathcal{F}$  is the unknown function that the learner is trying to learn. The objective of the learner is to minimize the number of mistakes it makes. This is measured using the regret (also known as the mistake bound in the realizable case), defined as
$$
\operatorname {R e g r e t} _ {T} = \sum_ {t = 1} ^ {T} \mathbb {1} [ \hat {y} _ {t} \neq y _ {t} ].
$$
The stochastic case of sequential binary prediction corresponds to the setting when the input  $x_{t}$  is sampled from a fixed  $\mathfrak{D}_t = \mathfrak{D}$  independently across time. This is the sequential analogue of the classical i.i.d. batch model of learning. We now introduce a fundamental notion of complexity for hypothesis classes, the VC dimension which captures the regret in this setting. The VC dimension measures how expressive a hypothesis class is by measuring the number of functions that can be realized by the class on a set of points
Definition A.1 (Shattering and VC Dimension). Let  $\mathcal{X}$  be a domain and  $\mathcal{F}$  be a binary function class on  $\mathcal{X}$  i.e.  $\mathcal{F} \subset \{0,1\}^{\mathcal{X}}$ . A set  $\{x_1,\ldots,x_k\} \subseteq \mathcal{X}$  is said to be shattered by  $\mathcal{F}$  if for all  $y \in \{0,1\}^k$  there exists a function  $f \in \mathcal{F}$  such that  $f(x_i) = y_i$ . The VC dimension of  $\mathcal{F}$  is defined as the maximum  $k$  such that there is a set of size  $k$  that is shattered and is denoted by  $\mathrm{VCDim}(\mathcal{F})$ .
The VC dimension was originally introduced in the context of binary classification in the batch setting but minor modifications of the argument show that even in the sequential prediction setting, the VC dimension characterizes the regret in stochastic sequential prediction.
Continuing the discussion about shattering, we will introduce a notion that corresponding to the probability of shattering a set of points. Though this notion does feature in usual analysis of sequential prediction, it will be a key to the design and analysis of algorithms in our setting. See Section 4 for further discussion.
Definition A.2 (Shattered  $k$ -tuples). Let  $k$  be a positive integer. The set of shattered  $k$ -tuples, denoted by  $\mathcal{S}_k$ , for hypothesis class  $\mathcal{F}$  over a domain  $\mathcal{X}$  is defined as
A case of particular interest in the above definition is  $k = 1$ . This notion is closely studied in active learning under the name of disagreement region.
Definition A.3 (Disagreement Region). Let  $\mathcal{F}$  be a hypothesis class on domain  $\mathcal{X}$ . The disagreement region of  $\mathcal{F}$  is defined as
$$
\mathcal {S} _ {1} \left(\mathcal {F}\right) = \left\{x \in \mathcal {X}: \exists f, g \in \mathcal {F} \text {s u c h t h a t} f (x) \neq g (x) \right\}.
$$
Analogously, we will the disagreement probability as
$$
\rho_ {1} \left(\mathcal {F}, \mathfrak {D}\right) = \mathfrak {D} \left(\mathcal {S} _ {1} \left(\mathcal {F}\right)\right) = \Pr_ {x \sim \mathfrak {D}} \left[ \exists f, g \in \mathcal {F} \text {s u c h t h a t} f (x) \neq g (x) \right].
$$
Going back to sequential prediction, in a seminal result [Lit87] showed that the mistake bound is characterized by a combinatorial property of the hypothesis class known as the Littlestone dimension. Though the definition would not be central to our work, we include it here for comparison to our results.
Definition A.4 (Littlestone Dimension). Let  $\mathcal{F}$  be a hypothesis class on domain  $\mathcal{X}$ . A mistake tree is a full binary decision of depth  $\ell$  tree whose internal nodes are labelled by elements of  $\mathcal{X}$ . Every root
to leaf path in the mistake tree corresponds to a sequence  $(x_{i},y_{i})_{i = 1}^{\ell}$  by associating a label  $y_{i}$  to a node depending on whether it is the left or right child of its parent on the path. A mistake tree of depth  $\ell$  is said to be shattered by a class  $\mathcal{F}$  if for any root to leaf path  $(x_{i},y_{i})_{i = 1}^{\ell}$ , there is a function  $f\in \mathcal{F}$  such that  $f(x_{i}) = y_{i}$  for all  $i\leq \ell$ . The Littlestone dimension of the class  $\mathcal{F}$  denoted by  $\mathrm{LDim}(\mathcal{F})$  is the largest depth of a mistake tree shattered by the class  $\mathcal{F}$ .
The Littlestone dimension completely characterizes the mistake bound of the class  $\mathcal{F}$  in the sense that any algorithm can be forced to have a mistake a bound of  $\Omega$  ( $\mathrm{LDim}(\mathcal{F})$ ) and there exists an algorithm known as the standard optimal algorithm (SOA) with a mistake bound of  $O$  ( $\mathrm{LDim}(\mathcal{F})$ ). Though this characterization is elegant, it is usually interpreted as a negative result since even simple classes the Littlestone dimension is infinite. This is exemplified by the class of thresholds i.e.  $\mathcal{F} = \left\{\mathbb{1}_{[a,1]} : a \in [0,1]\right\}$  over the domain  $\mathcal{X} = [0,1]$ , for which the VC dimension is 1 but the Littlestone dimension is infinite. As noted earlier, the VC dimension captures the number of mistakes when the inputs are i.i.d. while Littlestone captures the number of mistakes for the worst case, dealing to a disparity between achievable mistake bounds in the two settings. This discrepancy is the main motivation for the study of sequential prediction in beyond worst-case settings.
B.1 Properties of Higher-Order Disagreement: Proof of Lemma 4.2
We first begin by relating the probabilities of shattering  $k$  points for the classes gotten by restricting to evaluating to 0 and 1 at  $x$  respectively, for an arbitrary point  $x$ . The proof of the following lemma uses a simple inclusion exclusion argument.
Lemma B.1. For all  $x \in \mathcal{X}$  and any hypothesis class  $\mathcal{F}$ , we have
$$
\mathfrak {D} ^ {\otimes k} \left(\mathcal {S} _ {k} \left(\mathcal {F} ^ {x \rightarrow 1}\right)\right) + \mathfrak {D} ^ {\otimes k} \left(\mathcal {S} _ {k} \left(\mathcal {F} ^ {x \rightarrow 0}\right)\right) \leq \mathfrak {D} ^ {\otimes k} \left(\mathcal {S} _ {k} \left(\mathcal {F}\right)\right) + \mathfrak {D} ^ {\otimes k} \left(\mathcal {S} _ {k} \left(\mathcal {F} ^ {x \rightarrow 1}\right) \cap \mathcal {S} _ {k} \left(\mathcal {F} ^ {x \rightarrow 0}\right)\right).
$$
Equivalently,
$$
\rho_ {k} \left(\mathcal {F} ^ {x \rightarrow 1}\right) + \rho_ {k} \left(\mathcal {F} ^ {x \rightarrow 0}\right) \leq \rho_ {k} (\mathcal {F}) + \Pr_ {x _ {1}, \dots , x _ {k} \sim \mathfrak {D}} \left[ x _ {1}, \dots , x _ {k} \text {i s s h a t t e r e d b y b o t h} \mathcal {F} ^ {x \rightarrow 1} \text {a n d} \mathcal {F} ^ {x \rightarrow 0} \right].
$$
Proof. For  $x_{1}, \ldots, x_{n} \in \mathcal{X}$ , consider the four indicator random variables given by
$$
\begin{array}{l} A _ {1} = \mathbb {1} \left(x _ {1}, \dots , x _ {n} \text {i s s h a t t e r e d b y} \mathcal {F} ^ {x \rightarrow 1}\right) \\ A _ {2} = \mathbb {1} \left(x _ {1}, \dots , x _ {n} \text {i s s h a t t e r e d b y} \mathcal {F} ^ {x \rightarrow 0}\right) \\ A _ {3} = \mathbb {1} \left(x _ {1}, \dots , x _ {n} \text {i s s h a t t e r e d b y} \mathcal {F} ^ {x \rightarrow 1} \text {a n d} \mathcal {F} ^ {x \rightarrow 0}\right) \\ A _ {4} = \mathbb {1} \left(x _ {1}, \dots , x _ {n} \text {i s s h a t t e r e d b y} \mathcal {F}\right). \\ \end{array}
$$
Note that  $A_{1} + A_{2}\leq A_{3} + A_{4}$  . Taking expectations gives the desired result.
In order to prove the main lemma, we then take expectations with respect to the point  $x$  drawn independently from  $\mathfrak{D}$ . The key observation is to relate the probability of  $k$  point being shattered by both classes that evaluate to 0 and 1 at  $x$ , for a random point  $x$ , to the probability of shattering  $k + 1$  points.
Proof of Lemma 4.2. The proof follows by using Lemma B.1 to get (3) and Markov's inequality to get (4). The final line follows by noting that since  $x_{1},\ldots ,x_{k}$  and  $x$  are drawn from  $\mathfrak{D}$  and are
shattered, (4) computes the probability that  $k + 1$  points are shattered.
$$
\begin{array}{l} \Pr_ {x \sim \mathfrak {D}} \left[ \rho_ {k} \left(\mathcal {F} ^ {x \rightarrow 0}\right) + \rho_ {k} \left(\mathcal {F} ^ {x \rightarrow 1}\right) \geq 2 \eta \rho_ {k} (\mathcal {F}) \right] \\ \leq \Pr_ {x \sim \mathfrak {D}} \left[ \rho_ {k} (\mathcal {F}) + \Pr_ {x _ {1}, \dots , x _ {k} \sim \mathfrak {D}} \left[ x _ {1}, \dots , x _ {k} \text {i s s h a t t e r e d b y b o t h} \mathcal {F} ^ {x \rightarrow 1} \text {a n d} \mathcal {F} ^ {x \rightarrow 0} \right] \geq 2 \eta \rho_ {k} (\mathcal {F}) \right] (3) \\ \leq \Pr_ {x \sim \mathfrak {D}} \left[ \Pr_ {x _ {1}, \dots , x _ {k} \sim \mathfrak {D}} \left[ x _ {1}, \dots , x _ {k} \text {i s s h a t t e r e d b y b o t h} \mathcal {F} ^ {x \rightarrow 1} \text {a n d} \mathcal {F} ^ {x \rightarrow 0} \right] \geq (2 \eta - 1) \rho_ {k} (\mathcal {F}) \right] \\ \leq \frac {\cdot \mathbb {E} \left[ \Pr_ {x _ {1} , \dots , x _ {k} \sim \mathfrak {D}} \left[ x _ {1} , \dots , x _ {k} \text {i s s h a t t e r e d b y b o t h} \mathcal {F} ^ {x \rightarrow 1} \text {a n d} \mathcal {F} ^ {x \rightarrow 0} \right]\right]}{(2 \eta - 1) \rho_ {k} (\mathcal {F})} \tag {4} \\ \leq \frac {1}{2 \eta - 1} \cdot \frac {\rho_ {k + 1} (\mathcal {F})}{\rho_ {k} (\mathcal {F})}. \\ \end{array}
$$
B.2 Proof of Lemma 4.3
Proof. Let  $t \in [\ell_k, m_k]$ . Recall that  $\mathcal{F}_t$  denotes the class consistent with the data seen till time  $t$ . Recall that  $H_t$  denotes the history of the interaction till time  $t$ .
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {1} \left[ c _ {t} = 0 \wedge \hat {y} _ {t} (\hat {x} _ {t}) = \perp \right] \mid H _ {t} \right] = \mathbb {E} \left[ \mathbb {1} \left[ c _ {t} = 0 \wedge \hat {y} _ {t} (x _ {t}) = \perp \right] \mid H _ {t} \right] (5) \\ \leq \mathbb {E} \left[ \mathbb {1} \left[ \hat {y} _ {t} (x _ {t}) = \perp \right] \mid H _ {t} \right] \\ \leq \Pr \left[ \min  \left\{\rho_ {k} \left(\mathcal {F} _ {t} ^ {x _ {t} \rightarrow 1}\right), \rho_ {k} \left(\mathcal {F} _ {t} ^ {x _ {t} \rightarrow 1}\right)\right\} \geq 0. 6 \rho_ {k} (\mathcal {F} _ {t}) \mid H (6) \right] \\ \leq \Pr \left[ \rho_ {k} \left(\mathcal {F} _ {t} ^ {x _ {t} \rightarrow 1}\right) + \rho_ {k} \left(\mathcal {F} _ {t} ^ {x _ {t} \rightarrow 1}\right) \geq 1. 2 \rho_ {k} \left(\mathcal {F} _ {t}\right) \mid H _ {t} \right] (7) \\ \leq 5 \frac {\rho_ {k + 1} \left(\mathcal {F} _ {t}\right)}{\rho_ {k} \left(\mathcal {F} _ {t}\right)} (8) \\ \leq 5 \frac {\alpha_ {k + 1}}{\alpha_ {k}}. (9) \\ \end{array}
$$
The equality in (5) follows from the fact that in the uncorrupted rounds,  $\hat{x}_t = x_t$ . The inequality in (6) follows from the condition for abstention in Algorithm 1 and the fact that  $x_{t}\sim \mathfrak{D}$ . (7) follows from the fact that the min of two numbers is at most their average. The key step is (8) which follows from Lemma 4.2. (9) follows from the fact that at level  $k$  in Algorithm 1,  $\rho_{k + 1}(\mathcal{F}_t)\leq \alpha_k$  and  $\rho_{k}(\mathcal{F}_{t})\geq \alpha_{k + 1}$ . Summing this bound and noting that  $e_k - \ell_k$  is at most  $T$ , we get the required bound.
B.3 Proof of Lemma 4.4
Proof. Note that from definition of Algorithm 1, we have that when  $\hat{y}_t \neq \bot$ ,  $\min \left\{\rho_k\left(\mathcal{F}_t^{x_t \to 1}\right), \rho_k\left(\mathcal{F}_t^{x_t \to 0}\right)\right\} \leq 0.6\rho_k\left(\mathcal{F}_t\right)$ . Thus, since we predict with the label corresponding to  $\max \left\{\rho_k\left(\mathcal{F}_t^{x_t \to 1}\right), \rho_k\left(\mathcal{F}_t^{x_t \to 0}\right)\right\}$ , if we make a mistake, we have
$$
\rho_ {k} \left(\mathcal {F} _ {t + 1}\right) \leq 0. 6 \cdot \rho_ {k} \left(\mathcal {F} _ {t}\right).
$$
Also, note once end of the phase corresponding to  $k$ ,  $e_k$ , is reached when  $\rho_k(\mathcal{F}_t) \leq \alpha_k$ . This leads to the required bound.
B.4 Proof of Theorem 4.1
Proof. First, let us look at the misclassification error. Note that Algorithm 1 will not misclassify when  $k = 1$ . To see this, recall from Definition 4.2 that  $\hat{x}_t \backslash S_1$  implies that there is a unique label
consistent with the history. For the remaining levels, we sum the errors from Lemma 4.4 and recall that  $\alpha_{k} = T^{-k}$ , which gives us
$$
\begin{array}{l} \text {M i s c l a s s i f i c a t i o n E r r o r} = \sum_ {t = 1} ^ {T} \mathbb {1} \left[ \hat {y} _ {t} = 1 - f ^ {\star} \left(x _ {t}\right) \right] \\ \leq 2 \sum_ {k = 2} ^ {d} \log \left(\frac {1}{\alpha_ {k}}\right) \\ \leq 2 \sum_ {k = 2} ^ {d} k \log T \\ \leq d ^ {2} \log T. \\ \end{array}
$$
For the abstention error, we again begin with the case of  $k = 1$ . Note that for  $t \geq \ell_1$ , we have
$$
\Pr_ {x \sim \mathfrak {D}} \left[ x \in \mathcal {S} _ {1} \left(\mathcal {F} _ {t}\right) \right] \leq \alpha_ {1}.
$$
Thus, we have a bound of  $T\alpha_{1} \leq 1$  on the expected error in this case. For the remaining levels, we sum the error from (2) over all  $k$ , which gives us the bound
$$
\begin{array}{l} \text {A b s t e n t i o n E r r o r} = \sum_ {t = 1} ^ {T} \mathbb {1} \left[ c _ {t} = 0 \wedge \hat {y} _ {t} = \perp \right] \\ = \sum_ {k = 1} ^ {d} \mathbb {E} \left[ \sum_ {t = \ell_ {k}} ^ {e _ {k}} \mathbb {1} \left[ c _ {t} = 0 \wedge \hat {y} _ {t} = \bot \right] \right] \\ = 1 + 5 T \sum_ {k = 2} ^ {d} \frac {\alpha_ {k + 1}}{\alpha_ {k}} \\ = 1 + 5 T \sum_ {k = 2} ^ {d} \frac {1}{T} \\ <   5 d + 1 \\ \leq 6 d. \\ \end{array}
$$
This gives us the desired bounds.
C.1 Structure for VC Dimension One Classes
Definition C.1. Consider a domain  $\mathcal{X}$  and a partial order  $\prec$  on  $\mathcal{X}$ . We say that a set  $I$  is an initial segment of  $\prec$  if for all  $x \in I$  and  $y \in \mathcal{X}$  such that  $y \prec x$ , we have  $y \in I$ . We say that a partial order is a tree ordering if every initial segment  $I$  is a linear order i.e. for all  $x, y \in I$ , either  $x \prec y$  or  $y \prec x$ .
Theorem C.1 ([BD15]). Let  $\mathcal{F}$  be a hypothesis class over the domain  $\mathcal{X}$ . Then, the following are equivalent:
a.  $\mathcal{F}$  has VC dimension 1.
b. There is a tree ordering  $\prec$  on  $\mathcal{X}$  and a hypothesis  $f\in \mathcal{F}$  such that every element of the set
$$
\mathcal {F} _ {f} = \{h \oplus f: h \in \mathcal {F} \}
$$
is an initial segment of  $\prec$ .
The result above was initially observed in [BD15].
C.2 Proof of Lemma 5.2
Lemma C.2. For any  $t$ , we have that
$$
\gamma_ {t + 1} \leq \gamma_ {t} - \alpha \cdot \mathbb {1} [ \text {M i s c l a s s i f i c a t i o n a t t i m e} t ] + 1. \tag {10}
$$
Proof. First, note that in any round that a mistake was not made, we have that  $\gamma_{t + 1} \leq \gamma_t + 1$ . This is because at most one point is added to the data set in each round.
Note that  $\left|\Gamma\left(S_{t-1}, \mathcal{F}^{\hat{x}_t \to 1}\right)\right| + \left|\Gamma\left(S_{t-1}, \mathcal{F}^{\hat{x}_t \to 0}\right)\right| \leq \left|\Gamma\left(S_{t-1}, \mathcal{F}\right)\right|$ . Further, we have  $\left|\Gamma\left(S_t, \mathcal{F}\right)\right| \leq \left|\Gamma\left(S_{t-1}, \mathcal{F}^{\hat{x}_t \to y_t}\right)\right| + 1$ . From the condition for predicting, we have that
$$
\left. \right. \max  \left\{\left| \Gamma \left(S _ {t - 1}, \mathcal {F} ^ {\hat {x} _ {t} \rightarrow 1}\right)\right|, \left| \Gamma \left(S _ {t - 1}, \mathcal {F} ^ {\hat {x} _ {t} \rightarrow 0}\right)\right|\right\} \geq \alpha .
$$
When we make a misclassification, we go to the smaller value for  $\Gamma$ , therefore we get the desired bound.
Proof of Lemma 5.2. Note that MisclassificationError = ∑t=1T 1 [Misclassification at time t]. Rearranging (10) and summing gives us
$$
\text {M i s c l a s s i f i c a t i o n E r r o r} \leq \frac {1}{\alpha} \sum_ {t = 1} ^ {T} \left(\gamma_ {t} - \gamma_ {t + 1} + 1\right)
$$
Note that for all  $i$ , we have that  $\gamma_i \leq T$  which gives us the desired bound.
C.3 Proof of Lemma 5.3
Definition C.2 (Attachable Point). Let  $\mathcal{F}$  be a hypothesis class and let  $f\in \mathcal{F}$  be a representative function. Let  $S$  be a realizable dataset. We say that a point  $x$  is attachable with respect to a data set  $S$  if there exists a sequence of adversarial examples  $A_{x}$  such that algorithm abstains on example  $x$  when the history is  $S\cup A_x\setminus \{x\}$ . In other words,  $x$  is attachable if there is a set of adversarial examples  $A_{x}$  such that  $x\notin S_1\left(\mathcal{F}|_{S\cup A_x\setminus \{x\}}\right)$  and
$$
\left. \right. \max  \left\{\left| \Gamma (S \cup A _ {x} \backslash \{x \}, \mathcal {F} ^ {x \rightarrow 0}) \right|, \left| \Gamma (S \cup A _ {x} \backslash \{x \}, \mathcal {F} ^ {x \rightarrow 1}) \right|\right\} \leq \alpha .
$$
The key lemma for our analysis is that the number of attackable examples is bounded.
Lemma C.3. Let  $\mathcal{F}$  be a hypothesis and  $f\in \mathcal{F}$  be any reference function. Let  $S$  be any set of examples. Then, the number of attackable examples is at most  $\alpha$ .
The proof uses a structure theorem for classes with VC dimension one which states that they can be represented as initial segements of a tree order.