SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Faculty of Computer Science
We study a generalization of boosting to the multiclass setting. We introduce a weak learning condition for multiclass classification that captures the original notion of weak learnability as being "slightly better than random guessing". We give a simple and efficient boosting algorithm, that does not require realizability assumptions and its sample and oracle complexity bounds are independent of the number of classes.
In addition, we utilize our new boosting technique in several theoretical applications within the context of List PAC Learning. First, we establish an equivalence to weak PAC learning. Furthermore, we present a new result on boosting for list learners, as well as provide a novel proof for the characterization of multiclass PAC learning and List PAC learning. Notably, our technique gives rise to a simplified analysis, and also implies an improved error bound for large list sizes, compared to previous results.
Boosting is a powerful algorithmic approach used to boost the accuracy of weak learning models, transforming them into strong learners. Boosting was first studied in the context of binary classification in a line of seminal works which include the celebrated Adaboost algorithm, as well as many other algorithms with various applications (see e.g. [14, 22, 11, 12]).
The fundamental assumption underlying boosting is that a method already exists for finding poor, yet not entirely trivial classifiers. Concretely, binary boosting assumes there exists a learning algorithm that, when presented with training examples, can find a classifier  $h: \mathcal{X} \mapsto \{0,1\}$  that has classification error less than  $1/2$ . That is, it performs slightly better than random guessing. The intuition is that this is the most minimal assumption one can make about a learning algorithm, without it being impractical. This assumption is called the weak learning assumption, and it is central to the study of boosting.
While binary boosting theory has been extensively studied, extending it to the multiclass setting has proven to be challenging. In particular, it turns out that the original notion of weak learnability as being "slightly better than a random guess", does not easily extend to the multiclass case. For
example, perhaps the most natural extension is to assume that the learner has accuracy that is slightly better than  $1 / k$ , where  $\mathcal{Y} = \{1,\dots,k\}$ . However, this naive extension is in fact known to be too weak for boosting (see Section 2 below for a detailed discussion). Instead, previous works [19, 7, 23] have formulated various complex weak learning assumptions with respect to carefully tailored loss functions, and rely on restrictive realizability assumptions, making them less useful in practice.
A weak learning assumption. In this work, we generalize the classic formulation of boosting to the multiclass setting. We introduce a weak learning condition that captures the original intuition of weak learnability as "slightly better-than-random guessing". The key idea that renders this condition useful compared to previous attempts, is based on a "hint" given to the weak learner. The hint takes the form of a list of  $k$  labels per example, where  $k$  is possibly smaller than  $|\mathcal{V}|$ . Then, the assumption is that there exists a learner capable of producing not entirely trivial classifiers, if it was provided with a "good hint". In other words, if the list provided to the learner happens to contain the correct label, we expect the learner to perform slightly better than randomly guessing a label from the list. Specifically, the assumption is that for any  $k \geq 2$ , if the given lists of size  $k$  contain the true labels, the learner will output a classifier  $h: \mathcal{X} \mapsto \mathcal{Y}$  with error slightly better than random guessing among the  $k$  labels. Notice that this encompasses both the binary case when  $k = 2$ , as well as the naive extension mentioned above, when  $k = |\mathcal{V}|$ . We call this new condition the "better-than-random guess", or BRG condition.
The BRG condition also generalizes the classic binary case condition in a practical sense. Previous methods on multiclass boosting are framed within the PAC (Probably Approximately Correct) setting, correspond to a known hypothesis class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$ , and assume that weak learning hold for every distribution  $\mathcal{D}$  over the entire domain  $\mathcal{X}$ . Practically, these requirements can be very difficult to check or guarantee. In contrast, as in binary boosting, the BRG condition can be relaxed to a more benign empirical weak learning assumption, that can be verified immediately in an actual learning setting.
Recursive boosting. Our main contribution is a new boosting algorithm that is founded on the BRG condition. Our boosting methodology yields a simple and efficient algorithm. It is based on the key observation that even a naive weak learner can produce a useful hint. Recall that when given no hint at all, the naive weak learner can still find a hypothesis with a slight edge  $(\gamma > 0)$ , over random guessing among  $|\mathcal{V}|$  labels. Although this may result in a poor predictor, we prove that it effectively reduces the label space per example to approximately  $1 / \gamma$  labels. This initial hint serves as the starting point for subsequent iterations of the boosting algorithm. The process continues recursively until the label list per example is reduced to size 2, at which point any classic binary boosting can yield a strong classifier. Unlike previous methods, our boosting algorithm and guarantees do not rely on realizability assumptions nor do they scale with  $|\mathcal{V}|$ . In fact, we show that the sample and oracle-call complexity of our algorithm are entirely independent of  $|\mathcal{V}|$ , which implies our approach is effective even in cases where the label space  $\mathcal{V}$  is possibly infinite. Moreover, the overall running time of our algorithm is polynomial in the size of its input.
An important insight that underlies our approach is the link between the naive weak learning condition, which we term weak-BRG learning, to that of list learning. In list learning [5, 8, 18], rather than predicting a single outcome for a given unseen input, the goal is to provide a short list of predictions. Here we use this technique as an intermediate goal of the algorithm, by which effectively reducing the size of the label space in each round. The generalization analysis relies on sample compression arguments which result in efficient bounds on the sample and oracle complexities.
Perhaps surprisingly, the connection between weak learnability and list learnability is even more fundamental. We prove that there is an equivalence between these two notions. Specifically, we establish that a  $\gamma$ -weak learner is equivalent to an  $(1 / \gamma)$ -list learner.
Lastly, we demonstrate the strength of our boosting framework. First, we give a generalization of our boosting technique to hold for list PAC learning algorithms. Then, we showcase the effectiveness of the weak learning criteria in capturing learnability in two fundamental learning settings: PAC learning, and List PAC learning. Recently, [5] proved a characterization of multiclass PAC learning using the Daniely-Shwartz (DS) dimension. In a subsequent study, [8] gave a characterization of list learnability using a natural extension of the DS dimension. Here we show that in both cases, assuming the appropriate dimension is bounded, one can devise a simple weak learning algorithm. Thus, it is also amenable to a boosting method similarly to our approach, leading to a novel and alternative proof of the characterization of learnability. We note that for cases where the dimension
is much smaller than the list size, we have an improved result over previous bound. Moreover, our approach offers a simpler algorithm and analysis technique, potentially benefiting future applications as well.
The main contributions in this work are as follows.
1. Multiclass boosting framework. Our main result is a boosting framework for the multiclass setting, which is a natural generalization of binary boosting theory. We give a simple weak learning assumption that retains the notion of weak learnability as "slightly-better-than-random-guess" from the binary case. Furthermore, we give an efficient multiclass boosting algorithm, as formally stated in Theorem 1 below. Our boosting algorithm is given in Section 3 (Algorithm 3).
2. Applications: List PAC learning. First, we establish an equivalence between List PAC learning and Weak PAC learning, demonstrating the strong ties between List PAC learning and multiclass boosting theory. Furthermore, we present a new result on boosting for list learners. Lastly, we give a novel and alternative proof for characterization of PAC learning and List PAC learning. In particular, the results imply a simplified algorithmic approach compared to previous works, and improved error bound for cases where the list size is larger than the appropriate dimension [5, 8].
We will now introduce the main weak learning assumption, which we call the "better-than-random guess", or BRG condition, and state our main result in Theorem 1 below.
In its original form, the boosting question begins by assuming that a given hypothesis class  $\mathcal{H} \subseteq \{0,1\}^{\mathcal{X}}$  is weakly-PAC learnable. Similarly, here we present the BRG condition framed as weak (multiclass) PAC setting, followed by a relaxation to an empirical variant of the assumption.
Definition 1 (BRG condition). We say that an hypothesis  $h: \mathcal{X} \to \mathcal{Y}$  satisfies the  $\gamma$ -BRG condition with respect to a list function  $\mu: \mathcal{X} \to \mathcal{Y}^k$  on a distribution  $\mathcal{D}$  over examples if
$$
\Pr_ {(x, y) \sim \mathcal {D}} [ h (x) = y ] \geq \left(\frac {1}{k} + \gamma\right) \Pr_ {(x, y) \sim \mathcal {D}} [ y \in \mu (x) ]. \tag {1}
$$
We say that a learning rule  $\mathcal{W}$  satisfies the  $\gamma$ -BRG condition for a hypothesis class  $\mathcal{H}$  if for every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , for every  $k \geq 2$ , for every list function  $\mu : \mathcal{X} \to \mathcal{Y}^k$ , the output hypothesis  $h$  outputted by  $\mathcal{W}$  satisfies Equation (1) with probability  $1 - \delta$ , when given  $m_0(\delta)$  i.i.d. examples from  $\mathcal{D}$ , and given  $\mu$ .
In words, the condition determines that if  $y$  belongs to the set  $\mu(x)$ , then  $h$  has a higher probability of correctly classifying  $x$  by an additional factor of  $\gamma$ , compared to a random guess from the list  $\mu(x)$ .
However, requiring that the labels be deterministic according to a target function from a known class  $\mathcal{H}$ , and that weak learning hold for every distribution  $\mathcal{D}$  over the entire domain  $\mathcal{X}$  are impractical, as they can be very difficult to check or guarantee.
Instead, as in the binary boosting setting, our condition can be relaxed to a more benign empirical weak learning assumption, as given next.
Definition 2 (Empirical BRG condition). Let  $S \in (\mathcal{X} \times \mathcal{Y})^m$ . We say that a learning rule  $\mathcal{W}$  satisfies the empirical  $\gamma$ -BRG condition for  $S$  if there is an integer  $m_0$  such that for every distribution  $p$  over  $[m]$ , for every  $k \geq 2$ , for every list function  $\mu : \mathcal{X}|_S \to \mathcal{Y}^k$ , when given  $m_0$  examples from  $S$  drawn i.i.d. according to  $p$ , and given  $\mu$ , it outputs a hypothesis  $h$  such that,
$$
\sum_ {i = 1} ^ {m} p _ {i} \cdot \mathbb {1} [ h (x _ {i}) = y _ {i} ] \geq \left(\frac {1}{k} + \gamma\right) \sum_ {i = 1} ^ {m} p _ {i} \cdot \mathbb {1} [ y _ {i} \in \mu (x _ {i}) ]. \tag {2}
$$
Next, we give our main result of an efficient boosting algorithm, as stated in Theorem 1 below.
<sup>1</sup>We denote  $\mathcal{X}|_{S} = \{x\in \mathcal{X}:\exists y\in \mathcal{Y}\mathrm{s.t.}(x,y)\in S\}$ .
Theorem 1 (Boosting (Informal)). There exists a multiclass boosting algorithm  $\mathcal{B}$  such that for any  $\epsilon, \delta > 0$ , and any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , when given a training set  $S \sim \mathcal{D}^m$  and oracle access to a learning rule  $\mathcal{W}$  where  $m = \tilde{O}\left(\frac{m_0}{\gamma^3\epsilon}\right)$  and applying  $\mathcal{B}$  with a total of  $\tilde{O}\left(1 / \gamma^3\right)$  oracle calls to  $\mathcal{W}$ , it outputs a predictor  $\bar{H}: \mathcal{X} \mapsto \mathcal{Y}$  such that with probability at least  $1 - \delta$ , we get that if  $\mathcal{W}$  satisfies the empirical  $\gamma$ -BRG condition for  $S$  then,
$$
\operatorname * {P r} _ {(x, y) \sim \mathcal {D}} \left[ \bar {H} (x) \neq y \right] \leq \epsilon .
$$
Boosting theory has been extensively studied, originally designed for binary classification (e.g., AdaBoost and similar variants) [23]. There are various extensions of boosting to the multiclass setting.
The early extensions include AdaBoost.MH, AdaBoost.MR, and approaches based on Error-Correcting Output Codes (ECOC) [24, 1]. These works often reduce the  $k$ -class task into a single binary task. The binary reduction can have various problems, including increased complexity, and lack of guarantees of an optimal joint predictor.
Other works on multiclass boosting focus on practical considerations and demonstrate empirical performance improvements across various applications [29, 16, 15, 3, 6, 4, 21]. However, they lack a comprehensive theoretical framework for the multiclass boosting problem and often rely on earlier formulations such as one-versus-all reductions to the binary setting or multi-dimensional predictors and codewords.
Notably, a work by [19] established a theoretical framework for multiclass boosting, which generalizes previous learning conditions. However, this requires the assumption that the weak learner minimizes a complicated loss function, that is significantly different from simple classification error. Moreover, it is based on a restrictive realizability assumption with respect to a known hypothesis class. In contrast, we do not require realizability, and only consider the standard classification loss.
More recently, [7] followed a formulation for multiclass boosting similar to that of [19]. They proved a hardness result showing that a broad, yet restricted, class of boosting algorithms must incur a cost which scales polynomially with  $|\mathcal{V}|$ . Our approach does not fall in this class of algorithms. Moreover, our algorithm has sample and oracle complexity bounds that are entirely independent of  $|\mathcal{V}|$ .
2 Warmup: too-weak weak learning
When there are only 2 labels, the weak learner must find a hypothesis that predicts the correct label a bit better than a random guess. That is, with a success probability that is slightly more than  $1/2$ . When the number of labels  $k$  is more than 2, perhaps the most natural extension requires that the weak learner outputs hypotheses that predict the correct label a bit better than a random guess among  $k$  labels. That is, with a success probability that is slightly more than  $1/k$ .
However, this is in fact known to be too weak for boosting (see e.g., [23], Chapter 10). Here we first give a simple example that demonstrates that fact. However, we also show that all is not yet lost for the "better-than-random-guess" intuition. Specifically, we describe how this condition can still allow us to extract valuable knowledge about which labels are incorrect. This observation will serve as a foundation for our main results, which we will elaborate on in the next section.
We start by defining the notion of better-than-random weak learner that we term weak-BRG learning.
Definition 3 (weak-BRG learning). A learning algorithm  $\mathcal{W}$  is a weak-BRG learner for a hypothesis class  $\mathcal{H} \subseteq [k]^{\mathcal{X}}$  if there is  $\gamma > 0$  and  $m_0 : (0,1) \mapsto \mathbb{N}$  such that for any  $\delta_0 > 0$ , and any  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$  over  $\mathcal{X} \times [k]$ , when given  $m_0 \geq m_0(\delta_0)$  samples from  $\mathcal{D}$ , it returns  $h : \mathcal{X} \to \mathcal{Y}$  such that with probability  $1 - \delta_0$ ,
$$
\Pr_ {(x, y) \sim D} [ h (x) = y ] \geq \frac {1}{k} + \gamma . \tag {3}
$$
To get an intuition for why this definition is indeed too weak for boosting, consider the following simple example. Suppose that  $\mathcal{X} = \{a,b,c\}$ ,  $\mathcal{Y} = \{1,2,3\}$ , and that the training set consists of the
2The  $\tilde{O}$  notation conceals polylog  $(m,1 / \delta)$  factors.
three labeled examples  $(a,1)$ ,  $(b,2)$ , and  $(c,3)$ . Further, we suppose that we are using a weak learner which chooses weak classifiers that never distinguish between  $a$  and  $b$ . In particular, the weak learner always chooses one of two weak classifiers:  $h_1$  and  $h_2$ , defined as follows. For  $x \in \{a,b\}$  then  $h_1$  always returns 1 and  $h_2$  always returns 2. For  $x = c$  they both return 3.
Then, notice that for any distribution over the training set, either  $h_1$  or  $h_2$  must achieve an accuracy of at least  $1/2$ , which is significantly higher than the accuracy of  $1/k = 1/3$ . However, regardless of how weak classifiers are aggregated, any final classifier  $H$  that relies solely on the predictions of the weak hypotheses will unavoidably misclassify either  $a$  or  $b$ . As a result, the training accuracy of  $H$  on the three examples can never exceed  $2/3$ , making it impossible to achieve perfect accuracy through any boosting method.
Furthermore, we note that this simple example can also be extended to a case where the data is realizable by a hypothesis class which is not learnable by any learning algorithm (let alone boosting). For example, consider the hypothesis class  $\mathcal{H} = \{1,2,3\}^{\mathcal{X}}$  for  $\mathcal{X} = \mathbb{N}$ . Then,  $\mathcal{H}$  is not PAC learnable (e.g., via No-Free-Lunch ([26], Theorem 5.1)). However, similarly as above, one can construct a learning rule that returns a hypothesis with accuracy  $1/2 > 1/k$  over an  $\mathcal{H}$ -realizable distribution.
Next, we will examine a useful observation that will form the basic building block of our algorithmic methodology. We demonstrate that the natural weak learner given in Definition 3, while weak, is nonetheless useful. This can be shown by examining the guarantees obtained through its application in boosting. Specifically, we consider the following classic variant of boosting via the Hedge algorithm.
Algorithm 1 Boosting via Hedge
Given: Training data  $S \in (\mathcal{X} \times [k])^m$ , parameter  $\eta > 0$ .
Output: A predictor  $H:\mathcal{X}\times \mathcal{Y}\mapsto \mathbb{R}$
1: Initialize:  $w_{1}(i) = 1$  for all  $i = 1, \dots, m$ .
2: for  $t = 1, \dots, T$  do
3: Denote by  $\mathcal{D}_t$  the distribution over  $[m]$  obtained by normalizing  $w_{t}$ .
4: Draw  $m_0$  examples from  $\mathcal{D}_t$  and pass to the weak learner.
5: Get weak hypothesis  $h_t : \mathcal{X} \mapsto \mathcal{Y}$ , and update for  $i = 1, \dots, m$ :
$$
w _ {t + 1} (i) = w _ {t} (i) e ^ {- \eta \cdot \mathbb {1} \left[ h _ {t} \left(x _ {i}\right) = y _ {i} \right]}.
$$
7: Output  $H$  such that for all  $(x,y)\in \mathcal{X}\times [k]$
$$
H (x, y) = \sum_ {t = 1} ^ {T} \mathbb {1} [ h _ {t} (x) = y ].
$$
Notice that the output of Algorithm 1 returns a predictor that is not a classifier, but a scoring function with the aim of predicting the likelihood of a given label candidate  $y \in [k]$  for some  $x \in \mathcal{X}$ . Typically, boosting algorithms combine the weak hypothesis into such a scoring function yet their final output applies an argmax over it, to yield a valid classifier. However, since the weak learning assumption is too weak as we have shown above, taking the argmax is useless in this setting.
Instead, the following lemma shows that by boosting the "too-weak" learner, we can guarantee to eliminate one label for each example in the data. Towards that end, we consider a relaxed variant of the weak-BRG learner, to be defined over a data set  $S$ , which we term the empirical weak-BRG learner. Specifically, we say that a learner satisfies the empirical weak-BRG condition if there is an integer  $m_0$  such that for any distribution over the training examples, when given  $m_0$  examples drawn i.i.d from it, the learner outputs a hypothesis that satisfies Equation (3).
Proofs are deferred to the appendix.
Lemma 1 (Remove one label). Let  $S \in (\mathcal{X} \times [k])^m$ . Let  $\mathcal{W}$  be an empirical weak-BRG learner for  $S$  with respect to some  $\gamma$  and sample size  $m_0$ . Then, the output  $H: (\mathcal{X} \times [k]) \mapsto [0, T]$  obtained by running Algorithm 1 with  $T \geq \frac{8\log(m)}{\gamma^2}$  and  $\eta = \sqrt{\frac{\ln(m)}{2T}}$ , guarantees that for all  $(x, y) \in S$ ,
$\frac{H(x,y)}{T} \geq \frac{1}{k} + \frac{\gamma}{2}$ . Moreover, for all  $(x,y) \in S$  the minimally scored label  $\hat{\ell} = \arg \min_{\ell \in [k]} H(x,\ell)$  must be incorrect. That is  $\hat{\ell} \neq y$ .
Notice that if we were to take the argmax of  $H$  as is typically done in boosting, the guarantees given in Lemma 1 do not suggest this will result in the correct prediction. In fact, this approach might yield a rather bad classifier even for the set  $S$  on which it was trained. In other words, for any  $(x,y)\in S$  it may be that there is some incorrect label  $y^\prime \neq y$  with  $H(x,y^{\prime}) > H(x,y)$ .
However, notice that the lemma does suggest a good classifier of incorrect labels. That is, the lowest scored label will always be an incorrect one, over the training data. This property can be shown to generalize via compression arguments, as discussed in Section 5. This allows us to effectively reduce the size of the label space by one, and is used as the basic building of our algorithm, as detailed in the next section.
3 Multiclass boosting results
We start by introducing the notion of weak learnability that is assumed by our boosting algorithm. We note that it is a relaxation of the empirical BRG condition introduced in Definition 2 in the sense that it does not make any guarantees for the case that the given hint list does not contain the correct label. This may seem like significantly weakening the assumption, yet it turns out to be sufficient for our boosting approach to hold.
In the resulting fully relaxed framework, no assumptions at all are made about the data. Although the BRG condition is not explicitly assumed to hold, when this is the case, our final bound given in Theorem 2 implies a high generalization accuracy.
Definition 4 (Relaxed Empirical  $\gamma$ -BRG learning). Let  $S \in (\mathcal{X} \times \mathcal{Y})^m$ ,  $\gamma > 0$ , and integer  $m_0$ . Let  $M$  be a set of list functions of the form  $^3\mu : \mathcal{X} \mapsto \mathcal{Y}^k$  for any integer  $k$ , such that for each  $\mu \in M$  and  $i \in [m]$ , then  $y_i \in \mu(x_i)$ . A learning algorithm satisfies this condition with respect to  $(S, \gamma, m_0, M)$ , if for any distribution  $p$  over  $S$  and any  $\mu : \mathcal{X} \mapsto \mathcal{Y}^k$  such that  $\mu \in M$ , when given a sample  $S' \sim p^{m_0}$  and access to  $\mu$ , it returns  $h : \mathcal{X} \mapsto \mathcal{Y}$  such that,
$$
\Pr_ {(x, y) \sim p} [ h (x) = y ] \geq \frac {1}{k} + \gamma . \tag {4}
$$
Notice that when the list  $\mu$  returns the set of all possible labels  $\mathcal{V}$  and it is of size  $k$ , this condition is essentially equivalent to the empirical weak-BRG condition, which as shown above is too weak for boosting. Requiring that the condition will hold for any list size  $k \leq |\mathcal{V}|$  is sufficient to facilitate boosting, as shown in Theorem 2.
The starting point of our overall boosting algorithm (given in Algorithm 3), is a simple learning procedure specified in Algorithm 4 that is used to effectively reduce the size of the label space. In particular, it is used to produce the initial "hint" function that is used by the boosting method.
Algorithm 2 Initial hint