SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rui M. Castro
Eindhoven University of Technology,
Eindhoven Artificial Intelligence Systems Institute (EAISI)
rmcastro@tue.nl
Fredrik Hellström
f.hellstrom@ucl.ac.uk
Tim van Erven
tim@timvanerven.nl
We consider online prediction of a binary sequence with expert advice. For this setting, we devise label-efficient forecasting algorithms, which use a selective sampling scheme that enables collecting much fewer labels than standard procedures. For the general case without a perfect expert, we prove best-of-both-worlds guarantees, demonstrating that the proposed forecasting algorithm always queries sufficiently many labels in the worst case to obtain optimal regret guarantees, while simultaneously querying much fewer labels in more benign settings. Specifically, for a scenario where one expert is strictly better than the others in expectation, we show that the label complexity of the label-efficient forecaster is roughly upper-bounded by the square root of the number of rounds. Finally, we present numerical experiments empirically showing that the normalized regret of the label-efficient forecaster can asymptotically match known minimax rates for pool-based active learning, suggesting it can optimally adapt to benign settings.
This paper considers online prediction with expert advice in settings where collecting feedback might be costly or undesirable. In the classical framework of sequence prediction with expert advice, a forecasting algorithm aims to sequentially predict a stream of labels on the basis of predictions issued by a number of experts (see, for instance, [1, 2, 3] and references therein). Typically, the forecaster receives the correct label after making a prediction, and uses that feedback to update its prediction strategy. There are, however, situations where collecting labels is costly and potentially unnecessary. In the context of online prediction, this naturally leads to the notion of selective sampling strategies, also called label-efficient prediction [4, 5, 6, 7, 8, 9]. In this line of work, there is a natural tension between performance (in terms of regret bounds) and label complexity, i.e., the number of labels collected. For a worst-case scenario, the optimal label-efficient strategy amounts to "flipping a coin" to decide whether or not to collect feedback, irrespective of past actions and performance [5]. Indeed, in the worst case, the number of labels that one has to collect is linear in the number of rounds for any algorithm [10]. This is a rather pessimistic perspective, and can miss the opportunity to reduce label complexity when prediction is easy. With this in mind, the adaptive selective sampling algorithms we develop follow naturally from a simple design principle: optimize the label collection probability at any time while preserving worst-case regret guarantees. This principled perspective leads to a general way to devise simple but rather powerful algorithms. These are endowed with optimal worst-case
performance guarantees, while allowing the forecaster to naturally adapt to benign scenarios and collect much fewer labels than standard (non-selective sampling) algorithms.
From a statistical perspective, the scenario above is closely related to the paradigm of active learning [11, 12, 13, 14, 15, 16]. For instance, in pool-based active learning, the learner has access to a large pool of unlabeled examples, and can sequentially request labels from selected examples. This extra flexibility, when used wisely, can enable learning a good prediction rule with much fewer labeled examples than what is needed in a passive learning setting, where labeled examples are uniformly sampled from the pool in an unguided way [17, 14, 15, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. Our work is partly motivated by such active learning frameworks, with the aim of devising a simple and adaptive methodology that does not rely on intricate modeling assumptions.
The main contributions of this paper are novel label-efficient exponentially weighted forecasting algorithms, which optimally decide whether or not to collect feedback. The proposed approach confirms, in a sound way, the intuition that collecting labels is more beneficial whenever there is a lack of consensus among the (weighted) experts. The proposed algorithms are designed to ensure that, in adversarial settings, they retain the known worst-case regret guarantees for full-information forecasters (i.e., forecasters that collect all labels) while providing enough flexibility to attain low label complexity in benign scenarios. To characterize the label complexity of the label-efficient forecaster, we focus on a scenario where the expected loss difference between the best expert and all other experts for all  $n$  rounds is lower-bounded by  $\Delta$ , and show that the label complexity is roughly  $\sqrt{n} / \Delta^2$ , ignoring logarithmic factors. This shows that the label-efficient forecaster achieves the "best of both worlds": it smoothly interpolates between the worst case, where no method can have optimal regret with less than  $O(n)$  queries, and the benign, stochastic case, where it is sufficient to make  $O(\sqrt{n})$  queries. Finally, to further examine the performance of the label-efficient forecaster, we conduct a simulation study. We find that the performance of the label-efficient forecaster is comparable to its full-information counterpart, while collecting significantly fewer labels. Intriguingly, for a threshold prediction setting studied in [14], the numerical results indicate that the label-efficient forecaster optimally adapts to the underlying prediction problem, so that its normalized regret displays the same asymptotic behavior as known minimax rates for active learning.
Before formally introducing our setting, we discuss additional related work. Selective sampling for online learning was studied by [4, 5, 28], with a focus on probabilistic threshold functions and margin-based sampling strategies. Similarly, [8] consider kernel-based linear classifiers, and base their sampling procedure on the estimated margin of the classifier. For the same setting as we consider, [29, 30] propose a selective sampling approach based on the maximum (unweighted) prediction disagreement among the experts, and numerically demonstrate its merits. Finally, results in a similar spirit to ours have recently been established in different settings. Namely, for a strongly convex loss, [9] devised an algorithm for selective sampling with expert advice, which provably retains worst-case regret guarantees, where the sampling strategy is based on the variance of the forecaster's prediction. [31] study a setting with shifting hidden domains, and establish a tradeoff between regret and label complexity in terms of properties of these domains. For a setting where the hypothesis class has bounded VC dimension and the data satisfies a Tsybakov noise condition, [32] devise a sampling strategy, with bounds on the regret and label complexity, based on a notion of disagreement where hypotheses are discarded based on their discrepancy relative to the empirical risk minimizer.
2 Setting
Throughout, we focus on a binary prediction task with the zero-one loss as a performance metric. We refer to  $y_{t}$  as the outcome at time  $t \in [n] \coloneqq \{1, \dots, n\}$ . No assumptions are made on this sequence, which can potentially be created in an adversarial way. To aid in the prediction task, the forecaster has access to the predictions of  $N$  experts. The prediction of the forecaster at time  $t$  can only be a function of the expert predictions (up to time  $t$ ) and the observed outcomes up to time  $t - 1$ . Furthermore, the algorithm can make use of internal randomization.
Formally, let  $f_{i,t} \in \{0,1\}$ , with  $i \in [N] \coloneqq \{1,\dots ,N\}$  and  $t \in [n]$ , denote the advice of the experts. At every time  $t \in [n]$ , the forecasting algorithm must: (i) output a prediction  $\hat{y}_t$  of  $y_{t}$ ; (ii) decide whether or not to observe  $y_{t}$ . Specifically, for each round  $t \in [n]$ :
- The environment chooses the outcome  $y_{t}$  and the expert advice  $\{f_{i,t}\}_{i = 1}^{N}$ . Only the expert advice is revealed to the forecaster.
- The forecaster outputs a (possibly randomized) prediction  $\hat{y}_t$ , based on all of the information that it has observed so far.
- The forecaster decides whether or not to have  $y_{t}$  revealed. We let  $Z_{t}$  be the indicator of that decision, where  $Z_{t} = 1$  if  $y_{t}$  is revealed and  $Z_{t} = 0$  otherwise.
- A loss  $\ell(\hat{y}_t, y_t) \coloneqq \mathbb{1} \left\{\hat{y}_t \neq y_t\right\}$  is incurred by the forecaster and a loss  $\ell_{i,t} \coloneqq \ell(f_{i,t}, y_t)$  is incurred by expert  $i$ , regardless of the value of  $Z_t$ .
Our goal is to devise a forecaster that observes as few labels as possible, while achieving low regret with respect to any specific expert. Regret with respect to the best expert at time  $n$  is defined as
$$
R _ {n} := L _ {n} - \min _ {i \in [ N ]} L _ {i, n},
$$
where  $L_{n} \coloneqq \sum_{t=1}^{n} \ell(\hat{y}_{t}, y_{t})$  and  $L_{i,n} \coloneqq \sum_{t=1}^{n} \ell(f_{i,t}, y_{t})$ . Note that the regret  $R_{n}$  is, in general, a random quantity. In this work, we focus mainly on the expected regret  $\mathbb{E}[R_{n}]$ .
Clearly, when no restrictions are imposed on the number of labels collected, the optimal approach would be to always observe the outcomes (i.e., take  $Z_{t} = 1$  for all  $t \in [n]$ ). This is optimal in a worst-case sense, but there are situations where one can predict as efficiently while collecting much fewer labels. The main goal of this paper is the development and analysis of methods that are able to capitalize on such situations, while still being endowed with optimal worst-case guarantees.
2.1 Exponentially weighted forecasters
All proposed algorithms in this paper are variations of exponentially weighted forecasters [2]. For each time  $t \in [n]$ , such algorithms assign a weight  $w_{i,t} \geq 0$  to the  $i$ th expert. The forecast prediction at time  $t$  and decision whether to observe the outcome or not are randomized, and based exclusively on the expert weights and the expert predictions at that time. Therefore,  $\hat{y}_t \sim \mathrm{Ber}(p_t)$  and  $Z_t \sim \mathrm{Ber}(q_t)$  are conditionally independent Bernoulli random variables given  $p_t$  and  $q_t$ . Here,  $p_t$  and  $q_t$  depend on the past only via the weights  $\{w_{i,j-1}\}_{i \in [N], j \in [t]}$  and the current expert predictions  $\{f_{i,t}\}_{i \in [N]}$ . The exact specifications of  $p_t$  and  $q_t$  depend on the setting and assumptions under consideration.
After a prediction has been made, the weights for all experts are updated using the exponential weights update based on the importance-weighted losses  $\ell_{i,t}Z_t / q_t$ . Specifically,
$$
w _ {i, t} = w _ {i, t - 1} e ^ {- \eta \frac {\ell_ {i , t} Z _ {t}}{q _ {t}}} \tag {1}
$$
where  $\eta > 0$  is the learning rate. To ensure that the definition in (1) is sound for any  $q_{t} \geq 0$ , we set  $w_{i,t} = w_{i,t-1}$  if  $q_{t} = 0$ . Finally, we define the weighted average of experts predicting label 1 at time  $t$  as
$$
A _ {1, t} := \frac {\sum_ {i = 1} ^ {N} w _ {i , t - 1} f _ {i , t}}{\sum_ {i = 1} ^ {N} w _ {i , t - 1}}. \tag {2}
$$
This quantity plays a crucial role in our sampling strategy. We will use the name exponentially weighted forecaster liberally to refer to any forecaster for which  $p_t$  is a function of  $A_{1,t}$ . Throughout, we assume that the weights for all forecasters are uniformly initialized as  $w_{i,0} = 1 / N$  for  $i \in [N]$ .
3 Regret bounds with a perfect expert
In this section, we consider a very optimistic scenario where one expert is perfect, in the sense that it does not make any mistakes. The results and derivation for this setting are didactic, and pave the way for more general scenarios where this assumption is dropped. We say that the  $i$ th expert is perfect if  $\ell_{i,t} = 0$  for all  $t \in [n]$ . The existence of such an expert implies that  $\min_{i \in [N]} L_{i,n} = 0$ . Therefore, the regret of a forecaster is simply the number of errors it makes, that is,  $R_n = L_n$ . In such a scenario, any reasonable algorithm should immediately discard experts as soon as they make even a single mistake. For an exponentially weighted forecaster, this is equivalent to setting  $\eta = \infty$ . Due to the uniform weight initialization, the scaled weight vector  $N \cdot (w_{1,t}, \ldots, w_{N,t})$  is thus binary, and indicates which experts agree with all the observed outcomes up to time  $t$ .
First, consider a scenario where the forecaster always collects feedback, that is,  $q_{t} = 1$  for all  $t \in [n]$ . A natural forecasting strategy at time  $t$  is to follow the majority, that is, to predict according to the
majority of the experts that have not made a mistake so far. When the forecaster predicts the wrong label, this implies that at least half of the experts still under consideration are not perfect. Since the number of experts under consideration is at least halved for each mistake the forecaster incurs, this strategy is guaranteed to make at most  $\log_2N$  mistakes. Therefore, we have
$$
R _ {n} = L _ {n} \leq \log_ {2} N. \tag {3}
$$
Clearly, this implies the following bound on the expected cumulative loss, and thus the regret:
$$
\bar {L} _ {n} ^ {(N)} := \mathbb {E} \left[ L _ {n} \right] \leq \log_ {2} N. \tag {4}
$$
Here, the superscript  $(N)$  explicitly denotes the dependence on the number of experts. This bound is tight when the minority is always right and nearly equal in size to the majority.
A natural question to ask is if there exists an algorithm that achieves the expected cumulative loss bound (4) while not necessarily collecting all labels. This is, in fact, possible. The most naive approach is to not collect a label if all experts still under consideration agree on their prediction, as in that case, they must all be correct due to the existence of a perfect expert. However, a more refined strategy that can collect fewer labels is possible, leading to the following theorem.
Theorem 1. Consider the exponentially weighted follow-the-majority forecaster with  $\eta = \infty$ . Specifically, let  $p_t = \mathbb{1}\{A_{1,t} \geq 1/2\}$ , so that  $\hat{y}_t = \mathbb{1}\{A_{1,t} \geq 1/2\}$ . Furthermore, let
$$
q _ {t} = \left\{ \begin{array}{l l} 0 & \text {i f A _ {1 , t} \in \{0 , 1 \} ,} \\ - \frac {1}{\log_ {2} \min  (A _ {1 , t} , 1 - A _ {1 , t})} & \text {o t h e r w i s e .} \end{array} \right.
$$
For this forecaster, we have
$$
\bar {L} _ {n} ^ {(N)} \leq \log_ {2} N.
$$
Recall that  $A_{1,t}$  is simply the proportion of experts still under consideration that predict  $y_{t} = 1$ . It is insightful to look at the expression for  $q_{t}$ , as it is somewhat intuitive. The bigger the disagreement between the experts' predictions, the higher the probability that we collect a label. Conversely, when  $A_{1,t}$  approaches either 0 or 1,  $q_{t}$  quickly approaches zero, meaning we rarely collect a label. Theorem 1 tells us that, remarkably, we can achieve the same worst-case bound as the full-information forecaster while sometimes collecting much less feedback. The proof of this result, in Appendix A, uses a clean induction argument that constructively gives rise to the expression for  $q_{t}$ . This principled way of reasoning identifies, in a sense, the best way to assess disagreement between experts: the specified  $q_{t}$  is the lowest possible sampling probability that preserves worst-case regret guarantees.
A slightly better regret bound is possible by using a variation of follow the majority, called the boosted majority of leaders. For this algorithm, the upper bound is endowed with a matching lower bound (including constant factors). In Appendix B, we devise a label-efficient version of the boosted majority of leaders, retaining the same worst-case regret bound as its full-information counterpart.
4 General regret bounds without a perfect expert
In this section, we drop the assumption of the existence of a perfect expert. It is therefore no longer sensible to use an infinite learning rate  $\eta$ , since this would discard very good experts based on their first observed error. We consider the general exponentially weighted forecaster described in Section 2.1, now simply with  $p_t = A_{1,t}$ .
For the scenario where  $q_{t} = 1$  for all  $t$ , a classical regret bound is well-known (see, for instance, [3, Thm 2.2]). Specifically, for the general exponentially weighted forecaster, with  $p_{t} = A_{1,t}$ ,  $q_{t} = 1$ , and uniform weight initialization, we have
$$
\bar {R} _ {n} := \mathbb {E} [ R _ {n} ] = \mathbb {E} \left[ L _ {n} - \min  _ {i \in [ N ]} L _ {i, n} \right] \leq \frac {\ln N}{\eta} + \frac {n \eta}{8}. \tag {5}
$$
In Theorem 2 below, we prove a stronger version of (5) that allows for an adaptive label-collection procedure. As before, we focus on the expected regret,  $\bar{R}_n = \mathbb{E}[R_n]$ . As done in Section 3 for the case of a perfect expert, we identify an expression for  $q_{t}$ , which is not necessarily identically 1, but still ensures the bound in (5) is valid. To state our main result, we need the following definition, which is guaranteed to be sound by Lemma 1.
(b)
Figure 1: The function  $q^{*}(x,\eta)$  for various values of  $\eta$ . Panel (b) is a zoomed version of panel (a).
Definition 1. For  $x \in [0,1]$  and  $\eta > 0$ , define
$$
q ^ {*} (x, \eta) = \inf  \left\{q \in (0, 1 ]: x + \frac {q}{\eta} \ln \left(1 - x + x e ^ {- \eta / q}\right) \leq \frac {\eta}{8}, \right. \tag {6}
$$
$$
\left. 1 - x + \frac {q}{\eta} \ln \left(x + (1 - x) e ^ {- \eta / q}\right) \leq \frac {\eta}{8} \right\}.
$$
In the following theorem, we present the label-efficient version of (5).
Theorem 2. Consider an exponentially weighted forecaster with  $p_t = A_{1,t}$  and
$$
q _ {t} \geq q ^ {*} (A _ {1, t}, \eta) := q _ {t} ^ {*}.
$$
$$
\bar {R} _ {n} = \mathbb {E} \left[ L _ {n} - \min  _ {i \in [ N ]} L _ {i, n} \right] \leq \frac {\ln N}{\eta} + \frac {n \eta}{8}. \tag {7}
$$
The proof, which is deferred to Appendix C, is similar to that used for Theorem 1, but with key modifications to account for the lack of a perfect expert. In particular, we need to account for the finite, importance-weighted weight updates, and carefully select  $q_{t}^{*}$  accordingly. While the proof allows for non-uniform weight initializations, we focus on the uniform case, as this enables us to optimally tune the learning rate. The result for general weight initializations is given in Appendix C.
Theorem 2 shows that the proposed label-efficient forecaster satisfies the same expected regret bound as the exponentially weighted forecaster with  $q_{t} \coloneqq 1$ . While the expression for  $q^{*}(x,\eta)$  in (6) is somewhat opaque, the underlying motivation is constructive, and it arises naturally in the proof of the theorem. In fact,  $q_{t}^{*}$  is the smallest possible label-collection probability ensuring the regret bound (7). One may wonder if  $q_{t}^{*}$  is well defined, as it is the infimum of a set that may be empty. However, as shown in the following lemma, this set always contains the point 1, ensuring that  $q_{t}^{*} \leq 1$ .
Lemma 1. For all  $\eta >0$  and  $x\in [0,1]$  , we have
$$
1 \in \left\{q \in (0, 1 ]: x + \frac {q}{\eta} \ln \left(1 - x + x e ^ {- \eta / q}\right) \leq \frac {\eta}{8} \right\}.
$$
The proof is presented in Appendix D, and is essentially a consequence of Hoeffding's inequality. While  $q^{*}(x,\eta)$  does not admit an analytic solution, its behavior as a function of  $x$ , depicted in Figure 1, is rather intuitive. Since  $\eta = \sqrt{8(\ln N) / n}$  minimizes the regret bound (7), we are primarily interested in small values of  $\eta$ . When the learning rate  $\eta$  is not too large, the behavior of  $q_{t}^{*}$  can be interpreted as follows: the larger the (weighted) disagreement of the experts is, the closer the value of  $A_{1,t}$  is to the point  $1 / 2$ . In this case,  $q_{t}^{*}$  will be close to 1, and we collect a label with high probability. When  $A_{1,t}$  is close to 0 or 1, the (weighted) experts essentially agree, so the probability of collecting a label will be small. For large learning rates, the behavior of  $q_{t}^{*}$  appears a bit strange, but note that for  $\eta \geq 8$ , the regret bound is vacuous. Thus, for this case,  $q^{*}(x,\eta) = 0$  for all  $x\in [0,1]$ .
The regret guarantee in Theorem 2 is valid provided one uses any choice  $q_{t} \geq q_{t}^{*}$ . The following lemma provides both an asymptotic characterization of  $q_{t}^{*}$  as  $\eta \to 0$ , as well as a simple upper bound that can be used for both analytical purposes and practical implementations.
Lemma 2. For any  $x\in [0,1]$  , we have
$$
\lim  _ {\eta \to 0} q ^ {*} (x, \eta) = 4 x (1 - x).
$$
Furthermore, for any  $\eta > 0$  and  $x \in [0,1]$ ,
$$
q ^ {*} (x, \eta) \leq \min  \left(4 x (1 - x) + \eta / 3, 1\right). \tag {8}
$$
The proof of this result is somewhat technical and tedious, and deferred to Appendix E. In the remainder of this paper, we will use this upper bound extensively.
5 Label complexity
We now examine the label complexity, defined as  $S_{n} \coloneqq \sum_{t=1}^{n} Z_{t}$ . In [10, Thm. 13], it is shown that there exists a setting for which the expected regret of a forecaster that collects  $m$  labels is lower-bounded by  $c n \sqrt{\ln(N-1) / m}$  for some constant  $c$ . Hence, in the worst case, the number of collected labels needs to be linear in  $n$  in order to achieve an expected regret that scales at most as  $\sqrt{n}$ . However, since  $q_{t}^{*}$  can be less than 1, it is clear that the label-efficient exponentially weighted forecaster from Theorem 2 can collect fewer than  $n$  labels in more benign settings. To this end, we consider a scenario with a unique best expert, which at each round is separated from the rest in terms of its expected loss. To state the condition precisely, we need to define  $\mathbb{E}_{t} = \mathbb{E}[|\cdot| \mathcal{F}_{t-1}]$  as the expectation at time  $t$  conditional on all possible randomness up to time  $t-1$ , that is, for  $\mathcal{F}_{t} = \sigma(\{Z_{j}, y_{j}, f_{1,j}, \ldots, f_{N,j}\}_{j=1,\dots,t})$ . With this, we assume that there is a unique expert  $i^{*} \in [N]$  such that, for all  $i \neq i^{*}$  and  $t \in [n]$ ,
$$
\mathbb {E} _ {t} \left[ \ell_ {i, t} - \ell_ {i ^ {*}, t} \right] \geq \Delta > 0 \quad \text {a l m o s t s u r e l y}.
$$
The parameter  $\Delta$  characterizes the difficulty of the given learning problem. If  $\Delta$  is large, the best expert significantly outperforms the others, and is thus easily discernible, whereas if  $\Delta$  is small, the best expert is harder to identify. In particular, if the vectors  $(y_{t},f_{1,t},\ldots ,f_{N,t})$  are independent and identically distributed over rounds  $t\in [n]$ ,  $\Delta$  is just the difference in expected loss between the best and the second-best expert in a single round, which is a common measure of difficulty for stochastic bandits [33, Thm. 2.1]. This difficulty measure has also been used in the context of prediction with expert advice [34]. Similar stochastic assumptions are standard in (batch) active learning, and highly relevant in practical settings (see [14, 15, 18, 21, 27] and references therein). Strictly speaking, our result holds under a more general assumption, where the best expert emerges after a time  $\tau^{*}$  instead of being apparent from the first round. This means that the best expert is even allowed to perform the worst for some rounds, as long as it performs well in sufficiently many other rounds. While we state and prove the result under this more general condition in Appendix F, we present the simpler assumption here for clarity.
We now state our main result for the label complexity.
Theorem 3. Consider the label-efficient exponentially weighted forecaster from Theorem 2 with  $q_{t} = \min (4A_{1,t}(1 - A_{1,t}) + \eta /3,1)$  and any  $\eta >0$ . Suppose that there exists a single best expert  $i^{*}$  such that, for all  $i\neq i^{*}$  and all  $t\in [n]$ ,