Title: Adaptive Selective Sampling for Online Prediction with Experts
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online learning

prediction with experts

selective sampling

active learning
Abstract: We consider online prediction of a binary sequence with expert advice. For this setting, we devise label-efficient forecasting algorithms, which use a selective sampling scheme that enables collecting much fewer labels than standard procedures. For the general case without a perfect expert, we prove best-of-both-worlds guarantees, demonstrating that the proposed forecasting algorithm always queries sufficiently many labels in the worst case to obtain optimal regret guarantees, while simultaneously querying much fewer labels in more benign settings. Specifically, for a scenario where one expert is strictly better than the others in expectation, we show that the label complexity of the label-efficient forecaster is roughly upper-bounded by the square root of the number of rounds. Finally, we present numerical experiments empirically showing that the normalized regret of the label-efficient forecaster can asymptotically match known minimax rates for pool-based active learning, suggesting it can optimally adapt to benign settings.

Adaptive Selective Sampling for Online Prediction with Experts

Abstract

We consider online prediction of a binary sequence with expert advice. For this setting, we devise label-efficient forecasting algorithms, which use a selective sampling scheme that enables collecting much fewer labels than standard procedures. For the general case without a perfect expert, we prove best-of-both-worlds guarantees, demonstrating that the proposed forecasting algorithm always queries sufficiently many labels in the worst case to obtain optimal regret guarantees, while simultaneously querying much fewer labels in more benign settings. Specifically, for a scenario where one expert is strictly better than the others in expectation, we show that the label complexity of the label-efficient forecaster is roughly upperbounded by the square root of the number of rounds. Finally, we present numerical experiments empirically showing that the normalized regret of the label-efficient forecaster can asymptotically match known minimax rates for pool-based active learning, suggesting it can optimally adapt to benign settings.

1 Introduction

This paper considers online prediction with expert advice in settings where collecting feedback might be costly or undesirable. In the classical framework of sequence prediction with expert advice, a forecasting algorithm aims to sequentially predict a stream of labels on the basis of predictions issued by a number of experts (see, for instance, [1, 2, 3] and references therein). Typically, the forecaster receives the correct label after making a prediction, and uses that feedback to update its prediction strategy. There are, however, situations where collecting labels is costly and potentially unnecessary. In the context of online prediction, this naturally leads to the notion of selective sampling strategies, also called label-efficient prediction [4, 5, 6, 7, 8, 9]. In this line of work, there is a natural tension between performance (in terms of regret bounds) and label complexity, i.e., the number of labels collected. For a worst-case scenario, the optimal label-efficient strategy amounts to 'flipping a coin' to decide whether or not to collect feedback, irrespective of past actions and performance [5]. Indeed, in the worst case, the number of labels that one has to collect is linear in the number of rounds for any algorithm [10]. This is a rather pessimistic perspective, and can miss the opportunity to reduce label complexity when prediction is easy. With this in mind, the adaptive selective sampling algorithms we develop follow naturally from a simple design principle: optimize the label collection probability at any time while preserving worst-case regret guarantees. This principled perspective leads to a general way to devise simple but rather powerful algorithms. These are endowed with optimal worst-case

Fredrik Hellström

performance guarantees, while allowing the forecaster to naturally adapt to benign scenarios and collect much fewer labels than standard (non-selective sampling) algorithms.

From a statistical perspective, the scenario above is closely related to the paradigm of active learning [11, 12, 13, 14, 15, 16]. For instance, in pool-based active learning, the learner has access to a large pool of unlabeled examples, and can sequentially request labels from selected examples. This extra flexibility, when used wisely, can enable learning a good prediction rule with much fewer labeled examples than what is needed in a passive learning setting, where labeled examples are uniformly sampled from the pool in an unguided way [17, 14, 15, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. Our work is partly motivated by such active learning frameworks, with the aim of devising a simple and adaptive methodology that does not rely on intricate modeling assumptions.

The main contributions of this paper are novel label-efficient exponentially weighted forecasting algorithms, which optimally decide whether or not to collect feedback. The proposed approach confirms, in a sound way, the intuition that collecting labels is more beneficial whenever there is a lack of consensus among the (weighted) experts. The proposed algorithms are designed to ensure that, in adversarial settings, they retain the known worst-case regret guarantees for full-information forecasters (i.e., forecasters that collect all labels) while providing enough flexibility to attain low label complexity in benign scenarios. To characterize the label complexity of the label-efficient forecaster, we focus on a scenario where the expected loss difference between the best expert and all other experts for all n rounds is lower-bounded by ∆ , and show that the label complexity is roughly √ n/ ∆ 2 , ignoring logarithmic factors. This shows that the label-efficient forecaster achieves the 'best of both worlds': it smoothly interpolates between the worst case, where no method can have optimal regret with less than O ( n ) queries, and the benign, stochastic case, where it is sufficient to make O ( √ n ) queries. Finally, to further examine the performance of the label-efficient forecaster, we conduct a simulation study. We find that the performance of the label-efficient forecaster is comparable to its full-information counterpart, while collecting significantly fewer labels. Intriguingly, for a threshold prediction setting studied in [14], the numerical results indicate that the label-efficient forecaster optimally adapts to the underlying prediction problem, so that its normalized regret displays the same asymptotic behavior as known minimax rates for active learning.

Before formally introducing our setting, we discuss additional related work. Selective sampling for online learning was studied by [4, 5, 28], with a focus on probabilistic threshold functions and margin-based sampling strategies. Similarly, [8] consider kernel-based linear classifiers, and base their sampling procedure on the estimated margin of the classifier. For the same setting as we consider, [29, 30] propose a selective sampling approach based on the maximum (unweighted) prediction disagreement among the experts, and numerically demonstrate its merits. Finally, results in a similar spirit to ours have recently been established in different settings. Namely, for a strongly convex loss, [9] devised an algorithm for selective sampling with expert advice, which provably retains worst-case regret guarantees, where the sampling strategy is based on the variance of the forecaster's prediction. [31] study a setting with shifting hidden domains, and establish a tradeoff between regret and label complexity in terms of properties of these domains. For a setting where the hypothesis class has bounded VC dimension and the data satisfies a Tsybakov noise condition, [32] devise a sampling strategy, with bounds on the regret and label complexity, based on a notion of disagreement where hypotheses are discarded based on their discrepancy relative to the empirical risk minimizer.

2 Setting

Throughout, we focus on a binary prediction task with the zero-one loss as a performance metric. We refer to y t as the outcome at time t ∈ [ n ] := { 1 , . . . , n } . No assumptions are made on this sequence, which can potentially be created in an adversarial way. To aid in the prediction task, the forecaster has access to the predictions of N experts. The prediction of the forecaster at time t can only be a function of the expert predictions (up to time t ) and the observed outcomes up to time t -1 . Furthermore, the algorithm can make use of internal randomization.

Formally, let f i,t ∈ { 0 , 1 } , with i ∈ [ N ] := { 1 , . . . , N } and t ∈ [ n ] , denote the advice of the experts. At every time t ∈ [ n ] , the forecasting algorithm must: (i) output a prediction ˆ y t of y t ; (ii) decide whether or not to observe y t . Specifically, for each round t ∈ [ n ] :

The environment chooses the outcome y t and the expert advice { f i,t } N i =1 . Only the expert advice is revealed to the forecaster.

The forecaster outputs a (possibly randomized) prediction ˆ y t , based on all of the information that it has observed so far.

The forecaster decides whether or not to have y t revealed. We let Z t be the indicator of that decision, where Z t = 1 if y t is revealed and Z t = 0 otherwise.

̸

Our goal is to devise a forecaster that observes as few labels as possible, while achieving low regret with respect to any specific expert. Regret with respect to the best expert at time n is defined as

A loss ℓ (ˆ y t , y t ) := ✶ { ˆ y t = y t } is incurred by the forecaster and a loss ℓ i,t := ℓ ( f i,t , y t ) is incurred by expert i , regardless of the value of Z t .

where L n := ∑ n t =1 ℓ (ˆ y t , y t ) and L i,n := ∑ n t =1 ℓ ( f i,t , y t ) . Note that the regret R n is, in general, a random quantity. In this work, we focus mainly on the expected regret E [ R n ] .

Clearly, when no restrictions are imposed on the number of labels collected, the optimal approach would be to always observe the outcomes (i.e., take Z t = 1 for all t ∈ [ n ] ). This is optimal in a worst-case sense, but there are situations where one can predict as efficiently while collecting much fewer labels. The main goal of this paper is the development and analysis of methods that are able to capitalize on such situations, while still being endowed with optimal worst-case guarantees.

2.1 Exponentially weighted forecasters

All proposed algorithms in this paper are variations of exponentially weighted forecasters [2]. For each time t ∈ [ n ] , such algorithms assign a weight w i,t ≥ 0 to the i th expert. The forecast prediction at time t and decision whether to observe the outcome or not are randomized, and based exclusively on the expert weights and the expert predictions at that time. Therefore, ˆ y t ∼ Ber ( p t ) and Z t ∼ Ber ( q t ) are conditionally independent Bernoulli random variables given p t and q t . Here, p t and q t depend on the past only via the weights { w i,j -1 } i ∈ [ N ] ,j ∈ [ t ] and the current expert predictions { f i,t } i ∈ [ N ] . The exact specifications of p t and q t depend on the setting and assumptions under consideration.

After a prediction has been made, the weights for all experts are updated using the exponential weights update based on the importance-weighted losses ℓ i,t Z t /q t . Specifically,

where η > 0 is the learning rate. To ensure that the definition in (1) is sound for any q t ≥ 0 , we set w i,t = w i,t -1 if q t = 0 . Finally, we define the weighted average of experts predicting label 1 at time t as

This quantity plays a crucial role in our sampling strategy. We will use the name exponentially weighted forecaster liberally to refer to any forecaster for which p t is a function of A 1 ,t . Throughout, we assume that the weights for all forecasters are uniformly initialized as w i, 0 = 1 /N for i ∈ [ N ] .

3 Regret bounds with a perfect expert

In this section, we consider a very optimistic scenario where one expert is perfect, in the sense that it does not make any mistakes. The results and derivation for this setting are didactic, and pave the way for more general scenarios where this assumption is dropped. We say that the i th expert is perfect if ℓ i,t = 0 for all t ∈ [ n ] . The existence of such an expert implies that min i ∈ [ N ] L i,n = 0 . Therefore, the regret of a forecaster is simply the number of errors it makes, that is, R n = L n . In such a scenario, any reasonable algorithm should immediately discard experts as soon as they make even a single mistake. For an exponentially weighted forecaster, this is equivalent to setting η = ∞ . Due to the uniform weight initialization, the scaled weight vector N · ( w 1 ,t , . . . , w N,t ) is thus binary, and indicates which experts agree with all the observed outcomes up to time t .

First, consider a scenario where the forecaster always collects feedback, that is, q t = 1 for all t ∈ [ n ] . A natural forecasting strategy at time t is to follow the majority , that is, to predict according to the

majority of the experts that have not made a mistake so far. When the forecaster predicts the wrong label, this implies that at least half of the experts still under consideration are not perfect. Since the number of experts under consideration is at least halved for each mistake the forecaster incurs, this strategy is guaranteed to make at most log 2 N mistakes. Therefore, we have

Clearly, this implies the following bound on the expected cumulative loss, and thus the regret:

Here, the superscript ( N ) explicitly denotes the dependence on the number of experts. This bound is tight when the minority is always right and nearly equal in size to the majority.

A natural question to ask is if there exists an algorithm that achieves the expected cumulative loss bound (4) while not necessarily collecting all labels. This is, in fact, possible. The most naive approach is to not collect a label if all experts still under consideration agree on their prediction, as in that case, they must all be correct due to the existence of a perfect expert. However, a more refined strategy that can collect fewer labels is possible, leading to the following theorem.

Theorem 1. Consider the exponentially weighted follow-the-majority forecaster with η = ∞ . Specifically, let p t = ✶ { A 1 ,t ≥ 1 / 2 } , so that ˆ y t = { A 1 ,t ≥ 1 / 2 } . Furthermore, let

For this forecaster, we have

Recall that A 1 ,t is simply the proportion of experts still under consideration that predict y t = 1 . It is insightful to look at the expression for q t , as it is somewhat intuitive. The bigger the disagreement between the experts' predictions, the higher the probability that we collect a label. Conversely, when A 1 ,t approaches either 0 or 1 , q t quickly approaches zero, meaning we rarely collect a label. Theorem 1 tells us that, remarkably, we can achieve the same worst-case bound as the full-information forecaster while sometimes collecting much less feedback. The proof of this result, in Appendix A, uses a clean induction argument that constructively gives rise to the expression for q t . This principled way of reasoning identifies, in a sense, the best way to assess disagreement between experts: the specified q t is the lowest possible sampling probability that preserves worst-case regret guarantees.

A slightly better regret bound is possible by using a variation of follow the majority, called the boosted majority of leaders. For this algorithm, the upper bound is endowed with a matching lower bound (including constant factors). In Appendix B, we devise a label-efficient version of the boosted majority of leaders, retaining the same worst-case regret bound as its full-information counterpart.

4 General regret bounds without a perfect expert

In this section, we drop the assumption of the existence of a perfect expert. It is therefore no longer sensible to use an infinite learning rate η , since this would discard very good experts based on their first observed error. We consider the general exponentially weighted forecaster described in Section 2.1, now simply with p t = A 1 ,t .

For the scenario where q t = 1 for all t , a classical regret bound is well-known (see, for instance, [3, Thm 2.2]). Specifically, for the general exponentially weighted forecaster, with p t = A 1 ,t , q t = 1 , and uniform weight initialization, we have

In Theorem 2 below, we prove a stronger version of (5) that allows for an adaptive label-collection procedure. As before, we focus on the expected regret, ¯ R n = E [ R n ] . As done in Section 3 for the case of a perfect expert, we identify an expression for q t , which is not necessarily identically 1 , but still ensures the bound in (5) is valid. To state our main result, we need the following definition, which is guaranteed to be sound by Lemma 1.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

0

.

2

0

.

4

0

.

6

0

.

8

1

x

q

∗

(

x, η

)

η

= 0

.

1

η

= 0

.

8

η

= 2

η

= 6

(a)

0

1

2

3

4

5

·

10

-

2

0

0

.

2

0

.

4

0

.

6

0

.

8

1

x

q

∗

(

x, η

)

η

= 0

.

1

η

= 0

.

8

η

= 2

η

= 6

(b)

Definition 1. For x ∈ [0 , 1] and η > 0 , define

In the following theorem, we present the label-efficient version of (5).

Theorem 2. Consider an exponentially weighted forecaster with p t = A 1 ,t and

The proof, which is deferred to Appendix C, is similar to that used for Theorem 1, but with key modifications to account for the lack of a perfect expert. In particular, we need to account for the finite, importance-weighted weight updates, and carefully select q ∗ t accordingly. While the proof allows for non-uniform weight initializations, we focus on the uniform case, as this enables us to optimally tune the learning rate. The result for general weight initializations is given in Appendix C.

Theorem 2 shows that the proposed label-efficient forecaster satisfies the same expected regret bound as the exponentially weighted forecaster with q t := 1 . While the expression for q ∗ ( x, η ) in (6) is somewhat opaque, the underlying motivation is constructive, and it arises naturally in the proof of the theorem. In fact, q ∗ t is the smallest possible label-collection probability ensuring the regret bound (7). One may wonder if q ∗ t is well defined, as it is the infimum of a set that may be empty. However, as shown in the following lemma, this set always contains the point 1 , ensuring that q ∗ t ≤ 1 .

Lemma 1. For all η > 0 and x ∈ [0 , 1] , we have

The proof is presented in Appendix D, and is essentially a consequence of Hoeffding's inequality. While q ∗ ( x, η ) does not admit an analytic solution, its behavior as a function of x , depicted in Figure 1, is rather intuitive. Since η = √ 8(ln N ) /n minimizes the regret bound (7), we are primarily interested in small values of η . When the learning rate η is not too large, the behavior of q ∗ t can be interpreted as follows: the larger the (weighted) disagreement of the experts is, the closer the value of A 1 ,t is to the point 1 / 2 . In this case, q ∗ t will be close to 1 , and we collect a label with high probability. When A 1 ,t is close to 0 or 1, the (weighted) experts essentially agree, so the probability of collecting a label will be small. For large learning rates, the behavior of q ∗ t appears a bit strange, but note that for η ≥ 8 , the regret bound is vacuous. Thus, for this case, q ∗ ( x, η ) = 0 for all x ∈ [0 , 1] .

The regret guarantee in Theorem 2 is valid provided one uses any choice q t ≥ q ∗ t . The following lemma provides both an asymptotic characterization of q ∗ t as η → 0 , as well as a simple upper bound that can be used for both analytical purposes and practical implementations.

For this forecaster, we have

Lemma 2. For any x ∈ [0 , 1] , we have

Furthermore, for any η > 0 and x ∈ [0 , 1]

The proof of this result is somewhat technical and tedious, and deferred to Appendix E. In the remainder of this paper, we will use this upper bound extensively.

5 Label complexity

̸

We now examine the label complexity, defined as S n := ∑ n t =1 Z t . In [10, Thm. 13], it is shown that there exists a setting for which the expected regret of a forecaster that collects m labels is lower-bounded by cn √ ln( N -1) /m for some constant c . Hence, in the worst case, the number of collected labels needs to be linear in n in order to achieve an expected regret that scales at most as √ n . However, since q ∗ t can be less than 1 , it is clear that the label-efficient exponentially weighted forecaster from Theorem 2 can collect fewer than n labels in more benign settings. To this end, we consider a scenario with a unique best expert, which at each round is separated from the rest in terms of its expected loss. To state the condition precisely, we need to define E t = E [ · | F t -1 ] as the expectation at time t conditional on all possible randomness up to time t -1 , that is, for F t = σ ( { Z j , y j , f 1 ,j , . . . , f N,j } j =1 ,...,t ) . With this, we assume that there is a unique expert i ∗ ∈ [ N ] such that, for all i = i ∗ and t ∈ [ n ] ,

The parameter ∆ characterizes the difficulty of the given learning problem. If ∆ is large, the best expert significantly outperforms the others, and is thus easily discernible, whereas if ∆ is small, the best expert is harder to identify. In particular, if the vectors ( y t , f 1 ,t , . . . , f N,t ) are independent and identically distributed over rounds t ∈ [ n ] , ∆ is just the difference in expected loss between the best and the second-best expert in a single round, which is a common measure of difficulty for stochastic bandits [33, Thm. 2.1]. This difficulty measure has also been used in the context of prediction with expert advice [34]. Similar stochastic assumptions are standard in (batch) active learning, and highly relevant in practical settings (see [14, 15, 18, 21, 27] and references therein). Strictly speaking, our result holds under a more general assumption, where the best expert emerges after a time τ ∗ instead of being apparent from the first round. This means that the best expert is even allowed to perform the worst for some rounds, as long as it performs well in sufficiently many other rounds. While we state and prove the result under this more general condition in Appendix F, we present the simpler assumption here for clarity.

We now state our main result for the label complexity.

̸

Theorem 3. Consider the label-efficient exponentially weighted forecaster from Theorem 2 with q t = min(4 A 1 ,t (1 -A 1 ,t ) + η/ 3 , 1) and any η > 0 . Suppose that there exists a single best expert i ∗ such that, for all i = i ∗ and all t ∈ [ n ] ,

Then, for any n ≥ 4 , the expected label complexity is at most

̸

Proof sketch. Initially, the sampling probability q t is large, but as we collect more labels, it will become detectable that one of the experts is better than the others. As this happens, q t will tend to decrease until it (nearly) reaches its minimum value η/ 3 . We therefore divide the forecasting process into time t ≤ τ and t > τ . With a suitable choice of τ ≈ 1 / ( η ∆ 2 ) (up to logarithmic factors), we can guarantee that the sampling probability is at most q t ≤ 4 η/ 3 for all t > τ with sufficiently high probability. This is shown by controlling the deviations of the cumulative importance-weighted loss differences ˜ Λ i t = ∑ t j =1 ( l i,j -l i ∗ ,j ) /q j for i = i ∗ from their expected values by using an anytime version of Freedman's inequality. With this, we simply upper bound the label complexity for the first τ rounds by τ , and over the remaining rounds, the expected number of collected labels is roughly η ( n -τ ) ≤ ηn . This leads to a total expected label complexity of 1 / ( η ∆ 2 ) + ηn , up to logarithmic factors. The full proof is deferred to Appendix F.

As mentioned earlier, the learning rate optimizing the regret bound (7) is η = √ 8 ln( N ) /n . For this particular choice, the label complexity in (9) is roughly √ n/ ∆ 2 , up to constants and logarithmic factors. We have thus established that the label-efficient forecaster achieves the best of both worlds: it queries sufficiently many labels in the worst case to obtain optimal regret guarantees, while simultaneously querying much fewer labels in more benign settings. It is interesting to note that the label complexity dependence of 1 / ∆ 2 on ∆ is less benign than the dependence of the regret bound from, e.g., [34, Thm. 11], which is 1 / ∆ . The underlying reason for this is that, while the two are similar, the label complexity is not directly comparable to the regret. In particular, the label complexity has much higher variance.

The bound of Theorem 3 relies on setting the sampling probability q t to be the upper bound on q ∗ t from Lemma 2. This bound is clearly loose when q ∗ t is approximately zero, and one may wonder if the label complexity of the algorithm would be radically smaller when using a forecaster for which q t = q ∗ t instead. With the choice η = √ 8 ln( N ) /n , which optimizes the bound in (7), it seems unlikely that the label complexity will substantially change, as numerical experiments suggest that the label complexity attained with q t set as q ∗ t or the corresponding upper bound from (8) appear to be within a constant factor. That being said, for larger values of η , the impact of using the upper bound in (8) is likely much more dramatic.

6 Numerical experiments

To further assess the behavior of the label-efficient forecaster from Theorem 2, we consider a classical active learning scenario in a batch setting, for which there are known minimax rates for the risk under both active and passive learning paradigms. We will set the sampling probability to be

Let D n = (( X t , Y t )) n t =1 be an ordered sequence of independent and identically distributed pairs of random variables with joint distribution D . The first entry of ( X i , Y i ) represents a feature, and the second entry is the corresponding label. The goal is to predict the label Y i ∈ { 0 , 1 } based on the feature X i . Specifically, we want to identify a map ( x, D n ) ↦→ ˆ g n ( x, D n ) ∈ { 0 , 1 } such that, for a pair ( X,Y ) ∼ D that is drawn independently from D n , we have small (zero-one loss) expected risk

̸

Concretely, we consider the following scenario, inspired by the results in [14]. Let the features X i be uniformly distributed in [0 , 1] , and Y i ∈ { 0 , 1 } be such that P ( Y i = 1 | X i = x ) = ζ ( x ) . Specifically, let τ 0 ∈ [0 , 1] such that ζ ( x ) ≥ 1 / 2 when x ≥ τ 0 and ζ ( x ) ≤ 1 / 2 otherwise. Furthermore, assume that for all x ∈ [0 , 1] , ζ ( x ) satisfies

for some c, C > 0 and κ > 1 . If τ 0 is known, the optimal classifier is simply g ∗ ( x ) = ✶ { x ≥ τ 0 } . The minimum achievable excess risk when learning ˆ g n from D n in this type of problems has been studied in, e.g., [14, 35]. For this setting, it is known that

as n → ∞ . However, rather than the classical supervised learning setting above, we can instead consider active learning procedures. Specifically, consider a sequential learner that can generate feature-queries X ′ i and sample a corresponding label Y ′ i , such that P ( Y ′ i = 1 | X ′ i = x ) = ζ ( x ) . This is often referred to as pool-based active learning. At time t , the learner can choose X ′ t as a function of the past (( X ′ j , Y ′ j )) t -1 j =1 according to a (possibly random) sampling strategy A n . This extra flexibility allows the learner to carefully select informative examples to guide the learning process. Similarly to the passive learning setting, the ultimate goal is to identify a prediction rule ( x, D ′ n ) ↦→ ˆ g A n ( x, D ′ n ) ∈ { 0 , 1 } , where D ′ n = (( X ′ t , Y ′ t )) n t =1 . In [14], it is shown that for this active learning setting, the minimax rates are also known, and given by

0

1

2

3

4

5

·

10

4

0

50

100

150

200

t

E

[

Rt

]

Full information

Label efficient

(a)

κ

= 2

0

1

2

3

4

5

·

10

4

0

50

100

150

200

t

E

[

Rt

]

Full information

Label efficient

(b)

κ

= 1

.

5

0

1

2

3

4

5

·

10

4

0

2

,

000

4

,

000

6

,

000

8

,

000

t

E

[

St

]

(c)

κ

= 2

0

1

2

3

4

5

·

10

4

0

2

,

000

4

,

000

6

,

000

8

,

000

t

E

[

St

]

(d)

κ

= 1

.

5

as n → ∞ . This shows that there are potentially massive gains for active learning, particularly when κ is close to 1. A natural question is whether similar conclusions hold for streaming active learning. In this setting, instead of selecting which example X ′ i to query, the learner observes the features ( X 1 , . . . , X n ) sequentially, and decides at each time t whether or not it should query the corresponding label. This is analogous to the online prediction setting discussed in this paper.

We now study this setting numerically. For the simulations, we use the specific choice

to generate sequences ( Y 1 , . . . , Y n ) , based on a sequence of features ( X 1 , . . . , X n ) sampled from the uniform distribution on [0 , 1] . Furthermore, we consider the class of N experts such that with i ∈ [ N ] and t ∈ [ n ] .

6.1 Expected regret and label complexity

In the simulations, we set τ 0 = 1 / 2 and N = ⌈ √ n ⌉ + ✶ {⌈ √ n ⌉ is even } . This choice enforces that N is odd, ensuring the optimal classifier is one of the experts. Throughout, we set η = √ 8 ln( N ) /n , which minimizes the regret bound (7). First, we investigate the expected regret relative to the optimal prediction rule for the label-efficient exponentially weighted forecaster with q t given by (8), and compare it with the corresponding regret for the full-information forecaster that collects all labels. Specifically, the regret at time t of a forecaster that predicts { ˆ Y j } n j =1 is given by

10

0

10

1

10

2

10

3

10

4

10

-

3

10

-

2

10

-

1

E

[

St

]

E

[

Rt

]

t

Full information

Label efficient

(a)

κ

= 2

10

0

10

1

10

2

10

3

10

4

10

-

3

10

-

2

10

-

1

E

[

St

]

E

[

Rt

]

t

Full information

Label efficient

(b)

κ

= 1

.

5

̸

Furthermore, to study the potential reduction in the number of collected labels, we also evaluate the expected label complexity E [ S t ] of the label-efficient forecaster. To approximate the expectations above, we use Monte-Carlo averaging with 500 independent realizations. Further experimental details are given in Appendix G. The results are shown in Figure 2. We see that the regret is comparable for the full-information and label-efficient forecasters, albeit slightly higher for the latter. Since P ( g ∗ ( X ) = Y ) = 1 2 -1 κ 2 κ , the expected cumulative loss of the optimal classifier grows linearly with t . For instance, when κ = 2 , we have ∑ t j =1 E [ ℓ ( g ∗ ( X t ) , Y t )] = 3 t/ 8 . Hence, the regret relative to the best expert is much smaller than the pessimistic (i.e., worst-case for adversarial environments) bound in (7). We also observe that the expected label complexity grows sub-linearly with t , as expected, and that E [ S n ] ≪ n , demonstrating that a good prediction rule can be learned with relatively few labels. When κ = 1 . 5 , the number of collected labels is significantly smaller than when κ = 2 . This is in line with the known minimax rates for active learning from [14]. To further examine this connection, we now turn to normalized regret.

6.2 Normalized regret relative to the number of samples

To relate the results of the label-efficient forecaster with known minimax rates for active learning, we investigate the expected regret normalized by the number of samples. Specifically, let

For the full-information forecaster, we expect that r ( t ) ≍ t -κ/ (2 κ -1) as t →∞ . The same holds for the label-efficient forecaster, but in this case, the relation between r ( t ) and the expected number of collected labels E [ S t ] is more interesting. If the label-efficient forecaster performs optimally, we expect r ( t ) ≍ E [ S t ] -κ/ (2 κ -2) as t → ∞ . To examine this, we plot r ( t ) against E [ S t ] (which equals t for the full-information forecaster) in logarithmic scales, so the expected asymptotic behavior corresponds to a linear decay with slopes given by -κ/ (2 κ -1) for the full-information forecaster and -κ/ (2 κ -2) for the label-efficient forecaster. This is shown in Figure 3 for κ = 1 . 5 and κ = 2 .

We see that the observed behavior is compatible with the known asymptotics for active learning, and similar results arise when considering different values of κ . More importantly, it appears that the label-efficient forecaster optimally adapts to the underlying setting. This is remarkable, as the label-efficient forecaster does not rely on any domain knowledge. Indeed, it has no knowledge of the statistical setting, and in particular, it has no knowledge of the parameter κ , which encapsulates the difficulty of the learning task. Note that our regret bounds are too loose to provide a theoretical justification of these observations via an online-to-batch conversion, and that such theoretical analyses will only be fruitful when considering non-parametric classes of experts, for which the asymptotics of the excess risk are ω (1 / √ n ) .

7 Discussion and outlook

In this paper, we presented a set of adaptive label-efficient algorithms. These follow from a very straightforward design principle, namely, identifying the smallest possible label collection probability q t that ensures that a known worst-case expected regret bound is satisfied. This leads to simple, yet powerful, algorithms, endowed with best-of-both-worlds guarantees. We conjecture that a similar approach can be used for a broader class of prediction tasks and losses than what is considered in this paper. For instance, the results we present can be straightforwardly extended to a setting where the expert outputs take values in [0 , 1] , as long as the label sequence and forecaster prediction remain binary and take values in { 0 , 1 } . In fact, the same inductive approach can be used when y t ∈ [0 , 1] and one considers a general loss function. However, the resulting label collection probability will be significantly more complicated than that of Definition 1. An interesting side effect of our analysis is that it leads to an inductive proof of the regret bound for standard, full-information algorithms. Extending the label complexity result, and in particular connecting it with known minimax theory of active learning in statistical settings, remains an interesting avenue for future research. Finally, another intriguing direction is to extend our approach to the bandit setting. In the setting we consider in this paper, we observe the losses of all experts when observing a label. In contrast, in the bandit setting, only the loss of the selected arm would be observed for each round. This would necessitate the forecaster to incorporate more exploration in its strategy, and the analysis of a label-efficient version seems like it would be quite different from what is used in this paper, although some of the ideas may transfer.

References

Volodimir G. Vovk. Aggregating strategies. In Proc. Workshop on Computational Learning Theory (COLT) , Rochester, NY, USA, 1990.

N. Littlestone and M.K. Warmuth. The weighted majority algorithm. Information and Computation , 108(2):212-261, 1994.

N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

David Helmbold and Sandra Panizza. Some label efficient learning results. In Proc. Conference on Computational Learning Theory (COLT) , Nashville, TN, USA, July 1997.

N. Cesa-Bianchi, A. Conconi, and C. Gentile. Learning probabilistic linear-threshold classifiers via selective sampling. In Proc. Conference on Learning Theory (COLT) , Washington, DC, USA, Aug. 2003.

Bhuvesh Kumar, Jacob D Abernethy, and Venkatesh Saligrama. Activehedge: Hedge meets active learning. In Proc. International Conference on Machine Learning (ICML) , Edinburgh, Scotland, June 2012.

Ofer Dekel, Claudio Gentile, and Karthik Sridharan. Selective sampling and active learning from single and multiple teachers. The Journal of Machine Learning Research , 13(1):2655-2697, 2012.

Francesco Orabona and Nicolò Cesa-Bianchi. Better algorithms for selective sampling. In Proc. International Conference on Machine Learning (ICML) , Bellevue, WA, USA, June 2011.

Dirk van der Hoeven, Nikita Zhivotovskiy, and Nicoló Cesa-Bianchi. A regret-variance trade-off in online learning. In Proc. Conference on Neural Information Processing Systems (NeurIPS) , New Orleans, LA, USA, Nov. 2022.

N. Cesa-Bianchi, G. Lugosi, and G. Stoltz. Minimizing regret with label efficient prediction. IEEE Transactions on Information Theory , 51(6):2152-2162, 2005.

D. J. C. Mackay. Information-based objective functions for active data selection. Neural Computation , 4:698-714, 1991.

D. Cohn, Z. Ghahramani, and M. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research , pages 129-145, 1996.

Y. Freund, H. S. Seung, E. Shamir, and N. Tishby. Selective sampling using the query by committee algorithm. Machine Learning , 28(2-3):133-168, Aug. 1997.

Rui M. Castro and Robert D. Nowak. Minimax bounds for active learning. Transactions on Information Theory , 54(5):2339-2353, Apr. 2008.

N. Balcan, A. Beygelzimer, and J. Langford. Agnostic active learning. In Proc. International Conference on Machine Learning (ICML) , Pittsburgh, PA, USA, June 2006.

Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proc. International Conference on Machine Learning (ICML) , Montreal, June 2009.

R. Castro, R. Willett, and R. Nowak. Faster rates in regression via active learning. In Proc. Conference on Neural Information Processing Systems (NeurIPS) , 2005. extended version available at http://homepages.cae.wisc.edu/ ∼ rcastro/ECE-05-3.pdf.

S. Dasgupta, A. Kalai, and C. Monteleoni. Analysis of perceptron-based active learning. In Proc. Conference on Learning Theory (COLT) , Bertinoro, Italy, June 2005.

S. Dasgupta. Coarse sample complexity bounds for active learning. In Proc. Conference on Neural Information Processing (NeurIPS) , Vancouver, Canada, Dec. 2005.

M. F. Balcan, C. Berlind, A. Blum, E. Cohen, K. Patnaik, and L. Song. Active learning and bestresponse dynamics. In Proc. Conference on Neural Information Processing Systems (NeurIPS) , Montreal, Canada, Dec. 2014.

P. Awasthi, M. Balcan, and P. M. Long. The power of localization for efficiently learning linear separators with noise. In Proc. Annual ACM Symposium on Theory of Computing (STOC) , New York, NY, USA, June 2014.

A. Krause and C. Guestrin. Nonmyopic active learning of Gaussian processes: An explorationexploitation approach. In Proc. International Conference on Machine Learning (ICML) , 2007.

X. Zhu, J. Lafferty, and Z. Ghahramani. Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. In Workshop on The Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining, ICML , Washington, D.C., USA, Aug. 2003.

J. L. Williams, J. W. Fisher, and A. S. Willsky. Performance guarantees for information theoretic active inference. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS) , San Juan, Puerto Rico, Mar. 2007.

B. Chen, R. Castro, and A. Krause. Joint optimization and variable selection of high-dimensional gaussian processes. In Proc. International Conference on Machine Learning (ICML) , Edinburgh, Scotland, July 2012.

A. Epshteyn, A. Vogel, and G. DeJong. Active reinforcement learning. In Proc. International Conference on Machine Learning (ICML) , Helsinki, Finland, July 2008.

Steve Hanneke. Theory of disagreement-based active learning. Foundations and Trends in Machine Learning , 7(2-3):131-309, 2014.

Nicoló Cesa-Bianchi, Claudio Gentile, and Luca Zaniboni. Worst-case analysis of selective sampling for linear classification. Journal of Machine Learning Research , 7(44):1205-1230, 2006.

Peilin Zhao, Steven Hoi, and Jinfeng Zhuang. Active learning with expert advice. In Proc. Uncertainty in Artificial Intelligence (UAI) , Bellevue, WA, USA, July 2013.

Shuji Hao, Peiying Hu, Peilin Zhao, Steven C. H. Hoi, and Chunyan Miao. Online active learning with expert advice. ACM Trans. Knowl. Discov. Data , 12(5), June 2018.

Yining Chen, Haipeng Luo, Tengyu Ma, and Chicheng Zhang. Active online learning with hidden shifting domains. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS) , San Diego, CA, USA, Aug. 2021.

Boshuang Huang, Sudeep Salgia, and Qing Zhao. Disagreement-based active learning in online settings. IEEE Transactions on Signal Processing , 70:1947-1958, Mar. 2022.

Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning , 5(1), 2012.

Pierre Gaillard, Gilles Stoltz, and Tim van Erven. A second-order bound with excess losses. In Proc. Conference on Learning Theory (COLT) , Barcelona, Spain, June 2014.

A. B. Tsybakov. On nonparametric estimation of density level sets. Annals of Statistics , 25:948-969, 1997.

A.R. Karlin and Y. Peres. Game Theory, Alive . American Mathematical Society, 2017.

Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization, 2011.

Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proc. International Conference on Machine Learning (ICML) , Edinburgh, Scotland, UK, June 2012.

A Proof of Theorem 1

Theorem 1. Consider the exponentially weighted follow the majority forecaster with η = ∞ . Specifically, let p t = ✶ { A 1 ,t ≥ 1 / 2 } , so that ˆ y t = { A 1 ,t ≥ 1 / 2 } . Furthermore, let

For this forecaster, we have

Proof. The main idea is to proceed by induction on n . For n = 1 , the result holds trivially, regardless of the choice for q t . Now, suppose that ¯ L ( N ) t ≤ log 2 N for all values t ∈ [ n -1] , any sequence of observations and expert predictions, and any number of experts N . Based on this assumption, we will derive a bound for ¯ L ( N ) n . Let k := ∑ N i =1 ℓ i, 1 be the number of experts that make a mistake when t = 1 . Note that 0 ≤ k ≤ N -1 , as there is one perfect expert. When k < N/ 2 , the majority vote ˆ y 1 is necessarily equal to y 1 . Therefore,

where ¯ L ( N ) 2: n denotes the expected cumulative loss in rounds 2 , . . . , n with N experts. Because the internal state of the algorithm only consists of a list of experts that have made no errors so far, the task in rounds 2 , . . . , n is equivalent to a task over n -1 rounds starting with the experts that remain after round 1 . Therefore, we can apply the induction hypothesis to obtain ¯ L ( N -k ) 2: n ≤ log 2 ( N -k ) ≤ log 2 N and ¯ L ( N ) 2: n ≤ log 2 N , which justifies the last inequality. We conclude that, when k < N/ 2 , the bound holds regardless of the choice of q 1 .

On the other hand, if N/ 2 ≤ k ≤ N -1 , the forecaster incurs an error at time t = 1 . Using the induction hypothesis and an analogous reasoning as above, we find that

To ensure that the right-hand-side of (11) satisfies the desired bound, we need to select q 1 such that

which can be written equivalently as

If we do not observe a label, we do not know the value k . All we know is that, since y 1 ∈ { 0 , 1 } , we have k ∈ { ∑ N i =1 f i, 1 , N -∑ N i =1 f i, 1 } . Therefore, to ensure the induction proof works, we need

where A 1 , 1 = 1 N ∑ N i =1 f i, 1 . Note that the case k = N cannot occur as there is always a perfect expert, but to ensure that the above definition is sound, we define q 1 = 0 when A 1 , 1 ∈ { 0 , 1 } .

B Label-efficient boosted majority of leaders

Through a variation of follow the majority, a slightly better regret bound than the one in Theorem 1 can be obtained, as well as a matching lower bound. As shown by [36, Proposition 18.1.3], there exists an adversarial strategy for the environment such that any forecaster will incur at least ⌊ log 2 N ⌋ / 2 ≥ ⌊ log 4 N ⌋ errors in expectation. A matching upper bound can be obtained, when N is a power of two, by considering a forecaster that incorporates randomness in its predictions. This is referred to as the boosted majority of leaders . As shown in the following theorem, this procedure can be made label-efficient while ensuring the same expected regret bound.

Theorem 4. Consider an exponentially weighted forecaster for which η = ∞ and

and

For this forecaster, we have

Proof. The proof technique is analogous to that of Theorem 1: we use an induction argument to find a choice of q t that guarantees the desired regret bound. We will proceed by analyzing different cases, depending on the value of A 1 ,t . First, assume that A 1 , 1 ≤ 1 / 4 . If y 1 = 0 , then

Thus, taking q 1 ≥ 0 suffices. If y 1 = 1 , then

and, using the same reasoning as before, it suffices to take

An analogous reasoning applies when A 1 , 1 > 3 / 4 , so that for this case, it suffices to take

Now, consider the case 1 / 4 < A 1 , 1 ≤ 1 / 2 . If y 1 = 0 , then

which implies that

Similarly, if y 1 = 1 , we must have

Since we do not know y 1 before the decision, the only possibility is to take q 1 = 1 .

A similar reasoning applies to the case 1 / 2 < A 1 , 1 ≤ 3 / 4 . Therefore, the above relations determine the expression for q t in the theorem, while enforcing the desired regret bound.

C Proof of Theorem 2

As mentioned after Theorem 2, an analogous result holds for non-uniform weight initializations. We will state and prove this more general result below, from which Theorem 2 as stated in the main text follows as a special case.

Theorem 2 (with non-uniform weight initialization) . Consider an exponentially weighted forecaster with initial weight vector w · , 0 = ( w 1 , 0 , . . . , w N, 0 ) such that ∑ i ∈ [ N ] w i, 0 = 1 , p t = A 1 ,t and

For this forecaster, we have

where the superscript in L ( w · , 0 ) n := ∑ n t =1 ℓ (ˆ y t , y t ) makes the dependence on the initial weights explicit. In particular, for the choice of initial weights w i, 0 := 1 /N for all i ∈ [ N ] , we have

Proof. The proof strategy is similar to that used in Theorem 1. We begin by noting that at time t , the internal state of the label-efficient exponentially weighted forecaster is determined by the weight vector w · ,t -1 = ( w 1 ,t -1 , . . . , w N,t -1 ) . Therefore, it suffices to focus on the requirements for q 1 for an arbitrary weight vector. As in Theorem 1, we proceed by induction on n .

For n = 1 , the theorem statement is trivially true, as this algorithm coincides with the ordinary exponentially weighted forecaster (also, the right-hand-side of (12) is bounded from below by 1 / 2 ). Proceeding by induction in n , suppose (12) holds for 1 , . . . , n -1 outcomes. Let ¯ L ( w · , 0 ) n | 1 denote the expected cumulative loss of the forecaster given ( y 1 , { f i, 1 } N i =1 ) , i.e., the true label and the expert predictions for time t = 1 :

Then, we have

In order to prove the desired result, it is enough to show that

Here, we let L i, 2: n := ∑ n t =2 ℓ i,t and let i ′ denote the arg min of the right-hand side. Now, using the induction hypothesis, we obtain

In the last step, we used the fact that since the upper bound holds for the minimum i , it holds for i ′ in particular. To ensure that the bound in the theorem holds, it is thus sufficient to select A 1 , 1 such that it satisfies

Notice that, after the first observation, we are back in a situation similar to that at time t = 1 , but possibly with a different weight vector. Specifically, for i ∈ [ N ] ,

It is important at this point that w i, 1 depends on the choice q 1 , so we cannot simply solve the above equation for q 1 . To proceed, it is easier to consider the two possible values of y 1 separately.

Case y 1 = 0 : Note that

Therefore, it suffices to have

Case y 1 = 1 : Similarly as above,

Therefore, it suffices to have

As we do not know the values of y 1 when computing q 1 , we must simultaneously satisfy (13) and (14). Nevertheless, these two conditions involve only η and A 1 , 1 . Thus, we can identify the range of values that q 1 can take, as a function of η and A 1 , 1 , while still ensuring the desired regret bound. Specifically, we require that q 1 ≥ q ∗ 1 , where

At this point, it might be unclear if q ∗ 1 is well defined, namely, if there always exists q ∈ [0 , 1] satisfying both (13) and (14). This is indeed the case, as shown in Lemma 1. By noting that E [ ¯ L ( w · , 0 ) i,n | 1 ] = ¯ L ( w · , 0 ) i,n , we have completed the induction step.

As stated at the beginning of the proof, looking at q 1 suffices to determine the general requirements for q t , concluding the proof. The statement given in (7) follows from instantiating the general result with uniform initial weights.

D Proof of Lemma 1

Lemma 1. For all η > 0 and x ∈ [0 , 1] , we have

Proof. Let B ∼ Ber ( x ) , with x ∈ [0 , 1] . Note that E [ e -ηB ] = (1 -x ) + xe -η . Note also that, by [3, Lem. A.1] we have ln E [ e -ηB ] ≤ -ηx + η 2 / 8 . Putting these two facts together we conclude that

Therefore, the point 1 is always contained in Definition 1, concluding the proof.

E Proof of Lemma 2

Lemma 2. For any x ∈ [0 , 1] , we have

Furthermore, for any η > 0 and x ∈ [0 , 1] ,

Proof. As already shown in Lemma 1, we know that q ∗ ( x, η ) ≤ 1 . Let η > 0 be arbitrary. Note that q ∗ ( x, η ) needs to be a solution in q of the following equation:

Note that lim η → 0 g ( η ) = 0 , so we can extend the definition of g to 0 by continuity. Specifically,

We proceed by using a Taylor expansion of g ( η ) around 0 . Tedious, but straightforward computations yields

where ξ > 0 . In conclusion,

where ξ ∈ [0 , η ] . At this point we can examine the structure of the solution of (15) when η → 0 . Note that q ∗ ( x, η ) necessarily satisfies

implying that q ∗ ( x, η ) → 4 x (1 -x ) as η → 0 , proving the first statement in the lemma.

For the second statement in the lemma, one needs to more carefully control the error term g ′′′ ( ξ ) . We begin by noting that τ ∈ [0 , x ] . This implies that -τ +3 τ 2 -2 τ 3 ≤ x (1 -x ) (this can be checked by algebraic manipulation 1 ). Therefore, q ∗ ( x, η ) ≤ q , where q is the solution of

This is a simple quadratic equation in q , yielding the solution

Although the above expression is a valid upper bound on q ∗ ( x, η ) , it is not a very convenient one. A more convenient upper bound can be obtained by noting that

Thus, we have q ≤ 4 x (1 -x ) + η/ 3 , concluding the proof.

1 It suffices to check that the solutions in τ of -1 + 3 τ -2 τ 2 ≤ 1 -x , if they exist, satisfy τ ∈ [0 , x ] .

F Proof of Theorem 3

In the proof of Theorem 3, we will require the following anytime version of Freedman's inequality:

Lemma 3. Let X 1 , . . . , X n be a martingale difference sequence with respect to some filtration F 1 ⊂ · · · ⊂ F n and with | X t | ≤ b for all t almost surely. Let Σ 2 t = ∑ t j =1 E [ X 2 j |F j -1 ] . Then, for any δ < 1 /e and n ≥ 4 ,

A proof of this result for the special case that F t = σ ( X 1 , . . . , X t ) can be found in [37, Lemma 3], which is an extended version of [38]. Their proof goes through unchanged for general filtrations.

We are now ready to prove Theorem 3. As aforementioned, we will prove the result under a more general condition than given in the theorem statement. Specifically, instead of assuming that the best expert in expectation is apparent from the first round, we only require the best expert to emerge after a time τ ∗ . This condition is given in a precise form in (17). Clearly, the assumption stated in the main text implies that the condition in (17) holds with τ ∗ = 0 .

Theorem 3 (with the best expert emerging after a time τ ∗ ) . Consider the label-efficient exponentially weighted forecaster from Theorem 2 with q t = min(4 A 1 ,t (1 -A 1 ,t ) + η/ 3 , 1) and any η > 0 . Define τ as

̸

Suppose that there exists a single best expert i ∗ and a time τ ∗ ≤ τ such that, for all i = i ∗ ,

Then, for any n ≥ 4 , the expected label complexity is at most

Proof. Recall that the estimated losses are ˜ ℓ i,t = ℓ i,t Z t /q t . Let ˜ Λ i t = ∑ t j =1 ˜ l i,j -˜ l i ∗ ,j denote the cumulative estimated loss relative to that of the best expert, and let ˜ Λ min t = min i = i ∗ ˜ Λ i t .

̸

Our argument separates the analysis in two regimes: Regime 1, where t ≤ τ , and Regime 2, where t > τ . The specific choice of τ given above arises naturally later in the analysis. Regime 1, in which labels will be collected frequently, is expected to be relatively short, so we simply upper-bound S τ ≤ τ . It then remains to bound the number of collected labels in Regime 2. To this end, we need τ to be chosen such that

with probability at least 1 -δ 1 , where δ 1 ∈ (0 , 1] will be fixed later. Let E denote the event that (18) holds. It then follows that the expected number of labels collected in Regime 2 is at most

It remains to verify that the choice of τ in (16) leads to δ 1 being sufficiently small. To this end, let

denote the weighted proportion of experts that agrees with the best expert at time t . Then

̸

Note that A 1 ,t ∈ { A ∗ t +1 , 1 -A ∗ t +1 } . Consequently,

̸

The desired condition from (18) is therefore satisfied if, for all i = i ∗ ,

̸

To study the evolution of ˜ Λ i t , we use a martingale argument. Consider any fixed i = i ∗ , and define the martingale difference sequence

Without loss of generality, we may assume that η ≤ 3 , because otherwise (19) holds trivially for any pair of τ and δ 1 . Then q t ≥ η/ 3 , | X t | ≤ 2 /q t ≤ 6 /η , and

Hence, by Lemma 3, we have

Using Assumption (17), it follows that

̸

with probability at least 1 -(ln( n ) δ 2 ) for any δ 2 ∈ (0 , 1 /e ] . By taking δ 2 = min { δ 1 / ( N ln n ) , 1 /e } and applying the union bound, we can make (22) hold for all i = i ∗ simultaneously with probability at least 1 -δ 1 . A sufficient condition for (22) to imply (20) is then to take

matching the definition in (16). We prove this claim in Lemma 4 below. Note that if our choice of τ > n , then (19) still holds trivially, because S n ≤ n . Evaluating (19) with the given choice of τ and taking δ 1 = η (assuming η < 1 , since (19) holds trivially if δ 1 ≥ 1 ), we arrive at the following bound:

thus completing the proof.

Then, with we have

Proof. We will consider the two possible outcomes of the maximum separately. First, we consider the case where the first term is the maximum. Then, we need to find τ such that for t ≥ τ ,

A straightforward calculation shows that this is satisfied for

Since ( a + b ) 2 ≤ 2 a 2 +2 b 2 for a, b > 0 , this is satisfied given the simpler condition

Next, we assume that the second term is the maximum. Then, we have

where we used the assumption that t ≥ τ . Thus, for this case, the desired condition holds with the specified τ . Therefore, with the specified τ , the desired statement holds for t = τ, . . . , n -1 .

G Experimental details

In this section, we describe the simulation study in Section 6 in detail. The full code, which can be executed in less than one hour on an M1 processor, is provided in the supplementary material.

We consider a sequential prediction problem with n = 50000 total rounds and N = 225 experts. The number of experts is chosen to be higher than √ n and odd. Then, we repeat the following simulation for 500 independent runs. First, we generate n independent features { X i } i ∈ [ n ] from the uniform distribution on [0 , 1] . Then, for each feature X i , we randomly generate a label Y i , where the probability P ( Y i = 1 | X i = x ) = ζ ( x ) , where

Note that the optimal prediction of the label is simply ✶ { x ≥ 1 / 2 } . We run simulations both for κ = 1 . 5 and κ = 2 .

We set the experts to be threshold classifiers, with thresholds uniformly spaced across [0 , 1] . Specifically, for all i ∈ [ N ] and t ∈ [ n ] ,

In order to optimize the regret bound in (7), we set η = √ 8 ln( N ) /n . For each time t ∈ [ n ] , we consider two different weight vectors: { w P i,t } i ∈ [ N ] , corresponding to the passive, full-information forecaster, and { w A i,t } i ∈ [ N ] , corresponding to the active, label-efficient forecaster. The weight for each expert for both forecasters is uniformly initialized as w P i, 0 = w A i, 0 = 1 /N .

Then, for each timestep t ∈ [ n ] , we proceed as follows. First, we compute the probability of prediction the label 1 for each forecaster, given by p P t for the full-information forecaster and p A t for the label-efficient forecaster, as

where the expert predictions are computed based on X t . Here, the weighted proportion of experts that predict the label 1 are given by A P 1 ,t for the full-information forecaster and A A 1 ,t for the labelefficient forecaster. On the basis of this, each forecaster issues a prediction for the label, given by ˆ y P t ∼ Ber ( p P t ) for the full-information forecaster and ˆ y A t ∼ Ber ( p A t ) for the label-efficient one.

Finally, the weights for each forecaster are updated as follows. For the full-information forecaster, the weight assigned to each expert is updated as

where ℓ i,t = ✶ { f i,t = Y t } . In order to determine whether the label-efficient forecaster observes a label or not, we compute q t = min(4 A A 1 ,t (1 -A A 1 ,t ) -η/ 3 , 1) and generate Z t ∼ Ber ( q t ) . Then, the weights of the label-efficient forecaster are updated as

Hence, if Z t = 0 , the weights of the label-efficient forecaster are not updated, since the label is not observed.

After all n rounds have been completed, we compute the regret of the full-information forecaster R P n and the label-efficient forecaster R A n as

̸

̸

̸

̸

Here, for each t ∈ [ n ] , the optimal prediction is given by y ∗ t = ✶ { X t ≥ 1 / 2 } . Furthermore, we compute the label complexity S n as

The results of these simulations are presented in Figure 2 and 3.

̸

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an adaptive label-efficient forecasting technique for online binary prediction with expert advice. The proposed approach implements a label querying probability that is a function of the observed scenario, rather than based on pessimistic conditions. This enables the method to adapt, i.e., have lower label complexity, to benign environments while remaining robust to adversarial ones, unlike prior approaches in this label-efficient forecasting. Sharp analyses of the regret and label complexity and results on synthetic scenarios are provided in support of the method.\n\nSTRENGTHS:\n* The paper is very well-written and organized.\n* The introduced algorithm is novel and intuitive. The regret and label complexity analyses of the approach seem sound.\n* The method remedies the shortcoming (lack of adaptivity) of prior label-efficient prediction approaches. This enables it to query fewer labels in benign scenarios while remaining robust to adversarial ones.\n* The authors present empirical evaluations that demonstrate the effectiveness of the method.\n\nWEAKNESSES:\n* The method only applies to binary prediction tasks with zero-one loss.\n* Additional details on prior work on label-efficient prediction would be helpful in contextualizing the benefit (adaptivity) of the proposed approach.\n\nQUESTIONS:\n1. The authors mention that the same inductive approach can be used for general loss functions and $y_t \\in [0,1]$, but that the expression for the selective sampling probability would be really complicated. Do the authors conjecture that an efficiently-computable upper bound for the complicated function can be found as was done for the binary zero-one loss case?\n2. The authors mention active learning as a possible application in the introduction. Could this method (or its generalization) be applied to, e.g., a simple active learning scenario with a ResNet on CIFAR10?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a binary prediction game on $0-1$ loss, it proposed efficient sampling scheme via an modification of the exponentiated weight forecaster, which selectively acquire labels $y_t$ based on $Ber(q_t)$, where the design of $q_t$ is correlated to the disagreement among experts’ predictions at each round for the exponentiated weight forecaster. \n\nUltimately, the proposed algorithms achieves the regret of the exponentiated weight forecaster $O(\\ln N \\sqrt{ n} )$ over time horizon $n$ and $N$ number of experts, the design labelling acquisition parameter $q_t$ resultant to a labelling complexity of $O( \\frac{\\sqrt{n}}{\\Delta^2} )$, where $0 < \\Delta \\le \\mathbb{E} [\\ell_{t,i} - \\ell_{t,i^{\\ast}}], \\forall t \\in [n], i \\neq i^{\\ast}$ represents the lower bound expected loss gap comparing to the optimal expert with index $i^{\\ast}$, which also signifies the difficulty of identifying the optimal experts.\n\nSTRENGTHS:\nThe paper is well written and easy to follow. It first introduces the intuition of sampling parameter $q_t$ if there is a perfect expert, then extended to general case. The paper also provided an graphical illustration on the lower bound of $q_t$ which matches with expectation.\n\nThe result is novel comparing to previous result in two folds. (1) There is no assumption on how $y_t$ is generated. The proposed algorithm is able to attain the $O(\\ln N \\sqrt{n})$ regret without less labels. (2) In contrast to previous work on sampling by disagreement, the label complexity can be quantified as $O(\\frac{\\sqrt{n}}{ \\Delta^2})$. \n\n$q_t$ is easy to compute, numerical experiment with respect to time horizon $n$ shows the expected regret and expected number of labels which matches with theoretical results. Experiment with respect to number of label (number of weights update) shows labelling efficient algorithm proposed in the paper matches the minimax rate in active learning asymptotically.\n\nWEAKNESSES:\nIt seems that the assumption $0 > \\Delta \\ge \\mathbb{E} [\\ell_{t,i} - \\ell_{t,i^{\\ast}}], \\forall t \\in [n], i \\neq i^{\\ast}$ is required only for bounding the labelling complexity, ( in order to track how $q_{t+1}$ evolves in line 493), without this assumption the regret still holds. \n\nI am a bit concern such assumption is very strong, it generally asks the best expert $i^{\\ast}$ is wining over every other expert at every round. In addition, the assumption is $\\ell_{t,i} \\in \\{ 0, 1\\}$, if we are at a specific iteration $t$ where $\\ell_{t,i^{\\ast}} = 1$, then what values  can $\\ell_{t,i}, \\forall i \\neq i^{\\ast}$ take in order to satisfy the condition for a strictly positive $\\Delta$? \n\nWhether labelling complexity can be bounded without this assumption?\n\nQUESTIONS:\nAppendix C: \nFrom line 437 to line 438, whether intermediate steps could be further elaborated. I am concerned about the proof, which seems primarily confused on the relationship of combining minimum values: $\\min_i (a_i + b_i)$ VS $\\min_i(a_i)+ \\min(b_i)$. \n\nMy understanding is that from the last equation line of line 437, we have \n$$ L_{n |1}^{w_{.,0} }  \\le \\frac{n \\eta }{ 8 }  \\underbrace{  - \\frac{\\eta}{8 } + A_{1,1} +  (1 - 2 A_{1,1})y_1  + \\mathbb{E} \\left[ \\min_{i} \\left( L_{i, n} + \\frac{\\ln w_{i,0}}{\\eta} + \\frac{q_1}{\\eta} ( \\ln w_{i,0} - \\ln w_{i,1} )  - \\ell_{i,1}  \\right)   \\right]}_{A} $$\n\nTo show the desired result claimed in the theorem, we aim to show \n$$ A \\le \\mathbb{E} \\left[ \\min_{i} \\left( L_{i,n} - \\frac{\\ln w_{i,0}}{\\eta} \\right) \\right] $$\n\nThat is to show \n$$  A_{1,1} +  (1 - 2 A_{1,1})y_1  + \\mathbb{E} \\left[ \\min_{i} \\left( L_{i, n} + \\frac{\\ln w_{i,0}}{\\eta} + \\frac{q_1}{\\eta} ( \\ln w_{i,0} - \\ln w_{i,1} )  - \\ell_{i,1}  \\right)   \\right]  \\le  \\frac{\\eta}{8 } + \\mathbb{E} \\left[ \\min_{i} \\left( L_{i,n} - \\frac{\\ln w_{i,0}}{\\eta} \\right) \\right] $$\n\nFrom the claim in line 438, it seems we need to show above inequality should hold even without expectation, that is:\n $$  A_{1,1} +  (1 - 2 A_{1,1})y_1  + \\min_{i} \\left( L_{i, n} + \\frac{\\ln w_{i,0}}{\\eta} + \\frac{q_1}{\\eta} ( \\ln w_{i,0} - \\ln w_{i,1} )  - \\ell_{i,1}  \\right) -  \\min_{i} \\left( L_{i,n} - \\frac{\\ln w_{i,0}}{\\eta} \\right)      \\le  \\frac{\\eta}{8 }  $$\n\nIt seems the paper used \n$$\\min_{i} \\left( L_{i, n} + \\frac{\\ln w_{i,0}}{\\eta} + \\frac{q_1}{\\eta} ( \\ln w_{i,0} - \\ln w_{i,1} )  - \\ell_{i,1}  \\right) -  \\min_{i} \\left( L_{i,n} - \\frac{\\ln w_{i,0}}{\\eta} \\right) \\le \\min_{i} \\left(  \\frac{q_1}{\\eta} ( \\ln w_{i,0} - \\ln w_{i,1} )  - \\ell_{i,1}  \\right) $$\nwhich doesn't seem to be true. These intermediate steps are just my personal guess. I wonder whether details to understand those two lines in the manuscript can be further explained.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an interesting novel approach to prediction with expert advise. In the standard prediction with expert advise setup, the learner receives experts' predictions, commits to its own and then sees the true outcome as produces by the (possibly adversarial) nature. Suppose that obtaining the true outcome is costly; do we really need to do this all the time? Clearly, if all (or most) experts agreed on the same prediction, the value of the true outcome for adjusting our trust in them is negligible; in the standard weight-based algorithms with multiplicative update the contribution of this round will simply be eliminated by normalisation.\n\nThe paper takes an algorithm from Cesa-Bianchi and Lugosi (exponential weighting with fixed $\\eta$) and shows that its regret bound stands as it is if the true outcome is requested with certain probability.\n\nIt is then shown that the expectation of the number of outcomes actually requested is upper bounded by $3\\eta T + O(\\log\\log T)$, so for small $\\eta$ there is linear improvement in the number of requested outcomes. If $\\eta$ is chosen to minimise the regret bound using prior knowledge of $T$, we take $\\eta\\propto 1/\\sqrt{T}$ and thus the bound reduces to $O(\\sqrt{T})$. This results holds under some conditions on experts' behaviour  though and they seem restrictive.\n\nSTRENGTHS:\nI think this is an interesting take on the well-known problem and should be published.\n\nWEAKNESSES:\nNo obvious weaknesses.\n\nQUESTIONS:\nThe authors chose to present the results in the probabilistic setup. Typically predicting $0$ or $1$ in a probabilistic setup is equivalent to predicting $\\gamma\\in [0,1]$ under a deterministic setup; the later seems more important to me (this is my judgement of course). Will the results of the paper stand?\n\nI think the final version would benefit from a better discussion of the $O(T)$ lower bound, which is mentioned but not discussed in detail.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the PEA problem in the context of online binary classification where the cost of obtaining labels for streaming data is high, necessitating selective label collection adaptively. To this end, the authors introduce a carefully designed label collection strategy based on the classical Hedge algorithm. The resultant label-efficient forecaster has a best-of-both-worlds theoretical guarantee. The authors further demonstrate that regret of their label-efficient forecaster asymptotically reaches the minimax rates of pool-based active learning.\n\nSTRENGTHS:\n* The paper is well-structured and easy to comprehend.\n* The theoretical analysis provided in the paper appears to be solid and sound.\n\nWEAKNESSES:\n* The primary conclusion of the paper, Theorem 3, relies on a core assumption that there exists a unique optimal expert whose expected loss in each round surpasses all other experts by a specific margin $\\Delta$. Is this assumption too strong? Are there real-world application scenarios that satisfy such an assumption? (To my knowledge, this assumption has only been used in the COLT'14 paper: A Second-order Bound with Excess Losses).\n\n* Does this specific PEA setting investigated in the paper relates to the bandit setting? Both are concerned with identifying the optimal expert (arm). How then do the setting and techniques used in this paper differ or relate to those in bandit scenarios? Does the problem studied in this paper present novel challenges in comparison to the bandit setting?\n\n\n* The paper states that the online prediction setting is similar to streaming active learning. If this is the case, should the numerical experiment compare the regret convergence rate of the proposed algorithm with that of the streaming active learning algorithm?\n\n\n* Some writing aspects could be improved:\n\n  * In line 227, the symbol $\\tilde l$ is undefined.\n  * In line 264, the regret bound (7) is termed \"pessimistic\" without explanation.\n\nQUESTIONS:\nSee Weaknesses above.\n\nLIMITATIONS:\nThe paper should further clarify the method and theory part, see Weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}