SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then, for any  $n \geq 4$ , the expected label complexity is at most
$$
\mathbb {E} \left[ S _ {n} \right] \leq \frac {5 0}{\eta \Delta^ {2}} \ln \left(\frac {N \ln n}{\eta}\right) + 3 \eta n + 1. \tag {9}
$$
Proof sketch. Initially, the sampling probability  $q_{t}$  is large, but as we collect more labels, it will become detectable that one of the experts is better than the others. As this happens,  $q_{t}$  will tend to decrease until it (nearly) reaches its minimum value  $\eta / 3$ . We therefore divide the forecasting process into time  $t \leq \tau$  and  $t > \tau$ . With a suitable choice of  $\tau \approx 1 / (\eta \Delta^{2})$  (up to logarithmic factors), we can guarantee that the sampling probability is at most  $q_{t} \leq 4\eta / 3$  for all  $t > \tau$  with sufficiently high probability. This is shown by controlling the deviations of the cumulative importance-weighted loss differences  $\tilde{\Lambda}_{t}^{i} = \sum_{j=1}^{t} (l_{i,j} - l_{i^*,j}) / q_{j}$  for  $i \neq i^*$  from their expected values by using an anytime version of Freedman's inequality. With this, we simply upper bound the label complexity for the first  $\tau$  rounds by  $\tau$ , and over the remaining rounds, the expected number of collected labels is roughly  $\eta (n - \tau) \leq \eta n$ . This leads to a total expected label complexity of  $1 / (\eta \Delta^{2}) + \eta n$ , up to logarithmic factors. The full proof is deferred to Appendix F.
As mentioned earlier, the learning rate optimizing the regret bound (7) is  $\eta = \sqrt{8\ln(N) / n}$ . For this particular choice, the label complexity in (9) is roughly  $\sqrt{n} /\Delta^2$ , up to constants and logarithmic factors. We have thus established that the label-efficient forecaster achieves the best of both worlds: it queries sufficiently many labels in the worst case to obtain optimal regret guarantees, while simultaneously querying much fewer labels in more benign settings. It is interesting to note that the label complexity dependence of  $1 / \Delta^{2}$  on  $\Delta$  is less benign than the dependence of the regret bound from, e.g., [34, Thm. 11], which is  $1 / \Delta$ . The underlying reason for this is that, while the two are similar, the label complexity is not directly comparable to the regret. In particular, the label complexity has much higher variance.
The bound of Theorem 3 relies on setting the sampling probability  $q_{t}$  to be the upper bound on  $q_{t}^{*}$  from Lemma 2. This bound is clearly loose when  $q_{t}^{*}$  is approximately zero, and one may wonder if the label complexity of the algorithm would be radically smaller when using a forecaster for which  $q_{t} = q_{t}^{*}$  instead. With the choice  $\eta = \sqrt{8\ln(N) / n}$ , which optimizes the bound in (7), it seems unlikely that the label complexity will substantially change, as numerical experiments suggest that the label complexity attained with  $q_{t}$  set as  $q_{t}^{*}$  or the corresponding upper bound from (8) appear to be within a constant factor. That being said, for larger values of  $\eta$ , the impact of using the upper bound in (8) is likely much more dramatic.
To further assess the behavior of the label-efficient forecaster from Theorem 2, we consider a classical active learning scenario in a batch setting, for which there are known minimax rates for the risk under both active and passive learning paradigms. We will set the sampling probability to be
$$
q _ {t} = \min  \left(4 A _ {1, t} \left(1 - A _ {1, t}\right) + \eta / 3, 1\right).
$$
Let  $D_{n} = ((X_{t},Y_{t}))_{t = 1}^{n}$  be an ordered sequence of independent and identically distributed pairs of random variables with joint distribution  $\bar{D}$ . The first entry of  $(X_{i},Y_{i})$  represents a feature, and the second entry is the corresponding label. The goal is to predict the label  $Y_{i}\in \{0,1\}$  based on the feature  $X_{i}$ . Specifically, we want to identify a map  $(x,D_n)\mapsto \hat{g}_n(x,D_n)\in \{0,1\}$  such that, for a pair  $(X,Y)\sim D$  that is drawn independently from  $D_{n}$ , we have small (zero-one loss) expected risk
$$
\operatorname {R i s k} \left(\hat {g} _ {n}\right) := \mathbb {P} \left(\hat {g} _ {n} \left(X, D _ {n}\right) \neq Y\right).
$$
Concretely, we consider the following scenario, inspired by the results in [14]. Let the features  $X_{i}$  be uniformly distributed in  $[0, 1]$ , and  $Y_{i} \in \{0, 1\}$  be such that  $\mathbb{P}(Y_{i} = 1 | X_{i} = x) = \zeta(x)$ . Specifically, let  $\tau_{0} \in [0, 1]$  such that  $\zeta(x) \geq 1/2$  when  $x \geq \tau_{0}$  and  $\zeta(x) \leq 1/2$  otherwise. Furthermore, assume that for all  $x \in [0, 1]$ ,  $\zeta(x)$  satisfies
$$
c | x - \tau_ {0} | ^ {\kappa - 1} \leq | \zeta (x) - 1 / 2 | \leq C | x - \tau_ {0} | ^ {\kappa - 1},
$$
for some  $c, C > 0$  and  $\kappa > 1$ . If  $\tau_0$  is known, the optimal classifier is simply  $g^*(x) = \mathbb{1}\{x \geq \tau_0\}$ . The minimum achievable excess risk when learning  $\hat{g}_n$  from  $D_n$  in this type of problems has been studied in, e.g., [14, 35]. For this setting, it is known that
$$
\inf_{\hat{g}_{n}}\sup_{\tau_{0}\in [0,1]}\operatorname {Rik}(\hat{g}_{n}) - \operatorname {Rik}(g^{*})\asymp n^{\frac{-\kappa}{2\kappa - 1}},
$$
as  $n\to \infty$ . However, rather than the classical supervised learning setting above, we can instead consider active learning procedures. Specifically, consider a sequential learner that can generate feature-queries  $X_{i}^{\prime}$  and sample a corresponding label  $Y_{i}^{\prime}$ , such that  $\mathbb{P}(Y_i' = 1|X_i' = x) = \zeta (x)$ . This is often referred to as pool-based active learning. At time  $t$ , the learner can choose  $X_{t}^{\prime}$  as a function of the past  $((X_j',Y_j'))_{j = 1}^{t - 1}$  according to a (possibly random) sampling strategy  $\mathcal{A}_n$ . This extra flexibility allows the learner to carefully select informative examples to guide the learning process. Similarly to the passive learning setting, the ultimate goal is to identify a prediction rule  $(x,D_n')\mapsto \hat{g}_n^A (x,D_n')\in \{0,1\}$ , where  $D_n' = ((X_t',Y_t'))_{t = 1}^n$ . In [14], it is shown that for this active learning setting, the minimax rates are also known, and given by
$$
\inf_{\hat{g}_{n}^{A},\mathcal{A}_{n}}\sup_{\tau_{0}\in [0,1]}\operatorname {Risk}(\hat{g}_{n}^{A}) - \operatorname {Risk}(g^{*})\asymp n^{\frac{-\kappa}{2\kappa - 2}},
$$
(a)  $\kappa = 2$
(b)  $\kappa = 1.5$
(c)  $\kappa = 2$
(d)  $\kappa = 1.5$
Figure 2: Numerical results for expected regret and label complexity when  $n = 50000$  and  $N = 225$ . Panels (a) and (b) depict the expected regret  $\mathbb{E}[R_t]$  as a function of  $t$ , for  $\kappa = 2$  and  $\kappa = 1.5$  respectively. Panels (c) and (d) depict the expected label complexity of the label-efficient forecaster,  $\mathbb{E}[S_t]$ , as a function of  $t$  for  $\kappa = 2$  and  $\kappa = 1.5$  respectively. The expectations were estimated from 500 independent realizations of the process and the shaded areas indicate the corresponding pointwise  $95\%$  confidence intervals.
as  $n \to \infty$ . This shows that there are potentially massive gains for active learning, particularly when  $\kappa$  is close to 1. A natural question is whether similar conclusions hold for streaming active learning. In this setting, instead of selecting which example  $X_{i}^{\prime}$  to query, the learner observes the features  $(X_{1},\ldots ,X_{n})$  sequentially, and decides at each time  $t$  whether or not it should query the corresponding label. This is analogous to the online prediction setting discussed in this paper.
We now study this setting numerically. For the simulations, we use the specific choice
$$
\zeta (x) = \frac {1}{2} + \frac {1}{2} \operatorname {s i g n} \left(x - \tau_ {0}\right) | x - \tau_ {0} | ^ {\kappa - 1},
$$
to generate sequences  $(Y_{1},\ldots ,Y_{n})$ , based on a sequence of features  $(X_{1},\ldots ,X_{n})$  sampled from the uniform distribution on  $[0,1]$ . Furthermore, we consider the class of  $N$  experts such that
$$
f _ {i, t} = \mathbb {1} \left\{X _ {t} \geq \frac {i - 1}{N - 1} \right\},
$$
with  $i\in [N]$  and  $t\in [n]$
6.1 Expected regret and label complexity
In the simulations, we set  $\tau_0 = 1/2$  and  $N = \lceil \sqrt{n} \rceil + \mathbb{1}\left\{\lceil \sqrt{n} \rceil \text{ is even}\right\}$ . This choice enforces that  $N$  is odd, ensuring the optimal classifier is one of the experts. Throughout, we set  $\eta = \sqrt{8\ln(N)/n}$ , which minimizes the regret bound (7). First, we investigate the expected regret relative to the optimal prediction rule for the label-efficient exponentially weighted forecaster with  $q_t$  given by (8), and compare it with the corresponding regret for the full-information forecaster that collects all labels. Specifically, the regret at time  $t$  of a forecaster that predicts  $\{\hat{Y}_j\}_{j=1}^n$  is given by
$$
\mathbb {E} \left[ R _ {t} \right] = \sum_ {j = 1} ^ {t} \mathbb {E} \left[ \ell \left(\hat {Y} _ {t}, Y _ {t}\right) - \ell \left(g ^ {*} \left(X _ {t}\right), Y _ {t}\right) \right].
$$
(b)
$\kappa = 1.5$
Figure 3: Numerical results for the normalized regret as a function of the expected label complexity when  $n = 50000$  and  $N = 225$ . The straight dotted lines are displayed for comparison, and have slopes given by  $-\kappa / (2\kappa - 1)$  (full information) and  $-\kappa / (2\kappa - 2)$  (label efficient). The expectations were estimated from 500 independent realizations of the process and the shaded areas indicate the corresponding pointwise  $95\%$  confidence intervals.
Furthermore, to study the potential reduction in the number of collected labels, we also evaluate the expected label complexity  $\mathbb{E}[S_t]$  of the label-efficient forecaster. To approximate the expectations above, we use Monte-Carlo averaging with 500 independent realizations. Further experimental details are given in Appendix G. The results are shown in Figure 2. We see that the regret is comparable for the full-information and label-efficient forecasters, albeit slightly higher for the latter. Since  $P(g^{*}(X)\neq Y) = \frac{1}{2} -\frac{1}{\kappa 2^{\kappa}}$  , the expected cumulative loss of the optimal classifier grows linearly with  $t$  . For instance, when  $\kappa = 2$  , we have  $\sum_{j = 1}^{t}\mathbb{E}[\ell (g^{*}(X_{t}),Y_{t})] = 3t / 8$  . Hence, the regret relative to the best expert is much smaller than the pessimistic (i.e., worst-case for adversarial environments) bound in (7). We also observe that the expected label complexity grows sub-linearly with  $t$  , as expected, and that  $\mathbb{E}[S_n]\ll n$  , demonstrating that a good prediction rule can be learned with relatively few labels. When  $\kappa = 1.5$  , the number of collected labels is significantly smaller than when  $\kappa = 2$  . This is in line with the known minimax rates for active learning from [14]. To further examine this connection, we now turn to normalized regret.
6.2 Normalized regret relative to the number of samples
To relate the results of the label-efficient forecaster with known minimax rates for active learning, we investigate the expected regret normalized by the number of samples. Specifically, let
$$
r (t) = \frac {1}{t} \mathbb {E} [ R _ {t} ] = \frac {1}{t} \sum_ {j = 1} ^ {t} \mathbb {E} [ \ell (\hat {Y} _ {t}, Y _ {t}) - \ell (g ^ {*} (X _ {t}), Y _ {t}) ].
$$
For the full-information forecaster, we expect that  $r(t) \asymp t^{-\kappa/(2\kappa-1)}$  as  $t \to \infty$ . The same holds for the label-efficient forecaster, but in this case, the relation between  $r(t)$  and the expected number of collected labels  $\mathbb{E}[S_t]$  is more interesting. If the label-efficient forecaster performs optimally, we expect  $r(t) \asymp \mathbb{E}[S_t]^{-\kappa/(2\kappa-2)}$  as  $t \to \infty$ . To examine this, we plot  $r(t)$  against  $\mathbb{E}[S_t]$  (which equals  $t$  for the full-information forecaster) in logarithmic scales, so the expected asymptotic behavior corresponds to a linear decay with slopes given by  $-\kappa/(2\kappa-1)$  for the full-information forecaster and  $-\kappa/(2\kappa-2)$  for the label-efficient forecaster. This is shown in Figure 3 for  $\kappa = 1.5$  and  $\kappa = 2$ .
We see that the observed behavior is compatible with the known asymptotics for active learning, and similar results arise when considering different values of  $\kappa$ . More importantly, it appears that the label-efficient forecaster optimally adapts to the underlying setting. This is remarkable, as the label-efficient forecaster does not rely on any domain knowledge. Indeed, it has no knowledge of the statistical setting, and in particular, it has no knowledge of the parameter  $\kappa$ , which encapsulates the difficulty of the learning task. Note that our regret bounds are too loose to provide a theoretical justification of these observations via an online-to-batch conversion, and that such theoretical analyses will only be fruitful when considering non-parametric classes of experts, for which the asymptotics of the excess risk are  $\omega(1/\sqrt{n})$ .
7 Discussion and outlook
In this paper, we presented a set of adaptive label-efficient algorithms. These follow from a very straightforward design principle, namely, identifying the smallest possible label collection probability  $q_{t}$  that ensures that a known worst-case expected regret bound is satisfied. This leads to simple, yet powerful, algorithms, endowed with best-of-both-worlds guarantees. We conjecture that a similar approach can be used for a broader class of prediction tasks and losses than what is considered in this paper. For instance, the results we present can be straightforwardly extended to a setting where the expert outputs take values in [0, 1], as long as the label sequence and forecaster prediction remain binary and take values in  $\{0, 1\}$ . In fact, the same inductive approach can be used when  $y_{t} \in [0, 1]$  and one considers a general loss function. However, the resulting label collection probability will be significantly more complicated than that of Definition 1. An interesting side effect of our analysis is that it leads to an inductive proof of the regret bound for standard, full-information algorithms. Extending the label complexity result, and in particular connecting it with known minimax theory of active learning in statistical settings, remains an interesting avenue for future research. Finally, another intriguing direction is to extend our approach to the bandit setting. In the setting we consider in this paper, we observe the losses of all experts when observing a label. In contrast, in the bandit setting, only the loss of the selected arm would be observed for each round. This would necessitate the forecaster to incorporate more exploration in its strategy, and the analysis of a label-efficient version seems like it would be quite different from what is used in this paper, although some of the ideas may transfer.
The authors would like to thank Wojciech Kotlowski, Gábor Lugosi, and Menno van Eersel for fruitful discussions that contributed to this work. The algorithmic ideas underlying this work were developed when R. Castro was a research fellow at the University of Wisconsin – Madison. This work was partly done while F. Hellström was visiting the Eindhoven University of Technology and the University of Amsterdam supported by EURANDOM and a STAR visitor grant, the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and the Chalmers AI Research Center (CHAIR). T. van Erven was supported by the Netherlands Organization for Scientific Research (NWO), grant number VI.Vidi.192.095.
[1] Volodimir G. Vovk. Aggregating strategies. In Proc. Workshop on Computational Learning Theory (COLT), Rochester, NY, USA, 1990.
[2] N. Littlestone and M.K. Warmuth. The weighted majority algorithm. Information and Computation, 108(2):212-261, 1994.
[3] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
[4] David Helmbold and Sandra Panizza. Some label efficient learning results. In Proc. Conference on Computational Learning Theory (COLT), Nashville, TN, USA, July 1997.
[5] N. Cesa-Bianchi, A. Conconi, and C. Gentile. Learning probabilistic linear-threshold classifiers via selective sampling. In Proc. Conference on Learning Theory (COLT), Washington, DC, USA, Aug. 2003.
[6] Bhuvesh Kumar, Jacob D Abernethy, and Venkatesh Saligrama. Activehedge: Hedge meets active learning. In Proc. International Conference on Machine Learning (ICML), Edinburgh, Scotland, June 2012.
[7] Ofer Dekel, Claudio Gentile, and Karthik Sridharan. Selective sampling and active learning from single and multiple teachers. The Journal of Machine Learning Research, 13(1):2655-2697, 2012.
[8] Francesco Orabona and Nicolò Cesa-Bianchi. Better algorithms for selective sampling. In Proc. International Conference on Machine Learning (ICML), Bellevue, WA, USA, June 2011.
[28] Nicoló Cesa-Bianchi, Claudio Gentile, and Luca Zaniboni. Worst-case analysis of selective sampling for linear classification. Journal of Machine Learning Research, 7(44):1205-1230, 2006.
[29] Peilin Zhao, Steven Hoi, and Jinfeng Zhuang. Active learning with expert advice. In Proc. Uncertainty in Artificial Intelligence (UAI), Bellevue, WA, USA, July 2013.
[30] Shuji Hao, Peiying Hu, Peilin Zhao, Steven C. H. Ho, and Chunyan Miao. Online active learning with expert advice. ACM Trans. Knowl. Discov. Data, 12(5), June 2018.
[31] Yining Chen, Haipeng Luo, Tengyu Ma, and Chicheng Zhang. Active online learning with hidden shifting domains. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), San Diego, CA, USA, Aug. 2021.
[32] Boshuang Huang, Sudeep Salgia, and Qing Zhao. Disagreement-based active learning in online settings. IEEE Transactions on Signal Processing, 70:1947-1958, Mar. 2022.
[33] Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5(1), 2012.
[34] Pierre Gaillard, Gilles Stoltz, and Tim van Erven. A second-order bound with excess losses. In Proc. Conference on Learning Theory (COLT), Barcelona, Spain, June 2014.
[35] A. B. Tsybakov. On nonparametric estimation of density level sets. Annals of Statistics, 25:948-969, 1997.
[36] A.R. Karlin and Y. Peres. Game Theory, Alive. American Mathematical Society, 2017.
[37] Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization, 2011.
[38] Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proc. International Conference on Machine Learning (ICML), Edinburgh, Scotland, UK, June 2012.
A Proof of Theorem 1
Theorem 1. Consider the exponentially weighted follow the majority forecaster with  $\eta = \infty$ . Specifically, let  $p_t = \mathbb{1}\{A_{1,t} \geq 1/2\}$ , so that  $\hat{y}_t = \mathbb{1}\{A_{1,t} \geq 1/2\}$ . Furthermore, let
Proof. The main idea is to proceed by induction on  $n$ . For  $n = 1$ , the result holds trivially, regardless of the choice for  $q_{t}$ . Now, suppose that  $\bar{L}_{t}^{(N)} \leq \log_{2} N$  for all values  $t \in [n - 1]$ , any sequence of observations and expert predictions, and any number of experts  $N$ . Based on this assumption, we will derive a bound for  $\bar{L}_{n}^{(N)}$ . Let  $k := \sum_{i=1}^{N} \ell_{i,1}$  be the number of experts that make a mistake when  $t = 1$ . Note that  $0 \leq k \leq N - 1$ , as there is one perfect expert. When  $k < N / 2$ , the majority vote  $\hat{y}_{1}$  is necessarily equal to  $y_{1}$ . Therefore,
$$
\begin{array}{l} \bar {L} _ {n} ^ {(N)} = \mathbb {E} [ \ell (\hat {y} _ {1}, y _ {1}) ] + \sum_ {t = 2} ^ {n} \mathbb {E} [ \ell (\hat {y} _ {t}, y _ {t}) ] = \sum_ {t = 2} ^ {n} \mathbb {E} [ \ell (\hat {y} _ {t}, y _ {t}) ] \\ = \sum_ {t = 2} ^ {n} q _ {1} \mathbb {E} [ \ell (\hat {y} _ {t}, y _ {t}) | Z _ {1} = 1 ] + (1 - q _ {1}) \mathbb {E} [ \ell (\hat {y} _ {t}, y _ {t}) | Z _ {1} = 0 ] \\ = q _ {1} \bar {L} _ {2: n} ^ {(N - k)} + (1 - q _ {1}) \bar {L} _ {2: n} ^ {(N)} \\ \leq \log_ {2} N, \tag {10} \\ \end{array}
$$
where  $\bar{L}_{2:n}^{(N)}$  denotes the expected cumulative loss in rounds  $2, \ldots, n$  with  $N$  experts. Because the internal state of the algorithm only consists of a list of experts that have made no errors so far, the task in rounds  $2, \ldots, n$  is equivalent to a task over  $n - 1$  rounds starting with the experts that remain after round 1. Therefore, we can apply the induction hypothesis to obtain  $\bar{L}_{2:n}^{(N - k)} \leq \log_2(N - k) \leq \log_2N$  and  $\bar{L}_{2:n}^{(N)} \leq \log_2N$ , which justifies the last inequality. We conclude that, when  $k < N/2$ , the bound holds regardless of the choice of  $q_1$ .
On the other hand, if  $N / 2 \leq k \leq N - 1$ , the forecaster incurs an error at time  $t = 1$ . Using the induction hypothesis and an analogous reasoning as above, we find that
$$
\bar {L} _ {n} ^ {(N)} = 1 + q _ {1} \bar {L} _ {2: n} ^ {(N - k)} + (1 - q _ {1}) \bar {L} _ {2: n} ^ {(N)} \leq 1 + q _ {1} \log_ {2} (N - k) + (1 - q _ {1}) \log_ {2} (N). \tag {11}
$$
To ensure that the right-hand-side of (11) satisfies the desired bound, we need to select  $q_{1}$  such that
$$
1 + q _ {1} \log_ {2} (N - k) + (1 - q _ {1}) \log_ {2} (N) \leq \log_ {2} N,
$$
which can be written equivalently as
$$
\frac {1}{q _ {1}} \leq \log_ {2} N - \log_ {2} (N - k).
$$
If we do not observe a label, we do not know the value  $k$ . All we know is that, since  $y_{1} \in \{0, 1\}$ , we have  $k \in \left\{\sum_{i=1}^{N} f_{i,1}, N - \sum_{i=1}^{N} f_{i,1}\right\}$ . Therefore, to ensure the induction proof works, we need
$$
q _ {1} \geq - \frac {1}{\log_ {2} \min (A _ {1 , 1} , 1 - A _ {1 , 1})},
$$
where  $A_{1,1} = \frac{1}{N}\sum_{i=1}^{N}f_{i,1}$ . Note that the case  $k = N$  cannot occur as there is always a perfect expert, but to ensure that the above definition is sound, we define  $q_1 = 0$  when  $A_{1,1} \in \{0,1\}$ .
B Label-efficient boosted majority of leaders
Through a variation of follow the majority, a slightly better regret bound than the one in Theorem 1 can be obtained, as well as a matching lower bound. As shown by [36, Proposition 18.1.3], there exists an adversarial strategy for the environment such that any forecaster will incur at least  $\lfloor \log_2N\rfloor /2\geq \lfloor \log_4N\rfloor$  errors in expectation. A matching upper bound can be obtained, when  $N$  is a power of two, by considering a forecaster that incorporates randomness in its predictions. This is referred to as the boosted majority of leaders. As shown in the following theorem, this procedure can be made label-efficient while ensuring the same expected regret bound.
Theorem 4. Consider an exponentially weighted forecaster for which  $\eta = \infty$  and
$$
p _ {t} = \left\{ \begin{array}{l l} 0 & \text {i f A _ {1 , t} \leq 1 / 4} \\ 1 + \log_ {4} A _ {1, t} & \text {i f 1 / 4 <   A _ {1 , t} \leq 1 / 2} \\ - \log_ {4} (1 - A _ {1, t}) & \text {i f 1 / 2 <   A _ {1 , t} \leq 3 / 4} \\ 1 & \text {i f A _ {1 , t} > 3 / 4} \end{array} \right.
$$
$$
q _ {t} = \left\{ \begin{array}{l l} 0 & \text {i f A _ {1 , t} = 0} \\ - 1 / \log_ {4} A _ {1, 1} & \text {i f 0 <   A _ {1 , t} <   1 / 4} \\ 1 & \text {i f 1 / 4 \leq A _ {1 , t} \leq 3 / 4} \\ - 1 / \log_ {4} (1 - A _ {1, 1}) & \text {i f 3 / 4 <   A _ {1 , t} <   1} \\ 0 & \text {i f A _ {1 , t} = 1} \end{array} \right..
$$
$$
\bar {L} _ {n} ^ {(N)} \leq \log_ {4} N.
$$
Proof. The proof technique is analogous to that of Theorem 1: we use an induction argument to find a choice of  $q_{t}$  that guarantees the desired regret bound. We will proceed by analyzing different cases, depending on the value of  $A_{1,t}$ . First, assume that  $A_{1,1} \leq 1/4$ . If  $y_{1} = 0$ , then
$$
\bar {L} _ {n} ^ {(N)} = 0 + q _ {1} \bar {L} _ {2: n} ^ {(N (1 - A _ {1, 1}))} + (1 - q _ {1}) \bar {L} _ {2: n} ^ {(N)}.
$$
Thus, taking  $q_{1} \geq 0$  suffices. If  $y_{1} = 1$ , then