SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to optimize the regret bound in (7), we set  $\eta = \sqrt{8\ln(N) / n}$ . For each time  $t\in [n]$ , we consider two different weight vectors:  $\{w_{i,t}^P\}_{i\in [N]}$ , corresponding to the passive, full-information forecaster, and  $\{w_{i,t}^A\}_{i\in [N]}$ , corresponding to the active, label-efficient forecaster. The weight for each expert for both forecasters is uniformly initialized as  $w_{i,0}^{P} = w_{i,0}^{A} = 1 / N$ .
Then, for each timestep  $t \in [n]$ , we proceed as follows. First, we compute the probability of prediction the label 1 for each forecaster, given by  $p_t^P$  for the full-information forecaster and  $p_t^A$  for the label-efficient forecaster, as
$$
p _ {t} ^ {P} = A _ {1, t} ^ {P} = \frac {\sum_ {i = 1} ^ {N} w _ {i , t - 1} ^ {P} f _ {i , t}}{\sum_ {i = 1} ^ {N} w _ {i , t - 1} ^ {P}}, \qquad p _ {t} ^ {A} = A _ {1, t} ^ {A} = \frac {\sum_ {i = 1} ^ {N} w _ {i , t - 1} ^ {A} f _ {i , t}}{\sum_ {i = 1} ^ {N} w _ {i , t - 1} ^ {A}},
$$
where the expert predictions are computed based on  $X_{t}$ . Here, the weighted proportion of experts that predict the label 1 are given by  $A_{1,t}^{P}$  for the full-information forecaster and  $A_{1,t}^{A}$  for the label-efficient forecaster. On the basis of this, each forecaster issues a prediction for the label, given by  $\hat{y}_t^P\sim \mathrm{Ber}(p_t^P)$  for the full-information forecaster and  $\hat{y}_t^A\sim \mathrm{Ber}(p_t^A)$  for the label-efficient one.
Finally, the weights for each forecaster are updated as follows. For the full-information forecaster, the weight assigned to each expert is updated as
$$
w _ {i, t} ^ {P} = w _ {i, t - 1} ^ {P} e ^ {- \eta \ell_ {i, t}}, \tag {23}
$$
where  $\ell_{i,t} = \mathbb{1}\left\{f_{i,t}\neq Y_t\right\}$ . In order to determine whether the label-efficient forecaster observes a label or not, we compute  $q_{t} = \min (4A_{1,t}^{A}(1 - A_{1,t}^{A}) - \eta /3,1)$  and generate  $Z_{t}\sim \mathrm{Ber}(q_{t})$ . Then, the weights of the label-efficient forecaster are updated as
$$
w _ {i, t} ^ {A} = w _ {i, t - 1} ^ {A} e ^ {- \eta \frac {\ell_ {i , t} Z _ {t}}{q _ {t}}} . \tag {24}
$$
Hence, if  $Z_{t} = 0$ , the weights of the label-efficient forecaster are not updated, since the label is not observed.
After all  $n$  rounds have been completed, we compute the regret of the full-information forecaster  $R_{n}^{P}$  and the label-efficient forecaster  $R_{n}^{A}$  as
$$
R _ {n} ^ {P} = \sum_ {t = 1} ^ {n} \mathbb {1} \left\{\hat {y} _ {t} ^ {P} \neq Y _ {t} \right\} - \mathbb {1} \left\{y _ {t} ^ {*} \neq Y _ {t} \right\}, \quad R _ {n} ^ {A} = \sum_ {t = 1} ^ {n} \mathbb {1} \left\{\hat {y} _ {t} ^ {A} \neq Y _ {t} \right\} - \mathbb {1} \left\{y _ {t} ^ {*} \neq Y _ {t} \right\}.
$$
Here, for each  $t \in [n]$ , the optimal prediction is given by  $y_{t}^{*} = \mathbb{1}\{X_{t} \geq 1/2\}$ . Furthermore, we compute the label complexity  $S_{n}$  as
$$
S _ {n} = \sum_ {t = 1} ^ {n} Z _ {t}.
$$
The results of these simulations are presented in Figure 2 and 3.