SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now we analyze reward estimates at each round  $t$ . When given history  $H_{t}$  and arm  $i \in [M]$ , each observation  $y \in H_{t,i}$  is an unbiased estimate of  $\theta^i$  as  $y \sim \mathbb{P}_{\theta^i}$ . Therefore, for any fixed  $v^{t}$ ,  $\langle v^{t},y\rangle$  is an unbiased estimate of the expected reward  $\langle v^{t},\theta^{i}\rangle$ , which we denote by  $\mu^{t,i}$ .
For each arm  $i$ , we can then obtain empirical reward estimate  $\bar{\mu}^{t,i}$  of the true expected reward  $\mu^{t,i}$  by  $\bar{\mu}^{t,i} \coloneqq \frac{1}{1 \vee |H_{t,i}|} \sum_{y \in H_{t,i}} \langle v^t, y \rangle$  where  $\bar{\mu}^{t,i} = 0$  if  $|H_{t,i}| = 0$ . Since expected rewards and reward estimates are bounded by  $[-1,1]$ , by standard sub-Gaussian tail bounds, we can then construct confidence interval,
$$
\mathbb {P} \left(\forall i \in [ M ], t \in [ T ], | \bar {\mu} ^ {t, i} - \mu^ {t, i} | \leq d ^ {t, i}\right) \geq 1 - \frac {1}{T}
$$
where  $d^{t,i} \coloneqq \sqrt{\frac{8\log(MT^2)}{1\vee|H_{t,i}|}}$ . Additionally, we define upper confidence bound as  $U^{t,i} = \mathrm{clip}_{[-1,1]}\left(\bar{\mu}^{t,i} + d^{t,i}\right)$ .
At each iteration  $t$ , we have the posterior distribution  $\mathbb{P}(\Theta = \cdot |H_t)$  of the ground truth  $\Theta = \{\theta^i\}_{i=1}^M$ .  $\widehat{\Theta} = \{\widehat{\theta^i}\}_{i=1}^M$  is sampled from this posterior. Consider  $i_\star^t = \arg \max_{i \in M} \langle v^t, \theta^i \rangle$  and  $\alpha^{t,j} = \arg \max_{i \in M} \langle v^t, \widehat{\theta^i} \rangle$ . The distribution of  $i_\star^t$  is determined by the posterior  $\mathbb{P}(\Theta = \cdot | H_t)$ . The distribution of  $\alpha^{t,j}$  is determined by the distribution of  $\widehat{\Theta}$ , which is also  $\mathbb{P}(\Theta = \cdot | H_t)$ . Therefore,  $i_\star^t$  and  $\alpha^{t,j}$  are identically distributed. Furthermore, since the upper confidence bounds are deterministic functions of  $i$  when given  $H_t$ , we then have  $\mathbb{E}[U^{t,\alpha^{t,j}} | H_t] = \mathbb{E}[U^{t,i_\star^t} | H_t]$ .
As a result, we upper bound the Bayesian regret by
$$
\begin{array}{l} B R (\text {T A I L O R}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \mu^ {t, i _ {\star} ^ {t}} - \mu^ {t, \alpha^ {t, j}} \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \left(\mu^ {t, i _ {\star} ^ {t}} - U ^ {t, i _ {\star} ^ {t}}\right) + \left(U ^ {t, \alpha^ {t, j}} - \mu^ {t, \alpha^ {t, j}}\right) \right]. \\ \end{array}
$$
Now, note that since  $\bar{\mu}^{t,i}\in [-1,1]$  we have  $\mathrm{clip}_{[-1,1]}\left(\bar{\mu}^{t,i} + d^{t,i}\right) = \mathrm{clip}_{[-\infty ,1]}\left(\bar{\mu}^{t,i} + d^{t,i}\right)$ , where only the upper clip takes effect. Based on the sub-Gaussian confidence intervals  $\mathbb{P}\left(\forall i\in [M],t\in [T],|\bar{\mu}^{t,i} - \mu^{t,i}|\leq d^{t,i}\right)\geq 1 - \frac{1}{T}$ , we can derive the following two confidence bounds:
$$
\begin{array}{l} \mathbb {P} (\forall i \in [ M ], t \in [ T ], \mu^ {t, i} > U ^ {t, i}) = \mathbb {P} (\forall i \in [ M ], t \in [ T ], \mu^ {t, i} > \operatorname {c l i p} _ {[ - 1, 1 ]} (\bar {\mu} ^ {t, i} + d ^ {t, i})) \\ = \mathbb {P} (\forall i \in [ M ], t \in [ T ], \mu^ {t, i} > \bar {\mu} ^ {t, i} + d ^ {t, i}), \text {s i n c e} \mu^ {t, i} \leq 1 \\ = \mathbb {P} (\forall i \in [ M ], t \in [ T ], \mu^ {t, i} - \bar {\mu} ^ {t, i} > d ^ {t, i}) \leq \frac {1}{2 T} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} (\forall i \in [ M ], t \in [ T ], U ^ {t, i} - \mu^ {t, i} > 2 d ^ {t, i}) = \mathbb {P} (\forall i \in [ M ], t \in [ T ], \operatorname {c l i p} _ {[ - 1, 1 ]} (\bar {\mu} ^ {t, i} + d ^ {t, i}) - \mu^ {t, i} > 2 d ^ {t, i}) \\ \leq \mathbb {P} (\forall i \in [ M ], t \in [ T ], \bar {\mu} ^ {t, i} + d ^ {t, i} - \mu^ {t, i} > 2 d ^ {t, i}) \\ = \mathbb {P} (\forall i \in [ M ], t \in [ T ], \bar {\mu} ^ {t, i} - \mu^ {t, i} > d ^ {t, i}) \leq \frac {1}{2 T}. \\ \end{array}
$$
Now with the decomposition,
$$
\begin{array}{l} B R (\text {T A I L O R}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \mu^ {t, i _ {\star} ^ {t}} - \mu^ {t, \alpha^ {t, j}} \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \mu^ {t, i _ {\star} ^ {t}} - U ^ {t, i _ {\star} ^ {t}} \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} U ^ {t, \alpha^ {t, j}} - \mu^ {t, \alpha^ {t, j}} \right] \\ \end{array}
$$
we can bound the two expectations individually.
First, to bound  $\mathbb{E}\left[\sum_{t = 1}^{T}\sum_{j = 1}^{B}\mu^{t,i_{\star}^{t}} - U^{t,i_{\star}^{t}}\right]$ , we note that  $\mu^{t,i_{\star}^{t}} - U^{t,i_{\star}^{t}}$  is negative with high probability. Also, the maximum value this can take is bounded by 2 as  $\mu^{t,i}, U^{t,i} \in [-1,1]$ . Therefore, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \mu^ {t, i _ {\star} ^ {t}} - U ^ {t, i _ {\star} ^ {t}} \right] \leq \left(\sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} 0 \cdot \mathbb {P} \left(\mu^ {t, i _ {\star} ^ {t}} <   = U ^ {t, i _ {\star} ^ {t}}\right) + 2 \cdot \mathbb {P} \left(\mu^ {t, i _ {\star} ^ {t}} > U ^ {t, i _ {\star} ^ {t}}\right)\right) \leq 2 T B \cdot \frac {1}{2 T} = B.
$$
Next, to bound  $\mathbb{E}\left[\sum_{t = 1}^{T}\sum_{j = 1}^{B}U^{t,\alpha^{t,j}} - \mu^{t,\alpha^{t,j}}\right]$  we decompose it similar to the above:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} U ^ {t, \alpha^ {t, j}} - \mu^ {t, \alpha^ {t, j}} \right] \leq \left(\sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} 2 \mathbb {P} (U ^ {t, \alpha^ {t, j}} - \mu^ {t, \alpha^ {t, j}} > 2 d ^ {t, i})\right) + \left(\sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} 2 d ^ {t, i}\right) \\ \leq B + \left(\sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \sqrt {\frac {3 2 \log (M T ^ {2})}{1 \vee | H _ {t , \alpha^ {t , j}} |}}\right) \\ \end{array}
$$
where recall that  $|H_{t,i}|$  is the number of samples collected using algorithm  $i$  in rounds  $\leq t$ .
To bound the summation, we utilize the fact that  $\frac{1}{1\vee|H_{t,i}|}\leq \frac{B}{k}$  for each  $k\in [|H_{t,i}|,|H_{t + 1,i}|]$ , since  $|H_{t + 1,i}| - |H_{t,i}|\leq B$ . As a result, we get
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \sqrt {\frac {3 2 \log (M T ^ {2})}{1 \vee | H _ {t , \alpha^ {t , j}} |}} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \sum_ {k = 1} ^ {\left| H _ {T, i} \right|} \sqrt {\frac {3 2 \log (M T ^ {2}) \cdot B}{k}} \\ \leq O (\sqrt {B (\log T + \log M)} \sum_ {i = 1} ^ {M} \sqrt {| H _ {T , i} |}) \\ \leq O \left(\sqrt {B (\log T + \log M)}\right) \cdot O (\sqrt {B M T}) = O (B \sqrt {M T (\log T + \log M)}) \\ \end{array}
$$
where last two inequalities follow from simple algebra and the fact that  $\sum_{i=1}^{M}|H_{T,i}| = TB$ .
Finally, to combine all of the bounds above, we get  $BR(\text{TAILOR}) \leq B + B + O(B\sqrt{MT(\log T + \log M)}) = O(B\sqrt{MT(\log T + \log M)})$ .
C Study of Candidate Algorithms
We compare the performance when we use the following two sets of candidate algorithms:
1. Active learning algorithms only: Uncertainty sampling, GALAXY and EMAL for multi-label classification; Uncertainty sampling, GALAXY and BADGE for multi-class classification.
2. Active learning and search algorithms: Uncertainty sampling, GALAXY, MLP, EMAL and Weak Sup for multi-label classification; Uncertainty sampling, GALAXY, MLP, BADGE and Modified Submodular for multi-class classification.
Note Modified Submodular is classified as an active search algorithms since we are using a balanced set of training examples as the conditioning set. We are effectively searching for examples similar to the ones that are annotated in these classes.
As shown in Figures 5 and 6, regardless of the meta algorithm, the performance is better when using active learning algorithms as candidates only. Nonetheless, even with active search algorithms as candidates, TAILOR still outperforms other meta active learning algorithms.
Figure 5: SVHN, Balanced Accuracy
Figure 6: CelebA, mAP
D Full Results
All of the results below are averaged from four individual trials except for Imagenet, which is the result of a single trial.
D.1 Multi-label Classification
(a) Mean Average Precision
(b) Number of Labels in Rarest Class
(c) Imbalance Ratio
Figure 7:CelebA
(c) Imbalance Ratio
Figure 8: COCO
(c) Imbalance Ratio
Figure 9: VOC
(c) Imbalance Ratio
Figure 10: Stanford Car
D.2 Multi-class Classification
(a) Balanced Accuracy
(c) Imbalance Ratio
Figure 11: CIFAR-10, 2 classes
(c) Imbalance Ratio
Figure 12: CIFAR-100, 10 classes
(c) Imbalance Ratio
Figure 13: SVHN, 2 classes
(c) Imbalance Ratio
Figure 14: Kuzushiji-49
Figure 15: Caltech256
(a) Imbalance Ratio
Figure 16: Caltech256
D.3 Multi-label Search
Figure 17: CelebA, Total Number of Positive Labels
Figure 18: COCO, Total Number of Positive Labels
Figure 19: VOC, Total Number of Positive Labels
D.4 Rarest Class Accuracy
We conduct an ablation study of the accuracy on the rarest class (determined by the ground truth class distribution). TAILOR significantly outperform baselines.
Figure 20: Stanford Car, Total Number of Positive Labels
Figure 21: CelebA, Rarest Class Accuracy
E What Algorithms Does TAILER Choose?
In the following two figures, we can see TAILOR chooses a non-uniform set of algorithms to focus on for each dataset. On CelebA, TAILOR out-perform the best baseline, EMAL sampling, by a significant margin. As we can see, TAILOR rely on selecting a combination of other candidate algorithms instead of only selecting EMAL.
On the other hand, for the Stanford car dataset, we see TAILOR's selection mostly align with the baselines that perform well especially in the later phase.
Figure 22: TAILOR Top-10 Most Selected Candidate Algorithms on CelebA Dataset
Figure 23: TAILOR Top-10 Most Selected Candidate Algorithms on Stanford Car Dataset
In the following figures, we plot the number of times the most frequent candidate algorithm is chosen. As can be shown, TAILOR chooses candidate algorithm much more aggressively than other meta algorithms in eight out of the ten settings.
Figure 24: CIFAR-10, 2 Classes, Number of Pulls of The Most Frequent Selection
Figure 25: CIFAR-100, 10 Classes, Number of Pulls of The Most Frequent Selection
Figure 26: SVHN, 2 Classes, Number of Pulls of The Most Frequent Selection
Figure 27: ImageNet-1k, Number of Pulls of The Most Frequent Selection
Figure 28: Kuzushiji-49, Number of Pulls of The Most Frequent Selection
Figure 29: Caltech256, Number of Pulls of The Most Frequent Selection
Figure 30: CelebA, Number of Pulls of The Most Frequent Selection
Figure 31: COCO, Number of Pulls of The Most Frequent Selection
Figure 32: VOC, Number of Pulls of The Most Frequent Selection
Figure 33: Stanford Car, Number of Pulls of The Most Frequent Selection