SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Over time, we keep an posterior distribution over each ground truth mean  $\theta^i$  for each algorithm  $i\in [M]$ . With a uniform prior, the posterior distribution is an instance of either element-wise Beta distribution<sup>1</sup> for multi-label classification or Dirichlet distribution for multi-class classification. During each round  $t$ , we draw samples from the posteriors, which are then used to choose the best action (i.e., candidate algorithm) that has the largest predicted reward. After the batch of  $B$  candidate algorithms are chosen, we then sequentially run each algorithm to collect the batch of unlabeled examples. Upon receiving the batch of annotations, we then update the posterior distribution for each algorithm. Lastly, the neural network model is retrained on all labeled examples thus far.
5 Analysis
In this section, we present regret upper bound of TAILOR and compare against a linear contextual bandit upper bound from Russo and Van Roy [2014]. Our time complexity analysis is in Appendix 5.1.
For  $z\in [0,1]^d$  and  $a,b\in \mathcal{Z}^{+^d}$  , we say  $z\sim \mathrm{Beta}(a,b)$  if for each  $i\in [d],z_i\sim \mathrm{Beta}(a_i,b_i)$
Given an algorithm  $\pi$ , the expected regret measures the difference between the expected cumulative reward of the optimal action and the algorithm's action. Formally for any fixed instance with  $\Theta = \{\theta^1, \dots, \theta^M\}$ , the expected regret is defined as
$$
R (\pi , \Theta) := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {j = 1} ^ {B} \max _ {i \in [ M ]} \langle v ^ {t}, \theta^ {i} - \theta^ {\alpha^ {t, j}} \rangle \right]
$$
where the expectation is over the randomness of the algorithm, e.g. posterior sampling in TAILOR.
Bayesian regret simply measures the average of expected regret over different instances
$$
B R (\pi) := \mathbb {E} _ {\theta^ {i} \sim \mathbb {P} _ {0} (\Omega), i \in [ M ]} \left[ R \left(\pi , \left\{\theta^ {i} \right\} _ {i = 1} ^ {M}\right) \right]
$$
where  $\Omega$  denotes the support of each  $\theta^i$  and  $\mathbb{P}_0(\Omega)$  denotes the prior. Recall  $\Omega = [0,1]^K$  for multi-label classification and  $\Omega = \Delta^{(K - 1)}$  for multi-class classification. While TAILOR is proposed based on uniform priors  $\mathbb{P}_0(\Omega) = \mathrm{uniform}(\Omega)$ , our analysis in this section holds for arbitrary  $\mathbb{P}_0$  as long as the prior and posterior updates are modified accordingly in TAILOR.
First, we would like to mention a Bayesian regret upper bound for the contextual bandit formulation mentioned in 4.2. This provides one upper bound for TAILOR. As mentioned, the reduction to a contextual bandit is valid, but is only based on observing rewards and ignores the fact that TAILOR observes rewards and the full realizations  $y^{t,j}$  of  $\theta^{\alpha^{t,j}}$  that generate them. So one anticipates that this bound may be loose.
Proposition 5.1 (Russo and Van Roy [2014]). Let  $\pi_{\text{context}}$  be the posterior sampling algorithm for linear contextual bandit presented in Russo and Van Roy [2014], the Bayesian regret is bounded by
$$
B R (\pi_ {\text {c o n t e x t}}) \leq \widetilde {O} (B M ^ {\frac {3}{4}} K ^ {\frac {3}{4}} \log T \sqrt {T})
$$
where  $B$  is the batch size,  $M$  is the number of candidate algorithms,  $K$  is the number of classes, and  $T$  is the number of rounds.
We omit the proof in this paper and would like to point the readers to section 6.2.1 in Russo and Van Roy [2014] for the proof sketch. As mentioned in the paper, detailed confidence ellipsoid of least squares estimate and ellipsoid radius upper bound can be recovered from pages 14-15 of Abbasi-Yadkori et al. [2011].
We now present an upper bound on the Bayesian regret of TAILOR, which utilizes standard sub-Gaussian tail bounds based on observations of  $y^{t,j}$  instead of confidence ellipsoids derived from only observing reward signals of  $r^{t,j}$ .
Theorem 5.2 (Proof in Appendix B). The Bayesian regret of TAILOR is bounded by
$$
B R (T A I L O R) \leq O \left(B \sqrt {M T (\log T + \log M)}\right)
$$
where  $B$  is the batch size,  $M$  is the number of candidate algorithms and  $T$  is total number of rounds.
We delay our complete proof to Appendix B. To highlight the key difference of our analysis from Russo and Van Roy [2014], their algorithm only relies on observations of  $r^{t,j}$  for each round  $t \in [T]$  and element  $j \in [B]$  in a batch. To estimate  $\theta^1, \dots, \theta^M$ , they use the least squares estimator to form confidence ellipsoids. In our analysis, we utilize observations of  $y$ 's up to round  $t$  and form confidence intervals directly around each of  $\langle v^t, \theta^1 \rangle, \dots, \langle v^t, \theta^M \rangle$  by unbiased estimates  $\{\langle v^t, y \rangle\}$ .
5.1 Time Complexity
Let  $N_{train}$  denote the total neural network training. The time complexity of collecting each batch for each active learning algorithm  $\mathcal{A}_i$  can be separated into  $P_i$  and  $Q_i$ , which are the computation complexity for preprocessing and selection of each example respectively. As examples of preprocessing, BADGE [Ash et al., 2019] computes gradient embeddings, SIMILAR [Kothawade et al., 2021] further also compute similarity kernels, GALAXY [Zhang et al., 2022] constructs linear graphs, etc. The selection complexities are the complexities of each iteration of K-means++ in BADGE, greedy submodular optimization in SIMILAR, and shortest shortest path computation in GALAXY. Therefore, for any individual algorithm  $\mathcal{A}_i$ , the computation complexity is then
DATASET	K	N	CLASS IMB RATIO	BINARY IMB RATIO
CELEBA [LIU ET AL., 2018]	40	162770	.0273	.2257
COCO [LIN ET AL., 2014]	80	82081	.0028	.0367
VOC [EVERINGHAM ET AL., 2010]	20	10000	.0749	.0721
CAR [KRAUSE ET AL., 2013]	10	12948	.1572	.1200
IMAGENET [DENG ET AL., 2009]	1000	1281167	.5631	—
KUZUSHII-49 [CLANUWAT ET AL., 2018]	49	23236	.0545	—
CALTECH256 [GRIFFIN ET AL., 2007]	256	24486	.0761	—
IMB CIFAR-10 [KRIZHEVSKY ET AL., 2009]	2	50000	.1111	—
IMB CIFAR-100 [KRIZHEVSKY ET AL., 2009]	10	50000	.0110	—
IMB SVHN [NETZER ET AL., 2011]	2	73257	.0724	—
Table 1: Details for multi-label and multi-class classification datasets.  $K$  and  $N$  denote the number of classes and pool size respectively. Class Imbalance Ratio represents the class imbalance ratio between the smallest and the largest class. We also report Binary Imbalance Ratio for multi-label datasets, which is defined as the average positive ratio over classes, i.e.,  $\frac{1}{K}\sum_{i\in [K]}(N_i / N)$  where  $N_{i}$  denotes the number of examples in class  $i$ .
$O(N_{train} + TP_i + TBQ_i)$  where  $T$  is the total number of rounds and  $B$  is the batch size. When running TAILOR, as we do not know which algorithms are selected, we provide a worst case upper bound of  $O(N_{train} + T \cdot (\sum_{i=1}^{M} P_i) + TB \cdot \max_{i \in [M]} Q_i)$ , where the preprocessing is done for every candidate algorithm. In practice, some of the preprocessing operations such as gradient embedding computation could be shared among multiple algorithms, thus only need to be computed once. As a practical note in all of our experiments, TAILOR is more than  $20\%$  faster than the slowest candidate algorithm as it selects a diverse set of candidate algorithms instead of running a single algorithm the entire time. Also, the most significant time complexity in practice often lies in neural network retraining. The retraining time dominates the running time of all algorithms including reward and Thompson sampling complexities.
In this section, we present results of TAILOR in terms of classification accuracy, class-balance of collected labels, and total number of positive examples for multi-label active search. Motivated by the observations, we also propose some future directions at the end.
6.1 Setup
Datasets. Our experiments span ten datasets with class-imbalance as shown in Table 1. For multi-label experiments, we experiment on four datasets including CelebA, COCO, VOC and Stanford Car datasets. While the Stanford Car dataset is a multi-class classification dataset, we transform it into a multi-label dataset as detailed in Appendix A.3. For multi-class classification datasets, ImageNet, Kuzushiji-49 and Caltech256 are naturally unbalanced datasets, while CIFAR-10 with 2 classes, CIFAR-100 with 10 classes and SVHN with 2 classes are derived from the original dataset following Zhang et al. [2022]. Specifically, we keep the first  $K - 1$  classes from the original dataset and treat the rest of the images as a large out-of-distribution  $K$ -th class.
Implementation Details. We conduct experiments on varying batch sizes anywhere from  $B = 500$  to  $B = 10000$ . To mirror a limited training budget [Citovsky et al., 2021, Emam et al., 2021], we allow 10 or 20 batches in total for each dataset, making it extra challenging for our adaptive algorithm selection due to the limited rounds of interaction.
Moreover, we assumed observations of  $y$  are sampled from stationary distributions  $\mathbb{P}_{\theta^1},\dots,\mathbb{P}_{\theta^M}$  in our analysis. However, these distributions could be dynamically changing. In our implementation, we use a simple trick to discount the past observations, where we change the posterior update in Algorithm 2 to  $a^i\gets \gamma a^i +\sum_{j:\alpha^{t,j} = i}y^{t,j}$  and  $b^{i}\gets \gamma b^{i} + \sum_{j:\alpha^{t,j} = i}(1 - y^{t,j})$  . We set the discounting factor  $\gamma$  to be .9 across all experiments. As will be discussed in Section 6.3, we find non-stationarity in
$\{\mathbb{P}_{\theta^k}\}_{k = 1}^M$  an interesting future direction to study. Lastly, we refer the readers to Appendix A for additional implementation details.
Baseline Algorithms. In our experiments, we choose a representative and popular subset of the deep AL algorithms and active search algorithms discussed in Section 2 as our baselines. To demonstrate the ability of TAILOR, number of candidate algorithms  $M$  ranges from tens to hundreds for different datasets. The baselines can be divided into three categories:
- We include off-the-shelf active learning and active search algorithms such as EMAL [Wu et al., 2014] and Weak Supervision [Ranganathan et al., 2018] for multi-label classification and Confidence sampling [Settles, 2009], BADGE [Ash et al., 2019], Modified Submodular optimization motivated by Kothawade et al. [2021] for multi-class classification. More implementation details can be found in Appendices A.1 and A.2.
- We derive individual candidate algorithms based on a per-class decomposition [Boutell et al., 2004]. For most likely positive sampling [Warmuth et al., 2001, 2003, Jiang et al., 2018], an active search strategy and abbreviated as MLP, we obtain  $K$  algorithms where the  $i$ -th algorithm selects examples most likely to be in the  $i$ -th class.
For multi-label classification, we also include  $K$  individual GALAXY algorithms [Zhang et al., 2022] and  $K$  Uncertainty sampling algorithms. To further elaborate, the original GALAXY work by Zhang et al. [2022] construct  $K$  one-vs-rest linear graphs, one for each class. GALAXY requires finding the shortest shortest path among all  $K$  graphs, an operation whose computation scales linearly in  $K$ . When  $K$  is large, this becomes computationally prohibitive to run. Therefore, we instead include  $K$  separate GALAXY algorithms, each only bisecting on one of the one-vs-rest graphs. This is equivalent with running  $K$  GALAXY algorithms, one for each binary classification task between class  $i \in [K]$  and the rest.
For Uncertainty sampling in multi-label settings, we similarly have  $K$  individual uncertainty sampling algorithms, where the  $i$ -th algorithm samples the most uncertain example based only on the binary classification task of class  $i$ .
As baselines for each type of algorithms above, we simply interleave the set of  $K$  algorithms uniformly at random.
- We compare against other adaptive meta selection algorithms, including Random Meta which chooses candidate algorithms uniform at random and ALBL Sampling [Hsu and Lin, 2015]. The candidate algorithms include all of the active learning baselines. In Appendix C, we also provide an additional study of including active search baselines as candidate algorithms.
(a) Balanced Accuracy, CIFAR-10, 2 classes
(b) Balanced Accuracy, Kuzushiji-49
Figure 3: Performance of TAILER against baselines on selected settings. (a) and (b) shows accuracy metrics of the algorithms. (c) shows class-balancedness of labeled examples. All performances are averaged over four trials with standard error plotted for each algorithm. The curves are smoothed with a sliding window of size 3.
(c) #Labels in Rarest Class, CelebA
6.2 Results
Multi-class and Multi-label Classification. For evaluation, we focus on TAILOR's comparisons against both existing meta algorithms and the best baseline respectively. In all classification experiments, TAILOR uses the class diversity reward in Section 4.1. For accuracy metrics, we utilize mean average precision for multi-label classification and balanced accuracy for multi-class classification. As a class diversity metric, we look at the size of the smallest class based on collected labels. All experiments are measured based on active annotation performance over the pool [Zhang et al., 2022].
As shown in Figures 2, 3 and Appendix D, when comparing against existing meta algorithms, TAILOR performs better on all datasets in terms of both accuracy and class diversity metrics. ALBL sampling performs similar to Random Meta in all datasets, suggesting the ineffectiveness of training accuracy based rewards proposed in Hsu and Lin [2015] and Pang et al. [2018]. When comparing against the best baseline algorithm, TAILOR performs on par with the best baseline algorithm on nine out of ten datasets in terms of accuracy and on all datasets in terms of class diversity. On the CelebA dataset, TAILOR even outperforms the best baseline by significant margin in accuracy. As discussed in Appendix E, TAILOR achieves this by selecting a combination of other candidate algorithms instead of choosing only the best baseline. On four out of the ten datasets, TAILOR outperforms the best baseline in class diversity. Collectively, this shows the power of TAILOR in identifying the best candidate algorithms over different dataset scenarios. Moreover in Appendix D.4, we conduct an ablation study of the accuracy of the rarest class (determined by the ground truth class distribution). TAILOR significantly outperform baselines suggesting its advantage in improving the accuracy on all classes. Lastly, shown in Appendix E, we also find TAILOR selects algorithms more aggressively than existing meta algorithms. The most frequent algorithms also align with the best baselines.
On the other hand for the Caltech256 dataset shown in Figure 16, TAILOR under-performs confidence sampling in terms of accuracy. We conjecture this is because the larger classes may not have sufficient examples and have much space for improvement before learning the smaller classes. Nevertheless, TAILOR was able to successfully collect a much more class-diverse dataset while staying competitive to other baseline methods.
Multi-label Search. We use the multi-label search reward proposed in Section 4.1. As shown in Figure 4 and Appendix D.3, on three of the four datasets, TAILOR performs better than the best baseline algorithm in terms of total collected positive labels. On the fourth dataset, TAILOR performs second to and on par with the best baseline. This shows TAILOR's ability in choosing the best candidate algorithms for active search.
Figure 4: Total positive labels for active search, CelebA
6.3 Future Work
While our experiments focus on class-imbalanced settings, TAILOR's effectiveness on balanced datasets warrants future study through further experiments and alternative reward design. We also find studying non-stationarity in label distributes  $\{\mathbb{P}_{\theta_i}\}_{i = 1}^M$  an interesting next step.
7 Choosing Candidate Algorithms
Our paper proposes an adaptive selection procedure over candidate deep AL algorithms. When judging individual deep AL algorithms, current standards in the research community tend to focus on whether an algorithm performs well on all dataset and application instances. However, we see value in AL algorithms that perform well only in certain instances. Consider, for example, an AL algorithm that performs well on  $25\%$  of previous applications, but poorly on the other  $75\%$ . One may wish to include this algorithm in TAILOR because the new application might be similar to those where it performs well. From the perspective of TAILOR, a "good" AL algorithm need not perform well on all or even most of a range of datasets, it just needs to perform well on a significant number of datasets.
On the other hand, as suggested by our regret bound that scales with  $M$ , one should not include too many algorithms. In fact, there are exponential numbers of possible AL algorithms, which could easily surpass our labeling budget and overwhelm the meta selection algorithm. In practice, one could leverage extra information such as labeling budget, batch size and model architecture to choose proper set of candidate algorithms to target their settings.
We would like to thank Aude Hofleitner and Shawndra Hill for the support of this research project, and Kevin Jamieson, Yifang Chen and Andrew Wagenmaker for insightful discussions. Robert Nowak would like to thank the support of NSF Award 2112471.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1183-1192. PMLR, 2017.
Yonatan Geifman and Ran El-Yaniv. Deep active learning over the long tail. arXiv preprint arXiv:1711.00941, 2017.
Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint arXiv:1907.06347, 2019.
Julius Gonsior, Maik Thiele, and Wolfgang Lehner. Imital: Learning active learning strategies from synthetic data. arXiv preprint arXiv:2108.07670, 2021.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Wei-Ning Hsu and Hsuan-Tien Lin. Active learning by learning. In Twenty-Ninth AAAI conference on artificial intelligence, 2015.
Shali Jiang, Gustavo Malkomes, Matthew Abbott, Benjamin Moseley, and Roman Garnett. Efficient nonmyopic batch active search. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), 2018.
Qiuye Jin, Mingzhi Yuan, Haoran Wang, Manning Wang, and Zhijian Song. Deep active learning models for imbalanced image classification. Knowledge-Based Systems, 257:109817, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. Advances in neural information processing systems, 30, 2017.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems, 34:18685-18697, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
Jan Kremer, Kim Steenstrup Pedersen, and Christian Igel. Active learning with support vector machines. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 4(4):313-326, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
Christoffer Löffler and Christopher Mutschler. Iale: Imitating active learner ensembles. Journal of Machine Learning Research, 23(107):1-29, 2022.
A.1 Deep Active Learning Decomposition
For any uncertainty sampling algorithm, picking the top-  $B$  most uncertain examples can be easily decomposed into an iterative procedure that picks the next most uncertain example. Next, for diversity based deep active learning algorithms, one usually relies on a greedy iterative procedure to collect a batch, e.g. K-means++ for BADGE [Ash et al., 2019] and greedy K-centers for Coreset [Sener and Savarese, 2017]. Lastly, deep active learning algorithms such as Cluster-Margin [Citovsky et al., 2021] and GALAXY [Zhang et al., 2022] have already proposed their algorithms as iterative procedures that select unlabeled examples sequentially.
A.2 Implementation of Modified Submodular
Instead of requiring access to a balanced holdout set [Kothawade et al., 2021], we construct the balanced set using training examples. We use the Submodular Mutual Information function FLQMI as suggested by Table 1 of Kothawade et al. [2021]. The proposed greedy submodular optimization is itself an iterative procedure that selects one example at a time. While SIMILAR usually performs well, our modification that discards the holdout set is unfortunately ineffective in our experiments. This is primarily due to the lack of the holdout examples, which may often happen in practical scenarios.
A.3 Stanford Car Multi-label Dataset
We transform the original labels into 10 binary classes of
1. If the brand is "Audi".
2. If the brand is "BMW".
3. If the brand is "Chevrolet".
4. If the brand is "Dodge".
5. If the brand is "Ford".
6. If the car type is "Convertible".
7. If the car type is "Coupe".
8. If the car type is "SUV".
9. If the car type is "Van".
10. If the car is made in or before 2009.
A.4 Negative Weighting for Common Classes
For multi-label classifications, for some classes, there could be more positive associations (label of 1s) than negative associations (label of 0s). Therefore, in those classes, the rarer labels are negative. In class diverse reward  $\langle v_{div}^t,y\rangle$  in Section 4.1, we implement an additional weighting of  $\mathbb{I}_{rare}^{t}*v_{div}^{t}$ , where  $^*$  denotes an elementwise multiplication. Here, each element  $\mathbb{I}_{rare,i}^{t}\in \{1, - 1\}$  takes value  $-1$  when  $\mathrm{COUNT}^t (i)$  is larger than half the size of labeled set. This negative weighting can be seen as upsampling negative class associations when positive associations are the majority.
A.5 Model Training
All of our experiments are conducted using the ResNet-18 architecture [He et al., 2016] pretrained on ImageNet. We use the Adam optimizer [Kingma and Ba, 2014] with learning rate of 1e-4 and weight decay of 5e-5.
B Proof of Theorem 5.2
Our proof follows a similar procedure from regret analysis for Thompson Sampling of the stochastic multi-armed bandit problem [Lattimore and Szepesvári, 2020]. Let  $\alpha^t \coloneqq \{\alpha^{t,j}\}_{j=1}^B$  and  $y^t \coloneqq$
$\{y^{t,j}\}_{j = 1}^{B}$  denote the actions and observations from the  $i$ -th round. We define the history up to  $t$  as  $H_{t} = \{\alpha^{1}, y^{1}, \alpha^{2}, y^{2}, \dots, \alpha^{t - 1}, y^{t - 1}\}$ . Moreover, for each  $i \in [M]$ , we define  $H_{t,i} = \{y^{t',j} \in H_{t} : \alpha^{t',j} = i\}$  as the history of all observations made by choosing the  $i$ -th arm (algorithm).