SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jifan Zhang
Madison, WI 53715
jifan@cs.wisc.edu
Shuai Shao
Meta Inc.
Menlo Park, CA 94025
sshao@meta.com
Saurabh Verma
saurabh08@meta.com
Robert Nowak
rdnowak@wisc.edu
Label efficiency has become an increasingly important objective in deep learning applications. Active learning aims to reduce the number of labeled examples needed to train deep networks, but the empirical performance of active learning algorithms can vary dramatically across datasets and applications. It is difficult to know in advance which active learning strategy will perform well or best in a given application. To address this, we propose the first adaptive algorithm selection strategy for deep active learning. For any unlabeled dataset, our (meta) algorithm TAILOR (Thompson ActIve Learning algORithm selection) iteratively and adaptively chooses among a set of candidate active learning algorithms. TAILOR uses novel reward functions aimed at gathering class-balanced examples. Extensive experiments in multi-class and multi-label applications demonstrate TAILOR's effectiveness in achieving accuracy comparable or better than that of the best of the candidate algorithms. Our implementation of TAILOR is open-sourced at https://github.com/jifanz/TAILOR.
Active learning (AL) aims to reduce data labeling cost by iteratively and adaptively finding informative unlabeled examples for annotation. Label-efficiency is increasingly crucial as deep learning models require large amount of labeled training data. In recent years, numerous new algorithms have been proposed for deep active learning [Sener and Savarese, 2017, Gal et al., 2017, Ash et al., 2019, Kothawade et al., 2021, Citovsky et al., 2021, Zhang et al., 2022]. Relative label efficiencies among algorithms, however, vary significantly across datasets and applications [Beck et al., 2021, Zhan et al., 2022]. When it comes to choosing the best algorithm for a novel dataset or application, practitioners have mostly been relying on educated guesses and subjective preferences. Prior work [Baram et al., 2004, Hsu and Lin, 2015, Pang et al., 2018] have studied the online choice of active learning algorithms for linear models, but these methods become ineffective in deep learning settings (see Section 2). In this paper, we present the first principled approach for automatically selecting effective deep AL algorithms for novel, unlabeled datasets.
We reduce the algorithm selection task to a multi-armed bandit problem. As shown in Figure 1, the idea may be viewed as a meta algorithm adaptively choosing among a set of candidate AL algorithms (arms). The objective of the meta algorithm is to maximize the cumulative reward incurred from running the chosen candidate algorithms. In Section 4, we propose reward functions that encourage
the collection of class-balanced labeled set. As mentioned above, deep AL algorithms are generally proposed to maximize different notions of informativeness. As a result, by utilizing our algorithm selection strategy TAILOR, we annotate examples that are both informative and class-diverse.
To highlight some of our results, as shown in Figure 2 for the CelebA dataset, TAILOR outperforms all candidate deep AL algorithms and collects the least amount of labels while reaching the same accuracy level. TAILOR achieves this by running a combination of candidate algorithms (see Appendix E) to yield an informative and class-diverse set of labeled examples (see Figure 3(c)).
Figure 1: Adaptive active learning algorithm selection framework for batch size 1. Our framework proposed in Section 3.2 is a batched version that chooses multiple candidate algorithms and unlabeled examples in every iteration. Labels and rewards are revealed all at once at the end of the iteration.
Figure 2: Mean Average Precision, CelebA
Our key contributions are as follows:
- To our knowledge, we propose the first adaptive algorithm selection strategy for deep active learning. Our algorithm TAILOR works particularly well on the challenging and prevalent class-imbalance settings [Kothawade et al., 2021, Emam et al., 2021, Zhang et al., 2022].
- Our framework is general purpose for both multi-label and multi-class classification. Active learning is especially helpful for multi-label classification due to the high annotation cost of obtaining multiple labels for each example.
- TAILOR can choose among large number (e.g. hundreds) of candidate deep AL algorithms even under limited (10 or 20) rounds of interaction. This is particularly important since limiting the number of model retraining steps and training batches is essential in large-scale deep active learning [Citovsky et al., 2021].
- In Section 5, we provide regret analysis of TAILER. Although TAILER can be viewed as a sort of contextual bandit problem, our regret bound is better than that obtained by a naive reduction to a linear contextual bandit reduction [Russo and Van Roy, 2014].
- We provide extensive experiments on four multi-label and six multi-class image classification datasets (Section 6). Our results show that TAILOR obtains accuracies comparable or better than the best candidate strategy for nine out of the ten datasets. On all of the ten datasets, TAILOR succeeds in collecting datasets as class-balanced as the best candidate algorithm. Moreover, with a slightly different reward function designed for active search, TAILOR performs the best in finding the highest number of positive class labels on all multi-label datasets.
Adaptive Algorithm Selection in Active Learning. Several past works have studied the adaptive selection of active learning algorithms for linear models. Donmez et al. [2007] studied the limited setting of switching between two specific strategies to balance between uncertainty and diversity. To choose among off-the-shelf AL algorithms, Baram et al. [2004] first proposed a framework that reduced the AL algorithm selectino task to a multi-armed bandit problem. That approach aims to maximize the cumulative reward in a Classification Entropy Maximization score, which measures the class-balancedness of predictions on unlabeled examples, after training on each newly labeled example. However, this becomes computationally intractable for large datasets with computationally expensive models. To remedy this problem, Hsu and Lin [2015] and Pang et al. [2018] proposed the use of importance weighted training accuracy scores for each newly labeled example. The training accuracy, however, is almost always  $100\%$  for deep learning models due to their universal approximation capability, which makes the reward signals less effective. Moreover, Hsu and Lin
[2015] reduced their problem to an adversarial multi-armed bandit problem while Pang et al. [2018] also studied the non-stationarity of rewards over time.
Lastly, we would like to distinguish the goal of our paper from the line of Learning Active Learning literature [Konyushkova et al., 2017, Shao et al., 2019, Zhang et al., 2020, Gonsior et al., 2021, Löffler and Mutschler, 2022], where they learn a single paramtereized policy model from offline datasets. These policies can nonetheless serve as individual candidate algorithms, while TAILOR aims to select the best subsets that are adapted for novel dataset instances.
Multi-label Deep Active Learning. Many active learning algorithms for multi-label classification based on linear models have been proposed [Wu et al., 2020], but few for deep learning. Multi-label active learning algorithms are proposed for two types of annotation, example-based where all associated labels for an example are annotated, and example-label-based where annotator assigns a binary label indicating whether the example is positive for the specific label class.
While Citovsky et al. [2021], Min et al. [2022] both propose deep active learning algorithms for example-label-based labels, we focus on example-based annotation in this paper. To this end, Ranganathan et al. [2018] propose an uncertainty sampling algorithm that chooses examples with the lowest class-average cross entropy losses after trained with weak supervision. We find the EMAL algorithm by Wu et al. [2014] effective on several datasets, despite being proposed for linear models. EMAL is based on simple uncertainty metric where one averages over binary margin scores for each class. Lastly, a multi-label task can be seen as individual single-label binary classification tasks for each class [Boutell et al., 2004]. By adopting this view, one can randomly interleave the above-mentioned AL algorithms for every class. In this paper, we include baselines derived from least confidence sampling [Settles, 2009], GALAXY [Zhang et al., 2022] and most likely positive sampling [Warmuth et al., 2001, 2003, Jiang et al., 2018].
Balanced Multi-class Deep Active Learning. Traditional uncertainty sampling algorithms have been adopted for deep active learning. These algorithms select uncertain examples based on scores derived from likelihood softmax scores, such as margin, least confidence and entropy [Tong and Koller, 2001, Settles, 2009, Balcan et al., 2006, Kremer et al., 2014]. The latter approaches leverage properties specific to neural networks by measuring uncertainty through dropout [Gal et al., 2017], adversarial examples [Ducoffe and Precioso, 2018] and neural network ensembles [Beluch et al., 2018]. Diversity sampling algorithms label examples that are most different from each other, based on similarity metrics such as distances in penultimate layer representations [Sener and Savarese, 2017, Geifman and El-Yaniv, 2017, Citovsky et al., 2021] or discriminator networks [Gissin and Shalev-Shwartz, 2019]. Lastly, gradient embeddings, which encode both softmax likelihood and penultimate layer representation, have become widely adopted in recent approaches [Ash et al., 2019, 2021, Wang et al., 2021, Elenter et al., 2022, Mohamadi et al., 2022]. As an example, Ash et al. [2019] uses k-means++ to query a diverse set of examples in the gradient embedding space.
Unbalanced Multi-class Deep Active Learning. More general and prevalent scenarios, such as unbalanced deep active classification, have received increasing attention in recent years [Kothawade et al., 2021, Emam et al., 2021, Zhang et al., 2022, Coleman et al., 2022, Jin et al., 2022, Aggarwal et al., 2020, Cai, 2022]. For instance, Kothawade et al. [2021] label examples with gradient embeddings that are most similar to previously collected rare examples while most dissimilar to out-of-distribution ones. Zhang et al. [2022] create linear one-vs-rest graphs based on margin scores. To collect a more class-diverse labeled set, GALAXY discovers and labels around the optimal uncertainty thresholds through a bisection procedure on shortest shortest paths.
3 Problem Statement
3.1 Notation
In pool based active learning, one starts with a large pool of  $N$  unlabeled examples  $X = \{x_{1}, x_{2}, \ldots, x_{N}\}$  with corresponding ground truth labels  $Y = \{y_{1}, y_{2}, \ldots, y_{N}\}$  initially unknown to the learner. Let  $K$  denote the total number of classes. In multi-label classification, each label  $y_{i}$  is denoted as  $y_{i} \in \{0, 1\}^{K}$  with each element  $y_{i,j}$  representing the binary association between class  $j$  and example  $x_{i}$ . On the other hand, in a multi-class problem, each label  $y_{i} \in \{e_{j}\}_{j \in [K]}$  is denoted by a canonical one-hot vector, where  $e_{j}$  is the  $j$ -th canonical vector representing the  $j$ -th class. Furthermore, at any time, we denote labeled and unlabeled examples by  $L, U \subset X$  correspondingly,
where  $L \cap U = \emptyset$ . We let  $L_0 \subset X$  denote a small seed set of labeled examples and  $U_0 = X \backslash L_0$  denote the initial unlabeled set. Lastly, an active learning algorithm  $\mathcal{A}$  takes as input a pair of labeled and unlabeled sets  $(L, U)$  and returns an unlabeled example  $\mathcal{A}(L, U) \in U$ .
3.2 Adaptive Algorithm Selection Framework
In this section, we describe a generic framework that encompasses the online algorithm selection settings in Baram et al. [2004], Hsu and Lin [2015] and Pang et al. [2018]. As shown in Algorithm 1, the meta algorithm has access to  $M$  candidate algorithms. At the beginning of any round  $t$ , a multi-set of  $B$  algorithms are chosen, where the same algorithm can be chosen multiple times. One example is selected by each algorithm in the multiset sequentially, resulting in a total of  $B$  unique examples. The batch of examples are then labeled all at once. At the end of the round, their corresponding rewards are observed based on the newly annotated examples  $\{(x^{t,j}, y^{t,j})\}_{j=1}^{B}$  selected by the algorithms. The model is also retrained on labeled examples before proceeding to the next round.
Overall, the meta algorithm aims to maximize the future cumulative reward based on noisy past reward observations of each candidate algorithm. The reward function  $r: X \times Y \to \mathbb{R}$  is measured based on an algorithm's selected examples and corresponding labels. There are two key components to this framework: the choice of reward function and a bandit strategy that optimizes future rewards. Our particular design will be presented in Section 4.
Algorithm 1 General Meta Active Learning Framework for Baram et al. [2004], Hsu and Lin [2015], Pang et al. [2018]
Define:  $M$  candidate algorithms  $A = \{\mathcal{A}_i\}_{i\in [M]}$ , pool  $X$ , total number of rounds  $T$ , batch size  $B$ .
Initialize: Labeled seed set  $L_0 \subset X$ , unlabeled set  $U_0 = X \backslash L_0$  and initial policy  $\Pi^1$ .
for  $t = 1,\dots,T$  do
Meta algorithm  $\Pi^t$  chooses multiset of algorithms  $\mathcal{A}_{\alpha_{t,1}}, \mathcal{A}_{\alpha_{t,2}}, \dots, \mathcal{A}_{\alpha_{t,B}}$ , where indexes  $\alpha_{t,1}, \dots, \alpha_{t,B} \in [M]$ . Initialize selection set  $S_t \gets \emptyset$ .
for  $j = 1,\dots,B$  do
Run algorithm to select unlabeled example  $x^{t,j} \coloneqq \mathcal{A}_{\alpha_{t,j}}(L_{t-1}, U_{t-1} \backslash S_t)$  that is unselected.
Insert the example  $x^{t,j}\colon S_t\gets S_t\cup \{x^{t,j}\}$
Annotate  $\{x^{t,j}\}_{j = 1}^{B}$  and observe labels  $\{y^{t,j}\}_{j = 1}^{B}$ . Update sets  $L_{t}\gets L_{t - 1}\cup S_{t},U_{t}\gets U_{t - 1}\backslash S_{t}$ .
Observe reward  $r^{t,j} = r(x^{t,j},y^{t,j})$  for each algorithm  $\mathcal{A}_{\alpha_{t,j}}$ , where  $j\in [B]$ .
Update policy statistics based on  $x^{t,j}, y^{t,j}$  and  $r^{t,j}$  to obtain  $\Pi^{t+1}$  and retrain model on  $L_t$ .
Objective: Maximize cumulative reward  $\sum_{t=1}^{T} \sum_{j=1}^{B} r^{t,j}$ .
We make the following two crucial assumptions for our framework:
Assumption 3.1. Any candidate batch active learning algorithm  $\bar{A}$  can be decomposed into an iterative selection procedure  $A$  that returns one unlabeled example at a time.
The assumption has been inherently made by our framework above where an deep active learning algorithm returns one unlabeled example at a time. It entails that running  $\bar{A}$  once to collect a batch of  $B$  examples is equivalent with running the iterative algorithm  $A$  for  $B$  times. As noted in Appendix A.1, most existing deep active learning algorithms can be decomposed into iterative procedures and thus can serve as candidate algorithms in our framework.
Assumption 3.2. For each round  $t \in [T]$ , we assume there exist ground truth reward distributions  $\mathbb{P}_{t,1}, \ldots, \mathbb{P}_{t,M}$  for each candidate algorithm. Furthermore, for each element  $j \in [B]$  in the batch, we make the iid assumption that reward  $r^{t,j} \stackrel{iid}{\sim} \mathbb{P}_{t,\alpha_{t,j}}$  is sampled from the distribution of the corresponding selected algorithm.
The iid assumption is made for theoretical simplicity by all of Baram et al. [2004], Hsu and Lin [2015], Pang et al. [2018]. We say the distributions are non-stationary if for any  $i \in [M]$ ,  $P_{t,i}$  varies with respect to time  $t$ . Both this paper and Pang et al. [2018] study non-stationary scenarios, whereas Baram et al. [2004] and Hsu and Lin [2015] assume the distributions are stationary across time.
4 Thompson Active Learning Algorithm Selection
In this section, we present the two key components of our design, reward function and bandit strategy. In Section 4.1, we first present a class of reward functions designed for deep active learning under class imbalance. In Section 4.2, by leveraging the structure of such reward functions, we reduce the adaptive algorithm selection framework from Section 3.2 into a novel bandit setting. In Section 4.3, we then propose our algorithm TAILOR which is specifically designed for this setting. When using TAILOR on top of deep AL algorithms, the annotated examples are informative and class-diverse.
4.1 Reward Function
We propose reward functions that encourage selecting examples so that every class is well represented in the labeled dataset, ideally equally represented or "class-balanced". Our reward function works well even under practical scenarios such as limited number of rounds and large batch sizes [Citovsky et al., 2021]. The rewards we propose can be efficiently computed example-wise as opposed to Baram et al. [2004] and are more informative and generalizable than Hsu and Lin [2015] and Pang et al. [2018]. Our class-balance-based rewards are especially effective for datasets with underlying class imbalance. Recall  $y \in \{0,1\}^{K}$  for multi-label classification and  $y \in \{e_i\}_{i=1}^{K}$  for multi-class classification. We define the following types of reward functions.
- Class Diversity: To encourage better class diversity, we propose a reward that inversely weights each class by the number of examples already collected. For each round  $t \in [T]$ ,
$$
r _ {d i v} ^ {t} (x, y) = \frac {1}{K} \sum_ {i = 1} ^ {K} \frac {1}{\max (1 , \mathrm {C O U N T} ^ {t} (i))} y _ {: i} =: \langle v _ {d i v} ^ {t}, y \rangle
$$
where  $\mathrm{COUNT}^t (i)$  denotes the number of examples in class  $i$  after  $t - 1$  rounds and  $y_{:i}$  denotes the  $i$ -th element of  $y$ . We let  $v_{div}^{t}$  denote the inverse weighting vector.
- Multi-label Search: As shown in Table 1, multi-label classification datasets naturally tend to have sparse labels (more 0's than 1's in  $y$ ). Therefore, it is often important to search for positive labels. To encourage this, we define a stationary reward function for multi-label classification:
$$
r _ {s e a r c h} (x, y) = \frac {1}{K} \sum_ {i = 1} ^ {K} y _ {: i} =: \langle v _ {p o s}, y \rangle \quad \text {w h e r e} v _ {p o s} = \frac {1}{K} \vec {1}.
$$
- **Domain Specific:** Lastly, we would like to note that domain experts can define specialized weighting vectors of different classes  $v_{dom}^{t} \in [-\frac{1}{K}, \frac{1}{K}]^{K}$  that are adjusted over time  $t$ . The reward function simply takes the form  $r_{dom}^{t}(x,y) = \langle v_{dom}^{t}, y \rangle$ . As an example of multi-label classification of car information, one may prioritize classes of car brands over classes of car types, thus weighting each class differently. They can also adjust the weights over time based on needs.
4.2 Novel Bandit Setting
We now present a novel bandit reduction that mirrors the adaptive algorithm selection framework under this novel class of rewards. In this setup,  $v_{t} \in [-\frac{1}{K}, \frac{1}{K}]^{K}$  is arbitrarily chosen and nonstationary. On the other hand, for each candidate algorithm  $\mathcal{A}_{i} \in A$ , we assume the labels  $y$  are sampled iid from a stationary 1-sub-Gaussian distribution  $\mathbb{P}_{\theta^i}$  with mean  $\theta^i$ . Both the stationary assumption in  $P_{\theta^i}$  and the iid assumption are made for simplicity of our theoretical analysis only. We will describe our implementation to overcome the non-stationarity in  $\mathbb{P}_{\theta^i}$  in Section 6.1. Although we make the iid assumption analogous to Assumption 3.2, we demonstrate the effectiveness of our algorithm in Section 6 through extensive experiments. Additionally, note that  $\theta^i \in [0,1]^K$  for multi-label classification and  $\theta^i \in \Delta^{(K-1)}$  takes value in the  $K$  dimensional probability simplex for multi-class classification. In our bandit reduction, at each round  $t$ ,
1. Nature reveals weighting vector  $v^t$ ;
2. Meta algorithm chooses algorithms  $\alpha^{t,1},\dots,\alpha^{t,B}$ , which sequentially select unlabeled examples;
3. Observe batch of labels  $y^{t,1},\dots,y^{t,B}$  all at once, where  $y^{t,j}\stackrel {iid}{\sim}\mathbb{P}_{\theta^{\alpha_{t,j}}}$
4. Objective: maximize rewards defined as  $r^{t,j} = \langle v^t, y^{t,j} \rangle$ .
This setting bears resemblance to a linear contextual bandit problem. Indeed, one can formulate such a problem close to our setting by constructing arms  $\phi_i^t = \mathrm{vec}(v^t e_i^\top) \in [-\frac{1}{K}, \frac{1}{K}]^{KM}$ . Here,  $\mathrm{vec}(\cdot)$  vectorizes the outer product between  $v^t$  and the  $i$ -th canonical vector  $e_i$ . A contextual bandit algorithm observes reward  $r = \langle \phi_i^t, \theta^\star \rangle + \varepsilon$  after pulling arm  $i$ , where  $\theta^\star = \mathrm{vec}([ \theta^1, \theta^2, \dots, \theta^M ]) \in [0, 1]^{KM}$  and  $\varepsilon$  is some sub-Gaussian random noise. However, this contextual bandit formulation does not take into account the observations of  $\{y^{t,j}\}_{j=1}^B$  at each round, which are direct realizations of  $\theta^1, \dots, \theta^M$ . In fact, standard contextual bandit algorithms usually rely on least squares estimates of  $\theta^1, \dots, \theta^M$  based on the reward signals [Russo and Van Roy, 2014]. As will be shown in Proposition 5.1, a standard Bayesian regret upper bound from Russo and Van Roy [2014] is of order  $\widetilde{O}(BM^{\frac{3}{4}}K^{\frac{3}{4}}\sqrt{T})$ . Our algorithm TAILOR, on the other hand, leverages the observations of  $y^{t,j} \sim \mathbb{P}_{\theta^{\alpha^{t,j}}}$  and has regret upper bounded by  $\widetilde{O}(B\sqrt{MT})$  (Theorem 5.2), similar to a stochastic multi-armed bandit.
Algorithm 2 TAILER : Thompson Active Learning Algorithm Selection
Input:  $M$  candidate algorithms  $A = \{\mathcal{A}_i\}_{i\in [M]}$  , pool  $X$  , total number of rounds  $T$  , batch size  $B$    
Initialize: For each  $i\in [M]$ $a^i = b^i = \vec{1}\in \mathbb{R}^{+K}$    
for  $t = 1,\dots,T$  do Nature reveals  $v^{t}\in [-\frac{1}{K},\frac{1}{K}]^{K}$  Choose candidate algorithms: for  $j = 1,\ldots ,B$  do For each  $i\in [M]$  sample  $\widehat{\theta}^i\sim \mathrm{Beta}(a^i,b^i)$  for multi-label or  $\widehat{\theta}^i\sim \mathrm{Dir}(a^i)$  for multi-class. Choose  $\alpha^{t,j}\gets \arg \max_{i\in [M]}\langle v^t,\widehat{\theta}^i\rangle .$    
end for Run chosen algorithms to collect batch: for  $j = 1,\dots,B$  do Run algorithm  $\mathcal{A}_{\alpha^{t,j}}$  to select unlabeled example  $x^{t,j}$  and insert into  $S_{t}$    
end for   
Annotate examples in  $S_{t}$  to observe  $y^{t,j}$  for each  $j\in [B]$  Update posterior distributions: For each algorithm  $i\in [M]$  ..  $a^i\gets a^i +\sum_{j:\alpha^{t,j} = i}y^{t,j}$ $b^{i}\gets b^{i} + \sum_{j:\alpha^{t,j} = i}(1 - y^{t,j})$  Retrain neural network to inform next round.   
end for
4.3 TAILER
We are now ready to present TAILOR, a Thompson Sampling [Thompson, 1933] style meta algorithm for adaptively selecting active learning algorithms. The key idea is to maintain posterior distributions for  $\theta^1,\dots,\theta^M$ . As shown in Algorithm 2, at the beginning we utilize uniform priors  $\mathrm{Unif}(\Omega)$  over the support  $\Omega$ , where  $\Omega = \Delta^{(t - 1)}$  and  $[0,1]^K$  respectively for multi-label and multi-class classification. We note that the choice of uniform prior is made so that it is general purpose for any dataset. In practice, one may design more task-specific priors.