Title: Algorithm Selection for Deep Active Learning with Imbalanced Datasets
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Active Learning
Abstract: Label efficiency has become an increasingly important objective in deep learning applications. Active learning aims to reduce the number of labeled examples needed to train deep networks, but the empirical performance of active learning algorithms can vary dramatically across datasets and applications. It is difficult to know in advance which active learning strategy will perform well or best in a given application. To address this, we propose the first adaptive algorithm selection strategy for deep active learning. For any unlabeled dataset, our (meta) algorithm TAILOR (Thompson ActIve Learning algORithm selection) iteratively and adaptively chooses among a set of candidate active learning algorithms. TAILOR uses novel reward functions aimed at gathering class-balanced examples. Extensive experiments in multi-class and multi-label applications demonstrate TAILOR's effectiveness in achieving accuracy comparable or better than that of the best of the candidate algorithms. Our implementation of TAILOR is open-sourced at https://github.com/jifanz/TAILOR.

Algorithm Selection for Deep Active Learning with Imbalanced Datasets

Abstract

Label efficiency has become an increasingly important objective in deep learning applications. Active learning aims to reduce the number of labeled examples needed to train deep networks, but the empirical performance of active learning algorithms can vary dramatically across datasets and applications. It is difficult to know in advance which active learning strategy will perform well or best in a given application. To address this, we propose the first adaptive algorithm selection strategy for deep active learning. For any unlabeled dataset, our (meta) algorithm TAILOR ( T hompson A ct I ve L earning alg OR ithm selection) iteratively and adaptively chooses among a set of candidate active learning algorithms. TAILOR uses novel reward functions aimed at gathering class-balanced examples. Extensive experiments in multi-class and multi-label applications demonstrate TAILOR 's effectiveness in achieving accuracy comparable or better than that of the best of the candidate algorithms. Our implementation of TAILOR is open-sourced at https://github.com/jifanz/TAILOR .

1 Introduction

Active learning (AL) aims to reduce data labeling cost by iteratively and adaptively finding informative unlabeled examples for annotation. Label-efficiency is increasingly crucial as deep learning models require large amount of labeled training data. In recent years, numerous new algorithms have been proposed for deep active learning [Sener and Savarese, 2017, Gal et al., 2017, Ash et al., 2019, Kothawade et al., 2021, Citovsky et al., 2021, Zhang et al., 2022]. Relative label efficiencies among algorithms, however, vary significantly across datasets and applications [Beck et al., 2021, Zhan et al., 2022]. When it comes to choosing the best algorithm for a novel dataset or application, practitioners have mostly been relying on educated guesses and subjective preferences. Prior work [Baram et al., 2004, Hsu and Lin, 2015, Pang et al., 2018] have studied the online choice of active learning algorithms for linear models, but these methods become ineffective in deep learning settings (see Section 2). In this paper, we present the first principled approach for automatically selecting effective deep AL algorithms for novel, unlabeled datasets.

We reduce the algorithm selection task to a multi-armed bandit problem. As shown in Figure 1, the idea may be viewed as a meta algorithm adaptively choosing among a set of candidate AL algorithms (arms). The objective of the meta algorithm is to maximize the cumulative reward incurred from running the chosen candidate algorithms. In Section 4, we propose reward functions that encourage

Saurabh Verma Meta Inc. Menlo Park, CA 94025

the collection of class-balanced labeled set. As mentioned above, deep AL algorithms are generally proposed to maximize different notions of informativeness. As a result, by utilizing our algorithm selection strategy TAILOR , we annotate examples that are both informative and class-diverse .

To highlight some of our results, as shown in Figure 2 for the CelebA dataset, TAILOR outperforms all candidate deep AL algorithms and collects the least amount of labels while reaching the same accuracy level. TAILOR achieves this by running a combination of candidate algorithms (see Appendix E) to yield an informative and class-diverse set of labeled examples (see Figure 3(c)).

MetaAlgorithm

Meta

Algorithm

5

<>>

<>

choose

/>

ALCandidate

Algorithm

口

ALCandidate

Algorithm M

口

ALCandidate

Algorithmi

口

Selectunlabeled

examplex

Annotatefor

labely

Our key contributions are as follows:

To our knowledge, we propose the first adaptive algorithm selection strategy for deep active learning. Our algorithm TAILOR works particularly well on the challenging and prevalent classimbalance settings [Kothawade et al., 2021, Emam et al., 2021, Zhang et al., 2022].

Our framework is general purpose for both multi-label and multi-class classification. Active learning is especially helpful for multi-label classification due to the high annotation cost of obtaining multiple labels for each example.

TAILOR can choose among large number (e.g. hundreds) of candidate deep AL algorithms even under limited (10 or 20) rounds of interaction. This is particularly important since limiting the number of model retraining steps and training batches is essential in large-scale deep active learning [Citovsky et al., 2021].

In Section 5, we provide regret analysis of TAILOR . Although TAILOR can be viewed as a sort of contextual bandit problem, our regret bound is better than that obtained by a naive reduction to a linear contextual bandit reduction [Russo and Van Roy, 2014].

We provide extensive experiments on four multi-label and six multi-class image classification datasets (Section 6). Our results show that TAILOR obtains accuracies comparable or better than the best candidate strategy for nine out of the ten datasets. On all of the ten datasets, TAILOR succeeds in collecting datasets as class-balanced as the best candidate algorithm. Moreover, with a slightly different reward function designed for active search, TAILOR performs the best in finding the highest number of positive class labels on all multi-label datasets.

2 Related Work

Adaptive Algorithm Selection in Active Learning. Several past works have studied the adaptive selection of active learning algorithms for linear models. Donmez et al. [2007] studied the limited setting of switching between two specific strategies to balance between uncertainty and diversity. To choose among off-the-shelf AL algorithms, Baram et al. [2004] first proposed a framework that reduced the AL algorithm selectino task to a multi-armed bandit problem. That approach aims to maximize the cumulative reward in a Classification Entropy Maximization score, which measures the class-balancedness of predictions on unlabeled examples, after training on each newly labeled example. However, this becomes computationally intractable for large datasets with computationally expensive models. To remedy this problem, Hsu and Lin [2015] and Pang et al. [2018] proposed the use of importance weighted training accuracy scores for each newly labeled example. The training accuracy, however, is almost always 100% for deep learning models due to their universal approximation capability, which makes the reward signals less effective. Moreover, Hsu and Lin

20000

40000

60000

80000

100000

Number of Labels

0.78

0.80

0.82

0.84

0.86

0.88

0.90

Mean Average Precision

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (81 algs)

[2015] reduced their problem to an adversarial multi-armed bandit problem while Pang et al. [2018] also studied the non-stationarity of rewards over time.

Lastly, we would like to distinguish the goal of our paper from the line of Learning Active Learning literature [Konyushkova et al., 2017, Shao et al., 2019, Zhang et al., 2020, Gonsior et al., 2021, Löffler and Mutschler, 2022], where they learn a single paramtereized policy model from offline datasets. These policies can nonetheless serve as individual candidate algorithms, while TAILOR aims to select the best subsets that are adapted for novel dataset instances.

Multi-label Deep Active Learning. Many active learning algorithms for multi-label classification based on linear models have been proposed [Wu et al., 2020], but few for deep learning. Multilabel active learning algorithms are proposed for two types of annotation, example-based where all associated labels for an example are annotated, and example-label-based where annotator assigns a binary label indicating whether the example is positive for the specific label class.

While Citovsky et al. [2021], Min et al. [2022] both propose deep active learning algorithms for example-label-based labels, we focus on example-based annotation in this paper. To this end, Ranganathan et al. [2018] propose an uncertainty sampling algorithm that chooses examples with the lowest class-average cross entropy losses after trained with weak supervision. We find the EMAL algorithm by Wu et al. [2014] effective on several datasets, despite being proposed for linear models. EMAL is based on simple uncertainty metric where one averages over binary margin scores for each class. Lastly, a multi-label task can be seen as individual single-label binary classification tasks for each class [Boutell et al., 2004]. By adopting this view, one can randomly interleave the above-mentioned AL algorithms for every class. In this paper, we include baselines derived from least confidence sampling [Settles, 2009], GALAXY [Zhang et al., 2022] and most likely positive sampling [Warmuth et al., 2001, 2003, Jiang et al., 2018].

Balanced Multi-class Deep Active Learning. Traditional uncertainty sampling algorithms have been adopted for deep active learning. These algorithms select uncertain examples based on scores derived from likelihood softmax scores, such as margin, least confidence and entropy [Tong and Koller, 2001, Settles, 2009, Balcan et al., 2006, Kremer et al., 2014]. The latter approaches leverage properties specific to neural networks by measuring uncertainty through dropout [Gal et al., 2017], adversarial examples [Ducoffe and Precioso, 2018] and neural network ensembles [Beluch et al., 2018]. Diversity sampling algorithms label examples that are most different from each other, based on similarity metrics such as distances in penultimate layer representations [Sener and Savarese, 2017, Geifman and El-Yaniv, 2017, Citovsky et al., 2021] or discriminator networks [Gissin and Shalev-Shwartz, 2019]. Lastly, gradient embeddings, which encode both softmax likelihood and penultimate layer representation, have become widely adopted in recent approaches [Ash et al., 2019, 2021, Wang et al., 2021, Elenter et al., 2022, Mohamadi et al., 2022]. As an example, Ash et al. [2019] uses k-means++ to query a diverse set of examples in the gradient embedding space.

Unbalanced Multi-class Deep Active Learning. More general and prevalent scenarios, such as unbalanced deep active classification, have received increasing attention in recent years [Kothawade et al., 2021, Emam et al., 2021, Zhang et al., 2022, Coleman et al., 2022, Jin et al., 2022, Aggarwal et al., 2020, Cai, 2022]. For instance, Kothawade et al. [2021] label examples with gradient embeddings that are most similar to previously collected rare examples while most dissimilar to out-of-distribution ones. Zhang et al. [2022] create linear one-vs-rest graphs based on margin scores. To collect a more class-diverse labeled set, GALAXY discovers and labels around the optimal uncertainty thresholds through a bisection procedure on shortest shortest paths.

3 Problem Statement

3.1 Notation

In pool based active learning, one starts with a large pool of N unlabeled examples X = { x 1 , x 2 , ..., x N } with corresponding ground truth labels Y = { y 1 , y 2 , ..., y N } initially unknown to the learner. Let K denote the total number of classes. In multi-label classification, each label y i is denoted as y i ∈ { 0 , 1 } K with each element y i,j representing the binary association between class j and example x i . On the other hand, in a multi-class problem, each label y i ∈ { e j } j ∈ [ K ] is denoted by a canonical one-hot vector, where e j is the j -th canonical vector representing the j -th class. Furthermore, at any time, we denote labeled and unlabeled examples by L, U ⊂ X correspondingly,

where L ∩ U = ∅ . We let L 0 ⊂ X denote a small seed set of labeled examples and U 0 = X \ L 0 denote the initial unlabeled set. Lastly, an active learning algorithm A takes as input a pair of labeled and unlabeled sets ( L, U ) and returns an unlabeled example A ( L, U ) ∈ U .

3.2 Adaptive Algorithm Selection Framework

In this section, we describe a generic framework that encompasses the online algorithm selection settings in Baram et al. [2004], Hsu and Lin [2015] and Pang et al. [2018]. As shown in Algorithm 1, the meta algorithm has access to M candidate algorithms. At the beginning of any round t , a multi-set of B algorithms are chosen, where the same algorithm can be chosen multiple times. One example is selected by each algorithm in the multiset sequentially, resulting in a total of B unique examples. The batch of examples are then labeled all at once. At the end of the round, their corresponding rewards are observed based on the newly annotated examples { ( x t,j , y t,j ) } B j =1 selected by the algorithms. The model is also retrained on labeled examples before proceeding to the next round.

Overall, the meta algorithm aims to maximize the future cumulative reward based on noisy past reward observations of each candidate algorithm. Th reward function r : X × Y → R is measured based on an algorithm's selected examples and corresponding labels. There are two key components to this framework: the choice of reward function and a bandit strategy that optimizes future rewards. Our particular design will be presented in Section 4.

Algorithm 1 General Meta Active Learning Framework for Baram et al. [2004], Hsu and Lin [2015], Pang et al. [2018]

Define: M candidate algorithms A = {A i } i ∈ [ M ] , pool X , total number of rounds T , batch size B . Initialize: Labeled seed set L 0 ⊂ X , unlabeled set U 0 = X \ L 0 and initial policy Π 1 .

for

t

= 1

, ..., T

do

Meta algorithm Π t chooses multiset of algorithms A α t, 1 , A α t, 2 , ..., A α t,B , where indexes α t, 1 , ..., α t,B ∈ [ M ] . Initialize selection set S t ←∅ .

for

j

= 1

, ..., B

do

Run algorithm to select unlabeled example x t,j := A α t,j ( L t -1 , U t -1 \ S t ) that is unselected. Insert the example x t,j : S t ← S t ∪ { x t,j } .

end for

Annotate { x t,j } B j =1 and observe labels { y t,j } B j =1 . Update sets L t ← L t -1 ∪ S t , U t ← U t -1 \ S t . Observe reward r t,j = r ( x t,j , y t,j ) for each algorithm A α t,j , where j ∈ [ B ] .

Update policy statistics based on x t,j , y t,j and r t,j to obtain Π t +1 and retrain model on L .

Objective: Maximize cumulative reward ∑ T t =1 ∑ B j =1 r t,j .

t end for

We make the following two crucial assumptions for our framework:

Assumption 3.1. Any candidate batch active learning algorithm ¯ A can be decomposed into an iterative selection procedure A that returns one unlabeled example at a time.

The assumption has been inherently made by our framework above where an deep active learning algorithm returns one unlabeled example at a time. It entails that running ¯ A once to collect a batch of B examples is equivalent with running the iterative algorithm A for B times. As noted in Appendix A.1, most existing deep active learning algorithms can be decomposed into iterative procedures and thus can serve as candidate algorithms in our framework.

Assumption 3.2. For each round t ∈ [ T ] , we assume there exist ground truth reward distributions P t, 1 , ..., P t,M for each candidate algorithm. Furthermore, for each element j ∈ [ B ] in the batch, we make the iid assumption that reward r t,j iid ∼ P t,α t,j is sampled from the distribution of the corresponding selected algorithm.

The iid assumption is made for theoretical simplicity by all of Baram et al. [2004], Hsu and Lin [2015], Pang et al. [2018]. We say the distributions are non-stationary if for any i ∈ [ M ] , P t,i varies with respect to time t . Both this paper and Pang et al. [2018] study non-stationary scenarios, whereas Baram et al. [2004] and Hsu and Lin [2015] assume the distributions are stationary across time.

4 Thompson Active Learning Algorithm Selection

In this section, we present the two key components of our design, reward function and bandit strategy. In Section 4.1, we first present a class of reward functions designed for deep active learning under class imbalance. In Section 4.2, by leveraging the structure of such reward functions, we reduce the adaptive algorithm selection framework from Section 3.2 into a novel bandit setting. In Section 4.3, we then propose our algorithm TAILOR which is specifically designed for this setting. When using TAILOR on top of deep AL algorithms, the annotated examples are informative and class-diverse .

4.1 Reward Function

We propose reward functions that encourage selecting examples so that every class is well represented in the labeled dataset, ideally equally represented or 'class-balanced". Our reward function works well even under practical scenarios such as limited number of rounds and large batch sizes [Citovsky et al., 2021]. The rewards we propose can be efficiently computed example-wise as opposed to Baram et al. [2004] and are more informative and generalizable than Hsu and Lin [2015] and Pang et al. [2018]. Our class-balance-based rewards are especially effective for datasets with underlying class imbalance. Recall y ∈ { 0 , 1 } K for multi-label classification and y ∈ { e i } K i =1 for multi-class classification. We define the following types of reward functions.

Class Diversity : To encourage better class diversity, we propose a reward that inversely weights each class by the number of examples already collected. For each round t ∈ [ T ] ,

where COUNT t ( i ) denotes the number of examples in class i after t -1 rounds and y : i denotes the i -th element of y . We let v t div denote the inverse weighting vector.

Multi-label Search : As shown in Table 1, multi-label classification datasets naturally tend to have sparse labels (more 0 's than 1 's in y ). Therefore, it is often important to search for positive labels. To encourage this, we define a stationary reward function for multi-label classification:

Domain Specific : Lastly, we would like to note that domain experts can define specialized weighting vectors of different classes v t dom ∈ [ -1 K , 1 K ] K that are adjusted over time t . The reward function simply takes the form r t dom ( x, y ) = ⟨ v t dom , y ⟩ . As an example of multi-label classification of car information, one may prioritize classes of car brands over classes of car types, thus weighting each class differently. They can also adjust the weights over time based on needs.

4.2 Novel Bandit Setting

We now present a novel bandit reduction that mirrors the adaptive algorithm selection framework under this novel class of rewards. In this setup, v t ∈ [ -1 K , 1 K ] K is arbitrarily chosen and nonstationary. On the other hand, for each candidate algorithm A i ∈ A , we assume the labels y are sampled iid from a stationary 1-sub-Gaussian distribution P θ i with mean θ i . Both the stationary assumption in P θ i and the iid assumption are made for simplicity of our theoretical analysis only. We will describe our implementation to overcome the non-stationarity in P θ i in Section 6.1. Although we make the iid assumption analogous to Assumption 3.2, we demonstrate the effectiveness of our algorithm in Section 6 through extensive experiments. Additionally, note that θ i ∈ [0 , 1] K for multi-label classification and θ i ∈ ∆ ( K -1) takes value in the K dimensional probability simplex for multi-class classification. In our bandit reduction, at each round t ,

Nature reveals weighting vector v t ;

Meta algorithm chooses algorithms α t, 1 , ..., α t,B , which sequentially select unlabeled examples;

Observe batch of labels y t, 1 , ..., y t,B all at once, where y t,j iid ∼ P θ α t,j ;

Objective: maximize rewards defined as r t,j = ⟨ v t , y t,j ⟩ .

This setting bears resemblance to a linear contextual bandit problem. Indeed, one can formulate such a problem close to our setting by constructing arms ϕ t i = vec ( v t e ⊤ i ) ∈ [ -1 K , 1 K ] KM . Here, vec ( · ) vectorizes the outer product between v t and the i -th canonical vector e i . Acontextual bandit algorithm observes reward r = ⟨ ϕ t i , θ ⋆ ⟩ + ε after pulling arm i , where θ ⋆ = vec ( [ θ 1 , θ 2 , ..., θ M ] ) ∈ [0 , 1] KM and ε is some sub-Gaussian random noise. However, this contextual bandit formulation does not take into account the observations of { y t,j } B j =1 at each round, which are direct realizations of θ 1 , ..., θ M . In fact, standard contextual bandit algorithms usually rely on least squares estimates of θ 1 , ..., θ M based on the reward signals [Russo and Van Roy, 2014]. As will be shown in Proposition 5.1, a standard Bayesian regret upper bound from Russo and Van Roy [2014] is of order ˜ O ( BM 3 4 K 3 4 √ T ) . Our algorithm TAILOR , on the other hand, leverages the observations of y t,j ∼ P θ α t,j and has regret upper bounded by ˜ O ( B √ MT ) (Theorem 5.2), similar to a stochastic multi-armed bandit.

Algorithm 2 TAILOR : Thompson Active Learning Algorithm Selection Input: M candidate algorithms A = {A i } i ∈ [ M ] , pool X , total number of rounds T , batch size B . Initialize: For each i ∈ [ M ] , a i = b i = ⃗ 1 ∈ R + K . for t = 1 , ..., T do Nature reveals v t ∈ [ -1 K , 1 K ] K . Choose candidate algorithms: for j = 1 , ..., B do For each i ∈ [ M ] , sample ̂ θ i ∼ Beta ( a i , b i ) for multi-label or ̂ θ i ∼ Dir ( a i ) for multi-class. Choose α t,j ← arg max i ∈ [ M ] ⟨ v t , ̂ θ i ⟩ . end for Run chosen algorithms to collect batch: for j = 1 , ..., B do Run algorithm A α t,j to select unlabeled example x t,j and insert into S t . end for Annotate examples in S t to observe y t,j for each j ∈ [ B ] . Update posterior distributions: For each algorithm i ∈ [ M ] : a i ← a i + ∑ j : α t,j = i y t,j , b i ← b i + ∑ j : α t,j = i (1 -y t,j ) . Retrain neural network to inform next round. end for

4.3 TAILOR

We are now ready to present TAILOR , a Thompson Sampling [Thompson, 1933] style meta algorithm for adaptively selecting active learning algorithms. The key idea is to maintain posterior distributions for θ 1 , ..., θ M . As shown in Algorithm 2, at the beginning we utilize uniform priors Unif (Ω) over the support Ω , where Ω = ∆ ( t -1) and [0 , 1] K respectively for multi-label and multi-class classification. We note that the choice of uniform prior is made so that it is general purpose for any dataset. In practice, one may design more task-specific priors.

Over time, we keep an posterior distribution over each ground truth mean θ i for each algorithm i ∈ [ M ] . With a uniform prior, the posterior distribution is an instance of either element-wise Beta distribution 1 for multi-label classification or Dirichlet distribution for multi-class classification. During each round t , we draw samples from the posteriors, which are then used to choose the best action (i.e., candidate algorithm) that has the largest predicted reward. After the batch of B candidate algorithms are chosen, we then sequentially run each algorithm to collect the batch of unlabeled examples. Upon receiving the batch of annotations, we then update the posterior distribution for each algorithm. Lastly, the neural network model is retrained on all labeled examples thus far.

5 Analysis

In this section, we present regret upper bound of TAILOR and compare against a linear contextual bandit upper bound from Russo and Van Roy [2014]. Our time complexity analysis is in Appendix 5.1.

Given an algorithm π , the expected regret measures the difference between the expected cumulative reward of the optimal action and the algorithm's action. Formally for any fixed instance with Θ = { θ 1 , ..., θ M } , the expected regret is defined as

where the expectation is over the randomness of the algorithm, e.g. posterior sampling in TAILOR .

Bayesian regret simply measures the average of expected regret over different instances

where Ω denotes the support of each θ i and P 0 (Ω) denotes the prior. Recall Ω = [0 , 1] K for multilabel classification and Ω = ∆ ( K -1) for multi-class classification. While TAILOR is proposed based on uniform priors P 0 (Ω) = uniform(Ω) , our analysis in this section holds for arbitrary P 0 as long as the prior and posterior updates are modified accordingly in TAILOR .

First, we would like to mention a Bayesian regret upper bound for the contextual bandit formulation mentioned in 4.2. This provides one upper bound for TAILOR . As mentioned, the reduction to a contextual bandit is valid, but is only based on observing rewards and ignores the fact that TAILOR observes rewards and the full realizations y t,j of θ α t,j that generate them. So one anticipates that this bound may be loose.

Proposition 5.1 (Russo and Van Roy [2014]) . Let π context be the posterior sampling algorithm for linear contextual bandit presented in Russo and Van Roy [2014], the Bayesian regret is bounded by

where B is the batch size, M is the number of candidate algorithms, K is the number of classes, and T is the number of rounds.

We omit the proof in this paper and would like to point the readers to section 6.2.1 in Russo and Van Roy [2014] for the proof sketch. As mentioned in the paper, detailed confidence ellipsoid of least squares estimate and ellipsoid radius upper bound can be recovered from pages 14-15 of Abbasi-Yadkori et al. [2011].

We now present an upper bound on the Bayesian regret of TAILOR , which utilizes standard subGaussian tail bounds based on observations of y t,j instead of confidence ellipsoids derived from only observing reward signals of r t,j .

Theorem 5.2 (Proof in Appendix B) . The Bayesian regret of TAILOR is bounded by

where B is the batch size, M is the number of candidate algorithms and T is total number of rounds.

We delay our complete proof to Appendix B. To highlight the key difference of our analysis from Russo and Van Roy [2014], their algorithm only rely on observations of r t,j for each round t ∈ [ T ] and element j ∈ [ B ] in a batch. To estimate θ 1 , ..., θ M , they use the least squares estimator to form confidence ellipsoids. In our analysis, we utilize observations of y 's up to round t and form confidence intervals directly around each of ⟨ v t , θ 1 ⟩ , ..., ⟨ v t , θ M ⟩ by unbiased estimates {⟨ v t , y ⟩ .

5.1 Time Complexity

Let N train denote the total neural network training. The time complexity of collecting each batch for each active learning algorithm A i can be separated into P i and Q i , which are the computation complexity for preprocessing and selection of each example respectively. As examples of preprocessing, BADGE [Ash et al., 2019] computes gradient embeddings, SIMILAR [Kothawade et al., 2021] further also compute similarity kernels, GALAXY [Zhang et al., 2022] constructs linear graphs, etc. The selection complexities are the complexities of each iteration of K-means++ in BADGE, greedy submodular optimization in SIMILAR, and shortest shortest path computation in GALAXY. Therefore, for any individual algorithm A i , the computation complexity is then

Table 1: Details for multi-label and multi-class classification datasets. K and N denote the number of classes and pool size respectively. Class Imbalance Ratio represents the class imbalance ratio between the smallest and the largest class. We also report Binary Imbalance Ratio for multi-label datasets, which is defined as the average positive ratio over classes, i.e., 1 K ∑ i ∈ [ K ] ( N i /N ) where N i denotes the number of examples in class i .

O ( N train + TP i + TBQ i ) where T is the total number of rounds and B is the batch size. When running TAILOR , as we do not know which algorithms are selected, we provide a worst case upper bound of O ( N train + T · ( ∑ M i =1 P i ) + TB · max i ∈ [ M ] Q i ) , where the preprocessing is done for every candidate algorithm. In practice, some of the preprocessing operations such as gradient embedding computation could be shared among multiple algorithms, thus only need to be computed once. As a practical note in all of our experiments, TAILOR is more than 20% faster than the slowest candidate algorithm as it selects a diverse set of candidate algorithms instead of running a single algorithm the entire time. Also, the most significant time complexity in practice often lies in neural network retraining. The retraining time dominates the running time of all algorithms including reward and Thompson sampling complexities.

6 Experiments

In this section, we present results of TAILOR in terms of classification accuracy, class-balance of collected labels, and total number of positive examples for multi-label active search. Motivated by the observations, we also propose some future directions at the end.

6.1 Setup

Datasets. Our experiments span ten datasets with class-imbalance as shown in Table 1. For multilabel experiments, we experiment on four datasets including CelebA, COCO, VOC and Stanford Car datasets. While the Stanford Car dataset is a multi-class classification dataset, we transform it into a multi-label dataset as detailed in Appendix A.3. For multi-class classification datasets, ImageNet, Kuzushiji-49 and Caltech256 are naturally unbalanced datasets, while CIFAR-10 with 2 classes, CIFAR-100 with 10 classes and SVHN with 2 classes are derived from the original dataset following Zhang et al. [2022]. Specifically, we keep the first K -1 classes from the original dataset and treat the rest of the images as a large out-of-distribution K -th class.

Implementation Details. We conduct experiments on varying batch sizes anywhere from B = 500 to B = 10000 . To mirror a limited training budget [Citovsky et al., 2021, Emam et al., 2021], we allow 10 or 20 batches in total for each dataset, making it extra challenging for our adaptive algorithm selection due to the limited rounds of interaction.

Moreover, we assumed observations of y are sampled from stationary distributions P θ 1 , ..., P θ M in our analysis. However, these distributions could be dynamically changing. In our implementation, we use a simple trick to discount the past observations, where we change the posterior update in Algorithm 2 to a i ← γa i + ∑ j : α t,j = i y t,j and b i ← γb i + ∑ j : α t,j = i (1 -y t,j ) . We set the discounting factor γ to be . 9 across all experiments. As will be discussed in Section 6.3, we find non-stationarity in

{ P θ k } M k =1 an interesting future direction to study. Lastly, we refer the readers to Appendix A for additional implementation details.

Baseline Algorithms. In our experiments, we choose a representative and popular subset of the deep AL algorithms and active search algorithms discussed in Section 2 as our baselines. To demonstrate the ability of TAILOR , number of candidate algorithms M ranges from tens to hundreds for different datasets. The baselines can be divided into three categories:

We include off-the-shelf active learning and active search algorithms such as EMAL [Wu et al., 2014] and Weak Supervision [Ranganathan et al., 2018] for multi-label classification and Confidence sampling [Settles, 2009], BADGE [Ash et al., 2019], Modified Submodular optimization motivated by Kothawade et al. [2021] for multi-class classification. More implementation details can be found in Appendices A.1 and A.2.

We derive individual candidate algorithms based on a per-class decomposition [Boutell et al., 2004]. For most likely positive sampling [Warmuth et al., 2001, 2003, Jiang et al., 2018], an active search strategy and abbreviated as MLP , we obtain K algorithms where the i -th algorithm selects examples most likely to be in the i -th class.

For multi-label classification, we also include K individual GALAXY algorithms [Zhang et al., 2022] and K Uncertainty sampling algorithms. To further elaborate, the original GALAXY work by Zhang et al. [2022] construct K one-vs-rest linear graphs, one for each class. GALAXY requires finding the shortest shortest path among all K graphs, an operation whose computation scales linearly in K . When K is large, this becomes computationally prohibitive to run. Therefore, we instead include K separate GALAXY algorithms, each only bisecting on one of the one-vs-rest graphs. This is equivalent with running K GALAXY algorithms, one for each binary classification task between class i ∈ [ K ] and the rest.

For Uncertainty sampling in multi-label settings, we similarly have K individual uncertainty sampling algorithms, where the i -th algorithm samples the most uncertain example based only on the binary classification task of class i .

As baselines for each type of algorithms above, we simply interleave the set of K algorithms uniformly at random.

We compare against other adaptive meta selection algorithms, including Random Meta which chooses candidate algorithms uniform at random and ALBL Sampling [Hsu and Lin, 2015]. The candidate algorithms include all of the active learning baselines. In Appendix C, we also provide an additional study of including active search baselines as candidate algorithms.

5000

10000 15000 20000 25000 30000 35000 40000

Number of Labels

0.75

0.80

0.85

0.90

0.95

1.00

Balanced Accuracy

Random (1 alg)

Confidence (1 alg)

GALAXY (2 algs)

MLP (2 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (7 algs)

CIFAR-10, 2 classes

2000

4000

6000

8000

10000

Number of Labels

0.6

0.7

0.8

0.9

1.0

Balanced Accuracy

Random (1 alg)

Confidence (1 alg)

GALAXY (49 algs)

MLP (49 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (101 algs)

Kuzushiji-49

20000

40000

60000

80000

100000

Number of Labels

500

1000

1500

2000

2500

3000

3500

Number of Positive Labels

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (81 algs)

(c) #Labels in Rarest Class,

CelebA

6.2 Results

Multi-class and Multi-label Classification. For evaluation, we focus on TAILOR 's comparisons against both existing meta algorithms and the best baseline respectively. In all classification experiments, TAILOR uses the class diversity reward in Section 4.1. For accuracy metrics, we utilize mean average precision for multi-label classification and balanced accuracy for multi-class classification. As a class diversity metric, we look at the size of the smallest class based on collected labels. All experiments are measured based on active annotation performance over the pool [Zhang et al., 2022].

As shown in Figures 2, 3 and Appendix D, when comparing against existing meta algorithms, TAILOR performs better on all datasets in terms of both accuracy and class diversity metrics. ALBL sampling performs similar to Random Meta in all datasets, suggesting the ineffectiveness of training accuracy based rewards proposed in Hsu and Lin [2015] and Pang et al. [2018]. When comparing against the best baseline algorithm, TAILOR performs on par with the best baseline algorithm on nine out of ten datasets in terms of accuracy and on all datasets in terms of class diversity. On the CelebA dataset, TAILOR even outperforms the best baseline by significant margin in accuracy. As discussed in Appendix E, TAILOR achieves this by selecting a combination of other candidate algorithm instead of choosing only the best baseline. On four out of the ten datasets, TAILOR outperforms the best baseline in class diversity. Collectively, this shows the power of TAILOR in identifying the best candidate algorithms over different dataset scenarios. Moreover in Appendix D.4, we conduct an ablation study of the accuracy of the rarest class (determined by the ground truth class distribution). TAILOR significantly outperform baselines suggesting its advantage in improving the accuracy on all classes. Lastly, shown in Appendix E, we also find TAILOR selects algorithms more aggressively than existing meta algorithms. The most frequent algorithms also align with the best baselines.

On the other hand for the Caltech256 dataset shown in Figure 16, TAILOR under-performs confidence sampling in terms of accuracy. We conjecture this is because the larger classes may not have sufficient examples and have much space for improvement before learning the smaller classes. Nevertheless, TAILOR was able to successfully collect a much more class-diverse dataset while staying competitive to other baseline methods.

Multi-label Search. We use the multi-label search reward proposed in Section 4.1. As shown in Figure 4 and Appendix D.3, on three of the four datasets, TAILOR performs better than the best baseline algorithm in terms of total collected positive labels. On the fourth dataset, TAILOR performs second to and on par with the best baseline. This shows TAILOR 's ability in choosing the best candidate algorithms for active search.

6.3 Future Work

While our experiments focus on class-imbalanced settings, TAILOR 's effectiveness on balanced datasets warrants future study through further experiments and alternative reward design. We also find studying non-stationarity in label distributes { P θ i } i M =1 an interesting next step.

7 Choosing Candidate Algorithms

Our paper proposes an adaptive selection procedure over candidate deep AL algorithms. When judging individual deep AL algorithms, current standards in the research community tend to focus on whether an algorithm performs well on all dataset and application instances. However, we see value in AL algorithms that perform well only in certain instances. Consider, for example, an AL algorithm that performs well on 25% of previous applications, but poorly on the other 75%. One may wish to include this algorithm in TAILOR because the new application might be similar to those where it performs well. From the perspective of TAILOR , a 'good" AL algorithm need not perform well on all or even most of a range of datasets, it just needs to perform well on a significant number of datasets.

On the other hand, as suggested by our regret bound that scales with M , one should not include too many algorithms. In fact, there are exponential number of possible AL algorithms, which could easily surpass our labeling budget and overwhelm the meta selection algorithm. In practice, one could leverage extra information such as labeling budget, batch size and model architecture to choose proper set of candidate algorithms to target their settings.

20000

40000

60000

80000

100000

Number of Labels

100000

200000

300000

400000

500000

600000

700000

800000

900000

Number of Positive Labels

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (122 algs)

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Umang Aggarwal, Adrian Popescu, and Céline Hudelot. Active learning for imbalanced datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1428-1437, 2020.

Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems , 34:89278939, 2021.

Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671 , 2019.

Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. In Proceedings of the 23rd international conference on Machine learning , pages 65-72, 2006.

Yoram Baram, Ran El Yaniv, and Kobi Luz. Online choice of active learning algorithms. Journal of Machine Learning Research , 5(Mar):255-291, 2004.

Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh Iyer. Effective evaluation of deep active learning on image classification tasks. arXiv preprint arXiv:2106.15324 , 2021.

William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9368-9377, 2018.

Matthew R Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown. Learning multi-label scene classification. Pattern recognition , 37(9):1757-1771, 2004.

Xinmeng Cai. Active learning for imbalanced data: The difficulty and proportions of class matter. Wireless Communications and Mobile Computing , 2022, 2022.

Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems , 34:11933-11944, 2021.

Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718 , 2018.

Cody Coleman, Edward Chou, Julian Katz-Samuels, Sean Culatana, Peter Bailis, Alexander C Berg, Robert Nowak, Roshan Sumbaly, Matei Zaharia, and I Zeki Yalniz. Similarity search for efficient active learning and search of rare concepts. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6402-6410, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Pinar Donmez, Jaime G Carbonell, and Paul N Bennett. Dual strategy active learning. In European Conference on Machine Learning , pages 116-127. Springer, 2007.

Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841 , 2018.

Juan Elenter, Navid NaderiAlizadeh, and Alejandro Ribeiro. A lagrangian duality approach to active learning. arXiv preprint arXiv:2202.04108 , 2022.

Zeyad Ali Sami Emam, Hong-Min Chu, Ping-Yeh Chiang, Wojciech Czaja, Richard Leapman, Micah Goldblum, and Tom Goldstein. Active learning at the imagenet scale. arXiv preprint arXiv:2111.12880 , 2021.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2): 303-338, 2010.

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning , pages 1183-1192. PMLR, 2017.

Yonatan Geifman and Ran El-Yaniv. Deep active learning over the long tail. arXiv preprint arXiv:1711.00941 , 2017.

Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint arXiv:1907.06347 , 2019.

Julius Gonsior, Maik Thiele, and Wolfgang Lehner. Imital: Learning active learning strategies from synthetic data. arXiv preprint arXiv:2108.07670 , 2021.

Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Wei-Ning Hsu and Hsuan-Tien Lin. Active learning by learning. In Twenty-Ninth AAAI conference on artificial intelligence , 2015.

Shali Jiang, Gustavo Malkomes, Matthew Abbott, Benjamin Moseley, and Roman Garnett. Efficient nonmyopic batch active search. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018) , 2018.

Qiuye Jin, Mingzhi Yuan, Haoran Wang, Manning Wang, and Zhijian Song. Deep active learning models for imbalanced image classification. Knowledge-Based Systems , 257:109817, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. Advances in neural information processing systems , 30, 2017.

Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems , 34:18685-18697, 2021.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013.

Jan Kremer, Kim Steenstrup Pedersen, and Christian Igel. Active learning with support vector machines. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 4(4):313-326, 2014.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

Christoffer Löffler and Christopher Mutschler. Iale: Imitating active learner ensembles. Journal of Machine Learning Research , 23(107):1-29, 2022.

Xue-Yang Min, Kun Qian, Ben-Wen Zhang, Guojie Song, and Fan Min. Multi-label active learning through serial-parallel neural networks. Knowledge-Based Systems , 251:109226, 2022.

Mohamad Amin Mohamadi, Wonho Bae, and Danica J Sutherland. Making look-ahead active learning strategies feasible with neural tangent kernels. arXiv preprint arXiv:2206.12569 , 2022.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy M Hospedales. Dynamic ensemble active learning: A non-stationary bandit with expert advice. In 2018 24th International Conference on Pattern Recognition (ICPR) , pages 2269-2276. IEEE, 2018.

Hiranmayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Panchanathan. Multi-label deep active learning with label correlation. In 2018 25th IEEE International Conference on Image Processing (ICIP) , pages 3418-3422. IEEE, 2018.

Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017.

Burr Settles. Active learning literature survey. 2009.

Jingyu Shao, Qing Wang, and Fangbing Liu. Learning to sample: an active learning framework. In 2019 IEEE International Conference on Data Mining (ICDM) , pages 538-547. IEEE, 2019.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. Journal of machine learning research , 2(Nov):45-66, 2001.

Haonan Wang, Wei Huang, Andrew Margenot, Hanghang Tong, and Jingrui He. Deep active learning by leveraging training dynamics. arXiv preprint arXiv:2110.08611 , 2021.

Manfred K Warmuth, Gunnar Rätsch, Michael Mathieson, Jun Liao, and Christian Lemmen. Active learning in the drug discovery process. In NIPS , pages 1449-1456, 2001.

Manfred K Warmuth, Jun Liao, Gunnar Rätsch, Michael Mathieson, Santosh Putta, and Christian Lemmen. Active learning with support vector machines in the drug discovery process. Journal of chemical information and computer sciences , 43(2):667-673, 2003.

Jian Wu, Victor S Sheng, Jing Zhang, Pengpeng Zhao, and Zhiming Cui. Multi-label active learning for image classification. In 2014 IEEE international conference on image processing (ICIP) , pages 5227-5231. IEEE, 2014.

Jian Wu, Victor S Sheng, Jing Zhang, Hua Li, Tetiana Dadakova, Christine Leon Swisher, Zhiming Cui, and Pengpeng Zhao. Multi-label active learning algorithms for image classification: Overview and future promise. ACM Computing Surveys (CSUR) , 53(2):1-35, 2020.

Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan. A comparative survey of deep active learning. arXiv preprint arXiv:2203.13450 , 2022.

Jifan Zhang, Lalit Jain, and Kevin Jamieson. Learning to actively learn: A robust approach. arXiv preprint arXiv:2010.15382 , 2020.

Jifan Zhang, Julian Katz-Samuels, and Robert Nowak. Galaxy: Graph-based active learning at the extreme. arXiv preprint arXiv:2202.01402 , 2022.

A Implementation Details

A.1 Deep Active Learning Decomposition

For any uncertainty sampling algorithm, picking the topB most uncertain examples can be easily decomposed into an iterative procedure that picks the next most uncertain example. Next, for diversity based deep active learning algorithms, one usually rely on a greedy iterative procedure to collect a batch, e.g. K-means++ for BADGE [Ash et al., 2019] and greedy K-centers for Coreset [Sener and Savarese, 2017]. Lastly, deep active learning algorithms such as Cluster-Margin [Citovsky et al., 2021] and GALAXY [Zhang et al., 2022] have already proposed their algorithms as iterative procedures that select unlabeled examples sequentially.

A.2 Implementation of Modified Submodular

Instead of requiring access to a balanced holdout set [Kothawade et al., 2021], we construct the balanced set using training examples. We use the Submodular Mutual Information function FLQMI as suggested by Table 1 of Kothawade et al. [2021]. The proposed greedy submodular optimization is itself an iterative procedure that selects one example at a time. While SIMILAR usually performs well, our modification that discards the holdout set is unfortunately ineffective in our experiments. This is primarily due to the lack of the holdout examples, which may often happen in practical scenarios.

A.3 Stanford Car Multi-label Dataset

We transform the original labels into 10 binary classes of

If the brand is 'Audi".

If the brand is 'BMW".

If the brand is 'Chevrolet".

If the brand is 'Dodge".

If the brand is 'Ford".

If the car type is 'Convertible".

If the car type is 'Coupe".

If the car type is 'SUV".

If the car type is 'Van".

If the car is made in or before 2009.

A.4 Negative Weighting for Common Classes

For multi-label classifications, for some classes, there could be more positive associations (label of 1 s) than negative associations (label of 0 s). Therefore, in those classes, the rarer labels are negative. In class diverse reward ⟨ v t div , y ⟩ in Section 4.1, we implement an additional weighting of ✶ t rare ∗ v div t , where ∗ denotes an elementwise multiplication. Here, each element ✶ t rare,i ∈ { 1 , -1 } takes value -1 when COUNT t ( i ) is larger than half the size of labeled set. This negative weighting can been seen as upsampling negative class associations when positive associations are the majority.

A.5 Model Training

All of our experiments are conducted using the ResNet-18 architecture [He et al., 2016] pretrained on ImageNet. We use the Adam optimizer [Kingma and Ba, 2014] with learning rate of 1e-4 and weight decay of 5e-5.

B Proof of Theorem 5.2

Our proof follows a similar procedure from regret analysis for Thompson Sampling of the stochastic multi-armed bandit problem [Lattimore and Szepesvári, 2020]. Let α t := { α t,j } B j =1 and y t :=

{ y t,j } B j =1 denote the actions and observations from the i -th round. We define the history up to t as H t = { α 1 , y 1 , α 2 , y 2 , ..., α t -1 , y t -1 } . Moreover, for each i ∈ [ M ] , we define H t,i = { y t ′ ,j ∈ H t : α t ′ ,j = i } as the history of all observations made by choosing the i -th arm (algorithm).

Now we analyze reward estimates at each round t . When given history H t and arm i ∈ [ M ] , each observation y ∈ H t,i is an unbiased estimate of θ i as y ∼ P θ i . Therefore, for any fixed v t , ⟨ v t , y ⟩ is an unbiased estimate of the expected reward ⟨ v t , θ i ⟩ , which we denote by µ t,i .

For each arm i , we can then obtain empirical reward estimate ¯ µ t,i of the true expected reward µ t,i by ¯ µ t,i := 1 1 ∨| H t,i | ∑ y ∈ H t,i ⟨ v t , y ⟩ where ¯ µ t,i = 0 if | H t,i | = 0 . Since expected rewards and reward estimates are bounded by [ -1 , 1] , by standard sub-Gaussian tail bounds, we can then construct confidence interval,

where d t,i := √ 8 log( MT 2 ) 1 ∨| H t,i | . Additionally, we define upper confidence bound as U t,i = clip [ -1 , 1] ( ¯ µ t,i + d t,i ) .

At each iteration t , we have the posterior distribution P (Θ = ·| H t ) of the ground truth Θ = { θ i } i M =1 . ̂ Θ = { ̂ θ i } i M =1 is sampled from this posterior. Consider i t ⋆ = arg max i ∈ M ⟨ v t , θ i ⟩ and α t,j = arg max i ∈ M ⟨ v t , ̂ θ i ⟩ . The distribution of i t ⋆ is determined by the posterior P (Θ = ·| H t ) . The distribution of α t,j is determined by the distribution of ̂ Θ , which is also P (Θ = ·| H t ) . Therefore, i t ⋆ and α t,j are identically distributed. Furthermore, since the upper confidence bounds are deterministic functions of i when given H t , we then have E [ U t,α t,j | H t ] = E [ U t,i t ⋆ | H t ] .

As a result, we upper bound the Bayesian regret by

Now, note that since ¯ µ t,i ∈ [ -1 , 1] we have clip [ -1 , 1] ( ¯ µ t,i + d t,i ) = clip [ -∞ , 1] ( ¯ µ t,i + d t,i ) , where only the upper clip takes effect. Based on the sub-Gaussian confidence intervals P ( ∀ i ∈ [ M ] , t ∈ [ T ] , | ¯ µ t,i -µ t,i | ≤ d t,i ) ≥ 1 -1 T , we can derive the following two confidence bounds:

Now with the decomposition,

we can bound the two expectations individually.

First, to bound E [ ∑ T t =1 ∑ B j =1 µ t,i t ⋆ -U t,i t ⋆ ] , we note that µ t,i t ⋆ -U t,i t ⋆ is negative with high probability. Also, the maximum value this can take is bounded by 2 as µ t,i , U t,i ∈ [ -1 , 1] . Therefore, we have

Next, to bound E [ ∑ T t =1 ∑ B j =1 U t,α t,j -µ t,α t,j ] we decompose it similar to the above:

where recall that | H t,i | is the number of samples collected using algorithm i in rounds ≤ t .

To bound the summation, we utilize the fact that 1 1 ∨| H t,i | ≤ B k for each k ∈ [ | H t,i | , | H t +1 ,i | ] , since | H t +1 ,i | - | H t,i | ≤ B . As a result, we get

M

where last two inequalities follow from simple algebra and the fact that ∑ i =1 | H T,i | = TB . Finally, to combine all of the bounds above, we get BR ( TAILOR ) ≤ B + B O ( B √ MT (log T +log M )) = O ( B √ MT (log T +log M )) .

+

C Study of Candidate Algorithms

We compare the performance when we use the following two sets of candidate algorithms:

Active learning algorithms only: Uncertainty sampling, GALAXY and EMAL for multilabel classification; Uncertainty sampling, GALAXY and BADGE for multi-class classification.

Active learning and search algorithms: Uncertainty sampling, GALAXY, MLP, EMAL and Weak Sup for multi-label classification; Uncertainty sampling, GALAXY, MLP, BADGE and Modified Submodular for multi-class classification.

Note Modified Submodular is classified as an active search algorithms since we are using a balanced set of training examples as the conditioning set. We are effectively searching for examples similar to the ones that are annotated in these classes.

As shown in Figures 5 and 6, regardless of the meta algorithm, the performance is better when using active learning algorithms as candidates only. Nonetheless, even with active search algorithms as candidates, TAILOR still outperforms other meta active learning algorithms.

10000

20000

30000

40000

50000

Number of Labels

0.90

0.92

0.94

0.96

0.98

1.00

Balanced Accuracy

ALBL Meta, (AL algs)

ALBL Meta, (all algs)

20000

40000

60000

80000

100000

Number of Labels

0.76

0.78

0.80

0.82

0.84

0.86

0.88

0.90

Mean Average Precision

ALBL Meta, (AL algs)

ALBL Meta, (all algs)

D Full Results

All of the results below are averaged from four individual trials except for Imagenet, which is the result of a single trial.

D.1 Multi-label Classification

20000

40000

60000

80000

100000

Number of Labels

0.78

0.80

0.82

0.84

0.86

0.88

0.90

Mean Average Precision

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (81 algs)

20000

40000

60000

80000

100000

Number of Labels

500

1000

1500

2000

2500

3000

3500

Number of Positive Labels

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (81 algs)

(b) Number of Labels in Rarest Class

20000

40000

60000

80000

100000

Number of Labels

0.030

0.035

0.040

0.045

0.050

0.055

0.060

0.065

Imbalance Ratio

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (81 algs)

(c) Imbalance Ratio

10000

20000

30000

40000

50000

Number of Labels

0.60

0.65

0.70

0.75

Mean Average Precision

Random (1 alg)

Uncertainty (80 algs)

GALAXY (80 algs)

MLP (80 algs)

EMAL (1 alg)

ALBL Meta (161 algs)

(a) Mean Average Precision

10000

20000

30000

40000

50000

Number of Labels

20

40

60

80

100

Number of Positive Labels

Random (1 alg)

Uncertainty (80 algs)

GALAXY (80 algs)

MLP (80 algs)

EMAL (1 alg)

ALBL Meta (161 algs)

10000

20000

30000

40000

50000

Number of Labels

0.0032

0.0034

0.0036

0.0038

0.0040

0.0042

0.0044

0.0046

Imbalance Ratio

Random (1 alg)

Uncertainty (80 algs)

GALAXY (80 algs)

MLP (80 algs)

EMAL (1 alg)

ALBL Meta (161 algs)

(c) Imbalance Ratio

2000

4000

6000

8000

10000

Number of Labels

0.75

0.80

0.85

0.90

0.95

1.00

Mean Average Precision

Random (1 alg)

Uncertainty (20 algs)

GALAXY (20 algs)

MLP (20 algs)

EMAL (1 alg)

ALBL Meta (41 algs)

(a) Mean Average Precision

2000

4000

6000

8000

10000

Number of Labels

50

100

150

200

250

Number of Positive Labels

Random (1 alg)

Uncertainty (20 algs)

GALAXY (20 algs)

MLP (20 algs)

EMAL (1 alg)

ALBL Meta (41 algs)

(b) Number of Labels in Rarest Class

2000

4000

6000

8000

10000

Number of Labels

0.07

0.08

0.09

0.10

0.11

0.12

0.13

Imbalance Ratio

Random (1 alg)

Uncertainty (20 algs)

GALAXY (20 algs)

MLP (20 algs)

EMAL (1 alg)

ALBL Meta (41 algs)

(c) Imbalance Ratio

2000

4000

6000

8000

10000

Number of Labels

0.5

0.6

0.7

0.8

0.9

1.0

Mean Average Precision

Random (1 alg)

Uncertainty (10 algs)

GALAXY (10 algs)

MLP (10 algs)

EMAL (1 alg)

ALBL Meta (21 algs)

(a) Mean Average Precision

2000

4000

6000

8000

10000

Number of Labels

100

200

300

400

500

600

Number of Positive Labels

Random (1 alg)

Uncertainty (10 algs)

GALAXY (10 algs)

MLP (10 algs)

EMAL (1 alg)

ALBL Meta (21 algs)

2000

4000

6000

8000

10000

Number of Labels

0.18

0.20

0.22

0.24

0.26

0.28

Imbalance Ratio

Random (1 alg)

Uncertainty (10 algs)

GALAXY (10 algs)

MLP (10 algs)

EMAL (1 alg)

ALBL Meta (21 algs)

(c) Imbalance Ratio

D.2 Multi-class Classification

5000

10000 15000 20000 25000 30000 35000 40000

Number of Labels

0.75

0.80

0.85

0.90

0.95

1.00

Balanced Accuracy

Random (1 alg)

Confidence (1 alg)

GALAXY (2 algs)

MLP (2 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (7 algs)

(a) Balanced Accuracy

5000

10000 15000 20000 25000 30000 35000 40000

Number of Labels

1000

2000

3000

4000

5000

Number of Positive Labels

Random (1 alg)

Confidence (1 alg)

GALAXY (2 algs)

MLP (2 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (7 algs)

(b) Number of Labels in Rarest Class

5000

10000 15000 20000 25000 30000 35000 40000

Number of Labels

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Imbalance Ratio

Random (1 alg)

Confidence (1 alg)

GALAXY (2 algs)

MLP (2 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (7 algs)

(c) Imbalance Ratio

5000

10000

15000

20000

25000

30000

35000

40000

Number of Labels

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Balanced Accuracy

Random (1 alg)

Confidence (1 alg)

GALAXY (10 algs)

MLP (10 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (23 algs)

(a) Balanced Accuracy

5000

10000

15000

20000

25000

30000

35000

40000

Number of Labels

100

200

300

400

500

Number of Positive Labels

Random (1 alg)

Confidence (1 alg)

GALAXY (10 algs)

MLP (10 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (23 algs)

(b) Number of Labels in Rarest Class

5000

10000 15000 20000 25000 30000 35000 40000

Number of Labels

0.010

0.012

0.014

0.016

0.018

0.020

0.022

0.024

Imbalance Ratio

Random (1 alg)

Confidence (1 alg)

GALAXY (10 algs)

MLP (10 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (23 algs)

(c) Imbalance Ratio

10000

20000

30000

40000

50000

60000

70000

Number of Labels

0.90

0.92

0.94

0.96

0.98

1.00

Balanced Accuracy

Random (1 alg)

Confidence (1 alg)

GALAXY (2 algs)

MLP (2 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (7 algs)

(a) Balanced Accuracy

10000

20000

30000

40000

50000

60000

70000

Number of Labels

1000

2000

3000

4000

5000

Number of Positive Labels

Random (1 alg)

Confidence (1 alg)

GALAXY (2 algs)

MLP (2 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (7 algs)

(b) Number of Labels in Rarest Class

10000

20000

30000

40000

50000

60000

70000

Number of Labels

0.08

0.10

0.12

0.14

0.16

0.18

0.20

0.22

0.24

Imbalance Ratio

Random (1 alg)

Confidence (1 alg)

GALAXY (2 algs)

MLP (2 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (7 algs)

(c) Imbalance Ratio

2000

4000

6000

8000

10000

Number of Labels

0.6

0.7

0.8

0.9

1.0

Balanced Accuracy

Random (1 alg)

Confidence (1 alg)

GALAXY (49 algs)

MLP (49 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (101 algs)

(a) Balanced Accuracy

2000

4000

6000

8000

10000

Number of Labels

5

10

15

20

25

30

35

Number of Positive Labels

Random (1 alg)

Confidence (1 alg)

GALAXY (49 algs)

MLP (49 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (101 algs)

(b) Number of Labels in Rarest Class

2000

4000

6000

8000

10000

Number of Labels

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Imbalance Ratio

Random (1 alg)

Confidence (1 alg)

GALAXY (49 algs)

MLP (49 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (101 algs)

(c) Imbalance Ratio

2500

5000

7500

10000

12500

15000

17500

20000

Number of Labels

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Balanced Accuracy

Random (1 alg)

Confidence (1 alg)

GALAXY (256 algs)

MLP (256 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (515 algs)

2500

5000

7500

10000

12500

15000

17500

20000

Number of Labels

0

10

20

30

40

50

60

Number of Positive Labels

Random (1 alg)

Confidence (1 alg)

GALAXY (256 algs)

MLP (256 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (515 algs)

(a) Balanced Accuracy

(b) Number of Labels in Rarest Class

2500

5000

7500

10000 12500 15000 17500 20000

Number of Labels

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Imbalance Ratio

Random (1 alg)

Confidence (1 alg)

GALAXY (256 algs)

MLP (256 algs)

BADGE (1 alg)

Modified Submodular (1 alg)

ALBL Meta (515 algs)

(a) Imbalance Ratio

D.3 Multi-label Search

20000

40000

60000

80000

100000

Number of Labels

100000

200000

300000

400000

500000

600000

700000

800000

900000

Number of Positive Labels

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (122 algs)

10000

20000

30000

40000

50000

Number of Labels

20000

40000

60000

80000

100000

120000

140000

160000

Number of Positive Labels

Random (1 alg)

Uncertainty (80 algs)

GALAXY (80 algs)

MLP (80 algs)

EMAL (1 alg)

ALBL Meta (242 algs)

2000

4000

6000

8000

10000

Number of Labels

2000

4000

6000

8000

10000

12000

14000

Number of Positive Labels

Random (1 alg)

Uncertainty (20 algs)

GALAXY (20 algs)

MLP (20 algs)

EMAL (1 alg)

ALBL Meta (62 algs)

D.4 Rarest Class Accuracy

We conduct an ablation study of the accuracy on the rarest class (determined by the ground truth class distribution). TAILOR significantly outperform baselines.

2000

4000

6000

8000

10000

Number of Labels

2000

4000

6000

8000

10000

12000

Number of Positive Labels

Random (1 alg)

Uncertainty (10 algs)

GALAXY (10 algs)

MLP (10 algs)

EMAL (1 alg)

ALBL Meta (32 algs)

20000

40000

60000

80000

100000

Number of Labels

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

Average Precision of Rarest Class

Random (1 alg)

Uncertainty (40 algs)

GALAXY (40 algs)

MLP (40 algs)

EMAL (1 alg)

ALBL Meta (81 algs)

E What Algorithms Does TAILOR Choose?

In the following two figures, we can see TAILOR chooses a non-uniform set of algorithms to focus on for each dataset. On CelebA, TAILOR out-perform the best baseline, EMAL sampling, by a significant margin. As we can see, TAILOR rely on selecting a combination of other candidate algorithms instead of only selecting EMAL.

On the other hand, for the Stanford car dataset, we see TAILOR 's selection mostly align with the baselines that perform well especially in the later phase.

0

5000

10000

15000

20000

Number of Selections

GALAXY, Class #14

GALAXY, Class #4

GALAXY, Class #29

Uncertainty, Class #13

Uncertainty, Class #30

Uncertainty, Class #14

Uncertainty, Class #26

Uncertainty, Class #22

Uncertainty, Class #29

Uncertainty, Class #4

Algorithms

0

200

400

600

800

1000

1200

Number of Selections

GALAXY, Class #4

Uncertainty, Class #6

Uncertainty, Class #5

Uncertainty, Class #2

EMAL

Uncertainty, Class #3

Uncertainty, Class #8

Uncertainty, Class #0

Uncertainty, Class #1

Uncertainty, Class #4

Algorithms

In the following figures, we plot the number of times the most frequent candidate algorithm is chosen. As can be shown, TAILOR chooses candidate algorithm much more aggressively than other meta algorithms in eight out of the ten settings.

5000

10000

15000

20000

25000

30000

35000

Total Number of Selection

0

2500

5000

7500

10000

12500

15000

17500

20000

#Selection of Most Frequent Algorithm

ALBL Meta (7 algs)

5000

10000

15000

20000

25000

30000

35000

Total Number of Selection

0

1000

2000

3000

4000

5000

6000

7000

8000

#Selection of Most Frequent Algorithm

ALBL Meta (23 algs)

10000

20000

30000

40000

50000

60000

Total Number of Selection

5000

10000

15000

20000

25000

30000

35000

#Selection of Most Frequent Algorithm

ALBL Meta (7 algs)

20000 40000 60000 80000100000120000140000160000180000

Total Number of Selection

100

200

300

400

500

600

700

#Selection of Most Frequent Algorithm

ALBL Meta (2003 algs)

1000

2000

3000

4000

5000

6000

7000

8000

9000

Total Number of Selection

0

100

200

300

400

500

600

700

#Selection of Most Frequent Algorithm

ALBL Meta (101 algs)

2500

5000

7500

10000

12500

15000

17500

Total Number of Selection

20

40

60

80

100

120

140

#Selection of Most Frequent Algorithm

ALBL Meta (515 algs)

10000 20000 30000 40000 50000 60000 70000 80000 90000

Total Number of Selection

0

2500

5000

7500

10000

12500

15000

17500

20000

#Selection of Most Frequent Algorithm

ALBL Meta (83 algs)

5000

10000 15000 20000 25000 30000 35000 40000 45000

Total Number of Selection

0

500

1000

1500

2000

2500

3000

#Selection of Most Frequent Algorithm

ALBL Meta (163 algs)

2000

4000

6000

8000

Total Number of Selection

0

100

200

300

400

500

600

700

#Selection of Most Frequent Algorithm

ALBL Meta (43 algs)

2000

4000

6000

8000

Total Number of Selection

0

200

400

600

800

1000

1200

1400

#Selection of Most Frequent Algorithm

ALBL Meta (23 algs)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm called TAILOR  that adaptively chooses the active learning algorithm from the candidate set of multiple algorithms.  Novel reward functions are proposed that encourage choosing the algorithm such that both class diversity, as well as informativeness, are maintained while selecting unlabeled samples for the annotation. A strong theoretical justification is provided to demonstrate the effectiveness of the proposed TAILOR algorithm using regret analysis. The extensive evaluation conducted on various multiclass (imbalanced) and multilabel datasets demonstrates the ability of the proposed technique to produce comparable/better accuracy compared to the data-specific best candidate active learning strategy.\n\nSTRENGTHS:\n1. This paper has unified the strength of existing active learning algorithms into the single TAILOR framework. This avoids the burden of finding an effective active learning algorithm for a given specific problem.\n2. The authors have provided sufficient novelty in this paper. Specifically, considering the deep active learning setting, the reward function design along with the bandit setting seems novel and non-trivial to me.\n3. The extensive experimentation is conducted considering multiple datasets under  multiclass and multilabel settings. This enhances the credibility of the effectiveness of the proposed TAILOR framework.\n4. The theoretical analysis showing better regret bounds for the proposed TAILOR algorithm compared to linear contextual bandits seems to be a very important contribution.\n\nWEAKNESSES:\n1. The experimentation is conducted using a single ResNet-18 architecture. Having experimental results on multiple architectures, especially more powerful ones, such as ResNet101, ViT would be helpful to even strengthen the effectiveness of the proposed technique. \n2. The impact of \\gamma (line 281)  is missing. I would expect to see the performance variation with respect to different \\gamma values (some ablation study would be helpful).\n3. One of the crucial components is the reward that is designed to maximize the participation of the data samples from the minority (rarest) classes in an imbalanced data setting. It would be interesting to see how the involvement of those minority (rarest) class data samples affects minority class performance. An ablation study comparing the rarest class performance of the proposed TAILOR framework with regard to different baselines would be interesting.\n\nQUESTIONS:\n1. In the rebuttal, it would be interesting to see experimental results mentioned in the weakness.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper proposes a new method for algorithm selection in active learning. The problem is treated as a multi-armed bandit problem and the proposed method is based on Thompson sampling. Extensive experiments are conducted on several datasets, including both multi-class and multi-label classification setting.

STRENGTHS:
1. The paper is well-written and easy to follow.
2. The new reward functions designed for label balance are interesting. 
3. The authors provide regret analysis and the proposed method enjoys better regret bound than the standard linear contextual bandit algorithm.
4. The experimental results are comprehensive, including several benchmarks and two settings. TAILOR is comparable to the best candidate algorithm and even performs better than the best candidate for multi-label setting.

WEAKNESSES:
1. My major concern is the motivation behind active learning algorithm selection. For all the multi-class datasets, BADGE is almost always the best algorithm. Are there any datasets where BADGE does not perform well?
2. What's the time efficiency of TAILOR? Is TAILOR less efficient than the single algorithm?

Minor:
The figure is somewhat difficult to read the specific numbers. It would be better to provide an additional table.

QUESTIONS:
1. Are there any datasets where BADGE does not perform well?
2. Can BADGE be adapted for multi-label setting?
3. What's the time efficiency of TAILOR? Is TAILOR less efficient than the single algorithm?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed TAILOR, a Thompson Sampling framework for active learning algorithm selection for unlabeled, possibly imbalanced datasets by framing it as a multi-arm bandit problem. The authors compared with random Meta and another meta-framework ALBL, along with other AL methods on 10 multi-class multi-label vision datasets and concluded their framework new art.\n\nSTRENGTHS:\nOriginality:\n\nThe work is deemed original based on its discussion with related work.\n\nClarity & quality:\n\nThe work is well written with great graphics like Figure 1 to illustrate key ideas. The appendix is also very helpful information. \n\nSignificance:\n\nThe work is relevant to the broader community of active learning and computer vision. The technical contribution it makes is solid in terms that the TAILOR does show improved AL efficacy compared with individual AL methods.\n\nWEAKNESSES:\nI have a suggestion for clarity: Add / explain what is the pool of AL algorithms that TAILOR considers at the very beginning of the paper (perhaps the intro part) as the plot shows 81 algs are considered (which can raise confusion for readers)\n\nMy major concern about the paper lies in the necessity of the different pieces of the reward function. I think an ablation study on the three different types of rewards function would do great help towards readers' understanding of relative contributions of their modules.\n\nQUESTIONS:\n1. As an active learning practitioner, my experience with active learning strongly signifies a non-stationarity in the effectiveness of different AL methods at different label levels (they are also path dependent, not only the number of labels but which labels were annotated till this iteration matters for the next round reward). Therefore I would like to ask how the authors bake this non-stationarity into the current formation.\n\n2. What exactly are the 40 algs in the uncertainty (40 algs)? I didn't find it in Appendix F.\n\nLIMITATIONS:\nNo potential negative social impact of this work is observed. The authors are encouraged to address their limitations more.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSelecting the most appropriate active learning algorithm for a given dataset poses a significant challenge when applying active learning in real-world scenarios. This challenge stems from the fact that the performance of different active learning strategies could vary significantly across various scenarios and datasets. The paper introduces an interactive and adaptive algorithm selection strategy inspired by the concept of multi-armed bandit. By employing carefully designed reward functions, the proposed strategy can identify a subset of acquisition functions that yield maximum rewards. The authors prove that their selection algorithm exhibits a tighter bound on the Bayesian regret. Experimental evaluations conducted on both multi-class and multi-label classification tasks illustrate the promising performance of the proposed approach in terms of accuracy.\n\nSTRENGTHS:\n* Given the diverse nature of datasets, it is crucial to carefully consider and evaluate various active learning algorithms to achieve optimal performance and effectiveness. The concept of reducing the selection of acquisition functions to a multi-armed bandit problem is interesting as it allows the election process to incorporate accumulated rewards, providing a valuable approach for algorithm selection.\n* The proposed method demonstrates adaptability to both multi-class and multi-label classification tasks by utilizing well-designed reward functions that suitably capture the task-specific requirements.\n* Additionally, the authors present a proof establishing that the proposed selection algorithm outperforms an existing algorithm with a higher bound on Bayesian regret, although the reviewer did not thoroughly examine the soundness of the proof.\n\nWEAKNESSES:\n* The class imbalance issue is dealt with through a proper designed reward function, discussed in section 4.1. The idea of the inversely weighting each class based on its samples is not supervising, given that the weight scheme has been widely used in imbalanced semi supervised learning. The main concern is rather the empirical studies. Even though Figure 3(c) seems to show that the proposed method could deal with the imbalance problem, there is lack of studies about how tolerate the proposed method to different class imbalance ratios. Instead of showing the number of samples in the rarest class, it is better to plot the imbalance ratio. Furthermore, it is not clear how the imbalance issue is handled in the multi-label classification tasks.\n* With respect to Assumption 3.1, it basically assumes to choose the top B samples ranked based on, for example uncertainty/least confidence scores. However, there are other methods, in particularly, some diversity-based AL methods, that choose the samples using clustering, which cannot be directly converted into an iterative selection process.\n* The efficiency aspect of the proposed method is not adequately discussed in the paper. Consideration of efficiency factors, such as computational cost or time complexity, would provide a more holistic evaluation of the proposed method's practicality and real-world applicability.\n\nQUESTIONS:\n* For each iteration of acquisition, B samples are selected from B acquisition strategies, it sounds like an ensemble approach, doesn’t it? Different acquisition functions might capture, for example, different types of uncertainties, some to do with the model parameters, some to do directly with the model performance. Will this contribute to the performance difference?\n* The reward function focuses on the class diversity in order to deal with the class imbalanced issue. It would be interesting to design some reward function that is closely related to the model performance.\n* It is good to show the higher bound. However, can the author prove the convergence of the classifier, as done in WMOCU (Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xiaoning Qian. Uncertaintyaware active learning for optimal Bayesian classifier. In International Conference on Learning Representations, ICLR 2021, 2021)\n* In figure 3, there is no substantial difference between TAILOR and some other methods, including BADGE, can the author explain this? Meanwhile, BADGE is a diversity-based method using clustering, how this can be adopted in TAILOR with Assumption 3.1?\n\nLIMITATIONS:\nThe author briefly touches on the limitations in section 6.3 and section 7. As mentioned above, it is also good to discuss more about the computational cost which could potential benefit the AL practitioners.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm, TAILOR, that iteratively and adaptively selects candidate active learning algorithm to gather class-balanced examples. Experimental results demonstrate that TAILOR achieves comparable or better performance than the best of the candidate algorithms.\n\nSTRENGTHS:\nThe paper presents a new active learning algorithm and provides some theoretical analysis. Good empirical results are reported.\n\nWEAKNESSES:\nThe presentation of the paper leaves much to be desired, and its contribution appears to be quite limited. Additionally, the paper contains improper or insufficiently supported claims, such as “meta algorithm”, “focus on class imbalanced setting”, and “the first adaptive algorithm selection strategy”. \n\n\n1. The organization and content selection are unsatisfactory. \n\n(1) The introduction is not well motivated or presented. The framework illustrated in Figure 1 is an adaptive active learning procedure, and it is difficult to build connection with a multi-armed bandit problem from such a figure. It is not very proper to report a result figure in the introduction as well. \n\n(2) Algorithm 1 is not the contribution of this paper but was included on page 4 with a lot of space.  Since most contents of section 3.2 are from the literature, the presentation can be largely compressed. \n\n(3) Much space of the paper is not on the main contribution of the paper. The proposed algorithm TAILOR is only presented in 4.3.  \n\n(4) Throughout the overall paper (in particular section 4), method presentation, previous work review, and contribution claims are jumbled together without a clear and logical structure of presentation.\n\n\n\n2. The presentation of the paper lacks sufficient clarity.   \n\n(1) In 4.2, the notations and concepts are not well explained. For example, what is the 1-sub-Gaussian distribution?  What does it mean by stating “Nature reveals weighting vector v^t”?  \n\n(2) Section 4.2 also fails to present a clear and principled connection of the proposed active learning setting to the bandit problem. \n\n(3) The Equations are not well explained. For example, the “\\lor” operator in the class diversity reward is not described. \n\n\n\n3. Authors claim the algorithm as a meta algorithm. The TAILOR algorithm in Algorithm 2 didn’t show any meta-learning process. The meta concept needs to be clarified. \n\n\n\n4. The paper claims on focusing on class imbalanced setting,  but  fails to provide any substantial discussion or analysis on this aspect, except for conducting experiments on datasets with class-imbalance. \n\n\n\n5. The reward functions proposed in 4.1 are heuristic and lack principled connections to the model performance. There are no discussions on how to choose reward functions on different datasets.   \n\n\n\n6. The claim of “the first adaptive algorithm selection strategy for deep active learning” is not proper. Adaptive active learning has been studied in the active learning literature. Most active learning strategies developed in the literature are not dependent on the classifiers, and hence there is no need to distinguish linear models from deep models from the active learning perspective. In particular, the proposed active learning method has no special ties to deep models. \n\n\n\n7. Many related works on adaptive active learning or meta active learning are missing, e.g., [1,2,3]. \n[1] “Adaptive Active Learning for Image Classification”. 2013. \n[2] “Active Learning with Multi-label SVM Classification”. 2013. \n[3] “Meta-Learning for Batch Mode Active Learning”, 2018.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}