Title: How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budget
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Active learning

Low budget

High budget

Deep learning
Abstract: In the domain of Active Learning (AL), a learner actively selects which unlabeled examples to seek labels from an oracle, while operating within predefined budget constraints. Importantly, it has been recently shown that distinct query strategies are better suited for different conditions and budgetary constraints. In practice, the determination of the most appropriate AL strategy for a given situation remains an open problem. To tackle this challenge, we propose a practical derivative-based method that dynamically identifies the best strategy for a given budget. Intuitive motivation for our approach is provided by the theoretical analysis of a simplified scenario. We then introduce a method to dynamically select an AL strategy, which takes into account the unique characteristics of the problem and the available budget. Empirical results showcase the effectiveness of our approach across diverse budgets and computer vision tasks.

How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budget

Abstract

In the domain of Active Learning (AL), a learner actively selects which unlabeled examples to seek labels from an oracle, while operating within predefined budget constraints. Importantly, it has been recently shown that distinct query strategies are better suited for different conditions and budgetary constraints. In practice, the determination of the most appropriate AL strategy for a given situation remains an open problem. To tackle this challenge, we propose a practical derivative-based method that dynamically identifies the best strategy for a given budget. Intuitive motivation for our approach is provided by the theoretical analysis of a simplified scenario. We then introduce a method to dynamically select an AL strategy, which takes into account the unique characteristics of the problem and the available budget. Empirical results showcase the effectiveness of our approach across diverse budgets and computer vision tasks.

1 Introduction

Active learning emerged as a powerful approach for promoting more efficient and effective learning outcomes. In the traditional supervised learning framework, active learning enables the learner to actively engage in the construction of the labeled training set by selecting a fixed-sized subset of unlabeled examples for labeling by an oracle, where the number of labels requested is referred to as the budget . Our study addresses the task of identifying in advance the most appropriate active learning strategy for a given problem and budget.

The selection of an active learning strategy is contingent on both the learner's inductive biases and the nature of the problem at hand. But even when all this is fixed, recent research has shown that the most suited active learning strategy varies depending on the size of the budget. When the budget is large, methods based on uncertainty sampling are most effective. When the budget is small, methods based on typicality are most suitable (see Fig. 1). In practice, determining the appropriate active learning strategy based on the budget size is challenging, as a specific budget can be considered either small or large depending on the problem at hand. This challenge is addressed in this paper.

Specifically, we start by analyzing a simplified theoretical framework (Section 2), for which we can explicitly select the appropriate AL strategy using a derivative-based test. Motivated by the analysis of this model, we propose SelectAL (Section 3), which incorporates a similar derivative-based test to select between active learning strategies. SelectAL aims to provide a versatile solution for any budget, by identifying the budget domain of the problem at hand and picking an appropriate AL method from the set of available methods. SelectAL is validated through an extensive empirical study using several vision datasets (Section 4). Our results demonstrate that SelectAL is effective in identifying the best active learning strategy for any budget, achieving superior performance across all budget ranges.

High budget

Uncertainty sampling

Low budget

Typicality sampling

Focus onrareexamples

Focusontypicalexamples

Relation to prior work. Active learning has been an active area of research in recent years [32, 34]. The traditional approach to active learning, which is prevalent in deep learning and recent work, focuses on identifying data that will provide the greatest added value to the learner based on what it already knows. This is typically achieved through the use of uncertainty sampling [11, 24, 31, 36], diversity sampling [8, 20, 33, 35], or a combination of both [1, 10, 22].

However, in small-budget settings, where the learner has limited prior knowledge and effective training is not possible before the selection of queries, such active learning methods fail [4, 27]. Rather, [13, 30, 37] have shown that in this domain, a qualitatively different family of active learning methods should be used. These methods are designed to be effective in this domain and tend to seek examples that can be easily learned rather than confusing examples [5, 25, 39].

With the emergence of this distinction between two separate families of methods to approach active learning, the question arises as to which approach should be preferred in a given context, and whether it is possible to identify in advance which approach would be most effective. Previous research has explored methods for selecting between various AL strategies [2, 19, 29, 40]. However, these studies assumed knowledge of the budget, which is often unrealistic in practice. To our knowledge, SelectALis the first work that addresses this challenge.

We note that several recent works focused on the distinctions between learning with limited versus ample data, but within the context of non-active learning [6, 9, 12, 14, 15, 16, 21, 26]. These studies often highlight qualitative differences in model behavior under varying data conditions. While our results are consistent with these observations, our focus here is to suggest a practical approach to identify the budget regime in advance, allowing for the selection of the most appropriate AL strategy.

2 Theoretical analysis

The aim of this section is to derive a theoretical decision rule that can be translated into a practical algorithm, thus enabling practitioners to make data-driven decisions on how to allocate their budget in active learning scenarios. Given an AL scenario, this rule will select between a high-budget approach, a low-budget approach, or a blend of both. In Section 3, the theoretical result motivates the choice of a decision rule, in a practical method that selects between different active learning strategies.

To develop a test that can decide between high and low-budget approaches for active learning, we seek a theoretically sound framework in which both approaches can be distinctly defined, and show how they can be beneficial for different budget ranges. To this end, we adopt the theoretical framework introduced by [13]. While this framework is rather simplistic, it allows for precise formulation of the distinctions between the high and low-budget approaches. This makes it possible to derive a precise decision rule for the theoretical case, giving some insights for the practical case later on.

In Section 2.1, we establish the necessary notations and provide a brief summary of the theoretical framework used in the analysis, emphasizing the key assumptions and highlighting the results that are germane to our analysis. We then establish in Section 2.2 a derivative-based test, which is a novel contribution of our work. In Section 2.3, we utilize this test to derive an optimal active learning strategy for the theoretical framework. To conclude, in Section 2.4 we present an empirical validation of these results, accompanied by visualizations for further clarification.

2.1 Preliminaries

Notations. We consider an active learning scenario where a learner, denoted by L , is given access to a set of labeled examples L , and a much larger pool of unlabeled examples U . The learner's task is to select a fixed-size subset of the unlabeled examples, referred to as the active set and denoted by A ⊆ U , and obtain their labels from an oracle. These labeled examples are then used for further training of the learner. The goal is to optimally choose the active set A , such that the overall performance of learner L trained on the combined labeled dataset T = A ∪ L is maximal.

We refer to the total number of labeled examples as the budget , denoted by B = | T | = | A ∪ L | . This concept of budget imposes a critical constraint on the active learning process, as the learner must make strategic selections within the limitations of the budget.

Model definition and assumptions. We now summarize the abstract theoretical framework established by [13], which is used in our analysis. While this framework contains simplistic assumptions, its insights are shown empirically to be beneficial in real settings. This framework considers two independent general learners L low and L high , each trained on a different data distribution D low and D high respectively. Intuitively, one can think of D low as a distribution that is easier to learn than D high , such that L low requires fewer examples than L high to achieve a similar generalization error.

The number of training examples each learner sees may vary between the learners. To simplify the analysis, the framework assumes that data naturally arrives from a mixture distribution D , in which an example is sampled from D low with probability p and from D high with probability 1 -p .

We examine the mean generalization error of each learner denoted by E low , E high : R → [0 , 1] respectively, as a function of the number of training examples it sees. The framework makes several assumptions about the form of these error functions: (i) Universality: Both E low and E high take on the same universal form E ( x ) , up to some constant α > 0 , where E ( x ) = E low ( x ) = E high ( αx ) . (ii) Efficiency: E ( x ) is continuous and strictly monotonically decreasing, namely, on average each learner benefits from additional examples. (iii) Realizability: lim x →∞ E ( x ) = 0 , namely, given enough examples, the learners can perfectly learn the data.

To capture the inherent choice in active learning, a family of general learners is considered, each defined by a linear combination of L low and L high . [13] showed that when fixing the number of examples available to the mixture model, i.e, fixing the budget B , it is preferable to sample the training data from a distribution that differs from D . Specifically, it is preferable to skew the distribution towards D low when the budget is low and towards D high when the budget is high.

2.2 Derivative-based query selection decision rule

We begin by asking whether it is advantageous to skew the training distribution towards the low or high-budget distribution. Our formal analysis gives a positive answer. It further delivers a derivativebased test, which can be computed in closed form for the settings above. Importantly, our empirical results (Section 4) demonstrate the effectiveness of an approximation of this test in deep-learning scenarios, where the closed-form solution could not be obtained.

We begin by defining two pure query selection strategies - one that queries examples only from D low , suitable for the low-budget regime, and a second that queries examples only from D high , suitable for the high-budget regime. Given a fixed budget B , we analyze the family of strategies obtained by a linear combination of these pure strategies, parameterized by q ∈ [0 , 1] , in which qB points are sampled from D low and (1 -q ) B points are sampled using D high .

The mean generalization error of combined learner L on the original distribution D , where L is trained on B examples picked by a mixed strategy q , is denoted E L ( B,q ) = p · E ( qB )+(1 -p ) · E ( α (1 -q ) B ) , as the combined learner simply sends qB examples to L low and (1 -q ) B examples to L high . By differentiating this result (see derivation in Suppl. A), we can find an optimal strategy for this family, and denote it by ˆ q . This strategy is defined as the one that delivers the lowest generalization error when using a labeled set of size B . The strategy is characterized by ˆ q ∈ [0 , 1] , implicitly defined as follows:

If Eq. (1) has a unique and minimal solution for ˆ q , it defines an optimal mixed strategy ˆ q E ( B,p,α ) for a given set of problem parameters { B,p,α } . As p and α are fixed for any specific problem, we get that different budgets B may require different optimal AL strategies. Notably, if budget B satisfies ˆ q E ( B,p,α ) = p , then the optimal strategy for it is equivalent 1 to selecting samples directly from the original distribution D . We refer to such budgets as B eq . With these budgets, active learning is no longer helpful. While B eq may not be unique, for the sake of simplicity, we assume that it is unique in our analysis, noting that the general case is similar but more cumbersome.

We propose to use Eq. (1) as a decision rule, to determine whether a low-budget or high-budget strategy is more suitable for the current budget B . Specifically, given budget B , we can compute ˆ q E ( B,p,α ) and B eq in advance and determine which AL strategy to use by comparing B and B eq . In the current framework, this rule is guaranteed to deliver an optimal linear combination of strategies, as we demonstrate in Section 2.3. Visualization of the model's error as a function of the budget, for different values of q , can be seen in Fig. 2a.

improvement

1.0

0.01

0.15

0.8

0.3

0

0.3

0.5

Optimal

0.6

1

Bequiv

Performancei

0.6

0.8

0.4

1

Bequiv

0.2

0.0

101

102

103

101

102

103

Number of examples (B)

Number of examples (B)

(a) Accuracy gain of mixed strategies q , where mixture coefficient q is indicated in the legend.

(b) Optimal

ˆ

q

a

(

B,p,α

)

.

2.3 Best mixture of active learning strategies

With the aim of active learning (AL) in mind, our objective is to strategically select the active set A in order to maximize the performance of learner L when trained on all available labels T = A ∪ L . To simplify the analysis, we focus on the initial AL round, assuming that L is sampled from the underlying data distribution D . The analysis of subsequent rounds can be done in a similar manner.

To begin with, let us consider the entire available label set T = A ∪ L . Since T comprises examples from D , D low and D high , we can represent it using the (non-unique) notation S ( r rand , r low , r high ) , where r rand , r low , r high denote the fractions of T sampled from each respective distribution. Based on the definitions of q and ˆ q , we can identify an optimal combination for T , denoted T ∗ :

Attainable optimal mixed strategy. It is important to note that T = A ∪ L , where the active learning (AL) strategy can only influence the sampling distribution of A . Consequently, not every combination of T is attainable. In fact, the feasibility of the optimal combination relies on the size of the set A . Specifically, utilizing (1), we can identify two thresholds, B low and B high , such that if B < B low ≤ B eq or B > B high ≥ B eq , it is not possible to achieve the optimal T ∗ = A ∗ ∪ L . The derivation of both thresholds can be found in Suppl. A.

|A|

B

=0.01

Errorreduction

Y

=0.3

B

2

1

0

0

25

50

75

100

125

150

Numberofexamples(B)

(a) CIFAR-10

(b) CIFAR-100

Random

S(0.9,0,0.1)

S(0.5,0,0.5)

5(0.1,0,0.9)

Random

S(0.7,0.3,0)

S(0.3,0.7,0)

S(0.9,0.1,0)

S(0.5,0.5,0)

S(0.1,0.9,0)

S(0.9,0.1,0)

S(0.5,0.5,0)

S(0.1,0.9,0)

Accuracyimprovement

Accuracyimprovement

4

3

3

2

2

1

1

0

D

1

103

104

103

104

Numberofexamples(B)

Numberofexamples(B)

With (2) and the thresholds above, we may conclude that the optimal combination of A ∗ is:

In other words, we observe that the optimal strategy (2) is only attainable in non-extreme scenarios. Specifically, in cases of very low budgets ( B < B low ), it is optimal to sample the active set purely from D low because more than | A | points from D low are needed to achieve optimal performance. Similarly, in situations of very high budgets ( B > B high ), it is optimal to select the active set solely from D high since more than | A | points from D high are necessary for optimal performance. We note that even if B eq is not unique, in the limit of | A |→ 0 , (3)-(4) are still locally optimal, segment by segment.

Small active set A . Upon examining the definitions of B low and B high (refer to Suppl. A), we observe that lim | A |→ 0 B low = lim | A |→ 0 B high = B eq . Consequently, as indicated in (4), the optimal mixed strategy, in this scenario, actually becomes a pure strategy. When the budget is low, the entire active set A should be sampled from D low . Similarly, when the budget is high, A should be sampled solely from D high . In the single point where B low = B high = B eq , A should be sampled from D . Fig. 2b provides a visualization of these strategies as the size of A decreases.

Iterative querying. When | A | is not small enough to justify the use of the limiting pure strategy (4), we propose to implement query selection incrementally. By repeatedly applying (4) to smaller segments of length m , we can iteratively construct A , with each iteration becoming more computationally feasible. This strategy not only enhances robustness but also yields comparable performance to the mixed optimal strategy (3), as demonstrated in Section 2.4.

Based on the analysis presented above, it becomes apparent that in practical scenarios, the optimal combination of active learning strategies can be achieved by sequentially sampling from pure strategies , while utilizing a derivative-based test to determine the most effective strategy at each step. This concept forms the core motivation behind the development of the practical algorithm SelectAL, which is presented in Section 3.

2.4 Validation and visualization of theoretical results

Visualization. In Fig. 3, we illustrate the error of the strategies defined in (3) using the same exponential example as depicted in Fig. 2. The orange curve represents a relatively large active set, where | A | is equal to 30% of the budget B , while the blue curve represents a significantly smaller active set, where | A | is equal to 1% of the budget B . It is evident that as the size of A decreases, the discrepancy between the optimal strategy and the attainable strategy diminishes.

Fig. 2b shows the optimal mixture coefficient ˆ q as a function of the budget size B for various values of | A | , namely 1% , 30% , and 100% of B . According to our analysis, as the size of | A | decreases, the optimal ˆ q should exhibit a more pronounced step-like behavior. This observation suggests that in the majority of cases, it is possible to sample the entire active set A from a single strategy rather than using a mixture of strategies.

Validation. Our theoretical analysis uses a mixture model of idealized general learners. We now validate that similar phenomena occur in practice when training deep networks on different computer vision tasks. Since sampling from D low and D high is not feasible in this case, we instead choose low and high-budget deep AL strategies from the literature. Specifically, we choose TypiClust as the low-budget strategy and BADGE as the high-budget strategy, as explained in detail in Section 4.1. We note that other choices for the low and high-budget strategies yield similar qualitative results, as is evident from Tables 1-2. Fig. 4 shows results when training 20 ResNet-18 networks using mixed strategies S (1 -r | T | | A | , 0 , r | T | | A | ) and S (1 -r | T | | A | , r | T | | A | , 0) on CIFAR-10 and CIFAR-100. We compare the mean performance of each strategy to the performance of 20 ResNet-18 networks trained with the random query selection strategy S (1 , 0 , 0) .

Inspecting these results, we observe similar trends to those shown in the theoretical analysis. As the budget increases, the most beneficial value of mixture coefficient r decreases until a certain transition point (corresponding to B eq ). From this transition point onward, the bigger the budget is, the more beneficial it is to select additional examples from one of the high-budget strategies. As in the theoretical analysis, when the budget is low it is beneficial to use a pure low-budget strategy, and when the budget is high it is beneficial to use a pure high-budget strategy. The transition area, corresponding to segment B low ≤ B ≤ B high , is typically rather short (see Figs. 4 and 6).

2.5 Discussion: motivation from the theoretical analysis

The theoretical framework presented here relies on a simplified model, enabling a rigorous analysis with closed-form solutions. While these solutions may not be directly applicable in real-world deep learning scenarios, they serve as a solid motivation for the SelectAL approach. Empirical evidence demonstrates their practical effectiveness.

In Section 2.2, we introduced a derivative-based test, demonstrating that measuring the model's error function on small data perturbations suffices to determine the appropriate strategy for each budget. This concept serves as the fundamental principle of SelectAL, a method that chooses an active learning strategy by analyzing the model's error function in response to data perturbations.

In Section 2.3, we showed that while the optimal mixed AL strategy can be determined analytically, a practical approximation can be achieved by choosing a single pure AL strategy for each round in active learning. This approach significantly simplifies SelectAL, as it doesn't require the creation of a distinct mixed AL strategy tailored separately to each budget. Instead, SelectAL selects a single pure AL strategy for each active learning round, from a set of existing "pure" AL strategies. In Fig. 5, we observe that this selection consistently outperforms individual pure strategies, confirming our analytical predictions. We also explored various combinations of existing AL strategies, none of which showed a significant improvement over SelectAL, further reinforcing the relevance of predictions of the theoretical analysis.

3 SelectAL: automatic selection of active learning strategy

In this section, we present SelectAL - a method for automatically selecting between different AL strategies in advance, by estimating dynamically the relative budget size of the problem at hand. The decision whether B should be considered 'large" or 'small" builds on the insights gained from the theoretical analysis presented in section 2. The suggested approach approximates the derivative-based test suggested in (1), resulting in a variation of the attainable optimal mixed strategy (3).

SelectAL consists of two steps. The first step involves using a version of the derivative-based rule from Section 2.2 to assess whether the current budget B for the given problem should be classified as 'high" ( B ≥ B high ) or 'low" ( B ≤ B low ). This is accomplished by generating small perturbations of L based on both low and high-budget strategies, and then predicting if the current quantity of labeled examples is adequate to qualify as a high-budget or low-budget.

In the second step, we select the most competitive AL strategy from the relevant domain and use it to select the active set A . This approach not only ensures good performance within the specified budget constraints but also provides robustness across a wide range of budget scenarios. It is scalable in its ability to incorporate any future development in active learning methods.

3.1 Deciding on the suitable budget regime

In the first step, SelectAL must determine whether the current labeled set L should be considered low-budget ( B ≤ B low ) or high-budget ( B ≥ B high ) for the problem at hand. The challenge is to make this decision without querying additional labels from U .

To accomplish this, SelectAL requires access to a set of active learning strategies for both low and high-budget scenarios. We denote such strategies by S ′ low and S ′ high , respectively. Additionally, a random selection strategy, denoted by S rand , is also considered.

To determine whether the current budget is high or low, SelectAL utilizes a surrogate test. Instead of requiring additional labels, we compute the result of the test with a derivative-like approach. Specifically, for each respective strategy separately, we remove a small set of points (size ϵ > 0 ) from L , and compare the reduction in generalization error to the removal of ϵ randomly chosen points. To implement this procedure, we remove the active sets chosen by either S ′ low or S ′ high , when trained using the original labeled set L as the unlabeled set and an empty set as the labeled set.

The proposed surrogate test presents a new challenge: in most active learning strategies, particularly those suitable for high budgets, the outcome relies on a learner that is trained on a non-empty labeled set. To overcome this, we restrict the choice of active learning strategies S ′ low and S ′ high to methods that rely only on the unlabeled set U . This added complexity is crucial : using the labels of set L by either S ′ low or S ′ high results in a bias that underestimates the cost of removing known points, as is demonstrated in our empirical study (see Fig. 7). No further restriction on S ′ low and S ′ high is needed. Empirically, we evaluated several options for S ′ low and S ′ high , including TypiClust and ProbCover for S ′ low and inverse-TypiClust and CoreSet for S ′ high , with no qualitative difference in the results.

Our final method can be summarized as follows (see Alg. 1): We generate three subsets from the original labeled set L : data low , data high and data rand . Each subset is obtained by asking the corresponding strategy S ′ low , S ′ high , and S rand - to choose a class-balanced subset of ϵ · | L | examples from L . Note that this is unlike their original use, as AL strategies are intended to select queries from the unlabeled set U . Importantly, since set L is labeled, we can guarantee that the selected subset is class-balanced. Subsequently, the selected set is removed from L , and a separate learner is trained on each of the 3 subsets (repeating the process multiple times for S rand ). Finally, we evaluate the accuracy of each method using cross-validation, employing 1% of the training data as a validation set in multiple repetitions. The subset with the lowest accuracy indicates that the subset lacks examples that are most critical for learning. This suggests that the corresponding strategy queries examples that have the highest impact on performance .

Algorithm 1 SelectAL

1: Input: S ′ high , S ′ low , S high , S low , S rand , labeled data L , ϵ > 0

2: Output: AL strategy

3: c ← max { ⌊ ϵ # of classes ⌋ , 1 }

4: for i ∈ {low, high, rand} do

5: removed_examples ← query c examples from each class from L using S ′ i

6: data i ← L \ removed_examples

7: acc i ← model's accuracy on data i

8: end for

9: strategy ← arg min ( acc i )

i

∈{

low,high,rand

10: return S strategy

}

(a) CIFAR-10

(b) CIFAR-100

3.2 Selecting the active learning strategy

In its second step, SelectAL selects between two active learning strategies, S low , and S high , which are known to be beneficial in the low and high-budget regimes respectively. Unlike S ′ low and S ′ high , there are no restrictions on { S low , S high } , from which Alg. 1 selects, and it is beneficial to select the SOTA active learning strategy for the chosen domain.

Somewhat counter-intuitively, SelectAL is likely to use different pairs of AL strategies, one pair to determine the budget regime, and possible a different one for the actual query selection. This is the case because in its first step, S ′ low and S ′ high are constrained to use only the unlabeled set U , which may eliminate from consideration the most competitive strategies. In contrast, and in order to achieve the best results, S low and S high are chosen to be the most competitive AL strategy in each domain. This flexibility is permitted because both our theoretical analysis in Section 2 and our empirical analysis in Section 4 indicate that the transition points B low and B high are likely to be universal, or approximately so, across different strategies. This is especially important in the high-budget regime, where the most competitive strategies often rely on both L and U .

4 Empirical results

We now describe the results of an extensive empirical evaluation of SelectAL. After a suitable AL strategy is selected by Alg. 1, it is used to query A unlabeled examples. The training of the deep model then proceeds as is customary, using all the available labels in A ∪ L .

4.1 Methodology

Our experimental framework is built on the codebase of [28], which allows for fair and robust comparison of different active learning strategies. While the ResNet-18 architecture used in our experiments may not achieve state-of-the-art results on CIFAR and ImageNet, it provides a suitable platform to evaluate the effectiveness of active learning strategies in a competitive environment, where these strategies have been shown to be beneficial. In the following experiments, we trained ResNet-18 [18] on CIFAR-10, CIFAR-100 [23] and ImageNet-50 - a subset of ImageNet [7] containing 50 classes as done in [38]. We use the same hyper-parameters as in [28], as detailed in Suppl. B.

SelectAL requires two types of active learning strategies: restricted strategies that use only the unlabeled set U for training, and unrestricted competitive strategies that can use both L and U (see discussion above). Among the restricted strategies, we chose TypiClust [13] to take the role of S ′ low , and inverse TypiClust for S ′ high . In the latter strategy, the most atypical examples are selected. Note that inverse TypiClust is an effective strategy for high budgets, while relying solely on the unlabeled set U (see Suppl. C.1 for details). Among the unrestricted strategies, we chose ProbCover [39] for the role of the low-budget strategy S low , and BADGE [1] for the high-budget strategy S high . Other choices yield similar patterns of improvement, as can be verified from Tables 1-2.

In the experiments below, we use several active learning strategies, including Min margin , Max entropy , Least confidence , DBAL [10], CoreSet [33], BALD [22], BADGE [1], TypiClust [13] and ProbCover [39]. When available, we use for each strategy the code provided in [28]. For low-budget strategies, which are not implemented in [28], we use the code from the repository of each paper.

4.2 Evaluating the removal of examples with AL in isolation

We isolate the strategy selection test in Alg. 1 as described in Section 3.1. To generate the 3 subsets of labeled examples data low , data high and data rand , we remove 5% of the labeled data, ensuring that we never remove less than one data point per class. In the low-budget regime, removing examples according to S ′ low yields worse performance as compared to the removal of random examples, while better performance is seen in the high-budget regime. The opposite behavior is seen when removing examples according to S ′ high . Different choices of strategies for S ′ low and S ′ high yield similar qualitative results, see Suppl. C.2.

More specifically, we trained 10 ResNet-18 networks on each of the 3 subsets, for different choices of budget B . In Fig. 6, we plot the difference in the mean accuracy of networks trained on data low and data high , compared to networks trained on data rand , similarly to the proposed test in Alg. 1. In all

the budgets that are smaller than the orange dashed line, SelectAL chooses S low . In budgets between the orange and the green dashed lines, SelectAL chooses S rand . In budgets larger than the green dashed line, SelectAL chooses S high .

(a) CIFAR-10

(b) CIFAR-100

(c) ImageNet-50

Srand

Biow

Bhigh

Sow

Shnigh

Srand

Biow

u6Ng

Slow

十

Snigh

Srand

Biow

Bnigh

+

Siow

Shigh

Accuracyimprovement

4

hprovement

1.5

Accuracyimprovement

1.0

2

0.5

0

0.0

0

2

-0.5

-2

cu

1.0

-1

U

1.5

103

104

103

104

103

104

Number of examples (B)

Number of examples (B)

Number of examples (B)

4.3 SelectAL: results

In Fig. 5, we present the average accuracy of a series of experiments involving 10 ResNet-18 networks trained over 10 consecutive AL rounds using three AL strategies: S low (ProbCover), S high (BADGE), and our proposed method SelectAL. We compare the accuracy improvement of each strategy to training without any AL strategy. In each AL round, SelectAL selects the appropriate AL strategy based on Alg. 1. Specifically, SelectAL selects S low when the budget is below the orange dashed line, S rand when the budget is between the orange and green dashed lines, and S high when the budget is above the green dashed line. We observe that while S low and S high are effective only for specific budgets, SelectAL performs well across all budgets. In all the experiments we performed, across all datasets, we always observed the monotonic behavior predicted by the theory - SelectAL picks S low for several AL iterations, followed by S rand and then S high .

It is important to note that, unlike Fig. 6, where the data distribution of set L is sampled from the original distribution D because we analyze the first round (see Section 2.3), in the current experiments the distribution is unknown apriori L in each iteration is conditioned on the results of previous iterations, and is therefore effectively a combination of S low , S high , and S rand . Consequently, the transition point determined automatically by Alg. 1 occurs earlier than the one detected in Fig. 6. Notably, while SelectAL chooses an existing AL strategy at each iteration, the resulting strategy outperforms each of the individual AL strategies when trained alone.

In Tables 1-2, we show the performance of SelectAL in comparison with the performance of the baselines (Section 4.1). In all these experiments, SelectAL succeeds to identify a suitable budget regime. As a result, it works well both in the low and high-budget regimes, matching or surpassing both the low and high-budget strategies at all budgets. Note that as SelectAL chooses an active learning strategy dynamically for each budget, any state-of-the-art improvements for either low or high budgets AL strategies can be readily incorporated into SelectAL.

Why S ′ low and S ′ high are Restricted? As discussed in Section 3.1, we have made a deliberate decision to exclude strategies that rely on the labeled set L while deciding which family of strategies is more suited to the current budget. We now demonstrate what happens when the selection is not restricted in this manner, and in particular, if S ′ high is chosen to be a competitive AL strategy that relies on the labeled set L for its successful outcome. Specifically, we repeat the experiments whose results are reported in Fig. 6a, but where strategy S ′ high - the one used for the removal of examples is BADGE. Results are shown in Fig. 7. Unlike Fig. 6a, there is no transition point, as it is always beneficial to remove examples selected by BADGE rather than random examples. This may occur because the added value of all points used for training diminishes after training is completed.

Computational time of SelectAL SelectAL entails the training of three active learning strategies on a small dataset. Consequently, the computational duration of SelectAL is contingent upon the user's strategy choices. It is worth noting that, in practice, the additional computation time is negligible in comparison to conventional AL techniques. This is due to the fact that S ′ low and S ′ high are exclusively trained on L , which is typically significantly smaller than U - the set used by pure AL strategies. To further expedite computation, one might contemplate training on data perturbations using a more compact model, an issues which we defer the to future research.

5 Summary and discussion

We introduce SelectAL, a novel method for selecting active learning strategies that perform well for all training budgets, low and high. We demonstrate the effectiveness of SelectAL through a combination of theoretical analysis and empirical evaluation, showing that it achieves competitive results across a wide range of budgets and datasets. Our main contribution lies in the introduction of the first budget-aware active learning strategy. Until now, selecting the most appropriate active learning strategy given some data was left to the practitioner. Knowing which active learning strategy is best suited for the data can be calculated after all the data is labeled, but predicting this in advance is a difficult problem. SelectAL offers a solution to this challenge, by determining beforehand which active learning strategy should be used without using any labeled data.

Acknowledgement This work was supported by grants from the Israeli Council of Higher Education and the Gatsby Charitable Foundations.

References

Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Yoram Baram, Ran El Yaniv, and Kobi Luz. Online choice of active learning algorithms. Journal of Machine Learning Research , 5(Mar):255-291, 2004.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 9630-9640. IEEE, 2021.

Yao-Chun Chan, Mingchen Li, and Samet Oymak. On the marginal benefit of active learning: Does self-supervision eat its cake? In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3455-3459. IEEE, 2021.

Liangyu Chen, Yutong Bai, Siyu Huang, Yongyi Lu, Bihan Wen, Alan L Yuille, and Zongwei Zhou. Making your first choice: To address cold start problem in vision active learning. arXiv preprint arXiv:2210.02442 , 2022.

Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar-learning trajectories of neural language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8281-8297, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Ehsan Elhamifar, Guillermo Sapiro, Allen Yang, and S Shankar Sasrty. A convex optimization framework for active learning. In Proceedings of the IEEE International Conference on Computer Vision , pages 209-216, 2013.

Noam Fluss, Guy Hacohen, and Daphna Weinshall. Semi-supervised learning in the few-shot zero-shot scenario. arXiv preprint arXiv:2308.14119 , 2023.

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning , pages 1183-1192. PMLR, 2017.

Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint arXiv:1907.06347 , 2019.

Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datasets. In International Conference on Machine Learning , pages 3950-3960. PMLR, 2020.

Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. In International Conference on Machine Learning . PMLR, 2022.

Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning , pages 2535-2544. PMLR, 2019.

Guy Hacohen and Daphna Weinshall. Principal components bias in over-parameterized linear models, and its manifestation in deep neural networks. The Journal of Machine Learning Research , 23(1):6973-7018, 2022.

Guy Hacohen and Daphna Weinshall. Pruning the unlabeled data to improve semi-supervised learning. arXiv preprint arXiv:2308.14058 , 2023.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016.

Wei-Ning Hsu and Hsuan-Tien Lin. Active learning by learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Rong Hu, Brian Mac Namee, and Sarah Jane Delany. Off to a good start: Using clustering to select the initial training set in active learning. In Twenty-Third International FLAIRS Conference , 2010.

Achin Jain, Gurumurthy Swaminathan, Paolo Favaro, Hao Yang, Avinash Ravichandran, Hrayr Harutyunyan, Alessandro Achille, Onkar Dabeer, Bernt Schiele, Ashwin Swaminathan, et al. A meta-learning approach to predicting performance and data requirements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3623-3632, 2023.

Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems , 32:7026-7037, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Online , 2009.

David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR'94 , pages 3-12. Springer, 1994.

Rafid Mahmood, Sanja Fidler, and Marc T. Law. Low-budget active learning via wasserstein distance: An integer programming approach. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Rafid Mahmood, James Lucas, Jose M Alvarez, Sanja Fidler, and Marc Law. Optimizing data collection for machine learning. Advances in Neural Information Processing Systems , 35:29915-29928, 2022.

Sudhanshu Mittal, Maxim Tatarchenko, Özgün Çiçek, and Thomas Brox. Parting with illusions about deep active learning. arXiv preprint arXiv:1912.05361 , 2019.

Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 223-232. IEEE, 2022.

Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy M Hospedales. Dynamic ensemble active learning: A non-stationary bandit with expert advice. In 2018 24th International Conference on Pattern Recognition (ICPR) , pages 2269-2276. IEEE, 2018.

Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active learning is a strong baseline for data subset selection. In Has it Trained Yet? NeurIPS 2022 Workshop , 2022.

Hiranmayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Panchanathan. Deep active learning for image classification. In 2017 IEEE International Conference on Image Processing (ICIP) , pages 3934-3938. IEEE, 2017.

Christopher Schröder and Andreas Niekler. A survey of active learning for text classification using deep neural networks. arXiv preprint arXiv:2008.07267 , 2020.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations , 2018.

Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009.

Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. In International Conference on Artificial Intelligence and Statistics , pages 1308-1318. PMLR, 2020.

Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5972-5981, 2019.

Alexandru Tifrea, Jacob Clarysse, and Fanny Yang. Uniform versus uncertainty sampling: When being active is less efficient than staying passive. arXiv preprint arXiv:2212.00772 , 2022.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conference on Computer Vision , pages 268-285. Springer, 2020.

Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active learning through a covering lens. In NeurIPS , 2022.

Jifan Zhang, Shuai Shao, Saurabh Verma, and Robert Nowak. Algorithm selection for deep active learning with imbalanced datasets. arXiv preprint arXiv:2302.07317 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSome recent works have shown that typical examples are more beneficial than more uncertain ones under low budget setting in active learning. However, there has been no prior works that address how to distinguish whether a budget is in low or high budget regime, or how to determine which approach should be taken.\n\nThe authors propose SelectAL, an algorithm that handles the tasks using derivative-based test, for the first time. The algorithm is theoretically and empirically supported in Section 2 and 4, respectively.\n\nSTRENGTHS:\n1. This is the first paper that addresses how to automatically determine selection strategies based on the budget in active learning.\n2. The authors provide some theoretical justification for the derivative-based test method.\n\nWEAKNESSES:\n1. Although Strength 1 is a great contribution of this paper, there are several assumptions some of which may not be realistic. Intuitively, efficiency and realizability are reasonable assumptions but it would be hard to check if universality is generally true or not. Furthermore, the uniqueness of $B_{eq}$ (line 120-121) seems like a strong assumption. \n2. It is somewhat hard to follow the flow of this paper. It is partially because of the complexity of theoretical work, but also because there are many assumptions to justify the theory. But upon the clarification of the questions below, I am willing to increase the score.\n\nQUESTIONS:\n1. I am very confused with the definition of the generalization errors. If my understanding is correct, $E_{low}$ and $E_{high}$ are the generalization errors on the same unseen test set using $\\mathcal{L}_{low}$ and $\\mathcal{L}_{hight}$, each of which are trained on $\\mathcal{D}_{low}$ and $\\mathcal{D}_{high}$. How is it related to the equation in line 111? \n2. What the equation in line 111 is saying is that the generalization error of $\\mathcal{L}$ trained on $B$ examples can be expressed as the convex combination of $E(qB)$ and $E(\\alpha(1-q)B)$. But, $E(qB)$ is from a learner trained only on $qB$ examples and $E(\\alpha(1-q)B)$ is from the other learner trained only on $(1-q)B$ examples. Is it generally true or even close to the reality? If the generalization error is a function of the test examples given the same learner, I think the convex combination makes sense but since it is a function of training examples, I am not sure it is reasonably true. \n3. Where are equalities for q coming from for $B_{low}$ and $B_{high}$ in  Appendix A?\n4. If the uniqueness of $B_{eq}$ does not hold, what happens to the Eq.(3), particularly, for the case $|A| \\rightarrow 0$?\n5. Could the authors provide more details about the experiment setting used to produce Figure 2 such as a linear $\\mathcal{L}$, the dataset, and reasoning behind of choosing exponential error function?\n6. For selectAL, did the authors observe some cases where {low, high, rand} are selected alternatively unlike the expected monotonic behavior? Can the authors provide the sequence of selections on CIFAR-10 and ImageNet-50?\n\nLIMITATIONS:\nPlease refer to the weakness and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the selection of active learning methods under different budgets. Previous studies have shown that different active learning methods perform differently under different labeling budgets, and incorrect selection can lead to model performance inferior to that of random sampling baselines. To address this issue, the authors conducted theoretical analysis and proposed the SelectAL method, and related experiments showed that this method can achieve consistent performance improvements under different budgets.\n\nSTRENGTHS:\n1. The idea presented is interesting and it tackles an important issue in active learning. \n2. The experimental results indicate that the proposal consistently improved performance, supporting the claims made in the paper.\n\nWEAKNESSES:\n1. There is room for improvement in the technological representation.  \n2. The connection between theoretical results and practical methods seems to be weak.  \nMiscellaneous: The text font in the figures should be kept the same as the main body.\n\nQUESTIONS:\n1. What is the physical meaning of $E_{low}(x)=E_{high}(\\alpha x)$ in the assumptions (Section 2.1)? Can you provide a more intuitive explanation?\n2. The theoretical results (Section 2) assume that there is a unique and minimal solution for Eq.(1). What kind of E satisfies this assumption, and is this condition general enough to support practical applications? \n3. The proposed method (Section 3) seems weakly connected to the theoretical analysis. The theoretical results show that under specific assumptions, there exist sampling boundaries B_low and B_high for the Low-Budget Method and High-Budget Method, which can be calculated through closed-form solutions. However, the SelectAL method used in the proposed method relies on a training-evaluating method to identify the type of samples that are currently lacking. Can you provide more connections between theoretical results and practical methods?\n4. In the experiments (Figure 6), there are indeed performance differences between different AL methods, but the differences are small (≤2%). Have you observed more severe phenomena? In Tables 1 and 2, there are similar phenomena, and the performance differences between all methods are not significant. Specifically, in the low-budget scenario, the classification performance is so poor that the model cannot be deployed. Is it necessary to introduce complex methods to deal with this? Perhaps, random sampling followed by the consideration of high-budget methods is already sufficient.\n\n---  \nThanks for the detailed clarifications, which have addressed my concerns. I would like to keep my score.\n\nLIMITATIONS:\nThe authors do not provide a discussion about the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an algorithm selection problem for active learning under different labeling budgets. A theoretical analysis is conducted The proposed algorithm uses a derivative approach by removing small sets of labeled examples to approximate the effectiveness of different active learning algorithms. Empirical results are shown for different labeling budget that SelectAL chooses the best active learning algorithm. Experiments are conducted on CIFAR-10, CIFAR-100 and ImageNet-50.\n\nSTRENGTHS:\n1. The paper studies an important problem of how to choose the best active learning algorithm for a labeling budget.\n2. Experiments demonstrates the effectiveness of the method in specific settings.\n3. It is interesting to empirically see that random sampling performs as good as other active learning algorithms when given a moderate number of labeling budget.\n\nWEAKNESSES:\nI have major concerns with the writing and structure of the paper.\n\n1. The theoretical analysis and the actual algorithm seem to be very loosely connected. Currently, section 3 contains very little motivation from section 2. Also, the analysis framework makes assumption on universality making it unlikely to reflect the performance in practice.\n\n2. The data removal process from a labeled set is not immediate to the reader but is the key underlying tool for the derivate approximation. How does one remove labeled examples from a labeled set using an active learning algorithm?\n\n3. Throughout the paper, the data removal algorithms are used as TypiClust and inverse-TypiClust. Instead of formulating the algorithm as using general active learning algorithms for removal, the authors may discuss more in-depth why these two algorithms are actually effective in predicting low and high budget scenarios.\n\nFurthermore, the experiments seem to be inconsistent. Specifically, even though CIFAR-10 and CIFAR-100 have the same pool size, the experiments are conducted with inconsistent budgets. For example, 7k+1k and 25k+5k are used for CIFAR-10 while 9k+1k and 30k+7k are used for CIFAR-100. Since the paper's focus is on different labeling budgets, I believe more budget settings are also needed beyond just three.\n\nLastly, algorithm selection methods have been studied before, but not for different budgets. The paper could benefit from the following related work:\n1. Baram, Y., Yaniv, R. E., & Luz, K. (2004). Online choice of active learning algorithms. Journal of Machine Learning Research, 5(Mar), 255-291.\n2. Hsu, W. N., & Lin, H. T. (2015, February). Active learning by learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 29, No. 1).\n3. Pang, K., Dong, M., Wu, Y., & Hospedales, T. M. (2018, August). Dynamic ensemble active learning: A non-stationary bandit with expert advice. In 2018 24th International Conference on Pattern Recognition (ICPR) (pp. 2269-2276). IEEE.\n4. (Note the first version came out Feb 2023.) Zhang, J., Shao, S., Verma, S., & Nowak, R. (2023). Algorithm Selection for Deep Active Learning with Imbalanced Datasets. arXiv preprint arXiv:2302.07317.\n\nQUESTIONS:\n1. What's the underlying intuition and reasoning of why TypiClust and inverse-TypiClust can correctly predict the budget scenarios.\n2. Why are the particular initial seed set size and batch size selected for CIFAR-10 and CIFAR-100.\n\nLIMITATIONS:\nSufficient.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe literature has shown that the optimal type of active learning (AL) strategy depends on the type of budget. Methods based on uncertainty sampling are most effective when the budget is large, and methods based on typicality work better in the low budget setting. However, what qualifies \"large\" or \"low\" depends on different criteria such as the data set and its size, the type of neural network architecture that is trained etc. \nThe paper proposes a general approach that automatically determines whether a given setting is low or high budget by decoupling the data set in different parts. The method is divided in two parts. In the first part, a different active learning strategy is used depending on whether a labeled image belongs to the low or high budget setting. In the second part, unlabeled images are added to the labeled set.\n\nSTRENGTHS:\nQuality and clarity: The general motivation of the approach (i.e. combining low and high budget settings and using different strategies for each) is clear and makes sense given the recent AL literature. The first part of the algorithm is clear and more or less described in Algorithm 1. The clarity of the second part could be improved by writing its pseudo-code (in the main paper or appendix).\n\nOriginality: Similar ideas that consider different strategies for different parts of the data set were recently published in the literature (e.g. [A,B]). \nJain et al. [A] show that low-budget settings follow a linear law whereas high-budget settings follow a power law in the standard classification setting. They also try to estimate whether a setting is low or high budget.\nMahmood et al. [B] also consider to learn the optimal budget over multiple datasets (in the multi-variate case) to improve model performance while minimizing labeling cost. This is related to the case where each dataset corresponds to a type of budget. \nHowever, [A,B] do not consider active learning strategies.\n\n[A] Jain et al., A Meta-Learning Approach to Predicting Performance and Data Requirements, CVPR 2023\n[B] Mahmood et al., Optimizing data collection for machine learning, NeurIPS 2022\n\nSignificance: The idea of the paper is interesting as it allows to automatically find the optimal AL strategy and type of budget. However, I wonder how much it would be useful in real-world applications since it seems much more expensive to run than standard AL approaches (by constantly checking how much samples improve the performance, and then to determine whether they have to be added to the low or high budget).\n\nWEAKNESSES:\n- As mentioned above, the whole method should be clearly written in the main paper or appendix to improve clarity.\n\n- Is the method scalable to real-world applications? If so, isn't it cheaper to select a few uncertainty strategies (in large scale applications) to optimize the general labeling cost?\n\n- Due to its nature, the proposed method SelectAL returns the same scores as the best baselines (TypiClust and ProbCover, according to Table 1 and 2). Would there be a way to combine low and high budget strategies to obtain significantly better performance than a single baseline?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}