How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budget
Authors: Guy Hacohen
Daphna Weinshall
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Active learning
Low budget
High budget
Deep learning
TL;DR: A method for selecting the appropriate active learning strategy for a given data and budget, before labeling any examples.

Abstract: In the domain of Active Learning (AL), a learner actively selects which unlabeled examples to seek labels from an oracle, while operating within predefined budget constraints. Importantly, it has been recently shown that distinct query strategies are better suited for different conditions and budgetary constraints. In practice, the determination of the most appropriate AL strategy for a given situation remains an open problem. To tackle this challenge, we propose a practical derivative-based method that dynamically identifies the best strategy for a given budget. Intuitive motivation for our approach is provided by the theoretical analysis of a simplified scenario. We then introduce a method to dynamically select an AL strategy, which takes into account the unique characteristics of the problem and the available budget. Empirical results showcase the effectiveness of our approach across diverse budgets and computer vision tasks.

How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budget
Guy Hacohen Daphna Weinshall School of Computer Science & Engineering' Edmond and Lily Safra Center for Brain Sciences
The Hebrew University of Jerusalem
Jerusalem 91904, Israel
{guy hacohen, daphna) @mail huu.hlhhujii.. acuji.ac... ac. i1
Abstract
In the domain of Active Learning (AL), learner actively selects which unlabeled examples to seek labels from an oracle, while operating within predefined budget constraints. Importantly, it has been recently shown that distinct query strategies are better suited for different conditions and budgetary constraints. In practice, the determination of the most appropriate AL strategy for a given situation remains an open problem. To tackle this challenge, we propose a practical derivative-basee method that dynamically identifies the best strategy for given budget. Intuitive motivation for our approach is provided by the theoretical analysis of simplified scenario. We then introduce a method to dynamically select an AL strategy, which takes into account the unique characteristics of the problem and the available budget. Empirical results showease the effectiveness of our approach across diverse budgets and computer vision tasks.
1 Introduction
Active learning emerged as a powerful approach for promoting more efficient and effective learning outcomes. In the traditional supervised learning framework, active learning enables the learner to actively engage in the construction of the labeled training set by selecting fixed-sized subset of unlabeled examples for labeling by an oracle, where the number of labels requested is referred to as the budget. Our study addresses the task of identifying in advance the most appropriate active learning strategy for a given problem and budget.
The selection of an active learning strategy is contingent on both the learner's inductive biases and the nature of the problem hand. But when all this is fixed, recent research has shown that the most suited active learning strategy varies depending on the size the budget. When the budget is large, methods based on uncertainty sampling are most effective. When the budget is small, methods based on typicality are most suitable (see Fig. 1). In practice, determining the appropriate active learning strategy based on the budget size is challenging, or large depending on the problem at hand. This challenge is addressed in this paper.
specific budget can be considered either small
Specifically, we start by analyzing a simplified theoretical framework (Section 2), for which explicitly seleet the appropriate AL strategy using a derivative- of this model, propose SelectAL (Section 3), which incorporates similar derivative-basee test to select between active learning strategies. SelectAL aims to provide a versatile solution for any budget, by identifying the budget domain of the problem at hand and picking appropriate AL method from the set of available methods. SelectAL is validated through an extensive empirical study using several vision datasets (Section 4). Our results demonstrate that SelectAL is effective in identifying the best active learning strategy for any budget, achieving superior performance across all budget ranges.
test. Motivated by the analysis
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

High budget
Uncertainty sampling Focus rare examples
Low budget
Typicality sampling Focus on typical examples
Figure I: Qualitatively different AL strategies are suited for different budgets. SelectAl. determines in advance which family of strategies should be used for the problem at hand. Left: when the labeled training set (aka budget) is large, uncertainty sampling, which focuses on unusual examples, provides the most added value. Right: when the budget is small, the learner benefits most from seeing characteristic examples.
Relation to prior work. Active learning has been an active area of research in recent years [32, 34). The traditional approach to active learning. which is prevalent in deep learning and recent work, focuses identifying data that will provide the greatest added value to the learner based on what it already knows. This is typically achieved through the use of uncertainty sampling [11, 24, 31, 36], diversity sampling [8, 20, 33, 35], a combination of both [1, 10, 22).
However, in small-budget settings, where the learner has limited prior knowledge and effective training is not possible before the selection of queries, such active learning methods fail [4, 27]. Rather, [13, 30, 37J have shown that this domain, a qualitatively different family of active learning methods should be used. These methods are designed to be effective in this domain and tend to seek examples that can be easily learned rather than confusing examples [5, 25.39] 39].
With the emergence of this distinction between two separate families of methods to approach active learning, the question arises as to which approach should be preferred in a given context, and whether it is possible to identify in advance which approach would be most effective. Previous research has explored methods for selecting between various AL strategies [2, 19, 29, 40]. However, these studies assumed knowledge of the budget, which is often unrealistic in practice. To our knowledge, SelectALis the first work that addresses this challenge.
We note that several recent works focused on the distinctions between learning with limited versus ample data, but within the context of non-active learning [6. 9, 12, 14, 15, 16, 21, 26). These studies often highlight qualitative differences in model behavior under varying data conditions. While our results are consistent with these observations, our focus here is suggest a practical approach to identify the budget regime in advance, allowing for the selection of the most appropriate AL strategy.
2 Theoretical analysis
The aim of this section is to derive a theoretical decision rule that can be translated into practical algorithm, thus enabling practitioners to make data-driven decisions on how to allocate their budget in active learning scenarios. Given an AL scenario, this rule will select between a high-budget approach, a low-budget approach, or a blend of both. In Section 3, the theoretical result motivates the choice of decision rule, in practical method that selects between different active learning strategies.
To develop test that can decide between high and luw-budget approaches for active learning, we seek theoretically sound framework in which both approaches can be distinctly defined, and show how they can be beneficial for different budget ranges. To this end, we adopt the theoretical framework introduced by [13]. While this framework is rather simplistic, it allows for precise formulation of the distinctions between the high and low-budget approaches. This makes it possible to derive a precise decision rule for the theoretical case, giving some insights for the practical case later on.
In Section 2.1, we establish the necessary notations and provide a brief summary of the theoretical framework used in the analysis, emphasizing the key assumptions and highlighting the results that are germane to our analysis. We then establish in Section 2.2 a derivative- contribution of our work. In Section 2.3, we utilize this test to derive optimal active learning strategy for the theoretical framework. To conclude, in Section 2.4 we present an empirical validation of these results, accompanied by visualizations for further clarification.
test, which a novel

---

2.1 Preliminaries
Notations. We consider an active learning scenario where a learner, denoted by L. is given access to set of laheled examples L,, and much larger pool of unlabeled examples W. The learner's task is to select a fixed-size subset of the unlabeled examples, referred to as the active set and denoted by A _ U, and obtain their labels from oracle. These labeled examples are then used for further training of the learner. The goal is to optimally choose the active set A, such that the overall performance of learner L trained on the combined labeled dataset T = A U L is maximal.
We refer to the total number of labeled examples as the budget, denoted by = |T]] = |A ULJ. This concept of budget imposes critical constraint on the active learning process, as the learner must make strategic selections within the limitations of the budget.
Model definition and assumptions. We now summarize the abstract theoretical framework estab- lished by [13], which is used in our analysis. While this framework contains simplistic assumptions, its insights are shown empirically to be beneficial in real settings. This framework considers two independent general learners Llow and Lhigh, each trained on different data distribution Diow and Duigh respectively. Intuitively, one can think of Diow as a distribution that is easier to learn than Dhigh, such that Liow requires fewer examples than Liigh to achieve a similar generalization error.
The number of training examples each learner sees may vary between the learners. To simplify the analysis, the framework assumes that data naturally arrives from a mixture distribution D, in which an example is sampled from Diow with probability p and from Phigh with probability - p.
We examine the mean generalization error of each learner denoted by Etow Ehigh : R - [0, respectively, function of the number of training examples it sees. The framework makes several assumptions about the form of these error functions: (i) Universality: Both Elow and Ehigh take on the same universal form E(x), up to some constant a > O, where E(x) Elow (x) = Ehigh (ax). (ii) Efficiency: (zz) is continuous and strictly monotonically decreasing, namely, on average each learner benefits from additional examples. (iii) Realizability: lim E(x) = O, namely, given enough examples, the learners can perfectly learn the data.
To capture the inherent choice in active learning, family of general learners is considered, each defined by a linear combination of Liow and Lhigh- [13] showed that when fixing the number of examples available to the mixture model, i., fixing the budget B, it is preferable sample the training data from a distribution that differs from D. Specifically, it is preferable to skew the distribution towards Diow when the budget is low and towards Drigh when the budget high.
2.2 Derivative-base query selection decision rule
We begin by asking whether it is advantageous to skew the training distribution towards the low or high-budget distribution. Our formal analysis gives a positive answer. It further delivers a derivative- based test, which can be computed in closed form for the settings above. Importantly, our empirical results (Section 4) demonstrate the effectiveness of an approximation of this test in deep-learning scenarios, where the closed-form solution could not be obtained.
We begin by defining two pure query selection strategies - one that queries examples only from Diow- suitable for the low-budget regime, and a second that queries examples only from Prigh. suitable for the budget regime. Given a fixed budget B, we analyze the family of strategies obtained by a linear combination of these pure strategies, parameterized by q E [0, 1]. in which qB points are sampled from Diow and (1 B points are sampled using Dhigh-
The mean generalization error of combined learner L on the original distribution D. where C is trained (11))))))+())))))(((())))))))))l
on B examples picked by mixed strategy q. is denoted Ez (B, (B)g) as the combined learner simply sends B examples to Ltow and (1 - B examples to Lhigh- By differentiating this result (see derivation in Suppl. A), we can find optimal strategy for this family, and denote it by q. This strategy is defined as the one that delivers the lowest generalization error when using a labeled set of size B. The strategy characterized by a E [0, 1], implicitly defined as follows:
E' (qB) E' (a - B)
(1 -
(1)
3

---

If Eq. (1) has a unique and minimal solution for 4, it defines optimal mixed strategy ( a) for a given set of problem parameters {B. a... As p and a are fixed for any specific problem. we get that different budgets B may require different optimal AL strategies. Notably, if budget B satisfies (E(B, p. a) = p. then the optimal strategy for it is equivalent' to selecting samples directly from the original distribution D. We refer to such budgets as Beg- With these budgets, active learning no longer helpful. While Beg may not be unique, for the sake of simplicity, we assume that is unique in analysis, noting that the general case is similar but more cumbersome.
We propose to use Eq. (1) a decision rule, determine whether low-budget or high-budger strategy is more suitable for the current budget B. Specifically, given budget B, we can compute og (B, p. a) and Beg in advance and determine which AL strategy to use by comparing In the current framework, this rule is guaranteed to deliver an optimal linear combination of strategies, as we demonstrate in Section 2.3. Visualization of the model's error as a function of the budget, for different values of q, can be seen in Fig. 2a.
and Beg.
0.03
0.3
222
22
06-
Performance
044
02-
00
10'
10-
Number of examples (B)
Number of examples (B)
(a) Accuracy gain of mixed strategies q, where mixture coefficient is indicated in the legend.
(b) Optimal (B a).
Figure 2: Visualization of Ec(B, q). where training of size B is selected for different values of q. We adopt the example from [13], choosing E(x)
use a 0.1, p = 1 and a 0.05, as exponential functions approximate well the error functions of real networks. (a) A plot of accuracy gain when using strategy (indicated in legend) compared to random query selection: k(p) - E(q). a function of budget B. Since p 2. the plot corresponding q is always 0. (b) Plots of á as function of B. Beg. which corresponds q = 2. is indicated by vertical dashed line. Each plot corresponds a different fraction (see legend).
2.3 Best mixture of active learning strategies
With the aim of active learning (AL) in mind, our objective is to strategically select the active set A in order to maximize the performance of learner L when trained all available labels T = A U L. To simplify the analysis, we focus on the initial AL round, assuming that IL is sampled from the underlying data distribution D. The analysis of subsequent rounds can be done in similar manner.
To begin with, let us consider the entire available label set T == A UL. Since T comprises examples from D, Diow and Drigh, we can represent using the (non-unique) notation S(rrond, Flow Thigh) ) where Trand, Tlows Thigh denote the fractions of T sampled from each respective distribution. Based on the definitions of q and è, we can identify an optimal combination for T, denoted T*:
F,F,0
<>p = B < Beg
T-p p)
4> P
T
S(1,0,0)
-=p = B = Beg
(2)
i á<p
((1-
i<p - B> Beg
Attainable optimal mixed strategy. It is important to note that T = AUU. L, where the active learning (AL) strategy can only influence the sampling distribution A. Consequently, not every combination of T is attainable. In fact, the feasibility of the optimal combination relies on the size of the set A. Specifically, utilizing (1), we can identify two thresholds. Blow and Bligh, such that B < Blow S. Beg B > Brigh Beg. it is not possible to achieve the optimal T = A U L. The derivation both thresholds can be found in Suppl. A.
"ddentity (in probability) is achieved when B -

---

# ==0.01
Gokin
5050011 W333
5099131
915050
023333 903676 901496
Etror reduction
H =0,3
Arturacy improvement
Accuracy improvemint
25
130
150
Number exampies
Figure 3: Visualization of strategy (3) using the example from Fig. 2. We plot the gain in error reduction pared to random query selection. We show results with active sets that are 30% and I % of B. The smaller the ac- tive set is, the closer the performance the pure optimal strategy.
Number of examples (B)
Number axamples
(a) CIFAR-10
(b) CIFAR-100
Figure 4: Empirical validation of the theoretical results. We plot the
racy gains 20 ResNet--18 networks trained using strategy (3), compared to using no active learning at all. We used TypiClust as Stow and BADGE as Shigh. We see that as predicted by the theoretical analysis. the best mixture coefficient increases the difference B Begl increases.
With (2) and the thresholds above, we may conclude that the optimal combination of A is:
1,0)
Blow
S(1 - + +
TAP
[A]
Biow S < Beq
1,0,
B < Biow
A-
0,),0,0)
B = Beg
(3)
IA A
0(1,0,0)
B = Beg
(4)
5(1 - 0
Bligh 2 B >>> Bag
1(),0,1)
B > Bhigh
1,)
B > Bhigh
In other words, we observe that the optimal strategy (2) is only attainable in non-extreme scenarios. Specifically, in cases of very low budgets (B < Biow), it is optimal to sample the active purely from Diow because more than [ points from Diom are needed to achieve optimal performance. Similarly, in situations of very high budgets (B > Bhigh), it optimal select the active set solely from Dhigh since more than [ points from Drigh are necessary for optimal performance. We note that even if Bug is not unique, in the limit of | 0, (3)-(4) are still locally optimal, segment by segment.
Small active set A. Upon examining the definitions of Blow and Bhigh (refer to Suppl. A), observe that lim Blow = AJ=s Beigh = Beg. Consequently, as indicated in (4), the optimal mixed strategy, in this scenario. actually becomes pure strategy. When the budget is low, the entire active set A should be sampled from Diow- Similarly, when the budget is high, & should sampled solely from Dnigh- In the single point where Brow Bhigh Beg. A should be sampled from D. Fig. 2b provides a visualization of these strategies as the size of & decreases.
Iterative querying. When |&] is not small enough to justify the use of the limiting pure strategy (4), we propose to implement query selection incrementally. By repeatedly applying (4) to smaller segments of length we can iteratively construct A, with each iteration becoming more computa- tionally feasible. This strategy not only enhances robustness but also yields comparable performance to the mixed optimal strategy (3), demonstrated in Section 2.4.
Based on the analysis presented above, becomes apparent that in practical scenarios, the optimal combination of active learning strategies can be achieved by sequentially sampling from pure strategies, while utilizing a derivative-bar based test to determine the most effective strategy each step. This concept forms the core motivation behind the development of the practical algorithm SelectAL, which is presented in Section 3.
2.4 Validation and visualization of theoretical results
Visualization. In Fig. 3, we illustrate the eгror of the strategies defined in (3) using the same exponential example as depicted Fig. 2. The orange curve represents a relatively large active set, where |A| is equal to 30% of the budget B, while the blue curve represents significantly smaller active set, where |A] equal to 1% of the budget B. evident that as the size of A decreases, the discrepancy between the optimal strategy and the attainable strategy diminishes.
5

---

Fig. 2b shows the optimal mixture coefficient ② as a function of the budget size B for various values of [A], namely 1%. 30%, and 100% of B. According to our analysis, as the size of [A] decreases, the optimal / should exhibit a more pronounced step-like behavior. This observation suggests that in the majority of cases, possible to sample the entire active set A. from a single strategy rather than using a mixture of strategies.
Validation. Our theoretical analysis uses mixture model of idealized general learners. We now validate that similar phenomena occur in practice when training deep networks different computer vision tasks. Since sampling from Drow and Dhigh is not feasible in this case, we instead choose low and high-budget deep AL strategies from the literature. Specifically, we choose TypiClust the low-budget strategy and BADGE as the high-budget strategy, as explained in detail in Section 4.1. We note that other choices for the low and high-budget strategies yield similar qualitative results, as is evident from Tables 1-2. Fig. 4 shows results when training 20 ResNet- 18 networks using mixed strategies 5 r [ 0, r
[T|
(T) (T) [A] and S r riaj." JA 0) on CIFAR-10 and CIFAR-100 100. We compare
the mean performance of each strategy to the performance of 20 ResNet- 18 networks trained with the random query selection strategy S(1, 0, 0).
Inspecting these results, we observe similar trends to those shown in the theoretical analysis. As the budget increases, the most beneficial value of mixture coefficient decreases until certain transition point (corresponding to Beg). From this transition point onward, the bigger the budget is, the more beneficial to select additional examples from one of the high-budget strategies. As in the theoretical analysis, when the budget is low it is beneficial to use pure low-budget strategy, and when the budget high it is beneficial to use a pure high-bure buuget strategy. The transition area, corresponding to segment Brow S B S Brigh is typically rather short (see Figs. 4 and 6).
2.5 Discussion: motivation from the theoretical analysis
The theoretical framework presented here relies simplified model, enabling a rigorous analysis with closed-fore solutions. While these solutions may not be directly applicable in real-worke deep learning scenarios, they serve as solid motivation for the SelectAl. approach. Empirical evidence demonstrates their practical effectiveness.
In Section 2.2, we introduced a berivative-bbsed test, demonstrating that measuring the model's error function on small data perturbations suffices to determine the appropriate strategy for each budget. This concept serves as the fundamental principle of SelectAL, method that chooses an active learning strategy by analyzing the model's error function in response to data perturbations.
In Section 2.3, we showed that while the optimal mixed AL strategy can be determined analytically, a practical approximation can be achieved by choosing a single pure AL strategy for each round in active learning. This approach significantly simplifies SelectAL, as it doesn't It require the creation of a distinct mixed AL strategy tailored separately to each budget. Instead, SelectAI. selects a single pure AL strategy for each active learning round, from set of existing "pure" AL strategies. In Fig. 5, we observe that this selection consistently outperforms individual pure strategies, confirming our analytical predictions. We also explored various combinations of existing AL strategies, none of which showed a significant improvement over SelectAL, further reinforcing the relevance of predictions of the theoretical analysis.
3 SelectAL: automatic selection of active learning strategy
In this section, we present SelectAL - a method for automatically selecting between different AL strategies in advance, by estimating dynamically the relative budget size of the problem hand. The decision whether B should be considered "large" or "small" builds on the insights gained from the theoretical analysis presented in section 2. The suggested approach approximates the derivative-based test suggested in (1), resulting in a variation of the attainable optimal mixed strategy (3).
SelectAL consists of two steps. The first step involves using a version of the derivative based rule from Section 2.2 to assess whether the current budget B for the given problem should be classified as "high" (B > Brigh) or "low" (B < Blow). This is accomplished by generating small perturbations of based on both low and high-budget strategies, and then predicting the current quantity of labeled examples is adequate to qualify as a high-budget or ludget.

---

In the second step, we select the most competitive AL strategy from the relevant domain and use it to select the active set A. This approach not only ensures good performance within the specified budget constraints but also provides robustness across a wide range of budget scenarios. It is scalable in its ability to incorporate any future development in active learning methods.
3.1 Deciding on the suitable budget regime
In the first step, SelectAL must determine whether the current labeled set L should be considered low-budget (B < Btow) or high-budget (B Brigh) for the problem at hand. The challenge is to make this decision without querying additional labels from U.
To accomplish this, SelectAL requires access to set of active learning strategies for both low and high-budget scenarios. We denote such strategies by Slow and Shigh respectively. Additionally, a random selection strategy, denoted by Srand, is also considered.
To determine whether the current budget is high or low, SelectAL utilizes a surrogate test. Instead of requiring additional labels, we compute the result of the test with derivative-like approach. Specifically, for each respective strategy separately, we remove a small set of points (size e > 0) from L. and compare the reduction in generalization error the removal of randomly chosen points. To implement this procedure, we remove the active sets chosen by either Slow or high+ when trained using the original labeled set II as the unlabeled set and an empty set as the labeled set.
The proposed surrogate test presents new challenge: in most active learning strategies, particularly those suitable for high budgets, the outcome relies on learner that is trained on a non-empty labeled set. To overcome this, we restrict the choice of active learning strategies Snow and Shigh methods that rely only on the unlabeled set U. This added complexity is crucial: using the labels of set L. by either Stow or Shigh results in a bias that underestimates the cost of removing known points, as demonstrated in our empirical study (see Fig. 7). No further restriction on Skow and Shigh is needed. Empirically, we evaluated several options for Syow and Sigh. including TypiClust and ProbCover for Slow and inverse- Typiclut and CoreSet for Shigh, with no qualitative difference in the results.
We generate three subsets from the original labeled set L: datatou. datahigh and datarand- Each subset is obtained by asking the corresponding strategy - Stow. Shighh and Syand - to choose a class-balanced subset of 6 [IL] examples from L. Note that this is unlike their original use, as AL strategies are intended to select queries from the unlabeled set U. Importantly, since set L is labeled, we can guarantee that the selected subset is balance. Subsequently, the selected is removed from L, and a separate leamnr is trained on each of the 3 subsets (repeating the process multiple times for Srand). Finally, we evaluate the accuracy of each method using vols-validation, employing 1% of the training data as a validation set in multiple repetitions. The subset with the lowest accuracy indicates that the subset lacks examples that are most critical for learning. This suggests that the corresponding strategy queries examples that have the highest impact on performance.
Our final method can be summarized as follows (see Alg. 1):
Accumey fncurnyyy Ieporneeeett
Algorithm 1 SelectAL
I: Input:
Shighs Stow- Snigh. Stows Svand, labeled data L, e 0
Accurscy
2: Output: AL strategy
3: ct max/ # classess
11
Dudge
Budge::
4: for i high, rand) do
(a) CIFAR-10
(b) CIFAR- 100
5: removed_ serredd__xaplessseee + query exam- ples from cach class from L using S;
Figure S: Accuracy improvement. relative to no active learning (random sampling), for 10 rounds of different AL strategics. We see that while some methods work well only in low budgets and some in high budgets, SelectAL works well all budgets. Before the first dashed line, SelectAL picked a low-budget strat- egy, between the dashed lined it picked random strategy, and after the second dashed line it picked high- strategy. In both experiments, SelectAL picked Stow for the first 3 iterations. followed by Smand for iteration and then Stagh-
6: data; < L,, removed_ _examlles
7: acc; model's accuracy data;
9: strategy +
argmin min (aces) higgh
arg
10: return Satrategy
7

---

3.2 Selecting the active learning strategy
In second step, SelectAL selects between two active learning strategies, Stown and Shigh which are known to be beneficial in the low and high-budget regimes respectively. Unlike Stow and highe there restrictions on {Stow, Shigh J. from which Alg. 1 selects, and it is beneficial to select the SOTA active learning strategy for the chosen domain.
Somewhat counter- -ntuter-ntuuiiivlly. SelectAL is likely to use different pairs of AL strategies, one pair to determine the budget regime, and possible a different one for the actual query selection. This the case because in its first step, Slow and Shigh are constrained to use only the unlabeled set U, which may eliminate from consideration the most competitive strategies. In contrast, and in order to achieve the best results, Stow and Shigh are chosen to be the most competitive AL strategy in each domain. This flexibility is permitted because both our theoretical analysis in Section 2 and our empirical analysis in Section 4 indicate that the transition points Blow and Bligh are likely be universal, or approximately so, across different strategies. This is especially important in the high-budget regime, where the most competitive strategies often rely on both IL and U.
4 Empirical results
We now describe the results of an extensive empirical evaluation of SelectAL. After suitable AL strategy is selected by Alg. 1, it is used to query A unlabeled examples. The training of the deep model then proceeds as customary, using all the available labels A U T.
4.1 Methodology
Our experimental framework built the codebase of [28), which allows for fair and robust comparison of different active learning strategies. While the ResNet-18 18 architecture used in our experiments may not achieve state-of-the-art results CIFAR and ImageNet, it provides suitable platform to evaluate the effectiveness of active learning strategies in a competitive environment, where these strategies have been shown be beneficial. the following experiments. we trained ResNet- [18] on CIFAR-10. 10, CIFAR- 100 [23] and ImageNet-50 a subset of ImageNet [7] containing 50 classes done in [38]. We use the same hyper-parameters as in [28], as detailed in Suppl. B.
SelectAL requires two types of active learning strategies: restricted strategies that use only the unlabeled set U for training, and unrestricted competitive strategies that can use both LL and U (see discussion above). Among the restricted strategies, we chose TypiClust [13] to take the role of Stow- and inverse TypiClust for Syigh- In the latter strategy, the most atypical examples are selected. Note that inverse TypiClust is effective strategy for high budgets, while relying solely on the unlabeled set U (see Suppl. C. for details). Among the unrestricted strategies, we chose ProbCover [39] for the role of the low-budget strategy Stow: and BADGE [1] for the high-budget strategy Snigh- Other choices yield similar patterns of improvement, as can be verified from Tables 1-2.
In the experiments below, we use several active learning strategies. including Min margin, Max entropy, Least confidence, DBAL [10], CoreSet [33], BALD [22), BADGE [1], TypiClust [13] and ProbCover [39]. When available, we use for each strategy the code provided in [28]. For low-budger strategies, which are not implemented in [28], we use the code from the repository of each paper.
4.2 Evaluating the removal of examples with AL in isolation
We isolate the strategy selection test in Alg. as described in Section 3.1. To generate the subsets of labeled examples datatow. data and datayand. we remove 5% of the labeled data, ensuring that never remove less than one data point per class. In the low-budget regime, removing examples according to Siow yields worse performance as compared to the removal of random examples, while better performance seen in the high-budget regime. The opposite behavior seen when removing examples according to Shigh. Different choices of strategies for Sy... and Shigh yield similar qualitative results, see Suppl. C.2.
More specifically, we trained 10 ResNet-18 18 networks on each of the 3 subsets, for different choices of budget B. Fig. 6, we plot the difference in the mean accuracy of networks trained on data and datanigh, compared to networks trained on datarand, similarly to the proposed test in Alg. 1. In all
8

---

the budgets that are smaller than the orange dashed line, SelectAL chooses Stow. In budgets between the orange and the green dashed lines, SelectAL chooses Srand- In budgets larger than the green dashed line, SelectAl. chooses Shigh-
recuracy improvement
Focureeyy inprovement
0.5
-02-
Rccuarety
-3.0
Number examples
Numrber (R]
excmmpee
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet-50
Figure Accuracy gain when using Slow to select points for removal as compared random selection (orange), or Shigh to select points for removal (green). Negative gain implies that the strategy is beneficial, and vice versa.
Table I: Mean accuracy and standard error of 10 ResNet-18 18 networks trained on CIFAR- 10 and CIFAR 100. using various budgets and active learning strategics. In each dataset, display results for 3 budget choices: one smaller than Biow (left column), one between Biow and Brigh (middle column), one larger than Blach (right column). We highlight in boldface the best result each column, and additionally the results that lic within its interval of confidence (the standard error bar). While most strategies are effective only in low or in high budgets, SelectAL is effective in both regimes, As predicted, between Blow and Bhigh, most AL strategies do not significantly outperform random query selection.
CIFAR-10 7k+lk
CIFAR-100
Budget (L + A)
100+100
25k+5k
100+100
9k+1k
30k+7k
random
31.8 0.3
76 0.3
87.2 0.2
0.2
39,9 0.3
60.7 0.3
TypiClust
34.1
0.3
87.1 0.2
7.1
39.7 0.4
60.4 + 0.2
BADGE
31.3 0.4
76.5 1: 0.4
88.1 0.1
5.3
39.5 0.3
61.9 1 0.1
DBAL
30.2 10.20...
76.4 0.4
4.2
3.4
61.5 1 0.2
BALD
3.3
76.3
0.2
4.2
1
61.5 1 0.2
CoreSet
0.4
0.3
0.2
0.4
0.3
- 0.2
ProbCover
0.3
0.3
0.11
8.2
4.4
61.4 0.3
Min Margin
30.7
0.2
87.9 = 0.2
5.2
0.2
61.6 - 0.3
Max Entropy
30.2
76.1
87.8 + 0.2
4.9
+.2
61.6 + 0.2
Least Confilence
0.2
76.1
88.1 0.2
4.7 0.4
38.9
61.5 + 0.2
SelectAL
76 0.3
88.1 1.110.11 0.1
8.2 = 0.1
39.9 0.3
61.9 1 0.1
4.3 SelectAL: results
In Fig. 5, we present the average accuracy of series of experiments involving 10 ResNet-18 networks trained over 10 consecutive AL rounds using three AL strategies: Slow (ProbCover). Shigh (BADGE), and our proposed method SelectAL. We compare the accuracy improvement of each strategy to training without any AL strategy. In each AL round, SelectAL selects the appropriate AL strategy based on Alg. 1. Specifically, SelectAL selects Stow when the budget is below the orange dashed line, Srand when the budget is between the orange and green dashed lines, and S Shigh when the budget above the green dashed line. We observe that while Stow and Shigh are effective only for specific budgets, SelectAL performs well across all budgets. all the experiments we performed, across all datasets, we always observed the monotonic behavior predicted by the theory SelectAL picks Stow for several AL iterations, followed by Srand and then Shigh.
It is important to note that, unlike Fig. 6, where the data distribution of set L is sampled from the original distribution because we analyze the first round (see Section 2.3), in the current experiments the distribution is unknown apriori L in each iteration is conditioned on the results of previous iterations, and is therefore effectively a combination of Stow. Shigh, and Srand- Consequently. the transition point determined automatically by Alg. occurs earlier than the one detected in Fig. 6. Notably, while SelectAL. chooses an existing AL. strategy at each iteration, the resulting strategy outperforms each of the individual AL strategies when trained alone.

---

In Tables 1-2, we show the performance of SelectAL in comparison with the performance of the baselines (Section 4.1). In all these experiments, SelectAL succeeds to identify a suitable budget regime. As result, it works well both in the low and high-budget regimes, matching surpassing both the low and bigh-budget strategies at all budgets. Note that as SelectAL. chooses an active learning strategy dynamically for each budget, any state-of-the-art improvements for either low or high budgets AL strategies can be readily incorporated into SelectAL.
Table 2: Same as Table mean and standard error of 10 ResNet-18 18 networks trained on ImageNet 50 using dif- ferent AL strategies, at low, medium, and high budgets.
Accuracy difference
ImageNet-50 7k+lk
(L + A)
100+100
25k+5k
-1
Random
random
9.3 1 0.2
61.8 1 0,4
79.8 1 0.2
BADGE
TypiClust
11.3 0.3
61.8 1 0.5
80.1 1 0.2
BADGE
9.4 0.2
61.3 + 0.5
80.8 0.2
Number of examples (B)
DBAL
0.4
61.1 + 0.6
80.1 0.2
BALD
9.4 0.4
61.7 + 0.3
80.7 0.1
Figure 7: To assess the suitability of BADGE as an example removal strategy Shigh. we com- pare it with the original Shigh approach whose performance is reported in Fig. 6a, for CIFAR- 10. Unlike the original Shygh (depicted in green), BADGE (depicted in orange) exhibits a lack of a distinct transition point. Consequently, BADGE is not well-suited as choice for Shigh.
CoreSet
8.6 0.3
61.7 11.7021 0.2
80.7 0.3
ProbCover
61.6 0.8 1.64
7..7
Min Margin
9.2
61.2
80.4
Max Entropy
8.9
8..1
Least Conf.
0.1
0.4
19.6:0.55
SelectAL
6.48 1..4
80.8 - 0.2
Why Slow and Shigh are Restricted? As discussed in Section 3.1, we have made a deliberate decision to exclude strategies that rely on the labeled set IL while deciding which family of strategies is more suited to the current budget. We now demonstrate what happens when the selection is not restricted in this manner, and in particular, if Shigh is chosen to be a competitive AL strategy that relies on the labeled set for its successful outcome. Specifically, we repeat the experiments whose results are reported in Fig. 6a, but where strategy Shigh. the one used for the removal of examples is BADGE. Results are shown in Fig. 7. Unlike Fig. 6a, there is no transition point, as it is always beneficial remove examples selected by BADGE rather than random examples. This may occur because the added value of all points used for training diminishes after training is completed.
Computational time of SelectAL SelectAL entails the training of three active learning strategies on a small dataset. Consequently, the computational duration of SelectAL. is contingent upon the user's strategy choices. It is worth noting that, in practice, the additional computation time is negligible in comparison to conventional AL techniques. This is due to the fact that Stow and Shigh are exclusively trained on L, which is typically significantly smaller than U the set used by pure AL strategies. To further expedite computation, one might contemplate training on data perturbations using more compact model, an issues which we defer the to future research.
5 Summary and discussion
We introduce SelectAL., a novel method for selecting active learning strategies that perform well for all training budgets, low and high. We demonstrate the effectiveness of SelectAL through combination of theoretical analysis and empirical evaluation, showing that achieves competitive results across wide range of budgets and datasets. Our main contribution lies in the introduction of the first budget-aware active learning strategy. Until now, selecting the most appropriate active learning strategy given some data was left to the practitioner. Knowing which active learning strategy best suited for the data can be calculated after all the data is labeled, but predicting this in advance is a difficult problem. SelectAL offers solution to this challenge, by determining beforehand which active learning strategy should be used without using any labeled data.
This work was supported by grants from the Israeli Council of Higher Education and the Gatsby Charitable Foundations.
Acknowledgement
10

---

References
[1] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[2] Yoram Baram, Ran El Yaniv, and Kobi Luz. Online choice of active learning algorithms. Journal of Machine Learning Research, 5(Mar):255-29) 2004.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEF/CVE International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9630-9640. IEEE, 2021.
[4] Yao- Mho-Chun Chan, Mingehen Li, and Samet Oymak. On the marginal benefit of active learning: Does self-supervision eat its cake? In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3455-3459. IEEE, 2021.
[5] Liangyu Chen, Yutong Bai, Siyu Huang, Yongyi Lu, Bihan Wen, Alan L Yuille, and Zongwei Zhou. Making your first choice: To address cold start problem in vision active learning. arXiv preprint arXiv:2210.02444 2022.
[6] Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar-leariigg trajectories of neural language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 8281-8297 2022.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[8] Ehsan Elhamifar, Guillermo Sapiro, Allen Yang, and S Shankar Sasrty. A convex optimization framework for active learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 209-216, 2013.
[9] Noam Fluss, Guy Hacohen, and Daphna Weinshall. Semi-supervised learning in the few-shot zero-shot scenario. arXiv preprint arXiv:2308. 14119, 2023.
[10] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1183-1192. PMLR, 2017.
[11] Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint
1971v:190..06347, 2019.
[12] Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree agree: Neural networks share classification order on real datasets. In International Conference on Machine Learning, pages 3950-3960 PMLR, 2020.
[13] Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning a budget: Opposite strategies suit high and low budgets. In International Conference on Machine Learning. PMLR, 2022.
[14] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning, pages 2535-2544 PMLR, 2019.
[15] Guy Hacohen and Daphna Weinshall. Principal components bias in over-parameterized linear models, and its manifestation in deep neural networks. The Journal of Machine Learning Research, (511)66773-7118 2022.
[16] Guy Hacohen and Daphna Weinshall. Pruning the unlabeled data to improve semi-supervised learning, arXiv preprint 2.KV::2308.14033 2023,
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.

---

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEER Computer Society, 2016.
[19] Wei-Ning Hsu and Hsuan-Tiee Lin. Active learning by learning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[20] Rong Hu, Brian Mac Namee, and Sarah Jane Delany. Off good start: Using clustering to select the initial training set in active learning. In Twenty-Third International FLAIRS Conference, 2010.
[21] Achin Jain, Gurumurthy Swaminathan, Paolo Favaro, Hao Yang, Avinash Ravichandran, Hrayr Harutyunyan, Alessandro Achille, Onkar Dabeer, Bernt Schiele, Ashwin Swaminathan, et al. A meta-lcarning approach to predicting performance and data requirements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3623-3632, 2023.
[22] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32:7026-7037, 2019.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Online, 2009.
[24] David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR '9, pages 3-12. Springer, 1994.
[25] Rafid Mahmood, Sanja Fidler, and Mare T. Law. Low-budget active learning via wasserstein distance: An integer programming approach. In The Tenth International Conference Learning Representations, ICLR 2022. Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[26] Rafid Mahmood, James Lucas, Jose M Alvarez, Sanja Fidler, and Marc Law. Optimizing data collection for machine learning. Advances in Neural Information Processing Systems, 35:29915-29923 2022.
[27] Sudhanshu Mittal, Maxim Tatarchenko, Özgün Çiçek, and Thomas Brox. Parting with illusions about deep active learning. arXiv preprint arXiv: 19/2.053// 2019.
[28] Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 223-232 IEEE, 2022.
[29] Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy M Hospedales. Dynamic ensemble active learning: A non-stationary bandit with expert advice. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2269-2276. IEEE, 2018.
[30] Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active learning is a strong baseline for data subset selection. In Has it Trained Yer? NeurlPS 2022 Workshop, 2022.
[31] Hirammayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Pan- chanathan. Deep active learning for image classification. In 2017 IEEE International Conference on Image Processing (ICIP), pages 3934-3938. IEEE, 2017.
[32] Christopher Schröder and Andreas Niekler. A survey of active learning for text classification using deep neural networks. arXiv preprint Sttiv::000.0076666 2020.
[33] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018
[34] Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin Wisconsin-Madiionn 2009.
[35] Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. International Conference on Artificial Intelligence and Statistics, pages 1318-1318. PMLR, 2020.
12

---

[36] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972-5981, 2019.
[37] Alexandru Tifrea, Jacob Clarysse, and Fanny Yang. Uniform versus uncertainty sampling: When being active less efficient than staying passive. arXiv preprint arXiv:2212.00772. 2022.
[38] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conference Computer Vision, pages 268-285. Springer, 2020.
[39] Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active learning through covering lens. In NeurIPS, 2022.
[40] Jifan Zhang, Shuai Shao, Saurabh Verma, and Robert Nowak. Algorithm selection for deep active learning with imbalanced datasets. arXiv preprint 17Xiv:2302.07317, 2023.
13


Reviews:
[{"review_id": "6LYlt8KbtW", "reviewer": "Reviewer_uJGw", "summary": "The literature has shown that the optimal type of active learning (AL) strategy depends on the type of budget. Methods based on uncertainty sampling are most effective when the budget is large, and methods based on typicality work better in the low budget setting. However, what qualifies \"large\" or \"low\" depends on different criteria such as the data set and its size, the type of neural network architecture that is trained etc. \nThe paper proposes a general approach that automatically determines whether a given setting is low or high budget by decoupling the data set in different parts. The method is divided in two parts. In the first part, a different active learning strategy is used depending on whether a labeled image belongs to the low or high budget setting. In the second part, unlabeled images are added to the labeled set. ", "questions": "See above", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Quality and clarity: The general motivation of the approach (i.e. combining low and high budget settings and using different strategies for each) is clear and makes sense given the recent AL literature. The first part of the algorithm is clear and more or less described in Algorithm 1. The clarity of the second part could be improved by writing its pseudo-code (in the main paper or appendix).\n\nOriginality: Similar ideas that consider different strategies for different parts of the data set were recently published in the literature (e.g. [A,B]). \nJain et al. [A] show that low-budget settings follow a linear law whereas high-budget settings follow a power law in the standard classification setting. They also try to estimate whether a setting is low or high budget.\nMahmood et al. [B] also consider to learn the optimal budget over multiple datasets (in the multi-variate case) to improve model performance while minimizing labeling cost. This is related to the case where each dataset corresponds to a type of budget. \nHowever, [A,B] do not consider active learning strategies.\n\n[A] Jain et al., A Meta-Learning Approach to Predicting Performance and Data Requirements, CVPR 2023\n[B] Mahmood et al., Optimizing data collection for machine learning, NeurIPS 2022\n\nSignificance: The idea of the paper is interesting as it allows to automatically find the optimal AL strategy and type of budget. However, I wonder how much it would be useful in real-world applications since it seems much more expensive to run than standard AL approaches (by constantly checking how much samples improve the performance, and then to determine whether they have to be added to the low or high budget).", "weaknesses": "- As mentioned above, the whole method should be clearly written in the main paper or appendix to improve clarity.\n\n- Is the method scalable to real-world applications? If so, isn't it cheaper to select a few uncertainty strategies (in large scale applications) to optimize the general labeling cost?\n\n- Due to its nature, the proposed method SelectAL returns the same scores as the best baselines (TypiClust and ProbCover, according to Table 1 and 2). Would there be a way to combine low and high budget strategies to obtain significantly better performance than a single baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the references; indeed a discussion of low and high budget, independently of active learning, will put the present work in greater context. We will modify our related work section accordingly, and cite the appropriate references. \n\nSpecific comments raised by the reviewer:\n\n**Clarity**:\n\nTo improve the clarity of our paper, we will add pseudo-code for the second part of the algorithm to the appendix, and will reference it in the main text.\n\n**Scalability**:\n\nEmpirically, our proposed method -- SelectAL -- demonstrates robustness and scalability to real-world applications. The total cost of SelectAL involves training a neural network once on two variants of the labeled set, denoted in the text by $\\mathbb{L}$. While training two networks incurs this overhead, it is comparable to the training cost of the model that would use the labeled data afterward. Hearing the reviewer's concern and in order to further reduce the method's cost, in our future work we will examine the option to perform the test in Alg. 1 using a significantly smaller model. This discussion will be added to the summary and discussion section.\n\nIt is worth clarifying that as currently presented, SelectAL already chooses a single low-budget and a single high-budget strategy to represent a larger family of active learning algorithms. In this way, we reduce the complexity of the method, as users are not required to separately examine each AL strategy they wish to consider. This already significantly contributes to reducing the method's computational cost. This heuristics was evaluated empirically, and the results reported in the paper show its effectiveness. After SelectAL returns its trinary decision, a user may decide to select a few uncertainty strategies, or a mixture of such strategies, if the high-budget regime is identified.\n\n**Combining low and high-budget strategies**:\n\nIn a single round within the active learning framework, our theoretical analysis in Section 2.3 indicates that when the number of added examples is relatively small as compared to the total number of examples in the dataset, there is no clear advantage to mixing strategies, and specifically to combining low and high-budget strategies. Empirically, we validated this finding through a grid search, which showed that while combining several strategies does offer a minor improvement, it does not yield a significant boost in learning beyond any single active learning strategy. As a result, and since the use of multiple strategies adds complexity to the suggested approach, we propose to use of a single strategy.\n\nWe note that the scenario changes in the context of multi-round active learning. With SelectAL's capability to switch strategies in each round, we can continually build the final labeled set, incorporating examples from different strategies along the way. As shown in Fig. 5, this dynamic approach leads to more substantial performance improvements as compared to any single uncertainty-based strategy. Thus there is a trade-off between the number of rounds and performance, where improved performance can be obtained with increased complexity by way of additional rounds, and when fewer examples are chosen in each round. We will add a discussion to this effect in the camera-ready revision."}, {"review_id": "GqW6zaO3oe", "reviewer": "Reviewer_GcEN", "summary": "The paper studies an algorithm selection problem for active learning under different labeling budgets. A theoretical analysis is conducted The proposed algorithm uses a derivative approach by removing small sets of labeled examples to approximate the effectiveness of different active learning algorithms. Empirical results are shown for different labeling budget that SelectAL chooses the best active learning algorithm. Experiments are conducted on CIFAR-10, CIFAR-100 and ImageNet-50.", "questions": "1. What's the underlying intuition and reasoning of why TypiClust and inverse-TypiClust can correctly predict the budget scenarios.\n2. Why are the particular initial seed set size and batch size selected for CIFAR-10 and CIFAR-100.", "limitations": "Sufficient.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper studies an important problem of how to choose the best active learning algorithm for a labeling budget.\n2. Experiments demonstrates the effectiveness of the method in specific settings.\n3. It is interesting to empirically see that random sampling performs as good as other active learning algorithms when given a moderate number of labeling budget.", "weaknesses": "I have major concerns with the writing and structure of the paper.\n\n1. The theoretical analysis and the actual algorithm seem to be very loosely connected. Currently, section 3 contains very little motivation from section 2. Also, the analysis framework makes assumption on universality making it unlikely to reflect the performance in practice.\n\n2. The data removal process from a labeled set is not immediate to the reader but is the key underlying tool for the derivate approximation. How does one remove labeled examples from a labeled set using an active learning algorithm?\n\n3. Throughout the paper, the data removal algorithms are used as TypiClust and inverse-TypiClust. Instead of formulating the algorithm as using general active learning algorithms for removal, the authors may discuss more in-depth why these two algorithms are actually effective in predicting low and high budget scenarios.\n\nFurthermore, the experiments seem to be inconsistent. Specifically, even though CIFAR-10 and CIFAR-100 have the same pool size, the experiments are conducted with inconsistent budgets. For example, 7k+1k and 25k+5k are used for CIFAR-10 while 9k+1k and 30k+7k are used for CIFAR-100. Since the paper's focus is on different labeling budgets, I believe more budget settings are also needed beyond just three.\n\nLastly, algorithm selection methods have been studied before, but not for different budgets. The paper could benefit from the following related work:\n1. Baram, Y., Yaniv, R. E., & Luz, K. (2004). Online choice of active learning algorithms. Journal of Machine Learning Research, 5(Mar), 255-291.\n2. Hsu, W. N., & Lin, H. T. (2015, February). Active learning by learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 29, No. 1).\n3. Pang, K., Dong, M., Wu, Y., & Hospedales, T. M. (2018, August). Dynamic ensemble active learning: A non-stationary bandit with expert advice. In 2018 24th International Conference on Pattern Recognition (ICPR) (pp. 2269-2276). IEEE.\n4. (Note the first version came out Feb 2023.) Zhang, J., Shao, S., Verma, S., & Nowak, R. (2023). Algorithm Selection for Deep Active Learning with Imbalanced Datasets. arXiv preprint arXiv:2302.07317.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Connections between the theoretical analysis and the actual algorithm presented:**\n\nThe theoretical analysis presented in our work conveys two main take-home messages:\n\n 1. The effectiveness of a derivative-like test, highlighted in Section 2.2 and emphasized in Equation 1, proves to be valuable in determining the budget that best suits the problem at hand. This theoretical result is put into action in the suggested algorithm, which perturbs the given labeled set to discern the appropriate budget regime.\n2. The theoretical analysis in Section 2.3 demonstrates that choosing a mixture of active learning (AL) strategies in each AL round can be well approximated by selecting a single AL strategy in each round and allowing for the flexibility to switch strategies between rounds. This insight significantly simplifies the algorithm, as it shifts the focus from selecting the best mixture of several strategies to making a binary choice between two distinct strategies in a pure manner. Additionally, as shown in Fig. 5, allowing for a different pure AL strategy in each round outperforms the use of a single pure strategy throughout, validating the predictions made in our analysis.\n\nWe understand from the reviews that these connections, between the analysis and the empirical section, could be better elucidated in the current manuscript. Accordingly, for the camera-ready version, we will add a comprehensive discussion that explicitly emphasizes these points, thereby reinforcing the cohesion between the theoretical foundations and the practical ingredients of the proposed approach.\n\n---\n**Removing data using AL algorithm, and the choice of TypiClust and inverse TypiClust:**\n\nThe rationale for choosing the specific methods of TypiClust and inverse-TypiClust is the following:\n\nTo remove examples from a labeled set using an active learning algorithm, we must be able to consider the AL algorithm as if it operates with an empty labeled set, denoted by $\\mathbb{L}$, while its unlabeled pool $\\mathbb{U}$ is composed of the original labeled dataset. This restricts our choice of AL algorithms quite drastically. This is because most (if not all) uncertainty-based algorithms are not  well-defined in such cases, since they typically make use of a classifier trained on $\\mathbb{L}$, which in our scenario is empty. In contrast, TypiClust [1] and Inverse-TypiClust present suitable solutions for these settings, as they tackle a covering problem directly on the data itself, independent of any specific classifier trained on $\\mathbb{L}$. Additionally, in the original paper it was shown that typiclust is a very good performer when given an empty $\\mathbb{L}$.\n\nBased on the above, and the lack of competitive alternative methods that can be effective when $\\mathbb{L}$ is empty, we chose TypiClust and inverse-TypiClust. This choice was empirically tested (see Fig. 6), where it was shown to be effective.\n\nWhile this concept is presently explained in the manuscript in the paragraph starting at line 218, we will emphasize and clarify this point in future revisions, as it is an important part of our suggested algorithm.\n\nNevertheless, it is essential to highlight that the proposed algorithm is not exclusive to this choice of TypiClust and Inverse-TypiClust. Alternative AL strategies, such as ProbCover [2] as a low-budget option or CoreSet [3] as a high-budget option, can also be effective choices in place of TypiClust and Inverse-TypiClust, respectively. In Fig. 1 in the attached PDF, we report the decisions made by SelectAL when using ProbCover and CoreSet instead of TypiClust and inverse-Typiclust, demonstrating this point. We will add this figure to the appendix, and discuss it in the main paper.\n\n---\n**Different budget settings for CIFAR-10 and CIFAR-100**\n\nThe size of the low-budget regime varies across different tasks. Intuitively, in the low-budget regime, you only have enough examples to provide a coarse description of the underlying problem. As the difficulty of the task increases, the low-budget regime becomes larger. Since CIFAR-100 is a more challenging problem compared to CIFAR-10 due to its tenfold increase in the number of classes, it has a larger low-budget regime despite both datasets sharing the same pool size.\n\nTable 1 in the manuscript showcases specific settings for a single-round active learning scenario, selected to be representative of different budget levels one might encounter. Considering that the low budget in CIFAR-100 is larger, we appropriately adjusted the settings for the mid-budget and high-budget scenarios to reflect this difference.\n\n\nHaving said all that, it is important to note that Table 1 is a **showcase**, and by no means shows all the conditions tested. For example, Fig. 5 shows results with many more cases. It summarizes the results of experiments with SelectAL in multi-round AL settings, effectively testing it in a large array of possible budgets, including low, mid, and high budgets, for both CIFAR-10 and CIFAR-100. This specific figure clearly shows that SelectAL works well in many settings, where it outperforms other AL strategies by a significant margin.\n\n---\n**Related work**\n\nThank you for bringing this set of interesting papers to our attention. Indeed they are not directly related to our work, as they focus on selecting AL algorithms under different settings than ours. Nevertheless, it will strengthen the paper to discuss our method in a broader context. This will be done for the camera-ready revision.\n\n---\n[1] Hacohen, Guy, et al. \"Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Yehuda, Ofer, et al. \"Active learning through a covering lens.\" Advances in Neural Information Processing Systems 35 (2022): 22354-22367.\n\n[3] Sener, Ozan, and Silvio Savarese. \"Active Learning for Convolutional Neural Networks: A Core-Set Approach.\" International Conference on Learning Representations. 2018."}, {"review_id": "0Y0zxl1pNz", "reviewer": "Reviewer_5Syc", "summary": "This paper considers the selection of active learning methods under different budgets. Previous studies have shown that different active learning methods perform differently under different labeling budgets, and incorrect selection can lead to model performance inferior to that of random sampling baselines. To address this issue, the authors conducted theoretical analysis and proposed the SelectAL method, and related experiments showed that this method can achieve consistent performance improvements under different budgets.", "questions": "1. What is the physical meaning of $E_{low}(x)=E_{high}(\\alpha x)$ in the assumptions (Section 2.1)? Can you provide a more intuitive explanation?\n2. The theoretical results (Section 2) assume that there is a unique and minimal solution for Eq.(1). What kind of E satisfies this assumption, and is this condition general enough to support practical applications? \n3. The proposed method (Section 3) seems weakly connected to the theoretical analysis. The theoretical results show that under specific assumptions, there exist sampling boundaries B_low and B_high for the Low-Budget Method and High-Budget Method, which can be calculated through closed-form solutions. However, the SelectAL method used in the proposed method relies on a training-evaluating method to identify the type of samples that are currently lacking. Can you provide more connections between theoretical results and practical methods?\n4. In the experiments (Figure 6), there are indeed performance differences between different AL methods, but the differences are small (≤2%). Have you observed more severe phenomena? In Tables 1 and 2, there are similar phenomena, and the performance differences between all methods are not significant. Specifically, in the low-budget scenario, the classification performance is so poor that the model cannot be deployed. Is it necessary to introduce complex methods to deal with this? Perhaps, random sampling followed by the consideration of high-budget methods is already sufficient.\n\n---  \nThanks for the detailed clarifications, which have addressed my concerns. I would like to keep my score.", "limitations": "The authors do not provide a discussion about the limitations and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea presented is interesting and it tackles an important issue in active learning. \n2. The experimental results indicate that the proposal consistently improved performance, supporting the claims made in the paper.", "weaknesses": "1. There is room for improvement in the technological representation.  \n2. The connection between theoretical results and practical methods seems to be weak.  \nMiscellaneous: The text font in the figures should be kept the same as the main body.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**$B_{low}$ and $B_{high}$, and the connection between the theoretical analysis and the practical methods**\n\nIndeed, the theoretical framework presented in this study relies on a simplified model. This makes possible a rigorous analysis and the derivation of closed-form solutions, unlike the involved analysis of real-world deep network training scenarios. While these closed-form expressions may be irrelevant in real deep learning scenarios, approximating $B_{low}$ and $B_{high}$ plays a vital role in SelectAL. The removal of examples in Algorithm 1 serves the purpose of calculating whether the current budget falls above or below these thresholds, as explained in detail in Figure 6. Overall, our whole approach is greatly motivated by the theoretical results; in effect, the main algorithm puts the theoretical results into actual use by empirically evaluating the thresholds and boundaries identified in the analysis.\n\nThe following discussion may clarify further connections between the theoretical results and the actual method. We note that the proposed theory conveys two main such insights, which are implemented and tested in our suggested method:\n\n1. The effectiveness of a derivative-like test, highlighted in Section 2.2 and emphasized in Equation 1, proves to be valuable in determining the budget that best suits the problem at hand. This theoretical result is put into action in the suggested algorithm, which perturbs the given labeled set to discern the appropriate budget regime.\n\n2. The theoretical analysis in Section 2.3 demonstrates that choosing a mixture of active learning (AL) strategies in each AL round can be well approximated by selecting a single AL strategy in each round and allowing for the flexibility to switch strategies between rounds. This insight significantly simplifies the algorithm, as it shifts the focus from selecting the best mixture of several strategies to making a binary choice between two distinct strategies in a pure manner. Additionally, as observed in Fig. 5, allowing for a different pure AL strategy in each round outperforms the use of a single pure strategy throughout, validating the predictions made in our analysis.\n\nFor the camera-ready version, we will add a comprehensive discussion that explicitly emphasizes these points, thereby reinforcing the cohesion between the theoretical foundations and the practical ingredients of the proposed approach.\n\n---\n**$E_{low}(x)=E_{high}(\\alpha x)$**\n\nThe error functions $E_{low}$ and $E_{high}$ measure the generalization error of a classifier as a function of the size of its training data. For example, it could be some exponential decreasing function: the error drops fast if you increase the size of the training data at first, but as more data is added there is some diminishing returns effect.\n\n$E_{low}$ and $E_{high}$ are 2 such functions, for 2 similar classifiers, trained on either an easy dataset, corresponding to the low-budget, or a hard dataset, corresponding to the high budget. The $E_{low}(x)=E_{high}(\\alpha x)$ states that the error functions in such case would be from a similar family of functions, i.e if $E_{low}$ decreases exponentially, then $E_{high}$ also does, but with a different rate of decay. This assumption is rather realistic -- measuring the error functions of neural networks on different datasets indeed shows that they decay in an exponential-like manner but with different decay rates.\n\n---\n**Uniqueness of Eq. 1**\n\nIn the case where there are multiple solutions for Eq. 1, the core analysis and its qualitative implications remain unchanged. In such case, we observe the existence of several thresholds $B_{eq}$, each having its distinct $B_{low}$ and $B_{high}$. To maintain consistency in our analysis, we can employ the minimal threshold as $B_{low}$ and the maximal threshold as $B_{high}$, while adopting a random AL strategy within this range. It's important to highlight that in practical scenarios when training deep networks on real datasets, the observed solutions have consistently been unique (see Fig. 6 for a concrete example).\n\n---\n**Size of the differences observed**\n\nIn Tables 1 and 2, we acknowledge that the differences between models trained using active learning (AL) strategies and those trained on random examples might appear relatively small. This is due to the fact that we consider a single round of AL, where only a limited number of examples are chosen by the AL strategy. Consequently, the impact of these few added examples on the overall training process may not be substantial. However, it is important to note that the primary objective in this context is to assess whether there is an improvement or deterioration resulting from the AL approach.\n\nIn contrast, in Fig. 5, we demonstrate the results of a multi-round AL experiment, where a larger fraction of examples are selected by AL strategies. In this multi-round setting, the improvements achieved through active learning are significant and noticeable.\n\nRegarding the low-budget results, it is essential to consider that we are dealing with a dataset containing only 200 examples. In such limited data scenarios, achieving substantial improvements beyond what is reported in the tables can be challenging. This closely mirrors real-life situations where data can be sparse, making active learning techniques all the more valuable.\n\nHowever, we also recognize that there are potential ways to enhance results in low-budget scenarios, especially when unlabeled examples are available, and a semi-supervised approach is employed. In such cases, the separation between low and high-budget regimes has been demonstrated, and the use of SelectAL may be advantageous. While this is a promising avenue, we want to emphasize that our manuscript's main focus is not on semi-supervised scenarios but on showcasing the effectiveness of SelectAL in identifying the budget regime of the problem in advance."}, {"review_id": "R618SUCrIC", "reviewer": "Reviewer_mgMr", "summary": "Some recent works have shown that typical examples are more beneficial than more uncertain ones under low budget setting in active learning. However, there has been no prior works that address how to distinguish whether a budget is in low or high budget regime, or how to determine which approach should be taken.\n\nThe authors propose SelectAL, an algorithm that handles the tasks using derivative-based test, for the first time. The algorithm is theoretically and empirically supported in Section 2 and 4, respectively. ", "questions": "1. I am very confused with the definition of the generalization errors. If my understanding is correct, $E_{low}$ and $E_{high}$ are the generalization errors on the same unseen test set using $\\mathcal{L}_{low}$ and $\\mathcal{L}_{hight}$, each of which are trained on $\\mathcal{D}_{low}$ and $\\mathcal{D}_{high}$. How is it related to the equation in line 111? \n2. What the equation in line 111 is saying is that the generalization error of $\\mathcal{L}$ trained on $B$ examples can be expressed as the convex combination of $E(qB)$ and $E(\\alpha(1-q)B)$. But, $E(qB)$ is from a learner trained only on $qB$ examples and $E(\\alpha(1-q)B)$ is from the other learner trained only on $(1-q)B$ examples. Is it generally true or even close to the reality? If the generalization error is a function of the test examples given the same learner, I think the convex combination makes sense but since it is a function of training examples, I am not sure it is reasonably true. \n3. Where are equalities for q coming from for $B_{low}$ and $B_{high}$ in  Appendix A?\n4. If the uniqueness of $B_{eq}$ does not hold, what happens to the Eq.(3), particularly, for the case $|A| \\rightarrow 0$?\n5. Could the authors provide more details about the experiment setting used to produce Figure 2 such as a linear $\\mathcal{L}$, the dataset, and reasoning behind of choosing exponential error function?\n6. For selectAL, did the authors observe some cases where {low, high, rand} are selected alternatively unlike the expected monotonic behavior? Can the authors provide the sequence of selections on CIFAR-10 and ImageNet-50?", "limitations": "Please refer to the weakness and questions above. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is the first paper that addresses how to automatically determine selection strategies based on the budget in active learning.\n2. The authors provide some theoretical justification for the derivative-based test method. ", "weaknesses": "1. Although Strength 1 is a great contribution of this paper, there are several assumptions some of which may not be realistic. Intuitively, efficiency and realizability are reasonable assumptions but it would be hard to check if universality is generally true or not. Furthermore, the uniqueness of $B_{eq}$ (line 120-121) seems like a strong assumption. \n2. It is somewhat hard to follow the flow of this paper. It is partially because of the complexity of theoretical work, but also because there are many assumptions to justify the theory. But upon the clarification of the questions below, I am willing to increase the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks much for the detailed and precise list of questions. We wish to emphasize that some brevity in the manuscript was called for because the general framework is not original - it is adopted with all its assumptions from the work of [1]. This study demonstrated that the model, despite its simplified assumptions, allows for a clear differentiation between low and high budgets, and can further be subjected to rigorous analysis. We note that due to the inherent simplifications in the model, some assumptions are hard to justify in realistic settings and therefore any conclusion derived from the model requires exhaustive empirical testing to validate its relevance in real settings.\n\nAnswers to questions:\n\n---\n1+2)\n\nThe model under analysis consists of two learners, $L_{low}$ and $L_{high}$, trained on separate datasets $D_{low}$ and $D_{high}$, respectively. $L_{low}$ is trained and evaluated solely on $D_{low}$, while $L_{high}$ on $D_{high}$.\n\nWe then define a mixture model, combining $L_{low}$ and $L_{high}$. The model is trained and tested on a mixture distribution, where each example is drawn from $L_{low}$ with probability $p$ and from $L_{high}$ with probability $(1-p)$.\n\nThe main question such a framework can address is whether changing the training distribution of the combined model, while keeping the test set unchanged, is beneficial. Under certain assumptions, the results suggest that in low-budget scenarios, more examples should come from $D_{low}$, and in high-budget scenarios, more examples should come from $D_{high}$.\n\nSince the distribution of the data is a convex combination of $2$ distributions, so is the error of the combined model, which is depicted in line 111. A combined model that gets $qB$ examples from $D_{low}$ and $(1-q)B$ examples from $D_{high}$ is expected to have a generalization error of $pE(qB) + (1-p)E(\\alpha (1-q)B)$ on the mixture distribution.\n\n---\n3)\n\nIn Eq. 4, with specific values for $p$ and $q$, solving for $B$ can determine the budget at which selecting $qB$ examples from $D_{low}$ yields the best generalization error. By setting $p=q$, we find a budget ($B_{eq}$) where the optimal strategy is to keep the distribution unchanged, namely, active learning is not needed. Any smaller budget would necessitate a low-budget strategy, while a higher budget would require a high-budget strategy.\n\nIn practice, we only control the source from which the active set of examples $\\mathbb{A}$ is sampled, but not the source of the remaining $B-|\\mathbb{A}|$ examples. Sampling from $D_{low}$ all the examples in $|\\mathbb{A}|$, results in a total of $\\left(p+\\frac{\\mathbb{A}(1-p)}{B}\\right)B$ examples being sampled from $D_{low}$ overall. Plugging such $q$ in Eq. 4, we obtain the maximal budget $B$ for which this strategy is favorable. We refer to this budget as $B_{low}$. Similarly, the concept applies to $B_{high}$, which defined the smallest budget for which sampling $\\mathbb{A}$ from $D_{high}$ is optimal.\n\n---\n4)\n\nIf $B_{eq}$ is not unique, it implies that Eq. 4 have multiple solutions, suggesting the presence of a multitude of low and high-budget regimes. However, in our empirical investigations, we have not observed this phenomenon.\n\nWhen multiple $B_{low}$ and $B_{high}$ values exist, they delineate distinct budget thresholds. For budgets below the smallest $B_{low}$ value, the best strategy involves sampling the entire active set from $D_{low}$. Conversely, for budgets surpassing the largest $B_{high}$ value, the optimal strategy is to sample the entire active set from $D_{high}$. In the intermediate range between these two thresholds, determining the precise optimal strategy becomes more challenging, and random sampling may be preferred. As $|\\mathbb{A}|\\leftarrow 0$, the minimum $B_{low}$ approaches the maximum $B_{high}$, resulting in the same optimal strategy in the limit as described in Eq. 3.\n\n---\n5)\n\nThere is no specific dataset or learner in the experiment in Fig 2. Once the error function is defined, datasets and learners are not explicitly required -- the analysis is done based on the error function alone.\n\nThe reason we chose the exponential form for the error function is twofold. Firstly, when examining the error functions of neural networks on real datasets, they often exhibit behavior that resembles an exponential pattern. Thus, using an exponential form in this example allows us to analyze a scenario that aligns with the behavior of empirical error functions in realistic scenarios. Secondly, we opted to utilize the same example from [1], as it was a recognized and relevant illustration, ensuring continuity with previous art.\n\n---\n6)\n\nIn all our experiments and across various settings, we consistently observed a monotonic pattern in SelectAL's behavior. Specifically, SelectAL followed a sequence of low-budget strategy for several rounds, then a random strategy for another set of rounds, then a high-budget strategy for the remaining rounds.\n\nFor both CIFAR-10 and CIFAR-100 datasets, as depicted in Fig 5, the selection sequence of SelectAL was as follows: It initially employed the low-budget strategy for 3 iterations. then random strategy for 1 iteration and rom that point onwards, it favored the high-budget strategy. The specific values for these iterations are highlighted in Fig 5, using orange and green dashed lines.\n\n---\nIn postfix, we sincerely wish to thank you for your valuable feedback. This thorough review of our work and the insightful questions you raised help us recognize the areas that need further clarification, strengthening our work.\n\nIn the camera-ready revision, we will carefully address each of the concerns you have raised, aiming to enhance the overall coherence and comprehensibility of the manuscript. We are committed to incorporating your feedback to deliver a more polished and reader-friendly paper.\n\n---\n[1] Hacohen, Guy, et al. \"Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets.\" ICML 2022."}]

Author rebuttal:
We thank all the reviewers for the significant time and effort they made in reviewing our work.

In the attached PDF, we added an experiment suggested by Reviewer GcEN, in which we use other active learning strategies, namely ProbCover [1] and CoreSet [2], as the decision rules for SelectAL. Specifically, this experiment can be viewed as an extension of Figure 6 in the original manuscript, showing that SelectAL will make similar choices for a variety of active learning strategies. We believe that this addition enhances the comprehensiveness of our findings and strengthens the overall contribution of our work.

---

[1] Yehuda, Ofer, et al. "Active learning through a covering lens." Advances in Neural Information Processing Systems 35 (2022): 22354-22367.

[2] Sener, Ozan, and Silvio Savarese. "Active Learning for Convolutional Neural Networks: A Core-Set Approach." International Conference on Learning Representations. 2018.

Meta-review: This paper was reviewed by four experts in the field and received one Weak Accept and three Borderline Accept as the ratings. The reviewers have agreed that the paper addresses an important problem in active learning research experimental results demonstrate the effectiveness of the proposed method. 

The reviewers have mentioned that the theoretical analysis and the presented algorithm seem to be loosely connected, which has been addressed by the authors in the rebuttal. Concerns were also raised about the inconsistencies in the experimental setup and budget settings, which were also addressed convincingly by the authors in the rebuttal. 

The reviewers, in general, have a positive opinion about the paper and its contributions. Most of them have been satisfied with the authors’ rebuttal and have recommended acceptance. Based on the reviewers’ feedback, the decision is to recommend the paper for acceptance to NeurIPS 2023. The reviewers have provided some valuable suggestions, such as including a pseudo-code of the proposed method in the Appendix, clearly explaining the connection between the theoretical analysis and the algorithm proposed and adding a few more reference papers in the Related Work section. The authors are encouraged to address these in the final version of their paper. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)