SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The theoretical framework presented here relies on a simplified model, enabling a rigorous analysis with closed-form solutions. While these solutions may not be directly applicable in real-world deep learning scenarios, they serve as a solid motivation for the SelectAL approach. Empirical evidence demonstrates their practical effectiveness.
In Section 2.2, we introduced a derivative-based test, demonstrating that measuring the model's error function on small data perturbations suffices to determine the appropriate strategy for each budget. This concept serves as the fundamental principle of SelectAL, a method that chooses an active learning strategy by analyzing the model's error function in response to data perturbations.
In Section 2.3, we showed that while the optimal mixed AL strategy can be determined analytically, a practical approximation can be achieved by choosing a single pure AL strategy for each round in active learning. This approach significantly simplifies SelectAL, as it doesn't require the creation of a distinct mixed AL strategy tailored separately to each budget. Instead, SelectAL selects a single pure AL strategy for each active learning round, from a set of existing "pure" AL strategies. In Fig. 5, we observe that this selection consistently outperforms individual pure strategies, confirming our analytical predictions. We also explored various combinations of existing AL strategies, none of which showed a significant improvement over SelectAL, further reinforcing the relevance of predictions of the theoretical analysis.
3 SelectAL: automatic selection of active learning strategy
In this section, we present SelectAL - a method for automatically selecting between different AL strategies in advance, by estimating dynamically the relative budget size of the problem at hand. The decision whether  $B$  should be considered "large" or "small" builds on the insights gained from the theoretical analysis presented in section 2. The suggested approach approximates the derivative-based test suggested in (1), resulting in a variation of the attainable optimal mixed strategy (3).
SelectAL consists of two steps. The first step involves using a version of the derivative-based rule from Section 2.2 to assess whether the current budget  $B$  for the given problem should be classified as "high" ( $B \geq B_{high}$ ) or "low" ( $B \leq B_{low}$ ). This is accomplished by generating small perturbations of  $\mathbb{L}$  based on both low and high-budget strategies, and then predicting if the current quantity of labeled examples is adequate to qualify as a high-budget or low-budget.
In the second step, we select the most competitive AL strategy from the relevant domain and use it to select the active set  $\mathbb{A}$ . This approach not only ensures good performance within the specified budget constraints but also provides robustness across a wide range of budget scenarios. It is scalable in its ability to incorporate any future development in active learning methods.
3.1 Deciding on the suitable budget regime
In the first step, SelectAL must determine whether the current labeled set  $\mathbb{L}$  should be considered low-budget ( $B \leq B_{low}$ ) or high-budget ( $B \geq B_{high}$ ) for the problem at hand. The challenge is to make this decision without querying additional labels from  $\mathbb{U}$ .
To accomplish this, SelectAL requires access to a set of active learning strategies for both low and high-budget scenarios. We denote such strategies by  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$ , respectively. Additionally, a random selection strategy, denoted by  $S_{rand}$ , is also considered.
To determine whether the current budget is high or low, SelectAL utilizes a surrogate test. Instead of requiring additional labels, we compute the result of the test with a derivative-like approach. Specifically, for each respective strategy separately, we remove a small set of points (size  $\epsilon > 0$ ) from  $\mathbb{L}$ , and compare the reduction in generalization error to the removal of  $\epsilon$  randomly chosen points. To implement this procedure, we remove the active sets chosen by either  $S_{low}^{\prime}$  or  $S_{high}^{\prime}$ , when trained using the original labeled set  $\mathbb{L}$  as the unlabeled set and an empty set as the labeled set.
The proposed surrogate test presents a new challenge: in most active learning strategies, particularly those suitable for high budgets, the outcome relies on a learner that is trained on a non-empty labeled set. To overcome this, we restrict the choice of active learning strategies  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$  to methods that rely only on the unlabeled set  $\mathbb{U}$ . This added complexity is crucial: using the labels of set  $\mathbb{L}$  by either  $S_{low}^{\prime}$  or  $S_{high}^{\prime}$  results in a bias that underestimates the cost of removing known points, as is demonstrated in our empirical study (see Fig. 7). No further restriction on  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$  is needed. Empirically, we evaluated several options for  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$ , including TypiClust and ProbCover for  $S_{low}^{\prime}$  and inverse-TipiClust and CoreSet for  $S_{high}^{\prime}$ , with no qualitative difference in the results.
Our final method can be summarized as follows (see Alg. 1): We generate three subsets from the original labeled set  $\mathbb{L}$ : data $_{low}$ , data $_{high}$  and data $_{rand}$ . Each subset is obtained by asking the corresponding strategy  $-S_{low}^{\prime}$ ,  $S_{high}^{\prime}$ , and  $S_{rand}^{\prime}$  to choose a class-balanced subset of  $\epsilon \cdot |\mathbb{L}|$  examples from  $\mathbb{L}$ . Note that this is unlike their original use, as AL strategies are intended to select queries from the unlabeled set  $\mathbb{U}$ . Importantly, since set  $\mathbb{L}$  is labeled, we can guarantee that the selected subset is class-balanced. Subsequently, the selected set is removed from  $\mathbb{L}$ , and a separate learner is trained on each of the 3 subsets (repeating the process multiple times for  $S_{rand}$ ). Finally, we evaluate the accuracy of each method using cross-validation, employing  $1\%$  of the training data as a validation set in multiple repetitions. The subset with the lowest accuracy indicates that the subset lacks examples that are most critical for learning. This suggests that the corresponding strategy queries examples that have the highest impact on performance.
Algorithm 1 SelectAL
1: Input:  $S_{high}^{\prime}, S_{low}^{\prime}, S_{high}, S_{low}, S_{rand}$ , labeled data  $\mathbb{L}, \epsilon > 0$
2: Output: AL strategy
3:  $c\gets \max \{\left\lfloor \frac{\epsilon}{\text{of classes}}\right\rfloor ,1\}$
4: for  $i \in \{\text{low, high, rand}\}$  do
5: removedexamples  $\leftarrow$  query c examples from each class from  $\mathbb{L}$  using  $S_{i}^{\prime}$
6: data  $i\gets \mathbb{L}\backslash$  removedexamples
7:  $acc_{i}\gets$  model's accuracy on data
8: end for
9: strategy  $\leftarrow$  arg min  $i\in \{low,high,rand\}$ $(acc_{i})$
10: return  $S_{strategy}$
(b) CIFAR-100
Figure 5: Accuracy improvement, relative to no active learning (random sampling), for 10 rounds of different AL strategies. We see that while some methods work well only in low budgets and some in high budgets, SelectAL works well in all budgets. Before the first dashed line, SelectAL picked a low-budget strategy, between the dashed lined it picked a random strategy, and after the second dashed line it picked a high-budget strategy. In both experiments, SelectAL picked  $S_{low}$  for the first 3 iterations, followed by  $S_{rand}$  for 1 iteration and then  $S_{high}$ .
3.2 Selecting the active learning strategy
In its second step, SelectAL selects between two active learning strategies,  $S_{low}$ , and  $S_{high}$ , which are known to be beneficial in the low and high-budget regimes respectively. Unlike  $S_{low}'$  and  $S_{high}'$ , there are no restrictions on  $\{S_{low}, S_{high}\}$ , from which Alg. 1 selects, and it is beneficial to select the SOTA active learning strategy for the chosen domain.
Somewhat counter-intuitively, SelectAL is likely to use different pairs of AL strategies, one pair to determine the budget regime, and possible a different one for the actual query selection. This is the case because in its first step,  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$  are constrained to use only the unlabeled set  $\mathbb{U}$ , which may eliminate from consideration the most competitive strategies. In contrast, and in order to achieve the best results,  $S_{low}$  and  $S_{high}$  are chosen to be the most competitive AL strategy in each domain. This flexibility is permitted because both our theoretical analysis in Section 2 and our empirical analysis in Section 4 indicate that the transition points  $B_{low}$  and  $B_{high}$  are likely to be universal, or approximately so, across different strategies. This is especially important in the high-budget regime, where the most competitive strategies often rely on both  $\mathbb{L}$  and  $\mathbb{U}$ .
We now describe the results of an extensive empirical evaluation of SelectAL. After a suitable AL strategy is selected by Alg. 1, it is used to query  $\mathbb{A}$  unlabeled examples. The training of the deep model then proceeds as is customary, using all the available labels in  $\mathbb{A} \cup \mathbb{L}$ .
4.1 Methodology
Our experimental framework is built on the codebase of [28], which allows for fair and robust comparison of different active learning strategies. While the ResNet-18 architecture used in our experiments may not achieve state-of-the-art results on CIFAR and ImageNet, it provides a suitable platform to evaluate the effectiveness of active learning strategies in a competitive environment, where these strategies have been shown to be beneficial. In the following experiments, we trained ResNet-18 [18] on CIFAR-10, CIFAR-100 [23] and ImageNet-50 - a subset of ImageNet [7] containing 50 classes as done in [38]. We use the same hyper-parameters as in [28], as detailed in Suppl. B.
SelectAL requires two types of active learning strategies: restricted strategies that use only the unlabeled set  $\mathbb{U}$  for training, and unrestricted competitive strategies that can use both  $\mathbb{L}$  and  $\mathbb{U}$  (see discussion above). Among the restricted strategies, we chose TypiClust [13] to take the role of  $S_{low}^{\prime}$ , and inverse TypiClust for  $S_{high}^{\prime}$ . In the latter strategy, the most atypical examples are selected. Note that inverse TypiClust is an effective strategy for high budgets, while relying solely on the unlabeled set  $\mathbb{U}$  (see Suppl. C.1 for details). Among the unrestricted strategies, we chose ProbCover [39] for the role of the low-budget strategy  $S_{low}$ , and BADGE [1] for the high-budget strategy  $S_{high}$ . Other choices yield similar patterns of improvement, as can be verified from Tables 1-2.
In the experiments below, we use several active learning strategies, including Min margin, Max entropy, Least confidence, DBAL [10], CoreSet [33], BALD [22], BADGE [1], TypiClust [13] and ProbCover [39]. When available, we use for each strategy the code provided in [28]. For low-budget strategies, which are not implemented in [28], we use the code from the repository of each paper.
4.2 Evaluating the removal of examples with AL in isolation
We isolate the strategy selection test in Alg. 1 as described in Section 3.1. To generate the 3 subsets of labeled examples data<sub>low</sub>, data<sub>high</sub> and data<sub>Rand</sub>, we remove  $5\%$  of the labeled data, ensuring that we never remove less than one data point per class. In the low-budget regime, removing examples according to  $S_{low}^{\prime}$  yields worse performance as compared to the removal of random examples, while better performance is seen in the high-budget regime. The opposite behavior is seen when removing examples according to  $S_{high}^{\prime}$ . Different choices of strategies for  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$  yield similar qualitative results, see Suppl. C.2.
More specifically, we trained 10 ResNet-18 networks on each of the 3 subsets, for different choices of budget  $B$ . In Fig. 6, we plot the difference in the mean accuracy of networks trained on data<sub>low</sub> and data<sub>high</sub>, compared to networks trained on data<sub>Rand</sub>, similarly to the proposed test in Alg. 1. In all
the budgets that are smaller than the orange dashed line, SelectAL chooses  $S_{low}$ . In budgets between the orange and the green dashed lines, SelectAL chooses  $S_{rand}$ . In budgets larger than the green dashed line, SelectAL chooses  $S_{high}$ .
(b) CIFAR-100
Figure 6: Accuracy gain when using  $S_{low}^{\prime}$  to select points for removal as compared to random selection (orange), or  $S_{high}^{\prime}$  to select points for removal (green). Negative gain implies that the strategy is beneficial, and vice versa.
(c) ImageNet-50
Table 1: Mean accuracy and standard error of 10 ResNet-18 networks trained on CIFAR-10 and CIFAR-100, using various budgets and active learning strategies. In each dataset, we display results for 3 budget choices: one smaller than  $\bar{B}_{low}$  (left column), one between  $B_{low}$  and  $B_{high}$  (middle column), and one larger than  $B_{high}$  (right column). We highlight in boldface the best result in each column, and additionally all the results that lie within its interval of confidence (the standard error bar). While most strategies are effective only in low or in high budgets, SelectAL is effective in both regimes. As predicted, between  $B_{low}$  and  $B_{high}$ , most AL strategies do not significantly outperform random query selection.
Budget (L + A)	CIFAR-10	CIFAR-100
100+100	7k+1k	25k+5k	100+100	9k+1k	30k+7k
random	31.8 ± 0.3	76 ± 0.3	87.2 ± 0.2	5.3 ± 0.2	39.9 ± 0.3	60.7 ± 0.3
TypiClust	34.1 ± 0.4	75.7 ± 0.3	87.1 ± 0.2	7.1 ± 0.1	39.7 ± 0.4	60.4 ± 0.2
BADGE	31.3 ± 0.4	76.5 ± 0.4	88.1 ± 0.1	5.3 ± 0.2	39.5 ± 0.3	61.9 ± 0.1
DBAL	30.2 ± 0.3	76.4 ± 0.4	87.8 ± 0.1	4.6 ± 0.2	38.9 ± 0.4	61.5 ± 0.2
BALD	30.8 ± 0.3	76.3 ± 0.2	88 ± 0.2	4.9 ± 0.2	39.8 ± 0.5	61.5 ± 0.2
CoreSet	29.4 ± 0.4	75.8 ± 0.3	87.7 ± 0.2	5.6 ± 0.4	39.1 ± 0.3	61.4 ± 0.2
ProbCover	35.1 ± 0.3	76.1 ± 0.3	87.1 ± 0.1	8.2 ± 0.1	40 ± 0.4	61.4 ± 0.3
Min Margin	30.7 ± 0.4	71.1 ± 0.2	87.9 ± 0.2	5.2 ± 0.1	39.2 ± 0.2	61.6 ± 0.3
Max Entropy	30.2 ± 0.3	76.1 ± 0.3	87.8 ± 0.2	4.9 ± 0.2	39 ± 0.2	61.6 ± 0.2
Least Confidence	29.7 ± 0.2	76.1 ± 0.3	88.1 ± 0.2	4.7 ± 0.4	38.9 ± 0.4	61.5 ± 0.2
SelectAL	35.1 ± 0.3	76 ± 0.3	88.1 ± 0.1	8.2 ± 0.1	39.9 ± 0.3	61.9 ± 0.1
4.3 SelectAL: results
In Fig. 5, we present the average accuracy of a series of experiments involving 10 ResNet-18 networks trained over 10 consecutive AL rounds using three AL strategies:  $S_{low}$  (ProbCover),  $S_{high}$  (BADGE), and our proposed method SelectAL. We compare the accuracy improvement of each strategy to training without any AL strategy. In each AL round, SelectAL selects the appropriate AL strategy based on Alg. 1. Specifically, SelectAL selects  $S_{low}$  when the budget is below the orange dashed line,  $S_{rand}$  when the budget is between the orange and green dashed lines, and  $S_{high}$  when the budget is above the green dashed line. We observe that while  $S_{low}$  and  $S_{high}$  are effective only for specific budgets, SelectAL performs well across all budgets. In all the experiments we performed, across all datasets, we always observed the monotonic behavior predicted by the theory - SelectAL picks  $S_{low}$  for several AL iterations, followed by  $S_{rand}$  and then  $S_{high}$ .
It is important to note that, unlike Fig. 6, where the data distribution of set  $\mathbb{L}$  is sampled from the original distribution  $\mathcal{D}$  because we analyze the first round (see Section 2.3), in the current experiments the distribution is unknown apriori -  $\mathbb{L}$  in each iteration is conditioned on the results of previous iterations, and is therefore effectively a combination of  $S_{low}$ ,  $S_{high}$ , and  $S_{rand}$ . Consequently, the transition point determined automatically by Alg. 1 occurs earlier than the one detected in Fig. 6. Notably, while SelectAL chooses an existing AL strategy at each iteration, the resulting strategy outperforms each of the individual AL strategies when trained alone.
In Tables 1-2, we show the performance of SelectAL in comparison with the performance of the baselines (Section 4.1). In all these experiments, SelectAL succeeds to identify a suitable budget regime. As a result, it works well both in the low and high-budget regimes, matching or surpassing both the low and high-budget strategies at all budgets. Note that as SelectAL chooses an active learning strategy dynamically for each budget, any state-of-the-art improvements for either low or high budgets AL strategies can be readily incorporated into SelectAL.
Table 2: Same as Table 1, mean and standard error of 10 ResNet-18 networks trained on ImageNet 50 using different AL strategies, at low, medium, and high budgets.
B (L + A)	ImageNet-50
100+100	7k+1k	25k+5k
random	9.3 ± 0.2	61.8 ± 0.4	79.8 ± 0.2
TypiClust	11.3 ± 0.3	61.8 ± 0.5	80.1 ± 0.2
BADGE	9.4 ± 0.2	61.3 ± 0.5	80.8 ± 0.2
DBAL	9 ± 0.4	61.1 ± 0.6	80.1 ± 0.2
BALD	9.4 ± 0.4	61.7 ± 0.3	80.7 ± 0.1
CoreSet	8.6 ± 0.3	61.7 ± 0.2	80.7 ± 0.3
ProbCover	11.4 ± 0.6	61.6 ± 0.8	77.7 ± 0.3
Min Margin	9.8 ± 0.2	61.2 ± 0.5	80.4 ± 0.1
Max Entropy	8.9 ± 0.2	61.8 ± 0.4	80.1 ± 0.1
Least Conf.	8.9 ± 0.1	61.5 ± 0.4	79.6 ± 0.5
SelectAL	11.4 ± 0.6	61.8 ± 0.4	80.8 ± 0.2
Figure 7: To assess the suitability of BADGE as an example removal strategy  $S_{high}^{\prime}$ , we compare it with the original  $S_{high}^{\prime}$  approach whose performance is reported in Fig. 6a, for CIFAR-10. Unlike the original  $S_{high}^{\prime}$  (depicted in green), BADGE (depicted in orange) exhibits a lack of a distinct transition point. Consequently, BADGE is not well-suited as a choice for  $S_{high}^{\prime}$ .
Why  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$  are Restricted? As discussed in Section 3.1, we have made a deliberate decision to exclude strategies that rely on the labeled set  $\mathbb{L}$  while deciding which family of strategies is more suited to the current budget. We now demonstrate what happens when the selection is not restricted in this manner, and in particular, if  $S_{high}^{\prime}$  is chosen to be a competitive AL strategy that relies on the labeled set  $\mathbb{L}$  for its successful outcome. Specifically, we repeat the experiments whose results are reported in Fig. 6a, but where strategy  $S_{high}^{\prime}$  – the one used for the removal of examples – is BADGE. Results are shown in Fig. 7. Unlike Fig. 6a, there is no transition point, as it is always beneficial to remove examples selected by BADGE rather than random examples. This may occur because the added value of all points used for training diminishes after training is completed.
Computational time of SelectAL SelectAL entails the training of three active learning strategies on a small dataset. Consequently, the computational duration of SelectAL is contingent upon the user's strategy choices. It is worth noting that, in practice, the additional computation time is negligible in comparison to conventional AL techniques. This is due to the fact that  $S_{low}^{\prime}$  and  $S_{high}^{\prime}$  are exclusively trained on  $\mathbb{L}$ , which is typically significantly smaller than  $\mathbb{U}$  - the set used by pure AL strategies. To further expedite computation, one might contemplate training on data perturbations using a more compact model, an issues which we defer the to future research.
5 Summary and discussion
We introduce SelectAL, a novel method for selecting active learning strategies that perform well for all training budgets, low and high. We demonstrate the effectiveness of SelectAL through a combination of theoretical analysis and empirical evaluation, showing that it achieves competitive results across a wide range of budgets and datasets. Our main contribution lies in the introduction of the first budget-aware active learning strategy. Until now, selecting the most appropriate active learning strategy given some data was left to the practitioner. Knowing which active learning strategy is best suited for the data can be calculated after all the data is labeled, but predicting this in advance is a difficult problem. SelectAL offers a solution to this challenge, by determining beforehand which active learning strategy should be used without using any labeled data.
Acknowledgement This work was supported by grants from the Israeli Council of Higher Education and the Gatsby Charitable Foundations.
[36] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972-5981, 2019.
[37] Alexandru Tifrea, Jacob Clarysse, and Fanny Yang. Uniform versus uncertainty sampling: When being active is less efficient than staying passive. arXiv preprint arXiv:2212.00772, 2022.
[38] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conference on Computer Vision, pages 268-285. Springer, 2020.
[39] Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active learning through a covering lens. In NeurIPS, 2022.
[40] Jifan Zhang, Shuai Shao, Saurabh Verma, and Robert Nowak. Algorithm selection for deep active learning with imbalanced datasets. arXiv preprint arXiv:2302.07317, 2023.