SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guy Hacohen†‡, Daphna Weinshall†
School of Computer Science & Engineering†
Edmond and Lily Safra Center for Brain Sciences†
The Hebrew University of Jerusalem
Jerusalem 91904, Israel
{guy.hacohen,daphna}@mail.huji.ac.il
In the domain of Active Learning (AL), a learner actively selects which unlabeled examples to seek labels from an oracle, while operating within predefined budget constraints. Importantly, it has been recently shown that distinct query strategies are better suited for different conditions and budgetary constraints. In practice, the determination of the most appropriate AL strategy for a given situation remains an open problem. To tackle this challenge, we propose a practical derivative-based method that dynamically identifies the best strategy for a given budget. Intuitive motivation for our approach is provided by the theoretical analysis of a simplified scenario. We then introduce a method to dynamically select an AL strategy, which takes into account the unique characteristics of the problem and the available budget. Empirical results showcase the effectiveness of our approach across diverse budgets and computer vision tasks.
Active learning emerged as a powerful approach for promoting more efficient and effective learning outcomes. In the traditional supervised learning framework, active learning enables the learner to actively engage in the construction of the labeled training set by selecting a fixed-sized subset of unlabeled examples for labeling by an oracle, where the number of labels requested is referred to as the budget. Our study addresses the task of identifying in advance the most appropriate active learning strategy for a given problem and budget.
The selection of an active learning strategy is contingent on both the learner's inductive biases and the nature of the problem at hand. But even when all this is fixed, recent research has shown that the most suited active learning strategy varies depending on the size of the budget. When the budget is large, methods based on uncertainty sampling are most effective. When the budget is small, methods based on typicality are most suitable (see Fig. 1). In practice, determining the appropriate active learning strategy based on the budget size is challenging, as a specific budget can be considered either small or large depending on the problem at hand. This challenge is addressed in this paper.
Specifically, we start by analyzing a simplified theoretical framework (Section 2), for which we can explicitly select the appropriate AL strategy using a derivative-based test. Motivated by the analysis of this model, we propose SelectAL (Section 3), which incorporates a similar derivative-based test to select between active learning strategies. SelectAL aims to provide a versatile solution for any budget, by identifying the budget domain of the problem at hand and picking an appropriate AL method from the set of available methods. SelectAL is validated through an extensive empirical study using several vision datasets (Section 4). Our results demonstrate that SelectAL is effective in identifying the best active learning strategy for any budget, achieving superior performance across all budget ranges.
Figure 1: Qualitatively different AL strategies are suited for different budgets. SelectAL determines in advance which family of strategies should be used for the problem at hand. Left: when the labeled training set (aka budget) is large, uncertainty sampling, which focuses on unusual examples, provides the most added value. Right: when the budget is small, the learner benefits most from seeing characteristic examples.
Relation to prior work. Active learning has been an active area of research in recent years [32, 34]. The traditional approach to active learning, which is prevalent in deep learning and recent work, focuses on identifying data that will provide the greatest added value to the learner based on what it already knows. This is typically achieved through the use of uncertainty sampling [11, 24, 31, 36], diversity sampling [8, 20, 33, 35], or a combination of both [1, 10, 22].
However, in small-budget settings, where the learner has limited prior knowledge and effective training is not possible before the selection of queries, such active learning methods fail [4, 27]. Rather, [13, 30, 37] have shown that in this domain, a qualitatively different family of active learning methods should be used. These methods are designed to be effective in this domain and tend to seek examples that can be easily learned rather than confusing examples [5, 25, 39].
With the emergence of this distinction between two separate families of methods to approach active learning, the question arises as to which approach should be preferred in a given context, and whether it is possible to identify in advance which approach would be most effective. Previous research has explored methods for selecting between various AL strategies [2, 19, 29, 40]. However, these studies assumed knowledge of the budget, which is often unrealistic in practice. To our knowledge, SelectALis the first work that addresses this challenge.
We note that several recent works focused on the distinctions between learning with limited versus ample data, but within the context of non-active learning [6, 9, 12, 14, 15, 16, 21, 26]. These studies often highlight qualitative differences in model behavior under varying data conditions. While our results are consistent with these observations, our focus here is to suggest a practical approach to identify the budget regime in advance, allowing for the selection of the most appropriate AL strategy.
2 Theoretical analysis
The aim of this section is to derive a theoretical decision rule that can be translated into a practical algorithm, thus enabling practitioners to make data-driven decisions on how to allocate their budget in active learning scenarios. Given an AL scenario, this rule will select between a high-budget approach, a low-budget approach, or a blend of both. In Section 3, the theoretical result motivates the choice of a decision rule, in a practical method that selects between different active learning strategies.
To develop a test that can decide between high and low-budget approaches for active learning, we seek a theoretically sound framework in which both approaches can be distinctly defined, and show how they can be beneficial for different budget ranges. To this end, we adopt the theoretical framework introduced by [13]. While this framework is rather simplistic, it allows for precise formulation of the distinctions between the high and low-budget approaches. This makes it possible to derive a precise decision rule for the theoretical case, giving some insights for the practical case later on.
In Section 2.1, we establish the necessary notations and provide a brief summary of the theoretical framework used in the analysis, emphasizing the key assumptions and highlighting the results that are germane to our analysis. We then establish in Section 2.2 a derivative-based test, which is a novel contribution of our work. In Section 2.3, we utilize this test to derive an optimal active learning strategy for the theoretical framework. To conclude, in Section 2.4 we present an empirical validation of these results, accompanied by visualizations for further clarification.
Notations. We consider an active learning scenario where a learner, denoted by  $\mathcal{L}$ , is given access to a set of labeled examples  $\mathbb{L}$ , and a much larger pool of unlabeled examples  $\mathbb{U}$ . The learner's task is to select a fixed-size subset of the unlabeled examples, referred to as the active set and denoted by  $\mathbb{A} \subseteq \mathbb{U}$ , and obtain their labels from an oracle. These labeled examples are then used for further training of the learner. The goal is to optimally choose the active set  $\mathbb{A}$ , such that the overall performance of learner  $\mathcal{L}$  trained on the combined labeled dataset  $\mathbb{T} = \mathbb{A} \cup \mathbb{L}$  is maximal.
We refer to the total number of labeled examples as the budget, denoted by  $B = |\mathbb{T}| = |\mathbb{A} \cup \mathbb{L}|$ . This concept of budget imposes a critical constraint on the active learning process, as the learner must make strategic selections within the limitations of the budget.
Model definition and assumptions. We now summarize the abstract theoretical framework established by [13], which is used in our analysis. While this framework contains simplistic assumptions, its insights are shown empirically to be beneficial in real settings. This framework considers two independent general learners  $\mathcal{L}_{low}$  and  $\mathcal{L}_{high}$ , each trained on a different data distribution  $\mathcal{D}_{low}$  and  $\mathcal{D}_{high}$  respectively. Intuitively, one can think of  $\mathcal{D}_{low}$  as a distribution that is easier to learn than  $\mathcal{D}_{high}$ , such that  $\mathcal{L}_{low}$  requires fewer examples than  $\mathcal{L}_{high}$  to achieve a similar generalization error.
The number of training examples each learner sees may vary between the learners. To simplify the analysis, the framework assumes that data naturally arrives from a mixture distribution  $\mathcal{D}$ , in which an example is sampled from  $\mathcal{D}_{low}$  with probability  $p$  and from  $\mathcal{D}_{high}$  with probability  $1 - p$ .
We examine the mean generalization error of each learner denoted by  $E_{low}, E_{high} : \mathbb{R} \rightarrow [0,1]$  respectively, as a function of the number of training examples it sees. The framework makes several assumptions about the form of these error functions: (i) Universality: Both  $E_{low}$  and  $E_{high}$  take on the same universal form  $E(x)$ , up to some constant  $\alpha > 0$ , where  $E(x) = E_{low}(x) = E_{high}(\alpha x)$ . (ii) Efficiency:  $E(x)$  is continuous and strictly monotonically decreasing, namely, on average each learner benefits from additional examples. (iii) Realizability:  $\lim_{x \to \infty} E(x) = 0$ , namely, given enough examples, the learners can perfectly learn the data.
To capture the inherent choice in active learning, a family of general learners is considered, each defined by a linear combination of  $\mathcal{L}_{low}$  and  $\mathcal{L}_{high}$ . [13] showed that when fixing the number of examples available to the mixture model, i.e., fixing the budget  $B$ , it is preferable to sample the training data from a distribution that differs from  $\mathcal{D}$ . Specifically, it is preferable to skew the distribution towards  $\mathcal{D}_{low}$  when the budget is low and towards  $\mathcal{D}_{high}$  when the budget is high.
2.2 Derivative-based query selection decision rule
We begin by asking whether it is advantageous to skew the training distribution towards the low or high-budget distribution. Our formal analysis gives a positive answer. It further delivers a derivative-based test, which can be computed in closed form for the settings above. Importantly, our empirical results (Section 4) demonstrate the effectiveness of an approximation of this test in deep-learning scenarios, where the closed-form solution could not be obtained.
We begin by defining two pure query selection strategies - one that queries examples only from  $\mathcal{D}_{low}$ , suitable for the low-budget regime, and a second that queries examples only from  $\mathcal{D}_{high}$ , suitable for the high-budget regime. Given a fixed budget  $B$ , we analyze the family of strategies obtained by a linear combination of these pure strategies, parameterized by  $q \in [0,1]$ , in which  $qB$  points are sampled from  $\mathcal{D}_{low}$  and  $(1 - q)B$  points are sampled using  $\mathcal{D}_{high}$ .
The mean generalization error of combined learner  $\mathcal{L}$  on the original distribution  $\mathcal{D}$ , where  $\mathcal{L}$  is trained on  $B$  examples picked by a mixed strategy  $q$ , is denoted  $E_{\mathcal{L}}(B, q) = p \cdot E(qB) + (1 - p) \cdot E(\alpha(1 - q)B)$ , as the combined learner simply sends  $qB$  examples to  $\mathcal{L}_{low}$  and  $(1 - q)B$  examples to  $\mathcal{L}_{high}$ . By differentiating this result (see derivation in Suppl. A), we can find an optimal strategy for this family, and denote it by  $\hat{q}$ . This strategy is defined as the one that delivers the lowest generalization error when using a labeled set of size  $B$ . The strategy is characterized by  $\hat{q} \in [0, 1]$ , implicitly defined as follows:
$$
\frac {E ^ {\prime} (\hat {q} B)}{E ^ {\prime} (\alpha (1 - \hat {q}) B)} = \frac {\alpha (1 - p)}{p}. \tag {1}
$$
If Eq. (1) has a unique and minimal solution for  $\hat{q}$ , it defines an optimal mixed strategy  $\hat{q}_E(B, p, \alpha)$  for a given set of problem parameters  $\{B, p, \alpha\}$ . As  $p$  and  $\alpha$  are fixed for any specific problem, we get that different budgets  $B$  may require different optimal AL strategies. Notably, if budget  $B$  satisfies  $\hat{q}_E(B, p, \alpha) = p$ , then the optimal strategy for it is equivalent<sup>1</sup> to selecting samples directly from the original distribution  $\mathcal{D}$ . We refer to such budgets as  $B_{eq}$ . With these budgets, active learning is no longer helpful. While  $B_{eq}$  may not be unique, for the sake of simplicity, we assume that it is unique in our analysis, noting that the general case is similar but more cumbersome.
We propose to use Eq. (1) as a decision rule, to determine whether a low-budget or high-budget strategy is more suitable for the current budget  $B$ . Specifically, given budget  $B$ , we can compute  $\hat{q}_E(B,p,\alpha)$  and  $B_{eq}$  in advance and determine which AL strategy to use by comparing  $B$  and  $B_{eq}$ . In the current framework, this rule is guaranteed to deliver an optimal linear combination of strategies, as we demonstrate in Section 2.3. Visualization of the model's error as a function of the budget, for different values of  $q$ , can be seen in Fig. 2a.
(a) Accuracy gain of mixed strategies  $q$ , where mixture coefficient  $q$  is indicated in the legend.
(b) Optimal  $\hat{q}_a(B,p,\alpha)$
Figure 2: Visualization of  $E_{\mathcal{L}}(B,q)$ , where training set of size  $B$  is selected for different values of  $q$ . We adopt the example from [13], choosing  $E(x) = e^{-ax}$ ,  $a = 0.1$ ,  $p = \frac{1}{2}$  and  $\alpha = 0.05$ , as exponential functions approximate well the error functions of real networks. (a) A plot of accuracy gain when using strategy  $q$  (indicated in legend) as compared to random query selection:  $E(p) - E(q)$ , as a function of budget  $B$ . Since  $p = \frac{1}{2}$ , the plot corresponding to  $q = \frac{1}{2}$  is always 0. (b) Plots of  $\hat{q}$  as a function of  $B$ .  $B_{eq}$ , which corresponds to  $\hat{q} = \frac{1}{2}$ , is indicated by a vertical dashed line. Each plot corresponds to a different fraction  $\frac{|A|}{B}$  (see legend).
2.3 Best mixture of active learning strategies
With the aim of active learning (AL) in mind, our objective is to strategically select the active set  $\mathbb{A}$  in order to maximize the performance of learner  $\mathcal{L}$  when trained on all available labels  $\mathbb{T} = \mathbb{A} \cup \mathbb{L}$ . To simplify the analysis, we focus on the initial AL round, assuming that  $\mathbb{L}$  is sampled from the underlying data distribution  $\mathcal{D}$ . The analysis of subsequent rounds can be done in a similar manner.
To begin with, let us consider the entire available label set  $\mathbb{T} = \mathbb{A}\cup \mathbb{L}$ . Since  $\mathbb{T}$  comprises examples from  $\mathcal{D}$ ,  $\mathcal{D}_{low}$  and  $\mathcal{D}_{high}$ , we can represent it using the (non-unique) notation  $S(r_{rand},r_{low},r_{high})$ , where  $r_{rand},r_{low},r_{high}$  denote the fractions of  $\mathbb{T}$  sampled from each respective distribution. Based on the definitions of  $q$  and  $\hat{q}$ , we can identify an optimal combination for  $\mathbb{T}$ , denoted  $\mathbb{T}^*$ :
$$
\mathbb {T} ^ {*} = \left\{ \begin{array}{l l} \mathcal {S} (1 - \hat {r}, \hat {r}, 0) & \hat {q} > p \Rightarrow B <   B _ {e q} \\ \mathcal {S} (1, 0, 0) & \hat {q} = p \Rightarrow B = B _ {e q} \\ \mathcal {S} (1 - \hat {r}, 0, \hat {r}) & \hat {q} <   p \Rightarrow B > B _ {e q} \end{array} \right. \quad \hat {r} = \left\{ \begin{array}{l l} \frac {1}{1 - p} (\hat {q} - p) & \hat {q} > p \\ \frac {1}{1 - p} (p - \hat {q}) & \hat {q} <   p \end{array} \right. \tag {2}
$$
Attainable optimal mixed strategy. It is important to note that  $\mathbb{T} = \mathbb{A}\cup \mathbb{L}$ , where the active learning (AL) strategy can only influence the sampling distribution of  $\mathbb{A}$ . Consequently, not every combination of  $\mathbb{T}$  is attainable. In fact, the feasibility of the optimal combination relies on the size of the set  $\mathbb{A}$ . Specifically, utilizing (1), we can identify two thresholds,  $B_{low}$  and  $B_{high}$ , such that if  $B < B_{low}\leq B_{eq}$  or  $B > B_{high}\geq B_{eq}$ , it is not possible to achieve the optimal  $\mathbb{T}^{*} = \mathbb{A}^{*}\cup \mathbb{L}$ . The derivation of both thresholds can be found in Suppl. A.
Identity (in probability) is achieved when  $B\to \infty$
Figure 3: Visualization of strategy (3) using the example from Fig. 2. We plot the gain in error reduction as compared to random query selection. We show results with active sets that are  $30\%$  and  $1\%$  of  $B$ . The smaller the active set is, the closer the performance is to the pure optimal strategy.
(b) CIFAR-100
Figure 4: Empirical validation of the theoretical results. We plot the accuracy gains of 20 ResNet-18 networks trained using strategy (3), compared to using no active learning at all. We used TypiClust as  $S_{low}$  and BADGE as  $S_{high}$ . We see that as predicted by the theoretical analysis, the best mixture coefficient  $r$  increases as the difference  $|B - B_{eq}|$  increases.
With (2) and the thresholds above, we may conclude that the optimal combination of  $\mathbb{A}^*$  is:
$$
\mathbb {A} ^ {*} = \left\{ \begin{array}{l l} S (0, 1, 0) & B <   B _ {l o w} \\ S (1 - \hat {r} \frac {| \mathbb {T} |}{| \mathbb {A} |}, \hat {r} \frac {| \mathbb {T} |}{| \mathbb {A} |}, 0) & B _ {l o w} \leq B <   B _ {e q} \\ S (1, 0, 0) & B = B _ {e q} \\ S (1 - \hat {r} \frac {| \mathbb {T} |}{| \mathbb {A} |}, 0, \hat {r} \frac {| \mathbb {T} |}{| \mathbb {A} |}) & B _ {h i g h} \geq B > B _ {e q} \\ S (0, 0, 1) & B > B _ {h i g h} \end{array} \right.
$$
$$
\begin{array}{l}\text {(3)} \quad \xrightarrow {| \mathbb {A} | \rightarrow 0} \mathbb {A} ^ {*} = \left\{\begin{array}{l l}S (0, 1, 0)&B <   B _ {l o w}\\S (1, 0, 0)&B = B _ {e q}\\S (0, 0, 1)&B > B _ {h i g h}\end{array}\right.\end{array}\tag {4}
$$
In other words, we observe that the optimal strategy (2) is only attainable in non-extreme scenarios. Specifically, in cases of very low budgets ( $B < B_{low}$ ), it is optimal to sample the active set purely from  $\mathcal{D}_{low}$  because more than  $|\mathbb{A}|$  points from  $\mathcal{D}_{low}$  are needed to achieve optimal performance. Similarly, in situations of very high budgets ( $B > B_{high}$ ), it is optimal to select the active set solely from  $\mathcal{D}_{high}$  since more than  $|\mathbb{A}|$  points from  $\mathcal{D}_{high}$  are necessary for optimal performance. We note that even if  $B_{eq}$  is not unique, in the limit of  $|\mathbb{A}| \to 0$ , (3)-(4) are still locally optimal, segment by segment.
Small active set  $\mathbb{A}$ . Upon examining the definitions of  $B_{low}$  and  $B_{high}$  (refer to Suppl. A), we observe that  $\lim_{|\mathbb{A}| \to 0} B_{low} = \lim_{|\mathbb{A}| \to 0} B_{high} = B_{eq}$ . Consequently, as indicated in (4), the optimal mixed strategy, in this scenario, actually becomes a pure strategy. When the budget is low, the entire active set  $\mathbb{A}$  should be sampled from  $\mathcal{D}_{low}$ . Similarly, when the budget is high,  $\mathbb{A}$  should be sampled solely from  $\mathcal{D}_{high}$ . In the single point where  $B_{low} = B_{high} = B_{eq}$ ,  $\mathbb{A}$  should be sampled from  $\mathcal{D}$ . Fig. 2b provides a visualization of these strategies as the size of  $\mathbb{A}$  decreases.
Iterative querying. When  $|\mathbb{A}|$  is not small enough to justify the use of the limiting pure strategy (4), we propose to implement query selection incrementally. By repeatedly applying (4) to smaller segments of length  $m$ , we can iteratively construct  $\mathbb{A}$ , with each iteration becoming more computationally feasible. This strategy not only enhances robustness but also yields comparable performance to the mixed optimal strategy (3), as demonstrated in Section 2.4.
Based on the analysis presented above, it becomes apparent that in practical scenarios, the optimal combination of active learning strategies can be achieved by sequentially sampling from pure strategies, while utilizing a derivative-based test to determine the most effective strategy at each step. This concept forms the core motivation behind the development of the practical algorithm SelectAL, which is presented in Section 3.
2.4 Validation and visualization of theoretical results
Visualization. In Fig. 3, we illustrate the error of the strategies defined in (3) using the same exponential example as depicted in Fig. 2. The orange curve represents a relatively large active set, where  $|\mathbb{A}|$  is equal to  $30\%$  of the budget  $B$ , while the blue curve represents a significantly smaller active set, where  $|\mathbb{A}|$  is equal to  $1\%$  of the budget  $B$ . It is evident that as the size of  $\mathbb{A}$  decreases, the discrepancy between the optimal strategy and the attainable strategy diminishes.
Fig. 2b shows the optimal mixture coefficient  $\hat{q}$  as a function of the budget size  $B$  for various values of  $|\mathbb{A}|$ , namely  $1\%$ ,  $30\%$ , and  $100\%$  of  $B$ . According to our analysis, as the size of  $|\mathbb{A}|$  decreases, the optimal  $\hat{q}$  should exhibit a more pronounced step-like behavior. This observation suggests that in the majority of cases, it is possible to sample the entire active set  $\mathbb{A}$  from a single strategy rather than using a mixture of strategies.
Validation. Our theoretical analysis uses a mixture model of idealized general learners. We now validate that similar phenomena occur in practice when training deep networks on different computer vision tasks. Since sampling from  $\mathcal{D}_{low}$  and  $\mathcal{D}_{high}$  is not feasible in this case, we instead choose low and high-budget deep AL strategies from the literature. Specifically, we choose TypiClust as the low-budget strategy and BADGE as the high-budget strategy, as explained in detail in Section 4.1. We note that other choices for the low and high-budget strategies yield similar qualitative results, as is evident from Tables 1-2. Fig. 4 shows results when training 20 ResNet-18 networks using mixed strategies  $S(1 - r\frac{|\mathbb{T}|}{|\mathbb{A}|},0,r\frac{|\mathbb{T}|}{|\mathbb{A}|})$  and  $S(1 - r\frac{|\mathbb{T}|}{|\mathbb{A}|},r\frac{|\mathbb{T}|}{|\mathbb{A}|},0)$  on CIFAR-10 and CIFAR-100. We compare the mean performance of each strategy to the performance of 20 ResNet-18 networks trained with the random query selection strategy  $S(1,0,0)$ .
Inspecting these results, we observe similar trends to those shown in the theoretical analysis. As the budget increases, the most beneficial value of mixture coefficient  $r$  decreases until a certain transition point (corresponding to  $B_{eq}$ ). From this transition point onward, the bigger the budget is, the more beneficial it is to select additional examples from one of the high-budget strategies. As in the theoretical analysis, when the budget is low it is beneficial to use a pure low-budget strategy, and when the budget is high it is beneficial to use a pure high-budget strategy. The transition area, corresponding to segment  $B_{low} \leq B \leq B_{high}$ , is typically rather short (see Figs. 4 and 6).
2.5 Discussion: motivation from the theoretical analysis