AutoAL: Automated Active Learning with Differentiable Query Strategy Search

## Abstract

Abstract As deep learning continues to evolve, the need for data efficiency becomes increasingly important. Considering labeling large datasets is both time-consuming and expensive, active learning (AL) provides a promising solution to this challenge by iteratively selecting the most informative subsets of examples to train deep neural networks, thereby reducing the labeling cost. However, the effectiveness of different AL algorithms can vary significantly across data scenarios, and determining which AL algorithm best fits a given task remains a challenging problem.
This work presents the first differentiable AL strategy search method, named AutoAL , which is designed on top of existing AL sampling strategies.
AutoAL consists of two neural nets, named SearchNet and FitNet, which are optimized concurrently under a differentiable bi-level optimization framework.
For any given task, SearchNet and FitNet are iteratively co-optimized using the labeled data, learning how well a set of candidate AL algorithms perform on that task.
With the optimal AL strategies identified, SearchNet selects a small subset from the unlabeled pool for querying their annotations, enabling efficient training of the task model.
Experimental results demonstrate that AutoAL consistently achieves superior accuracy compared to all candidate AL algorithms and other selective AL approaches, showcasing its potential for adapting and integrating multiple existing AL methods across diverse tasks and domains.
Code will be available at: https://github.com/haizailache999/AutoAL.

## 1 Introduction

With the development of deep learning (DL) techniques, large volumes of high-quality labeled data have become increasingly crucial as the model training processes are usually data hungry  Ren et al. (); Zhan et al. ().
Active learning (AL) addresses this challenge by iteratively selecting the most informative unlabeled samples for annotation, thereby boosting model efficiency Xie et al. (); Wang et al. ().
Deep active learning strategies are typically divided into two categories: uncertainty-based and representativeness/diversity-based approaches Sener & Savarese (); Zhu & Bento (); Ash et al. (). Uncertainty-based methods focus on querying samples the model is most uncertain about, while diversity-based strategies aim to select a diverse subset of samples that effectively represent the entire dataset Citovsky et al. (). Both strategies have limitations, especially in batch selection and varying dataset complexities, leading to the inconsistency of the strategy efficiency Ren et al. (). Hybrid strategies, which balance uncertainty and diversity Zhan et al. (; ), offer partial solutions. However, these strategies rely on strategy choices and subjective experiment settings, which can be ineffective in real-life applications.

Previous works Baram et al. (); Hsu & Lin (); Pang et al. () have explored selecting among different AL strategies to achieve optimal selections without human effort.
SelectAL Hacohen & Weinshall () introduces a method for dynamically choosing AL strategies for different deep learning tasks and budgets by estimating the relative budget size of the problem. It involves evaluating the impact of removing small subsets of data points from the unlabeled pool to predict how different AL strategies would perform. However, this strategy depends on approximating the reduction in generalization error on small subsets, which would not fully capture the complexities of real-world tasks.
Zhang et al. () proposes to select the optimal batch of AL strategy from hundreds of candidates, aiming to maximize future cumulative rewards based on noisy past reward observations of each candidate algorithm.
However, both the computational cost and the lack of differentiability make the optimization of these works inefficient.

To address the challenges mentioned above, we propose AutoAL, an automated active learning framework that leverages differentiable query strategy search.
AutoAL offers two key advantages: 1) it enables automatic and efficient updates during optimization, allowing the model to adapt seamlessly; 2) it is inherently data-driven, leveraging the underlying data distribution to guide the AL strategy selection.
However, integrating multiple AL strategies into the candidate pool makes achieving differentiability difficult.
To overcome this issue, AutoAL leverages a bi-level optimization framework that smoothly integrates and optimizes different AL strategies. We design two neural networks: SearchNet and FitNet.
FitNet is trained on the labeled dataset to adaptively yield the informativeness of each unlabeled sample.
Guided by FitNet’s task loss, SearchNet is efficiently optimized to accurately select the best AL strategy.
Instead of searching over a discrete set of candidate AL strategies, we relax the search space to a continuous domain, allowing SearchNet to be optimized via gradient descent based on FitNet’s output.
The gradient-based optimization, which is more data-efficient than traditional black-box searches, enables AutoAL to achieve state-of-the-art performance in strategy selection with significantly reduced computational costs. Our key contributions are as follows:

- •
We introduce a novel AutoAL framework for differentiable AL query strategy search.
To the best of our knowledge, AutoAL is the first automatic query strategy search algorithm that can be trained in a differientiable way, achieving highly efficient update of query strategy in a data-driven manner.
- •
AutoAL is built upon existing AL strategies (i.e., uncertainty-based and diversity-based approaches). By incorporating most AL strategies into its search space, AutoAL leverages the advanages of these strategies through finding the most effective ones for specific AL settings, objectives, or data distributions.
- •
We evaluate AutoAL across various AL tasks. Given the importance of AL in domain-specific tasks, particularly where data quality and imbalance are concerns,
we conducte extensive experiments on both nature image datasets and medical datasets.
The results demonstrate that AutoAL consistently outperforms all candidate strategies and other state-of-the-art AL methods across these datasets.

## 2 Related Work

#### Active Learning.

Active Learning has been studied for decades to improve the model performance using small sets of labeled data, significantly reducing the annotation cost Cohn et al. (); Settles (); Zhan et al. ().
Most AL research focuses on pool-based AL, which identifies and selects the most informative samples from a large unlabeled pool iteratively.
Pool-based AL sampling strategies are generally categorized into two primary branches: diversity-based and uncertainty-based. The diversity-based methods select the batch of samples that can best represent the entire dataset distribution.
CoreSet Sener & Savarese () selects a batch of representative points from a core set, which is a subsample of the dataset that effectively serves as a proxy for the entire set.
In contrast, uncertainty-based methods focus on selecting the samples with high uncertainty, which is due to the data generation or the modeling/learning process.
Bayesian Active Learning by Disagreements (BALD) Gal et al. () chooses data points that are expected to maximize the mutual information between predictions and model posterior.
Yoo & Kweon () incorporates a loss prediction strategy by appending a compact parametric module designed to forecast the loss of unlabeled inputs relative to the target model. This is achieved by minimizing the discrepancy between the predicted loss and the actual target loss.
It requires subjective judgment and strategic selection of active learning methods to be effective on specific datasets in real-world settings.

#### Adaptive Sample Selection in AL.

Neither diversity-based nor uncertainty-based methods are perfect in all data scenarios. Diversity-based methods tend to perform better when the dataset contains rich category content and large batch size, while uncertainty-aware methods typically perform better in opposite settings Zhan et al. (); Citovsky et al. ().
Several past works have explored adaptive selection in active learning to solve this problem.
A common approach is to choose the best AL strategy from a set of candidate methods and use it to query unlabeled data, such as Hacohen & Weinshall () and Zhang et al. () introduced in Section 1.
Active Learning By Learning (ALBL) Hsu & Lin () adaptively selects from a set of existing algorithms based on their estimated contribution to learning performance. It frames each candidate as a bandit machine, converting the problem into a multi-armed bandit scenario.
However, none of these methods leverage the differentiable framework for automatic AL selection.
Compared to the methods mentioned above, AutoAL first relaxes the search space to be continuous, thus can optimize automatically via gradient descent.
This framework enables AutoAL to seamlessly integrate multiple existing AL strategies and rapidly adapt to select the optimal strategy based on the labeled pool.
The simple gradient-based optimization is much more data-efficient than traditional black-box searches. Therefore, AutoAL can significantly reduce computational costs compared to these methods.

#### Bilevel Optimization.

Bilevel optimization is a hierarchical mathematical framework where the feasible region of one optimization task is constrained by the solution set of another optimization task Liu et al. ().
It has gained significant attention due to its nested problem structure, which allows the two tasks to optimize jointly. Bilevel optimization adapts to many DL applications Chen et al. (), such as hyperparameter optimization Chen et al. (); MacKay et al. (), meta-knowledge extraction Finn et al. (), neural architecture search Liu et al. (); Hu et al. (), and active learning Sener & Savarese (). For instance, Sener & Savarese () formulates AL as a bi-level coreset selection problem and designs a 2-approximation method to select a subset of data that represents the entire dataset. In this work, we novelly formulate the Al query strategy search as a bi-level optimization problem, i.e., iteratively updating FitNet and using its task loss to guide the optimization of SearchNet. We further reformulate it as a differentiable learning paradigm to enable a more efficient selection of examples across various objectives and data distributions.

## 3 Methodology

### 3.1 Problem Setting

Pool-based AL focuses on selecting the most informative data iteratively from a large pool of unlabeled independent and identically distributed (i.i.d.) data samples until a fixed budget is exhausted or the expected model performance has been reached.
Specifically, in AL processes, assuming that we have an initial seed labeled set $L=\{(x_{j},y_{j})\}_{j=1}^{M}$ and a large unlabeled data pool $U=\{x_{i},\}_{i=1}^{N}$, where $M\ll N$, $y_{i}\in\{0,1\}$ is the class label of $x_{i}$ for binary classification and $y_{i}\in\{1,...,p\}$ for multi-class classification.
In each iteration, we select a batch of the most informative data $Q^{*}=\arg\max^{b}_{x\in U}\alpha(x,\Omega)$ with batch size $b$ from $U$ based on the basic learned model $\Omega$ and the acquisition function $\alpha(x,\Omega)$, and query their labels from oracle/annotator.
With these samples, the expected loss function $f$ of the basic learned model
can be minimized. $L$ and $U$ are then updated.

The performance of existing AL methods relies on the choice of query strategies, which should be carefully adapted to different tasks or applications.
To achieve a robust and adaptive AL performance, this work novelly creates an automated AL strategy selection framework.
AutoAL integrates two neural nets, FitNet $\Omega_{F}$ and SearchNet $\Omega_{S}$ upon each basic learned model $\Omega$ to select the best AL strategy before selecting $Q$. To facilitate automatic selection, AutoAL incorporates them into a bi-level optimization framework and relaxes the search space to enable differentiable updates. We discuss more details of AutoAL framework in the following sections.

Figure: Figure 1: Overall framework of differentiable query strategy search for automated active learning (AutoAL). AutoAL leverages labeled pool to train the FitNet and SearchNet in a bi-level optimization mode. Data samples with the largest search score are then selected from the unlabeled pool.
Refer to caption: x1.png

### 3.2 Automated Active Learning

We propose Automated Active Learning (AutoAL), which aims to adaptively deliver the optimal query strategy for each sample in a given unlabeled data pool. Specifically, AutoAL consists of two neural networks: FitNet $\Omega_{F}$ and SearchNet $\Omega_{S}$.
$\Omega_{S}$ selects the optimal AL strategy from a set $\mathcal{A}$, which contains $K$ candidate AL sampling strategies $\{\mathcal{A}_{\kappa}\}_{\kappa\in[K]}$ (e.g., BALD, Maximum Entropy, etc).
$\Omega_{F}$ models the data distribution within the unlabeled dataset and guides the training of SearchNet $\Omega_{S}$. Since annotations for the unlabeled data cannot be accessed directly by the model $\Omega$, AutoAL requires no data from the unlabeled pool $U$, using only the labeled dataset $L$ for training.

In AutoAL, each AL iteration consists of $\mathcal{C}$ cycles. In each cycle $c$, the labeled data $L$ is randomly split into two equal-sized subsets: training and validation sets. FitNet $\Omega_{F}$ computes the cross-entropy loss $\mathcal{L}$  Zhang & Sabuncu () for data. Meanwhile, SearchNet $\Omega_{S}$ generates scores for the samples, which are used to rank them. This raises a key question: If only the training set is available, without access to the unlabeled pool, can $\Omega_{S}$ still effectively select the optimal samples?

Our key design is intuitive: SearchNet $\Omega_{S}$ treats the training batch as the labeled pool and the validation batch as the unlabeled pool. This allows $\Omega_{S}$ to simulate the process of AL selection, without direct access to the actual unlabeled data pool.
The optimization goal of $\Omega_{S}$ is to select data with higher losses, as they are expected to provide more informative updates to the neural network.
For $\Omega_{F}$, the training objective is to minimize the loss of the selected samples, particularly those *prioritized* by $\Omega_{S}$.
AutoAL is formulated as a bi-level optimization problem, as the optimization of $\Omega_{F}$ is dependent only on itself, but the optimization of $\Omega_{S}$ depends on the optimal $\Omega_{F}$.

$$ $\Omega_{S}^{*}=\arg\max_{\Omega_{S}}\sum\nolimits_{j=1}^{M/2}\mathcal{L}_{S}(( x_{j},y_{j}),\Omega_{S},\Omega_{F}^{*}),$ (1) $$

$$ $\text{s.t.}\quad\Omega_{F}^{*}=\arg\min_{\Omega_{F}}\sum\nolimits_{j=1}^{M/2} \mathcal{L}_{F}((x_{j},y_{j}),\Omega_{F}).$ $$

$\mathcal{L}_{F}$ and $\mathcal{L}_{S}$ represent the losses of $\Omega_{F}$ and $\Omega_{S}$, respectively.
The neural nets $\Omega_{F}$ and $\Omega_{S}$ are optimized jointly as outlined in Eq. 1.
At the lower level of the nested optimization framework, $\Omega_{F}$ is optimized to approach the distribution of the dataset. At the upper level, $\Omega_{S}$ is optimized to output the informativeness of each data, based on the optimal distribution returned from $\Omega_{F}$.

### 3.3 Differentiable Query Strategy Optimization

Although Eq. 1 shows an effective framework for automatically deriving the optimal query strategy, solving the bi-level optimization problem is inefficient in practice. To address this, we derive a probabilistic query strategy coupled with a differentiable optimization framework, improving the efficiency of the optimization process.

#### Probabilistic query strategy.

During training, for each labeled data $x_{j}$,we query a score $\mathcal{S}{\kappa}(x_{j})$ from each candidate AL sampling strategy $\mathcal{A}i$, where $\mathcal{S}{\kappa}(x_{j})\in{0,1}$ indicates whether the sample is selected.
To model the overall score $\mathcal{S}(x_{j})$, which is a combination of all the AL search scores $\mathcal{S}{\kappa}(x_{j})$, we adopt a Gaussian Mixture approach Reynolds et al. ():

$$ $p(\mathcal{S})=\sum\nolimits_{k=1}^{k}\pi_{k}\,\mathcal{N}(\mathcal{S}\,|\,\mu _{k},\Sigma_{k}),$ (2) $$

$$ $\{\hat{\pi}_{k},\hat{\mu}_{k},\hat{\Sigma}_{k}\}=\text{arg max}_{\pi_{k},\mu_{ k},\Sigma_{k}}\,\prod_{j=1}^{M}p(\mathcal{S}(x_{j})),$ (3) $$

where $\pi_{k}$ is the weight of the $k$-th gaussian component and $\mathcal{N}$ is the probability density function. Then we generate samples according to the Gaussian Mixture model and get the $t$-th maximum value, where $t$ is the ratio of batch size $b$ to the total pool size $M+N$. $W_{\kappa,j}$ is the probability of each query strategy for each sample predicted from the neural network $\Omega_{S}$.

$$ $S_{\text{sample}}\sim\sum\nolimits_{k=1}^{K}\hat{\pi}_{k}\,\mathcal{N}( \mathcal{S}\,|\,\hat{\mu}_{k},\hat{\Sigma}_{k}),$ (4) $$

$$ $\hat{\mathcal{S}}_{\kappa}(x_{j},\Omega_{S})=(\mathcal{S}_{\kappa}(x_{j})- \vartheta_{t}(S_{\text{sample}}))*W_{\kappa,j}~{}.$ (5) $$

A sample with a higher score $\hat{\mathcal{S}}$ indicates that it has been selected by more candidate AL strategies, and therefore should be given higher priority for labeling.

#### Differentiable acquisition function optimization.

If we limit the individual query score $\mathcal{S}_{\kappa}(x_{j})$ to discrete values $\{0,1\}$, the bi-level optimization objective of Eq. 1 becomes non-differentiable and still difficult to optimize.
To enable differentiable optimization, we relax the categorical selection of strategies into a continuous space, we apply a *Sigmoid* function over all possible strategies.:

$$ $\bar{\mathcal{S}}(x_{j})=\sum_{\kappa\in K}\frac{\lambda}{1+\exp(-\Theta^{(j)} _{\mathcal{S}_{\kappa}^{\prime}})}\hat{\mathcal{S}}_{\kappa}(x_{j},\Omega_{S}),$ (6) $$

where the strategy mixing weights for each sample $j$ is parameterized by a vector $\Theta^{(j)}$ of dimension |$\mathcal{A}$|.
$\lambda$ is a scaling vector.
$\bar{\mathcal{S}}(x_{j})$ is the final differentiable learning objective function, which can be optimized with back-propagation.
Therefore, $\Omega_{F}$ and $\Omega_{S}$ can be efficiently optimized under Eq. 1.

### 3.4 Learning and Algorithm of AutoAL

With differentiable query strategy optimization in Sec. 3.3, we reformulate the bi-level optimization problem in Sec. 3.2 as efficient optimization objectives as follows:

$$ $\mathcal{L}_{F}=\frac{1}{B}\sum_{j^{\prime}=1+(n+1)B}^{(n+2)B}\bar{\mathcal{S} }_{detach}(x^{\prime}_{j})\cdot\mathcal{L}(x^{\prime}_{j},y^{\prime}_{j})+\bar {\lambda}\mathcal{L}_{re}(t,B),$ (7) $$

$$ $\mathcal{L}_{S}=-\frac{1}{B}\sum_{j=1+nB}^{(n+1)B}\bar{\mathcal{S}}(x_{j}) \cdot\mathcal{L}_{detach}(x_{j},y_{j})-\bar{\lambda}\mathcal{L}_{re}(t,B),$ (8) $$

As described in section 3.2, in cycle $c$-2, $\Omega_{F}$ is optimized to get the data distribution. The final loss is calculated according to Eq. 7.
In cycle $c$-1, AutoAL optimizes $\Omega_{S}$ to get the data sample with highest loss. The loss function from $\Omega_{F}$ is detached to avoid updating itself, and the final loss is calculated according to Eq. 8. Here we use negative loss to achieve gradient ascent.
For $n\in\left\{0,1,\ldots,\left\lfloor\frac{M}{2B}\right\rfloor\right\}$, AutoAL additionally integrates a loss prediction module according to Yoo & Kweon () to help update $\Omega_{S}$, as

$$ $\mathcal{L}_{re}(t,B)=\left(\frac{1}{1+\exp\left(0.5\cdot\left|\alpha-t\cdot B \right|\right)}-0.5\right).$ (9) $$

$\mathcal{L}_{re}$ represents a regularization loss that limits the number of selected samples, as shown in Eq. 9. Here, $\alpha$ is the number of selected samples, $t$ is the ratio of the query batch size $b$ in each AL iteration to the total pool size $M+N$, and $\bar{\lambda}$ is a scaling vector.
The final learning objective is to select the most informative samples while effectively leveraging the underlying data distribution.

The AutoAL algorithm is illustrated in Algorithm 1.
In each AL iteration, the labeled set is first used to train the FitNet and SearchNet as described in section 3.2.
Then the optimal SearchNet $\Omega^{*}_{S}$ is applied to the unlabeled pool to select a new batch of data, where $\Omega_{S}^{*}$ is used to generate score $\bar{\mathcal{S}}$ for each sample $x_{i}$, and $Q^{*}=\arg\max^{b}_{x\in U}\bar{\mathcal{S}}(x_{i})$ with top-$b$ highest scoring samples are selected and labeled. The labeled set $L$ and the unlabeled set $U$ are updated as $L=L+Q^{*}$ and $U=U-Q^{*}$, respectively. The updated labeled set is then used for task model training.

Figure: Algorithm 1 AutoAL: Automated Active Learning with Differentiable Query Strategy Search

## 4 Experiments

We empirically evaluate the performance of AutoAL on a wide range of datasets with both nature and medical images.
These experiments demonstrate that AutoAL outperforms the baselines by fully automating the training within the differentiable bi-level optimization framework. Additionally, we conduct ablation studies to analyze the contribution of each module and examine the effects of the candidate AL strategies, including different numbers of candidates.

### 4.1 Experiment Settings

**Table 1: Overview of dataset information.**
|  | Dataset | # of training set | # of test set | # of classes |
| --- | --- | --- | --- | --- |
| Nature image | Cifar-10 | $50,000$ | $10,000$ | $10$ |
| Cifar-100 | $50,000$ | $10,000$ | $100$ |  |
| SVHN | $73,257$ | $26,032$ | $10$ |  |
| Medical image | OrganCMNIST | $12,975$ | $8,216$ | $11$ |
| PathMNIST | $89,996$ | $7,180$ | $9$ |  |
| TissueMNIST | $165,466$ | $47,280$ | $8$ |  |

#### Datasets and Baselines.

We conduct AL experiments on six datasets: Cifar-10 and Cifar-100 Krizhevsky et al. (), SVHN Netzer et al. () in the nature image domain, and OrganCMNIST, PathMNIST, and TissueMNIST from MedMNIST Yang et al. () in the medical image domain.
We compare our method against multiple AL sampling strategies, including Maximum Entropy Sampling Shannon (), Margin Sampling Netzer et al. (), Least Confidence Wang & Shang (), KMeans Sampling Ahmed et al. (),
Bayesian Active Learning by Disagreements (BALD) Gal et al. (), Variation Ratios (VarRatio) Freeman () and Mean Standard Deviation (MeanSTD) Kampffmeyer et al. ().
We also consider the state-of-the-art deep AL methods as baselines, including Batch Active learning by Diverse Gradient Embeddings (BADGE) Ash et al. () and Loss Prediction Active Learning (LPL) Yoo & Kweon ().
We also include ALBL Hsu & Lin () to compare AutoAL with existing selective AL strategies.

#### Implementation Details.

For $\Omega_{F}$ and $\Omega_{S}$, we build the backbone using ResNet-18 He et al. ().
We also employ ResNet-18 as the classification model on all baselines and AutoAL for fair comparison.
Since AutoAL is built upon existing AL strategies and focuses on selecting the optimal strategy, we integrate seven AL methods into AutoAL: Maximum Entropy, Margin Sampling, Least Confidence, KMeans, BALD, VarRatio, and MeanSTD.
For the optimization of $\Omega_{S}$ and loss prediction module, we use SGD optimizer Ruder ().
For the optimization of $\Omega_{F}$, we use Adam optimizer Kingma (). Both use $0.005$ as the learning rate.
While training, FitNet will first update for $200$ epochs using the validation queue, then $\Omega_{F}$, $\Omega_{S}$ and the loss prediction module will update iteratively with a total of $400$ epochs.
All experiments are repeated *three times* with different randomly selected initial labeled pools, reporting mean and standard deviation.

Figure: Figure 2: Overall performance on six benchmark datasets: natural image datasets (top) and medical image datasets (bottom).
Refer to caption: x2.png

### 4.2 Evaluation of Active Learning Performance

Fig. 2 shows the overall performance comparison of different AL methods, where AutoAL consistently outperforms the baselines across all datasets. Additionally, we have the following observations:

- 1.
Our method consistently outperforms the other approaches in terms of rounds, a crucial attribute for a successful active learning method. This adaptability is particularly valuable in real-world applications, where the labeling budget may vary significantly across different tasks. For instance, one might only have the resources to annotate 10% of the data, rather than 20% or 40%.
- 2.
Our method demonstrates robust performance not only on easier datasets but also on more challenging ones, such as Cifar-100 and OrganCMNIST. Cifar-100 has significantly more classes than other datasets, while OrganCMNIST has smaller data pool, with only $12,975$ images. These challenges make active learning more difficult, yet our method’s superior performance across these datasets highlights its robustness.
- 3.
The labeled data vs. accuracy curves of our method are relatively smooth compared to those of other methods. The baseline methods sometimes show significant accuracy drops during certain active learning rounds.
This mainly dues to harmful data selection Koh & Liang () or overfitting problem.
However, by selecting the optimal strategy thus selecting the informative data, AutoAL can alleviate the problem and make the curve much more smooth than baseline strategies.
- 4.
Our method shows smaller standard deviations, indicating that it is robust across repeated experiments. This robustness is crucial for AL applications, especially in a real-world setting, where the AL process is typically conducted only once.
- 5.
Different AL strategies produce varying results across datasets. For instance, Margin Sampling underperforms on the SVHN dataset, while KMeans Sampling yields the worst performance on the OrganCMNIST and Cifar-10 datasets. This variability highlights the significance of our approach, which aims to identify and select the optimal strategy for each dataset.

### 4.3 Ablation Studies

Figure: Figure 3: Ablation study on three components of AutoAL. ‘ResNet Backbone’: without the loss prediction module, updating only the ResNet. ‘Loss Prediction Module’: without updating the ResNet backbone, optimizing solely with the loss prediction module. ‘ResNet+Loss Prediction’: the full AutoAL pipeline.
Refer to caption: x3.png

Figure: Figure 4: Ablation study on the size of the candidate pool in AutoAL strategy selection.
Refer to caption: x4.png

We perform the ablation study on two key components of AutoAL: the SearchNet architecture and the AL candidate strategies.
Experiments are conducted on three datasets: Cifar-100, SVHN, and OrganCMNIST, which are chosen for their mix of natural and medical images, as well as the increased difficulty they present for active learning applications.

#### SearchNet Architecture.

Our SearchNet consists of two main components: a ResNet-18 backbone and a loss prediction module for sample loss prediction  Yoo & Kweon (). In this ablation study, we disable one component at a time and record the results.
Fig. 3 illustrates the performance, revealing the effectiveness of both ResNet-18 backbone and the loss prediction module. However, when only the loss prediction module is used to optimize the network, performance drops significantly.
This aligns with our intuition, as although the loss prediction module can assist with network optimization, the main focus shifts from selecting the best AL strategy to merely minimizing the sample loss. This misalignment will result in incorrect AL strategy weight estimation (See Fig. 1).

#### Size of Active Learning Candidate Pool.

Since AutoAL is built on multiple AL strategies, we study how the size of the AL candidate pool impacts performance. We start from the Maximum Entropy as the sole candidate for the initial baseline, then expand the candidate pool to three strategies by adding Margin Sampling and Least Confidence. For the pool of five candidates, we further include KMeans and BALD.
As shown in Fig. 4, for all three datasets, the single candidate variation performs the worst but still outperforms the Entropy Sampling baseline in Fig. 2. This is because AutoAL does not directly query data samples with maximum entropy but queries the data with the highest score from SearchNet, which has been optimized with the help of the loss prediction module and FitNet.
For Cifar-100 and OrganCMNIST, AutoAL can perform well when there are at least three candidates in the selection pool. However, for SVHN, more AL candidates result in better performance. This indicates that the upper bound of candidate pool size varies across different datasets. In certain datasets, incorporating more active learning strategies yields better results, but this is not the case for other datasets.
Additionally, pools with more candidates exhibit lower standard deviations. This demonstrates strong capability of AutoAL in selecting the best active learning strategy. It can consistently choose the optimal one and exclude the less effective options.

### 4.4 AL Selection Strategy

Here, we examine which AL strategy is prioritized by AutoAL in different active learning rounds. For each AL strategy candidate, we calculate the AL score of each image according to Eq. 6. Notice that we don’t sum all the scores according to the dimension of each active learning, but compute the average value of all images over a round, and normalize the results for visualization. As illustrated in Fig. 5,
in both experiments, KMeans dominate the sample selection during the initial rounds, such as the first round, but be de-prioritized in subsequent rounds. This indicates that diversity-based measures, which select samples that represent diverse regions of the data distribution, are more critical in the early stages of active learning. In the early rounds, when only a small percentage of the data has been labeled, the model’s understanding of the entire data distribution is limited. As a result, selecting samples that cover a broad spectrum of the dataset becomes crucial to provide the model with a more comprehensive view of the data.

On Cifar-100 dataset, both Least Confidence and MeanSTD, two uncertainty-based measures, demonstrate improved performance in the final rounds. AutoAL assigns them higher priority to enhance overall performance. As the AL process progresses and more diverse data become available, the model attempts to develop a better understanding of the underlying structure of the target data. Uncertainty-based measures are more effective now because the major challenge switches to refining the decision boundaries. The transition from diversity to uncertainty-based strategies is consistent with the model’s evolving needs. In early rounds, the model requires diversity to build a broad understanding of the data, but once that foundation is established, its focus shifts toward uncertainty, which helps fine-tune model decision-making ability. This adaptive approach ensures that the model targets the most diverse subsets at each stage of the learning process, from broad exploration in the early rounds to fine-grained exploitation in the later rounds. By automatically adjusting the priorities of these strategies, AutoAL improves the overall accuracy and shows stronger robustness.

Figure: Figure 5: AL strategy scores across different AL rounds on CIFAR-100 and OrganCMNIST datasets.
Refer to caption: x5.png

## 5 Conclusion

In this paper, we have introduced AutoAL, the first automated search framework for deep active learning.
AutoAL employs two neural networks, SearchNet and FitNet, integrated into a bilevel optimization framework to enable joint optimizations.
To efficiently solve the bi-level optimization problem, we have proposed a probabilistic query strategy that relaxes the search space from discrete to continuous, enabling a differentiable and data-driven learning paradigm for AutoAL.
This framework not only accelerates the search process but also ensures that AutoAL can generalize across a wide range of tasks and datasets.
Furthermore, AutoAL is flexible, allowing for easy integration of most of the existing active learning strategies into the candidate pool, making it adaptable to evolving active learning techniques.
Our extensive empirical evaluation, conducted across both natural and medical image datasets, has demonstrated the robustness and generalizability of AutoAL. The results show that AutoAL consistently outperforms baselines, highlighting its capability to optimize sample selection and enhance model performance.
In future, we plan to apply AutoAL to boarder machine learning tasks such as structured prediction and consider more sophisticated AL method combinations.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: See weaknesses
soundness: 3
strengths: - The work handles an important ML problem; Active Learning (AL) with differentiable strategy search.
- The proposed method is technically sound
- Writing is clear and easy-to-follow
summary: This work introduces AutoAL, a differentiable active learning (AL) strategy search method that builds on existing AL sampling strategies. AutoAL contains two neural networks, SearchNet and FitNet, which are co-optimized through a differentiable bi-level optimization framework to identify optimal AL strategies for different tasks. Experimental results show that AutoAL outperforms individual AL algorithms and other selective approaches.
weaknesses: - Hybrid AL methods that combine uncertainty and diversity have been demonstrated to perform effectively in a variety of situations. It would be beneficial to include examples where the proposed AutoAL approach is particularly necessary or advantageous for specific applications.
- The bi-level optimization within AutoAL relies on labeled data. How does the algorithm perform if the labeled data is skewed or imbalanced? For instance, if the initial labeled set suffers from class imbalance, might this severely impair the algorithm? The assumption of a randomly selected initial set, as used in the current experiments, appears to be less practical.
- Similarly, is there a guarantee that the AutoAL approach, trained with labeled data from the current AL round, will identify the most informative samples from the unlabeled pool in the subsequent AL round? A more detailed analysis of the algorithm's guarantees is necessary.
- The approach of training an additional network for sample selection shows similarities to [1] employing meta-learning with an additional network for querying.
- Can the proposed method be applied to the open-set AL problem [1]?
- The datasets used in the experiments are of small scale. It is imperative to validate the performance on large-scale datasets, such as ImageNet.

---
[1] Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning, NeurIPS, 2022

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - How does AutoAL perform in terms of computational efficiency compared to traditional methods on large-scale datasets?
- What mechanisms ensure robustness against convergence issues and local minima in the bi-level optimization?
- Can AutoAL be extended to generate new active learning strategies dynamically rather than relying solely on a predefined candidate pool?
soundness: 2
strengths: - A new approach that automates active learning strategy selection through differentiable optimization, surpassing manual and non-differentiable methods.
- Effective integration of strategy selection and data modeling via the bi-level optimization of SearchNet and FitNet.
- Flexibility and adaptability, allowing incorporation of multiple existing strategies and tailoring to specific tasks and data distributions.
summary: The paper presents AutoAL, a framework for automated active learning that optimizes query strategy selection using differentiable methods. Traditional active learning approaches often rely on predefined strategies like uncertainty sampling or diversity sampling, which may not perform optimally across different datasets or tasks. AutoAL addresses this limitation by integrating existing active learning strategies into a unified framework. It employs two neural networks, SearchNet and FitNet, within a bi-level optimization structure to automate the selection process. By relaxing the discrete search space of active learning strategies into a continuous domain, AutoAL enables gradient-based optimization, enhancing computational efficiency and adaptability. Experimental results demonstrate that AutoAL consistently outperforms individual strategies and other selective methods across various natural and medical image datasets, highlighting its effectiveness and versatility.
weaknesses: - Increased complexity and computational overhead due to the additional neural networks and bi-level optimization, potentially challenging scalability on large datasets.
- Dependence on a predefined pool of candidate strategies, which may limit performance if optimal strategies are not included.
- Lack of in-depth theoretical analysis explaining the method's effectiveness and the conditions under which it performs best, possibly affecting generalizability.

---

confidence: 1
contribution: 2
initial_rating: 3
presentation: 2
questions: see above
soundness: 2
strengths: 1. The studied strategy selection problem for active learning is important.
2. The bi-level optimization strategy is rational.
summary: This paper proposes an active strategy query algorithm where the optimal query strategy is selected by a bi-level optimization network. In particular, the authors aggregates the query strategies by a scoring function implemented as a Gaussian Mixture Model. Then, the authors split out a validation set from the labeled samples to guide scoring function calculation. Experimental results show that the proposed method supasses the baselines.
weaknesses: 1. There is still room for improvement in the paper writing.

   1.1. It's unnecessary to name the two networks "fitnet" and "searchnet," as it seems intended to make people think this is a significant innovation. However, in meta-learning, this kind of separated network design and bi-level optimization paradigm is very common.

   2.1. The notations are somewhat confusing. For example, the authors didn't clearly define the output of the search net in Section 3.2, making it hard to understand Sec 3.2. It wasn’t until I finished reading the method section that I realized the output is actually a sample-wise score, forming an aggregation of scores for different queries.

2. The novelty of this paper is relatively limited. The proposed meta-learning/bi-level optimization has been applied to AL [1,2]. Also, I think the algorithm design is too complicated.

3. The motivation for modeling the scores by GMM distributions is unclear. Why is the score function of each strategy distributed as a Gaussian Distribution? Why is the final score function a linear weighted aggregation of different strategies? The authors should provide a concrete application or example.

4. The comparison methods are too outdated, with the latest ones being LPL and BADGE from 2019. Additionally, the datasets are quite limited; despite the complexity of the method design, only CIFAR and MNIST datasets were used. Validation should be conducted on the ImageNet dataset (at least Image100). Otherwise, given that the algorithm design is much more complex than the baselines, its effectiveness cannot be convincingly demonstrated.

[1] Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy Hospedales. 2018. Meta-learning transferable active learning policies by deep reinforcement learning. International Workshop on Automatic Machine Learning (ICML AutoML 2018).

[2] https://grlplus.github.io/papers/96.pdf

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See the Weaknesses.
soundness: 3
strengths: 1. The problem studied in this paper is valuable. This paper presents the first differentiable AL strategy search method. 
2. The proposed AutoAL approach is interesting and easily followed.
3. The paper is well organized. 
4. The experiments validate the effectiveness of the proposed approach, and the ablation study in Figure 5 is insightful.
summary: This paper attempts to tackle the "generalization problem" of active learning (AL) algorithms across data scenarios. I believe this is a core issue in the current active learning field. This paper proposes AutoAL, a differentiable AL strategy search method to select the most effective AL sampling strategies in each iteration. It consists of two neural nets, named SearchNet and FitNet, which are optimized concurrently under a differentiable bi-level optimization framework. The experiments on multiple datasets validate the effectiveness of the proposed approach.
weaknesses: 1. My only concern is the efficiency of the AutoAL algorithm. Although more efficient solutions have been proposed to solve second-order optimization problems, I cannot find any relevant experiments to verify them.

Meta-review: This work introduces AutoAL, the first differentiable active learning strategy search method, which uses a bi-level optimization framework to adaptively identify optimal AL strategies, achieving superior accuracy and efficiency across diverse tasks and domains.

There reviewers reviewed this paper. All of them agree that this paper is not yet ready for publication at this stage. We recommend the authors go through all the reviews and address it in the next version.

Decision: Reject