Reliable Active Learning via Influence Functions
Authors: Meng Xia
Ricardo Henao
Venue: Accepted by TMLR ()

Abstract: Due to the high cost and time-consuming nature of collecting labeled data, having insufficient labeled data is a common challenge that can negatively impact the performance of deep learning models when applied to real-world applications. Active learning (AL) aims to reduce the cost and time required for obtaining labeled data by selecting valuable samples during model training. However, recent works have pointed out the performance unreliability of existing AL algorithms for deep learning (DL) architectures under different scenarios, which manifests as their performance being comparable (or worse) to that of basic random selection. This behavior compromises the applicability of these approaches. We address this problem by proposing a theoretically motivated AL framework for DL architectures. We demonstrate that the most valuable samples for the model are those that, unsurprisingly, improve its performance on the entire dataset, most of which is unlabeled, and present a framework to efficiently estimate such performance (or loss) via influence functions, pseudo labels and diversity selection. Experimental results show that the proposed reliable active learning via influence functions (RALIF) can consistently outperform the random selection baseline as well as other existing and state-of-the art active learning approaches.

Published in Transactions on Machine Learning Research (11/2023)
Reliable Active Learning via Influence Functions
Meng Xia
meng.ria@@duke..cm edu
Deportment of Electrical & Computer Engineering University of Duke
Ricardo Henao
Bukke, edu
Deportment of Electrical & Computer Engineering
University of Duke
Biological, Environmental Sciences and Engincering Division
King Abdullah University of Science and Technology (KAUST)
Reviewed on OpenReview: https: // openreview. net/ forum? id= 4N9YICB6NN
Abstract
Due to the high cost and time-consuming nature of collecting labeled data, having insufficient labeled data a common challenge that can negatively impact the performance of deep learning models when applied to real-world applications. Active learning (AT) aims to reduce the cost and time required for obtaining labeled data by selecting valuable samples during model training. However, recent works have pointed out the performance unreliability of existing AL algorithms for deep learning (DL.) architectures under different scenarios, which manifests as their performance being comparable (or worse) to that of basic random selection. This behavior compromises the applicability of these approaches. We address this problem by proposing a theoretically motivated AL framework for DL architectures. We demonstrate that the most valuable samples for the model are those that, unsurprisingly, improve its performance on the entire dataset, most of which is unlabeled, and present framework to efficiently estimate such performance (or loss) via influence functions, pseudo labels and diversity selection. Experimental results show that the proposed retiable active learning via influence functions (RALIF) can consistently outperform the random selection baseline as well as other existing and state-of-the art active learning approaches.
1 Introduction
In the last decade, deep learning (DL) models have achieved record-breaking performance on datasets with millions even billions of samples (Chen et al., 2022; 2023; Jain al., 2022). However, their broad practical applicability in scenarios where data is scarce has been limited mainly due the lack of sufficient (and high- quality) labeled data for training. This is often due to the high cost time-consuming nature of manual labeling (or sample annotation) often requiring subject-matter expertise. Aiming to address this gap in the field, active learning (AL), which has been historically applied to traditional machine learning approaches, emerged as a means to improve the applicability of deep learning models in such scenarios. The main objective of AL to design strategies to seleetively annotate (label) small subsets of a large (unlabeled) dataset that model performance can be maximized while controlling the cost. associated with annotations (Settles, 2009; Sener & Savarese, 2017). It is worth noting that there are also situations where both data (e.g., images) and labels are scarce, however, such scenario is outside of the scope of this work.
Early AL algorithms (Lewis & Gale, 1994; Dagan & Engelson, 1995; McCallum et al., 1998; Tong & Koller, 2001; Freytag et al., 2014) were originally developed for traditional machine learning models, such as logistic regression (Kleinbaum et al., 2002), support vector machines (SVMs) (Hearst al., 1998), or naive Bayes classifiers (Rish al., 2001). Unfortunately, applying these algorithms directly to deep neural networks is challenging because they rely unique characteristics of traditional machine learning models (e.g., the

---

Published in Transactions on Machine Learning Research (11/2023)
new selected data with labels provided by oracle
Obtain Pseudo Labels
Generalization Loss (Estimated via Influence Functions)
Oracle (Get True Labels)
Pesudo labels
Estimated loss for unlabeled data.
Select Top-K Samples
Labeled Dalaset
DL Model
for unlabeled data
Rest unlabled data
Unlabeled Dataset
The basic reliable active learning via influence functions (RALIF) framework. Starting with a small labeled dataset and a much larger unlabeled dataset, al each selection cycle, we first train the larget DL model with the labeled dataset and generate pseudo labels for the entire unlabeled datasot using the trained target DL model. We then use the selection mechanism based on influence functions to rank the most likely useful samples from the unlabeled dataset, query the oracle to obtain the true labels for the top K ranked samples, and update the labeled and unlabeled datasets accordingly. This process is repeated for a fixed number of cycles or until the labeling budget has been spent or performance target has been achieved.
Figure 1:
convexity their learning objective), and the widely known, but incompatible, need for large datasets when training DL models. As an example of the former, Tong & Koller (2001) leveraged the version space duality in SVMs consisting in that each training sample corresponds of SVMs (Vapnik, 1999; Herbrich et al., 2001), and aimed to select and label samples that have the most significant impact on shrinking such solution space, which is sensible approach considering that SVMs are known to have a unigue optimum classification hyperplane. Unfortunately, there is no theoretical basis or direct evidence to support the assumption that each sample in a deep neural network corresponds to a hyperplane in their solution space. In fact, neural networks have in general non-convex optimization objectives known for not yielding a unique global optimum, and instead being overwhelmed by local optima.
hyperplane in the version (solution) space
In contrast, numerous DL-based AL algorithms have been proposed specifically for deep learning models (Gal et al., 2017; Beluch et al., 2018; Sinha et al., 2019; Yoo & Kweon, 2019; Liu et al., 2021; Wang et al., 2022; Yi et al., 2022; Vo et al., 2022). While DL-based AL algorithms have demonstrated strong performance in some scenarios, recent work by Ash et al. (2019); Munjal et al. (2022); Wang et al. (2022) revealed the performance unreliability of many DL-based AL methods across different scenarios. Specifically and quite often, existing DL-based AL algorithms (Wang & Shang, 2014; Sener & Savarese, 2017; Gal et al., 2017; Sinha et al., 2019) fail to consistently outperform the most basic (no skill) random AL selection mechanism, which assumes that all (unlabeled) samples are equally likely to improve the performance of the model if labeled and added to the training set. This unreliability undermines their applicability and adoption, particularly in situations when labeling costs are high.
Motivated by such findings, we propose a more reliable AL algorithm for DL architectures we call reliable active learning via influence functions (RALIF), which benefits from a theoretically grounded selection mechanism that increases the likelihood of selecting samples that will yield performance improvements when labeled and included dataset used for model training. The RALIF framework illustrated in Figure introduces a new objective for AL that leverages the concept of influence function (Hampel, 1974) to make the sample selection process more efficient. Specifically, Koh & Liang (2017) showed that influence functions approximate the changes in model parameters caused by incorporating new samples into the training dataset, but without the need for explicitly retraining the deep neural network. This enables RALIF, in principle, to efficiently quantily the generalization ability of a model by estimating the influence of new samples on the complete-dataset performance, thus enabling informed decisions during the AL sample selection process. Our contribution consists on leveraging pscudo labels to repurpose these influence functions for the AL case where labels are not available for most of the dataset. Experimental results on three datasets (CIFAR10, FICAR100 and iNaturalist) demonstrate that. RALIF achieves consistent and superior performance compared to random
2

---

Published in Transactions on Machine Learning Research (11/2023)
selection (as no-skill baseline), well as existing AL approaches across various practical scenarios, thus providing strong empirical evidence of the reliability and effectiveness of RALIF.
The rest of this paper is organized as follows, in Section 2 we discuss the related work in the context of the RALIF framework. In Section 3, we provide a detailed description of the notation used in this paper and define the standard setting for a (sequential) AL algorithms. In Section 4, we introduce the proposed AL objective and describe the strategies we employ to implement RALIF with deep learning neural networks. Subsequently, in Section 5, present the results of our experiments, which highlight the performance reliability of the proposed AL framework across different scenarios, and conclude in Section 6 with a discussion and directions for future work.
2 Related Work
AL for traditional machine learning models Among the early AL algorithms, uncertainly sampling stands out as one of the most commonly used. This strategy aims to selecting samples for which predictions with the model trained on the current labeled dataset exhibit the highest uncertainty. The rationale behind this approach is that selecting samples with uncertain predictions is believed to provide additional information to the model that can likely improve its performance relative to the current version of the trained model. For example, Lewis & Gale (1994) introduced probabilistic binary classifier that utilized a bag-of-words representation for text data and proposed selecting samples with posterior predictive probabilities close to the decision boundary. In another example involving multi-class text classification tasks, Settles & Craven (2008) suggested selecting samples for which the entropy of the predicted outputs of conditional random field (Tafferty et. al., 2001) is high, as a proxy for uncertainty.
Another popular early introduced AL strategy is query-by-conee bymommmttee (Seung et al., 1992). This approach consists of constructing committee of models trained on a labeled dataset. The selection process typically involves choosing samples that elicit the most diverse members. Importantly, constructing the committee is a key aspect of this approach. For instance, McCallum et al. (1998) constructed the commitlee by training naive Bayes classifier with a labeled text dalaset and sampled multiple naive Bayes classifier instances from the posterior distribution of the trained parameters. Alternatively, Dagan & Engelson (1995) trained a hidden Markov model (HMM) (Rabiner, 1989) with a labeled text dataset and constructed the committee by sampling a collection of HMMs based on the posterior distribution of the trained HMM parameters. Moreover, there are several other early AL strategies, such as Variance Reduction (Cohn et al., 1996) or Fisher Information Ratio (Zhang & Oles, 2000), which aim to minimize the expected generalization error of the model by selecting unlabeled samples that minimize the variance or the Fisher information ratio estimated for the model including such samples. See Settles (2009) for a comprehensive overview of these methods.
heterogeneous predictions among the committee
AL for deep neural networks Over the past few years, an abundance of DL-based AL algorithms have been proposed, most of which can be categorized into three main strategies, namely, i) based on uncertainty (Wang& & Shang, 2014; Beluch et al., 2018), ti) based on diversity (Sener & Savarese, 2017; Geifman & El-Yaniv, 2017), and dii) hybrid approaches (Tughoofer, 2012; Ash et al., 2019; Wu et al., 2021).
Uncertainty-based AL hims for DL architectures built upon the ideas of the early uncertainty sampling strategies for AL (Hwa, 2004; Settles & Craven, 2008), where uncertainty is typically estimated by calculating the entropy of the model predictions for each unlabeled sample (Wang & Shang, 2014). In addition to directly using the model's predictions to estimate uncertainty, several other methods have been proposed to estimate the uncertainty of deep neural networks for unlabeled samples. One of such methods, proposed by Gal & Ghahramani (2016), involved applying dropout (Srivastava et al., 2014) before each layer of the neural network. By sampling from the dropout masks, multiple predictions could be obtained for cach unlabeled sample. These predictions were then averaged to obtain the final prediction for the sample, and the entropy of the expected prediction could be calculated as a measure of uncertainty. Another approach, presented by Beluch et al. (2018), followed a similar framework to Gal & Ghahramani (2016), but usod a different method to obtain multiple predictions for unlabeled samples. Instead of using dropout, Beluch et al. (2018) trained multiple models with identical architecture but different. random initialization seeds using the same
3

---

Published in Transactions on Machine Learning Research (11/2023)
labeled dataset. By obtaining predictions from these independently trained models, multiple predictions for unlabeled samples could be obtained, which could then be used to estimate uncertainty. Note that this approach is more computationally expensive than simply using dropout like in Gal & Ghahramani (2016).
In bavee AL algorithms, samples that are representative of the entire dataset distribution are preferred. For instance, in Sener & Savarese (2017), features extracted from the model trained on the labeled dataset were used as representations for the unlabeled samples. Then, their goal was to select a subset (batch) of samples that minimized the maximum distance of any arbitrary sample from the dataset to its nearest selected sample. Similarly, Geifman & Yaniv (2017) also used features extracted from the model trained on the labeled dataset as represcntations for the unlabeled samples. They focused on selecting samples that were farthest away from the labeled dataset. By selecting samples that were farthest from the labeled dataset, they aimed to capture the most diverse and informative samples that are distinct from those already labeled.
Hybrid strategies typically combine uncertainty and diversity approaches to select samples that are both uncertain and highly representative of the entire unlabeled dataset. For instance, Lughofer (2012) proposed a two-stage hybrid active learning algorithm. In the first stage, they employed the evolving vector quantization approach (Lughofer, 2008) to cluster the data. In the second stage, they selected samples that were either close to the cluster centers (highly representative) or near the boundaries of clusters (highly uncertain). Tn Ash et al. (2019), a similar approach to the diversity strategy presented in Sener & Savarese (2017) was employed, but with a consideration of the trade-off between uncertainty and diversity. Pseudo labels were generated for the unlabeled samples using the model trained on the labeled dataset, and the gradients of the loss function with respect to the model parameters were computed. The gradients of the unlabeled samples were then used as representations for clustering algorithm. The first cluster center was selected as the unlabeled sample with the largest norm of gradients (uncertainty), and subsequent centers were selected based on their distance to the nearest center (diversity). Another study by Wu et al. (2021) introduced a hybrid active learning approach using two deep neural networks. One network was trained to estimate the diversity of the unlabeled samples, while the other network estimated the loss for each unlabeled sample. Samples with high diversity (representativeness) and high loss (uncertainty) were then selected for labeling.
There are also AL algorithms specifically designed based on the understanding or techniques specific to deep neural networks. For example, Sinha et. al. (2019) leveraged adversarial training to select unlabeled samples that were most dissimilar from labeled samples. Yoo & Kweon (2019) posited that samples with the largest loss indicated the largest informativeness, thus they trained module to estimate loss for unlabeled samples, then selected samples with the largest predicted loss. Furthermore, in their study, Liu et al. (2021) aimed to select samples capable of reducing the loss of a. labeled reference set. More recently, Wang et al. (2022) proved that samples that mostly change the DNN weights were most useful, and accordingly, they selected samples that generated the largest approximate gradients.
It is important to highlight that while our work shares similarities with some existing AL algorithms for DNNs, there significant differences exist between our approach and others. For example, though both the approaches introduced by Ash et al. (2019) and Wang el al. (2022), as well as the proposed RALIF, involve the utilization of sample gradients, our motivations for sample selection differ significantly. In Ash et al. (2019), the authors argue that samples for which the current model produce uncertain predictions will yield larger gradients. Thus, they prefer to select samples with larger gradients. On the other hand, Wang et al. (2022) adopts a different perspective. They propose that the magnitude of gradients introduced by samples, i.e., the extent to which samples influence the DNN weights, can serve unobserved test data. As such, they consider that lowering the amount of samples with large gradients will result in a higher upper bound for the loss of the unobserved test dataset, implying they too favor selecting samples with larger gradient magnitudes. In contrast, the proposed RALIF leverages sample gradients and influence functions to assess the impact of samples on the entire dataset. To further enhance the estimation of the influence of unlabeled samples on the entire dataset, we introduce the use of using pseudo labels to approximate the true dataset distribution.
upper bound for the loss within
When comparing our approach to Liu et al. (2021), the most siguificant distinction lies in the objective they use to select samples that reduce the task loss on a labeled reference set, whereas our approach prioritizes
4

---

Published in Transactions on Machine Learning Research (11/2023)
selecting samples that reduce the task loss of the entire dataset. It is crucial to note that their algorithm (TSAT: influence selection for active learning) necessitates a labeled reference set in addition to the initial labeled dataset, resulting in the need for annotations not only for the initial labeled dataset but also for this additional reference set at the outset of the active learning selection process. In contrast, RALIF only requires annotations for the initial labeled dataset, thus underscoring that in principle, we need fewer annotations overall. Instead of constructing a reference set with labels, we rely on pseudo-labels to effectively approximate the true dataset distributions, and we provide theoretical results in the following sections to substantiate this approximation. Another notable difference is the incorporation of a diversity selection operation in our methodoloyy. As demonstrated by the results in Table 1, this diversity operation ensures the selection of samples that are not only valuable but also diverse, resulting in an enhanced overall performance for our approach. We also provide experimental comparisons between RALIF and their approach under a specific active learning setting in Appendix A.6, The results indicate the superior performance of RALIF compared to their proposod ISAL method.
Reliability in active learning While most AL algorithms for traditional machine learning models with convex objectives enjoy theoretical guarantees for their performance (Tong & Koller, 2001; Brinker, 2003; He et al., 2004), this is in general not the case for DL-based AL algorithmss For instance, uncertainty- and DL- based AL algorithms (Wang& & Shang, 2014; Gal et al., 2017) have been shown to perform comparably to basic random selection in several scenarios (Sener & Savarese, 2017), whereas the performance of Wang & Shang (2014) was even slightly worse than random selection in the carly stages of the AL selection process (Ash et al., 2019). Studies by Yoo & Kweon (2019) and Wang et al. (2022) revealed that DL-based AL algorithums with theoretical support, such as Sener & Savarese (2017) and Gal et al. (2017), also failed to consistently outperform random selection. Further, experiments conducted by Munjal et al. (2022) demonstrated that many DL-based AL algorithms, including Wang & Shang (2014); Sener & Savarese (2017); Gal et al. (2017); Sinha et al. (2019), exhibited inconsistent performance when compared to random selection under the same experimental conditions. In their experiments, the conditions referred to applying these algorithms on the same datasets, using the same classification model, and employing the same sample sclection size at cach active learning selection cycle; all of this to make comparisons casier and fairer.
This reported inconsistency in performance undermines the applicability and adoption of AL algorithms in real-world scenarios. Therefore, we are motivated to propose DL-based AL algorithm that is not only theoretically grounded, but also demonstrates reliable performance across various AL selection scenarios, e.g. different datasets, model architectures and selected sample size (batch size) in each AL selection cycle.
3 Problem Definition (Active Learning)
We consider a hypothetical multi-class classification datasct D = De)% where n is the total number of samples, 2, E X' represents an input sample (e.g., images in our experiments), and Yi E ) = [1, represents its corresponding class label, which is not available for the majority of the dataset. D. In active learning, we begin with a small labeled subset Lo = (:),, containing mo < n labeled samples. The remaining samples, which are unlabeled, are denoted as the set Un = {z)}ī=mot1- Importantly, in active learning (AL),, we do not have access to the true labels of the samples in Un. denoted as Uy = (uyl ~ mo) Let M denote a classification learner, 6.9., a deep neural network, and let Mr denote the learner M trained with labeled dataset Lo. Further, (33 : X x y - R denotes the loss function used during the training process, e.g., the standard cross-entropy loss for classification tasks.
In each AL selection cycle (full model training iteration), sequential AL algorithm seleets a single sample Ts ~ U--1 from the unlabeled set U--1 at cycle e- - 1, and queries its true label ys with an oracle function resulting in a new labeled dataset Le = Le-1 (z., ya). The most basic (no skill) strategy for AL is random selection, where sample I, randomly chosen from U--1 and its true label is queried; this denotod as the Random strategy. Arguably the most common baseline strategy is the Uncerlainly strategy, which selects the sample Fs from U--1 with the largest predictive uncertainty under the current model M
In practice, this replaced by human expert observing the sample and producing the label of interest.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Such uncertainty is quantified in terms of the entropy (Shannon, 1948). A more detailed information about these two strategies can be found in Section 5.
4 Reliable Sample Selection for Active Learning
We propose a selection strategy that aims to pick samples from the unlabeled set Uc-1, denoted as 2g ~ Uc-1: such that the loss of Mr. on the entire set is minimized. This objective reflects the goal of improving the gencralization performance of ML, on the entire dataset D, which in turn a proxy for its gencralization ability. Formally then, the objective of the sequential AL algorithm in each cycle e can be formulated
arg mig RE..W.-DDEE 86 MLJ] 1 Σ M0, Wi ML.) v.-Uee1
There are two key barriers preventing us from using (1) directly, namely, 1) at cycle e we do not have access to the label for H, to be able to obtain Mt. via labeled set Le Le-1 U (#x. and ti) we do not have access to most of the labels in D, but only to those in L
Below, we describe how we propose approximating (1) to use it as a means to select samples that are likely to improve the generalization performance of the model ML. on D. Specifically, we will introduce a variant objective based on (1) as a modification to the original AT objective, which underscores the issue concerning lacking most of the labels in D. Then, we will propose strategies to address the practical implementation challenges that arisc when applying such an objective variant in practice. More preciscly, we will address the challenges associated with lacking labels for most samples and the expensive selection process resulting due to the time-consuming nature of DL model training. Finally, we motivate the need for selecting multiple samples (a batch) in each cycle in the context of deep learning. Taken together, the objective, implementation strategies and batch selection constitute the proposed RALIF which is conceptually summarized in Figure 1.
4.1 AL Objective Revisited
We can readily derive the following upper bound for (1):
B(o,a)>- ML.)]
1
a
Ux yss ML.) +
Σ 72
me
(xs yes ML.)
n +1
me
1x, Mos My +
Ux Ves Mt.)
(2)
71
Ma immet1
Me
Ba (x; Maa Vss ML.) ) + E (eran)t Mrx Vs: ML.)I.
Unlabeled subset loss
Training
in which we made explicit that the labels for Ue denoted as U. are not available in practice. From (2) we see that for the purpose of selecting F, ~ U--1 via B(s.n))=D (rx ye: ML,)), the training loss becomes irrelevant (a constant), so it can be safely ignored. Therefore, we can use the following objective in place of (1):
arg min U( [1x ya ML.)).
however, akin to that in (1), the label ys and label set UY are not available. In order to tackle this problem, below we present Theorem that bridges the gap between practical but estimated loss and the inaccessible but actual loss.
Theorem 1. Given the true data joint distribution PxPrIx, where X is the impul space and y is the label space, and an estimated data joint distribution pxprix (assuming we can access the input distribution px but not the true label conditional py x), any posilice and bounded loss function M) X x + [0, L] for model M must satisfy
B(ex)~pxPrix (I, yi M) S B(e.D)<pxpy (x 0: M) + 2L XxxxxxxxXxXxxxx
(4)
where (rv(),) denoles the total variation difference between two distributions.
6

---

Published in Transactions on Machine Learning Research (11/2023)
The detailed proof for Theorem 1 can be found in Appendix A.1. Conceptually, Theorem 1 shows that the difference between the estimated and true loss in (4), obtained via pyxx and PyІx, respectively, is bounded by two terms: i) the magnitude, L, of the loss function, and i)) the difference between the real and cstimated joint distributions via 8yv (pxPrix, PxйvIx). In most cases, for instance when using the standard cross-entropy ropy loss for classificatton, the total variation is bounded by some power r of the norm of the parameters |[W]l,, where W denotes all the parameters of the model, which itself is bounded via the use of regularization term during the training process (Figueiredo, 2001; Figueiredo & Jain, 2001; Figueiredo, 2003). We provide detailed proof for the cross-entropy loss in Appendix A.1.
With regard to the second term involving (rv(:;), we propose to generate pscudo labels from M as a means to intrinsically and effectively sample from the estimated label conditional distribution pyix implied by M Conveniently, for the purpose of sample selection, the second term in (4) becomes approximately a constant by assuming that 6TV PrxxuxPrixUe,,PPP PPXus,,PyXXUe - O, where XUx, and X Unr for T. # F.g are used (in slight abuse of notation) denote the estimated conditionals with Le = Lc-1 U (Zs, Ve) and Le 1 V Mer); respectively, which is reasonable long as m is not too small. Consequently, combining (4) with (3) results in
arg min (aar-Ue.errrriiie Ma,, Des M..)]. ---1111
(2,,
(5)
Unlike (3), we can readily evaluate the loss (x ys; ML.) on U2 for every sample in U2-1- Below, we will describe how to implement (5) provided that Mr. is not available when selecting Vs ~ Up-1.
4.2 Estimating Loss without Retraining
Though the AL selection objective introduced in (5) is thcoretically justified, its practical application to deep neural network architectures raises two key issues that need to be addressed. The first issue has to do with efficiency in that implementing equation (5) necessitates training ML. for each data, candidate (x. ~ 1, ~ 1), resulting in the training of n m distinct models. Unfortunately, the time- consuming nature of training deep neural networks makes this process highly impractical or even prohibitive in some cases. The second problem associated with labeling that obtaining ML. for each candidate data requires access to the true label y/2 U²-,, which is unavailable during AL selection cycles. To address these two issues, we propose the following strategies.
To tackle the efficiency problem due to the need of explicitly training n mc-1 models in each AL selection cycle, we leverage influence functions (IL) (Hampel, 1974). The influence function serves as proxy to quantify how the parameters of a model change when a training point, denoted as (I,, is weighted by an infinitesimal amount. To provide a fundamental understanding of the influence function, we present a simple example. Let us consider a model, denoted as M, trained with a labeled datasct L (2,, 9:) whose parameters E O are obtained via 0 = argmingge Σ I(x,, yc: 0). Now, if we incrementally increase the weight of specific sample (I., by small value e, the resulting parameters, denoted as d.,(r,...)) can be expressed as: (e,( = argmingge Σ/= [xx; ys: f) + cl(x,, 0): The influonce function, introduced by Hampel (1974), gives insight into the impact of this incremental weighting of (Ir,),),, which results in can be defined as:
i (e.(e..))) Formally, the inffuence function I[ 0)
d0 dDe((en)))) de
IL 0)
V 1s;
(6)
(P((,,,,3)) is the Hessian of model M. The detailed proof of (6) can be found in Koh where Hg - " + & Liang (2017).
In the context of active learning, we aim to obtain model Mr. in (5) by incorporating a sample (z., ys) ~ U-- into the training datasct Le-1- Assuming that we already have the parameters of the model trained with T denoted as 0 M which were obtained via 0 M argmingge 1 Em [(x, yr; 0). The parameters m.

---

Published in Transactions on Machine Learning Research (11/2023)
of ML. can be approximated as
me (x, Mii 0) + -(2ss No: 0).
ML. 2 argminggo
me
me
This approximation suggests that incorporating sample (Ix, Us) ~ Uc-1 into the training dataset T can be thought as to incrementally weighting the sample (I):Ws) by m²² Leveraging the influence function as in (6), we can estimate the parameters of Mr. denoted as OML. efficiently, that is via Oze. R 0M + IL (ML.. Conveniently, this allows to efficiently obtain the parameters for ML. in (5) without the need to explicitly train Mr. with - U (I...)s) for each data candidate (x,, ~ Uc-1, Us ~ U2-1). Additional details about the theory and efficient implementation of the IL can be found in Koh & Liang (2017).
To address the labeling problem associated with lacking access to ys UY we consider three approaches. The first one, which we refer to single-IL, uses pseudo labels generated by ML.-17 i.e., by simply replacing Vs with the (most likely) pseudo label Ú.- arg max ML (x.). This can be formulated as
Bag--Uesh Modierairyyyy (xx Fy: ML.)] ~ E(a,-U..dedi (x) (2,, Dc ML.., U (x.: 0z))].
Though conceptually simple, this method may introduce severe approximation bias if ys + 9s. Alternatively, to obtain a relaxed approximation of B( [[ ys! ML.)] for each zs: we account for all class labels for each Is, rather than relying solely on the single most probable (maximum likelihood) class ds
(7). One straightforward approach is to weight all possible classes for x, using the probabilistic predictions from Mr. (x.). We refer to this approach as expeclation-II. and formulate as
K
Bi Di Mr.))
Σ
pa (1ss) Ble Des Mr (t(r,
where 8 denotes setting the label for v, as class k, pk (Zx) is the predicted probability (weight) that v, is of class k which is obtained from ML.119 and K is the number of classes. Note that this is conceptually equivalent to considering all possible outcomes for Ia, but weighting them according to how likely they are under the current version of the model, i.e., ML....
In practice, we found that (8) has the potentially significant problem that for unlikely classes, (x.), is likely to be nonzero, thus introducing estimation noise. Moreover, we are mostly interested in classes for which (x.) is large or at least comparable to max (z,). With this in mind, we propose an alternative solution we call truncated-IL, IL, which involves considering several high-probability class labels for each x, instead of all classes. More formally write
K
El xx) (1 Di: ML.) ~
(pp()x)) (x())) Bea.-Ue. (a 0es M M ((.,)1 (9)
from which we see that compared with (8), (9) has a indicator function x()) defined as
{1 if pa(1))>T
a (p)(xx))
(10)
otherwise
where 7 is a threshold value set to filter out highly unlikely class assignments for Vg. For all experiments, we set 7 = , which we found empirically gives the best overall performance. Nevertheless and for illustration, we provide comparison of the results obtained by using different values of in one specific active learning scenario in Appendix A.3.
Having addressed the challenges involved in implementing (5) that are associated with efficiency and label- ing, we have developed a single-sample sequential DL-based AL algorithm. Conceptually, we will rank all unlabelod samples based on their ability to minimize the model loss on the current unlabeled datasct using single-IL IL in (7), expectation-II in (8) or truncated-II IL in (9), and then select the top-ranked sample at each AL selection cycle, however, as discussed below, this approach may not be practical for DL, models.
8

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 1 RALLF:: Reliable active learning via influence functions with truncated-IL selection.
Input: Lo: Initial labeled dataset, Uo: initial unlabeled dataset, M: model, C: AL cycles, B: query set, P: candidate set, @M (-)= convolutional encoder, and т: truncation hyperparameter.
1: forc=1, c=1,..., C do
2: Train model Mi.1 with labels set Le-1
3: IP| = -0.
4: Va; ~ U. generate pseudo-labels ū = arg max ML.-1 (2:)
5: for J, EU.-1 do
Generate probabilisttic predictions {px(x,)) from M (2x))
Calculate Bipr: Mi; Mt..)] via (9)
end for
9: Select top-|P| samples faitipt with minimum Bez,~UerdrBy Ma De; Mr,
10:
Run *-MEANS++ on {ØMe._. (ri))ir to select |B] diverse samples
11: L. - U {1 Us))
12:
end for
Return: Final labeled dataset Le (of size Lo + CJB))
4.3 Batch Selection AL for DL
In the previous subsection we introduced a practical single-sample sequential deep learning active learning algorithm. However, when dealing with deep neural networks, adding only one sample at the time to the training dataset is not likely to meaningfully affect performance between AL cycles?, but will definitely increase computational requirements to a point that in some scenarios it may render AL prohibitive. There- fore, we opt for batch selection in each AL cycle, which is a common practice in active learning selection mechanisms for deep neural networks (Kirsch et al., 2019; Sener & Savarese, 2017; Wang et al., 2022).
From (5) we can conceptualize that given sample x, that minimizes the loss, there are likely other samples in U--1 that being very similar to 2, also have comparably small losses. However, by virtue of being similar they are also likely to be redundant in their ability to improve model M. Though such redundancy is not likely to negatively impact performance, it certainly affects efficiency, by requiring unnecessary labeling efforts. This phenomenon of batch AL approaches has been described before (Beluch et al., 2018; Ash et al., 2019; Wang et al., 2022), and several approaches have been proposed to mitigate this problem. For example, Beluch et al. (2018) suggested selecting samples from a subset of the unlabeled dataset rather than considcring the entire dataset to avoid including highly similar samples in sclected batch.
To alleviate such issue, we propose a two-step approach. First, we utilize single-TT. in (7), especlation- in (8) or truncated-IL. in (9) to select pool of candidate samples with the smallest expected losses on the unlabeled dataset. In practice, we set the size of the candidate pool P to be 10% of the size of the unlabeled set, i.e., P == 0.1 x [Ecl- Second, to ensure the diversity of the final selected samples, we cluster the P candidates into B clusters, where B refers to the budget size for each AL selection cycle, and then select from P the centroids of these clusters as the samples selected for labeling. In order to ensure that selected samples are representative of the entire pool of candidates, we use the features extracted from the convolutional encoder as the representation space for clustering. Specifically, we use the K-MEANS+ (Arthur & Vassilvitskii, 2007) algorithm, a variant of the popular K-means (MacQueen, 1967) algorithm. This particular clustering algorithm was chosen because work by Arthur & Vassilvitskii (2007); Ash et al. (2019) demonstrated that +-MEANS++ has a fast convergence speed while generating relatively diverse clustering centroids, which encourages the diversity of the set of selected samples.
In practice, one cycle of the RALIF framework consists on selecting a diverse set of samples with the smallest loss the unlabeled data given in (7), (8) (9), while using pseudo labels from the current trained model as proxy for the true, but the unavailable, labels. The complete process for truneated-IL in (9) is summarized
in Algorithm 1. Further, we provide more complete algorithm in Appendix A.2, which includes all selection approaches, ie.,, single-IT. in (7), erpectation-II. in (8), and truncated-IT. in (9).
2In fact, it may be detrimental some cases due the stachastic gradient learning approaches.

---

Published in Transactions on Machine Learning Research (11/2023)
5 Experiments
Our experiments aim to demonstrate the reliability of RALIF on various selection scenarios. Below, we first introduce the baseline algorithms with which we compare. Then, we describe the different AL selection scenarios that we consider in our experiments, along with the model training details. Finally, we will present the results of our experiments, highlighting the performance of the proposed RALIF framework.
Baselines We consider the following baselines in our experiments: i) Random: selecting a random batch of unlabeled samples at each selection cycle, which serves as the naive, no skill, baseline for active learning; it) Uncertainty: we leverage the most commonly used uncertainty- based selection method introduced in Beluch et al. (2018), which involves selecting unlabeled samples with the highest entropy based on the classification probabilities produced by the model trained with the labeled dataset; iti) CoreSet (Sener & Savarese, 2017): utilizing the model trained with the labeled dalaset to extract features for all unlabeled samples, then applying the K- Center-Greedy (Wolf, 2011) algorithm on such feature space to select diverse batch of unlabeled samples; ir) BADGE (Ash et al., 2019): selecting unlabeled samples by applying the K-MEANS++ algorithu on the gradient norms calculated for all unlabeled samples based on the pseudo labels predicted by the model trained with the labeled data; and v) GradNorm (Wang et al., 2022): selecting unlabeled samples with the largest gradient norms by using their entropy as loss for backpropagation. Note that this is the same entropy used in Uncertainty.
The five baselines we consider in our experiments cover the most common types of active learning algorithms, namely, Random selection (as a naive baseline for AL), uncertainty-basee selection (Uncertainty)) diversity- based selection (Coreset), hybrid strategics (BADGE, which combines uncertainty and diversity), and DL- specific selection (GradNorm). In comparison to these baselines, RALIF can be considered a hybrid DL. specific AL algorithm as it combines the DL-specific selection (utilizing influence functions), but encouraging the selection of set of diverse samples that are likely to minimize the loss incurred by the model on the unlabeled dataset. By comparing RALIF with these baselines, we will also demonstrate how it fares with the state-of-the-art methods, i..., Ash et al. (2019); Wang et al. (2022).
AL Scenarios To simulate various selection scenarios, consider three main determinants in the active learning selection process, namely, dataset, model architecture, and budget size at each selection cycle. In our experiments, we use three datasets: Cifar10 (Krizhevsky et al., 2009), Cifar100 (Krizhevsky et al., 2009), and iNaturalist (14 super classes) (Van Horn et al., 2018), representing simple, medium complexity, and imbalanced datasots, respectively. In terms of model architectures, we consider three options: ResNet18 from scratch denoted as 1esNet18 (scratch), Res. with pretrained weights from ImageNet (Deng et al., 2009) denoted, and a simple linear classifier (a fully connected layer) with fixed but powerful encoder (Clip- Vit-patch16) denoted as CLIP. We use the standard ResNet18 (He et al., 2016) architecture for the pretrained ResNet18, the standard Clip-Vit-patch16 (Radford et al., 2021) for CLIP, and a customized version of 18 for ResNet18 (scratch)*, where the kernel size of the first convolution layer is modified from 7 to 3 and the max pooling layer is removed. Moreover, we also consider two different budget sizes: 1000 or 2000 samples at cach selection cycle. In Appendix A.5, we included additional active learning scenarios considering other datasets and models for more comprehensive evaluation.
Training Details We set different initial labeled dataset sizes for each selection scenario based on model capacity and the complexity of the dataset. For the CLIP model, which has shown excellent performance with smaller training dataset size compared to other models, we set the initial labeled dataset size to 100 samples. This choice is made to ensure that the initial accuracy is not excessively high, allowing us to observe the effectiveness of the AL algorithms.. For all other models, we set the initial labeled dataset size to 1000 samples for Cifar10 and 6000 samples for 10 and iNaturalist. This decision takes into account the higher complexity of the iNaturalist and Cifar100 datasets relative to Cifar10. We trained our models using stochastic gradient descent (Sutskever et al., 2013) with a weight decay of 10 and a momentum of 0.9. The initial learning rate, learning rate scheduler, and training epochs were set based on the model capacity. For pretrained 18 and CLIP, we employed Cosine (Loshchilov & Hutter, 2016)
3 Source code available attps://githbb con httpa//c///nnnggggorrorro------
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: Ablation study results. We present the mean and standard deviation of the performance metrics for all algorithms (different selection strategics and diversity selection) over three trials. Best results are highlighted in green.
Diversity
Cycle
Candidate Selection
AUCA Ratio
k- 4-MEANS++
First (1*t)
Last (9th)
Random
85.31 + 0.29
92.29 + 0.09
1.000 + 0.000
single-IL
85.60 + 0.60
98.86 = 0.04
1.014 + 0.000
expectation-II
85.62 + 0.38
93.00 + 0.09
1.004 + 0.001
truncated-IL.
86.15 + 0.20
9..1
1.017 = 0.002
single-IL
86.23 + 0.49
94.05
1.017 = 0.001
expectation-IL
85.66 + 0.18
0.16
1.006 = 0.001
truncated-IT.
86.92 + 0.15
+.005
1.020 + 0.002
the learning rale scheduler and set. the initial learning rate 0.01. The pretrained ResNet18 and CLIP models were trained for 200 and 100 epochs, respectively, the latter accounting for the excellent performance of the CLIP model. For ResNet18 (scratch), following the suggestions in He et al. (2016), we set the initial learning rate to 0.1 and decrease it by a factor of 10 at the 160-th epoch, for a total of 200 training epochs. In our experiments, we consider basic data augmentations, including random horizontal flip, random crop, and normalization. For the imput images, the model is ResNet18 (scratch), we resize the inputs to 32 x 32, otherwise, we resize the inputs to 224 224. This choice is based on the different architectures of the models and their specific requirements for input sizes. For each selection scenario, we each algorithm three times over 9 AL selection cycles and report the mean and standard deviation of the results as the final performance of the algorithms. Source code is available at https:/ //gi thub. (aamm/x41--aa http:/Attvveeeeeeeeeee git
Evaluation Metrics In our experiments, we classification accuracy, denoted as ACC, as the primary evaluation metric for comparing the performance of the AL algorithms. Specifically, we train a model with the labeled dataset selected by each AL algorithm at each selection cycle and evaluate the classification accuracy of the model on the corresponding test dataset. To provide a more comprehensive metric of the performance of each AL algorithm over the entire selection process (9 selection cycles), we introduce summary performance metric we call AUCA Ratio representing the ratio between the area under the classification accuracy curve over all selection cycles for a model and the random selection approach. The AUCA Ratio is calculated using the trapezoidal rule (Yeh et al., 2002). Note that AUCA Ratio larger than 1 indicates that the corresponding AL algorithm achieves a consistent improvement over the random selection baseline throughout the entire selection process.
5.1 Ablation Study
The proposed RALIF consists of mainly two components. We first utilize results from Theorem 1 and the influence function formulation in (6) via (5) to select the most likely informative and useful sample candidates for labeling. We then use the k-MAA NS algorithm on the features of the selected pool of candidates to down-select a batch encouraging sample diversity. We consider three options for the first component of RALIF: single-IL in (7), truncated-II IL in (9), and truncated-II in (8). For the second component, we can choose (or not) to favor diversity by applying the &-MEANS++ If we choose not to apply k-MEANS+ NS++, we simply seleet a batch of the top-ranked B samples obtained using the ranked loss from the first component.
To investigate the contribution of different choices, we conduct ablation study scenario: using the Cifar10 dataset, the pretrained ResNet18 model architecture, and budget size of 1000. We compare performance metrics, ACC and AUCA Ratio, after the first and last selection cycles, first and ninth, respectively. Table underscores that each component. of our algorithm contributes to the selection process by prioritizing more useful samples compared to the random selection baseline. From Table 1, we also observe that, compared with single-IL and truncated-II, II, expectation-II. is less successful at selecting useful samples. This is likely to be caused by samples with small but nonzero probabilities assigned during predictions that negatively bias their influence estimation, thus undermining the selection
general selection
11

---

Published in Transactions on Machine Learning Research (11/2023)
process. Complementary, the results suggest that applying the k-MEANS++ algorithm in general improves the performance of the the AL, algorithm by alleviating information redundancy during the selection process. Overall, the combination of truncated-IL IL and +-MEANS++ achieved the best performance, so we will use this combination, simply referred to as truncated-Tl. IL for further comparisons with all AL baselines. Additionally, we have included the results of using different diversity operations Appendix A.4, providing further insights into the impact of these on the performance of our approach.
5.2 Performance Comparison with Baselines
We conduct a comprehensive comparison of the proposed truncated-II IL algorithm (with diversity selection) with other baselines in six simulated benchmark scenarios with different datasets (Cifar10, Cifar100 and iNaturalist), batch size (100, 1000 and 2000) and DL model (pretrained RestNet18, ResNet118 from scratch and pretrained CLIP). Table 2 provides summary of the results, focusing on ACC and AUCA Ratio comparisons for the first and last AI. selection cycles. For more detailed ACC comparisons across all selection cycles and boxplots summarizing the AUCA Ratio for different trials see Appendix A.8. In summary, the results presented in Table 2 clearly demonstrate that RALIF with truncaled-11, selection consistently achieved the best performance across all scenarios, highlighting its reliability and effectiveness.
More specifically, among all the basclines considered, BADGE fared as the sccond-best approach in three scenarios, followed by Grad-Norm in two scenarios, and Uncertainty in one scenario. However, it is impor- tant to note that all baselines performed worse than the Random baseline in at least one selection cycle (highlighted in red in Table 2). In contrast, RALIF with truncated-IL selection always outperformed Ran- dom. For instance, when considering the Cifar10 dataset with a budget size of 1000 and the ResNet18 (scratch) model, all baselines performed worse than Random in the first selection cycle. Notably, the Core- set algorithm exhibited worse performance than the Random baseline in two scenarios, while Uncertainty performed worse than Random in one scenario. As a summary of the experiments across different settings, we also present average metrics, from which we can see the consistency of RALIF relative to the competing approaches. These rosults highlight the reliability of RALIF compared to basclines including statc-of-the art methods. Further, to stress out the significance of the performance differences between RALIF and other approaches, we have included detailed results of statistical significance tests in Appendix A.7. Moreover, these significance results have been summarized in the bottom row of Table 2, which indicates the proportion of times that the proposed RALIF achieves statistically significant performance differences relative to other approaches across all AL scenarios, including the additional AL scenarios presented in Appendix A.5.
5.3 Qualitative results
To gain further insights into the samples selected by different AL approaches, we present visualizations of selected samples using UMAP embeddings for the uNatural dataset with AL approaches with a budget of 100 and a fixed CLIP encoder. Figure 2 shows UMAP visualizations of unlabeled and selected samples (in red) for Uncertainty, BADGE, and RALIF. We chose Uncertainty and BADGE for visualization because the former is the most commonly used AL algorithm, and the latter is the second-best AL algorithm among the baselines based on the experimental results in Table 2. In the figure, each point represents sample, different colors indicate true classes (first two rows) or entropy (third row). Note that true labels are not available to the AL methods.
From Figure 2, several observations can be made. First, compared to the other two AL algorithms, the samples selected by Uncertainty appear cluster in a smaller area consistent with high uncertainty (entropy), and with most of them closely packed together. This suggests that the Uncertainty algorithm tends to select similar samples in each iteration, leading to redundancy in the selected data and potentially undermining its performance, thus underscoring the benefits of diversity sampling. However, results for Coresct in Table 2 indicate that solely relying diversity during active learning selection cycles can lead to worse performance. Alternatively, BADGE combines uncertainty and diversity in its strategy. Examining Figure 2, we observe that the samples selected by BADGE are more diverse compared to those selected by Uncertainty. This indicates that BADGE effectively avoids selecting samples that are clustered together and rather focuses on selecting more diverse samples. From Figure 2, we also see that RALIF, similar to BADGE, successfully
12

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: Comparison with baselines in different scenarios. B: budget size at each selection eycle. Note that for RALIF we use truncated-IL for candidate selection and K- K-MEANS++ for diversity down-selection. We present the mean and standard deviation of the results for all algorithms over three trials. The best and worse than Random results highlighted in green and red, respectively. Underlined values indicate instances where the performance difference between RALIF and other approaches is not statistically signif- icant, t.e., the p-value for the significance test exceeds the 0.05 threshold. The Average summary the overall performance averaged of different algorithms across all active learning (AL) scenarios, including the additional AL. scenarios presented in Appendix A.5.
Setting
Cycles
Methods
AUCA Ratio
Datasets
B
models
First(1*t)
Last (9th)
Random
61.33 + 1.20
87.04 + 0.11
1.000 + 0.000
Uncertainty
60.27 +: 3.06
90.39 + 0.08
1.025 + 0.008
ResNet18 (seratch)
Cifar10
1000
Coreset
54.07 2.68
84.80 0.68
0.948 = 0.006
BADGE
60.21 + 0.88
89.79 0.12
1.024 = 0.006
Grad-Norm
57.90 +: 3.78
90.08 + 0.25
1,027 + 0.002
RALIF
62.05 + 0.75
90.53 + 0.13
1.041 + 0.003
Random
85.31 + 0.29
92.29 + 0.09
1.000 + 0.000
Uncertainty
85.55 +
94.29 0.24
1.066 = 0.003
Cifar10
1000
ResNet18
Coreset
85.59 = 0.11
93.04 + 0.12
1.008 = 0.002
BADGE
86.66 + 0.16
94.32 + 0.06
1.019 + 0.001
Grad-Norm
86.60 + 0.07
94.19 + 0.07
1,099 + 0.003
RALIF
86.92 = 0.15
94.35 + 0.05
1.021 = 0.002
Random
87.56 1 0.25
94.06 1 0.04
1.000 1 0.000
Uncertainty
88.64 + 0.29
95.57 + 0.03
1019 + 0.001
Cifar10
2000
ResNet18
Coreset
87.58 + 0.17
94.68 + 0.20
1.007 + 0.001
BADGE
88.88 + 0.19
95.69 + 0.16
1.018 + 0.001
Grad-Norm
88.46 + 0.11
95.70 + 0.03
1018 = 0.001
RALIF
88.96 = 0.05
95.72 0.07
1,020 = 0.001
Random
64.50 + 0.55
71.64 0.13
1.000 + 0.000
Uncertainty
64.81 + 0.13
72.41 0.25
1,011 = 0.003
Cifar100
1000
ResNet18
Coreset
64.86 = 0.10
71.67 + 0.37
1.003 = 0.001
BADGE
65.26 1 0.16
73.21 1 0.19
1.013 1 0.001
Grad-Norm
65.27 + 0.34
73.11 0.31
1.100 + 0,003
RALIF
73.26 0.08
1.015 + 0.002
Random
85.17 0.07
87.10
1.000 = 0.000
Uncertainty
85.14 0.08
$7.45
1.002 = 0.000
iNaturalist.
1000
ResNet188
Coreset
85.43 + 0.03
87.45 0.07
1.003 = 0.000
BADGE
85.53 + 0.14
88.40
1.010 = 0.001
Grad-Norm
85.56 = 0.05
88.47 0.08
1.010 = 0.001
RALIF
85.66 = 0.05
88.63 0.12
1.012 = 0.001
Random
90.85 1 0.90
93.98 1 0.18
1.000 1 0.000
Uncertainty
89.13 + 1.00
93.72 0.20
0.994 = 0.007
iNaturalist
100
CLIP
Coreset
89.64 + 0.90
93.44 0.18
0.990 + 0.004
BADGE
92.66 = 0.32
95.17 + 0.08
1015 = 0.004
Grad-Norm
92.71 + 0.38
95.05 + 0.02
1.05 = 0.004
RALIF
92.82 + 0.26
95.17 + 0.01
1.015 + 0.002
Random
81.67 + 10.29
89.58 + 6.92
1,000 = 0.000
Uncertainty
81.71 = 10.58
91.02 + 7.11
1012 = 0.010
Average
Coreset
80.76 +: 11.93
89.67 1 7.29
0.996 1 0.020
BADGE
82.62 +: 10.94
91.30 1 6.90
1.016 1 0.005
Grad-Norm
82.26 + 11.50
91.30 + 6.92
1.016 + 0.005
RALIF
83.05 + 10.66
91.53 + 6.93
1.020 + 0.008
RALIF is statistically significant.
33/45(73.33%)
19 19/45(42.22%) 34/45(75.56%)
13

---

Published in Transactions on Machine Learning Research (11/2023)
Uncertainty
BADGE
RALIF
First Cycle (cmap- temap-classes classes)
Last (emag--
Last Cycle (cmap-Entrop) Entropy
Figure 2: UMAP embeddings for the first and last selection cycles of different active learning approaches with a fixed CLIP encoder on the iNaturalist data. The first and second rows correspond to the first and last selection cycle, respectively, and with colors indicating true classes for each data point. The bottom row corresponds to the last cycle with colors denoting the normalized entropy for each data point. From left right, the visualizations are for Uncertainty, BADGE, and RALIF, Red dots highlight samples selected for labeling by each approach.
avoids selecting samples that cluster together and selects more diverse samples than Uncertainty. Notably, RALIF seems select samples more sparsely distributed relative BADGE, underscoring the effectiveness of the trancated-IL selection approach, by minding that both BADGE and RALIF the same diversity strategy. Further, comparing sample selections for the first. and last cycle, we see that Uncertainty naturally favors tightly packed samples with high entropy, BADGE consistently selects sparsely distributed samples, whereas RALIF initially favors more tightly packed samples and then (as the model improves) starts favoring more diverse samples, thus differentiating itself from both Uncertainty and BADGE.
6 Conclusion
In this paper, we proposed new AL framework for DL architectures called RALIF, which is motivated by thcoretical observations on the complete-data loss of classification model. We established that samples minimizing the model loss on the unlabeled dataset with generaled pseudo labels bound from above the loss on the entire dataset with true labels. To address the challenges associated with evaluating the impact of candidate samples on the loss of the model on the unlabeled datasct, we introduce three efficient selection strategies, namely, single-IL, expectation- IL and truncated-IL. IL. These strategies allow the quantification of the contribution of candidate samples to the loss the entire dataset without the need for explicit model
14

---

Published in Transactions on Machine Learning Research (11/2023)
retraining, Further, we incorporated a diversity prioritization based on K-MEANS++ similar to BADGE (Ash et al., 2019), to improve the diversity of selected samples when performing batch selection.
We evaluated the reliability and effectiveness of RALIF through comprehensive experiments across various scenarios. The experimental results clearly demonstrated that RALIF consistently outperformed the random baseline and other DI.-based AT algorithms terms of classification performance. Moreover, we provided qualitative results that highlighted RALIF's ability to trade-off high-uncertainty and diversity during the active selection process.
One limitation of RALIF is that the quality of the pseudo labels generated from the model M can affect its performance. There are two main reasons for this behavior. First, RALIF utilizes the model and its predictions as an approximation to the conditional distribution Prix in Theorem A.1. If the quality of the model and thus its pseudo labels is poor, the total variation difference (rv(::) between the estimated label conditional distribution pyjx and the true label conditional distribution Prix will be large. This can result in a loose bound in Theorem A.1, rendering it less effective for AL. Second, RALIF uses pseudo labels to approximate the influence function in (7). If the quality of the pseudo labels is low, the approximated influence loss for each candidate will be inaccurate. This can affect the selection process and potentially lead to suboptimal results, However, it is important to note that this limitation is not exclusive to RALIF but also applies to other AL. baselines since they also dependent on the quality and performance of the model at each selection cycle. For example, when considering Cifar10 with a budget size of 1000 and the 18 model trained from scratch, the results in Table 2 demonstrate that all other AL baselines perform worse than the Random baseline in the first cycle, when the initial model has poor performance. Another limitation of RALIF is that it is currently designed specifically for classification tasks, thus it may not be directly applicable to other types of tasks, such as object detection natural language processing problems. Consequently, further research and adaptation would be needed to extend RALIF to such tasks.
In future work, we plan to extend RALIF to real-world datasets that exhibit specific label structures, such as datasets with hierarchically organized labels. These datasets pose unique challenges for AL algorithms, as the label hierarchy introduces additional dependencies and relationships among the samples. Moreover, it will be interesting to explore how RALIF can be used address related tasks such as few-shot learning and test-time domain generalization.
Acknowledgements
The authors would like to thank the editor and the anonymous reviewers for their insightful comments. This research was supported by NIH/NINDS 1R61NS120246, NIH/NIDDK R01- DK123062, and ONR N00014- 18-1-2871-PP0000-3..
References
David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-STAM symposium on Discrete algorithms, pp. 1027-1035, 2007.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXi::1906.0300 2019.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Klaus Brinker. Incorporating diversity in active learning with support vector machines. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 59 59-66, 2003.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. ar XiXiv preprint ar arXiv:200000670 2022.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprinl arXiv:2302.0667 2023.
David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129 145, 1996.
Ido Dagan and Sean P Engelson. Committoc-based sampling for training probabilistic classifiers. In Machine Learning Proceedings 1995, pp. 150 157. Elsevier, 1995.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPRO9, 2009.
Mário Figuciredo. Adaptive sparseness using jeffreys prior. Advances in neural information processing systems, 14, 2001.
Mário AT Figuciredo.
Adaptive sparseness for supervised learning. IEEE transactions on pattern analysis and machine intelligence, 25(9):1150-11) 1159, 2003.
Mário AT Figueiredo and Anil K Jain. Bayesian learning of sparse classifiers. In Proceedings of the 2001 IEFE Computer Sociely Conference on Computer Vision and Paltern Recogrition. CVPR 2001, volume 1, pp. 1-1. IEEE, 2001.
Alexander Freytag, Erik Rodner, and Joachim Denzler. Selecting influential examples: Active learning with expected model output changes. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Swilzerland,, Seplember 6-12, 2014, Proceedings, Part IV 13, pp. 562 577. Springer, 2014.
Keinosuke Fukunaga and Larry Hostetler. The estimation of the gradient of a density function, with appli- cations in pattern recognition. IEFE Transactions on information theory, 21(1):32-M0 40, 1975.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp- 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In PMLR, 2017.
International conference on machine learning, pp.
Yonatan Geifman and Ran El- Eaniv... Deep active learning over the long tail. arXio preprint urXiv:1711.00044 2017.
Mark Greenwood. Intermediate Statistics with R. 2014.
Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, (9(36))3383 393, 1974.
Jingrui He, Hanghang Tong. Mingjing Li, Hong-Jiang Zhang, and Changshui Zhang. Mean version space: active learning method for content-based image retrieval. In Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, pp. 15-22, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Marti A. Hearst, Susan Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-2) 28, 1998.
Ralf Herbrich, Thore Graepel, and Colin Campbell. Bayes point machines. Journal of Machine Learning Research, 1(Aug):245 279, 2001.
Rebecca Hwa. Sample selection for statistical parsing. Computational linguistics, 30(3):253 2202266,, 2004.
Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. ar preprint ar 1.06220, 2022.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
David G Kleinbaum, K Dietz, M Gail, Mitchel Klein, and Mitchell Klein. Logistic regression. Springer, 2002.
Pang Wei Koh and Percy Liang. Understanding black-bor predictions via inffuence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.
David Lewis and William A Gale. sequential algorithm for training text classifiers. In SIGIR '94, pp- 3- 12. Springer, 1994.
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp- 9274 9283, 2021.
Nya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03900 2016.
Edwin Lughofer. Extensions of vector quantization for incremental clustering. Pottern recognition, 41(3): 995 091- 2008
Edwin Lughofer. Hybrid active learning for reducing the annotation effort of operators in classification systems. Pallern Recognition, 45(2):884 896, 2012.
MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pp. 281-297, University of California Los Angeles LA USA, 1967.
Andrew McCallum, Kamal Nigam, et al. Employing em and pool-based active learning for text classification. In ICML, volume 98, pp. 350-558. Citeseer, 1998.
Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. In Proceedings of the TEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 223 232, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Lawrence Rabiner. tutorial on hidden markov models and selected applications in speech rocognition. Proceedings of the IEEE, 77(2):257 286, 1989.
Alcc Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8763. PMLR, 2021.
Irina Rish et al. An empirical study of the naive bayes classifier. In IJCAI 2001 workshop on empirical methods in artificial intelligence, volume 3, pp. 41 46, 2001.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A corc-set approach. arXiv preprinl Xr 00089, 2017.
Burr Settles. Active learning literature survey. 2009.
Burr Settles and Mark Craven. An analysis of active learning strategies for soquence labeling tasks. In proceedings of the 2008 conference on empirical methods in natural language processing, pp. 1070-1079, 2008.
17

---

Published in Transactions on Machine Learning Research (11/2023)
H. Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Compulational learning theory, pp. 287-294, 1992.
Claude E Shannon. A mathematical theory communication. The Bell system technical journal, (7):: 379-423, 1948.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xr 1556, 2014.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 0972-5981, 2019.
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky; Nya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Πya Sutskover, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentumm deep learning. In International conference on machine learning, pp. 1139 1147. PMLR, 2013.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classifica- tion. Journal of machine learning research, 2(Nov)):45-0) 66, 2001.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Vladimir Vapnik. The nature of statistical learning theory. Springer science business media, 1999.
Huy V Vo, Oriane Siméoni, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Jean Ponce. Active learning strategies for seakly-supervised object detection. In European Conference Computer Vision, pp. 211 230. Springer, 2022.
Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference on neural networks (IJCNN) pp. 112-119. IEFE, 2014.
Tianyang Wang, Xingjian Li, Pengkun Yang, Guosheng Hu, Xiangrui Zeng, Siyu Huang, Cheng-Zhong Xu, and Min Xu. Boosting active learning via improving test performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8566-8574, 2022.
Gert W Wolf. Facility location: concepts, models, algorithms and case studies. series: Contributions management science: edited by zanjirani farahani, reza and hekmatfar, masoud, heidelberg, germany, physica-verlag, 2009, 549 E 171.15, $219.00,f. 144.00, isbn 978-3-7908-2150-5 (hardprint), 978-3-7908- 2151-2 (electronic), 2011.
Xing Wu, Cheng Chen, Mingyu Zhong, and Jianjia Wang. Hal: Hybrid active learning for efficient labeling in medical domain. Neurocomputing, 456:563 -72,6,3-77,,, 2021.
Shi-Tao Yeh et al. Using trapezoidal rule for the area under a curve calculation. Proceedings of the 27th Annual SASD User Group International (SUG)02), pp. 1-5, 2002.
John Seon Keun Yi, Minseok Seo, Jongchan Park, and Dong- Geol Choi. Pt4al: Using self-supervised pretext tasks for active learning. In European Conference on Computer Vision, pp. 696-612. Springer, 2022.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 93 102, 2019.
18

---

Published in Transactions on Machine Learning Research (11/2023)
Tong Zhang and F Oles. The value of unlabeled data for classification problems. In Proceedings of the Seventeenth International Conference on Machine Learning, (Tangley, P., ed.), volume 20, pp. 0. Citeseer, 2000.
A Appendix
A.1
Proof and explanation of Theorem
Proof. Let Px, PxPr|x- Px, = PxPrix:
Veer (px,r (2, (p
E P(o.))~xPV/x (xxx)~xx (2,
Px, (z, ()) uddrdy
I Xxx Ixx, - px, w)1 [( v) dxdy
(11)
L Sxxr (xx,((, - px, w)1 dody 2L Drv(pxPrxxxPxPe
In practice, the upper bound L of the loss functions can usually be established given that the network inputs are usually bounded and the network weights regularized during training. An example is the cross-entropy loss function used for classification. Denote the (before softmax activation) network output as x E R² with elements Ik: i.e., x = (I*)i and the ground truth label vector as E [0, 1] with elements yk: i.e. y (wn) and Ek=1 Uh =. The cross-entropy loss is defined as:
(CE(x.y) Σ yb log Pk:
(12)
where is the post-softmax prediction score for class k, i.e.,
exp(112)
Pk 2 exp(xz)
(13)
We can show that
(CE(X,y)
K
exp(xx)
K Σ log
Σ/ exp(ary)
Σ k=
log
|x||2
|x|12
Σ == exp(2;;)
||x||2 R=
exp(x*)
K
K eexp
max Iz Isisk
K . exp
max Kj 15jSK
yk log
Mk log
[
exp(x1)
|X|2
exp
max Ijj 15JSK
(14)
exp
max Ijj 1
1
log K + log
|x|2
exp
max T) 1SJSK
1
log K + max Iz
|x 2|
1SiSK"
log K
+
|
from which we can see that the magnitude of Lee bounded by a constant multiple of the 2-norm of When | - +00, although the exact form of how the norm of x is bounded by the model weights depend many network details, e.g., network architecture, activation functions.
19

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 2 Ralif Framework including all selecting approaches.
Input: Lo: Initial labeled dataset, Uo: initial unlabeled dataset, M: model, C: AL cycles, B: query set, P: candidate set, PM (-)= convolutional encoder, and т: truncation hyperparameter.
1: for c=1,..., C do
Train model MI.1 with labels set Le-1
3: IPI = =01
4: Vx; ~ Uc generate pseudo-labels ÿ = arg max ML.-1 (2:)
if single-IT. then
for Hg E U--1 do
Generate pseudo-label Us arg max M (2ia) Calculate D. (2,, dis ML.)] via (7)
end for end if
10:
11:
expectation-IL then
12:
for Ig E Up-1 do
13:
Generate probabilistic prodictions (x,))) from (
14:
Calculate & (wi, Des Mr.)) via (8)
15:
end for end if
16:
17:
if
truncated-I IL then for F, E Uc-1 do Generate probabilistic predictions (x.)), from Mi Calculate (aa (2, Ue ML.) ) via (9)
18:
19:
20:
21:
end for end if
22:
23: Select lop-IP| samples (aihep with minimum Ble,~Ue.Berrynn Ma, des M
24:
Run A-MEANS++ on {ome.-, (Iz))ir to select [B] diverse samples
25:
Le L (2),,
26:
end for
Return: Final labeled dataset Le (of size Lo + C)B))
A.2 Complete RALIF Algorithm
The complete Algorithm 2 including all three selecting approaches, iee, single-IL in (7), expectation-Il in (8) and truncated-IL in (9).
A.3 Different thresholds
To investigate the impact of different values of on the performance of RALIF, we conducted experiments on a specific selection scenario. The experiments were performed using the Cifar10 dataset, pretrained ResNet18 model architecture, and a budget size 1000. We varied the value of T across five different values: K 0, 0.2, K 0.4, 0.6, and 0.8. We then compared the resulting ACC and AUCA Ratio the first and last selection cycle. The results summariied in Table 3. It is interesting to note that regardless of the specific value of RALIF consistently outperforms the random baseline. Further, based the results shown in Table 3, observe that & yields the best performance among the tested values.
A.4 Diversity operation
Below we consider the same AL scenario as previously detailed in Section 5.1, i.e., using the Cifar10 dataset, the pretrained ResNet18 model architecture, and a budget size of 1000. In the experiments, we replace the k- A-MEANS++ algorithm RALIF with the mean shift algorithm Fukunaga & Hostetler (1975). Notably, the mean shift algorithm circumvents the need for an explicit specification of the number of clusters. Thus, first apply the mean shift. algorithm to the selected candidate pool and then select the samples closest
20

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: Comparison with different thresholds for the indicator function x(-) (9). We show means and standard deviations of the results for all algorithms over three trials. The first row provides results for the random baseline, which serves a reference for comparisons. The best results are highlighted in green.
Cycle
AUCA Ratio
First (1*t)
Last (gth)
Random
85.31 + 0.29
92.29 + 0.09
1.000 + 0.000
0.0
85.66 + 0.18
92.92 + 0.16
1.007 + 0.001
0.2
86.60 + 0.10
94.24 0.04
1.019 + 0.002
0.4
94.35 = 0.05
1.021 = 0.002
0.6
86.59 0.16
94.18 0.03
1.019 + 0.002
0.8
86.63 0.26
94.03 0.05
1.018 + 0.002
Table 4: Diversity operations comparison results. We present the mean and standard deviation of the
performance metrics for different diversity selection over three trials. Best results are highlighted in green. Diversity mean shift R-MEANS++
Cycle
Candidate Selection
AUCA Ratio
First (1st
Last (9th)
Random
85.31 + 0.29
92.29 = 0.09
1.000 + 0.000
trumcated-11
86.41 + 0.23
94.16 + 0.16
1.019 + 0.002
truncated-I1
86.92 + 0.15
94.35 = 0.05
1.020 + 0.002
to the centroid of each cluster, where the number of samples chosen from each cluster is determined by the proportion of samples in that cluster to the total number of candidate samples. Results presented in Table 4 demonstrate that both the mean shift algorithm and the 6-MEANS+ algorithm can enhance RALIF as a diversity operation. However, it is worth noting that the A-MEANS +++ algorithm yields better performance.
A.5 Additional AL scenarios
In Table 5, we present the results of additional experiments conducted under different active learning scenarios relative to those in Table 2. In these experiments, we consider ResNet18 (scratch), identical the one introduced in See 5, alongside two other model architectures: VGG16 Simonyan & Zisserman (2014) and EfficientNet-BO Tan & Le (2019), both of which are initialized with pretrained weights from ImageNet Deng et al. (2009). Additionally, analysis, we also provide a comprehensive view of the performance comparisons across all selection cycles and present boxplots summarizing the AUCA Ratio in Figure 3.
use different dataset, SVHN Netzer et al. (2011). For more detailed
A.6 Additional Comparison with Influence Selection for Active Learning (ISAL)
To compare the proposed RALIF with ISAL,, as presented Liu et al. (2021), we consider the active learning scenario described in Table 3 of Liu et al. (2021). In this scenario, the dataset is Cifar10, the initial labeled dataset size is 1000, and the active learning budget set 1000 samples. The model used here and the training details follow the Target Model specifications outlined in Liu et al. (2021). The results of the comparison between the proposed RALIF and ISAL are presented in Table 6. More details about the difference between ISAL, ISAL v2 and ISAL v3 can be found in The Selection of Reference Set Section in Liu et al. (2021). From the results, we see that in the same active learning scenario, the proposed RALIF performs better than ISAL, regardless of the selection of reference set used in ISAL.
A.7 Significance Tests
To quantify the significance of the performance differences between RALIF and other approaches, we employ two-sided permutation test (Greenwood, 2014). Specifically, we start with predictions for the test dataset
21

---

Published in Transactions on Machine Learning Research (11/2023)
Table 5: Comparison with baselines in different scenarios. B: budget size each selection cycle. Note that for RALIF we use truncated-IT. for candidate selection and K-MEANS++ for diversity down-sclection. We present the mean and standard deviation of the results for all algorithus over three trials. The best and worse than Random results are highlighted in green and red, respectively. Underlined values indicate instances where the performance differences between RALIF and other approaches is not statistically significant, i.c.,
the p-value for the significance test exceeds the 0.05 threshold.
Setting
Cycles
Methods
AUCA Ratio
Datasets
B
models
First(1*()
Last (9th)
Random
86.02 + 0.311
93.74 + 0.263
1.000 + 0.000
Uncertainty
85.47 + 2.049
95.58 + 0.138
1.015 + 0.001
ResNet18 (scratch)
SVHN
1000
Coreset
86.29 + 1.245
95.24 0.110
1.014 = 0.003
BADGE
87.55 0.701
95.38 + 0.148
1.018 + 0.003
Grad-Norm
87.31 + 0.965
95.55 + 0.172
1.018 + 0.004
RALIF
88.14 0.679
95.77 0.143
1.023 = 0.003
Random
84.26 + 0.665
91.33 = 0.248
1.000 + 0.000
Uncertainty
85.22 + 0.335
93.15 = 0.219
1.014 0.001
Cifar10
1000
VGG16
Coreset
82.58 + 1.016
90.49 = 0.194
0.984 + 0.001
BADGE
85.61 + 0.311
93.15 + 0.143
1.016 + 0.002
Grad-Norm
85.19 + 0.120
92.98 1 0.255
1.015 1 0.002
RALIF
86.16 0.162
93.47 = 0.167
1.020 + 0.002
Random
90.06 1 0.238
95.02 1 0.236
1.000 T 0.000
Uncertainty
91.20 + 0.127
96.58 + 0.039
1.015 + 0.001
Cifar10
1000
FfficientNNet-A
Coreset
90.78 + 0.210
96.18 0.101
1.010 + 0.002
BADGE
91.23 + 0.074
96.56 0.014
1.015 + 0.003
Grad-Norm
91.31 + 0.047
06.52
1.015 = 0.002
RALIF
91.61 + 0.172
96.88 0.019
1.018 = 0.003
the Cifar10 dataset, utilizing the results for ISAL. obtained from Table 9 in Liu al. (2021). For RALIF, we conducted the experiments three times for robustness and reported the performance mean, following Liu et al. (2021).
Table 6: We compared RALIF and ISAL
Method
1000
3000
5000
7000
9000
ISAL
45.52
67.72
81.24
85.96
89.26
ISAL v2
45.52
67.06
80.57
85.71
88.92
ISAL v3
45.52
67.12
80.11
84.88
88.71
RALIF
45.25
74.62
82.77
87.32
89.89
generated by different approaches. The test statistic, denoted as Tahs; represents the accuracy AUCA Ratio differences between RALIF and other approaches, calculated using true test dataset labels. To obtain permuted results, denoted as T* we perform random permutations of the true labels of the test dataset 10, 000 times. Each time, we calculate the accuracy or AUCA Ratio differences between RALIF and other approaches using the generated permuted test. dataset labels. Subsequently, we define the p-value as the proportion of times the absolute value of T greater than or equal the absolute value of Tobs- A smaller p-value indicates reduced likelihood that the observed accuracy or AUCA Ratio differences between RALIF and other approaches are due to random chance. In each active learning scenario, conducted permutation tests three times (one per trial) and to be conservative, we only report the largest p-value (worst case) for each scenario in Table 7. The p-values in Table 7 demonstrate that RALIF's performance is statistically superior to that of other mothods, further emphasizing the uniqueness of RALIF as an active learning algorithm with a strong emphasis on reliability. Additionally, for a concise overview of the frequency with which RALIF's performance is statistically significant, i.e., p-values are smaller than 0.05 (the significance level threshold), compared to other methods, we provide summary in Table
22

---

Published in Transactions on Machine Learning Research (11/2023)
Table We conducted permutation test three times (one per trial) for each setting and only report the largest p-value observed for each sctting. To improve readability, we have highlighted cases where the difference between RALIF and other approaches statistically significant. in green, t.e., where the p-value less than the 0.05 threshold.
Setting
Methods
First(1*))
Last (9th)
AUCA Ratio
Dalasets
models
Random
0.000
0.000
0.000
Uncertainty
0.000
0.858
0,965
ResNet18 (scratch)
Cifar10
1000
Coreset
0.000
0.000
0.000
BADGE
0.000
0.003
0.022
Grad-Norm
0.000
0.000
0.000
Random
0.000
0.000
0.003
Uncertainly
0.000
0.099
0.611
Cifar10
1000
ResNel18
Coreset
0.000
0.000
0.032
BADGE
0.434
0.692
0.987
Grad-Norm
0.664
0.756
0.645
Random
0.000
0.000
0.001
Uncertainty
0.695
0.000
0.667
Cifar10
2000
ResNet18
Coreset
0.000
0.000
0.000
BADGE
1.000
0.783
0.843
Grad-Norm
0.009
0.189
0.763
Random
0.020
0.000
0.608
Uncertainty
0.051
0.000
0.842
Cifar100
1000
ResNet18
Coreset
0.000
0.000
0.673
BADGE
0.127
0.740
0.907
Grad-Norm
0.948
0.000
0.838
Random
0.000
0.000
0.000
Uncertainty
0.000
0.000
0.000
iNaturalist
1000
ResNet18
Coreset
0.000
0.000
0.000
BADGE
0.000
0.001
1.000
Grad-Norm
0.053
0.000
0.366
Random
0.000
0.000
0.000
Uncertainty
0.000
0.000
0.000
iNaturalist
100
CLIP
Coreset
0.000
0.000
0.000
BADGE
0.000
0.216
1.000
Grad-Norm
0.000
0,013
0.362
Random
0.000
0.000
0.000
Uncertainty
0.000
0.924
0.050
SVHN
100
ResNet18
Coreset
0.000
0.000
0.011
BADGE
0.178
0.000
0.271
Grad-Norm
0.000
0.002
0.352
Random
0.000
0.000
0.009
Uncertainty
0.000
0.105
0.388
Cifar10
100
VGG16
Corcset
0.000
0.000
0,000
BADGE
0.022
0.064
0.657
Grad-Norm
0.000
0.000
0.673
Random
0.000
0.000
0.010
Uncertainty
0.096
0.002
0.648
Cifar10
100
EfficientNet-B0
Coreset
0.000
0.000
0.091
BADCE
0.152
0.000
0.495
Grad-Norm
0.328
0.001
0.537
23

---

Published in Transactions on Machine Learning Research (11/2023)
A.8
Detail results comparison
For a more detailed comparison wet the baselines and RALIF, we present the ACC across all AL selection cycles and AUCA Ratio boxplots for all baselines and RALIF under all six AL scenarios in Figure 3.
1000; models: ResNet18|scratch)
1.0275
0.950
1.0250
0.925
0.900
1.0225
0.875
AUICA Ratio
1.0200
0.850
0.825
1.0150
Uncertainty
Coreset
0.775
BADGE
1.0125
RALIF
Uncertainty
Coreset
BADGE
Grad-Norm RALIF
Datasets: Cifar10; B: 1000; models: VGG16
0.925
1.020
1.015
0.900
1.010
AUCA Ratio
0.850
1.000
0.995
Random
Uncertainty
0.800
Coreset
0.990
0.775
Gred-Norm
0.985
0.980
Uncortainty
Coreset
BADGE
Grad-Nom
RALIIF
Datasets: Cifar10; B: 1000; models: Efficient_B0
1.020
0.96
0.94
1.016
ACC
0.92
1.014
AUCA
1.012
0.90
Uncertainty
1.010
0.88
BADGE
Grad-Norm
0.86
RALIF
1.006
Uncertainty Coreset
BADGE
Grad-Norm
Cycles
Figure 3: Detailed AL results. Each row corresponds to AL scenario, following the same order in
Table 2. The title for each row provides description of the AT scenario, including the Dataset, B (budget size at each selection cycle), and the model used. Standard deviations are omitted in the ACC plots for clarity, and AUCA Ratio boxplots summarize results over the three trials.
24

---

Published in Transactions on Machine Learning Research (11/2023)
Datasets: Cifar10; B: 1000; models: ResNet1 B(scratch)))
1.02
AUCA Ratio
1.00
ACC
Random
Uncertainty
Coraset
0.96
50
BADGE
Grad-Norm
0.90
Uncertainty
Grad-Norm
Cycles
Datasets: Cifar10; B: 1000; madels: ResNet18
1.0200
AUCA Rato
ACC
1.0150
Randor
Uncertainty
1.0100
Coreset
BADGE
82
Grad-Norm
1.0075
Uncartainty Coreset
Cycles
Datasets: Cifar10; B: 2000; models: ResNet18
1.018
1.016
Ratio
1.044
ACC
1.012
Random
84
Uncertainty
Coreset
BADGE
Grad-Norm
RALIF
1.006
Uncortainty
BADGE
Cycles
Figure 3: Detailed AL results. Each row corresponds to AL scenario, following the same order in
Table 2. The title for cach row provides a description of the AL scenario, including the Dataset, B (budget size at each selection cycle), and the model used. Standard deviations are omitted in the ACC plots for clarity, and AUCA Ratio boxplots summarize results over the three trials.
25

---

Published in Transactions on Machine Learning Research (11/2023)
Datasets: Cifar100; B: 1000; models: ResNet18
Uncertainty
1.016
BADGE
RAUF
1.012
AUCA Razo
1.010
ACC
1.008
1.006
1.004
1,002
Uncertainty
Coresat
BADGE
Grad-Norm
Cycles
Datasets: INaturalist; 1000; models: ResNet18
88.5 -
Uncertainty
Random
1.012
Coreset BADGE
88.0
RALIF
AUCA Ratoo
1.008
86.5
1.006
86.0
85.0
1.002
Uncertainty
Cycles
Datasets: |Naturalist; B: 1000; models: CLIP
1.020
1.015
1,010
AUCA Ratio
ACC
91
1.000
Random Uncertainty
0.995
Coreset
0.990
BADGE
Grad-Norm RALIF
Uncortainty
BADGE
Cycles
Figure 3: Detailed AL results. Each row corresponds to AL scenario, following the same order in
Table 2. The title for cach row provides a description of the AL scenario, including the Dataset, B (budget size at each selection cycle), and the model used. Standard deviations are omitted in the ACC plots for clarity, and AUCA Ratio boxplots summarize results over the three trials.
26


Reviews:
[{"review_id": "DY4snyYvfv", "reviewer": "Reviewer_RLtK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While this work introduces a theoretically motivated active learning framework based on estimating sample value through influence functions, pseudo-labels, and diversity selection, a significant issue in this paper is the absence of a discussion regarding the state-of-the-art active learning methods and how this work differs from them. The lack of a comparison against existing active learning methods leaves us uncertain about where this paper stands in the literature and its relative contributions.\n\nThe primary novelty of this paper revolves around the selection of the most valuable samples for labeling using influence functions. However, it's worth noting that similar approaches have been proposed in the past, like Liu, Zhuoming, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. \"Influence selection for active learning.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9274-9283. 2021. What is the major difference between the proposed influence function and that of the referenced paper? It would be valuable for the authors to clarify the distinctions and provide an experimental comparison if applicable.\n\nA bunch of literature is missing, what is the major difference between the propose method with those works?\n\nLiu, Zhuoming, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. \"Influence selection for active learning.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9274-9283. 2021.\n\nYi, John Seon Keun, Minseok Seo, Jongchan Park, and Dong-Geol Choi. \"Pt4al: Using self-supervised pretext tasks for active learning.\" In European Conference on Computer Vision, pp. 596-612. Cham: Springer Nature Switzerland, 2022.\n\nVo, Huy V., Oriane Siméoni, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Jean Ponce. \"Active learning strategies for weakly-supervised object detection.\" In European Conference on Computer Vision, pp. 211-230. Cham: Springer Nature Switzerland, 2022.\n\nYoo, Donggeun, and In So Kweon. \"Learning loss for active learning.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102. 2019.\n\nFreytag, Alexander, Erik Rodner, and Joachim Denzler. \"Selecting influential examples: Active learning with expected model output changes.\" In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, pp. 562-577. Springer International Publishing, 2014.\n\nNo experimental comparison with the state of the arts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a theoretically motivated active learning framework that relies on estimating sample value through influence functions, pseudo-labels, and diversity selection. The authors provide comprehensive theoretical proofs and analyses for the proposed method, and they conducted extensive experiments to demonstrate its effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add comparison against state of the arts.\n2. Illustrate the difference of this work from other similar works.", "broader_impact_concerns": "NO", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WhEyQeStIz", "reviewer": "Reviewer_Frwm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is well-written and presented in a clear and understandable manner, making it easy for readers to follow the content.\n\n2. The concept is presented in a straightforward manner with a sound derivation.\n\n\nWeaknesses:\n\n1. The enhancements, when compared to other methods, show only marginal gains. Refer to the average performance presented in Table 2.\n\n2. Given the similarities between semi-supervised learning (SSL) and active learning, it would be pertinent to evaluate RALIF within a more sophisticated SSL framework. Potential SSL algorithms to consider are outlined in [1]. For RALIF to be deemed valuable, it should ideally outperform the SSL benchmarks presented in [1]; otherwise, its utility comes into question.\n\n3. The study lacks an ablation analysis concerning the substitution of K-means++ with alternative clustering methods, such as mean shift or DBSCAN.\n\n[1] USB: A Unified Semi-supervised Learning Benchmark for Classification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel active learning framework, Reliable Active Learning via Influence Functions (RALIF) which utilizes influence functions, offering a theoretically sound selection mechanism. This mechanism enhances the likelihood of choosing samples that, when labeled and incorporated into the training dataset, will boost model performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V4E61Dfa5E", "reviewer": "Reviewer_n6fZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is theoretically solid and have improved performance.\n\nWeaknesses:\n1. limited experiments: most experiments are conduct on a relative small model: ResNet18. It would be better if the author try more models and datasets.\n2. Since IL is the most important part of the algorithm, the paper lack discussion on IL.\n3. It would be better if Figure 1 can add more text in the arrow lines to illustrate how the pipeline works.\n4. The pipeline seems simple and effective. I would like to see more ablation study about ranking score like IL to see the difference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of insufficient labeled data in deep learning models by introducing a theoretically motivated active learning framework called \"Reliable Active Learning via Influence Functions\" (RALIF). RALIF consistently outperforms random selection and existing active learning methods by efficiently identifying valuable data samples that improve model performance across the entire dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The detailed changes are listed on the previous block.", "broader_impact_concerns": "This paper provide a good pipeline for active learning. The following research can develop more effective estimation functions to improve this pipeline.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b7pdOKnto4", "reviewer": "Reviewer_Qj5H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n1. The paper is well-written and presented. The related work, the proposed method and the empirical results are presented in a clear and logical way. \n2. The proposed method is theoretical motivated and can be computed efficiently without retraining the model when selecting the unlabeled samples.\n\n- Weaknesses:\n1. The classification accuracy of the proposed method may not be very much advantageous when compared with baselines, e.g., BADGE and Grad-Norm, in most settings in Table 2. It seems that the proposed method is also based on the gradient information (i.e., influence function) which is quite similar with the baselines.\n2. It may be hard to quickly check the performance comparison of methods from the first glance in Table 2 due to the lack of tests for significance. \n3. The performance of the proposed method may be affected if the number of initial labeled data is too small to train a good enough model that could produce reliable pseudo-labels at the beginning of cycles.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The paper proposed a theoretical motivated method for active learning, i.e., using initial labeled data to pretrain a model for generating pseudo labels of unlabeled data; then select unlabeled data based on influence functions to acquire oracle labels. \n2. The proposed method is evaluated on CIFAR-10, CIFAR-100 and iNaturalist and compared with representative baselines in active learning. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explain the similarity & difference between the proposed method and gradient based baselines, e.g., BADGE and Grad-Norm.\n2. Add significance test for results in Table 2 and highlight the best and comparable methods.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A 'minor revision' for this paper is recommended due to several remaining concerns. While the proposed RALIF method consistently shows improvements, the reviewers found these gains to be relatively marginal. Additionally, the choice of baselines may not be the latest. The authors included a discussion with the related work "Influence Selection for Active Learning" by Liu et al. (2021) in the paper, but there is a lack of further in-depth empirical discussion and analysis. It's important to note that Liu et al. (2021) clarified in their paper that they "chose the validation set V created in the first step of active learning as the reference set, since this would not cause additional annotation," and they also included relevant studies in their experiments. The authors thus need to further clarify the distinction between the author's new work and existing works (Liu et al. 2021) and may consider including relevant empirical comparisons or discussion to emphasize the advantages of the proposed new algorithm.
Decision: Accept (poster)