Salutary Labeling with Zero Human Annotation

## Abstract

Abstract Active learning strategically selects informative unlabeled data points and queries their ground truth labels for model training. The prevailing assumption underlying this machine learning paradigm is that acquiring these ground truth labels will optimally enhance model performance. However, this assumption may not always hold true or maximize learning capacity, particularly considering the costly labor annotations required for ground truth labels. In contrast to traditional ground truth labeling, this paper proposes salutary labeling, which automatically assigns the most beneficial labels to the most informative samples without human annotation. Specifically, we utilize the influence function, a tool for estimating sample influence, to select newly added samples and assign their salutary labels by choosing the category that maximizes their positive influence. This process eliminates the need for human annotation. Extensive experiments conducted on nine benchmark datasets demonstrate the superior performance of our salutary labeling approach over traditional active learning strategies. Additionally, we provide several in-depth explorations and practical applications of large language model (LLM) fine-tuning.

## 1 Introduction

Active learning [, , ] is a specialized area in machine learning that focuses on effectively training models by enabling them to request the labeling of particularly informative data points with a certain budget. This approach arises from the challenge and expense involved in obtaining labeled data, which is often a major bottleneck in machine learning applications. The principle behind active learning is that a machine learning model can achieve higher accuracy with fewer ground truth labels if it is allowed to choose the data from which it learns. This makes active learning particularly valuable in fields where the labeling process is costly and time-consuming.

Consequently, significant research efforts have been dedicated to active learning in various research areas such as computer vision [, ], natural language processing [, ], and medical diagnosis [, ].
Traditionally, active learning works select data points based on uncertainty and representativeness. The early uncertainty-based methods mainly measure the data uncertainty with the posterior probability predicted by the model [, , ], while some recent works utilize auxiliary modules [, ] to estimate uncertainty. Solely focusing on the uncertainty might cause bias in sampling, therefore other methods [, ] aim to find the most representative subset of the full data. Recently, some works [, ] attempt to estimate the effect of integrating each data point on the training loss with the influence function [, ].

The above active learning approaches show promising results but hinge on a critical assumption that training with ground truth labels of the selected samples will optimally enhance model performance. However, this assumption may not always hold, as some human-annotated labels can be incorrect or misleading, potentially harming the model’s efficacy [, ]. Moreover, even the correct annotation might still harm or limit the model performance.
Therefore, we believe that the most valuable annotations may not always be the ground truth but the labels that most improve the model.
This perspective underscores the need for a strategy that not only identifies the most informative data points but also accurately determines the labels that offer the greatest benefit.

Contributions.
In this paper, we present salutary labeling, which aims to select the most informative samples and automatically annotate them with the most beneficial labels, enhancing training efficacy and minimizing the need for human intervention.
We summarize our contribution as follows:

- •
We consider a new task named salutary labeling, which integrates the querying and annotating processes of active learning into a single autonomous step. To the best of our knowledge, this is the first initiative aimed at both maximizing model performance and eliminating the need for ground truth in active learning with an automatic labeling strategy.
- •
We adapt the influence function to calculate the sample influence, which serves as a criterion for selecting the most influential sample for labeling. However, the label information is required during calculating sample influence. Our salutary labeling ingeniously addresses this challenge by assessing the impact of each sample across all possible labels and assigning the label that yields the greatest positive influence. This simple strategy allows the model to automatically select and label samples, maximizing their overall benefit without any human annotation.
- •
We validate the efficacy of our approach on nine benchmark datasets, comparing with four classical methods in active learning and two recent influence function-based methods. In addition to standard active learning experiments, we also conduct various in-depth explorations to address key questions concerning salutary labeling and extend its application to LLM fine-tuning.

## 2 Related Work

Our work intersects with several areas within machine learning. Among them, our work is most closely related but in contrast to existing active learning. Active learning [, ] selectively queries the user to annotate data points that are likely to be most beneficial for improving model performance, while our work introduces a new task named salutary labeling, aiming to accomplish the querying and labeling in one unified step without any human annotation. Traditionally, some strategies [, , ] select important data points with indirect criteria such as uncertainty or representativeness. Uncertainty-based methods define sample uncertainty in one of three main ways: the entropy of the posterior probability distribution [, , ], the probability of the predicted class [, , ], or the margin between the probabilities of the highest two predicted classes [, , ].
Beyond these, research works [, ] utilize consensus among multiple classifiers [, ], or employ an auxiliary module [] to measure uncertainty.
Another strand of active learning approaches focuses on selecting the most representative samples [, , ] through clustering [] or by maximizing the distances between selected samples [].
Alternatively, several methods [, , ] attempt to identify the most diverse subset to represent the full dataset.
Unlike these uncertainty-based and representativeness-based methods, our salutary labeling directly estimates each sample’s impact on model performance with influence function.

Technically, our work is therefore inherently related to influence function [], which measures the change in a model’s output due to an infinitesimal perturbation of one training data point. Following Koh and Liang [], significant research efforts [, , , ] are dedicated to quantifying the impact of individual or group of training samples on model performance. Recently, ISAL [] extends the influence function to active learning by utilizing pseudo labels to calculate the influence.
Alternatively, IBDS [] incorporates an auxiliary regression module, which is specifically trained on labeled data and their calculated influences, to estimate the impact of unlabeled samples. While these methods avoid the requirement of labels in calculating influence function, they still rely on human annotators to label the selected data. In contrast, our method eliminates the need for human annotation, thereby avoiding the labor-intensive process of annotations and the potential inaccuracies associated with detrimental ground truth labels.

Conceptually, our work is also related to several data-centric topics.
Data relabeling methods [, ] seek to relabel the harmful training samples for better model performance, while
partial label learning [, , ] aims to train a classifier to accurately predict the ground-truth label using partially labeled data, where each training instance is associated with multiple candidate labels.
Although both tasks involve automatically assigning labels to data points, neither of them is designed to query unseen samples for further improving model performance.
Data-efficient learning [, , , ] aims to accelerate model training by selecting a minimum subset of the data, which requires ground truth labels for all available data.
Antidote data [, ] overlaps with our method as it generates additional training data to modify specific model behaviors such as fairness. However, these approaches do not primarily focus on the context of active learning.

Figure: Figure 1: Experimental results on Diabetes [] dataset with ground truth and salutary labels. We select 300 samples with labels as the initial set and reserve the remaining training samples as the unlabeled pool set. In both figures, the X-axis represents the sample influence with salutary labels. According to this measurement, we divide the initial/pool set into 20 equal-sized bins. The red and dark blue solid lines denote the performance of adding each bin into the initial set with ground truth labels and salutary labels, respectively, and the dashed blue line denotes the performance with the original initial set.
Refer to caption: /html/2405.17627/assets/figures/moti_init_hticks.png

## 3 Motivation

Conventional active learning methods aim to strategically select unlabeled samples for annotation, assuming that correctly labeled samples inherently enhance model performance. However, this assumption may not always hold true. Research in the realm of noisy labels [, ] has revealed that even a small subset of samples with noisy labels can contribute positively to model improvement. Our own observations, depicted in Figure [1](#S2.F1) (top), further substantiate this claim. Leveraging the influence function, we discern the impact of individual samples on model performance. Based on this analysis, we calculate the sample influence with the most salutary label adjustment, maximizing its impact on model performance. Subsequently, we partition the entire training set into 20 equally-sized bins and replace the labels of samples within each bin with their optimal counterparts. Notably, the red line in the figure illustrates the model’s performance with the entire training set, but with the labels of samples within each bin adjusted accordingly. Note that the dots representing equally-sized samples along the red line do not have uniform intervals and do not align with the unevenly-sized histogram. Surprisingly, for bins with high influence scores, retraining the model with these adjusted labels results in a significant performance improvement. For instance, in the last bin, the accuracy increases from 69% to 74%. This underscores the presence of salutary labels that surpass ground truth labels in enhancing model performance.

Expanding on the concept of salutary labels, we apply it within the framework of active learning, as depicted in Figure [1](#S2.F1) (bottom). Analogous to our previous approach, we sort the data points in the pool set based on their influence when labeled with salutary labels, dividing the pool set into 20 equally-sized bins. The red and blue lines represent the performance when each bin is added to the initial set with ground truth labels and salutary labels, respectively. Our salutary labeling strategy consistently outperforms ground truth labels in most scenarios, particularly notable for samples with high influence estimates, which exhibit a remarkable 5% improvement over ground truth labels. It is noteworthy that the inclusion of bins with low influence leads to a decrease in prediction accuracy, highlighting the presence of detrimental samples. These findings motivate us to pursue active learning with salutary labels, a strategy that not only enhances performance compared to ground truth labels but also alleviates the need for costly annotation efforts.

## 4 Method

### 4.1 Preliminaries

Active learning. The active learning process begins with training a model on a small initial labeled dataset $L$$=$$\{(x_{i},y_{i})\}_{i=1}^{N_{L}}$. Guided by certain criteria, active learning selects an amount of the most informative unlabeled data points from a pool set $U$$=$$\{x_{j}\}_{j=1}^{N_{U}}$, queries their labels to obtain $B$$=$$\{(x_{j^{\prime}},y_{j^{\prime}})\}_{j^{\prime}=1}^{b}$, where $b$ represents the querying budget in each iteration, and updates the model with the newly labeled data $L\cup B$. These queried samples are then removed from the unlabeled pool for subsequent iterations. This learning cycle is repeated for multiple rounds, gradually enhancing model performance while minimizing labeling effort.

Influence function.
For a labeled training dataset $\{(x_{i},y_{i})\}_{i=1}^{N}$ and a model with a convex loss function $\ell(\cdot,\cdot)$, the optimized parameters for empirical risk minimization (ERM) can be represented as $\hat{\theta}$$=$${\arg\min}_{\theta\in\Theta}\frac{1}{N}\sum_{i}\ell{(x_{i},y_{i})}$$+$$\frac{\lambda}{2}\|\theta\|^{2}_{2}$. If one training point $(x_{j},y_{j})$ is down-weighted by infinitesimal $\epsilon$ during the training, the new optimized parameters change to $\hat{\theta}_{(x_{j},y_{j});-\epsilon}$$=$${\arg\min}_{\theta\in\Theta}\frac{1}{N}\sum_{i}\ell{(x_{i},y_{i})}-\epsilon\ell{(x_{j},y_{j})}+\frac{\lambda}{2}\|\theta\|^{2}_{2}$. Without actually retraining the model, the influence function [] estimates the actual change by $\hat{\theta}_{(x_{j},y_{j});-\epsilon}-\hat{\theta}=-\mathbf{H}_{\hat{\theta}}^{-1}\nabla_{\hat{\theta}}\ell{(x_{j},y_{j})}$,
where $\mathbf{H}_{\hat{\theta}}$$=$$\frac{1}{N}\sum_{i=1}\nabla^{2}_{\hat{\theta}}\ell{(x_{i},y_{i})}$$+$$\lambda\mathbf{I}$ is the positive definite Hessian matrix for ${\hat{\theta}}$.

By setting $\epsilon=1/N$, we can linearly approximate the change of $\hat{\theta}$ after removing a training sample, as removing sample $(x_{j},y_{j})$ is equivalent to down-weighting it with $\epsilon=1/N$.
If the validation set $V$ is taken into consideration, let the validation loss be $\mathcal{L}_{v}=\ell(V;{\hat{\theta}})$, the impact of a specific training data point $(x_{j},y_{j})$ on the validation loss can be estimated as follows []:

$$ $\mathcal{I}(x_{j},y_{j})=-\nabla_{\hat{\theta}}\mathcal{L}_{v}^{\top}\mathbf{H}_{\hat{\theta}}^{-1}\nabla_{\hat{\theta}}\ell{(x_{j},y_{j})}.$ (1) $$

Unlike traditional active learning methods that rely on indirect criteria such as uncertainty [, , ] or representativeness [, , ] to select informative samples, influence functions offer a more direct and precise assessment of a data point’s importance to the model. By quantifying the effect of each sample on the model loss on the validation set, the influence function provides a more accurate means of selecting the most informative data points for labeling. Despite their potential benefits, the influence function presents a crucial challenge in active learning. As shown in Eq. ([1](#S4.E1)), the influence function relies on having label information to estimate the impact of each data point, which poses a challenge when dealing with pool samples in the active learning task where such labels are unavailable. To overcome this obstacle, we introduce salutary labeling, which is a simple and effective labeling strategy and makes the influence function flexible for active learning.

### 4.2 Salutary Labeling for Active Learning

In this work, we propose salutary labeling for active learning, a novel approach that directly evaluates the impact of each unlabeled sample and automatically assigns labels to the selected data without any human annotation. This method circumvents the requirement for ground truth labels in influence function calculation, by systematically exploring all possible labels for each data point and calculating the influence corresponding to each label. The label with the highest influence estimation is then assigned to each sample as the salutary label.
This salutary influence, estimated using the salutary label, represents the maximum possible benefit when incorporating the data point into training. Subsequently, our method selects the unlabeled samples with the highest salutary influence and annotates them with salutary labels in a unified step, without requiring any human intervention. In the following section, we introduce the notations and provide technical details of our method.

Training protocol and technical notations.
In each iteration of active learning, the model is trained on the labeled training set $L$ with label space $\mathcal{C}$.
The optimized model parameters for the convex training loss function $\ell(\cdot,\cdot)$ are donated as $\hat{\theta}$.
To actively query the most beneficial samples from the unlabeled pool set $U=\{x_{i}\}_{i=1}^{N_{U}}$, our salutary labeling algorithm calculates the influence estimation of every data point $x_{i}$ with its salutary label on the validation loss $\mathcal{L}_{v}=\ell(V;{\hat{\theta}})$.
The samples with the highest influences are selected as the salutary set, donated as $B=\{(x_{j},y_{j}^{s})\}_{j=1}^{b}$, where $y_{j}^{s}\in\mathcal{C}$ represents the salutary label of the queried data and the superscript ‘s’ represents the salutary label.
After forming the salutary set, it is removed from the pool $U$, thus updating $U=U\setminus B$. Subsequently, the model is re-trained on the expanded labeled set $L=L\cup B$ for the next active learning cycle.

Salutary labeling with influence function. With the concept of the salutary label, we can handle the absence of label information when calculating the influence function. Specifically, for an unlabeled sample, we compute the influence estimations for each label and pick the one with the largest influence value as follows:

$$ $\mathcal{I}(x_{j},y_{j}^{s})=\mathcal{I}(x_{j},\hat{c}),\ \textup{where}\ \hat{c}=\operatorname*{arg\,max}_{c\in\mathcal{C}}\mathcal{I}(x_{j},c).$ (2) $$

Autonomous active learning. Eq. ([2](#S4.E2)) directly measures the impact of each unlabeled sample and automatically assigns the salutary label, enabling our method to query the unlabeled data without human intervention. Specifically, the model selects the top $b$ samples with the highest influences from the pool set $U$ and annotates them with salutary labels, to form an active salutary set $B=\{(x_{j},y_{j}^{s})\}_{j=1}^{b}$. This salutary set is then removed from $U$ and integrated into the labeled training set $L$, to update the model.

We summarize the full training protocol of our salutary labeling for active learning in Algorithm [1](#alg1). The time complexity of salutary labeling is bounded by the calculation of the influence function in Eq. ([2](#S4.E2)). For each label $c\in\mathcal{C}$, the calculation of gradients for all unlabeled samples will take $\mathcal{O}(nd)$, where $n$ is the number of samples and $d$ is the dimension of model parameter $\theta$.
Notice that the computation of the Hessian matrix and its inverse only involves the label information of the validation set. Therefore, these calculations only need to be performed once for all potential labels. The explicit computation of Hessian takes $\mathcal{O}(nd^{2})$ and its inversion takes $\mathcal{O}(d^{3})$, we apply conjugate gradients and stochastic estimations of Hessian-vector products [], reducing the time complexity to $\mathcal{O}(nd)$.

Figure: Algorithm 1 Salutary Labeling for Active Learning

## 5 Experiments

We demonstrate the performance of our method in this section.
We first introduce the experimental setup, then report the algorithmic performance of extended active learning experiments,
and finally provide various in-depth analyses of autonomous salutary labeling.

### 5.1 Experimental Setup

Datasets.
We use the seven tabulate datasets [] and two vision datasets in our experiments.
Bank [] dataset has a total of 30,488 records of bank telemarketing phone calls. Each sample contains 51 features which are used to predict if a client will subscribe to a term deposit or not.
Diabetic [] dataset contains 1,151 retina images of patients for predicting if the patients suffer from Diabetes or not. We use 19 features extracted by Antal and Hajdu [].
CelebA [] has a total of 104,163 samples of face images with 39 features from each sample image and we treat the features as tabulated data to predict if the person is smiling or not.
Musk_v2 [] dataset contains 6,598 instances of molecules, and 166 features to represent the low-energy conformations of the molecules, which is used to learn to predict whether new molecules will be musks or non-musks.
Electrical [] dataset contains 10,000 points and 11 attributes such as power consumption and price in a 4-node star electrical grid system, which is used to predict if the system is stable or not.
Wine [] dataset consists of the physicochemical test results for 4,898 variants of the Portuguese “Vinho Verde” wine. We use it to predict the quality scores (from 3 to 9) based on 11 physicochemical attributes, such as acidity, density, and alcohol rate.
Waveform [] dataset contains 5,000 instances of waveform records, each described by 21 attributes. We use it to classify each record into one of the three waveform classes.
MNIST [] is a collection of 70,000 handwritten digit images (0 through 9). We use a ResNet-34 [], which is pre-trained on the ImageNet [], to extract 512 deep features for each image.
CIFAR10 [] consists of 60,000 real-life images in 10 classes, with 6,000 images per class. Similar to the MNIST dataset, we also extract 512 features with the pre-trained ResNet-34. We summarize the datasets used in the experiments in Appendix [A](#A1).

**Table 1: Final predicting accuracy (%) of the logistic regression model on the test set after 10 rounds of active learning in 5 runs. We report only the average accuracy in this table due to the space limits. The standard deviations are presented in Figure [2](#S5.F2) as well as in Table [4](#A2.T4) in Appendix [B](#A2).**
| Method | Electric | Bank | ​Diabetic | CelebA | ​Musk_v2 | Wine | ​​​​Waveform | ​​CIFAR10 | MNIST |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Init | 63.85 | 65.89 | 56.43 | 73.33 | 73.45 | 44.76 | 79.11 | 46.74 | 77.75 |
| Random | 65.15 | 67.77 | 58.41 | 82.06 | 78.33 | 46.31 | 81.10 | 55.92 | 80.93 |
| Entropy [] | 69.72 | 73.84 | 65.34 | 81.23 | 79.11 | 45.00 | 83.23 | 53.91 | 83.77 |
| Margin [] | 69.72 | 73.84 | 65.34 | 81.23 | 79.11 | 47.30 | 82.26 | 56.95 | 83.72 |
| Uncertainty [] | 69.72 | 73.84 | 65.34 | 81.23 | 79.11 | 44.53 | 83.33 | 55.47 | 83.63 |
| ISLA [] | 67.98 | 64.41 | 61.38 | 84.71 | 77.72 | 47.15 | 79.40 | 53.91 | 79.35 |
| IBDS [] | 67.66 | 65.14 | 64.35 | 82.49 | 78.15 | 44.84 | 82.91 | 54.61 | 80.05 |
| Ours | 71.31 | 78.07 | 71.28 | 85.50 | 81.06 | 49.92 | 84.21 | 58.33 | 86.68 |

Figure: Figure 2: Comparison of salutary labeling and baseline active learning methods on nine datasets over 10 rounds of learning cycle. In all figures, the X-axis represents the training iterations, where round 0 is the initial training. The shaded area is the standard deviation across 5 different random runs. Notice that the entropy, margin and uncertainty sampling yield the same results for binary datasets.
Refer to caption: /html/2405.17627/assets/figures/results_err_oneline.png

Baseline methods.
We include the six baseline methods for active learning. Random sampling is the most intuitive baseline which randomly queries samples from the pool set. Entropy sampling [] selects the unlabeled samples with the highest entropy of the current model’s predictions. Margin sampling [] ranks all pool samples by the margin between the highest and second-highest values from the soft-max logits predicted by the model. Uncertainty sampling [] queries by the classification uncertainty, which is determined by the probability of the predicted class as assigned by the classifier. We also include two influence-based active learning methods, both of which choose the unlabeled data set with influence estimation. Influence Selection for Active Learning (ISLA) [] uses base model predictions as pseudo-labels to compute influence. Influence-based Data Selection (IBDS) [] uses an influence regressor, which is trained with labeled training data and their influences calculated with Eq. ([1](#S4.E1)), to predict the influence estimations for the unlabeled data. It is important to note that while all six baseline methods require human effort to annotate the queried unlabeled samples, our approach is completely human annotation-free.

Experimental protocol and implementation details.
In our experiments, we divide all datasets into training set (60%), validation set (20%), and test set (20%), except for Bank, CelebA and Diabetic datasets, which have predefined splits for training, validation, and testing. The influence-based models, including ISAL, IBDS, and our methods, exclusively utilize the validation set to compute influence estimations. This setup ensures that none of the methods access any information from the test set, maintaining the testing data unseen to the models during the evaluation. All experiments are repeated 5 times with different random seeds. In each run, we randomly choose 300 samples from the training set as the initial set and reserve the rest as the pool set.

We implement
our method with Scikit-learn [] and Pytorch []. All experiments are conducted on our workstation equipped with one 24GB NVIDIA TITAN RTX GPU.
We choose a logistic regression classification model that satisfies the convex requirement of the influence function.
We initiate the process by training this model with the initial set.
Subsequently, we conduct active learning for $R=10$ active rounds. In each round, the model queries 10 samples from the pool dataset $U$. For baseline methods, the ground truth labels of these selected samples are used, whereas our method automatically assigns salutary labels according to Eq. ([2](#S4.E2)). After labeling, the queried data points are integrated into the labeled set for re-training the model. After each round of learning, we evaluate the model’s performance by measuring prediction accuracy on the test set.

We set the query budget $b$ to 10 to maintain the distinction in performance between different models. Using a larger budget, such as 1% of the pool set, might cause the model to reach the performance ceiling on some datasets. We provide a detailed discussion and visualization on this in Appendix [D](#A4).

### 5.2 Algorithmic Performance

We evaluate the performance of our salutary labeling method alongside the active learning baselines. Note that the entropy, margin, and uncertainty samples yield the same results for the same random initial/pool splits in binary classification datasets, as these three metrics have the same rank for 2-dimensional logits.
As shown in Table [1](#S5.T1), our method shows significant improvements over the initial model despite a limited querying budget and achieves the highest accuracy among all active learning methods.
We notice that the two influence-based baselines do not perform well on datasets like Diabetic and Wine. This highlights the difficulties in estimating influence without access to label information, emphasizing the challenges and limitations of current influence-based approaches in handling complex datasets where salutary labeling shows a clear advantage.

Moreover, we also present the accuracy change throughout 10 learning rounds for all methods in Figure [2](#S5.F2). Notably, our method demonstrates a significant and steady improvement in accuracy, particularly in challenging datasets like Bank, Waveform, and Wine, where the baselines show limited progress.
This indicates the efficiency of salutary labeling in active learning, particularly noteworthy as it operates without the need for human annotation efforts. This capability to function autonomously underscores the potential of our method for practical applications.

### 5.3 In-depth Explorations

We would like to answer the following questions for salutary labeling in our in-depth exploration:

- •
The influence function has been demonstrated as an accurate estimation for leave-one-out influence [], which estimates the impact of removing a training sample. On the contrary, salutary labeling adapts this function to assess the effect of adding a sample unseen during model training, raising the question: How accurate is this estimation?
- •
As salutary labeling does not require human annotation, there is no budget constraint. Is it possible to achieve better performance training with more pool samples?
- •
The influence function requires the learning model to be convex, which limits its applied scenarios. Can we circumvent the convex requirement of influence function and extend the salutary labeling to applications involving non-convex deep models?

Influence estimation vs. add-one-in retraining.
We empirically verify how accurate is the influence function when estimating the impact of adding a new data point on three datasets, namely Diabetic, CelebA, and Bank.
For each dataset, we compare the predicted influence estimations with the actual changes in loss observed after adding a sample and re-training the model.
Using the initial set, we train a logistic regression model $\hat{\theta}$ and compute the influence $\mathcal{I}(x_{j},y_{j})$ for every data point in the pool set. Consequently, we individually add each pool sample $(x_{j},y_{j})$ to the training set and update the model parameters $\hat{\theta_{j}}$. We compare influence estimation $\mathcal{I}(x_{j},y_{j})$ and the validation loss difference after add a sample $\ell(V;{\hat{\theta_{j}}})-\ell(V;{\hat{\theta}})$. As shown in Figure [3](#S5.F3),
The influence estimation for new samples does not perfectly match the actual loss change, likely because they were unseen during initial training.
However, the influence estimations are highly correlated with actual loss differences, as measured by Spearman’s rank correlation coefficient.
Therefore, the influence function still provides an accurate indication of each sample’s relative impact.

Figure: Figure 3: Influence estimation vs. actual loss difference of add-one-in retraining on Diabetic (left), CelebA (middle), and Bank (right) datasets. In all plots, the horizontal axes represent the estimated influence on validation loss, while the vertical axes show the actual loss change. Their correlation is quantified with the Spearman’s rank correlation coefficient (Spearman-r). We randomly selected 300 samples in each plot to ensure clarity in visualization.
Refer to caption: /html/2405.17627/assets/figures/add_one_in_scale.png

Salutary labeling with more data points.
In Section [5.2](#S5.SS2), we demonstrated the efficacy of salutary labeling. The fact that salutary labeling requires zero human intervention allows our method to query even more unlabeled samples without incurring any annotation costs. Therefore, we conduct additional experiments to evaluate the effectiveness of our method with more pool samples. Following the setup described in Section [5.2](#S5.SS2), we split the data into an initial set for training the initial logistic regression model, along with a pool set, validation set, and test set. For each data set, the model queries and automatically annotates 10 samples from the pool set with salutary labeling in each active learning iteration. We allow the model to query up to 50% samples from the pool set and choose the iteration that has the best predicting accuracy on the validation set as the final model.

In addition to evaluating our salutary labeling, we also report the test accuracy obtained after training the model with all labeled data from both the initial and pool sets. This provides a reference point to the maximum achievable accuracy when the model is supervised by all available data. As demonstrated in Table [2](#S5.T2), our method outperforms supervised learning on four datasets, which validates that salutary labels can indeed provide superior guidance compared to ground truth labels under certain conditions. On Musk_v2 [], Wine [], and Waveform [] datasets, the fully supervised model only slightly lead our method by a margin less than 1%, the fully supervised model leads our method by a narrow margin of less than 1%. On CIFAR10 [] and MNIST [], our method trails the fully supervised model by approximately 3.5%, but it still boosts the accuracy by over 15% compared to the initial model. Notably, these gains are achieved without any human annotation, demonstrating the effectiveness of our approach in leveraging unlabeled data.

**Table 2: Predicting accuracy (%) of the logistic regression model on the test set after querying 50% of the pool set. The standard deviations (less than 0.3% for all datasets) are omitted due to space limits.**
| Method | Electrical | Bank | ​Diabetic | CelebA | ​Musk_v2 | Wine | ​​​​Waveform | ​​CIFAR10 | MNIST |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Init | 63.85 | 65.89 | 56.43 | 73.33 | 73.45 | 44.76 | 79.11 | 46.74 | 77.75 |
| Full Pool | 70.08 | 80.14 | 72.27 | 85.07 | 85.75 | 52.53 | 85.60 | 65.67 | 95.36 |
| Ours | 72.25 | 81.21 | 73.26 | 85.89 | 85.68 | 52.38 | 85.50 | 62.05 | 92.06 |

Salutary labeling for LLM fine-tuning.
In our above experiments, we use a logistic regression model to fulfill the convex requirement of the influence function. In this section, we aim to expand our salutary labeling method to practical applications with complex model structures. Specifically, we conduct the active learning experiments in the LLM fine-tuning with RoBERTa [] model on three datasets of GLUE [] repository, namely, WNLI [], MRPC [] and RTE []. We simulate an active learning scenario for fine-tuning the RoBERTa model, denoted by $g\circ h$, where $g$ represents the transformer layers and $h$ represents the classification head. Following the setting of Section [2](#S5.F2), we divide each dataset into the initial set, pool set, validation set, and test set.

During the whole training, we fix the transformer layers $g$ in RoBERTa and fine-tune the non-convex classification head $h$. Initially, we train the model using the initial set. Subsequently, in each learning cycle, we use the 768-dimensional hidden state extracted by $g$, along with predictions from of $h$, to train a surrogate logistic regression model $h^{\prime}(\cdot;\hat{\theta})$. This surrogate model was then used to identify and annotate 10 samples from the pool set, as detailed in Algorithm [1](#alg1). The newly annotated samples are used to update the classification head $h$.
We provide the training details in Appendix [C](#A3).

As illustrated in Figure [4](#S5.F4), our method outperforms all baseline approaches in all three tasks after 10 learning cycles. The performance advantage is consistent across most rounds, with detailed per-round results displayed in Figure [5](#A2.F5) of the Appendix [C](#A3). These findings underscore the potential of our method in practical applications, highlighting the adaptability and effectiveness of our approach in real-world settings, even when the model is not strictly convex.

Figure: Figure 4: Predicting accuracy of the final model after 10 rounds of active learning for LLM fine-tuning on WNLI (left), MRPC (middle) and RTE (right) datasets of GLUE repository.
Refer to caption: /html/2405.17627/assets/figures/llm_last_round_oneline.png

## 6 Conclusion

In this paper, we delved into the realm of active learning and proposed a novel concept called salutary labeling, which seamlessly merges the querying and annotating processes of active learning into a single autonomous step. Unlike traditional methods, our approach eliminates the need for human annotation; instead, it automatically assigns a salutary label, i.e., the label category that maximizes model performance. Technically distinct from conventional active learning approaches that rely on indirect measurements such as uncertainty and representativeness to select samples for labeling, we utilized the influence function to directly compute sample influence. However, a significant challenge arises when dealing with pool samples in active learning tasks, as label information may be unavailable. Our salutary labeling method adeptly overcomes this hurdle by evaluating the impact of each sample across all possible labels and assigning the label that generates the greatest positive influence. Extensive experimental results underscored the efficacy and advantages of our salutary labeling approach across various scenarios.

## Appendix

## Appendix A Dataset

We summarize some key statistics of nine datasets we use in experiments of Section [5.2](#S5.SS2) in Table [3](#A1.T3). For all datasets, we conduct 5 runs of experiments with different random seeds. In each experimental run, we fix the validation set and test set, then randomly choose 300 data points from the training samples as the initial set, and reserve the rest as the pool set. All datasets are publicly available with CC BY 4.0 license.

**Table 3: Dataset Statistics**
| Dataset | # of Training | # of Val | # of Test | # of Classes | # of Dim | Data Type |
| --- | --- | --- | --- | --- | --- | --- |
| Bank [] | 18,292 | 6,098 | 6,098 | 2 | 51 | tabulate |
| Diabetic [] | 950 | 100 | 100 | 2 | 19 | tabulate |
| CelebA [] | 62,497 | 20,833 | 20,833 | 2 | 39 | tabulate |
| Musk_v2 [] | 3,958 | 1,320 | 1,320 | 2 | 166 | tabulate |
| Electrical [] | 6,000 | 2,000 | 2,000 | 2 | 12 | tabulate |
| Waveform [] | 3,000 | 1,000 | 1,000 | 3 | 21 | tabulate |
| Wine [] | 3,896 | 1,300 | 1,300 | 7 | 11 | tabulate |
| MNIST [] | 54,000 | 6,000 | 10,000 | 10 | 512 | vision |
| CIFAR10 [] | 45,000 | 5,000 | 10,000 | 10 | 512 | vision |

## Appendix B Detailed algorithmic performance with standard deviation

We do not include the standard deviation in Table [1](#S5.T1) for better visualization.
Here we report the full experimental results with standard deviation in Table [4](#A2.T4), which includes the active learning experiments in Section [5.2](#S5.SS2) and the LLM fine-tuning experiments in Section [2](#S5.T2).
Our salutary labeling method outperforms all baseline methods across multiple datasets in the standard active learning setting for both convex logistic regression and non-convex LLM fine-tuning, all without requiring any human annotation.
Notably, our method not only achieves the highest final predicting accuracy across all datasets but also maintains relatively small standard deviations, keeping consistent performance across different experimental runs. These results highlight the efficacy of our method, emphasizing its potential in practical applications.

**Table 4: Final predicting accuracy (%) for all datasets on the test data after 10 learning cycles, alongside the standard deviations across five experimental runs with different random seeds. This table includes the results of all experimental in Section [5.2](#S5.SS2) and LLM fine-tuning in Section [2](#S5.T2).**
| Method | Electrical | Bank | ​Diabetic | CelebA | ​Musk_v2 | Wine |
| --- | --- | --- | --- | --- | --- | --- |
| Init | 63.85 | 65.89 | 56.43 | 73.33 | 73.45 | 44.76 |
| Random | 65.15±0.40 | 67.77±0.61 | 58.41±0.93 | 82.06±0.13 | 78.33±1.30 | 46.31±0.16 |
| Entropy [] | 69.72±0.55 | 73.84±1.33 | 65.34±0.11 | 81.23±2.11 | 79.11±0.60 | 45.00±0.41 |
| Margin [] | 69.72±0.55 | 73.84±1.33 | 65.34±0.11 | 81.23±2.11 | 79.11±0.60 | 47.30±0.25 |
| Uncertainty [] | 69.72±0.55 | 73.84±1.33 | 65.34±0.11 | 81.23±2.11 | 79.11±0.60 | 44.53±0.67 |
| ISLA [] | 67.98±0.74 | 64.41±0.54 | 61.38±0.80 | 84.71±0.41 | 77.72±0.14 | 47.15±0.65 |
| IBDS [] | 67.66±0.94 | 65.14±0.15 | 64.35±0.46 | 82.49±0.36 | 78.15±0.64 | 44.84±0.64 |
| Ours | 71.31±0.04 | 78.07±0.92 | 71.28±1.68 | 85.50±0.12 | 81.06±0.39 | 49.92±0.61 |

Figure: Figure 5: Comparison of salutary labeling and baseline methods on three datasets of GLUE [] repository over 10 rounds of learning cycle in LLM fine-tuning application.
Refer to caption: /html/2405.17627/assets/figures/roberta_results_err_oneline.png

## Appendix C Training details for active LLM fine-tuning

We conduct our LLM fine-tuning experiments on three datasets of GLUE [] repository, namely, WNLI [], MRPC [] and RTE [].
WNLI is a reading comprehension dataset, where the authors construct sentence pairs by replacing the ambiguous pronoun in the original sentence with each possible referent.
This dataset is used for predicting whether the sentence with the pronoun substituted is entailed by the original sentence or not. MRPC is a corpus of sentence pairs automatically extracted from online news sources,
and we use it to predict whether the sentences in the pair are semantically equivalent or not.
RTE are constructed based on news and Wikipedia text.
The task is to classify each sample into one of the two classes assigned by human annotators.

For each dataset, we randomly select 100 samples from the predefined training split to form the initial set and use the remaining data as the pool set. Half of the predefined validation split serves as the validation set for salutary labeling, with the other half used as the test set. We use the Hugging Face [] implementation of RoBERTa [], denoted by $g\circ h$. We fix the transformer layers $g$ while fine-tuning the classification head $h$, which is a two-layer multilayer perceptron model with dropout before both layers and $tahn$ activation function between the two layers.
Initially, the model is trained with the initial set. In each of the 10 active learning cycles, it annotates 10 samples. For sampling methods like entropy, margin, and uncertainty, the output of $h$ determines the pool set queries. For influence-based methods including ISAL [], IBDS [] and our method, we train a surrogate logistic regression model $h^{\prime}(\cdot;\hat{\theta})$ using the 768-dimensional hidden states extracted by $g$ and predictions from $h$. This surrogate model was then used to calculate the influence function and query pool samples for model re-training. We compute the accuracy on the test set after each round and plot the results in Figure [5](#A2.F5).

Figure: Figure 6: Prediction accuracy of salutary labeling and baseline methods with $b$ set at 1% of the pool samples on CelebA [] (left), Waveform [] (middle), and Electrical [] (right) datasets.
Refer to caption: /html/2405.17627/assets/figures/results_1percent_oneline.png

## Appendix D Choice of query budget b 𝑏 b

We set a relatively small query budget $b$ to maintain clear performance distinctions between different models. In our preliminary exploration stage, we found that a larger budget, such as 1% of the pool set, allows models to reach the performance ceiling on datasets like CelebA [], Waveform [], and Electrical [].
As shown in Figure [6](#A3.F6), such a budget causes different active learning methods to perform very similarly after several rounds. Consequently, we opted for a smaller budget in our experiments to better evaluate the distinct capabilities of each model.

## Appendix E Broader Impact and Limitations

This paper presents work whose goal is to advance the field of Machine Learning.
We broaden the scope of active learning with a novel approach called salutary labeling, which integrates the querying and annotating processes of active learning into a single, autonomous step.
The proposed salutary labeling method eliminates human annotation and maximizes benefits from queried data.
Beyond the impact mentioned above, there are also other potential societal consequences of our work, none of which we feel must be specifically highlighted here.

One potential limitation of our method stems from the influence function, one key component of salutary labeling.
The influence function requires the model to be convex, ensuring that its Hessian matrix is positive definite, and invertible after training to convergence. Despite the ongoing discussions [, , ] on the accuracy of the influence function on non-convex models, many research works have successfully applied the influence function across various applications [, , ]. In this work, we adopt the same strategy as in the work of Li and Liu [], which uses a surrogate convex model on the embeddings extracted by the non-convex model, and achieve promising results as illustrated in Section [5.3](#S5.SS3).
Further exploring the application of the influence function to non-convex models is not the focus of this study, so we defer this topic to future work.

## Appendix F Code and Reproducibility

The code for the implementation of our method will be released soon.

All experiments were conducted on a Linux workstation running Ubuntu 20.04.6 LTS. The CPU used was an Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz. Any experiments requiring GPU (such as multi-class influence calculation and LLM fine-tuning) were conducted with one NVIDIA TITAN RTX GPU with 24 VRAM and CUDA version 11.4.

All code is written in Python, and utilizes basic libraries such as NumPy [], scikit-learn [], PyTorch [], Pandas [], etc. Detailed package information will be provided in the code repository.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1)	In addition to the logistic regression model used, is the proposed method suitable for more complex models? In Appendix E, a surrogate model is employed to compute the influence, but could the ResNet itself be directly used for influence calculation? 
2)	Since the accuracy of the pseudo-labels depends on the quality of the validation set, how does the performance of the proposed method vary with different validation set sizes?
soundness: 3
strengths: Solid experiments are conducted on both tabular and image datasets, integrating recent data selection methods from AL and SSL. The proposed method demonstrates promising empirical results without the need for human-annotated labels. In addition, the method is validated on a LLM fine-tuning task, which further underscores its potential for application in different domains.
summary: This paper presents a novel pseudo-labeling approach for unlabeled data using influence functions. The proposed method estimates the influence of each possible label on the validation loss for a given unlabeled data point and assigns the label with the most significant improvement in loss as its pseudo-label. Subsequently, a subset of the unlabeled data with the highest improvement in validation loss is selected to update the model. Extensive experiments are conducted to validate the effectiveness of the method.
weaknesses: While the method is framed in part within the active learning context, its approach—assigning pseudo-labels to unlabeled data via a self-training mechanism—seems more aligned with semi-supervised learning. The introduction could benefit from adjustments to reflect this alignment more accurately. Another concern is the limited technical contributions. It uses the influence function to score and pseudo-labeling the unlabeled data. Although this is an interesting application, it may not represent a substantial methodological advance. Furthermore, in the experiments, a query budget of 10 examples is set, yet only the first 10 rounds of performance are reported. Providing results for additional rounds or scenarios with a larger query budget would offer a more comprehensive evaluation of the method’s long-term effectiveness. In Appendix D, results from querying 1% of the data reveal that the proposed method underperforms relative to other baseline methods. This needs further exploration and explanation. My last concern is that the paper’s approach relies on setting aside 20% of the data for validation, which may be impractical for certain active learning settings, where labeled data is typically scarce. Performances on different sizes of validation set should also be explored.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: My main concern is the weakness 1. I do not understand the value of their new proposed task. I think this setting is similar to the unsupervised learning with the pseudo labels. In my understanding, I think the human annotation is helpful during training. This is the reason why we study active learning. If we fully abandon human intervention, this is totally another area which is unsupervised learning.
soundness: 1
strengths: 1. The motivation and paper writing are clear. 
2. The experiment is sufficient 
3. The method is fully automatic without human annotation
summary: This paper proposes salutary labeling, which automatically assigns the most beneficial labels to the most informative samples without human annotation. Specifically, they utilize the influence function, a tool for estimating sample influence, to select newly added samples and assign their salutary labels by choosing the category that maximizes their positive influence.
weaknesses: 1. They do not discuss the difference with the unsupervised learning methods, such as 

[1] Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID
[2] Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification

If the human intervention is removed from active learning, it will be transformed to unsupervised learning that assigns the pseudo labels to the samples. Could you discuss the difference ?

2. How to tackle with the situation that the selected labels are wrong? Could you discuss potential error correction mechanisms, or to analyze the impact of incorrect labels on model performance?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: - The proposed approach appears closely related to training with noisy labeled data methods mentioned in the Weakness section. An analysis and comparison with this literature in the related work would be beneficial.
- A detailed analysis of the assigned salutary labels would be interesting. It would be helpful to know if labels different from the ground truth sometimes enhance performance or if performance improvement primarily comes from refining noisy labels.
soundness: 2
strengths: - The *Salutary Labeling for Active Learning* framework was new to me. If we can effectively annotate data automatically without human annotator, it could demonstrate remarkable potential for machine learning as a whole.

- Overall, the writing is well-structured and easy to follow, making it straightforward to understand the main concepts.

- The proposed method consistently improved performance across nine datasets. This study also conducted detailed ablation studies to analyze the effectiveness of *salutary labels* through the influence function.
summary: This study defines a task called salutary labeling, which involves selecting a subset of data from an unlabeled data pool predicted to be the most beneficial for training, similar to traditional active learning, and then using this selected data for model training. The key difference in the proposed salutary labeling approach is that, instead of using labels from human annotators, it assigns pseudo labels expected to improve performance on the validation set. Specifically, it measures the influence of each data point and assigns a salutary label that maximizes validation set performance. The data with the highest influence from the assigned salutary labels is then used for training. This method was tested on nine datasets and demonstrated improved results compared to traditional active learning approaches.
weaknesses: - The motivation behind combining salutary labeling with active learning is not fully clear to me.
   - The core motivation of active learning is to label only a small number of informative data points to reduce annotation costs. If automatic labeling without human annotation costs is feasible, applying salutary labels to all available data without the selection process in active learning should suffice.
   - In Algorithm 1, it seems that salutary labels are generated for the entire unlabeled pool during selection. Figure 5 also suggests that model performance improves as the number of salutary labels increases, so the necessity of sampling is unclear.
   - **Additional explanation on why the active learning framework remains effective despite not requiring human labor for labeling would help clarify my understanding.**

- I am not fully convinced about how the salutary labeling task is novel compared to existing label-efficient tasks.
   - To me, the salutary labeling task appears to combine elements from various existing label-efficient tasks. If the goal is to automatically label training data, this could be viewed as a pseudo-labeling process in semi-supervised or self-supervised learning. On the other hand, methods for correcting label noise align more closely with the learning with label noise task [a, b, c].
   - In Appendix A, a key difference from other label-efficient learning tasks is described as a focus on active learning (line 836). However, as mentioned above, without a clear reason for combining salutary labels with the active learning framework, I remain uncertain about the task’s distinction from others.
   - **A clearer explanation on why salutary labels need to be combined with active learning and how this approach fundamentally differs from other label-efficient learning methods would be helpful.**

- There seem to be potential fairness concerns in the comparative experiments with existing AL baselines.
   - The proposed experimental setup uses 20% of the dataset as a validation set and utilizes this validation set’s annotations for influence function calculations and labeling. If labeled data is essential for the proposed method, it would be fairer to allocate part of the training budget for this purpose.
   - Specifically, in Table 1, the budget sample size is set as low as 10, which makes it impractical to reserve 20% of the dataset as a validation set while only training on about 10 samples. This setup might give the impression that validation set labels are indirectly being used for salutary labeling of the unlabeled data.
   - As shown in Figure 5 of Appendix D, one potential reason for the reduced gain of the proposed method as the budget size increases may be the dilution of the validation set’s supervisory effect.
   - **The following additional experiments might strengthen the justification for salutary labeling:**
       - Using a smaller validation set (e.g., 1% to 5% instead of 20%).
       - Including the validation set size in the training budget for all methods (e.g., allowing AL baselines to train on the validation set or at least part of it).

[a] Xiao, Ruixuan, et al. "Promix: Combating label noise via maximizing clean sample utility." arXiv preprint arXiv:2207.10276 (2022).

[b] Chen, Wenkai, Chuang Zhu, and Mengting Li. "Sample prior guided robust model learning to suppress noisy labels." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer Nature Switzerland, 2023.

[c] Liu, Sheng, et al. "Robust training under label noise by over-parameterization." International Conference on Machine Learning. PMLR, 2022.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.In the last paragraph of Section 4, the authors mentioned that the time complexity of salutary labeling is O(nd). However, the proposed salutary labeling algorithm need to calculate the influence estimation of every data point in each iteration of active learning. How much will this slow down the entire training process? Can the authors provide the running time comparison results of each method in Table 1?

2.In the experiment, the author set active rounds R = 10 and query budget b = 10. When b and R are larger, is it impossible to prove that the proposed salutary labeling is effective?

3.The first paragraph of Section 2 is too long and a little bit difficult to read. It should be adjusted appropriately.

4.The legend in Figure 1 obscures part of the polyline and may need to be further refined.
soundness: 3
strengths: 1.The writing is good and the supplementary materials are relatively sufficient.

2.The authors propose a simple-sounding but effective active learning method which eliminates the need for human annotation. Judging from the comparative experimental results provided by the authors, this idea is effective.
summary: The authors propose salutary labeling for active learning which is human annotation-free. They adapt the influence function to calculate the sample influence by assessing the impact of each sample across all possible labels and assigning the label that yields the greatest positive influence. The authors conducted experiments on different datasets to verify the effectiveness of the simple idea.
weaknesses: 1.In the last paragraph of Section 4, the authors mentioned that the time complexity of salutary labeling is O(nd). However, the proposed salutary labeling algorithm need to calculate the influence estimation of every data point in each iteration of active learning. How much will this slow down the entire training process? Can the authors provide the running time comparison results of each method in Table 1?

2.In the experiment, the author set active rounds R = 10 and query budget b = 10. When b and R are larger, is it impossible to prove that the proposed salutary labeling is effective?

3.The first paragraph of Section 2 is too long and a little bit difficult to read. It should be adjusted appropriately.

4.The legend in Figure 1 obscures part of the polyline and may need to be further refined.

Meta-review: This paper proposes salutary labeling, which automatically assigns the most beneficial labels to the most informative samples without human annotation. Specifically, the authors utilize the influence function, a tool for estimating sample influence, to select newly added samples and assign their salutary labels by choosing the category that maximizes their positive influence. This process eliminates the need for human annotation. Intensive experiments conducted on nine benchmark datasets demonstrate the superior performance of the proposed salutary labeling approach over traditional active learning strategies. 

Generally, the reviewers consider that this paper is potentially interesting, but the model details as well as experiments need more and clearer explanations. I hope the authors can take the comments into consideration and submit their paper to somewhere else.

Decision: Reject