Venue: NAACL 2022

# AcTune: Uncertainty-Based Active Self-Training for

Active Fine-Tuning of Pretrained Language Models

 Yue Yu1, Lingkai Kong1, Jieyu Zhang2, Rongzhi Zhang1, Chao Zhang1

###### Abstract

While pre-trained language model (PLM) fine-tuning has achieved strong performance in many NLP tasks, the fine-tuning stage can be still demanding in labeled data. Recent works have resorted to active fine-tuning to improve the label efficiency of PLM fine-tuning, but none of them investigate the potential of unlabeled data. We propose AcTune, a new framework that leverages unlabeled data to improve the label efficiency of active PLM fine-tuning. AcTune switches between data annotation and model self-training based on uncertainty: it selects high-uncertainty unlabeled samples for active annotation and low-uncertainty ones for model self-training. Under this framework, we design (1) a region-aware sampling strategy that reduces redundancy when actively querying for annotations and (2) a momentum-based memory bank that dynamically aggregates the model's pseudo labels to suppress label noise in self-training. Experiments on 6 text classification datasets show that AcTune outperforms the strongest active learning and self-training baselines and improves the label efficiency of PLM fine-tuning by 56.2% on average. Our implementation is available at [https://github.com/yueyu1030/actune](https://github.com/yueyu1030/actune).

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Fine-tuning pre-trained language models (PLMs) has achieved much success in natural language processing (NLP) Devlin et al. (2019); Liu et al. (2019); Brown et al. (2020). One benefit of PLM fine-tuning is the promising performance it offers when consuming only a few labeled data Bansal et al. (2020); Gao et al. (2021). However, there are still significant gaps between few-shot and fully-supervised PLM fine-tuning in many tasks. Besides, the performance of few-shot PLM fine-tuning can be sensitive to different sets of training data Bragg et al. (2021). Therefore, there is a crucial need for approaches that make PLM fine-tuning more label-efficient and robust to selection of training data, especially for applications where labeled data are scarce and expensive to obtain.

Towards this goal, researchers have recently sorted to _active fine-tuning_ of PLMs and achieved comparable performance to fully-supervised methods with much less annotated samples Ein-Dor et al. (2020); Margatina et al. (2021a, b); Yuan et al. (2020). Nevertheless, they usually neglect unlabeled data, which can be useful for improving label efficiency for PLM fine-tuning Du et al. (2021). To incorporate unlabeled data into active learning, efforts have been made in the semi-supervised active learning literature Wang et al. (2016); Rottmann et al. (2018); Simeoni et al. (2020). However, the query strategies proposed in these works can return highly redundant samples due to limited representation power, resulting in suboptimal label efficiency. Moreover, they usually rely on pseudo-labeling to utilize unlabeled data, which requires greater (yet often absent) care to denoise the pseudo labels, otherwise the errors could accumulate and hurt model performance. This issue can be even more severe for PLMs, as the fine-tuning process is often sensitive to different weight initialization and data orderings Dodge et al. (2020). Thus, it still remains open and challenging to design robust and label efficient method for active PLM fine-tuning.

To tackle the above challenges, we propose AcTune, a new method that improves the label efficiency and robustness of active PLM fine-tuning. Based on the estimated model uncertainty, AcTune tightly couples _active learning_ with _self-training_ in each learning round: (1) when the average uncertainty of a region is low, we trust the model's predictions and select its most certain predictions within the region for self-training; (2) when the average uncertainty of a region is high, indicating inadequate observations for parameter learning, we actively annotate its most uncertain

samples within the region to improve model performance. Different from existing AL methods that only leverage uncertainty for querying labels, our uncertainty-driven self-training paradigm gradually leverages unlabeled data with low uncertainty via self-training, while reducing the chance of error propagation triggered by highly-uncertain mislabeled data.

To further boost model performance for AcTune, we design two techniques to improve the query strategy and suppress label noise, namely region-aware sampling (RS) and momentum-based memory bank (MMB). Inspired by the fact that existing uncertainty-based AL methods often end up with choosing uncertain yet repetitive data [15, 27], we design the region-aware sampling technique to promote both diversity and representativeness by leveraging the representation power of PLMs. Specifically, we first estimate the uncertainties of the unlabeled data with PLMs, then cluster the data using their PLM representations and weigh the data by the corresponding uncertainty. Such a clustering scheme partitions the embedding space into small sub-regions with an emphasis on highly-uncertain samples. Finally, by sampling over multiple high-uncertainty regions, our strategy selects data with high uncertainty and low redundancy.

To rectify the erroneous pseudo labels derived by self-training, we design a simple but effective way to select low-uncertainty data for self-training. Our method is motivated by the fact that fine-tuning PLMs suffer from instability issues -- different initializations and data orders can lead to large variance in model performance [30, 48, 69]. However, previous approaches only select pseudo-labeled data based on the prediction of the current round and are thus less reliable. In contrast, we maintain a dynamic memory bank to save the predictions of unlabeled samples for later use. We propose a momentum updating method to dynamically aggregate the predictions from preceding rounds [10] and select low-uncertainty samples based on aggregated prediction. As a result, only the samples with high prediction confidence over multiple rounds will be used for self-training, which mitigates the issue of label noise. We highlight that our active self-training approach is an efficient substitution to existing AL methods, requiring little extra computational cost.

Our key contributions are: (1) an active self-training paradigm AcTune that integrates self-training and active learning to minimize the labeling cost for fine-tuning PLMs; (2) a region-aware querying strategy to enforce both the informativeness and the diversity of queried samples during AL; (3) a simple and effective momentum-based method to leverage the predictions in preceding rounds to alleviate the label noise in self-training and (4) experiments on 6 benchmarks demonstrating AcTune improves the label efficiency over existing self-training and active learning baselines by 56.2%.

## 2 Uncertainty-aware Active Self-training

### Problem Formulation

We study active fine-tuning of pre-trained language models for text classification, formulated as follows: Given a small number of labeled samples \({\cal X}_{l}=\left\{(x_{i},y_{i})\right\}^{L}_{i=1}\) and unlabeled samples \({\cal X}_{u}=\left\{x_{j}\right\}^{U}_{j=1}\) (\(|{\cal X}_{l}|<|{\cal X}_{u}|\)), we aim to fine-tune a pre-trained language model \(f(\mathbf{x};\theta):{\cal X}\rightarrow{\cal Y}\) in an interactive way: we perform active self-training for \(T\) rounds with the total labeling budget \(b\). In each round, we aim to query \(B=b/T\) samples denoted as \({\cal B}\) from \({\cal X}_{u}\) to fine-tune a pre-trained language model \(f(\mathbf{x};\theta)\) with both \({\cal X}_{l},{\cal B}\) and \({\cal X}_{u}\) to maximize the performance on downstream text classification tasks. Here \({\cal X}={\cal X}_{l}\cup{\cal X}_{u}\) denotes all samples, and \({\cal Y}=\left\{1,2,\cdots,C\right\}\) is the label set where \(C\) is the number of classes.

### Overview of AcTune Framework

We now present our active self-training paradigm AcTune underpinned by estimated uncertainty. We begin the active self-training loop by fine-tuning a BERT \(f(\theta^{(0)})\) on the initial labeled data \({\cal X}_{L}\). Formally, we solve the following optimization problem

\[\min_{\theta}~{}\frac{1}{|{\cal X}_{L}|}\sum_{(\mathbf{x}_{i},y_{i})\in{\cal X}_{ L}}\ell_{\text{CE}}\left(f(\mathbf{x}_{i};\theta^{(0)}),y_{i}\right). \tag{1}\]

In round \(t\) (\(1\leq t\leq T\)) of active self-training, we first calculate the uncertainty score based on a given function \(a^{(t)}_{i}=a(\mathbf{x}_{i},\theta^{(t)})\)1 for all \(\mathbf{x}_{i}\in{\cal X}_{u}\). Then, we query labeled samples and generate pseudo-labels for unlabeled data \({\cal X}_{u}\) simultaneously to facilitate self-training. For each sample \(\mathbf{x}_{i}\), the pseudo-label \(\widetilde{y}\) is calculated based on the current 

model's output:

\[\widetilde{y}=\operatorname*{argmax}_{j\in\mathcal{Y}}\left[f(\mathbf{x}; \theta^{(t)})\right]_{j}, \tag{2}\]

where \(f(\mathbf{x};\theta^{(t)})\in\mathbb{R}^{C}\) is a probability simplex and \([f(\mathbf{x};\theta^{(t)})]_{j}\) is the \(j\)-th entry. The procedure of AcTune is summarized in Algorithm 1.

```
Input: Initial labeled samples \(\mathcal{X}_{l}\); Unlabeled samples \(\mathcal{X}_{u}\); Pre-trained LM \(f(\cdot;\theta)\), number of active self-training rounds \(T\).// Fine-tune the LM with initial labeled data.
1. Calculate \(\theta^{(0)}\) based on Eq. (1).
2. Initialize the memory bank \(g(\mathbf{x};\theta^{t})\) based on the current prediction. If \(\mathbf{c}_{i}\) are set to \(\mathbf{x}_{i}\)do
3. Compute the memory bank \(g(\mathbf{x};\theta^{t})\) by \(\mathcal{L}_{\text{SF}}\) ( \(\mathbf{x}_{i}\) ) using AdamW.
4. Update memory bank \(g(\mathbf{x};\theta^{t})\) with Eq. (1) or (1). Output: The final fine-tuned model \(f(\cdot;\theta^{T})\).
```

**Algorithm 1**Training Procedures of AcTune.

### Region-aware Sampling for Active Learning on High-uncertainty Data

After obtaining the uncertainty for unlabeled data, we aim to query annotation for high-uncertainty samples. However, directly sampling the most uncertain samples gives suboptimal results as it tends to query repetitive data (Ein-Dor et al., 2020) that represent the overall data distribution poorly.

To tackle this issue, we propose region-aware sampling to capture both _uncertainty_ and _diversity_ during active self-training. Specifically, in the \(t\)-th round, we first conduct the weighted K-means clustering (Huang et al., 2005), which weights samples based on their uncertainty. Denote by \(K\) the number of clusters and \(\mathbf{v}_{i}^{(t)}=\text{BERT}(\mathbf{x}_{i})\) the representation of \(\mathbf{x}_{i}\) from the penultimate layer of BERT. The weighted K-means process first initializes the center of each each cluster \(\mathbf{\mu}_{i}(1\leq i\leq K)\) via K-Means++ (Arthur and Vassilvitskii, 2007). Then, it jointly updates the centroid of each cluster and assigns each sample to cluster \(c_{i}\) as

\[c_{i}^{(t)}=\operatorname*{argmin}_{k=1,...,K}\|\mathbf{v}_{i}-\mathbf{\mu}_{k}\|^{2}\,, \tag{3}\]

\[\mathbf{\mu}_{k}^{(t)}=\frac{\sum_{\mathbf{x}_{i}\in\mathcal{C}_{k}^{(t)}}a(\mathbf{x}_{i },\theta^{(t)})\cdot\mathbf{v}_{i}^{(t)}}{\sum_{\mathbf{x}\in\mathcal{C}_{k}^{(t)}}a( \mathbf{x}_{i},\theta^{(t)})} \tag{4}\]

where \(\mathcal{C}_{k}^{(t)}=\{\mathbf{x}_{i}^{(t)}|c_{i}^{(t)}=k\}(k=1,\dots,K)\) stands for the \(k\)-th cluster. The above two steps in Eq. (3), (4) are repeated until convergence. Compared with vanilla K-Means method, the weighting scheme increases the density of the samples with high uncertainty, thus enabling the K-Means methods to discover clusters with high uncertainty. After obtaining \(K\) regions with the corresponding data \(\mathcal{C}_{k}^{(t)}\), we calculate the uncertainty of each region as

\[u_{k}^{(t)}=U(\mathcal{C}_{k}^{(t)})+\beta I(\mathcal{C}_{k}^{(t)}) \tag{5}\]

where

\[U(\mathcal{C}_{k}^{(t)})=\frac{1}{|\mathcal{C}_{k}^{(t)}|}\sum_{\mathbf{x}_{i}\in \mathcal{C}_{k}^{(t)}}a(\mathbf{x}_{i},\theta^{(t)}), \tag{6}\]

is the average uncertainty of samples and

\[I(\mathcal{C}_{k}^{(t)})=-\sum_{j\in C}f_{j}^{(t)}\log f_{j}^{(t)} \tag{7}\]

is the inter-class diversity within cluster \(k\) and \(f_{j}^{(t)}=\frac{\sum_{i}1\overline{\{y}_{i}=j\}}{|\mathcal{C}_{k}^{(t)}|}\) is the frequency of class \(j\) on cluster \(k\). Notably, the term \(U(\mathcal{C}_{k}^{(t)})\) assigns higher score for clusters with more uncertain samples, and \(I(\mathcal{C}_{k}^{(t)})\) grants higher scores for clusters containing samples with more diverse predicted classes from pseudo labels since such clusters would be closer to the decision boundary.

Then, we rank the clusters in an ascending order in \(u_{k}^{(t)}\). A high score indicates high uncertainty of the model in these regions, and we need to actively annotate the member instances to reduce uncertainty and improve the model's performance. We adopt a hierarchical sampling strategy: we first select the \(M\) clusters with the highest uncertainty, and then sample \(b^{\prime}=\lfloor\frac{B}{M}\rfloor\) data with the highest uncertainty to form the batch \(\mathcal{Q}^{(t)}\).2

Footnote 2: If the number of samples in the \(i\)-th cluster \(\mathcal{C}_{i}\) is smaller than \(b^{\prime}\), then we sample all the data within \(\mathcal{C}_{i}\) and increase the budget for the \((i+1)\)-th cluster by \(b^{\prime}-|\mathcal{C}_{i}|\).

\[\begin{split}&\mathcal{K}_{a}^{(t)}=\underset{k\in\{1,...,K\}}{ \text{top-M}}~{}u_{k}^{(t)},\\ &\mathcal{Q}^{(t)}=\bigcup_{k\in\mathcal{K}_{a}^{(t)}}\mathcal{C} _{a,k}^{(t)}~{}\mathrm{where}~{}\mathcal{C}_{a,k}^{(t)}=\underset{\mathbf{x}_{i}\in \mathcal{C}_{k}^{(t)}}{\text{Top-b}^{\prime}}~{}a(\mathbf{x}_{i},\theta^{(t)}). \end{split} \tag{8}\]

We remark that such a hierarchical sampling strategy queries most uncertain samples from _different_ regions, thus the uncertainty and diversity of queried samples can be both achieved 



### Self-training over Confident Samples from Low-uncertainty Regions

For self-training, we aim to select unlabeled samples which are _most likely_ to have been correctly classified by the current model. This requires the sample to have low uncertainty. Therefore, we select the top \(k\) samples from the \(M\) lowest uncertainty regions to form the acquired batch \({\mathcal{S}^{(t)}}\):

\[\begin{split}&{\mathcal{C}^{(t)}_{s}=\bigcup_{k\in{\mathcal{ C}^{(t)}_{k}}}{\mathcal{C}^{(t)}_{k}}~{}\text{where}~{}{\mathcal{K}^{(t)}_{ s}=\operatorname*{bottom-M}_{k}u^{(t)}_{k}},\\ &{\mathcal{S}^{(t)}=\operatorname*{bottom-k}_{\mathbf{x}_{i}\in{ \mathcal{C}^{(t)}_{s}}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!

of the memory bank \(g({\mathbf{x}};\theta)\) is proportional to \(|\mathcal{X}_{u}|\), while the extra computation of maintaining this dictionary is _ignorable_ since we do not inference over the unlabeled data for multiple times in each round as BALD Gal et al. (2017) does. The running time of AcTune will be shown in section C.

## 3 Experiments

### Experiment Setup

**Tasks and Datasets.** In our main experiments, we use 4 datasets, including _SST-2_Socher et al. (2013) for sentiment analysis, _AGNews_Zhang et al. (2015) for news topic classification, _PubmedRCT_Dernoncourt and Lee (2017) for medical abstract classification, and _DBpedia_Zhang et al. (2015) for wikipedia topic classification. For weakly-supervised text classification, we choose 2 datasets, namely _TREC_Li and Roth (2002) and _Chemprot_Krallinger et al. (2017) from the WRENCH benchmark Zhang et al. (2021) for evaluation. The statistics are shown in Table 1.

**Active Learning Setups.** Following Yuan et al. (2020), we set the number of rounds \(T=10\), the overall budget for all datasets \(b=1000\) and the initial size of the labeled \(|\mathcal{X}_{l}|\) is set to 100. In each AL round, we sample a batch of 100 samples from the unlabeled set \(\mathcal{X}_{u}\) and query their labels. Since large development sets are impractical in low-resource settings Kann et al. (2019), we keep the size of development set as 1000, which is the same as the labeling budget4. For weakly-supervised text classification, since the datasets are much smaller, we keep the labeling budget and the size of development set to \(b=500\).

Footnote 4: This is often neglected in previous low-resource AL studies, and we highlight it to ensure the true low-resource setting.

**Implementation Details.** We choose RoBERTbase Liu et al. (2019) from the HuggingFace codebase Wolf et al. (2020) as the backbone for AcTune and all baselines except for Pubmed and Chemprot, where we use SciBERT Beltagy et al. (2019), a BERT model pre-trained on scientific corpora. In each round, we train from scratch to avoid overfitting the data collected in earlier rounds as observed by Hu et al. (2019). More details are in Appendix B.

**Hyperparameters.** The hyperparameters setting is in SectionB.5. In the \(t\)-th round of active self-training, we increase the number of pseudo-labeled samples by \(k\), where \(k\) is \(500\) for TREC and Chemprot, \(3000\) for SST-2 and Pubmed-RCT, and \(5000\) for others. For the momentum factor, we tune \(m_{L}\) from \([0.6,0.7,0.8]\) and \(m_{H}\) from \([0.8,0.9,1.0]\) and report the best \(\{m_{L},m_{H}\}\) based on the performance of the development set.

**Baselines.**

**Self-training Methods:** (1) **Self-training (ST, Lee (2013))**: It is the vanilla self-training method that generates pseudo labels for unlabeled data. (2) **UST**Mukherjee and Awadallah (2020); Rizve et al. (2021): It is an uncertainty-based self-training method that only uses low-uncertainty data for self-training. (3) **COSINE**Yu et al. (2021): It uses self-training to fine-tune LM with weakly-labeled data, which achieves SOTA performance on various text datasets in WRENCH benchmark Zhang et al. (2021). Note that for these two baselines, we _randomly sample_\(b\) labeled data as the initialization.

**Active Learning Methods:** (1) **Random**: It acquires annotation randomly, which serves as a baseline for all methods. (2) **Entropy**Holub et al. (2008): It is an uncertainty-based method that acquires annotations on samples with the highest predictive entropy. (3) **BALD**Gal et al. (2017): It is also an uncertainty-based method, which calculates _model uncertainty_ using MC Dropout Gal and Ghahramani (2015). (4) **BADGE**Ash et al. (2020): It first selects high uncertainty samples then uses KMeans++ over the gradient embedding to sample data. (5) **ALPS**Yuan et al. (2020): It uses the masked language model (MLM) loss of BERT to query labels for samples. (6) **CAL**Margatina et al. (2021) is the most recent AL method for pretrained LMs. It calculates the uncertainty of each sample based on the KL divergence between the prediction of itself and its neighbors' prediction.

**Semi-supervised Active Learning (SSAL) Methods:** (1) **ASST**Tomanek and Hahn (2009); Simeoni et al. (2020) is an active semi-supervised learning method that jointly queries labels for AL and samples pseudo labels for self-training. (2) **CEAL**Wang et al. (2016) acquires annotations on informative samples, and uses high-confidence

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline
**Dataset** & **Label Type** & **\# Class** & **\# Train** & **\# Dev** & **\#Test** \\ \hline SST-2 & Sentiment & 2 & 60.6k & 0.8k & 1.8k \\ AG News & News Topic & 4 & 119k & 1k & 7.6k \\ Pubmed & Medical Abstract & 5 & 180k & 1k & 30.1k \\ DBpedia & Wikipedia Topic & 14 & 280k & 1k & 70k \\ \hline TREC & Question & 6 & 5.0k & 0.5k & 0.5k \\ Chemprot & Medical Abstract & 10 & 12.8k & 0.5k & 1.6k \\ \hline \end{tabular}
\end{table}
Table 1: Dataset Statistics. For DBPedia, we randomly sample 20k sample from each class due to the limited computational resource.

 

samples with predicted pseudo labels for weights updating. (3) **BASS**[14] is similar to CEAL, but use MC dropout for querying labeled sample. (4) **REVIVAL**[20] is the most recent SSAL method, which uses an adversarial loss to query samples and leverage label propagation to exploit adversarial examples.

**Our Method**: We experiment with both Entropy and CAL as uncertainty measures for AcTune. Note that when compared with active learning baselines, we do not augment the train set with pseudo-labeled data (Eq. (9)) to ensure fair comparisons.

### Main Result

Figure 1 reports the performance of AcTune and the baselines on 4 benchmarks. From the results, we have the following observations:

\(\bullet\)AcTune consistently outperforms baselines in most of the cases. Different from studies in the computer vision (CV) domain [56] where the model does not perform well in the low-data regime, pre-trained LM has achieved competitive performance with only a few labeled data, which makes further improvements to the vanilla fine-tuning challenging. Nevertheless, AcTune surpasses baselines in more than 90% of the rounds and achieves 0.4%-0.7% and 0.3%-1.5% absolute gain at the end of AL and SSAL respectively. Figure 3 quantitatively measures the number of labels needed for the most advanced active learning model and self-training model (UST) to outperform ActTune with 1000 labels. These baselines need >2000 clean labeled samples to reach the performance as ours. AcTune saves on average **56.2%** and **57.0%** of the labeled samples than most advanced active learning and self-training baselines respectively, which justifies its promising performance under low-resource scenarios. Such improvements show the merits of two key designs under our active self-training framework: the region-aware sampling for active learning and the momentum-based memory bank for robust self-training, which will be discussed in the section 3.5. \(\bullet\) Compared with the previous AL baselines, AcTune can bring consistent performance gain, while previous semi-supervised active learning methods cannot. For instance, BASS is based on BALD for active learning, but sometimes it performs even worse than BALD with the same number of labeled data (see Fig. 1(b) and Fig. 1(f)). This is mainly because previous methods simply combine noisy pseudo labels with clean labels for training without explicitly rectifying the wrongly-labeled data, which will cause the LM to overfit these hazardous labels. Moreover, previous methods do not exploit momentum updates to stabilize the learning process, as there are oscillations in the beginning rounds. In contrast, AcTune achieves a more stable learning process and enables an active self-training process to benefit from more labeled data. \(\bullet\) The self-training methods (ST & UST) achieve superior performance with limited labels. However, they mainly focus on leveraging unlabeled data for improving the performance, while our results demonstrate that adaptive selecting the most useful data for fine-tuning is also important for improving the performance. With a powerful querying policy, AcTune can improve these self-training baselines by 1.05% in terms of accuracy on average.

### Weakly-supervised Learning

ActTune can be naturally used for weakly-supervised classification, where \({\cal X}_{l}\) is a set of noisy labels derived from linguistic patterns or rules. Since the initial label set is noisy, the model trained with Eq. (1) can overfit the label noise [81], and we can actively query labeled data to refine the model. We conduct experiments on the TREC and Chemprot dataset5, where we first use Snorkel [10] to obtain weak label set \({\cal X}_{l}\), then fine-tune the pre-trained LM \(f(\theta^{(0)})\) on \({\cal X}_{l}\). After that, we adopt AcTune for active self-training.

Footnote 5: Details for labeling functions are in Zhang et al. (2021).

Fig. 2 shows the results of these two datasets6. When combining AcTune with CAL, the performance is unsatisfactory. We believe it is because CAL requires clean labels to calculate uncertainties. When labels are inaccurate, it will prevent Acute from querying informative samples. In contrast, AcTune achieves the best performance over baselines when using Entropy as the uncertainty measure. The performance gain is more notable on the TREC dataset, where we achieve 96.68% accuracy, close to the fully supervised performance (96.80%) with only \(\sim\)6% of clean labels.

Footnote 6: We omit AL methods since they perform worse than SSAL methods on these datasets in general.

### Combination with Other AL Methods

Fig. 5 demonstrates the performance of AcTune combined with other AL methods (e.g. BADGE, ALPS) on SST-2 dataset. It is clear that even if the AL methods are not uncertainty-based 

(e.g. BADGE), when using the _entropy_ as an uncertainty measure to select pseudo-labeled data for self-training, AcTune can further boost the performance. This indicates that AcTune is a general active self-training approach, as it can serve as an efficient plug-in module for existing AL methods.

### Ablation and Hyperparameter Study

**The Effect of Different Components in AcTune.** We inspect different components of AcTune, including the region-sampling (RS), momentum-based memory bank (MMB), and weighted clustering (WClus)7. Experimental results (Fig. 5(b)) shows that all the three components contribute to the final performance, as removing any of them hurts the classification accuracy. Also, we find that when removing MMB, the performance hurts most in the beginning rounds, which indicates that MMB effectively suppresses label noise when the model's capacity is weak. Conversely, removing WClus hurts the performance on later rounds, as it enables the model to select most informative samples.

Footnote 7: For models w/o RS, we directly select samples with highest uncertainty during AL. For models w/o MMB, we only use the prediction from the current round for self-training. For models w/o WClus, we cluster data with vanilla K-Means.

**Hyperparameter Study.** We study two hyperparameters, namely \(\beta\) and \(K\) used in querying labels. Figure 4(a) and 4(b) show the results. In general, the model is insensitive to \(\beta\) as the performance difference is less than 0.6%. The model cannot perform well with smaller \(K\) since it cannot pinpoint to high-uncertainty regions. For larger \(K\), the performance also drops as some of the high-uncertainty regions can be outliers and sampling from them would hurt the model performance [18].

**A Closer Look at the Momentum-based Memory Bank.** To examine the role of MMB, we show the overall accuracy of pseudo-labels on AG News dataset in Fig. 4(c). From the result, it is clear that the momentum-based memory bank can stabilize the active self-training process, as the accuracy of pseudo labels increases around 1%, especially in

Figure 3: The label-efficiency of AcTune compared with AL and self-training baselines. According to Fig. 1, the best AL method is Entropy for DBPedia and CAL for others.

Figure 2: The comparison of AcTune and baselines on weakly-supervised classification tasks.

 

later rounds. Fig 4(d) and 4(e) illustrates the model performance with different \(m_{L}\) and \(m_{H}\). Overall, we find that our model is robust to different choices as AcTune outperform the baseline without momentum update consistently. Moreover, we find that the larger \(m_{H}\) will generally lead to better performance in later rounds. This is mainly because in later rounds, the model's prediction is more reliable. Conversely, at the beginning of the training, the model's prediction might be oscillating on unlabeled data. In this case, using a smaller \(m_{L}\) will favor samples with consistent predictions to improve the robustness of active self-training.

Another finding is that for different AL methods, the optimal memory bank can be different. For Entropy, probability-based memory bank leads to a better result, while for CAL, simple aggregating over uncertainty score achieves better performance. This is mainly because the method used in CAL is more complicated, and using probability-based memory bank may hurt the uncertainty calculation.

### Case Study

We give an example of our querying strategy on AG News dataset for the 1st round of active self-training process in figure 6. Note that we use t-SNE algorithm Van der Maaten and Hinton (2008) for dimension reduction, and the black triangle stands for the queried samples while other circles stands for the unlabeled data. We can see that the existing uncertainty-based methods such as Entropy and CAL, are suffered from the issue of limited diversity. However, when combined with ActTune, the diversity is much improved. Such results, compared with the main results in figure 1, demonstrate the efficacy of AcTune empirically.

## 4 Related Work

**Active Learning.** Active learning has been widely applied to various NLP tasks Yuan et al. (2020); Shelmanov et al. (2021); Karamcheti et al. (2021). So far, AL methods can be categorized into uncertainty-based methods Gal et al. (2017); Margatina et al. (2021a, b), diversity-based methods Ru et al. (2020); Sener and Savarese (2018) and hybrid methods Yuan et al. (2020); Ash et al. (2020). Ein-Dor et al. (2020) offer an empirical study of active learning with PLMs. Very recently, there are also several works attempted to query labeling functions for weakly-supervised learning Boecking et al. (2020); Hsieh et al. (2022); Zhang et al. (2022b). In our study, we leverage the power of unlabeled instances via self-training to further promote the performance of AL.

**Semi-supervised Active Learning (SSAL).** Gao et al. (2020); Guo et al. (2021) design query strategies for specific semi-supervised methods, Zhang et al. (2020); Jiang et al. (2020) combine active learning with data augmentation and Tomanek and Hahn (2009); Rottmann et al. (2018); Simconi et al. (2020) exploit the most-certain samples from the unlabeled with pseudo-labeling to augment the training set. So far, most of the SSAL approaches are designed for CV domain and it remains unknown how this paradigm performs with PLMs on NLP tasks. In contrast, we propose AcTune to effectively leverage unlabeled data during finetuing PLMs for NLP tasks.

**Self-training.** Self-training is one of the earliest and simplest approaches to semi-supervised learning Lee (2013). It first generates pseudo labels for high-confidence samples, then fits a new model on pseudo labeled data to improve the generalization ability. However, it is known to be vulnerable to error propagation Arazo et al. (2020); Rizve et al. (2021); Zuo et al. (2021). To alleviate this, we adopt a simple momentum-based method to select high confidence samples, effectively reducing

Figure 4: Parameter study. Note the effect of different \(m_{L}\) and \(m_{H}\) is conducted on AG News dataset.

Figure 5: Results of AcTune with different AL methods (SST-2), ablation study (SST-2 with AcTune+Entropy).

 

the pseudo labels noise for active learning. Note that although Mukherjee and Awadallah (2020); Rizve et al. (2021) also leverage uncertainty information for self-training, their focus is on developing better self-training methods, while we aim to jointly query high-uncertainty samples and generate pseudo-labels for low-uncertainty samples. The experiments in Sec. 3 show that with appropriate querying methods, AcTune can further improve the performance of self-training.

## 5 Conclusion and Discussion

In this paper, we develop AcTune, a general active self-training framework for enhancing both label efficiency and model performance in fine-tuning pre-trained language models (PLMs). We propose a region-aware sampling approach to guarantee both the uncertainty the diversity for querying labels. To combat the label noise propagation issue, we design a momentum-based memory bank to effectively utilize the model predictions for preceding AL rounds. Empirical results on 6 public text classification benchmarks suggest the superiority of AcTune to conventional active learning and semi-supervised active learning methods for fine-tuning PLMs with limited resources.

There are several directions to improve AcTune. First, since our focus is on fine-tuning pre-trained language models, we use the representation of [CLS] token for classification. In the future work, we can consider using prompt tuning Gao et al. (2021); Schick and Schutze (2021), a more data-efficient method for adopting pre-trained language models on classification tasks to further promote the efficiency. Also, due to the computational resource constraints, we do not use larger pre-trained language models such as RoBERTa-large Liu et al. (2019) which shown even better performance with only a few labels Du et al. (2021). Moreover, we can explore more advanced uncertainty estimation approach Kong et al. (2020) into AcTune to further improve the performance. Last, apart from the text classification task, we can also extend our work into other tasks such as sequence labeling and natural language inference (NLI).

## Acknowledgements

We thank the anonymous reviewers for their feedback. This work was supported in part by NSF IIS-2008334, IIS-2106961, CAREER IIS-2144338, and ONR MURI N00014-17-1-2656.

Figure 6: Visualization of the querying strategy of AcTune. Black dots stand for the queried data points. Different colors indicates different categories.

 



## References

* Arazo et al. (2020) Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. 2020. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In _2020 International Joint Conference on Neural Networks (IJCNN)_, pages 1-8. IEEE.
* Arthur and Vassilvitskii (2007) David Arthur and Sergei Vassilvitskii. 2007. K-means++: The advantages of careful seeding. In _Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms_, page 1027-1035, USA.
* Ash et al. (2020) Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2020. Deep batch active learning by diverse, uncertain gradient lower bounds. In _International Conference on Learning Representations_.
* Bansal et al. (2020) Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020. Self-supervised meta-learning for few-shot natural language classification tasks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 522-534, Online. Association for Computational Linguistics.
* Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3615-3620.
* Boecking et al. (2020) Benedkit Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. 2020. Interactive weak supervision: Learning useful heuristics for data labeling. _arXiv preprint arXiv:2012.06046_.
* Bragg et al. (2021) Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. Flex: Unifying evaluation for few-shot nlp. _Advances in Neural Information Processing Systems_, 34.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, volume 33, pages 1877-1901.
* Dernoncourt and Lee (2017) Franck Dernoncourt and Ji Young Lee. 2017. PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 308-313, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. _CoRR_, abs/2002.06305.
* Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1383-1392.
* Du et al. (2021) Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021. Self-training improves pre-training for natural language understanding. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5408-5418. Association for Computational Linguistics.
* Ein-Dor et al. (2020) Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7949-7962. Association for Computational Linguistics.
* Gal and Shahramani (2015) Yarin Gal and Zoubin Shahramani. 2015. Bayesian convolutional neural networks with bernoulli approximate variational inference. _CoRR_, abs/1506.02158.
* Gal et al. (2017) Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep bayesian active learning with image data. In _International Conference on Machine Learning_, pages 1183-1192. PMLR.
* Gao et al. (2020) Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan O Ark, Larry S Davis, and Tomas Pfister. 2020. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. In _European Conference on Computer Vision_, pages 510-526. Springer.
* Gao et al. (2021) Tianyu Gao, Adam Fisch, and Dani Chen. 2021. Making pre-trained language models better few-shot learners. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3816-3830, Online. Association for Computational Linguistics.
* Guo et al. (2020) Jiannan Guo, Haochen Shi, Yangyang Kang, Kun Kuang, Siliang Tang, Zhuoren Jiang, Changlong

Sun, Fei Wu, and Yueting Zhuang. 2021. Semi-supervised active learning for semi-supervised models: Exploit adversarial examples with graph-based virtual labels. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_, pages 2896-2905.
* He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Holub et al. (2008) Alex Holub, Pietro Perona, and Michael C Burl. 2008. Entropy-based active learning for object recognition. In _2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_, pages 1-8. IEEE.
* Hsieh et al. (2022) Cheng-Yu Hsieh, Jieyu Zhang, and Alexander Ratner. 2022. Nemo: Guiding and contextualizing weak supervision for interactive data programming. _arXiv preprint arXiv:2203.01382_.
* Hu et al. (2019) Peiyun Hu, Zack Lipton, Anima Anandkumar, and Deva Ramanan. 2019. Active learning with partial feedback. In _International Conference on Learning Representations_.
* Huang et al. (2005) Joshua Zhexue Huang, Michael K Ng, Hongqiang Rong, and Zichen Li. 2005. Automated variable weighting in k-means type clustering. _IEEE transactions on pattern analysis and machine intelligence_, 27(5):657-668.
* Jiang et al. (2020) Zhuoren Jiang, Zhe Gao, Yu Duan, Yangyang Kang, Changlong Sun, Qiong Zhang, and Xiaozhong Liu. 2020. Camouflaged Chinese spam content detection with semi-supervised generative active learning. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3080-3085, Online. Association for Computational Linguistics.
* Kann et al. (2019) Katharina Kann, Kyunghyun Cho, and Samuel R. Bowman. 2019. Towards realistic practices in low-resource natural language processing: The development set. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3342-3349, Hong Kong, China. Association for Computational Linguistics.
* Karamcheti et al. (2021) Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. 2021. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 7265-7281, Online. Association for Computational Linguistics.
* Kong et al. (2020) Lingkai Kong, Jimeng Sun, and Chao Zhang. 2020. Sde-net: Equipping deep neural networks with uncertainty estimates. In _International Conference on Machine Learning_, pages 5405-5415. PMLR.
* Krallinger et al. (2017) Martin Krallinger, Obdulia Rabal, Saber A Akhondi, et al. 2017. Overview of the biocreative VI chemical-protein interaction track. In _BioCreative evaluation Workshop_, volume 1, pages 141-146.
* Laine and Aila (2016) Samuli Laine and Timo Aila. 2016. Temporal ensembling for semi-supervised learning. _arXiv preprint arXiv:1610.02242_.
* Lee (2013) Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In _ICML Workshop on challenges in representation learning_, volume 3, page 896.
* Li and Roth (2002) Xin Li and Dan Roth. 2002. Learning question classifiers. In _The 19th International Conference on Computational Linguistics_.
* Liang et al. (2020) Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1054-1064.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Margatina et al. (2021a) Katarina Margatina, Loic Barrault, and Nikolaos Aletras. 2021a. Bayesian active learning with pretrained language models. _arXiv preprint arXiv:2104.08320_.
* Margatina et al. (2021b) Katarina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021b. Active learning by acquiring contrastive examples. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 650-663, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines. In _International Conference on Learning Representations_.
* Mukherjee and Awadallah (2020) Subhabrata Mukherjee and Ahmed Awadallah. 2020. Uncertainty-aware self-training for few-shot text classification. _Advances in Neural Information Processing Systems_, 33.
* Ratner et al. (2017) Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Re. 2017. Snorkel: Rapid training data creation with weak supervision. In _Proceedings of the VLDB Endowment._, volume 11, page 269 

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. 2021. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In _International Conference on Learning Representations_.
* Rottmann et al. (2018) Matthias Rottmann, Karsten Kahl, and Hanno Gottschalk. 2018. Deep bayesian active semi-supervised learning. In _2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA)_, pages 158-164. IEEE.
* Ru et al. (2020) Dongyu Ru, Jiangtao Feng, Lin Qiu, Hao Zhou, Mingxuan Wang, Weinan Zhang, Yong Yu, and Lei Li. 2020. Active sentence learning by adversarial uncertainty sampling in discrete space. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4908-4917, Online. Association for Computational Linguistics.
* Schick and Schutze (2021) Timo Schick and Hinrich Schutze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 255-269, Online. Association for Computational Linguistics.
* Sener and Savarese (2018) Ozan Sener and Silvio Savarese. 2018. Active learning for convolutional neural networks: A core-set approach. In _International Conference on Learning Representations_.
* Shelmanov et al. (2021) Artem Shellanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, and Alexander Panchenko. 2021. Active learning for sequence tagging with deep pre-trained models and Bayesian uncertainty estimates. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1698-1712, Online. Association for Computational Linguistics.
* Simoni et al. (2020) Oriane Simoni, Mateusz Budnik, Yannis Avrithis, and Guillaume Gravier. 2020. Rethinking deep active learning: Using unlabeled data at model training. In _the 25th International Conference on Pattern Recognition (ICPR)_, pages 1220-1227. IEEE.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642. Association for Computational Linguistics.
* Tomanek and Hahn (2009) Katrin Tomanek and Udo Hahn. 2009. Semi-supervised active learning for sequence labeling. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 1039-1047.
* Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(11).
* Wang et al. (2020) Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. 2016. Cost-effective active learning for deep image classification. _IEEE Transactions on Circuits and Systems for Video Technology_, 27(12):2591-2600.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and Others. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. _Advances in Neural Information Processing Systems_, 33.
* Yu et al. (2021) Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. 2021. Fine-tuning pretrained language model with weak supervision: A contrastive-regularized self-training approach. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1063-1077. Association for Computational Linguistics.
* Yuan et al. (2020) Michelle Yuan, Hsuan-Tien Lin, and Jordan BoydGraber. 2020. Cold-start active learning through self-supervised language modeling. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7935-7948, Online. Association for Computational Linguistics.
* Zhang et al. (2022a) Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. 2022a. A survey on programmatic weak supervision. _arXiv preprint arXiv:2202.05433_.
* Zhang et al. (2021) Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. 2021. WRENCH: A comprehensive benchmark for weak supervision. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Zhang et al. (2022b) Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang. 2022b. Prboost: Prompt-based rule discovery and boosting for interactive weakly-supervised learning. _arXiv preprint arXiv:2203.09735_ 

Rongzhi Zhang, Yue Yu, and Chao Zhang. 2020a. SeqMix: Augmenting active sequence labeling via sequence mixup. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8566-8579, Online. Association for Computational Linguistics.
* Zhang et al. [2020b] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artemzi. 2020b. Revisiting few-sample bert fine-tuning. _arXiv preprint arXiv:2006.05987_.
* Zhang et al. [2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. _Advances in neural information processing systems_, 28:649-657.
* Zuo et al. [2021] Simiao Zuo, Yue Yu, Chen Liang, Haoming Jiang, Siawpeng Er, Chao Zhang, Tuo Zhao, and Hongyuan Zha. 2021. Self-training with differentiable teacher. _arXiv preprint arXiv:2109.07049_.

 

Datasets Details

### Data Source

The seven benchmarks in our experiments are all publicly available. Below are the links to downloadable versions of these datasets.

\(\diamond\)_SST-2_: We use the datasets from [https://huggingface.co/datasets/glue](https://huggingface.co/datasets/glue).

\(\diamond\)_AGNews_: We use the datasets from [https://huggingface.co/datasets/ag_news](https://huggingface.co/datasets/ag_news).

\(\diamond\)_Pubmed-RCT_: Dataset is available at [https://github.com/Franck-Dernoncourt/pubmed-rct](https://github.com/Franck-Dernoncourt/pubmed-rct).

\(\diamond\)_DBPedia_: Dataset is available at [https://huggingface.co/datasets/dbpedia_14](https://huggingface.co/datasets/dbpedia_14).

For two weakly-supervised classification tasks, we use the data from WRENCH benchmark [30].

\(\diamond\)_TREC_: Dataset is available at [https://drive.google.com/drive/u/1/folders/lv55IKG2JN9fMtKJWW48B_5_DCPWGnpTq](https://drive.google.com/drive/u/1/folders/lv55IKG2JN9fMtKJWW48B_5_DCPWGnpTq).

\(\diamond\)_ChemProt_: The raw dataset is available at [http://www.cbs.dtu.dk/services/ChemProt/ChemProt-2.0/](http://www.cbs.dtu.dk/services/ChemProt/ChemProt-2.0/). The preprocessed dataset is available at [https://drive.google.com/drive/u/1/folders/lv55IIKG2JN9fMtKJWU48B_5_DCPWGnpTq](https://drive.google.com/drive/u/1/folders/lv55IIKG2JN9fMtKJWU48B_5_DCPWGnpTq).

### Train/Test Split

For all the datasets, we use the original train/dev/test split from the web. To keep the size of the development set small, we randomly sample 1000 data for _SST-2, AGNews, Pubmed-RCT, DBPedia_ and randomly sample 500 samples for _TREC, ChemProt_.

## Appendix B Details on Implementation and Experiment Setups

### Computing Infrastructure

_System_: Ubuntu 18.04.3 LTS; Python 3.6; Pytorch 1.6.

_CPU_: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz.

_GPU_: NVIDIA 2080Ti.

### Number of Parameters

AcTune and all baselines use Roberta-base Liu et al. (2019) with a task-specific classification head on the top as the backbone, which contains 125M trainable parameters. We do not introduce any other parameters in our experiments.

### Experiment Setups

Following Ein-Dor et al. (2020); Yuan et al. (2020); Margatina et al. (2021), all of our methods and baselines are run with 3 different random seed and the result is based on the average performance on them. This indeed creates \(4\) (the number of datasets) \(\times 3\) (the number of random seeds) \(\times 11\) (the number of methods) \(\times 10\) (the number of fine-tuning rounds in AL) = \(1320\) experiments for fine-tuning PLMs, which is almost the limit of our computational resources, not to mention additional experiments on weakly-supervised text classification as well as different hyper-parameter tuning. We have show both the mean and the standard deviation of the performance in our experiment sections. All the results have passed a paired t-test with \(p<0.05\)Dror et al. (2018).

### Hyper-parameters for General Experiments

We use AdamW as the optimizer, and the learning rate is chosen from \(1\times 10^{-5},2\times 10^{-5}\)}. A linear learning rate decay schedule with warm-up \(0.1\) is used, and the number of training epochs is \(15\) for fine-tuning. For active self-training & SSAL baselines, we tune the model with 2000 steps, and evaluate the performance on the development set in every 50 steps. Finally, we use the model with best performance on the development set for testing.

### Hyper-parameters for AcTune

Although AcTune introduces several hyperparameters including \(K\), \(M\), \(m_{L}\), \(m_{H}\), \(\beta,\gamma\), \(\lambda\), most of them are keep fixed during our experiments, thus it does not require heavy hyper-parameter tuning. All results are reported as the average over three runs.

In our experiments, we keep \(\beta=0.5\), \(\lambda=1\) for all datasets. For other parameters, we use a grid search to find the optimal setting for each datasets. Specifically, we search \(\gamma\) from \([0.5,0.6,0.7]\), \(m_{L}\) from \([0.6,0.7,0.8]\), \(m_{H}\) from \([0.8,0.9,1]\). For AccTune with Entropy, we use probability based aggregation and for AcTune with CAL, we use value 

based aggregation by default.

## Appendix C Runtime Analysis

Table 3 shows the time in one active learning round of AcTune and baselines. Here we highlight that the additional time for region-aware sampling and momentum-based memory bank is _rather small_ compared with the inference time. Also, we find that BALD and REVIVAL are not so efficient. For BALD, it needs to infer the uncertainty of the model by passing the data to model with multiple times. Such an operation will make the total inference time for PLMs very long. For REVIVAL, we find that calculating the adversarial gradient needs extra forward passes and backward passes, which could be time-consuming for PLMs with millions of parameters8.

Footnote 8: The original model is proposed with CV tasks and they use ResNet-18 as the backbone which only contains 11M parameters (around 10% of the parameters of Roberta-base).

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c} \hline \hline
**Hyper-parameter** & **SST-2** & **AG News** & **Pubmed** & **DBpedia** & **TREC** & **Chemprot** \\ \hline Dropout Ratio & \multicolumn{6}{c}{0.1} \\ \hline Maximum Tokens & 32 & 96 & 96 & 64 & 64 & 128 \\ \hline Batch Size for \(\mathcal{X}_{l}\) & \multicolumn{6}{c}{8} \\ \hline Batch Size for \(\mathcal{X}_{u}\) in Self-training & 32 & 48 & 48 & 32 & 16 & 24 \\ \hline Weight Decay & \multicolumn{6}{c}{\(10^{-8}\)} \\ \hline Learning Rate & \multicolumn{6}{c}{\(2\times 10^{-5}\)} \\ \hline \(\beta\) & \multicolumn{6}{c}{0.5} \\ \hline \(M\) & 25 & 30 & 30 & 40 & 40 & 40 \\ \hline \(K\) & 5 & \multicolumn{6}{c}{10} \\ \hline \(\gamma\) & 0.7 & \multicolumn{6}{c}{0.6} \\ \hline \(m_{L}\) & 0.8 & 0.9 & 0.7 & 0.8 & 0.8 & 0.8 \\ \hline \(m_{H}\) & 0.9 & 0.9 & 0.8 & 0.9 & 0.9 & 1.0 \\ \hline \(\lambda\) & \multicolumn{6}{c}{1} \\ \hline \end{tabular}
\end{table}
Table 2: Hyper-parameter configurations. Note that we only keep certain number of tokens.

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Dataset} \\  & Pubmed & DBPedia \\ \hline Finetune (Random) & ¡0.1s & ¡0.1s \\ \hline Entropy [1] & 461s & 646s \\ \hline BALD [17] & 4595s & 6451s \\ \hline ALPS [16] & 488s & 677s \\ \hline BADGE [15] & 554s & 1140s \\ \hline CAL [14] & 493s & 688s \\ \hline REVIVAL [14] & 3240s & OOM \\ \hline AcTune + Entropy & 477s & 733s \\ \hline w/ RS for Active Learning & 15.8s & 44.9s \\ \hline w/ MMB for Self-training & 0.12s & 0.18s \\ \hline AcTune + CAL & 510s & 735s \\ \hline w/ RS for Active Learning & 16.6s & 46.4s \\ \hline w/ MMB for Self-training & 0.12s & 0.18s \\ \hline \end{tabular}
\end{table}
Table 3: The running time of different baselines. Note that for ASST, CEAL and BASS, they directly use existing active learning methods so we do not list the running time here.