Combining Active Learning and Task Adaptation with BERT for Cost-Effective Annotation of Social Media Datasets
Year: 2023

Abstract:
Social media provide a rich source of data that can be mined and used for a wide variety of research purposes. However, annotating this data can be expensive, yet necessary for state-ofthe-art pre-trained language models to achieve high prediction performance. Therefore, we combine pool-based active learning based on prediction uncertainty (an established method for reducing annotation costs) with unsupervised task adaptation through Masked Language Modeling (MLM). The results on three different datasets (two social media corpora, one benchmark dataset) show that task adaptation significantly improves results and that with only a fraction of the available training data, this approach reaches similar F1-scores as those achieved by an upper-bound baseline model fine-tuned on all training data. We hereby contribute to the scarce corpus of research on active learning with pre-trained language models and propose a cost-efficient annotation sampling and fine-tuning approach that can be applied to a wide variety of tasks and datasets.

heading: 1 Introduction
text: Approximately 59% of the population worldwide use social media (Chaffey, 2023). Collectively, they post more than half a million comments on Facebook each minute, and a grand total of 500 million tweets per day (Shepherd, 2023; Aslam, 2023). These statistics indicate that social media are a virtually inexhaustible source of data, and a large part of this data can be accessed for research purposes. However, annotating this data, which is often necessary to achieve high prediction performance with pre-trained language models, can be an expensive and time-consuming process. One approach that has been proposed in previous research to reduce annotation costs is active learning (AL), which aims at optimizing the annotation effort by selecting specific data points from an unlabeled data pool which are expected to contribute
the most to a model’s learning phase (Settles, 2009). Although AL has proven its usefulness throughout decades of research, it remains a data selection method, which makes it challenging to use as only tool for annotation cost reduction and to reach upper bound performance (achieved by supervised learning on all available data).
In this work, we therefore exploit the capabilities of transformer-based pre-trained language models to learn from unsupervised data through their pretraining task. Concretely, we combine AL based on the prediction uncertainty of a model with unsupervised task adaptation through masked language modeling (MLM) to investigate whether this combination of techniques allows reaching the upper bound and with how much data. We test this approach by using different AL protocols on three publicly available datasets (2 social media datasets and 1 benchmark) that are costly to annotate, e.g., because they contain many fine-grained labels or the task is difficult to learn with little data. The experiments presented in this work show that using task adaptation before AL has a significant effect on model performance, and that substantially less data is needed to reach upper bound performance, therefore reducing annotation expenses.

heading: 2 Related research

heading: 2.1 Active learning
text: Active learning, or "sequential sampling", has been studied since the 1990’s (Lewis and Gale, 1994; Lewis and Catlett, 1994; Cohn et al., 1994, 1996). Originally, AL referred to prioritizing certain entries in an unlabeled dataset during the annotation process, based on the prediction confidence of a model trained on a small initial subset of labeled data, as shown in Figure 1. The intuition behind this sampling strategy is that the less certain a model is about a prediction, the more the model can learn from this data point during training, thereby
237
maximizing the return on annotation investment. In the last decades, AL has shown improvements in different tasks and models, such as text classification with k-nearest neighbors (Shi et al., 2008), word sense disambiguation with support vector machines (Zhu and Hovy, 2007), and machine translation with recurrent neural networks (Vashistha et al., 2022). Although AL is most commonly based on prediction uncertainty, sampling can also be based on model disagreement, such as in BALD (Bayesian AL by Disagreement, (Houlsby et al., 2011)), gradient information, such as in BADGE (Batch AL by Diverse Gradient Embeddings, Ash et al. (2019)), typicality or density (Zhu et al., 2008), batch diversity or representativeness (Shi et al., 2021), and other metrics (or a combination of any of the aforementioned, Settles (2009)).

heading: 2.2 Active learning with language models
text: Although AL with pre-trained transformers has gained interest in recent years, the amount of research remains relatively scarce compared to AL with traditional machine learning or neural models. Existing work, e.g. Schröder et al. (2022), has examined the vanilla uncertainty-based query strategy for various binary and multi-class text classification experiments and shown that this strategy is also effective for pre-trained language models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). Similarly, Ein-Dor et al. (2020) investigate the effect of annotation sample selection
based on prediction uncertainty, expected gradient length and batch representativeness compared to random sample selection. They hereby focus on binary text classification tasks where the positive class is in the minority and show that all strategies perform substantially better than a random sampling strategy.
Recently, Rotman and Reichart (2022) were the first to explore multi-task AL with pre-trained language models, building upon the work of Reichart et al. (2008), who introduced the concept for traditional machine learning methods, and Ikhwantri et al. (2018), who used it for (non-pre-trained) neural architectures. Entropy-based confidence, both in isolation and when using dropout agreement, was used for multi-task AL, and compared to single-task AL and random sampling. Additionally, they investigated the effect of task weighting in ranking samples on informativeness. Their experiments showed that multi-task AL is an efficient way for annotation cost reduction, but that the precise method should depend on the task(s).
Finally, it is noteworthy that any model utilized in an AL setting can be trained in two ways: by updating it after each step, or by re-initializing it and training the entire model on all available annotated data (Schröder et al., 2022; Ein-Dor et al., 2020; Hu et al., 2018; Shen et al., 2017). In general, language models are more frequently re-initialized than updated, because they tend to be unstable when incrementally fine-tuned on low amounts of data, resulting in lower performance and higher standard deviations across different random seeds or hyperparameters (Dodge et al., 2020). However, the effect of re-initializing versus updating language models during AL is understudied. This work will therefore investigate whether re-initializing language models is indeed the preferred approach when using the standard uncertainty-based AL approach, and when combining it with task adaptation.

heading: 2.3 Task adaptation
text: As mentioned, this work combines AL with task adaptation. The latter refers to learning training data in an unsupervised manner before fine-tuning on it for a specific end task. For example, Buhmann et al. (2022) show that task adaptation has a positive effect on their question-answering model used for VaccinChat1: a user interface that answers Dutchlanguage user questions about the COVID-19 vac-
1https://vaccinchat.be/
cine, specifically for Flemish (Dutch speaking Belgian) users. Similarly, Mehri et al. (2020) show that task adaptation through performing MLM on the training data before fine-tuning increases the performance of their task-oriented dialogue system. In the experiments presented in this paper, we adopt the MLM approach as task adaptation step before commencing AL.

heading: 3 Methodology

heading: 3.1 Data
text: Existing AL research often relies on datasets that are inexpensive to annotate or tasks that are relatively easily learned by models, even when little data is available. Examples are the IMDB movie review, YELP polarity, SST-2, and TREC datasets for NLP (Maas et al., 2011; Zhang et al., 2015; Socher et al., 2013; Hovy et al., 2001), and the MNIST and Fashion-MNIST for computer vision (Deng, 2012; Xiao et al., 2017). This results in experiments where very small initial training samples are used, which are then increased in equally small steps, while still achieving relatively high prediction performances. Although it is necessary to create artificial AL setups, the aforementioned experimental settings are in our opinion inappropriate for research on AL, because the method is in reality the most effective when working with data that is expensive to annotate or when many examples are needed to gain high prediction accuracy.
Taking this into account, we use the FRENK (Ljubesic et al., 2019), and GoEmotions (Demszky et al., 2020) datasets for our experiments and validate the results on an additional benchmark: the 20 News Groups dataset (Lang, 1995). A detailed description of these datasets and why they are useful for AL experiments can be found below, and a summary of the statistics of each of these datasets can be found in Table 1. A fine-grained overview of the class distributions of the datasets can be found in Appendix A.

heading: 3.1.1 FRENK
text: The FRENK dataset2 consists of Facebook comment threads on news item posts about two topics: migrants and the LGBTQ+ community (Ljubesic et al., 2019). Although the dataset contains Slovene and English comments, the current work only utilizes the English partition. The data contains labels concerning the topic (LGBTQ+ or migrants), the target of the hate speech (topic, related to topic, journalist/medium, other, no target), and the type of hate speech, which is the task we tackle in this study. FRENK distinguishes six types of hate speech in its annotation scheme:
1. Background-violence consists of messages that call for violence based on the personal background of the target (e.g., religion, gender, race or ethnicity). 2. Other-violence contains messages that call for violence for another reason than the background of the target, e.g., an opinion expressed by the target. 3. Background-offensive refers to messages that contain offensive statements that are aimed at the background of the target. 4. Other-offensive are messages that contain offensive language towards any aspect of the speaker but their personal background. 5. Inappropriate speech pertains to messages that contain vulgar and/or other types of offensive language that is not aimed towards a specific target (hence this category is technically not hate speech). 6. Appropriate speech, which does not contain any form of offensive or violent language.
Since the violent language classes contain very few entries, and the experiments in this paper require using small samples of training data, labels (1) and (2) were combined to form one "violent language" class, and labels (3) and (4) were used to form an "offensive language" class for the purpose of this paper, resulting in a total of 4 labels (the label distribution with these newly formed classes can be found in Table 15, Appendix A). The training, validation, and test partitions contain 8,404, 933 and 2,301 entries, respectively.
Since hate speech is a term that is open for interpretation and its identification depends on the personal and cultural background of the annotator,
2https://huggingface.co/datasets/classla/ FRENK-hate-en
multiple annotators are needed to generate high quality labels and avoid bias (Sap et al., 2022). In addition, the labels in FRENK show strong class imbalances, which is why many comments are needed in order to collect sufficient annotations for the underrepresented classes, resulting in high annotation costs. Further, annotators were required to read the comments thoroughly, since the labels contain a hierarchy of importance in cases where multiple types of hate speech occur in one message (other < background; offensive language < violent language), which increases annotation time even more.

heading: 3.1.2 GoEmotions
text: The GoEmotions dataset3 contains Reddit comments annotated with 28 emotions (incl. "neutral") in a multi-label setting (Demszky et al., 2020). The dataset is divided in 43,410 cases for training, 5,426 for validating, and 5,427 for testing.
Due to its high number of classes and multilabel scheme, annotating the GoEmotions dataset is labor-intensive. Given its large class imbalances, it is particularly difficult to gain a performance increase in the smaller classes by annotating more samples, since many samples need to be annotated before collecting a substantial amount of messages that express emotions that are infrequent in the dataset.

heading: 3.1.3 20 News Groups
text: The 20 News Groups dataset4 contains approximately 20,000 news groups posts each associated with 1 out of 20 different topic classes (Lang, 1995). We use this benchmark with many fine-grained classes as an additional test for the proposed approach next to the above mentioned social media datasets, which are the focus of this paper.

heading: 3.2 Approaches

heading: 3.2.1 Baseline approaches
text: Random sampling For this baseline, the training data was sampled randomly so that the effect of the AL strategies could be measured.
Upper bound This approach refers to fine-tuning with all available training data in order to estimate the highest possible performance that can be achieved with standard fine-tuning.
3https://huggingface.co/datasets/go_emotions 4https://huggingface.co/datasets/SetFit/20_
newsgroups

heading: 3.2.2 AL approaches
text: Model re-initialization This method refers to the standard AL strategy for language models as proposed in (Schröder et al., 2022; Ein-Dor et al., 2020): An initial sample is used to fine-tune a model, which then predicts labels for the rest of the training data. A second sample is then selected based on the confidence of the model and a new model is initialized and fine-tuned using all annotated data. This process then repeats itself n times.
Checkpoint updating This method is identical to the approach above, with the only difference being that each time a new batch of annotated data is selected, the model is not re-initialized, but finetuning continues with the new annotated sample starting from the final checkpoint of the previous round of fine-tuning.
Two-step learning This approach is a specific form of AL where a model is first fine-tuned and then updated once. For the first fine-tuning stage, an initial random sample is used. After predicting the labels of the rest of the training data with this model, it is fine-tuned a second time using the top n most uncertain entries, where n is determined by the rest of the annotation budget. In contrast, "checkpoint updating" refers to annotating various batches of fixed size and updating the model after each batch (i.e., in more than two steps).

heading: 3.3 Experimental setup
text: The annotation process was replicated as follows: assuming that a labeled validation and test set are available, a random batch consisting of ca. 10% of all training data was used as an initial sample for fine-tuning. Then, the amount of training data was incrementally increased until approx. 50% of the available training data was used. For all classification experiments, a BERT-base-uncased model (Devlin et al., 2019) was fine-tuned for 5 epochs with a batch size of 32 and a learning rate of 5e-5. The model was evaluated after each epoch, and predictions on the test set were made with the checkpoint that yielded the highest macro-averaged F1-score on the validation set.
In experiments where task adaptation was used, the model was first fine-tuned on the entire unlabeled training dataset through MLM for the duration of 5 epochs with a batch size of 64 and learning rate of 1e-4. The model checkpoint with lowest validation loss was used for further experiments.

heading: 4 Results

heading: 4.1 FRENK
text: Table 2 shows the random and upper bound baselines on the FRENK dataset, whereas Table 3, 4, and 5 show the results achieved for AL with model re-initialization, checkpoint updating, and two-step learning, respectively. The upper bound F1-macro amounts to 59.1%, which was surpassed when using 2,500 training instance in an AL fine-tuning process with model re-initialization after task adaptation. Specifically, using 29.7% of the available labeled training data led to a performance of 59.4% F1-macro. When increasing the training data to 4,000 entries, 47.6% of all available training data, the F1-score reached 61.0%.
Overall, model re-initialization when fine-tuning after querying a new sample of annotated data achieves best results, followed by two-step learning, and finally incremental checkpoint updating. In the case of the latter, we even observe a small decrease in performance (from 55.3% to 54.8%) and high standard deviation of the F1-score (5.4%) when updating the model with 500 unseen entries after it was already fine-tuned cumulatively on 3,000 entries. This shows that updating model check-
points with smaller samples of data is inefficient, cf. Schröder et al. (2022); Ein-Dor et al. (2020). For the best AL approach, i.e. model re-initialization, a McNemar test was conducted for each sample size to determine whether the improvements after task adaptation are statistically significant (McNemar, 1947), as shown in Table 3. These tests indicate that the improvements in 4 out of 7 experiments are statistically significant, while the other three still show substantial improvements.
To illustrate where most improvements were made after using task adaptation, Table 19 (Appendix C), shows the results per class after finetuning on 2,500 training entries using different sampling approaches (random sampling, standard AL, AL after task adaptation). In this table, it can be observed that the highest improvements were made in the most difficult and underrepresented class, namely the "violent speech" class. More precisely, results improved from 18.3% to 32.3% when using standard AL instead of random sampling, and to 47.1% after using task adaptation before AL. Noteworthy is that the performance for the "inappropriate speech" class dropped with 1.4% when using standard AL (compared to random sampling), but when using task adaptation, this performance drop was no longer observed.

heading: 4.2 GoEmotions
text: The results of the GoEmotions dataset are shown in Table 6 (random and upper bound baselines), 7 (model re-initialization), 8 (model updating), and 9 (two-step learning). As shown, the upper bound baseline achieves an F1-macro score of 47.2% on the test set (using all 43,410 samples for finetuning). In contrast with the experiments on the FRENK dataset, we observe that the upper bound baseline could not be matched with the utilized sample sizes. Nevertheless, it can be observed that task adaptation improves results for all AL approaches and sample sizes. Similarly to the results of the FRENK dataset, model re-initialization yields better results than checkpoint updating (regardless of whether this is done incrementally or through two-step learning). In the case of AL by checkpoint updating, we even observe a decrease in performance compared to random sampling when using 22,000 entries cumulatively for
fine-tuning. This evidences the inefficiency of incremental fine-tuning of pre-trained language models on small data samples. Statistical significance for the experiments with model re-initialization in this multi-label setting was determined by conducting a McNemar test for each individual class for each sample size. A summary of these tests can be found in Appendix B, Table 18. The three emotions where statistically significant improvements were observed the most frequently were "approval", "confusion", and "amusement", and in total, 6 out of 28 emotions were never predicted more accurately with statistical significance.
Finally, Table 20 (Appendix C) shows that the improvements are primarily found in the most difficult classes, namely those that were not predicted in the experiments with random sampling: "annoyance", "caring", "confusion", "desire", "disappointment", "excitement", "fear" and "surprise". The bulk of these did also not get predicted when using standard AL, but improvements could be observed when using task adaptation.

heading: 4.3 20 News Groups
text: As shown in Table 10, the upper bound F1-macro score achieved on the 20 News Groups dataset is 69.1%. This score was (only) achieved when using the AL protocol with model re-initialization and task adaptation after sampling 5,500 training entries, which equals 54.0% of the available training data.
Similarly to the experiments on the other datasets, task adaptation improves all three explored AL protocols, although model reinitialization remains the best of the three. Interestingly, however, AL without task adaptation yields lower results than random sampling. For AL with model re-initialization, for example, there is an average decrease of 0.8% in F1-macro across all sample sizes. This shows that the standard uncertaintybased AL makes worse sampling choices than random sampling, and that uncertainty is therefore a suboptimal metric for measuring informativeness in this particular dataset. A possible explanation for this observation is that language models are
often ill-calibrated and tend to be overconfident, even if their prediction is false (Yuan et al., 2020; Park and Caragea, 2022). Additionally, low prediction confidence may indicate that an entry is noisy, not just difficult to predict. A qualitative analysis of the entries that are selected early in the AL process shows that this is the case for the 20 News Groups dataset: many sampled posts contain merely a few words that are irrelevant to the topic, whereas other posts are lengthy and discuss a multitude of (irrelevant) topics causing the low prediction confidence in the classifier. Since data selection based on prediction uncertainty collects more noise than a random selection of data in this case, it prevents the model from learning useful information, especially in earlier samples.
After task adaptation, however, there is an average improvement of 2.6% over the random baseline, and an average gain of 3.2% over standard AL (improvements for all sample sizes are statistically significant). This shows that although sample selection based on model prediction uncertainty is
suboptimal on some datasets, task adaptation can act as a safety net to avoid performance impairment due to the use of suboptimal metrics for measuring informativeness.
Finally, and similarly to the results observed on the other datasets, the highest performance increases are observed in the most difficult classes. For example, "talk.religion.misc" improves from 8.9% (random sampling) to 22.4% (AL with model re-initialization and task adaptation), as shown in Appendix C (Table 21).
The results of the experiments until this point have shown that task adaptation has a positive effect on AL: Significant improvements in F1-score could be observed, and in two datasets, the upper bound could be reached with a fraction of the annotations, while still showing substantial gains in the third dataset. In the case of 20 News Groups, we observed that traditional AL had a negative effect on model performance, although task adaptation countered this effect. Finally, the results indicate that AL with model re-initialization leads to more stable fine-tuning than with model updating.

heading: 4.4 Ablation study
text: In this section, we investigate the effect of task adaptation in isolation, and whether using AL still has beneficial effects on model performance after task adaptation. In order to gain insights into this matter, task adaptation was used without AL, i.e. with random sample selection. The result of this experiment was then compared to the result achieved with random sampling, standard AL, and task adaptation combined with AL (as reported in the previous subsections).
The results of the above mentioned experiments can be found in Table 14: For FRENK, it can be concluded that task adaptation alone does not improve results when using random sample selection, although using task adaptation and AL leads to improvements of 4.6% on average. In comparison, standard AL leads to average improvements of 1.7%. This surprising result indicates that BERT does not learn new knowledge from task adaptation, but that this technique causes better sample selection during the AL stage. There may be different reasons why task adaptation has less effect on FRENK than on the other datasets. For example, the data in FRENK could resemble the data used to pre-train BERT more than is the case for the other datasets, so that less new information is Ta bl e 14 :I m pr ov em en ts (F 1) of ta sk ad ap ta tio n an d
A L
ov er
ra nd
om sa
m pl
in g
ac ro
ss al
ls am
pl e
si ze
s (fi
rs ts
am pl
e si
ze w
as no
ti nc
lu de
d in
th e
ca lc
ul at
io ns
fo rt
he st
an da
rd A L ex pe ri m en ts ,s in ce th is ex pe ri m en ti s id en tic al to th at of ra nd om sa m pl e se le ct io n) .
FR E
N K
G oE
m ot
io ns
20 N
ew sG
ro up
s
n R
an do
m R
an do
m +
ad ap
ta tio
n A
L A
L +
ad ap
ta tio
n n
R an
do m
R an
do m + ad ap ta tio n
A L
A L + ad ap ta
tio n
n R
an do
m R
an do
m +
ad ap
ta tio
n A
L A
L +
ad ap
ta tio n 1, 00 0 43 .4 45 .7 (+ 2. 3) 43 .4 (+ 0. 0) 45 .7 (+ 2. 3) 4, 00 0 9. 7 12 .9 (+ 3. 2) 9. 7 (+ 0. 0) 12 .9 (+ 3. 2) 1, 00 0 51 .8 58 .3 (+ 7. 5) 51 .8 (+ 0. 0) 58 .3 (+ 7. 5) 1, 50 0 45 .9 46 .8 (+ 0. 9) 46 .9 (+ 1. 0) 48 .7 (+ 2. 8) 6, 00 0 14 .2 19 .9 (+ 5. 7) 17 .0 (+ 2. 8) 24 .7 (+ 10 .5 ) 1, 50 0 58 .0 62 .7 (+ 4. 7) 54 .7 (- 3. 3) 60 .2 (+ 2. 2) 2, 00 0 48 .0 48 .3 (+ 0. 3) 48 .5 (+ 0. 5) 51 .8 (+ 3. 8) 8, 00 0 21 .5 28 .6 (+ 7. 1) 25 .8 (+ 4. 3) 34 .3 (+ 12 .8 ) 2, 00 0 62 .4 64 .9 (+ 2. 5) 58 .9 (- 4. 0) 63 .4 (+ 1. 0) 2, 50 0 52 .0 50 .0 (- 2. 0) 54 .5 (+ 2. 5) 59 .4 (+ 7. 4) 10 ,0 00 26 .3 35 .9 (+ 9. 6) 34 .2 (+ 7. 9) 39 .3 (+ 13 .0 ) 2, 50 0 63 .1 66 .3 (+ 3. 2) 61 .4 (- 1. 7) 65 .3 (+ 2. 3) 3, 00 0 54 .2 53 .2 (- 1. 0) 55 .0 (+ 0. 8) 59 .4 (+ 5. 2) 12 ,0 00 32 .7 38 .9 (+ 6. 2) 38 .1 (+ 5. 4) 41 .9 (+ 9. 2) 3, 00 0 64 .3 67 .2 (+ 2. 9) 63 .5 (- 0. 8) 66 .5 (+ 2. 2) 3, 50 0 52 .9 52 .6 (- 0. 3) 57 .1 (+ 4. 2) 59 .3 (+ 6. 4) 14 ,0 00 37 .5 40 .5 (+ 3. 0) 40 .8 (+ 3. 3) 44 .2 (+ 7. 3) 3, 50 0 65 .3 67 .5 (+ 2. 2) 65 .1 (- 0. 2) 67 .4 (+ 2. 1) 4, 00 0 56 .4 56 .2 (- 0. 2) 57 .4 (+ 1. 0) 61 .0 (+ 4. 6) 16 ,0 00 39 .7 41 .2 (+ 1. 5) 43 .5 (+ 3. 8) 45 .2 (+ 5. 5) 4, 00 0 65 .2 67 .4 (+ 2. 2) 66 .1 (+ 0. 9) 67 .7 (+ 2. 5) Av g. im pr ov em en t +0 .0 +1 .7 +4 .6 18 ,0 00 40 .9 43 .3 (+ 2. 4) 43 .9 (+ 3. 0) 45 .5 (+ 3. 0) 4, 50 0 65 .9 67 .9 (+ 2. 0) 66 .3 (+ 0. 4) 68 .0 (+ 2. 1) 20 ,0 00 41 .8 43 .9 (+ 2. 1) 45 .1 (+ 4. 0) 46 .2 (+ 4. 4) 5, 00 0 66 .5 68 .0 (+ 1. 5) 67 .3 (+ 0. 8) 68 .7 (+ 2. 2) 22 ,0 00 43 .1 44 .5 (+ 1. 4) 45 .6 (+ 2. 5) 46 .3 (+ 3. 2) 5, 50 0 67 .1 67 .5 (+ 0. 4) 68 .1 (+ 1. 0) 69 .2 (+ 2. 1) Av g. im pr ov em en t +4 .2 +4 .1 +7 .2 Av g. im pr ov em en t +2 .9 -0 .8 +2 .6
being learned. Determining the exact cause of this observation, however, is beyond the scope of this paper and can be the subject of future work.
With respect to the GoEmotions dataset, which yielded the lowest F1-scores compared to FRENK and 20 News Groups, the improvements of using both task adaptation and AL was the highest of all datasets: 7.2% on average. Individually, task adaptation and AL showed improvements similar to each other, namely 4.2% and 4.1%, respectively.
Regarding 20 News Groups, we observe that standard active learning has a negative effect on performance overall, especially in the smaller sample sizes, as mentioned before. Task adaptation, however, shows improvements of 2.9% and 2.6%, without and with the additional use of AL, respectively. This shows that when active learning has a negative effect on performance, task adaptation can negate this effect and still improve upon standard fine-tuning with random data selection.

heading: 5 Conclusion
text: Main findings The current work combines uncertainty-based AL with task adaptation in order to learn from the data that could not be labeled due to limited annotation budget. It investigates the following research questions:
• What is the the effect of task adaptation on AL? • How much data is needed to achieve the same performance with the proposed approach as with standard fine-tuning on all data?
The results of the experiments conducted on multiple datasets provide the following answers to these questions:
• Task adaptation has a significantly positive effect on AL, regardless whether the model is re-initialized or updated (although reinitialization consistently leads to better results). An analysis of the performance per class shows that the improvements are highest in the most difficult or underrepresented classes, and that the most difficult tasks in general show the highest improvements. • In our experiments, 29.7% and 54% of all annotated data in the FRENK and 20 News Groups dataset, respectively, was needed to achieve the upper bound baseline. Although the proposed approach did not reach the upper bound baseline in the GoEmotions dataset
with 50% or less of the training data, these results show that the proposed combination of approaches can lower annotation costs substantially.
Future research directions As mentioned earlier, AL with pre-trained language models remains understudied, in spite of it being an efficient annotation cost reduction method. Future research directions for AL with language models may include investigating the effect of calibration quality on uncertainty-based AL. An additional direction worth investigating is combining AL with other methods, such as data augmentation, weak supervision and domain adaptation, which are until now topics that are studied more extensively in other machine learning fields, such as computer vision (Zhao et al., 2020; Biegel et al., 2021; Xie et al., 2021; Hao et al., 2021; Zhan et al., 2022).
Limitations
The current work focuses on AL with pre-trained language models based on lowest prediction confidence. In spite of the effectiveness of the strategy shown both in these experiments and in previous work (Schröder et al., 2022; Ein-Dor et al., 2020), neural models are often not calibrated well (Yuan et al., 2020; Park and Caragea, 2022), which implies that the output of the softmax function could be a suboptimal metric for measuring prediction confidence, i.e. informativeness, for a given training sample. Future work on this topic should therefore investigate whether other metrics work better for AL with pre-trained language models and whether these metrics also benefit from unsupervised task adaptation. Additionally, experiments could only be conducted on a limited amount of tasks and datasets. Future work should shed new light on the usefulness of the proposed approach in different settings.

heading: Acknowledgements
text: This research was funded by the Flemish government under FWO IRI project CLARIAH-VL and under the Research Program Artificial Intelligence Flanders.

heading: B GoEmotions significance per class

heading: A Label distribution per dataset

heading: C Results per class

heading: Future Work
text: Future research directions As mentioned earlier, AL with pre-trained language models remains understudied, in spite of it being an efficient annotation cost reduction method. Future research directions for AL with language models may include investigating the effect of calibration quality on uncertainty-based AL. An additional direction worth investigating is combining AL with other methods, such as data augmentation, weak supervision and domain adaptation, which are until now topics that are studied more extensively in other machine learning fields, such as computer vision (Zhao et al. , 2020; Biegel et al. , 2021; Xie et al. , 2021; Hao et al. , 2021; Zhan et al. , 2022). Limitations
The current work focuses on AL with pre-trained language models based on lowest prediction confidence. In spite of the effectiveness of the strategy shown both in these experiments and in previous work (Schröder et al. , 2022; Ein-Dor et al. , 2020), neural models are often not calibrated well (Yuan et al. , 2020; Park and Caragea, 2022), which implies that the output of the softmax function could be a suboptimal metric for measuring prediction confidence, i. e. informativeness, for a given training sample. Future work on this topic should therefore investigate whether other metrics work better for AL with pre-trained language models and whether these metrics also benefit from unsupervised task adaptation. Additionally, experiments could only be conducted on a limited amount of tasks and datasets. Future work should shed new light on the usefulness of the proposed approach in different settings