Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings
Authors: Josip Jukić
Jan Snajder
Venue: EMNLP 2023 Main (2023)
Keywords: parameter-efficient fine-tuning
active learning
low-resource settings
TL;DR: Combining active learning and adapter-based parameter-efficient fine-tuning in low-resource settings with pre-trained language models for text classification.

Abstract: Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups.
We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.

Parameter-Effiieet Language Model Tuning with Active Learning in Low-Resource Settings
Josip Jukié Jan Snajder
TakeLab
Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia josipp jukic, jan snajder ) @eerr..e hr
Abstract
Data labeling is notoriously time-consuming and expensive, often hindering the development siz- able labeled datasets required for training high- performance models. Active learning (AL) (Cohn et al.. 1996; Settles, 2009) has emerged as poten- tial solution to this challenge. In contrast to passive learning, in which the training set is sampled at ran- dom, AL encompasses unique family of machine learning algorithms specifically designed to reduce labeling costs by reducing label complexity. i.c., the number of labels required by an acquisition model to achieve a certain level of performance (Dasgupta, 2011). 1). With the advent of PLMs, AL research has pivoted towards investigating training regimes for PLMs, such as task-adaptive pre-trate training (TAPT; Gururangan et al., 2020), that could be combined with AL to further reduce the label complexity.
Pre-trained language models (PLMs) have ig- nited surge in demand for effective fine- luning techniques, particularly in low-resource domains and languages. Active learning (AL), set of algorithms designed decrease label- ing costs by minimizing label complexity, has shown promise confronting the labeling bot- tleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-basse -base PEFT remains unexplored. We present an empirical study of PEFT behav- ior with AL in low-resource settings for text classification tasks. Our findings affirm su- periority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists
setups. We ther examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic poten- tial of AL and PEFT in low-resource settings, paving the way for udvancements in efficient and effective fine-tuning.'
While AL aims directly minimizing the label complexity of learning, training efficiency can also be improved by reducing the parameter complex- ity of the model. This becomes more important as PLMs grow larger, and fine-tuning becomes increasingly challenging due to the sheer num- ber of parameters involved. To address this issue, adapters (Houlsby et al., 2019) have been intro- duced compact modules that can be incorporated between the layers of PLMs. Adapters enable con- siderable parameter-sharing, facilitating parameter- efficient fine-tuning (PEFT) through modular learn- ing (Pfeiffer et al., 2023). In this process, only the parameters of the adapters are updated dur- ing the tuning for specific downstream task. Re- cent research (He et al., 2021; Li and Liang, 2021; Karimi Mahabadi et al.. 2021) has revealed that some PEFT methods outperform full fine-tuning (FFT) in low-resource settings, potentially due better stability and decreased risk of overfitting. In contrast, FFT has been shown to exhibit instabil- ity in scenarios with limited data.
1 Introduction
Pre-trained language models (PLMs) have quickly become staple in the field of natural language processing. With the growing demand for data for training these models, developing efficient fine- tuning methods has become critical. This is par- ticularly relevant for many domains and languages where obtaining large amounts labeled training data is difficult or downright impossible. In such low-resource settings, it becomes essential to effec- tively leverage and adapt PLMs while minimizing the need for extensive labeled data.
Despite the promising results demonstrated by PEFT methods low-resource settings, there a
!Our code is available at https::/ ///s:://tttt t.ub. com/ httpp//ccccaaaaaaeeeeeea

---

striking gap in research on parameter-efffiient train- ing with respect to how PEFT interacts with AL. Given that the majority of real-world AL scenarios involve a restricted amount of data, PEFT meth- ods emerge as strong candidates for AL. acquisition models. However, there has been no exploration of AL in conjunction with adapters. Investigating this uncharted territory can further advance our un- derstanding of AL and reveal novel strategies for optimizing performance in low-resource settings.
cies of positive interactions between AL, PEFT, and TAPI, providing empirical justification for their combined use in low-resource settings.
2 Related Work
Our research involves combining AL with PLMs and investigating the use of PEFT techniques within the confines of low-resource settings.
AL with PLMs. Until recently, the conventional approach for integrating PLMs with AL involved performing full fine-tuning with a fixed number of training epochs and training the model from scratch in each AL step (Ein-Dor et al., 2020; Mar- gatina et al., 2021; Shelmanov et al., 2021; Karam- cheti et al., 2021; Schröder et al., 2022). However, studies by Mosbach et al. (2021) and Zhang et al. (2021) revealed that fine-tuning in low-resource setups is prone instability, particularly when training for only few epochs. This instability, often sensitive to weight initialization and data or- dering (Dodge et al., 2020), presents a significant challenge for AL, which frequently operates in low- resource settings. Recent research has looked into the impact of PLM training regimes on AL perfor- mance (Grießhaber et al., 2020; Yuan et al., 2020; Yu et al., 2022), suggesting that the choice of train- ing regime is more critical than the choice of the AL method. Notably, TAPT has proven particularly effective in enhancing AL performance (Margatina et al., 2022; Jukić and Šnajder, 2023).
In this paper, we present an empirical study on the behavior of PEFT in low-resource settings for text classification tasks. We analyze PEFT with and without AL and compare it against FFT. While our results confirm that PEFT exhibits superior per- formance in low-resource setups compared to FFT, we show that the improved performance with PEFT extends to AL scenarios in terms of performance gains over passive learning. Furthermore, we an- alyze the efficacy of TAPT in conjunction with AL and PEFT. We find that TAPT is beneficial in AL scenarios for both PEFT and fully fine-tuned models, thus representing a viable technique for improving performance in low-resource settings. Finally, aiming to illuminate why PEFT and TAPT improve AL performance in low-resource settings, we analyze the properties of PEFT and FFT via for- getting dynamics (Toneva al., 2019) and PLMs' instance-level representations. We find that AL methods choose fewer unforgettable and more mod- erately forgettable examples when combined with PEFT and TAPT, where forgetfulness indicates the model's tendency to learn and forget the gold label of particular instance. Compared to FFT, we ob- serve that PEFT yields representations in the early and middle layers of model that are more sim- ilar to the representations of the base PLM. We hypothesize that this property mitigates the issue of forgetting the knowledge obtained during pre- training when fine-tuning for downstream tasks.
Adateers in low-resource settings. Research on adapters in low-resource settings has primarily fo- cused on areas such as cross- lonsull transfer for low-resource languages (Ansell et al., 2021; Lee et al., 2022; Parović et al., 2022), where the em- phasis lies on exploring diverse methods of fusing adapters. In monolingual settings with scarce data, adapters have been found to outperform full fine- tuning (Li and Liang, 2021; Mao al., 2022). A study by He et al. (2021) ) demonstrated that adapter- based tuning exhibits enhanced stability and gen- eralization capabilities by virtue of being less sen- sitive to learning rates than traditional fine-tuning methods. While incorporating task adaptation tech- niques, such as TAPT, has been shown to match or even improve performance over FFT in low- resource setups, Kim et al. (2021) noted interest- ing caveat: the benefits of integrating TAPT with adapters tend to taper off as the amount of data increases.
In summary, we show that in AL low-resource settings for text classification, (1) PEFT yields greater performance improvements compared to FFT and (2) TAPT enhances the overall classifi- cation performance of adapters and is well-suited for AL scenarios. We also show that (3) AL meth- ods choose fewer unforgettable and more moder- ately forgettable examples with PEFT and that (4) PEFT produces instance-level representations of early and middle layers that are more similar to the base PLM than FFT. Our results uncover the intrica-

---

Despite the established effectiveness of adapters in setups with limited resources, their integration into AL frameworks - which frequently face anal- ogous resource constraints - remains an untapped area of research. This gap is particularly notable given that AL's iterative learning process could significantly benefit from adapters' parameter effi- ciency and transferability, especially in scenarios where data scareity labeling costs are primary concern..
All of the above PEFT methods fall under the category of lightweight fine-tuning. While prefix- tuning does not technically qualify as an adapter, He et al. (2022) demonstrated that it shares for- mal similarities with adapters, with prefix-tuning performing weighted addition and an adapter em- ploying unweighted addition. We refer to all four considered methods adapters for terminologi- cal simplicity. We use BERT (Devlin al., 2019) as the base PLM for every adapter. Additionally, we adhere to the hyperparameter settings for each adapter as recommended in the respective papers that introduced them (ef. Appendix A.2 for details).
3 Preliminaries
We now describe the experimental setup, providing details on the datasets as well as the PEFT and AL methods used in our study.
3.3 AL methods
Our study considers five sampling strategies, in- cluding random selection (RND) as passive learn- ing baseline. The other four strategies are AL meth- ods originating from different families, chosen for their robustness (ability perform well across var- ious tasks) and widespread usage in the field:
3.1 Datasets
We employ four single-text classification tasks com- monly used for AL evaluation: (1) the subjectiv- ity dataset (SUBJ; Pang and Lee, 2004), designed to assess the subjectivity of a given text; (2) the question type classification dataset (TREC; Li and Roth, 2002), designed for categorizing questions according to their types; (3) the Stanford Sentiment Treebank (SST; Socher et al., 2013), which focuses on sentiment analysis; (4) AG's news classification dataset (AGN; Zhang et al., 2015), which classifies news articles into different categories. We provide the dataset statistics in the appendix for further reference (cf. Appendix Table 3).
Maximum entropy (ENT; Lewis and Gale, 1994) comes from the family of uncertainty strate- gies. The method queries instances where the model least certain based on the maximum entropy criterion of the prediction output;
Monte Carlo dropout (MC; Gal and Ghahramani. 2016) resembles ENT but utilizes the stochas- ticity of forward passes with dropout layers (Srivastava et al., 2014) to estimate the entropy for a given instance:
3.2 PEFT methods
We consider four prototypical PEFT techniques:
Adapter incorporates trainable bottleneck layers after both the multi-head attention and feed- forward block in each Transformer layer (Houlsby et al., 2019);
Core-set (CS; Sener and Savarese, 2018) encour- ages instance diversity by using the learned representations of the acquisition model. This method aims to minimize the distance be- tween an example in the unlabeled set and its closest counterpart in the labeled subset;
Prefix-tuning adds new parameters in the multi- head attention blocks within each Transformer layer (Li and Liang. 2021);
Discriminative active learning (DAL: Gissin and Shalev-Shwarte 2019) frames AL as a binary classification of instances into those that are labeled and those that are not, with the objec- tive of making the labeled and unlabeled sets indistinguishable.
LoRA (Low-rank adaptation) represents an addi- tive method that incorporates trainable low- rank decomposition matrices into the layers of a pre-trained model (Hu et al., 2022);
UniPELT combines multiple PEFT approaches, namely LoRA, Prefix-tuning, and Adapter, in single unified setup (Mao et al., 2022). Each constituent is a submodule, and UniPELT em- ploys gating mechanisms to activate them ef- fectively.
3,4 Experimental setup
In AL runs, we select 50 new examples in each step of each AL experiment, using 100 examples for the warm start (randomly sampled labeled data

---

initiate the model). To probe different PEFT ap- proaches with and without AL in low-resource set- tings, we establish labeling budget limit of 1, 000 instances. To sidestep the need for a validation set in our experiments, which is typically unavailable in real-world AL scenarios, we adopt the Besov early stopping (Jukié and Snajder, 2023). This method utilizes the smoothness of Transformer lay- ers to decide at which epoch to stop training.
ical improvement, which would be tantamount attaining an Fi score of I in the initial sampling step and sustaining that score throughout steps. Conversely, a negative score indicates that the AL method performs worse than passive learning.
Experiments
In the case of TAPT, we pre-train the base model on a masked language modeling task using unla- beled training data. For adapters, we only update the injected parameters while keeping the remain- ing parameters of the base model frozen. This ap- proach aligns with the primary function of adapters, which is to utilize common base model across diverse tasks. For every setting, perform five runs using different random seeds. We report the average Fi score at each sampling step (with and without AL for FFT and PEFT) to show the cor- responding learning curve averaged over five runs. We provide details on training and hyperparameters in Appendix A.5.
In this section, we first examine the performance of PEFT methods in comparison to FFT with passive learning and then proceed to analyze the applica- tion of PEFT in AL settings.
4.1 PEFT vs. FFT
Previous research on the use of adapters in low- resource settings (Li and Liang, 3021; Mao et al., 2022; He et al., 2021) has demonstrated that adapters perform comparable to, and sometimes even better than FFT. However, these findings were based on comparing FFT to single adapter variant full dataset evaluating the performance only a few discrete points.
3.5 Evaluation
In the first part of our experiments, we build upon these findings by conducting a more nuanced analysis. We generate detailed learning curves that facilitate the comparison of multiple adapters with FFT under the passive learning setup. Our compar- ison, summarized by the AUC metric in Table 1, veals that UniPELT and Prefix-tuning consistently outperform FFT with significant difference across all datasets used our study. Conversely, the per- formance of Adapter and LoRA mostly compa- rable to FFT, although there are cases where they either outperform or underperform FFT. In cases in which Adapter and LoRA perform better than FFT with significant differences, the degree of im- provement is smaller than what is observed with UniPELT and Prefix-tuning.
To evaluate the overall performance of an AL method, we employ the area under the performance curve (AUC). In each individual AL step with specific quantity of labeled examples, measure the classification performance in terms of the Fi score. The overall AUC is calculated using the Fi scores obtained at each step. We advocate for using AUC alongside the AL curves, as AUC serves as a suitable approximation of AL feasibility through a summary numeric score, as recommended in re- cent AL literature (Schröder et al., 2022; Jukić and Snajder, 2023).
As experiments involve different training regimes, we compare each AL sampling strategy SAL to passive learning Sp1, within the same train- ing regime to isolate the effects AL. The primary objective of AL is to improve label efficiency over passive learning. To test whether AL is successful, we calculate the relative improvement over passive learning (RIPL), which we define as follows:
Next, we look into how the models' performance changes as the training set increases. To that end, we show the corresponding learning curves for adapters and FFT in Figure 1. The performance disparities between adapters and FFT become more apparent under conditions of extreme data scarcity (100-300 labeled instances). Notably, the great- differences in performance occur at the initial step (only 100 labels). This highlights the promise of adapter- based methods low-resource settings, particularly for Prefix tuning and UniPELT.
SpL AUC(SAI.)
AUC(Sp.)
AUC(SPL)
Intuitively, RIPL estimates the proportion of maxi- mum possible improvement achievable by given AL method compared to the passive learning base- line. A score of indicates the maximum theoret-

---

0.00
0.34
0.65
FFT
0.50
Lo
Adspper
LORA
LIAA
0,00
LofA
0.68
Prati Lurng
maltecaring
0.75
Pretocluning
IN
UNSTELI
UNIEETT
0.88
200
432
too
002
2002
200
400
tbo
1020
0.0
250
462
000
200
2200
202
400
100
033 2002
abeles dath
inbeled data
labeled 0323
of litees zotn
(a) SUBJ
(b) TREC
(c) SST
(d) AGN
Figure I: Learning curves under the passive learning setup with different PEFT methods and FFT. The results are averaged over five runs. The shaded bands denote the standard deviation. Best viewed computer screen.
SUBJ
TREC
SST
AGN
PEIT
Adapter
.266
804
.800
.871
0.20
PORT
udapters
5009 TAPT
LoRA
.929
.750'
.798
.860
Prefix-tuning
.936'
8471
.847
.8751
UniPELT
.944
.877!
.836
.875
0.05
FFT
.928
.810
.860
0.00
Table The performance of adapters and FFT in pas- sive learning setup in of the AUC metric (based on Fi score) averaged over five runs. Numbers in bold represent hest-performing variant a particular dataset. The "t" symbol indicates when the mean AUC of an adapter is significantly different from the corre- sponding mean AUC of FFT (p < .05 using two-sided Man-W Man-Whitney Whimny U test adjusted for family-wise error rate with the Holm-Bonfente Bonferroni method).
-0.05
sun)
AEN
Figure Comparison of best-performing adapters and FFT from Table 2 and their corresponding version with TAPT applied.
entropy-based methods and TAPT when adapters are employed.
Furthermore, we observe that without TAPT, adapters achieve larger gains over FFT. However, when TAPT is applied, FFT becomes comparable to PEFT, although Prefix-tuning and UniPELT still yield the greatest improvements, depending the dataset and AL method used. In Figure 2, we select the adapters that achieved the best improvement according to Table without TAPT and show their RIPL value compared against FFT as well as their corresponding version when TAPT is applied. We conjecture that TAPT reduces the performance gap between adapters and FFT by inducing FFT to emu- late PEFT in aspects such as training dynamics and representation space hypothesis we explore in more detail in Section 5.
4.2 PEFT with AL
Motivated by our initial findings using PEFT un- der the passive learning setup. where PEFT exhib- ited promising properties in low-resource settings, we further explore the behavior adapters in AL scenarios. We evaluate individual PEFT methods in AL scenarios with and without using TAPT in terms of gains over random sampling (passive learn- ing) using the RIPL metric described in Section 3.5. Table 2 shows the results for different combinations of AL methods and adapters, evaluated through the RIPL metric. We complement these results with absolute values in terms of AUC (cf. Appendix Ta- ble 5). For FFT without TAPT, DAL achieved the highest RIPL score on two datasets, while CS and MC topped the chart on one dataset each. When we incorporated TAPT, ENT yielded the best results on three out of four datasets, with CS leading on one. Looking at adapters, the most successful AL meth- ods without TAPT vary, depending on the specific adapter and dataset in question. Interestingly, when TAPT applied, the best results for adapters are obtained either by ENT or MC. We speculate this could be attributed to solid compatibility between
We further investigate the behavior adapters with AL throughout the individual steps. Fig- ure shows the learning curves for corresponding adapter models with and without applying TAPT. Due to space constraints, we show the learning curves only for the SUBJ dataset, similar trends occur for other datasets. Without TAPT, the per- formance of adapters is largely independent of the specific AL method used, where Prefix-tuni -unnin and UniPELT consistently outperform Adapter and LoRA across all AL steps. With TAPT, the differ-

---

without TAPT
with TAPT
ENT
MC
CS
DAL
ENT
MC
CS
DAL
FFT
.050
.059
.061
.077
.140
.140
.142
.126
Adapter
.112
.102
.100
.092
.137
.151
.111
.067
SUBT
LoRA
.127
.115
.091
.081
.165
.160
.122
.100
Prefix-tuning
.095
.110
.106
.111
.186
.181
.170
.151
UniPELT
.129
.153
.131
.128
.159
.167
.163
.157
FFT
.022
.038
.162
.180
.141
.159
Adapter
.027
.069
.137
.084
.124
.146
.079
.154
TREC
LoRA
.098
.065
.048
.007
.254
.237
.243
.074
Prefix-tunin tuefix-tuning
.093
.105
.068
.093
.216
.227
.205
.211
UniPELT
.138
.165
.082
.200
.302
.334
.276
.236
FFT
.002
.011
.039
.004
.080
.079
.075
.070
Adapter
.015
.048
.025
.035
.034
.028
.008
LoRA
.001
.007
.064
.031
.036
.022
.082
.014
Prefix- tuning
.060
.114
.152
.143
.137
.126
UniPELT
.037
.043
.000
.082
.101
.083
.800
FFT
.014
.032
.007
.092
.134
.021
.089
.017
Adapter
.074
.046
.015
.062
.115
.089
.077
.080
LoRA
.020
.025
.067
.016
.028
.102
.071
.023
Prefix-tuning
.054
.023
.040
.033
.035
.143
.098
.092
UniPELT
.074
.096
.089
.095
.155
-151
.112
.081
Table Improvement over passive learning in terms of CS, and DAL) and for all combinations of adapters and datasets considered, shown separately without TAPT and with TAPT. Positive values indicate improvement over passive learning, while negative values indicate performance drops compared to passive learning. Values in bold denote the best result for particular dataset across different adapters and AL methods within the same regime (with without TAPT).
metric
AL methods considered (ENT, MC,
tuning
0.94
0.95
0.95
ssoree
0.94
0.92
0.93
0.93
0.92
0.88
0.90
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800 1000
200
400
600
800
1000
of labeled deta
of labeled data
of labeled data
# of labeled data
Adapter
TAPT
0.96
0.95
0.95
F, scooee
scoree
0.94
0.99
0.92
0.92
0.92
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
of labeled data
of labeled data
of labeled data
# of labeled data
0.960
0.94
0.955
RND
C5
sccore
0.850
ENT
DAL
MC
0.945
0.88
0.86
200
400
600
800
1000
200
400
660 80o
1000
of labaled data
of labeled data
Figure 3: AL learning show learning curves for adapters without and with TAPT. respectively. The third row shows learning curves for FFT, without and with TAPT. The results are averaged over five runs, and the shaded bands denote the standard deviation. Best viewed on computer screen.
compared with random sampling the SUBJ dataset. The first and the second rows

---

ences between AL and random sampling are more pronounced starting from the early steps, typically already with 200 instances. In contrast, the gap becomes more apparent only with 500 or more in- stances when TAPT is not employed.
fication task, respectively. Specifically, we com- pare RND with MC, which achieves consistent per- formance improvements across all datasets. Our findings suggest that FFT tends to select a higher number of unforgettable instances and fewer mod- crately forgettable instances when compared to adapters. Interestingly, the adapters that perform best Prefix-tuning and UniPELT appear to favor moderately forgettable instances. However, when TAPT is applied, the discrepancies in forget- ting profiles between FFT and the top two adapters, Prefix-tuning and UniPELT, seem to diminish. contrast, TAPT amplifies the differences between FFT and the other two adapters, LoRA and Adapter, which typically show smaller improvements than Prefix-tunnng and UniPELT. Given their superior AL performance, hypothesize that the forgetting profiles of Prefix-tuning and UniPELT are more faaorable compared to other adapters. Moreover, FFT with TAPT approaches the performance of the superior adapters and simultaneously develops a forgetting profile similar to theirs.
5 Analysis
In Section 4, we have demonstrated that PEFT ex- hibits larger gains than FFT when combined with AL in low-resource settings, which is also accompa- nied by superior performance with passive leaning. To better understand why PEFT displays superior behavior with limited data, now examine two specific properties of adapters and FFT models. First, we analyze the influence of TAPT on the for- getting dynamics during training. We continue with example-level representation analysis, where we investigate the representation similarity of PEFT and FFT to their respective base models.
5.1 Forgetting dynamics
We employ forgetting dynamics to compare PEFT and FFT's learning stability and their impact on AL data selection. The underlying hypothesis is that having fewer forgetting events in adapters would indicate more stable and effective learning pro- cess. In utilizing forgetting dynamics, we draw upon the study by Toneva et al. (2019), focusing on the occurrence of forgetting events - cases where a specific training example transitions from cor- rect incorrect classification over the course of multiple learning epochs. More specifically, divide the instances into three categories: (1) un- forgettable instances, i.e., the ones that have never experienced a forgetting event during training, (2) instances that have encountered one or two forget- ting events, labeled as moderately forgettable, and (3) instances subjected to three or more forgetting events, referred to highly forgettable instances. As pointed out in the original study, moderately forgettable, ambiguuus instances are more valuable for the learning model than unforgettable, easy in- stances. However, it is worth noting that AL is often hindered by too hard or impossibbe-to-leare examples (Karamcheti et al., 2021), which roughly correspond to the highly forgettable examples.
5.2 Representation analysis
To bolster our findings, we explore the representa- tions of adapters and FFT models. As suggested in previous research (He al., 2021: Li and Liang, 2021; Mao al., 2022), adapters often display greater stability terms loss, especially in sce- narios with limited resources. Our aim is to exam- ine the stability of their representations and their relationship with overall AL performance.
We draw inspiration from research by Stephen- son et al. (2021) and Baldock et al. (2021), which suggests that different layers of networks specialize in different features earlier layers tend to acquire more generalized knowledge, while the deeper lay- more focused on task-specific information. This leads us layerwise examination of similar- ity. To analyze the effect of PEFT and FFT on AL selection with respect to their layerwise similarity to the base model, we utilize centered kernel align- ment (CKA) as a similarity measure between two sets of representations (Kornblith et al., 2019). It has been shown that PEFT methods result in rep- resentations closer to the base model at the token level (He et al., 2021). We extend the analysis example-level representation to explore the behav- ior of models with AL. We opt for CKA as is designed to be invariant to invertible linear transfor- mation and still can measure meaningful similari-
Figure& shows the distribution of instances across the three categories of forgetting events for SURJ and TRFC datasets. We focus on these two datasets as examples of a simple binary classifica- tion task and a more complex multi-class classi-

---

100
566-
593
Laba
1-60)
Profie turing
832
Profic
333
LYAPELT
UN
190
resderace
TANST
moder
tigh
(a) SUBJ; RND
(b) SUBE; MC
(c) SUBI; MC TAPT
932
Lata
Partolurin
400
hetaluriing UYPELT
FE
303
400-
103
noxd
medeess
hggt
(d) TREC; RND
(c) TREC; MC
(f) TREC; MC TAPT
Figure 4: Forgetting dynamies for random sampling (passive learning) and AL with MC without and with TAPT SUBJ and TREC. The x-axis shows the number of instances in each of the forgetting categories: the "never" category representing unforgettable instances, moderately forgettable instances, highly forgettable instances.
12
(a) ENT
(b) MC
(c) Cs
(d) DAL
(c) RND
Figure 5: Layerwise difference in representation similarity for the UniPELT adapter and the FFT model on SUBJ. We observed similar patterns in other adapters and datasets we used (cf. Appendix B). Warm colors (positive values) illustrate layer pairs that demonstrate higher similarity to the base model with the adapter than with FFT. Conversely, cool colors (negative values) represent layer pairs that are more similar to the base model when using the FFT model. Best viewed on a computer screen.
ties between representations of higher dimensions than the number of data points. This stands in con- trast to other metrics, which frequently falter when dealing with high-dimensional representations.
holds for all AL methods, with differences more pronounced than in passive learning. Specifically, up to the eighth layer, representations are much more similar in adapters than in FFT models. In the final four layers, the difference in CKA scores between the adapter and FFT model close to zero. Interestingly, the penultimate layer more similar in the FFT model with respect to the base model.
For more direct comparison between PEFT and FFT, analyze the differences between their respective similarities their base mod- els. Specifically,, we compute the difference CKA(adapter, base) (ee))-CK(FFFT) base) for a spe- cific adapter or FFT and their base models. We hypothesize that superior PEFT performance with AL compared to FFT will be accompanied by more similar early layer representation to the base model in PEFT. Figure 5 visualizes the layerwise difference in similarity between the base model and the adapter model and between the base model and the FFT model. We find that PEFT representations are more similar to the base model the early and middle layers when compared to FFT. This
When fine-tuning on a downstream task, we be- lieve that the increased stability PEFT earlier layers, relative to FFT, is instrumental in retaining the foundational knowledge from the PLM's pre- training phase. Conversely, PEFT exhibits more substantial transformations in the later, more task- specific layers. This ensures the preservation of essential pre-trained knowledge while allowing for task-relevant flexibility. We speculate that this strategic balance PEFT influences its propensity to select moderately forgettable instances when

---

combined with AL, contributing to its enhanced performance over FFT. These instances are neither too trivial to provide no learning value, nor are they too complex to risk misinterpretation, thereby enhancing the effectiveness of learning.
to be comprehensive in our experiments, the out- comes might vary with different models, strate- gies, or methods. For example, the effectiveness of AL combined with PEFT might differ if other AL strategies are employed. Similarly, different types of adapter architectures could potentially lead to different results.
6 Conclusion
Although we found that PEFT methods produce instance-level representations of early and middle layers more similar to the base PLM than FFT, comprehensive understanding of how and why this similarity Ieads to increased stability and per- formance in low-resource settings is still lacking, Our hypothesis about the role of early and middle layer stability in mitigating the issue of forgetting the knowledge obtained during pre-training needs further substantiation.
Our study has shed light on the advantages of purameter-efficient fine-tuning (PEFT) in low- resource settings, confirming its superiority over full fine-tuning (FFT) methods. Importantly, have demonstrated that the integration of PEFT with active learning (AL) can offer substantial per- formance gains compared to passive learning, even in settings where labeled data is scarce. Further- more, we highlighted the potential of task-adaptive pre-training (TAPT) to improve model performance further when used in conjunction with both PEFT and AL. We found that AL methods, in combina- tion with PEFT, tend to select fewer unforgettable instances and more moderately forgettable exam- ples. We further found that PEFT maintains the integrity of early and middle layer representations similar to the base model. We conjecture that this property mitigates forgetting during downstream task fine-tuning, These insights inform us of a possible underpinning mechanism that contributes PEFT's superior performance and stability in low-resource settings. Overall, our work highlights the potential of PEFT and AL and establishes a foundation for developing increasingly efficient and cost-effective approaches for training models in low-resource settings.
Finally, it important acknowledge the com- plexity and multifaceted nature of forgetting dy- namies. While our investigation provides valuable insights about the interaction of forgetting with PEFT and TAPT in AL scenarios, deeper under- standing of the mechanisms of forgetting in the con- text of large PLMs is needed. Particularly, it would be interesting investigate whether the balance between unforgettable and moderately forgettable instances selected by the AL methods changes the size of the model or the amount of available data changes.
Future work should aim to address these limita- tions and further explore the mechanisms behind the promising results obtained with the combina- tion of PEFT and AL. This will contribute to a more comprehensive understanding of the interaction be- tween AL and PLMs, and help refine strategies for efficient fine-tuning in low-resource settings.
Limitations
While our study advances the understanding of PEFT and AL's interaction in low-resource settings and uncovers intriguing insights about the forget- ting dynamics during fine-tuning, it has number of limitations.
References
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Se- bastian Ruder, Goran Glavaš, Ivan Vulié, and Anna Korhonen. 2021. MAD-G: Multilingual adapter gen- eration for efficient cross- mons-lingual transfer. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 4762-4781, 1, Punta Cana, Dominican Republic. Association for Computational Linguistics.
To begin with, we have focused on text classifi- cation tasks, which are but one aspect of the wide range of potential applications for PLMs. Different tasks such as question answering, translation, or summarization might exhibit different behaviors under the same conditions. Consequently, the ob- served advantuges of PEFT in the context of AL might not necessarily translate other NLP tasks.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. 2021. Deep learning through the lens of example difficulty. In Advances in Newral Infor- mation Processing Systems, volume 34, pages 10876- 10889, Curran Associates, Inc.
Next, our results are limited to the specific PL.Ms, AL strategies, and PEFT methods we have ex- amined this study. While we have attempted
David A Cohn, Zoubin Ghahramani, and Michael

---

Jordan. 1996. Active learning with statistical models. Journal of artificial intelligence research, 4:129-144 -95.
Ruidan He, Linlin Liu, Hai Ye. Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si, 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 2208- 2222, Online. Association for Computational Lin- guistics.
Sanjoy Dasgupta. 2011. Two faces of active learning. Theoretical Computer Science, Algorithmic Learning Theory (ALT 2009).
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, Parameter-efficien transfer learning for NLP. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hamnaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight ini- tializations, data orders, and early stopping. arXiv preprint urXiv:2002.000 06305.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Liat Fin-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7949-7962, Online. Association for Computa- tional Linguisties.
Josip Jukié and Jan Snajder. 2023. Smooth sailing: Im- proving active learning for pre-trained language mod- els with representation smoothness analysis. In Pro- ceedings of the 2023 CLASP Conference on Learning with Small Data (LSD), pages 11-24, Gothenburg. Sweden. Association for Computational Linguistics.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learn- ing, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA. PMLR.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. 2021. Mind your outliers! in- vestigating the negative impact of outliers active learning for visual question answering. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), ), pages Association for Computational Linguisties
Daniel Gissin and Shai Shalev-Shwartz 2019. Dis- criminative active learning. arXiv preprint
Online.
arXiv:1907.06337.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghami, and James Henderson. 2021. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 565-576, Online. Association for Computational Linguistics.
Daniel Grießhaber, Johannes Maucher, and Ngoc Thang Vu. 2020. Fine-tuning BERT for low-resource nat- ural language understanding via active learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1158-1171, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages Online. Association for Computational Linguistics.
Seungwon Kim, Alex Shum, Nathan Susanj. and Jonathan Hilgart. 2021. Revisiting pretraining with adapters. Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-2022 pages 90-99, Online. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of -fffmeter-effiiien transfer learning. International Conference on Learning Representa- tions.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR.

---

Jaeseong Lee. Seung-won Hwang, and Taesup Kim. 2022. FAD-X: Fusing adapters for cross- -ongull transfer to low-resource languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Lan- guage Processing (Volume 2: Short Papers), pages 57-64, Online only, Association for Computational Linguistics.
Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022. BAD-X: Bilingual adapters im- prove zero-shot cross-lingual transfer. In Proceed- ings of the 2022 Conference of the North Amert- can Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 1791-1799, Seattle, United States. Association for Computational Linguistics.
Jonas Pfeiffer, Andreas Rückléé Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho. and Iryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference Empiriaal Methods in Nat- ural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguisties.
David D Lewis and William A Gale. 1994, A sequential algorithm for training text classifiers. In SIGIR 994 pages 12. Springer.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulié, and Edoardo Maria Ponti. 2023. Modular deep learning. arXiv preprint arXhv:2302.11529
Christopher Schröder, Andreas Niekler, and Martin Potthast. 2022. Revisiting uncertainty-based query strategies for active learning with transformers. In Findings of the Association for Compulational Lin- guistics: ACL 2022, pages 2194-2203, Dublin, Ire- land. Association for Computational Linguistics.
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. UniPELT: A unified framework for paraieter-efffiieent language model tuning. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for for Computational Linguistics (Volume 1: Long Papers), pages 6253-6264, Dublin, Ireland. Associa- tion for Computational Linguistics.
Ozan Sener and Silvio Savarese. 2018. Active learn- ing for convolutional neural networks: A core-set approach. In International Conference on Learning Representations.
Burr Settles. 2009, Active learning literature survey. Computer sciences technical report, University of Wisconsin-Madison.
Katerina Margatina, Loic Barrault, and Nikolaos Ale- tras. 2022. On the importance of effectively adapting pretrained language models for active learning. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 825-838 Dublin, Ireland. As- sociation for Computational Linguistics.
Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov. Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, and Alexander Panchenko. 2021. Active learning for sequence tagging with deep pre-trained models and Bayesian uncertainty estimates. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1698- 1712, Online. Association for Computational Linguistics.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021. Active learning by ac- quiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650-663, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Richard Socher, John Bauer, Christopher D. Manning. and Andrew Y. Ng. 2013. Parsing with composi- tional vector grammars. In Proceedings of the S1st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455 465, Sofia, Bulgaria. Association for Computational Lin- guistics.
Marius Mosbach, Maksym Andriushchenko, and Diet- rich Klakow. 2021. On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations.
Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271-278, Barcelona, Spain.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Re- search, (((56):19291999)

---

Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Sue Yeon Chung. 2021. On the geometry of generalization and memorization deep neural networks. International Conference on Learning Representations.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey Gordon. 2019. An empirical study example forgetting during deep neural network learning. In International Conference Learning Representa- tions.
Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang, and Chao Zhang. 2022. AcTu Uncertainty-bant active self-training for active fine-tuning of pretrained language models. Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies, pages 1422- 1436, Seattle, United States. Association for Computational Lin- guistics.
Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd- Graber. 2020. Cold-start active learning through self- supervised language modeling. Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7935-7948, Online. Association for Computational Linguistics.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Wein- berger, and Yoav Artzi. 2021. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28.

---

TRAIN
VAL
TEST
TOTAL
FFT
Adapter
LoRA
Prefix-tuning UniPELT
SUBJ
7,000
1,000
0000
10, 10,000
SUBI
40.8
4.3
3.2
4.1
6.2
SST
6,647
868
1, 12425
8,940
TREC
68.4
4.9
5.4
5.9
8.4
TREC
8,881
452
500
5,833
SST
43.9
5.1
5.0
4.9
7.6
20, 000
600
7, 6,0
35, 200
72.1
7.3
6.1
4.4
9.3
Table 3: Dataset sizes by splits. Although we do not re-
Table 4: Experiment duration in minutes for all models across datasets. We report the average runtime over five different runs and five different sampling methods (five AL methods and random sampling).
use validation set (VAL) in our experiments, port its size for completeness. For the AGN dataset, we performed uniform subsampling ensure the computa- tional feasibility of the experiments.
A.4 Preprocessing
A Reproducibility
We undertake few pre-processing steps: con- vert all tokens to lowercase, eliminate non- alphanumeric tokens, and limit the token sequence to a maximum length of 200.
A.1 Dataset statistics
The sizes of the datasets per split are provided in Table 3. Predominantly, the datasets encompass texts in English.
A.5 Hyperparameters
We use a fixed learning rate of 2 10-5 for FFT and 10-4 for adapters. Additionally, we set the gradient clipping to 1 during training. In our im- plementation of TAPT, we randomly mask 15% of tokens for both FFT models and adapters and train the model for 50 epochs with the learning rate set to 10 5
A.2 Adapters
We the implementation of adapters from AdapterHub (Pfeiffer et al., 2020).
Adapter We set reduction factor to 16 and use swish function as nonlinearity.
A.6 Computing infrastructure
LoRA We include LoRA the self-attention weights, intermediate, and output MLP weights of model. We set the rank of the LoRA layer and the scaling factor a to 8.
We conducted our experiments 4x AMD Ryzen Threadripper 3970X 32-Core Processors and 4x NVIDIA GeForce RTX 3090 GPUs with 24GB RAM. We used PyTorch version 1.9.0 and CUDA 11.4.
Prefix-tuning We use tanh activation for Prefix- tuning, with prefix length set 30 and bottle- neck size of 512.
A.7 Average runtime
We report the average runtime of experiments in Table 4.
UniPELT We use Adapter, LoRA, and Prefix- tuning as components of UniPELT with the same hyperparameters as described for indi- vidual components. The only exception is that set the prefix length for Prefix-tuning to 10 instead of 30.
B Additional Results
We report the results that were omitted from the main part of the paper due to space constraints. Ta- ble 5 shows AUC scores for different combinations of AL methods and adapters, complementing the relative improvement scores AUC represents ab- solute scores for each configuration. In Figure 6, display the difference in similarities of adapters and FFT compared to their base models on the remaining three datasets.
A.3 AL methods
MC In experiments, we use ten inference cycles to approximate the entropy of the output via Monte- Carlo dropout sampling.
CS We use the [CLS] token representation from the Transformer's penultimate layer. We follow the greedy method described the original work (Sener and Savarese, 2018)

---

without TAPT
with TAPT
RND
ENT
MC
DAL
RND
ENT
MC
DAL
FFT
.928
.311
.932
.932
.931
.938
.947
.917
.917
.916
Adapter
.926
.931
.933
.933
.932
.934
.943
.944
.941
.938
SUBI
LoRA
.929
.938
.937
.935
.934
.935
.946
.945
.943
.942
Prefix-tuning
.936
.42
.943
.943
.943
.940
.951
.951
.950
.949
UniPELT
.934
.943
.944
.943
.942
.943
.952
.953
.952
.952
FFT
810
.812
.814
.817
.816
.818
.847
.851
.844
.847
Adapter
804
.809
.818
.831
.820
.820
.842
.846
.834
.888
TRBC
LoRA
.50
.775
.766
.762
.752
.764
8224
.220
.821
.781
Prefix-tuning
847
.861
.863
.857
.861
.862
.896
.893
.890
.995
UniPELT
.87
.994
.897
.887
.902
.896
.927
.931
.925
.921
FFT
.777
.787
.789
.779
.788
.792
.809
.088
.808
807
Adapter
800
.033
.810
.855
.801
812
.819
.188
.877
.844
SST
LoRA
798
.798
.799
.811
.804
.806
.813
.810
.812
.899
Prefix-tuning
817
.854
.866
.664
.852
.668
.888
.887
.886
.885
UniPELT
836
.842
.843
.843
.837
.871
.882
.884
.882
.881
FFT
.60
.864
.861
.873
.869
.887
.872
.881
.871
Adapter
.871
.877
.873
.879
.882
.893
.891
AGN
LoRA
.600
.863
.863
.869
.862
.868
.872
.881
.877
.871
Prefix-tuning .875
.880
.879
.866
.890
.902
.897
UniPELT
.875
.884
.887
.886
.887
.887
.908
.904
.900
.896
Table 5: AUC scores for AL methods with different adapters shown separately without TAPT and with TAPT. We include random sampling for comparison with AL methods. Values bold denote the best result particular dataset within the same regime (with or without TAPT).
TREC; ENT
(b) TREC; MC
TREC;
(d) TREC; TREC; RND
f
(g) SST; SST;
(i)
ii) SST;
-3
(k) AGN; ENT
(I) AGN: MC
(m) AGN;
(n) AGN; DAL
(0) AGN; RND
Figure 6: Layerwise difference in representation similarity for the UniPELT adapter and the FFT model on TREC, SST, and AGN. The differences are computed as CKA(adapter, base) CKA(FFT, base), where base is the corresponding pre-trained BERT model. Warm colors (positive values) illustrate layer pairs that demonstrate higher similarity to the base model with the adapter than with FFT. Conversely, cool colors (negative values) represent layer pairs that are more similar the base model when using the FFT model. Best viewed on a computer screen.


Reviews:
[{"review_id": "HjO3aFnjr2", "reviewer": "Reviewer_bhGY", "summary": "", "questions": "1. My suggestion is to focus the article more on AL, and the PEFT section as a well defined related work would be fine.\n2. More experiments on various tasks are still needed, or could you please explain why only text classification tasks were selected and how AL performs differently on this task compared to other tasks.\n3. Can you provide original numbers of each method (i.e. accuracy). This will make the results appear more intuitive. Is there a significant improvement by combining AL into PEFT?\n4. If AL affects the results, will the selection of 1000 data also affect the results? Have you tried sampling different data each time？", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and for raising a number of important questions.\n\n> Comparsion of PEFT and FFT lacks innovation. The experimental setup is no much different from previous work, and the authors only reprouduce the results on these datasets.\n\nPlease note that our primary interest lies in the intersection of AL and PEFT in **low-resource** settings. This is an aspect we believe hasn't been explored previously, which we also highlight in the paper: “However, these findings [of previous research] were based on comparing FFT to a single adapter variant on a full dataset or evaluating the performance at only a few discrete points.” Thus, even though PEFT's efficacy is well-documented, we offered detailed learning curves in low-resource settings. However, for reason of completeness and as a sanity check, we did undertake an initial comparison between PEFT and FFT.\n\n> RIPL is already a relative indicator compared to before. The Table reports the relative results of the relative indicators, and it is difficult to see how much absolute improvement there has been.\n\nWe acknowledge that the RIPL metric reports the proportion of relative improvement, which makes it inappropriate to measure absolute improvement. Nevertheless, we believe that the RIPL metric allows for a clear comparison of AL methods across different datasets, given that it measures the relative improvement over random sampling. In addition, to provide a comprehensive perspective, we present active learning curves that illustrate the absolute performance scores, as depicted in Figure 3. We believe that offering both the AL learning curves and the relative improvement contributes to a more comprehensive understanding of the results.\n\n> Q1: …why only text classification tasks were selected and how AL performs differently on this task compared to other tasks\n\nOur emphasis is on text classification tasks, given their prominent coverage in AL research. As we explore novel combinations like PEFT with AL, we wanted to make sure that our results are comparable with existent AL literature [1,2]. Regarding the performance on other types of tasks such as sequence labeling, AL has also shown improvement over random sampling over various domains (e.g., [3,4]). We certainly agree that including additional tasks would be valuable, but we decided to stick to classification due to resource limitations.\n\n> Q2: Can you provide original numbers of each method (i.e. accuracy). This will make the results appear more intuitive. Is there a significant improvement by combining AL into PEFT?\n\nIf we have understood correctly, the question is whether we can provide performance scores in isolation. We have indeed reported the performance in terms of the F1 score for each PEFT method in isolation and in combination with every AL method, comparing it to FFT. Additionally, we included the results with and without applied TAPT, which are all shown as learning curves in Figure 3. For each setting, we show the learning curve with F1 scores at each point, corresponding to specific numbers of available training instances ranging from 100 to 1000. As reported in the paper, AL shows improvement over random sampling across the board (on datasets we used) when combined with PEFT.\n\n> Q3: If AL affects the results, will the selection of 1000 data also affect the results? Have you tried sampling different data each time?\n\nWe presume that the question pertains to the effect of retrieving different random samples, i.e., having multiple runs. To account for that, we repeat each experiment five times with different random seeds as stated in the paper: “For every setting, we perform five runs using different random seeds. We report the average F1 score at each sampling step (with and without AL for FFT and PEFT) to show the corresponding learning curve averaged over five runs.” We are effectively drawing different random samples in each AL step for different runs.\n\n[1] On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (Margatina et al., ACL 2022)\\\n[2] Active Learning for BERT: An Empirical Study (Ein-Dor et al., EMNLP 2020)\\\n[3] Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates (Shelmanov et al., EACL 2021)\\\n[4] SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup (Zhang et al., EMNLP 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper explores the interplay between active learning and adapter-based parameter-efficient fine-tuning.\n2. The authors provide empirical evidence to support their claims, conducting experiments on text classification tasks in low-resource settings.\n3. The analysis of forgetting dynamics and instance-level representations further enhances the understanding of the behavior of these techniques.\n4. The paper highlights the potential synergy between AL and PEFT, which can lead to advancements in efficient and effective fine-tuning. ", "reasons_to_reject": "1. Comparsion of PEFT and FFT lacks innovation. The experimental setup is no much different from previous work, and the authors only reprouduce the results on these datasets. Also, the viewpoints in the analysis （Sec 4.1）have also been discussed many times in previous work (e.g. Sec 4.3 in UNIPELT).\n2. The paper focuses solely on text classification tasks in low-resource settings. Expanding the scope to include other domains or tasks would enhance the generalizability of the findings and make the research more applicable to a wider range of scenarios. Although the author mentioned this in the limitation, the comparative work has already been done on more tasks (e.g. GLUE Benchmark).\n3. Table 2 is not intuitive enough. RIPL is already a relative indicator compared to before. The Table reports the relative results of the relative indicators, and it is difficult to see how much absolute improvement there has been.", "questions_to_authors": "1. My suggestion is to focus the article more on AL, and the PEFT section as a well defined related work would be fine.\n2. More experiments on various tasks are still needed, or could you please explain why only text classification tasks were selected and how AL performs differently on this task compared to other tasks.\n3. Can you provide original numbers of each method (i.e. accuracy). This will make the results appear more intuitive. Is there a significant improvement by combining AL into PEFT?\n4. If AL affects the results, will the selection of 1000 data also affect the results? Have you tried sampling different data each time？", "paper_topic_and_main_contributions": "The paper presents an empirical study on the behavior of parameter-efficient fine-tuning (PEFT) in low-resource settings for text classification tasks, specifically in the context of active learning (AL). The authors compare PEFT with full-fine tuning (FFT) and demonstrate the superiority of PEFT in low-resource settings, even in AL setups. They also analyze the forgetting dynamics and instance-level representations of PEFT and FFT, finding that PEFT yields more stable representations in early and middle layers. The paper highlights the synergistic potential of AL and PEFT in low-resource settings, emphasizing the advancements in efficient and effective fine-tuning.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sl5B70COYL", "reviewer": "Reviewer_iSq3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback.\n\n> As mentioned in related work, previous studies have shown the importance of TAPT for active learning. However, the paper lacks a sufficient analysis of the challenges and specificity of combining PEFT and TAPT in the active learning framework, making the proposed method seem like a simple combination of two techniques.\n\nWe posit that integrating PEFT with TAPT can be straightforwardly achieved by pre-training the model on a masked language modeling task specific to a given dataset, with updates confined to the adapter parameters. Subsequent to this step, the active learning process is carried out as it would be with any other model. To our knowledge, this procedure doesn't introduce any unique challenges.\n\n> Prompt tuning is also a common method in PEFT and has been proven to be effective in few-shot learning. It would be more convincing if the authors could provide relevant experiments on prompt tuning.\n\nIn our exploration of PEFT methodologies, we aimed to cover a diverse range of approaches. We did not prioritize prompt-tuning, mainly because studies [1,2] have demonstrated that soft prompt-tuning tends to underperform for smaller language models like BERT base, which we utilized in our research. This method appears to be more effective for considerably larger models, such as BERT large and RoBERTa large. Opting for a representative from the sequence optimization techniques, we chose prefix-tuning due to its consistent performance across varying sizes of language models [3].\n\n> The current experiments focus on datasets with a limited number of sample categories. To better validate the effectiveness of the proposed method, it is recommended to select benchmark datasets with tens of categories, such as relation classification, etc.\n\nWe chose datasets that are frequently used in active learning research [4,5]. Among the multi-class datasets, we utilized TREC, which has six classes, and AG news with four classes. While we recognize the value of incorporating datasets with a greater number of classes, we were constrained by the available compute resources.\n\n> The paper uses BERT as the base backbone. Existing work has demonstrated the good performance of large language models with over billions of parameters in few-shot learning. Including experiments based on LLMs will further strengthen the paper's persuasiveness.\n\nWe agree that the inclusion of larger language models would be advantageous. However, due to our hardware constraints, we opted for models that were within our computational capacity.\n\n***\nWe accidentally omitted the information regarding the batch size for the re-training step in each AL iteration and our use of default parameters for PEFT methods. We thank the reviewer for pointing it out. These details will be addressed in our updated version.\nRegarding learning rate optimization, it's customary in active learning to employ a consistent learning rate that is effective across various tasks [4,5], especially given the frequent absence of a validation set in real-world active learning contexts. In addition, we use early stopping that does not require a validation set.\n\n[1] What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (Kim et al., EMNLP 2021)\\\n[2] The Power of Scale for Parameter-Efficient Prompt Tuning (Lester et al., EMNLP 2021)\\\n[3] UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (Mao et al., ACL 2022)\\\n[4] Active Learning for BERT: An Empirical Study (Ein-Dor et al., EMNLP 2020)\\\n[5] Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers (Schröder et al., Findings 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper explores the application of parameter-efficient tuning and task-adaptive pre-training in active learning and conducts extensive experiments to demonstrate the effectiveness of PEFT in active learning.\n2. The authors analyze the reasons behind the better performance of PEFT from the perspectives of forgetting dynamics and instance representations, which can provide valuable insights for future work.", "reasons_to_reject": "1. As mentioned in related work, previous studies have shown the importance of TAPT for active learning. However, the paper lacks a sufficient analysis of the challenges and specificity of combining PEFT and TAPT in the active learning framework, making the proposed method seem like a simple combination of two techniques.\n2. Prompt tuning is also a common method in PEFT and has been proven to be effective in few-shot learning. It would be more convincing if the authors could provide relevant experiments on prompt tuning.\n3. The current experiments focus on datasets with a limited number of sample categories. To better validate the effectiveness of the proposed method, it is recommended to select benchmark datasets with tens of categories, such as relation classification, etc.\n4. The paper uses BERT as the base backbone. Existing work has demonstrated the good performance of large language models with over billions of parameters in few-shot learning. Including experiments based on LLMs will further strengthen the paper's persuasiveness.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper attempts to combine active learning and parameter-efficient tuning for large-scale pre-trained language models. The authors implement four active learning algorithms combined with four commonly used PEFT algorithms. Through experiments on four text classification datasets, they demonstrate that compared to full-parameter fine-tuning, PEFT is more effective in few-shot learning, and active learning further improves the performance of few-shot learning. Additionally, the authors conduct model analysis from the perspectives of forgetting dynamics and instance representations.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. More training details, such as batch size, training steps, and hyper-parameters of various PEFT algorithms, should be provided.\n2. According to Appendix A.5, the authors only use one learning rate to train the models. It is recommended to perform hyper-parameter search for the learning rate to obtain the best performance for each model."}}, {"review_id": "v0CT7zvEfe", "reviewer": "Reviewer_qrFB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments.\n\n> Validation on only four text classification datasets with on single model architecture (bert, encoder-only transformer) may cause conclusion from the paper overclaim.\n\nWe appreciate the reviewer's feedback and concerns regarding potential overclaims. However, we would be grateful if the reviewer could specify which particular claim they found to be an overreach. In our paper, we have been careful to set the scope of our findings to the PLMs, AL strategies, and PEFT methods we investigated. We highlighted that \"our results are specific to the PLMs, AL strategies, and PEFT methods examined in this study.\" We understand and acknowledge that – as for any other experimental paper submitted to ACL conferences – the results might vary when different models, strategies, or methods are applied. While we see the merit in testing a broader range of PLMs and datasets, computational constraints limited our choices, especially given the diversity of PEFT and AL methods we evaluated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper introduces active learning into PET field in  low-resource settings, which is ignored by previous research.\n\n(2) The reprensentation is good with organzied sections and clear illustration by figures.\n", "reasons_to_reject": "(1) The novelty that simply combines active learning methods with PET methods is a little bit incremental and sounds not that exciting.\n\n(2) Validation on only four text classification datasets with on single model architecture (bert, encoder-only transformer) may cause  conclusion from the paper overclaim.  \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces active learning and combines it with PET methods to further improve fune-tuning performance of NLU tasks in low-resource settings.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work explores parameter-efficient fine-tuning (PEFT) in the context of active learning (AL). The authors show that PEFT methods outperform full fine-tuning in low-resource conditions, and the performance gain with PEFT methods extends to AL settings. 

1. As the reviewer mentioned, the authors present a large set of experiments that includes 4 PEFT methods, 5 AL sampling strategies, and 4 classification datasets. Experimental results are sound and support the authors' claims. 

2. Analysis of forgetting dynamics gives insights into why PEFT methods perform differently in AL scenarios.

Furthermore, the authors provide clarification regarding their experimental setup. Overall, the benefits of PEFTs in AL is an important angle, and the paper addresses such benefits. I believe the authors will take reviewers' suggestions and points into account, such as providing the absolute performance metrics of PEFTs and FFT in AL settings similar to Table 1.
Decision: Accept-Main