Parameter-Efficient Fine-Tuning without Introducing New Latency
Year: 2023

Abstract:
Parameter-efficient fine-tuning (PEFT) of pretrained language models has recently demonstrated remarkable achievements, effectively matching the performance of full fine-tuning while utilizing significantly fewer trainable parameters, and consequently addressing the storage and communication constraints. Nonetheless, various PEFT methods are limited by their inherent characteristics. In the case of sparse fine-tuning, which involves modifying only a small subset of the existing parameters, the selection of fine-tuned parameters is taskand domain-specific, making it unsuitable for federated learning. On the other hand, PEFT methods with adding new parameters typically introduce additional inference latency. In this paper, we demonstrate the feasibility of generating a sparse mask in a task-agnostic manner, wherein all downstream tasks share a common mask. Our approach, which relies solely on the magnitude information of pre-trained parameters, surpasses existing methodologies by a significant margin when evaluated on the GLUE benchmark. Additionally, we introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation, thereby achieving identical inference speed to that of full finetuning. Through extensive experiments, our proposed method attains a new state-of-the-art outcome in terms of both performance and storage efficiency, storing only 0.03% parameters of full fine-tuning.

heading: 1 Introduction
text: Pre-trained language models (PLMs) have served as a cornerstone for various natural language processing applications, favoring downstream tasks by offering a robust initialization (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019, 2020; Brown et al., 2020; Liao et al., 2022). Starting
1Code at https://github.com/baohaoliao/pafi_hiwi
with a pre-trained checkpoint, a model can achieve significantly better performance on tasks of interest than the one from scratch. The most historically common way to adapt PLMs to downstream tasks is to update all pre-trained parameters, full fine-tuning. While full fine-tuning produces numerous state-of-the-art results, it is impractical for storage-constrained and communication-frequent cases, like federated learning (McMahan et al., 2017), since it requires a full copy of the fine-tuned model for each task. This issue becomes more severe when PLMs are large-scale (Brown et al., 2020; Zhang et al., 2022; Hoffmann et al., 2022; Raffel et al., 2020; Scao et al., 2022; Touvron et al., 2023), the number of tasks in interest grows, or data are privately saved on hundreds of servers for federated learning.
An alternative approach popularized by Houlsby et al. (2019) is parameter-efficient fine-tuning (PEFT), where a small number of task-specific parameters is updated and the majority of PLM’s parameters is frozen. In this way, only one general PLM alongside the modified parameters for each task is saved or transferred. Except for saving
4242
memory and training cost, PEFT matches the performance of full fine-tuning with only updating less than 1% of the PLM parameters, quickly adapts to new tasks without catastrophic forgetting (Pfeiffer et al., 2021) and often exhibits robustness in out-ofdistribution evaluation (Li and Liang, 2021). These compelling advantages have sparked considerable interest in the adoption of PEFT.
PEFT methods can be split into two categories: sparse and infused fine-tuning. Sparse fine-tuning tunes a small subset of existing parameters without introducing new parameters. One typical example is BitFit (Zaken et al., 2022), where only the biases are updated. Nonetheless, BitFit is not scalable because of the fixed bias terms. Diff Pruning (Guo et al., 2021) and FISH Mask (Sung et al., 2021) alleviate this issue by learning and updating task-specific masked parameters with a specified sparsity ratio. However, different masks are learned under different tasks, making these two methods unsuitable for the federated learning setting, where data is rarely i.i.d. across servers.
Infused fine-tuning introduces new parameters to PLMs, and only updates these parameters during training. For example, adapter fine-tuning (Houlsby et al., 2019; Pfeiffer et al., 2021) inserts adapters to each layer of the PLM. Other methods, like Prefix Tuning (Li and Liang, 2021) and Prompt Tuning (Lester et al., 2021), append trainable vectors to input or hidden layers. However, inference latency is typically introduced by the newly added parameters and is nonnegligible for some complex tasks, like machine translation (MT) and summarization that add more than 4% of the PLM parameters (He et al., 2022).
In this paper, we address the above-mentioned challenges from sparse and infused fine-tuning by proposing two methods, PaFi and HiWi (illustrated in Figure 2). PaFi is a sparse fine-tuning method that selects trainable parameters in a task-agnostic way. I.e., we have the same mask for various downstream tasks. The mask generation of PaFi is also data-less. It doesn’t require any training on any data. HiWi is an infused fine-tuning method that applies the adapters directly to pre-trained weights or biases instead of to hidden representations. After training, the adapters are abandoned, therefore sharing the same inference speed as full fine-tuning.
Our main contributions in this paper are: (1) We introduce two novel transfer learning methods that solve the above-mentioned key challenges of sparse
and infused fine-tuning. (2) We empirically evaluate PaFi on the GLUE benchmark and show its effectiveness over existing sparse fine-tuning methods. (3) We compare our methods to a wide range of baselines on a newly constructed benchmark that contains tasks in different types and resources. HiWi outperforms all baselines and full fine-tuning, while requiring the minimum storage (see Figure 1). (4) Our proposed methods still show their effectiveness on a complex task, i.e. machine translation. And all PaFi and HiWi share the same inference speed as full fine-tuning.

heading: 2 Preliminaries
text: In this section, we give an overview of sparse finetuning and adapter fine-tuning, and highlight the key challenges of these PEFT methods.
Sparse Fine-Tuning. Sparse fine-tuning formulates a task-specific fine-tuning as a two-phase learning problem. In the first phase, one needs to determine which subset of the pre-trained parameters θ(0) can be modified by generating a sparse mask m ∈ {0, 1}∣θ(0)∣, where 1s in m denote the corresponding parameters are trainable. BitFit (Zaken et al., 2022) heuristically specifies the bias terms trainable. Diff Pruning (Guo et al., 2021) and LT-SFT (Ansell et al., 2022) fully fine-tune PLM on downstream task to obtain θ(1). And the k parameters with the greatest absolute difference∣θ(1) − θ(0)∣ are selected for updating in the next phase. FISH Mask (Sung et al., 2021) uses the gradient information of θ(1) to learn the mask.
After obtaining the mask, the PLM is finetuned and only the masked parameters are updated whereas the others are frozen. The learning procedure of the second phase is defined as
θ (2) = argmin θ L(D;m⊙ θ) (1)
with θ initialized by θ(0), where L and D are the objective and data of downstream task, respectively. In addition, (1−m)⊙θ(2) = (1−m)⊙θ(0), since we only update the masked parameters. In the end, only a common θ(0), the updated parameters m⊙ θ (2) and their indices are saved, which is storagefriendly with a large number of downstream tasks. Adapter Fine-Tuning. Adapter fine-tuning methods (Houlsby et al., 2019; Pfeiffer et al., 2021) insert one or multiple small MLP modules into each layer of the PLM. This MLP module consists
of a down (Wdown ∈ Rd×r) and up (Wup ∈ Rr×d) projection pair, where r is the bottleneck dimension, d is the dimension of hidden representation and r ≪ d. Most adapter fine-tuning methods can be fed into the formula of h ← h + f(hWdown)Wup (2) where h is the input to the adapter and f(⋅) is a nonlinear function.
The adapter fine-tuning methods in the formula of Equation 2 are module-wise, which means they consider the attention module or the feed-forward module as a unit and insert the adapter in between or after these units. In contrast, LoRA (Hu et al., 2022) inserts the adapter layer-wise as:
h ← hW + hWdownWup (3)
where W ∈ Rd×d is a pre-trained weight. LoRA has the same inference speed as full fine-tuning, since we can pre-compute W ← W+WdownWup and use the new W for inference. Both sparse finetuning and adapter fine-tuning can be fed into a unified framework.

heading: A Unified Framework for Sparse and Adapter
text: Fine-Tuning. Normally, we initialize Wdown and Wup (or at least one of them) close to 0 (Houlsby et al., 2019; Hu et al., 2022), so the initial state of h is close to the original state h of PLM, which makes the fine-tuning empirically perform better. This initialization is important for PEFT in case of the catastrophic forgetting of the pre-training knowledge. Supposed we defined the newly added
parameters (Wdowns and Wups) for adapter finetuning as δ, the initialization of δ (i.e. δ(0) ≈ 0) fulfills Lpre(⋅;ϕ(0)) ≈ Lpre(⋅;θ(0)), where Lpre is the pre-training objective2 and ϕ(0) = {θ(0), δ(0)}.
Straightforwardly, we can combine the second phase of sparse fine-tuning (Equation 1) and the adapter fine-tuning as:
ϕ̂ = argmin ϕ L(D; z ⊙ ϕ) (4) with ϕ initialized by ϕ(0), where z ∈ {0, 1}∣ϕ(0)∣. For sparse fine-tuning, the 1s in z only locates for the trainable parameters in θ(0), whereas all locations for δ(0) in z are 1s for adapter fine-tuning. In a word, the subset of the trainable parameters in ϕ is fixed for adapter fine-tuning, but task-specific for sparse fine-tuning.
Key Challenges. Sparse fine-tuning normally gains less attention than adapter fine-tuning. The reasons are two-fold: (1) In the first phase of sparse fine-tuning, the generation of the sparse mask is task-specific, which means different downstream tasks or the same task with different domain data might have different masks, whereas adapter finetuning always has fixed positions for adapters; (2) One needs some tricks to generate these masks, like a differential version of L0 norm (Guo et al., 2021) or Fisher Information (Sung et al., 2021), which requires more computation than a normal full fine-tuning for the same iterations.
2Lpre can’t be replaced with L that is the objective for downstream task, since most downstream tasks require a randomly initialized classifier, which makes L unpredictable.
Compared to sparse fine-tuning, adapter finetuning typically introduces additional inference latency from the newly added parameters. Though LoRA doesn’t have this issue, one can’t apply a nonlinear function in the adapter for LoRA since
hW + f(hWdown)Wup ≠h(W + f(Wdown)Wup) (5)
, which limits the learning capacity of this method.

heading: 3 Methodologies
text: Motivated by the key challenges stated in Section §2, we propose two methods in this section and illustrate them in Figure 2: one for sparse fine-tuning that generates a universal mask for various tasks without any training, and another one for adapter fine-tuning that has the same inference speed as full fine-tuning while requiring even less storage than BitFit (Zaken et al., 2022).

heading: 3.1 Task-Agnostic Mask Generation
text: Compared to adapter fine-tuning with fixed parameters to tune, existing sparse fine-tuning methods typically require extra training to determine which parameters are trainable. This procedure not only requires more computation than full fine-tuning but also hinders the application of this method to federated learning where data is rarely i.i.d. Based on this issue, we propose a research question: could we universally select a set of trainable parameters for different tasks?
To solve this question, we look into the benefit of sequential pre-training and fine-tuning. Aghajanyan et al. (2021) stated that PLM learns generic and distributed enough representations of language to facilitate downstream learning of highly compressed task representation. We hypothesize that the important (in some sense) parameters of a PLM learn a more generic representation of language and therefore favor downstream tasks more than the others. Therefore, we should fix these parameters and only update the unimportant ones. One might argue that the important parameters of a PLM could also be the ones important to downstream tasks and we should train them rather than fix them. We empirically justify our claim in Section §5.5.
Now the question goes to how to select the unimportant parameters so that we can fine-tune them on downstream tasks later. Here we offer two options: one with training on pre-training data and one in a data-less way. Inspired by FISH Mask (Sung et al.,
2021) where the unimportant parameters are the ones with less Fisher information, we can approximate the Fisher information matrix (Fisher, 1992; Amari, 1996) as
F̂θ = 1
N
N
∑ i=1 (▽θLpre(Di;θ))2 (6) where F̂θ ∈ R∣θ∣, Di is a sample from the pretraining corpus and Lpre is the pre-training objective. An intuitive explanation of Equation 6 is: The parameters with a larger value in F̂θ are more important since they cause larger gradient updates. Then the sparse mask comprises the parameters{θi∣F̂θi ≤ sort(F̂θ)k}, since we only update the unimportant parameters.
Another method is magnitude-based. We simply consider the pre-trained parameters with the smallest absolute magnitude as the unimportant ones, since they contribute the least to the pre-training loss. Then the sparse mask consists of the parameters {θi∣∣θi∣ ≤ sort(∣θ∣)k}.
In this paper, we only explore the magnitudebased method and leave another one for future work. The main reason is: The magnitude-based method doesn’t require any training on any data, whereas another method requires the calculation of gradients on pre-training data that are normally large-scale and private for some PLMs. We name our method as PaFi, since it follows a procedure of Pruning-and-Finetuning.

heading: 3.2 Adapter for Pre-trained Parameters
text: Compared to sparse fine-tuning that tunes a small portion of existing parameters, adapter fine-tuning introduces new parameters at some fixed positions for different tasks. Normally, the number of added parameters correlates with the complexity of the downstream task. For example, one can add 0.5% of the PLM parameters to achieve the same performance as full fine-tuning for the GLUE benchmark, while 4% is required for machine translation (He et al., 2022). The inference speed of a task is proportional to the number of added parameters, so the introduced inference latency is nonnegligible for complex tasks.
Inspired by LoRA (Hu et al., 2022) which has the same inference speed as full fine-tuning, we propose a new adapter fine-tuning method that applies an adapter directly to pre-trained parameters
instead of hidden representations as: W ← W + f(WWdown)Wup (7) If we neglect the nonlinear function, a representation h through our linear layer becomes hW (1 + WdownWup). Compared to LoRA (See Equation 3) where the learned diff matrix is △W = WdownWup, our learned diff matrix is △W = WWdownWup. Without considering the nonlinear function, the rank of the diff matrix from LoRA is the upper bound of the rank for our diff matrix:
rank(WWdownWup) ≤min(rank(W ), rank(WdownWup)) =rank(WdownWup) (8)
The equality holds since the rank of a pre-trained weight is empirically larger. To improve the learning capacity (it is related to the matrix rank) of our method, we input a nonlinear function between Wdown and Wup, which is not possible for LoRA if we want to maintain the same inference speed as full fine-tuning (see Equation 5).
One obvious advantage of our method is that it has the same inference speed as full finetuning, since we can compute W ← W + f(WWdown)Wup before the inference step. Another advantage is: we can replace the weight matrix in Equation 7 with the bias term. I.e. we input the pre-trained bias to an adapter to construct a new bias. In this way, we can solve the issue raised by BitFit (Zaken et al., 2022), where the number of bias terms is fixed and therefore BitFit is not scalable. When we apply the adapter to the weight matrix, we need to save Wdown and Wup, since their size are much smaller than W . However, when we apply the adapter to bias terms, we only need to save the new bias (b ← b+ f(bWdown)Wup) that requires much less storage than saving Wdown and Wup. It also means we require the same storage (for the bias terms) whatever the size of Wdown and Wup, and therefore we can use a large number of trainable parameters. We name our method as HiWi, since it Hides (throws away) the Weights from adapters.

heading: 4 Experimental Setup

heading: 4.1 Evaluation Tasks
text: Due to limited computation resources, we select six tasks from the GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) benchmarks:
two natural language inference tasks (MNLI and RTE), a similarity task (STS-B), a word sense disambiguation task (WiC), a coreference resolution task (WSC) and a causal reasoning task (COPA). For most tasks, we follow the RoBERTa paper (Liu et al., 2019), treating MNLI and RTE as sentencelevel classification tasks, WiC as a word-level classification task, STS-B as a regression task and WSC as a ranking task. Nevertheless, we implement COPA as a ranking classification task rather than a binary classification task in Liu et al. (2019), since it offers better performance for all methods.
We term our selected tasks VariousGLUE, since they cover a wide range of tasks (classification, ranking, regression) and include high-resource (MNLI), middle-resource (STS-B, WiC and RTE) and low-resource (WSC, COPA) tasks. For evaluation on VariousGLUE, we report accuracy for MNLI, WiC, RTE, WSC and COPA, and the Pearson correlation coefficient for STS-B on the development sets. More data statistics, implementation details and task selection criteria of VariousGLUE are in Appendix §A. Except for natural language understanding (NLU) tasks, we also evaluate our methods on a sequence-to-sequence task, i.e. English to Romanian translation with the WMT 2016 En-Ro dataset (Bojar et al., 2016), and report BLEU (Papineni et al., 2002) on the test set.

heading: 4.2 Baselines
text: To compare with other baselines broadly, we replicate their setups since most of them are not evaluated on the same tasks or use the same PLM (RoBERTaLARGE) as ours. If possible, we also report their scores.
Full fine-tuning (Full FT) updates all parameters. Linear fine-tuning (Linear FT) only tunes the added classifier. Linear fine-tuning with normalization (Linear FTnorm) fine-tunes the classifier and all normalization layers of the PLM. We borrow the fine-tuning recipe from Liu et al. (2019) for these three baselines.
Both Diff Pruning (Guo et al., 2021) and FISH Mask (Sung et al., 2021) are chosen as sparse finetuning baselines. We implement them on their own frameworks3 with their own recipes (combined with our recipe for middle-/low-resource tasks).
We select three adapter variants as our baselines: Adapter (Houlsby et al., 2019), Pfeiffer Adapter
3 https://github.com/dguo98/DiffPruning; https:
//github.com/varunnair18/FISH
(Pfeiffer et al., 2021) and LoRA (Hu et al., 2022). In addition, we also compare our methods to BitFit (Zaken et al., 2022), Prefix Tuning (Li and Liang, 2021) and MAM Adapter (He et al., 2022). MAM Adapter combines prefix tuning and adapter, offering a new state-of-the-art.
If not specified otherwise, we reproduce these baselines on RoBERTaLARGE with our own training recipe (see Section §4.3), if they don’t offer results on our selected tasks or use different PLMs. You can find more details about the calculation of trainable parameters and storage requirements of these methods in Appendix §B.

heading: 4.3 Implementation
text: We use the encoder-only RoBERTaLARGE model (Liu et al., 2019) as the underlying model for all NLU tasks and the encoder-decoder mBARTLARGE model (Liu et al., 2020) for MT. All our implementations are on the Fairseq framework (Ott et al., 2019). For NLU tasks, we sweep learning rates in {3, 4, 5, 6, 7} ⋅10−4 (inspired by the best results obtained in LoRA (Hu et al., 2022)), batch sizes in {16, 32}, and the number of epochs in {10, 20} (for tasks with the number of samples over 100K, we only train for 10 epochs). Other settings of the optimizer stay the same as the RoBERTa paper. For the En-Ro task, we borrow the same training recipe from He et al. (2022), i.e. setting the learning rate as 5 ⋅10−5, a mini-batch with 16384 tokens, a label smoothing factor of 0.1 (Szegedy et al., 2016; Gao et al., 2020) for 50K iterations.
We run all experiments on a single NVIDIA RTX A6000 GPU with 48G memory. In addition, we run the same task of a method in the above-mentioned grid search space three times with different random seeds, choose the best result from each run, and report the median and standard deviation of these
three best results. PaFi and HiWi. By default, we select the bottom-k parameters for PaFi group-wise rather than globally. I.e. we select k parameters with the smallest absolute magnitude within each group (a weight matrix or a bias term is considered as a group) and only fine-tune them. In addition, we fine-tune all parameters from normalization layers and don’t update the token and position embeddings at all. More discussion about this setting is in Section §5.5. For HiWi, the default setting is feeding the bias rather than the weight to an adapter, because it requires much less storage.

heading: 5 Result and Discussion
text: In this section, we present the results of baselines and our proposed methods on GLUE, VariousGLUE and translation tasks.

heading: 5.1 Sparse Fine-Tuning on GLUE
text: Since the frameworks of Diff Pruning and FISH Mask only support the GLUE benchmark, we evaluate our proposed sparse fine-tuning method, PaFi, on GLUE and show the results in Table 1, where we follow the sparsity setting of Diff Pruning and FISH Mask, and set it as 0.5%. In the RoBERTa paper (Liu et al., 2019), some low-resource tasks (RTE, MRPC and STS-B) are initialized from the fine-tuned model on MNLI rather than the PLM. We don’t follow this setup and always consider the PLM as an initialization as Houlsby et al. (2019). Surprisingly, our reproduction outperforms the reported score in the RoBERTa paper by 0.4%.
Compared to existing sparse and adapter finetuning methods, our PaFi obtains the best result, achieving the same score (89.3) as Full FT with only updating 0.5% parameters. PaFi also outperforms Diff Pruning and FISH Mask by a significant
margin, with at least a 1.3% improvement on average, which demonstrates the effectiveness of our simple parameter selection method.
In addition, PaFi also shows its efficiency in terms of storage and training compared to other sparse fine-tuning methods. The saving of the indices for updated parameters consumes the same memory as the saving of these parameters if both are in fp32. One only needs to save one mask for PaFi, but the same number of masks as the number of tasks for Diff Pruning and FISH Mask. Our one-mask-for-all setting is also suitable for federated learning, where non-i.i.d. data could use the same mask, which is impossible for Diff Pruning and FISH Mask. For training costs, PaFi doesn’t require any training in the mask generation phase, while Diff Pruning and FISH Mask require more computation (> 2 times for Diff Pruning) than Full FT because of the calculation of differential L0 norm or Fisher information matrix. Specifically on
the MNLI task, Diff Pruning, FISH Mask and PaFi spend 19.9h, 1m39s and 2s to generate the sparse mask, respectively. And both Diff Pruning and FISH Mask require the data of downstream task for the mask generation, while the mask generation of PaFi is data-less.

heading: 5.2 Results on VariousGLUE
text: Table 2 shows the results of different methods on VariousGLUE. For most methods, the number of trainable parameters stays the same as the number of stored parameters. However, Prefix Tuning and MAM Adapter apply a re-parameterization trick and throw away some trainable parameters after training, which might result in different numbers (see Appendix §B). In addition, the storage requirement of HiWi is invariant to the number of trainable parameters since we throw away all adapter parameters and only save the new bias terms.
Compared to Table 1, PaFi performs unexpectedly worse, which justifies the necessity of evaluation on various tasks for PEFT methods. On high-resource (MNLI) and most middle-resource (STS-B and RTE) tasks, PaFi is on par with Full FT. Though not the best, PaFi still outperforms some adapter fine-tuning methods (Adapter and Pfeiffer Adapter), which shows tuning existing parameters is enough for some tasks.
When using the same number of trainable parameters (0.5%), HiWi performs on par with the best baseline, MAM Adapter (87.3 vs 87.5). Notably, MAM Adapter is an additive work that combines Prefix Tuning and adapters together. We could also implement HiWi and Prefix Tuning in the same framework and leave this exploration to future work. If we increase the number of trainable
parameters to 2% (almost requiring the same training time and memory footprint as 0.5%), HiWi outperforms all baselines and Full FT (88.2 vs 88.0). Notably, HiWi only requires a fixed storage, around 0.03% of the PLM total parameters, which is also the least storage requirement among all methods.

heading: 5.3 Results on MT
text: Compared to NLU tasks, where the number of trainable parameters is negligible, MT task requires more trainable parameters to achieve a similar result as Full FT. We show the results of different methods on WMT 2016 En-Ro in Table 3. PaFi performs the best and outperforms Full FT (37.7 vs 37.3 with 4.5% parameters). Similar to the conclusion that we draw from Table 2, PaFi is good at high- and middle-resource tasks.
HiWi for bias works unexpectedly worse, while HiWi for weight is on par with LoRA. We argue: Though we can improve the learning capacity of bias terms with adapters, the limited amount of biases still hinders HiWi’s representative ability. HiWi for weight solves this issue by feeding a much larger amount of weights to adapters. In addition, most baselines, except for BitFit and LoRA, introduce nonnegligible inference latency (proportional to the stored parameters), while PaFi and HiWi share the same inference speed as Full FT. Specifically, the inference time on En-Ro is 110s for Full FT, LoRA, HiWi and PaFi, while it’s 125s for Pfeiffer Adapter (13.6% more inference latency).

heading: 5.4 Scalability
text: Not all PEFT methods benefit monotonically from having more trainable parameters. Li and Liang (2021) and Hu et al. (2022) have shown that Prefix Tuning can’t be scaled up well. Here we investigate the scalability of our methods and show the results in Figure 3. On average, all listed methods could be scaled well, and HiWi always outperforms other
methods. However, these methods show different scaling behaviors for tasks in different resources.
For the high-resource task, HiWi performs the worst and is stable with an increasing number of trainable parameters, while PaFi shows a wellbehaved pattern. This also explains the best performance of PaFi on En-Ro. I.e. PaFi could obtain stronger performance for the setting of a highresource task and a high number of trainable parameters. For middle-resource tasks, PaFi outperforms other methods and still has a good scaling behavior. For low-resource tasks, HiWi performs the best. The best overall performance of HiWi mainly benefits from these low-resource tasks, which shows HiWi is an effective option for the low-resource or few-shot learning setting.
In summary, PaFi shows its superiority when the task is high-resource and the number of tunable parameters is not too small, while the superiority of HiWi locates in the low-resource (for tasks and the number of tunable parameters) setting.

heading: 5.5 The Default Setting of PaFi
text: Table 4 shows our investigation of the PaFi’s settings. We implement four ablation experiments on how to choose the trainable parameters. Overall, updating the parameters with the smallest absolute magnitudes offers the best results. The gap between our default setting and other options is the largest for RTE. In addition, tuning the normalization layers is necessary for all tasks.
The results of tuning embeddings are similar to the results of without tuning embeddings, but a little lower. According to Figure 4 (in Appendix), the embedding layer always has a higher mean, twice as the mean from other layers. This is also one reason why we don’t tune parameters from the embedding layer, since most of them have a higher absolute magnitude, showing that they are important. Another reason is that the embedding layer occupies a large number of parameters. We can spare the budget for trainable parameters from this layer and share it with other layers.

heading: 6 Related Works
text: Parameter-Efficient Fine-Tuning for PLMs. Sparse fine-tuning (Zaken et al., 2022; Guo et al., 2021; Sung et al., 2021) offers a tool to explore the over-parameterization of PLMs and doesn’t introduce additional latency. Infused fine-tuning (Houlsby et al., 2019; Pfeiffer et al., 2021; Hu
et al., 2022; He et al., 2022) is highly modularized and has a lower task-switching overhead during inference. Our PaFi simplifies existing sparse finetuning methods and makes it more practical for the communication-frequent setting (federated learning). Our HiWi lowers the requirement for storage to a scale and doesn’t induce any inference latency.
Other efficient parameterization methods, like COMPACTER (Mahabadi et al., 2021), efficiently parametrize the adapter layers and reduce the number of trainable parameters. They are orthogonal to our work and can be combined with our HiWi. A recently proposed activation method, (IA)3 (Liu et al., 2022), achieves a new state-of-the-art on few-short learning and has a similar storage requirement as HiWi. However, it can’t be scaled up and performs worse than adapter fine-tuning methods when the number of samples is not too small4.
Pruning. Network pruning is a technique for sparsifying neural networks while sacrificing minimal performance (Theis et al., 2018; Liu et al., 2021). Though we borrow some methods from it, like parameter selection with magnitude or Fisher information, we don’t prune any parameter and maintain a similar performance as full fine-tuning.

heading: 7 Conclusion
text: In this work, we first propose PaFi as a novel but simple method for computing the sparse mask without any training and in a task-agnostic way. It selects the parameters with the lowest absolute magnitude from PLMs and tunes them, with keeping others frozen. We demonstrate the effectiveness of PaFi on the GLUE benchmark and translation task. Secondly, we propose HiWi as an adapter finetuning method that feeds pre-trained parameters instead of hidden representations to the adapters. It
4 https://adapterhub.ml/blog/2022/09/
updates-in-adapter-transformers-v3-1/
doesn’t introduce any inference latency. Furthermore, it requires the lowest storage while outperforming other strong baselines. In the future, we will try to compute the sparse mask with Fisher information and estimate our methods in a realistic few-shot learning setting.

heading: Acknowledgements
text: This research was partly supported by the Netherlands Organization for Scientific Research (NWO) under project number VI.C.192.080. We also gratefully acknowledge the valuable feedback from ACL2023 reviewers.
Limitations
We acknowledge the main limitation of this work is that we only evaluate our methods on some tasks from the GLUE and SuperGLUE benchmarks due to limited computation resources. And all tasks are not in a realistic few-shot setting, where the number of training samples is less than a few hundred and development sets are not offered. The benefit of PEFT methods could come from an exhaustive search of hyper-parameters for the development sets, while the realistic few-shot setting could solve this issue and shed more light on PEFT. It would be interesting to see how our methods and other baselines perform on a wide range of few-shot tasks.
In addition, current frameworks are not friendly for sparse fine-tuning methods. Most works (Diff Pruning, FISH Mask and our PaFi) still need to calculate a full gradient of all parameters and selectively update the masked parameters, which makes it cost the same training time as full fine-tuning.
Last but not least, we only estimate our methods on one single complex task, i.e. WMT 2016 En-Ro. One might not draw the same conclusion as ours on other complex tasks, like machine translation for
different languages and resources, summarization tasks, and so on.
Ethics Statement
The PLMs involved in this paper are RoBERTaLARGE and mBARTLARGE. RoBERTaLARGE was pre-trained on BookCorpus (Zhu et al., 2015), CC-News (Nagel, 2016), OpenWebText (Gokaslan and Cohen, 2019) and Stories (Trinh and Le, 2018). mBARTLARGE was pre-trained on CC25 extracted from Common Crawl (Wenzek et al., 2020; Conneau et al., 2020). All our models may inherit biases from these corpora.

heading: A Experimental Detail
text: A.1 Data Statistics We show the statistics of VariousGLUE and En-Ro in Table 5. We test our methods on four types of tasks that are high-resource (MNLI and En-Ro), middle-resource (STS-B, WiC and RTE) or lowresource (WSC and COPA).
A.2 Implementation Details of VariousGLUE Due to limited computation resources, we could not evaluate our methods and baselines on all tasks from the GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) benchmarks. We select six tasks from these two benchmarks. Except for COPA, we follow the same implementation as RoBERTa (Liu et al., 2019). We list the details as follows:
• MNLI (Williams et al., 2018) and RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009): The input format is “[CLS] sentence1 [SEP] [SEP] sentence2 [SEP]”. We input the representation of the [CLS] token from the encoder to a multi-class classifier for prediction.
• WiC (Pilehvar and Camacho-Collados, 2019): WiC has the same input format as MNLI and RTE. We feed the concatenation of the representation of the two marked words and the [CLS] token to a binary classifier.
• STS-B (Cer et al., 2017): STS-B also has the same input format as MNLI and RTE. We
feed the representation of the [CLS] token to a regression layer (similar to the classification layer with only one class).
• WSC (Levesque et al., 2012): We first detect all noun phrases from the sentence. Supposed n noun phrases are detected, we replace the pronoun with these phrases to construct n new sentences and input them to RoBERTa in a batch way in the format of “[CLS] sentence [SEP]”. After the masked word prediction layer, we take the corresponding logits for these n noun phrases. Some noun phrases might be a span. We average the logits in this span to obtain a single logit for each noun phrase. For the sample that offers a correct match between the noun phrase and the pronoun, we assign 1 as a label to the logit of this noun phrase and 0 to the other and calculate the cross-entropy loss. Even though we have to throw away the annotated samples that have incorrect matches, this method offers the best result. During inference, the noun phrase with the biggest logit is the prediction.
• COPA (Roemmele et al., 2011) The input format for a single sample is “[CLS] Because sentence1, so sentence2 [SEP]” or “[CLS] Because sentence2, so sentence1 [SEP]”. We feed these two inputs from the same sample to RoBERTa in a batch way, then input the representation of both [CLS]s to a classifier with only one class, making sure the logit from the input with the correct causal effect is larger than another one by calculating the cross-entropy loss.
Except for WSC, we always insert a classifier layer on top of the RoBERTa encoder. For WSC, we use the original masked word prediction layer from the PLM and keep it frozen. You can find more implementation details in our codebase.
A.3 Criteria for Task Selection
Most PEFT methods are evaluated on the GLUE benchmark. We argue that the GLUE tasks might be too easy for PLMs, since all GLUE tasks, except for STS-B, are classification tasks and some PLMs outperform our humans by a large margin according to the GLUE leaderboard5. In addition, the number of training samples for each GLUE task
5 https://gluebenchmark.com/leaderboard
is big (RTE is the task with the minimum number of samples, 2.5K). Most works have to construct few-shot learning tasks from GLUE by themselves. Compared to GLUE, the SuperGLUE benchmark offers more low-resource tasks and is much more difficult for PLMs. Due to computational resource limits, we can’t evaluate on all GLUE and SuperGLUE tasks, and therefore want to select some tasks from these two benchmarks.
Our selection criteria are three-dimensional: complexity, variety in task type and variety in task resource. Deciding whether a task is easy or complex could be subjective. Since our main research topic in this paper is PEFT, we determine the complexity of a task with its performance improvement from Linear FTnorm to Full FT (see Section §4.2 for these baselines). Linear FTnorm requires the minimum trainable parameters among all PEFT methods. If a task obtains a huge improvement from Linear FTnorm to Full FT, it means this task needs many parameters to tune, and therefore it is difficult for PEFT methods and a complex task.
Figure 5 shows the performance of Full FT, Linear FT and Linear FTnorm on the tasks from GLUE and SuperGLUE. Overall, Linear FTnorm outperforms Linear FT by a large margin, 75.8 vs. 57.8 on average (see Table 6). It means that only tuning the normalization layers is an efficient method6. With an increasing number of samples, the gap between Full FT and Linear FTnorm normally be-
6According to our observation, the tuning of the normalization layer becomes less important for recent infused finetuning methods. Pfeiffer Adapter and MAM Adapter obtain similar results w/o tuning normalization layers.
comes narrower, which shows the fine-tuning of high-resource tasks requires less trainable parameters than low-resource.
From the perspective of task variety, we want to make sure all task types appear in our VariousGLUE. COPA, WSC, STS-B and WiC are chosen because of their uniqueness. They are sentencelevel ranking, word-level ranking, regression and word-level classification tasks, respectively. The rest are all sentence-level classification tasks. From the perspective of complexity and variety in resources, we choose MNLI and RTE, since MNLI has the biggest number of samples and RTE is the most complex and a low-resource task (see right subplot of Figure 5).

heading: B Number of Trainable Parameters and Storage
text: We summarize the calculation for the number of trainable parameters and storage requirements in Table 7. We only show the calculation of the encoder-only model here. Notably, the newly added classifier for some tasks is excluded from the calculation, since all methods have this same setting. We also show the hyper-parameter values for all baselines used in this paper in Table 8.
Full FT: The number of parameters for the token and position embeddings is V d + 2d + nd, where V is the vocabulary size, d is the hidden dimension and n is the maximum sequence length. 2d here means the number of parameters from the normalization layer since RoBERTa applies a layer normalization after the embedding. For each encoder layer, RoBERTa has four projection layers for key, value, query and output (4d2 + 4d, 4d is the num-
ber of parameters for bias), and two feed-forward layers. The first feed-forward layer projects the representation from d to 4d. And the second one projects it back to d. So the number of parameters for these two feed-forward layers is 8d2 + 5d (5d = 4d + d for the bias terms). Each layer also has two normalization layers, including 2 × 2d parameters. Overall, the size of trainable parameters for Full FT is (V +2+n)d+(12d2+13d)L, where L is the number of encoder layers.
Linear FTnorm: Since we only tune the parameters from the normalization layers, so the number of trainable parameters is 2d + 4dL.
BitFit: We tune all bias terms from the normalization layers and the linear layers, so the size is d + 11dL.
Adapter: We insert two adapter layers to each encoder layer and also tune the normalization layers in the encoder layer, so the size of trainable parameters is (2(dr + r + rd + d) + 4d)L =(4dr + 2r + 6d)L, where r is the bottleneck dimension of the adapter.
Pfeiffer Adapter: Pfeiffer Adapter inserts a single adapter to each encoder layer and doesn’t tune the normalization layers, so the size is (dr + r + rd + d)L = (2dr + r + d)L.
LoRA: LoRA applies two adapters in parallel to the projection layers for query and value, respectively. Then the number of trainable parameters is 2(dr + rd)L = 4drL (no bias terms for the adapter).
Prefix Tuning: Prefix Tuning applies a reparameterization trick to expand the number of trainable parameters. Firstly, it defines an embedding in the size of l × d, where l is the length of
the prefix vectors. Then this embedding is fed into a large adapter from d to 2dL. After the adapter, the embedding is in the size of l × 2dL. We then reshape this matrix to L × 2 × ld, with one prefix vector in the size of ld for the key and another one for the value for each layer. So the number of trainable parameters is ld+ dm+m+ 2mdL+ 2dL = ld+ dm+m+ (2md+ 2d)L, where m is the bottleneck dimension for the adapter. However, it is not necessary for us to save all these parameters. We can compute the prefix vectors after training and throw away the embedding and adapter, so the size of stored parameters is 2ldL.
MAM Adapter: MAM adapter applies Prefix Tuning to the attention module and an adapter in parallel to the MLP module. The number of trainable parameters is the same as the sum of the one for Pfeiffer Adapter and the one for Prefix Tuning, which is ld+ dm+m+ (2dr + r + 3d+ 2md)L. Similar to Prefix Tuning, we can throw away the embedding and adapter for Prefix Tuning after training. So the stored size is (2dr + r + d + 2ld)L.
HiWi for Bias: HiWi applies one adapter to the bias term (in the size of 4d) of the first feed-forward layer and another adapter to the bias term (in the size of d) of the second feed-forward layer. To avoid allocating too many trainable parameters to the first adapter, we set the bottleneck dimension for the first adapter as 2r, and the one for the second adapter as r. So the size of trainable parameters is((8dr + 2r + 8dr + 4d)+ (dr + r + dr + d))L =(18dr+3r+5d)L. After training, we compute the new bias and throw away all adapter parameters. So the stored size is 5dL.
HiWi for Weight HiWi applies one adapter to the weight (in the size of 4d × d) of the first feedforward layer and another adapter to the weight (in the size of d × 4d) of the second feed-forward layer. We only need to swap the order of the above adapters for the bias terms. So the number of trainable parameters stays the same as above, i.e.(18dr + 3r + 5d)L. However, we don’t compute the new weight and throw away the adapters for this case, since the size of the weight matrix is larger than the adapter size. So the stored size is still(18dr + 3r + 5d)L.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4 and 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will try to compute the sparse mask with Fisher information and estimate our methods in a realistic few-shot learning setting