FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models
Year: 2023

Abstract:
With increasing concerns about data privacy, there is an increasing necessity of fine-tuning pre-trained language models (PLMs) for adapting to downstream tasks located in end-user devices or local clients without transmitting data to the central server. This urgent necessity therefore calls the research of investigating federated learning (FL) for PLMs. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we investigate the parameter-efficient tuning (PETuning) of PLMs and develop a corresponding federated benchmark for four representative PETuning methods, dubbed FedPETuning. Specifically, FedPETuning provides the first holistic empirical study of representative PLMs tuning methods in FL, covering privacy attacks, performance comparisons, and resource-constrained analysis. Intensive experimental results have indicated that FedPETuning can efficiently defend against privacy attacks and maintains acceptable performance with reducing heavy resource consumption. The open-source code and data are available at https://github. com/SMILELab-FL/FedPETuning.

heading: 1 Introduction
text: Pre-trained Language Models (PLMs), such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019a), have demonstrated exceptional performance on a multitude of natural language processing (NLP) benchmarks (e.g., GLUE (Radford et al., 2019)). Consequently, PLMs have become de facto backbones for real-world applications. Generally, in most real-world NLP applications, PLMs are centrally fine-tuned on the enormous quantity
*Equal contribution. †Corresponding authors.
of data collected from individual customers, small businesses, or large enterprises (Qu et al., 2021). However, with the rising privacy concerns and the enactment of data protection laws1, enterprises or institutions are not allowed to collect data from end devices or local clients to a centralized server for fine-tuning PLMs.
To break this barrier, federated learning (Konečnỳ et al., 2016; McMahan et al., 2017) (FL) has emerged as a privacy-aware technique designed to collaboratively train models without transmitting the numerous end-user or client data to a centralized place. In FL, the decentralized clients only need to periodically compute and send model information (i.e., parameters or gradients) to a server which is responsible for aggregating them to produce a global model. With the notion of privacy preserving, FL is appealing for privacy-sensitive NLP applications (Sui et al., 2020; Basu et al., 2021), as in the case of healthcare (Ge et al., 2020), finance (Long et al., 2020), and mobile keyboard (Ji et al., 2019).
1Such as the EU’s GDPR or the US’s HIPAA.
9963
Although fine-tuning PLMs in FL, namely FedFT, presents promising opportunities, there are two significant challenges that cannot be overlooked, including (1) communication overhead in the FL system, and (2) computational and storage costs for local clients. Fine-tuning such PLMs usually requires distributed clients and services highfrequently exchange model gradients or parameters which are usually in the scale of millions or even billions. The limited communication bandwidth2 in the FL system may cause excessive delays during frequent uploading and downloading phases of the federated fine-tuning procedure. Meanwhile, it is often impractical for local clients to fine-tune the entire PLMs because of their limited computing resources. Moreover, fully fine-tuning PLMs is extremely memory-intensive when a local client wants to store different instances for different tasks (Hu et al., 2022). As such, it is imperative to explore suitable PLMs-empowered FL methods under resource constraints (i.e., communication, parameters adaption, and storage).
To this end, we investigate parameter-efficient tuning (PETuning) methods of PLMs under the FL setting. PETuning methods, such as adapter tuning (Houlsby et al., 2019), prefix tuning (Li and Liang, 2021), LoRA (Hu et al., 2022), BitFit (Zaken et al., 2021), freeze most parameters of PLMs and update only a few additional parameters or a part of the original model parameters for downstream tasks (Ding et al., 2022). These properties make PETuning methods potentially appealing to satisfy the resource constraints since local clients just need to tune lightweight parameters and communicate with the server for updates. Nevertheless, it is crucial to address pertinent concerns. First, as the performance of federated models is greatly affected by the data heterogeneity (Kairouz et al., 2021) among clients, it is not yet known that PETuning methods can achieve acceptable performance in FL with such challenging data heterogeneity. Second, as private data could be recovered from model gradients uploaded by clients via gradient inversion attacks (Zhu et al., 2019), it is unclear whether PETuning methods that upload only partial parameters of the entire model can resist gradient inversion attacks.
To address these concerns, we present the framework of Federated Parameter-Efficient Tuning
2For example, the communication bandwidth between clients and server is constrained from a hundred Kbps to a few Mbps in most situations (Sui et al., 2020).
(named FedPETuning for short), as illustrated in Figure 1, and conduct in-depth experiments of various PETuning methods under the FL setting, measuring privacy-preserving capability, performance, and resource costs. Intensive experimental results on the GLUE benchmark reveal that (1) FedPETuning can reduce the considerable resource costs in FL settings while still achieving acceptable performance (e.g., FedAP reduces 97.4% of communication with only 0.7% performance degradation compared with FedFT) as shown in Table 2, and (2) FedPETuning has an appealing ability to defend against gradient inversion attacks, which can reduce the prevision of recovered words by an average of 40.7% compared to FedFT, as shown in Section 4.2. In summary, the major contributions of this paper are shown as follows:
• FedPETuning is the first benchmark to provide a holistic review of PETuning methods for PLMs under FL settings, covering privacy attacks, performance comparisons, and resource-constrained analysis.
• FedPEtuning can serve as a suite of baselines for efficient-parameter tuning of PLMs in a federated setting, and guide the community to design FL-tailored efficient parameter tuning algorithms.
Our research findings demonstrate the potential of combining large PLMs with FL, providing a promising training paradigm for privacypreserving learning in the era of large language models (Ouyang et al., 2022; OpenAI, 2023).

heading: 2 Related Work
text: Federated Learning Federated learning (Konečnỳ et al., 2016; McMahan et al., 2017) (FL), a widespread distributed learning technique used in privacy-sensitive tasks, has been hindered by not Independently and Identically Distributed (non-IID) (Kairouz et al., 2021), which results in accuracy discrepancies compared to centralized training. Extensive optimization studies have been conducted to address the non-IID issue, including data optimization (Zhao et al., 2018), model updating optimization (Chai et al., 2020), and model training optimization (Sahu et al., 2018). Recently, some work has tried to solve this problem from a pre-trained model initialization perspective (Chen et al., 2022b; Nguyen et al., 2022) and transformer model structure (Qu et al.,
2022). Weller et al. (2022) experimentally show that using PLMs could reduce non-IID adverse effects and narrow down its accuracy gap to centralized learning. However, the significant communication overhead of large-scale PLMs are less considered in FL systems, leading to slow and impractical federated training in real-world tasks. Additionally, PLMs can pose challenges for local clients with limited hardware capabilities for computation and storage. In contrast, our study investigates PLMs’ training in the FL context under resource constraints.
Injecting parameters-efficient tuning methods into federated learning Parameter-efficient tuning (PETuning) seeks to keep most parameters of PLMs frozen and fine-tune only additional lightweight parameters or a fraction of the parameters for downstream tasks (Houlsby et al., 2019; Li and Liang, 2021; Hu et al., 2022; Zaken et al., 2021). With this trait, PETuning methods can be utilized to mitigate the communication overhead in FL, which primarily relies on the size of model update parameters. In the field of computer vision, Sun et al. (2022) present the FedPEFT framework by injecting three PETuning methods (i.e., Bais, Adapter, Prompt) of the visual pre-trained models into FL, and find that lightweight PETuning methods in FL can significantly reduce the communication burden while maintaining performance and performing better in different FL settings. Meanwhile, Chen et al. (2022c) extend PETuning methods to the visual language model in FL and show that PETuning can facilitate a fast convergence rate. However, these studies ignore the important privacy attack issue existing in FL. With increasing attention to privacy concerns, validation of the privacy-preserving capabilities of federated PETuning methods is paramount and facilitates their practical deployment in real-world scenarios.
In the context of NLP, Zhao et al. (2022) first explore the effect of prompt-tuning under the FL setting and achieve acceptable performance results compared with fine-tuning. Xu et al. (2022) show that it is possible to train large vocabulary language models while preserving accuracy and privacy by adopting the low-rank adaptation (Hu et al., 2022). However, there has been no comprehensive investigation into the FL performance of the PETuning method for PLMs. Our research aims to bridge this gap and provide access to our code and data to inspire further exploration of the potential of this
new paradigm for efficient federated NLP.

heading: 3 Federated Parameter-Efficient Tuning
text: In this section, we present how different PETuning methods work, followed by the training process of FedPETuning.

heading: 3.1 PETuning Methods
text: Denote the original PLM parameters by Wp = {w1, w2, ..., wN} and the updated parameters by W ′p = {w ′ 1, w ′ 2, ..., w ′ M} after training on the dataset D. DefineWe as trainable model parameters. In vanilla fine-tuning, |We| are equal to the number of original model parameters and N = M , where | · | refers to the number of parameters. In the PETuning methods, most parameters of the PLM keep frozen and only a few added parameters or a part of the original model parameters are updated, that is, M ≥ N and |We| ≪ N . Following the taxonomy of Ding et al. (2022), PETuning methods could be divided into three groups, i.e., additionbased methods, specification-based methods, and reparameterization-based methods.
Addition-based methods introduce new trainable parameters into the frozen PLMs. These methods have two representative branches: Adapter tuning and Prompt tuning. The adapter tuning proposed by Houlsby et al. (2019) inserts adapter layers to vanilla Transformers. Specifically, two adapter layers are inserted into each Transformer block, wherein each adapter layer contains a downprojection and an up-projection. Given the input feature h ∈ Rd, the down-projectionWd ∈ Rd×r projects the input hin to a r-dimensional space and the up-projectionWu ∈ Rr×d maps back to the input size. Mathematically, the computation process is as follows,
h←WTu f ( WTd h ) , (1)
where f(·) is the nonlinear activation function. In this strategy, adapter tuning could only fine-tune adapter layersWe = {Wu,Wd} (about 0.5% to 8% parameters of the whole model) during the tuning process while keeping parameters of PLMs frozen.
On the contrary, prompt-tuning (Li and Liang, 2021; Liu et al., 2021) adds extra parameters without modifying the model architecture. Prompttuning converts the training objective of downstream tasks into a form similar to the pre-trained stage (Devlin et al., 2019; Liu et al., 2019b), by attaching trainable vectors P , namely prompt, to
the original input. During model training, prompttuning only adapts light-weight prompt vectors We = {P}, which scale to within 10% of the number of PLMs parameters.
Algorithm 1: Training process of FedPETuning
Parameters: Client set C; Communication round T ; Local epoch number E ; The PLMs parameters Wp; The local trainable and efficient parametersWe and the local dataset Dk of the k-th client; Local PETuning Method P;
Before Training: Initialize W0e on the server and Wp on each client in C. ServerGlobalAggregation: for each communication round t = 1 to T do Ct ← (randomly sample K clients from C) for each user k ∈ Ct in parallel do
ClientLocalTuning(k,Wt−1e ) end Receive local updated parametersWk,te Perform global aggregation by Eq. (3)
end
ClientLocalTuning (k,Wte): Wt ← (assembleWte andWp) for epoch e = 1 to E do Wk,t+1e ← P(Dk,Wt)
end SendWk,t+1e to the server
Specification-based methods aim to fine-tune a fraction of the parameters while keeping others frozen. In particular, Zaken et al. (2021) propose BitFit and empirically demonstrate that only tuning the bias termsWe = {bℓ,(·)(·) } of PLMs could still achieve competitive performance.
Reparameterization-based methods argue that the PLMs’ adaptions can be re-parameterized into optimization within a low-dimensional subspace (Aghajanyan et al., 2020). Based on this hypothesis, LoRA (Hu et al., 2022) optimizes the low-rank decomposition for weight update matrices ∆W during model training. For a pretrained weight matrixW ∈ Rd×k, we have
W +∆W =W + BA, (2)
where B ∈ Rd×r, A ∈ Rr×k, and the rank r ≪ min(d, k). In this way, LoRA could make training more efficient with less than 1% trainable parametersWe = {B,A} and match the finetuning performance.

heading: 3.2 FedPETuning
text: Our work considers a conventional FL system that comprises a central server responsible for manag-
ing participated clients for local training and distributing the shared model parameters. Instead of communicating all cumbersome PLMs parameters, FedPETuning resorts to PETuning methods for exchanging lightweight parameters, as illustrated in Figure 1.
Before training, FedPETuning initializes the backbone PLM withWp and the PETuning method P with efficient-parameter We. Then global aggregation in the server and local updating in local clients are executed alternately.
Server Global Aggregation. The third-party service first randomly selects K clients from C and distributes trainable parameterWe to these chosen clients. Then the server performs the federated aggregation based on the received parametersWk,te from clients, and updates theWe by
Wt+1e = K∑
k=1
|Dk|∑K k=1 |Dk| Wk,te . (3)
Client Local Tuning. When selected clients download the trainable parameters, they assemble a whole model with lightweight parameters We and local PLM parameters Wp. Then, selected clients train the assembled model with local private data Dk. After local training, the k-th client sends its updated efficient parametersWke to the central server for federated aggregation.
The training process described above is repeated until a specific criterion (e.g., the maximum number of communication rounds T ) is satisfied. This process can be summarized in Algorithm 1.

heading: 4 Experiments
text: In this section, we conduct extensive experiments for evaluating the performance of PETuning in the FL setting, covering privacy attacks (see Section 4.2), performance comparisons (see Section 4.3), and resource-constrained analysis (see Section 4.4). Besides, we also provide an in-depth ablation analysis of FedPETuning in terms of data heterogeneity (see Section 4.5), local training epochs (see Section 4.6), and different FL scenarios (see Section 4.7).

heading: 4.1 Experiments Setup
text: Dataset and non-IID partitioning Our experiments use six datasets from the GLUE benchmark. The reasons are as follows: (1) the GLUE datasets have emerged as the de facto standard benchmark for assessing PLMs’ effectiveness in natural language understanding; (2) They have been exten-
sively leveraged to validate various PETuning methods (Li and Liang, 2021; Liu et al., 2021; Zaken et al., 2021; Houlsby et al., 2019), and (3) these datasets are large enough and convenient for FL data partitioning (non-IID). Due to the limitation of the unpublished test set in GLUE, we follow the previous studies (Liu et al., 2022; Zhao et al., 2022) and use the original validation set as the new test set and split a part of the training set as the validation set. The data breakdown for this benchmark is in Table 1.
For the non-IID data partitioning, we follow Lin et al. (2021) and partition the datasets by using the Dirichlet distribution as the class priors. In particular, we sample D ∼ Dir(α) and allocate data Dk to k-th client. α determines the degree of non-IID, and a lower value of α generates a high label distribution shift. Unless otherwise specified, we maintain a default setting of α = 1.0 for the Dirichlet parameter throughout our experiments.
Implementation Details We adopt the benchmark FL system FedAvg to simulate the FL setting, which has been applied to commercial products (Bonawitz et al., 2019). In FedAvg, clients upload local model parameters to the central server and download the aggregated model for the next training round. Following Lin et al. (2021), we set the communication round to 100 and the local training epoch to 1 for all tuning methods. Following Chen et al. (2022a), we utilize Roberta-Base (Liu et al., 2019a) as the local model released by Huggingface3. The FedAvg implementation is based on FedLab (Zeng et al., 2023).
Under the FedAvg training protocol, we primarily evaluate the full fine-tuning (FedFT) and four representative PETuning methods, covering adapter tuning (FedAP), prefix tuning (FedPF), LoRA (FedLR), and BitFit (FedBF). For FedAP, we follow the architecture in Houlsby et al. (2019), which interposes adapter modules into both the multi-head attention module and the feed-forward
3https://github.com/huggingface/transformers
network in each Transformer layer. We use prefix tuning (Lester et al., 2021) as the representative of prompt tuning because it has better performance on the small PLMs, e.g., base versions of Roberta. For LoRA and BitFit, we take the architectures from their origin papers (Zaken et al., 2021; Hu et al., 2022). All PETuing methods are based on OpenDelta4, which is a plug-and-play framework for parameter-efficient tuning.
To make a fair and reasonable comparison, we run a hyperparameter sweep for each dataset and tuning method. We select the best model according to the metric on the validation set and report test set metric. Especially, the learning rate is selected from {5e-5, 1e-4, 5e-4, 1e-3, 5e-3, 1e-2}. We search the reduction factor from {16, 64} for FedAP, the prompt length from {8, 16, 64} for FedPF, and the scaling factor and rank from {8, 16} for FedLR. All experiments are done on a server with 8 Nvidia Tesla V100 GPUs with 32GB RAM each.

heading: 4.2 Privacy-Preserving Results
text: We first investigate the privacy-preserving capabilities of FedPETuing and FedFT. In this privacy attack experiment, we adopt DLG (Zhu et al., 2019) as our base attack method. Due to space limitations, we have omitted the working process of DLG and recommend the reader read the original paper (Zhu et al., 2019).
Setup and Metrics As the goal for the attacker with DLG is to recover text from client-uploaded gradients, we follow Song and Raghunathan (2020) and evaluate FedPETuning and FedFT in terms of precision (the average percentage of recovered words in the target texts), recall (the average percentage of words in the target texts are predicted) and F1 score ( the harmonic mean between preci-
4https://github.com/thunlp/OpenDelta
sion and recall). Specifically, we randomly selected 128 samples from the MNLI as the attack dataset.
Results Figure 2 shows the results of DLG on the attack dataset with different tuning methods. The results show that FedPETuning can effectively defend against data reconstruction attacks compared to FedFT. Since FedPETuning communicates a fraction of the entire model parameters with the server during the federated training process, it is intractable for the attacker to reconstruct the original text from such lightweight model parameters. Surprisingly, among FedPETuning, DLG is more likely to reconstruct private data from FedAP. We speculate that this result may be related to the design of PETuning methods. Adapter tuning inserts adapter layers to vanilla Transformers, which can encode the input data separately during model training. Compared to other PETuning methods, Adapter tuning is easier to “remember” the model input, thus causing more severe privacy leaks.
There is a clear and consistent trend across all tuning methods: DLG performance decreases as the batch size increases. The reason is that the DLG attack requires more parameters to be optimized in larger batch sizes. Therefore, increasing the batch size is a good defense strategy against the DLG attack. However, clients (e.g., mobile phone) in the FL system are usually resource-constrained in realworld applications. There is a trade-off between limited computing resources and large batches. In contrast, the FedPETuning method does not vary significantly across different batch sizes. In this sense, FedPETuning can provide a practical defense strategy for clients with restricted computing
resources.

heading: 4.3 Performance Comparison
text: Table 2 shows the performance for four PETuning and FT under the federated (upper half table) and centralized settings (bottom half table). The results demonstrate that FedPETuning maintains acceptable performance (more than 95% of FedFT’s performance) while reducing the communication overhead substantially.
From the upper half of Table 2, we find that although FedPETuning methods lag behind FedFT in the federated setting, the performance gap is relatively acceptable. For instance, FedAP reduces 60 times the communication overhead by only 0.7% performance degradation compared with FedFT. This result also shows that FedPETuning methods, especially FedAP and FedLR (more than 95% of FedFT’s performance), can achieve a good level of trade-off between performance and communication overhead in practice.
In the centralized setting, some PETuning methods achieve competitive performance compared with CenFT. Specifically, CenAP outperforms CenFT on five out of six datasets, and the CenAP and CenLR achieve better performances on average. This result also supports our motivation that PETuning, stimulating colossal models with only a small portion of tunable parameters, can naturally be a promising way for FL under communication and resource constraints.
Comparing the upper and bottom parts of Table 2, we find that all tuning methods endure a decline in performance under the federated setting. Remarkably, the FedPETuning exhibits a significant
102 103 104 105 Communication Budget / MB
0.5
0.6
0.7
0.8
0.9
Ac cu
ra cy
(% )
QNLI
FedLR FedPF FedAP FedBF FedFT
102 103 104 105 Communication Budget / MB
0.60
0.65
0.70
0.75
0.80
0.85
0.90
Ac cu
ra cy
(% )
QQP
FedLR FedPF FedAP FedBF FedFT
0 25 50 75 100 125 150 175
RTE
MNLI
190
190
12.1
12.1
140.1
140.4
103.8
41.9
1
1
FedBF FedPF FedLR FedAP FedFT
drop in performance in the data heterogeneous FL context, which aligns with the results observed in data heterogeneity experiments (see Section 4.5). This result suggests that the FL community may design FL-tailored PETuning methods to bridge the performance gap caused by the non-IID problem. This will be explored in our future work.

heading: 4.4 Resource Costs
text: We next show the resource cost by different tuning methods under FL settings, including communication budget and client storage overhead. Figure 3 shows accuracy versus communication budget for all tuning methods on RTE and MNLI5. As shown in Figure 3, the communication budgets are ranked as FedFT≫ FedPF > FedAP > FedLR > FedBF. The experimental results show that FedPETuning renders federated training remarkably efficient by significantly reducing the communication overhead as opposed to FedFT. More communication overhead entails increased training time during uploading and downloading. For FL that necessitates high-frequency communication,
5The plots of remaining tasks can be found in Appendix B, which have similar results.
FedFT FedAP FedLR FedPF FedBF 10
5
0
Re la
tiv e
Ac cu
ra cy
(% ) RTE
=0.1 =10.0 FedFT FedAP FedLR FedPF FedBF
4
2
0
MNLI
=0.1 =10.0
FedFT FedAP FedLR FedPF FedBF
40
20
0
Re la
tiv e
Ac cu
ra cy
(% ) QNLI
=0.1 =10.0 FedFT FedAP FedLR FedPF FedBF
20
10
0
QQP
=0.1 =10.0
Figure 5: Performance variation of FedPETuning and FedFT under different non-IID with respect to α = 1.0 on RTE and MNLI. FedPETuning is more susceptible to data heterogeneity than FedFT.
0.0 0.2 0.4 0.6 0.8 Distance
0
1
2
3
4
5
6
De ns
ity
RTE =0.1
0.0 0.2 0.4 0.6 0.8 Distance
0.0
0.5
1.0
1.5
2.0 RTE =1.0
0.0 0.1 0.2 0.3 0.4 0.5 Distance
0
2
4
6
8
10
RTE =10.0
0.0 0.2 0.4 0.6 0.8 Distance
0
2
4
6
8
De ns
ity
MNLI =0.1
0.0 0.2 0.4 0.6 0.8 Distance
0.00
0.25
0.50
0.75
1.00
1.25
1.50
MNLI =1.0
0.0 0.2 0.4 0.6 Distance
0
1
2
3
MNLI =10.0
Figure 6: Data distribution under different Dirichlet parameter α. We report the pairwise Jensen–Shannon distance of the label distribution between two clients.
FedFT is extremely time-consuming, making the federated training slow. In this regard, FedPETuning is more pragmatic in real-world applications, particularly in communication-constrained FL challenges.
Figure 4 shows normalized values of storage efficiencies of FedPETuning and FedFT on RTE and MNLI. FedPETuning lowers the storage cost to entry by 12~190 times. This appealing characteristic is practiced for local clients of real-world FL systems. When deploying multiple tasks on a local client, FedPETuning can share PLM between different tasks, enabling the client to maintain only a few parameters for each task, thus reducing storage requirements.

heading: 4.5 Impact of Data Heterogeneity
text: As data heterogeneity is a fundamental challenge in FL, we also evaluate the performance of FedPETuning and FedFT under different data heterogeneity. Following Lin et al. (2021), we consider three Dirichlet distributions in this experiment by choosing α from {0.1, 1.0, 10.0} where a smaller α indicates a sharper non-IID distribution among clients.
The performance of α = 1.0 has already been discussed, Figure 5 illustrates performance changes of FedPETuning and FedFT with respect to the
1 2 3 Epoch
60
65
70 75 Ac cu ra cy (%
) RTE
FedLR FedPF
FedAP FedBF
FedFT
1 2 3 Epoch
82
84
86
MNLI
FedLR FedPF
FedAP FedBF
FedFT
1 2 3 Epoch
80
85
90
Ac cu
ra cy
(% )
MRPC
FedLR FedPF
FedAP FedBF
FedFT
1 2 3 Epoch
86
88
90
QQP
FedLR FedPF
FedAP FedBF
FedFT
Figure 7: Performance comparison of FedPETuning and FedFT under different epochs on RTE and MNLI. Most tuning methods under the FL setting benefit from increased local training epochs.
other two α values. The outcomes corresponding to other tasks can be found in Appendix B. Figure 5 reveals that FedPETuning is more susceptible to data heterogeneity than FedFT. As shown in Figure 5, although the increase in data heterogeneity (α from 1.0 to 0.1) degrades the performance of all tuning methods, FedPETuning degrades more dramatically compared to FedFT. This suggests more the federated model necessitates more trainable parameters to tackle intricate data heterogeneity, and it may be arduous for FedPETuning to confront complex data heterogeneity in FL.
Another noteworthy phenomenon is the performance of FedPETuning and FedFT do not change much when increasing α from 1.0 to 10.0. To figure out this, we report the Probability Density Function (PDF) of the data distribution under different α in Figure 6 (See Appendix A for other datasets). As shown in Figure 6, the data heterogeneity is similar for α equal to 1.0 and 10.0, while the distribution gap is large for α equal to 0.1. We think similar data heterogeneity between α = 1.0 and α = 10.0 contributes to this result.

heading: 4.6 Impact of Local Training Epochs
text: Figure 7 shows the performance of FedPETuning and FedFT with different local training epochs on RTE and MNLI. More results are provided in Appendix B. This result reveals that most tuning methods under the FL setting benefit from increased local training epochs. Federated models trained with more local epochs consistently yield a performance gain across different tuning methods when clients with heigh-resource data (like MNLI). In contrast, training with more epochs may incur a drop in performance for the low-resource dataset (like FedLR and FedAP on RTE). This may be due to the probability of PLMs becoming more susceptible to overfitting when clients with few data undergo more training epochs. On the contrary,
clients with adequate data require more local training epochs to enhance the FL performance.
With the data scale into analysis, we also find that all tuning methods are more unstable on small datasets in Figure 7. For instance, the standard deviation performance of FedLR on RTE and MRPC is over 4.0% while other datasets are no more than 0.5%. This phenomenon is consistent with previous work (Chen et al., 2022a), and they experimentally show that increasing training steps can effectively mitigate the instability of the PETuning. However, this strategy fails in the FL setting. Training PLMs more stably in FL is a good research direction we will explore in our future work.

heading: 4.7 Various FL Scenarios
text: To verify the effectiveness of PETuning under different FL scenarios, we mainly consider two FL scenarios in this experiment, i.e., the cross-silo FL scenario (Kairouz et al., 2021) and the largescale cross-device FL scenario (Lai et al., 2022). Cross-silo (Kairouz et al., 2021) is a vital application scenario for FL, suitable for several users (no more than 100). In Cross-silo, the server selects all clients for training in each communication round. Large-scale cross-device FL (Lai et al., 2022) is another federated scenario for deployment across thousands of clients. In the large-scale cross-device FL, data held by the local client is more scarce. In
this experiment, we chose RTE, MRPC, and SST2 to simulate the cross-silo FL scenarios, while QNLI, QQP, and MNLI to simulate the large-scale cross-device FL scenarios. The total number of clients in the cross-silo and large-scale settings is set to 10 and 1000, respectively. For both FL scenarios, ten clients are involved in each communication round for local training and federated aggregation.
Table 3 and Table 4 show the performance results of PETuning and FT under these two FL scenarios, respectively. The results show that FedPETuning can significantly reduce the communication overhead while achieving acceptable performance in cross-silo and large-scale crossdevice FL scenarios. More noteworthy between the different FL scenes is the cross-silo setting. As shown in Table 3, the performance gap between the FedPETuning and FedFT diminishes under the cross-silo setting, compared to our standard setting in Section 4.3. For example, FedPF lags behind FedFT by 2.3% and 4.7% under the cross-silo FL scenario and the FL scenario of Table 2, respectively. We attribute this observation to relieving the data scarcity issue since each client has ten times more training samples in the cross-silo setting than in the standard setting.

heading: 5 Conclusion
text: This paper investigates parameter-efficient tuning methods of PLMs in the FL setting with extensive experiments for in-depth measurement of these methods under FL settings, covering privacy attacks, performance comparisons, and resourceconstrained analysis. Experimental results unveil that FedPETuning can (1) achieve acceptable performance while reducing the colossal communication overhead and local storage cost, and (2) provide strong privacy-preserving capacity under different FL settings. To facilitate FL-tailored PETuning research, we have released our code and partitioned datasets, aiming to facilitate the use of FedPETuning and inspire the broader community to develop more suitable PETuning methods in future federated learning research.
Limitation
One limitation of this paper is that we do not validate FedPETuning on large scale models, (e.g., T5 (Raffel et al., 2019), LLaMa (Touvron et al., 2023), Vicuna (Chiang et al., 2023), etc). Although
the parameter efficiency of PETuning is more impressive on larger pre-trained models, we have to consider the limited computational resources of clients in FL, making it challenging to deploy such a large-scale model. In addition, with the increasing size of modern pre-trained models, the community needs to design FL-friendly PETuning methods. In this sense, our work can serve as a benchmark and guide for future exploration of PETuning in FL.

heading: Acknowledgements
text: We’d like to thank all the anonymous reviewers for their careful readings and valuable comments. This work was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826), the Major Key Project of PCL (No. 2022ZD0115301), and an Open Research Project of Zhejiang Lab (NO.2022RC0AB04).

heading: A Non-IID Partitionings Results
text: Figure 8 illustrates the data distribution under different α. It is observed that data distributed in clients with α as 0.1 has a large distance with each other, which has impact on the performance. Both α as 1.0 and 10.0 are considered to produce a more uniform distribution.

heading: B Extra Results
text: Communication Analysis In this section, we illustrate the accuracy given the communication budget on the remaining four tasks in Figure 10. As can be seen, PETuning methods consistently reduce the communication budget over several orders of magnitude while providing comparable performance. Moreover, PETuning methods (apart from FedBF) achieve acceptable accuracy ( 90% of fine-tuning) on all the tasks, demonstrating the effectiveness of these methods.
Non-IID Analysis Figure 11 presents the comparisons of accuracy under different distributions by varying α. All the methods obtain better performance with α as 1.0 than α as 0.1, showing that non-IID distribution has a negative impact on the model’s performance. On the contrary, varying α from 1.0 to 10.0 has little impact in most circumstances.
Local Training Epoch Analysis In Figure 12, we show the accuracy of tuning methods trained with different numbers of training epochs on the local clients. The accuracy of the relatively small datasets (MRPC, SST-2) shows a faint decreasing trend because of the over-fitting issue. All the tuning methods benefit from more training epochs on relatively large datasets (QNLI, QQP).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation Section
7 A2. Did you discuss any potential risks of your work? Our work does not involve potential risks
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.1.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.1.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To facilitate FL-tailored PETuning research, we have released our code and partitioned datasets, aiming to facilitate the use of FedPETuning and inspire the broader community to develop more suitable PETuning methods in future federated learning research. Limitation
One limitation of this paper is that we do not validate FedPETuning on large scale models, (e. g. , T5 (Raffel et al. , 2019), LLaMa (Touvron et al. , 2023), Vicuna (Chiang et al. , 2023), etc). Although
the parameter efficiency of PETuning is more impressive on larger pre-trained models, we have to consider the limited computational resources of clients in FL, making it challenging to deploy such a large-scale model. In addition, with the increasing size of modern pre-trained models, the community needs to design FL-friendly PETuning methods. In this sense, our work can serve as a benchmark and guide for future exploration of PETuning in FL