Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization
Authors: Tianshi Che
Ji Liu
Yang Zhou
Jiaxiang Ren
jiwen zhou
Victor S. Sheng
Huaiyu Dai
Dejing Dou
Venue: EMNLP 2023 Main (2023)
Keywords: Federated learning
Large Language Models
Prompt tuning
Prefix tuning
TL;DR: This paper proposes a Parameter-efficient prompt Tuning approach with Adaptive Optimization, i.e., FedPepTAO, to enable efficient and effective federated learning of large language models.

Abstract: Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. However, the training process of Large Language Models (LLMs) generally incurs the update of significant parameters, which limits the applicability of FL techniques to tackle the LLMs in real scenarios. Prompt tuning can significantly reduce the number of parameters to update, but it either incurs performance degradation or low training efficiency. The straightforward utilization of prompt tuning in the FL often raises non-trivial communication costs and dramatically degrades performance. In addition, the decentralized data is generally non-Independent and Identically Distributed (non-IID), which brings client drift problems and thus poor performance. This paper proposes a Parameter-efficient prompt Tuning approach with Adaptive Optimization, i.e., FedPepTAO, to enable efficient and effective FL of LLMs. First, an efficient partial prompt tuning approach is proposed to improve performance and efficiency simultaneously. Second, a novel adaptive optimization method is developed to address the client drift problems on both the device and server sides to enhance performance further. Extensive experiments based on 10 datasets demonstrate the superb performance (up to 60.8% in terms of accuracy) and efficiency (up to 97.59% in terms of training time) of FedPepTAO compared with 9 baseline approaches. Our code is available at https://github.com/llm-eff/FedPepTAO.

Federated Learning of Large Language Models with Parameter-Effiient Prompt Tuning and Adaptive Optimization
Tianshi Che'; Ji Liu Yang Zhou',, Jiaxiang Ren', Jiwen Zhou', Victor S. Sheng", Huaiyu Dai², Dejing Dou
Auburn University, Auburn, United States,
Hithink RoyalFlush Information Network Co., Ltd., Hangzhou, Zhejiang, China, a "Baidu Inc. Beijing, China, "Texas Tech University, Lubbock, United States, North Carolina State University, United States, Boston Consulting Group, United States.
Abstract
(Official Journal of the European Union, 2016; Cal- ifornians for Consumer Privacy, 2020) have been implemented, aggregating the decentralized data into central server or data center becomes com- plicated or even impossible (Yang et al., 2019), FL generally exploits parameter server module (Liu et al., 2023b) to manage the distributed model updates in devices, which only exchanges the pa- rameters of the updated models instead of the raw data, between the parameter server and devices.
Federated learning (FL. promising paradigm to enable collaborative model train- ing with decentralized data. However, training process of Large Language Models (LLMs) generally incurs the update of signif- icant parameters, which limits the applicabil- ity of FL. techniques to tackle the L.LMs in real scenarios. Prompt tuning can significantly reduce the number of parameters to update, but it either incurs performance degradation or low training efficiency. The straightfor- ward utilization of prompt tuning in the FL often raises non-trivial communication costs and dramatically degrades performance. In addition, the decentralized data is generally non-Independent and Identically Distributed (non-IID), which brings client drift problems and thus poor performance. This paper pro- poses a Parameter-efficient prompt Tuning ap- proach with Adaptive Optimization, i.e., Fed- PepTAA, to enable efficient and effective FL L.MMs. First, an efficient partial prompt tun- ing approach is proposed improve perfor- mance and efficiency simultancously. Second, a novel adaptive optimization method devel- oped to address the client drift problems both the device and server sides to enhance performance further. Extensive experiments based on 10 datasets demonstrate the superb performance (up to 60.8% in terms of accu- racy) and efficiency (up to 97.59% in terms of training time) of FedPepTAG TAO compared with baseline approaches. Our code available at https://gii thub. www.cooo//////o -lm-efffie www.l//lmeeeeeppi//// FfF/FedPeTTAO.
Large Language Models (LLMs) (Devlin et al., 2018; Liu al., 2019b; Brown al., 2020; Lewis et al.. 2019) have achieved major advances in Nat- ural Language Processing (NLP) tasks. The scale of LLMs can range from 110 million parameters to 175 billion parameters, which correspond to huge communication and computation costs to update parameters during the pre-training process (Sanh et al., 2022; Wang et al., 2022) or fine-tuning pro- cess (Ding et al., 2023). Both pre-training and fine-tuning update the whole set of parameters of the language model. Thus, the upplication of FL in the pre-training or the fine-tuning process is most impossible due to significant communication burden brought by large amount of parameters.
Prompt design (Brown et al., 2020) can lead to excellent performance while freezing the orig- inal LLMs. When the LLMs are frozen, only the prompts or prefix are updated during the tuning process, which can significantly reduce the number of parameters to update. For instance, for sam- ple from sentiment analysis task (e.g.. "beautiful place!"), a discrete prompt "It [MASK]." for prompt tuning (Brown et al., 2020) and continu- ous task-specific vectors for prefix tuning (Li and Liang, 2021) can be concatenated to be sent to a LLM, which generates the label of the sample to be "terrible" or "great".
1 Introduction
As a promising paradigm to handle decentralized data, Federated Learning (FL) (Kairouz et al., 2021) enables collaborative model training without transferring the raw data across a massive num- ber of devices. As bunch of legal restrictions
Numerous parameter-efficient prompt or prefix tuning approaches have been proposed to tune the large language models through updating a few train-
Equal contribution. Corresponding author: yangzhou @ auburn.edu
jiliuwark@gmail.comm

---

able parameters while achieving comparable perfor- mance compared with fine-tuning. order avoid human involvement in the prompt design, prompt tuning methods (Shin et al., 2020) are proposed to search proper prompts within a discrete space of words, which corresponds to inferior performance compared with fine-tuning. Continuous prompts, i.., prefix tuning, can be updated to achieve better performance (Liu al., 2021; Lester al., 2021). However, this approach leads to sub-optimal perfor- mance for the models with less than 10 billion pa- rameters. Although P-tuning V2 (Liu et al., 2022d) achieves comparable performance compared with fine-tuning, it introduces more parameters, which may correspond to heavier communication costs in the setting of FL compared with other parameter- efficient tuning approaches. Some other parameter- efficient prompt tuning methods either suffer from low performance with the focus on low-rank hyper- complex adapter layers (Karimi Mahabadi et al., 2021a) or prompt with a single layer (Liu et al., 2022c), or introduce extra computation costs with attentions (Asai et al., 2022). In addition, existing FL techniques for fine-tuning large language mod- els typically incur performance degradation low efficiency due to huge communication costs (Tian al., 2022: Sun et al.. 2022; Zhao et al., 2023).
rameters in all the prompt layers corresponds heavy communication costs, we propose an effi- cient and effective method to choose proper layers of prompts based on the importance of each layer. We design a scoring method to identify the im- portance of cach layer according to the tuning im- pact of the layer on the final convergence accuracy. In addition, we propose an adaptive optimization method both server side and device side with control measures on each device achieve superb accuracy. We summarize out major contributions as follows:
We propose novel purameteer-effciiet prompt tuning method with an efficient and effective method to choose proper layers of prompts for FL. The subset of layers of prompts can reduce both the communication and computation costs within FL.
We provide original adaptive optimization method on both server side and device side with control measures on each device.
We carry out extensive experimentation based on 10 datasets, which demonstrates the advan- tages of FedPepTAO in terms of accuracy (up to 60.8% higher) and efficiency (up to 97.59% faster) compared with 9 baseline approaches.
Adaptive optimization methods, e.g., Adaptive Moment Estimation (Adam) and Stochastic Gra- dient Descent (SGD) with Momentum (SGDM) (Sutskever et al., 2013), have been utilized either on server side (Duchi et al., 2011; Reddi et al., 2018a) or on device side (Yuan et al., 2021; Liu et al., 2020; Gao et al., 2021a; Wang et al., 2020) to achieve superior performance in FL. However, the direct application of the adaptive optimization methods may incur problems of the convergence within the training process (Reddi et al., 2018b). Furthermore, the application of adaptive optimiza- tion on single side, i.e., either device or server, may correspond to poor performance. However, when the adaptive optimization is exploited on both sides (Jin et al., 2022a) may incur heavy communi- cation costs. In addition, client drift (Karimireddy al., 2020b) may exist in terms of the adaptive op- timization due non-Independent and Identically Distributed (nD)-IID) data among devices.
Related Work
As updating all the parameters of a pre-trained LLM consumes a large amount of memory and computation resources, prompt tuning (Brown et al., 2020) or prefix tuning (Li and Liang, 2021) proposed to update a few parameters with a frozen language model while achieving comparable perfor- mance compared with fine-tuning. While prompt tuning may correspond inferior performance with discrete space of words (Shin et al., 2020). prefix tuning (Liu al., 2021: Lester al., 2021) can deal with continuous prompts to achieve bet- ter performance. Adapter modules (Houlsby et al., 2019) are exploited to tune large language models with prompts, which may incur heavy computa- tion costs due to the calculation of feed-forward project and non-lincarity attention mechanism (Asai et al., 2022). Although efficient low-rank hypercomplex mechanism (Karimi Mahabadi al., 2021b) can be utilized to reduce parameters to up- date, the performance may degrade. P-tuning V2 achieves comparable performance compared with
In this paper, we propose a Parameter-efficient prompt Tuning approach with Adaptive Optimiza- tion, i.c., FedPepTAO, to tune large language mod- els with FL. As transferring the whole set of pa-

---

fine-tuning (Liu al., 2022d), which the prompts added into each layer of the large language model. Prompts can be added at a single layer to further reduce the computation costs (Liu et al., 2022c), which may incur performance degradation and de- pends on multiple trials with each layer. In ad- dition, the selection of the layer may incur long execution time to verify the impact on the final ac- curacy. Although the NASWOT algorithm (Mellor al., 2021) can be exploited to analyze the perfor- mance of a neural network architecture, it only compatible with the neural networks based on the ReLU activation function (Nair and Hinton, 2010; Glorot et al.. 101
within FL on server side (Duchi et al., 2011; Reddi et al., 2018: 2018a) or on device side (Yuan al., 2021; Liu et al., 2020; Gao et al., 2021a; Wang et al.. 2020) to address the client drift problem brought by the non-IID data in FI. (Karimireddy et al., 2020b). However, the direct application of adaptive opti- mization on devices may lead to convergence prob- lem (Reddi et al., 2018b). In addition, the appli- cation of adaptive optimization on both server and device sides may incur heavy communication costs (Jin al., 2022a).
Different from the previous work, we propose a general scoring method to analyze the correlation of each layer and the output of the large language model, which can represent the importance of each layer. Then, we select the prompt parameters of proper layers to be updated with FL while leav- ing other prompt parameters of other layers to be adjusted locally with a lossless method so as to achieve superb performance with limited commu- nication costs. In addition, we introduce control measures on each device to alleviate the client drift problem and propose novel adaptive optimization method on both server and device sides further improve the performance.
Parallel, distributed, and federated learning have been extensively studied in recent years (Liu ct al., 2023a; Chen et al., 2023b,a; Lee et al., 2019; Wu et al., 2021; Goswami et al., 2020; Zhang et al., 2021; Zhou et al., 2022: Guo et al., 2022; Jin et al., 2022a; Che al., 2022; Yan al., 2022a; Liu
et al,, 2022b; Yan al., 2022b,e; e; Jin et al., 2022b, 2021; Zhao al., 2021; Zhou and Liu, 2013; Lee et al., 2013; Zhang et al., 2013; Zhou et al., 2014; Zhang et al.. 2014; Bao et al., 2015; Zhou et al.. 2015a,b; Lee et al., 2015; Jiang et al., 2019; Zhang et al., 2022; Zhou, 2017; Hong et al., 2023; Chen et al., 201 1bba;; Gan et al., 2023; Che et al., 2023; Liu et al., 2023c, 2022a; Li et al., 2023; Oliveira al., 2019; Liu al., 2019a, 2016, 2015). Some existing FL techniques have been proposed to fine- tuning large language models, which may suffer from performance degradation or low efficiency due to huge communication costs (Tian et al., 2022; Sun et al., 2022; Zhao et al., 2023). FedBert (Tian et al., 2022) exploits split learning to split model into two parts, i.c., one with transformer and the other one with head and embedding. As the trans- former shared on the server, FedBert may cor- respond to inferior performance and huge com- munication costs compared with prompt-tuning or prefix-tuning. Some other FL methods only fine-tune a part of the model weights (Sun et al., 2022), which still suffer from heavy communica- tion costs for big language models. FedPrompt (Zhao al., 2023) enable FL based prompt tuning, which communicates the whole set of pa- rameters in prompts corresponding to huge com- munication costs.
Problem Formulation
The problem to address in this paper is how to efficiently tune a large language model based on prompt tuning in FL. Given a large language model M with L layers, in M and denote the set of parameters to generate prompts by Pt with representing the number of layer in M. The whole set of prompts is denoted by P During the tuning process, the parameters in M are frozen and cannot be updated while the param- eters in P are updated to improve the performance of M.
add prompts for each layer
We consider FL setting with a parameter server and M devices. We assume that the data for the tun- ing process of M distributed among multiple de- vices. On each Device i, a dataset D, = {s,, m is located with 8;, m,, and ny representing a sample, the corresponding label of s;, and the number of samples in D; We denote the total number of the samples on all the devices by N, the set of all the samples by S and that of labels by M. Due the limited computation capacity, each Device can only perform the inference of M while updating P; with P; representing the prompt parameters in De- vice i. In order to reduce communication costs, we
Adaptive Moment Estimation (Adam) and Stochastic Gradient Descent (SGD) with Momen- tum (SGDM) (Sutskever et al., 2013) are exploited

---

(CLS)
Prety e(Prety))
dull
@ Upload and v, to the server becore efficient toning plass)
# (GLS))
(ddull)
ed
Laper1 Prompts-
① Downloed and distritute St. to all clients Decore efficient tuning phase)
Layer Prompts
Selected, Promots
Layer Prompts
Uploading and A ggregation
Update Wi+ with (
Act with mn
Selected, Prompts for communication
Production (with linear head)
Promytts
Local Prompts
@ Prompis tuning uptistt
Downloading
Frozen FLM backbons
[CLS] el (CLS)
Nice
visual
@ Upload v to
Ignooee Prompts on server
e(Nice)
elvisusl]
e(l
( becore efficiest tuning plase)
i Download and distrbute ST.
Leyer Prompts
clients (before efficiest tuning plase)
Layer 2 Prompis-
X Selected Prompts
Layer Prompts-
Upleading Aggmeptionn
Calculata and for each cleat
Prediction (with linear bead)
@1 Prompts update
① Selected Prompts
Downloading
Figure 1: The system model of FedPepTTA.
enable the exchange the parameters of the prompts within a subset of selected layers ST,, between De- vice and the parameter server while the other only updated within each device. We denote the set of prompt parameters in all the devices by P. The problem address in this paper can be formulated as how to efficiently generate such that the global loss is minimized:
of FedPepTAO. Then, we propose parameter- efficient prompt tuning method and the adaptive optimization method, respectively,
prompt parameters
3.1 System Model
As shown in Figure we consider parameter server and multiple devices for the tuning process of FedPepTAO. We assume that large language model is deployed on each device. For each layer, we insert a prompt module. During the tuning process, the large language model only perform inference while the prompt modules of cach layer perform both the inference of the input and the update of parameters. Within the FL tuning pro- cess, the prompt parameters of specific layers are communicated between the device and the server. During the FL tuning process of FedPepTAO, the prompt parameters in each device are updated with multiple rounds. Each round consists of five steps. First, set of devices are selected to perform the update of prompt parameters. Second, these devices receive the corresponding updated prompt parameters of specific layers from the server (O). The selection the specific layers is based on our parameter
M
min F(M.P)
T.F.(M,
PEP
(1)
where F(M,P) P) represents the global loss, Fi(M,ps) 1. (e..m..) PL,,P..s; m)) refers to the loss function on Device k with ['M, pi, Si, ma) calculating the local loss of the combination of the large language model M and prompt parameters p; on {sk, mg).
For NLP tasks, each sample 8ę E is the input the large language model and my E M is the cor- responding label. Each sample is composed of multiple tokens, i.e., 8 = {sk ..., st), where represents the length of the input. The prompt p consists of multiple tokens p (Pi, Po...., ). and the corresponding prompt parameters can be trained. The prompts differ according layers. We denote the template by T(-). which defines how to concatenate the input tokens with the prompt. For instance, & T (sx.P) represents the sam- ple combined with the prompt, which contains one [MASK] token. The output of the large lan- guage model with the prompts predicts the label mg. which corresponds to the [MASK] token after applying verbalizer VV-), i.e. nix V(Ok)) with og representing the output of the model and ni
prompt tuning method (see de- tails Scction 3.2) (2) 3. Third, the prompt parameters are updated with our adaptive optimiza- tion method (see details in Section 3.3) based on the data on each device (4). Fourth, the prompt parameters of specific layers sent back the server (5). Fifth, the prompt parameters aggre- gated on the server with the adaptive optimization method (6) 7)
3.2 Parameter-efffcieet Prompt Tuning
referring to the predicted label.
We propose a parameter-efficient prompt tuning method to efficiently tune the language model with
In this section, we first present the system model

---

FL. Instead of synchronizing the full set of prompt parameters, we select a proper set of layers for each device and only exchange the prompt param- eters of these layers during the tuning process. In this section, we propose a scoring method to mea- sure the importance of cach layer. Then, we pro- pose lossless layer selection method to select the proper layers, which reduces the communication costs without performance degradation.
Algorithm Federated Parameter--fficient Prompt Tuning
Require:
L: The list of layers a large language model
M: The set of devices
w: The prompt parameters of the initial model
w": The prompt parameters of the current model in Round t
Ensure:
Given the prompt parameters based on any acti- vation function, we can calculate the hidden states of each parameter at each layer of the large lan- guage model. With a batch of local data samples S; - { mapped through the large language model and the prompt parameters corresponding to the function fp(s)), the hidden state corresponding Node A at 1-th layer is pp.. Then, the hid- den states of Layer corresponding to Sample 8; is h = [Sp.. (si), (8:) representing the number of nodes at Layer L.
SL: The set of selected layers
SL 0
for i M (on each device) do
3:
A f- w - w²
4:
H Hww)) + Get Hessian matrix of us
A ((( + Eigenvalues(II (w+))
5:
6:
{M , AI ,.... A, + Sort in ascending or-
der of A (((o))
7:
B(A,)
+ H Hww)) V--A, + w!)
fpx.. (s:)), with Ki
8:
L <- Get Lipschitz constant of B(A;)
k; + Get the first k that meets k+1 A
As the difficulty for a network to learn to sepa- rate the input samples has positive correlation with the similarity of the hidden states (Mellor al., 2021), we examine the correlation between the hidden states of any two layers by computing the following kernel matrix:
4.%:
Ki-ki K,
10: Ti
11:
for I do
12: Calculate Gi according to Formula 3
13: end for
14: end for
15: Aggregate and each Ç based on Formula 4
Cos,,,,1 haa)
Cos(, hat)
in descending order La) 17: while 1 E ( and Para(L) r do
16: < Sort (S1, ..., Cy (..
Kly,
Cos h,11)
Cos(hi, b.t))
18: ST. < SLUI
(2) where Cos(ba, 1a,1) represents the cosine sim- ilarity between two vectors (Dehak et al., 2010). Then, we calculate the eigenvalues of the kernel matrix A; = { (Ar,1 Ar,,,., .., ), with x represent- ing the distinction of Layer compared with other layers based on Sample s;. Afterward, we compute the score (GL)) of Layer with the local dataset on Device i using the Formula 3, which can avoid the possibility of unacceptable performance penalty due to abnormal eigenvalues (Gao et al., 2021b).
19: end while
where y represents the variable.
In order to efficiently tune the large language model without performance degradation, we ex- ploit a lossless method as shown in Algorithm I to select the set of proper layers within FL. Within first i rounds, the prompt parameters of all the lay- communicated between the server and each device. can be small, c.g., 10. At f-th round, we perform the layer selection. First, we calculate A; as the changement of the prompt parameters (Line 3). Then, we calculate the Hessian matrix (based an efficient PyHessian library (Yao et al., 2020)) of the current model (Line 4), the corre- sponding eigenvalues (Line 5), and sort the eigen- values in ascending order (Line 6). Afterward, we construct base function in Line 7 with V.F rep- resenting the gradients and calculate the Lipschitz constant of the base function We take the first k (4) that can meet the constraint in Line 9, and calculate the minimum remaining prompt parameter ratio in
n G.11
Σ 1=1
1og()x (Ais + + (3)
Th
where E refers to a small positive value, e.g., 1 e 5 We calculate the global score of each layer leveraging Formula 4 with = Cr.1-
N
n4y
Vi
N

---

the selected layers Ri, inspired by (Zhang al., 2021), which can achieve lossless compared with those at all the layers. We calculate the score of each layer in Lines 11 - 13. The execute of Lines 3 13 can be carried out in parallel on each de- vice. We aggregate the prompt parameter ratio and the scores based on Formula 4 from each device to the server (Line 15). Then, we sort the layers according the scores in descending order (Line 16). Finally, we add the layers into the selected layer set based on the scores in descending order (Lines 17- - 19), with Para(SL:) representing the number of parameters in the selected layer set. In the following rounds. the prompt parameters in SL are communicated between devices and the server
Algorithm 2 Communication-Lffiiieenn Adaptive Optimization
Require:
M: The set of devices
w: The prompt parameters of the initial model
R: The maximum number of global round
0: The local step size
3: The momentum parameter
7 {n',, R] The set of global learn-
ing rates
T T2, TMr: The set of local epoch
T, on each Device
Ensure:
" The final model
1: wD + w. 0 + 0, cg + 0. mn + 0,ViE
2: for r - ... R do
3.3 Communication-Effiiiae Adaptive Optimization
3: Randomly sample subset S of devices M
for i S (on each device) do u0 r. <- w" m,
r.0 0, o
r.0
0
While data is generally non-llD, we propose novel communication--fficien adaptive optimiza- tion to achieve superb performance without intro- ducing extra communication costs. In order to achieve excellent performance, we propose apply- ing adaptive optimization on both server based on momentum (Cutkosky and Mehta, 2020) and de- vice sides based Adam (Kingma and Ba, 2015). We reset the first and the second momentum buffers zero at the beginning of local update (Wang et al., 2021) to avoid extra communication of the momen- tum variables between the server and the device. In addition, we maintain a state for each device on the server to avoid possible client drift problem incurred by non-III (Karimireddy et al., 2020b).
for 1,2,
do oft V " Fi
n m, ni nit t Adam update with
8:
w/ m rit 1 o
and g,"
9:
end for
Auf = wi w r:0 end for
10:
11:
12: Aggregate Aw" based on Formula 4
13:
for i E S (on the server) do
14:
of =c c
cg
To Aw!
15:
Ac = cf c
16:
end for
17:
Aggregate Ac based on Formula 4
oy o dr + Ac" [
18:
M
The algorithm of communication-efficin adap- tive optimization for FL prompt tuning is shown in Algorithm 2. Within each round, we first ran- domly sample subset of devices (Line 3). Then, the prompt parameters corresponding to the model in the last round is sent to each device (Line 5), and each selected device perform local update based on Adam (Lines 7 - 8). Afterward, each selected device returns the accumulated difference of the prompt parameters to the server (Line 10). Please note that the execution of Lines 4 11 can be per- formed in parallel on each selected device. We aggregate the differences based on Formula 4 (Line 12). Inspired by (Karimireddy et al., 2020b), we calculate the control variate of (Line 14) and the corresponding difference Act (Line 15) for each device on the server. We aggregate the control vari- ate differences based on Formula 4 (Line 17), and
19:
by Aw"
Aw"
20:
for i S (on the server) do
21:
m. 3 m - +1 (1- B)
g + cg c
22:
w = w/ 17" * m
23:
end for
24: Aggregate w" based on Formula 4
25: Aggregate m² based on Formula 4
26:
end for
calculate the global control variate (Line 18) and global gradients (Line 19). Afterword, we update the momentum (Line 21) and prompt parameters (Line 22) for each selected device. Finally, ag- gregate the global prompt parameters (Line 24) and momentum (Line 25). The communication cost of Algorithm depends on the size of prompt parame- ters, which is similar to that of FedAvg (McMahan et al., 2017), while achieving superb performance.

---

Method
Comm, Params
QNLI
SST-2
CoLA
MPRC
RTE
BoolQ
MPQA
Subj
Trec
MR
Avg
Adapter
7.4M
87.79
94.04
30.96
71.81
68.59
75.11
90.97
94.6
79
91.9
75.36
FedPrompp
131K
85,91
94.84
33.05
77.87
61.73
74.77
90.45
94.25
95
91.65
76.85
P-tuning v2
6.3M
85.19
95.3
41.82
82.78
79.42
79.66
91
96.9
96.4
91.45
82.05
Prompt Tuning
20K
51.62
61.01
3,36
48
52.35
59.72
81,65
65.2
36.4
63.25
42,56
IDPG
137K
72.2
93.01
4.59
70.83
70.4
71.96
90.07
94
78.4
91.4
60.63
ATTEMPT
207k
54.93
85,89
4.63
78.65
58.48
73.49
91.05
88.95
82.2
91.9
58,28
LPT
792k
89.2
94.84
53.7
82.07
79.7
62.7
90.55
96.5
96.4
91.4
82.35
MomD
6.3M
67.42
93.92
1.64
75.17
75.45
62.02
89.05
49.95
38.6
83
63.62
MomS+AdamD
6.3M
87.85
95.18
42.96
80.15
78.1
90.95
96.75
96.8
91.55
84.26
FedPepTAO
492K
89.57
95.87
56.35
87.52
85.56
79.72
91.4
97.1
97.2
93
86.4
Table 1: The accuracy with TAO and diverse baseline approaches. All the methods from GLUE benchmark are evaluated on development sets while other tasks are evaluated with test sets. The best results are highlighted in bold and the second bests are marked with underline. All the results are obtained using RoBERTaLARGE.
4 Experiments
ity of FedPepTAAO, we carried out extra experi- ments with three additional prooode-bbssed models, i.e., GPT2 model (Radford et al., 2019) with 774M parameters on MRPC, MR, SST-2 dataset,
In this section, we present the experimental results over 9 baselines and 10 commonly-use tasks to demonstrate the advantages of FedPepTAO.
LLaMA 3B model (Touvron et al., 2023) on RTE, MRPC dataset, and LLaMA 7B model (Touvron et al., 2023) on MRPC dataset. The backbones of these models are frozen for all methods.
4.1 Experimental Setup
We consider an FL environment with 100 devices and a parameter server. In each epoch, we ran- domly sample 10 devices to perform the local up- date. We exploit 10 widely used NLP tasks includ- ing QNLI (Rajpurkar et al., 2016) SST-2 (Socher et al., 2013), CoLA (Warstadt et al., 2019), MRPC (Dolan and Brockett, 2005), RTE (Giampiccolo et al., 2007), and BoolQ (Clark et al., 2019) from the GLUE benchmark, and other tasks includ- ing MPQA (Wiebe et al., 2005), Subj (Pang and Lee, 2004), TREC (Voorhees and Tice, 2000), and MR (Pang and Lee, 2005) (Please see details in Appendix A.2). We take existing approaches as baselines, including an adapter-based method, i.e.,
4.2 Evaluation of FedPepTAAO
As shown in Table 1, FedPepTAO significantly outperforms baseline methods in terms of the best accuracy (up to 25.39%, 23.83%, 14.53%. 60.8%, 51.76%, 51.72%, 17.02%, 54.71%, 13.39% compared with Adapter, FedPrompt, P-tuning v2, Ptompt Tuning, IDPG, ATTEMPT, LTP, MomD, and MomS+AdamD, respectively). In addition, the average of the best accuracy (average accu- racy) for each task is shown in the last column. The advantage of FedPepTAAO is obvious in terms of the average accuracy as well, i.e., 11.04%, 9.55%, 4.35%, 43.84%, 25.77%, 18%, 4.05%, 58.6%, 13.39%, higher compared with Adapter, FedPrompt, P-tuning v2, Ptompt Tuning, IDPG, ATTEMPT, LTTP, MomD, and MomS+AdamD, re- spectively, Although FedPrompt, Prompt Tuning, IDPG, and ATTEMPT exploit fewer parameters, the corresponding accuracy inferior. FedPrompt. Prompt Tuning, and ATTEMPT only update the soft prompt for the first layer, which cannot opti- mize other important layers and incurs sub-optimal performance. IDPG shares a single generator for each layer, which cannot address the characteris- tics of diverse layers and leads to inferior accuracy. Different from these methods, Fe well optimize the prompt parameters for each layer based on P-tuning v2, while choosing the proper layers for aggregation within FL so as to achieve excellent performance. In addition, we exploit the adaptive optimization on both server and device
Adapter (Houlsby et al., 2019). prompt-based tun- ing methods, i.e., FedPrompt (Zhao et al., 2023), P-tuming v2 (Liu et al., 2022d), Prompt Tuning (Lester et al., 2021), IDPG (Wu et al., 2022), AT- TEMPT (Asai et al., 2022), LPT (Liu et al., 2022c), and optimization approaches, i.e., momentum on the device side with simple SGD on the server side (MomD) (Karimireddy et al., 2020a) and momen- tum on the server side with Adam on the device side without control variate (domS+AdamD). We adapt the centralized methods, i.e., Adapter, P-tuning v2, Prompt Tuning, IDPG (S-IDPG-PHM), ATTEMPT, and L.T,, with FedAvg (McMahan et al., 2017) to the FL setting for a fair comparison.
We evaluate FedPepTAO and all other meth- ods on ROBERTaLARGE (Liu et al., 2019b), which consists of 24 layers of transformers followed by a large language model head and 355M pre- trained parameters. To demonstrate the adaptabil-

---

Method
QNLI
SST-2
CoLA
MPRC
RTE
BoolQ
MPQA
Subj
Trec
MR
Adapter
8096
1065
2655
2218
937
1178
1388
1797
FedPrompt
12987
1471
1824
/
1485
412
1284
336
618
P-tuning v2
10780
489
154
201
135
105
132
748
Prompt Tuning
7
/
IDPG
3322
689
3254
908
1347
1220
1912
ATTEMPT
1774
/
973
438
2573
1028
1221
LPT
2918
650
733
156
270
/
162
155
112
860
MomD
/
1328
/
838
537
/
MomS AdamD
697
209
488
1178
192
139
141
166
100
610
TedPepTAO 781
97
219
230
186
129
31
62
76
610
Table 2: The tuning time (8) to achieve target accuracy (85% for QNLI, 92.5% for SST-2, 3% for CoLA, 77%
for MRPC, 65% for RTE, 71% for BoolQ, 85% for MPQA, 88% for Subj. 78% for Trec, 91% for MR)with FedPepTAO and diverse baseline approaches. "/" represents that training does not achieve the target accuracy. The best results highlighted in bold and the second bests are marked with underline. All the results obtained using RoBERTaLARGE
MRPC
MR
SST-2
RTE
MRPC
Method
Method
Acc
Time
Acc
Time
Acc
Time
Acc
Time
Acc
Time
FedPrompt
74.98
76.49
FedPrompt
78.34
540
81.86
459
P-tuning v2
74.8
74.77
P-tuning v2
56.68
75.17
/
ATTEMPT
37.91
ATTEMPT
64.98
81.18
718
LPT
74.98
1455
77.06
LPT
64.98
79.77
789
MomD
77.23
503
92.55
MomD
80.87
360
80.26
669
MumS+AdandD
76.89
291
262
92.55
MomS+AdamD
80.87
360
75.58
/
PеdPepTAO
81.23
273
89.5
222
93
2248
FeaPepTAO 83.59
325
86.46
409
Table 3: Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC, 81% for MR, and 92.5% for SST-2) on GPT2LARGE model. "/" represents that training does not achieve the target accuracy.
Table 4: Accuracy and tuning time (s) to achieve target accuracy (75% for RTE, 81% % for MRPC) LLAMM 3B model. "/" represents that training does not achieve the target accuracy.
sides to achieve superb accuracy. Compared with Adapter (93.4%), P-tuning v2 (92.19%), and LPT (37.98%), our methods can well reduce the number of parameters to transfer between devices and the server because of the proper layer selection, which corresponds to smaller communication costs, As result, the efficiency of FedPepTAO is significantly higher than baseline approaches (up to 95.91% %, 95. 95%, 92.76%, 99%, 97.28%, 97.59%, 85.8%, 94.23%, 80.48% faster compared with Adapter, FedPrompt, P-tuning v2, Ptompt Tuning, IDPG, ATTEMPT, LTP, MomD, and MomS+AdamD, re- spectively).
respectively). Furthermore, the efficiency of FedPepTAAO is significantly higher than base- line approaches (up to 84.74%, 86.89% and 15.27% faster compared to LPT, MomD, and MomS+AdamD, respectively).
When the model becomes larger, i..., LLaMA 3B with 3 billion parameters, FedPepTAO still achieves the best accuracy (up to 4.6%, 11.29%. 5.28%, 6.69% 6.2%, 10.88% higher compared FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and better effi- ciency (up to 39.81 %, 43.04%, 48.16%, 38.86%, 9.72% faster compared to FedPrompt, ATTEMPT, LPT, MomD, and MomS+ respectively) illustrated in Table 4.
4.3 Evaluation of TAO on Extra LLMs
To demonstrate the adaptability of FedPepTAO, we carried out extra experiments with three additional decoder-based models, ie.. GPT2LARGE model (774M) on MRPC, MR, SST-2 dataset, LLaMA 3B model on RTE, MRPC dataset, and LLaMA 7B model on MRPC dataset.
We verify the performance of our method with another large model, i.e., LLaMA 7B with 7 billion parameters. As shown in Table 5, FedPepTAAO outperforms baseline methods in terms of accuracy (up to 5.05%, 26.71%. 7,,1 18.41%, 18.41% %, 6.2%. 10.88% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and efficiency (up to 15.77% 75.14%. 32.06%, 67.04% faster compared to ATTEMPT, LPT, MomD, and MomS+AdamD, respectively), which demonstrates the scalability of FedPepTAO.
As shown in Table below, FAO signifi- cantly outperforms baseline methods in terms of the best accuracy on the decoder-based GPT2LARGE model (up to 32.3%, 18.23% 43.32%, 15.94% 4%, 4.34% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD,

---

MRPC
in FL settings and lead to excellent performance. In addition, please note that FedPepTAO calculates the control variate on the server with updated gra- dients (Algorithm 2), without extra communication cost.
Method
Acc
Time
FedPrompt
76. 18
/
P-tuning v2
74.98
/
ATTEMPT
81.52
317
LPT
81.95
1074
MomD
76.2
393
4.6 Hyperparameter Evaluation
AdmS+
75.75
810
PedPepTAO 82.34
267
In this section, we evaluate the performance of Fed- Pep with divers hyperparameters. Additional experiments are in the Appendix B.
Table 5: Accuracy and tuning time (8) to achieve target accuracy (75% for MRPC) on LLaMA 7B model. "/" represents that training does not achieve the target accu- racy.
Impact of server learning rate Due to the high sensitivity of the hyperparameters in NLP tasks, we investigate the impact of divers server learning rates on the RTE dataset. We analyze the accuracy with the learning rates 1 2, 5e 3, 1e 3 55c We find that the best performance was achieved when Ir 1e 1e-3, which only slightly outperforms Ir 5e by 0.37% (see Figure in the Appendix). This demonstrates that FedPepTAA is easy to fine-tune in practice.
4.4 Evaluation of Parameter-efficient Tuning
Formula calculates the global score Çi of each transformer layer in the model, based on which the proper layers are selected to enable the com- munication between devices and the server. The selection of layers is eritical the performance of FL. In this section, we compare our parameter- efficient prompt tuning (PEPT) with three other se- lection strategies, i.e., select with ascending order, descending order ((Liu et al., 2022d)), and random order. Our PEPT method can significantly outper- form ascending order (up to 5.78%), descending or- der (up to 1.81 )), and random order (up to 2.89%) (see Figure 5 in Appendix). In addition, we con- duct experiments and demonstrate that PEPT method outperforms random layer selection strat- egy by 2.93% on RTE, 4.73% on MRPC, and 7.29% on CoLA dataset (see Appendix B.6 for details).
Impact of heterogeneous data distribution Data heterogeneity has always been a common challenge in FL. To investigate the robustness of our approach to different degrees of non-lID data, conduct experiments under various levels of non-IID degrees. We observe that the accuracy of FedPepTAO is the best with ranging from 1.0 to 5.0 (see Figure 3 in Appendix). A smaller a represents a higher degree of heterogeneity. This demonstrates that our approach is robust to differ- ent data heterogeneity.
4.5 Evaluation of Adaptive Optimization
5 Conclusion
To demonstrate the effectivvness of Algorithm 2, we compare our adaptive optimization method with six baseline methods, i.e., FedAvg (McMahan et al., 2017), MomD, momentum on the server side with SGD on the device side (MomS) (Reddi et al., 2018a), momentum the server side with control variate and SGD on the device side (MomS+Con) (Reddi al., 2018a), Adam on device side with simple SGD on the server side (AdamD), and MomS+AdamD, on the RTE task (see Figure in Appendix). FedPepTAO corresponds to the high- est accuracy compared to baseline methods (up to 8.3% compared with MomD, 29.24% compared with MomS, 29.24% compared with MomS+Con, 2.16 2.16% compared with AdamD, 5.05% compared with MomS+A and 29.24% compared with FedAvg). The advantage of FedPepTAO up to 5.05% compared with MomS+ Addmm, which shows that the control variate can avoid client drift
In this paper, we propose an original parameter- efficient prompt tuning with adaptive optimization approach for large language models with FL, i.e., FedPepTAO. We dynamically caleulate the score of each layer choose proper layers of prompts for FL without accuracy degradation. In addition, we provide a novel adaptive optimization method with control variate on the server to achieve superb performance without extra communication costs. We carry out extensive experiments based on 10 tasks and 9 state-of-thes -hrtr--rttttt baseline approaches, which demonstrate significant ad advantages of Fed- PepTAAO in terms of accuracy (up 60.8% higher) and efficiency (up 97.59% faster).
A Akknowledgements
This research partially sponsored by the National Science Foundation (NSF) under Grant No. OAC- 2313191.

---

Limitations
Zhuo Chen, Gang Feng, Bei Liu, and Yang Zhou. 2018b. Delay optimization oriented service function chain migration and re-deployment in operator network. Acta Electronica Simica, 46(9):222--2237.
While our method can significantly enhance the per- formance and efficiency of federated prompt tuning in large language models (LLMs), we should ac- knowledge the sharing of prompts between clients and the server. Previous research has demonstrated that transferring additional sensitive information in Federated Learning (FL), such as predictions or embeddings, can lead to potential privacy con- cerns (Che et al., 2022). These concerns can be even more critical in prompt tuning scenarios, as prompts explicitly tuned with private local data. We anticipate that evaluating and mitigating pri- vacy risks in federated LLM prompt tuning will be an intriguing research area the future.
Zhuo Chen, Xiaoxiang Tan, Zhiyuan Zhou, and Yang Zhou, 2023a. A channel aggregation based dynamic pruning method in federated learning. In IEEE Global Communications Conference (GLOBECOM). To appear.
Zhuo Chen, Chuan Zhou, and Yang Zhou. 2023b. A hierarchical federated learning model with adaptive model parameter aggregation. Computer Science and Information Systems, 20(3): 1037-1600
Christopher Clark, Kenton Lee, Ming-Wer Weng-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2924-2936.
References
Akari Asai, Mohammadreza Salchi, Matthew E Peters, and Hannaneh Hajishirzi. 2022. Attempt: Parameter- efficient multi-task tuning via attentional mixtures of soft prompts. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 6655- 6672.
Ashok Cutkosky and Harsh Mchta. 2020. Momentum improves normalized SGD. In Int. Conf. Machine Learning (ICML), volume I 119, pages 2260-2268. PMIR.
Xianqiang Bao, Ling Liu, Nong Xiao, Yang Zhou, and Qi Zhang. 2015. Policy-driven autonomic configu- ration management for nosql. In IEEE Int. Conf. on Cloud Computing (CLOUD), pages 245-252.
Najim Dehak, Reda Dehak, James R Glass, Douglas A Reynolds, Patrick Kenny, et al. 2010. Cosine similar- ity scoring without score normalization techniques. In Odyssey, page 15.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 33:1877- 877-1901.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. 8. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv.1810004805.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. 2023. Parameter- fine-tuning of large-scale pre- trained language models. Nature Machine Intelli- gence, 5()::220-235.
Californians for Consumer Privacy. 2020. California consumer privacy act home page. https://wwa caprivacy. org/ Online; accessed 09/05/2022.
Tianshi Che, Zijie Zhang, Yang Zhou, Zhao, Ji Liu, Zhe Jiang, Da Yan, Ruoming Jin, and Dejing Dou. 2022. Federated fingerprint learning with hetero- geneous architectures. In IEEE Int. Conf. on Data Mining (ICDM), pages 31-40.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan. Dejing Dou, and Jun Huan. 2023. Fast federated machine unlearning with nonlinear functional theory. In Int. Conf. Machine Learn- ing (ICML), volume 202 of Proceedings of Machine Learning Research, pages 4241-4266 PMLR.
John Duchi, Elad Hazan, and Yoram Singer. 2011. 1. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7).
Xinbiao Gan, Guang WU, Ruigeng Zeng, Jiaqi Si, Ji Liu, Daxiang Dong, Chunye Gong, Cong Liu, and Tiejun Li. 2023. Ft-topo: Architecture-driven folded- triangle partitioning for communication--efccieett graph processing. In Int. Conf. on Supercomputing (ICS), page 240-250.
Zhuo Chen, Gang Feng, Bei Liu, and Yang Zhou. 2018a. Construction policy of network service chain oriented to resource fragmentation optimization in operator network. ournal of Electronics and Information Tech- nology, 40(4):763-76.

---

Hongchang Gao, An Xu, and Heng Huang. 2021a. On the convergence of communication-effiee for federated learning. In AAAI Conf. on Artificial Intelligence, 9, pages 7510-7518.
Ruoming Jin, Dong Li, Jing Gao, Zhi Liu, Li Chen, and Yang Zhou. 2021. Towards a better understanding of linear models for recommendation. In ACM SIGKDD Conf. on Knowledge Discovery and Data Mining (KDD), pages 776-785.
local sgd
Wei Gao, Shangwei Guo, Tianwei Zhang. Han Qiu, Yonggang Wen, and Yang Liu. 2021b. Privacy- preserving collaborative learning with automatic transformation search. In IEEF/CVF Conf. on Com- puter Vision and Pattern Recognition (CVPR), pages 114-123.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, and Mehdi Bennis et al. 2021. Ad- vances and open problems in federated learning, Foundations and Trendy in Machine Learning, 14(1- 2):1-210.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021a. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems (NeurIPS), 34:1022- 1035.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. Int. Conf. on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceed- ings.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021b. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems, pages 1022 1035.
Sayan Goswami, Ayam Pokhrel, Kisung Lee, Ling Liu, Qi Zhang, and Yang Zhou. 2020. Lightwieight in- dexing and querying services for big spatial data. The Journal of Supercomputing (TJSC), 76(9);6619- 6647.
Sai Pranceth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. 2020a. Mime: Mim- icking centralized stochastic algorithms in federated learning, arXiv preprint arXiv:2008.0336
Guimu Guo, Da Yan, Lyuheng Yuan, Jalal Khalil, Cheng Long, Zhe Jiang, and Yang Zhou. 2022. Maximal directed quasi-clique mining. IEEE Int. Conf. on Data Engineering (ICDE), pages 1900-1913.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020b. Scaffold: Stochas- tic controlled averaging for federated learning. In Int. Conf. on Machine Learning (ICML), pages 5132- 5143. PMLR.
Junyuan Hong, Zhuangdi Zhu, Lingjuan Lyu, Yang Zhou, Vishnu Naresh Boddeti, and Jiayu Zhou, 2023. Int. workshop on federated learning for distributed data mining. In ACM SIGKDD Conf. on Knowl- edge Discovery and Data Mining (KDD), pages 5861 5862, Long Beach, CA.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Int. Conf. on Learning Representations (ICLR).
Kisung Lee, Ling Liu, Raghu I.. Ganti, Mudhakar Sri- vatsa, Qi Zhang. Yang Zhou, and Qingyang Wang. 2019. Lightwieight indexing and querying services for big spatial data. IEEE Transactions Services Computing (ISC), 12(3):33-3355
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, Parameter-efficient transfer learning for nlp. In Int. Conf. on Machine Learning (ICML), pages 2790- 2799. PMLR.
Kisung Lee, Ling Liu, Karsten Schwan, Calton Pu, Qi Zhang. Yang Zhou, Emre Yigitoglu, and Pingpeng Yuan. 2015. Scaling iterative graph computations with graphmap. In IEEE Int. Conf. for High Perfor- mance Computing, Networking, Storage and Analysis (SC), pages 57:1-57:15
Yexi Jiang, Chang- Shingg-Siggggg Perng, Anca Sailer, Ignacio Silva-Lepe, Yang Zhou, and Tao Li. 2019. Csm: A cloud service marketplace for complex service acqui- sition. ACM Transactions on Intelligent Systems and Technology (TIST), 1-1):1225.
Jiayin Jin, Jiaxiang Ren, Yang Zhou, Lingjuan Lyu, Ji Liu, and Dejing Dou. 2022a. Accelerated federated learning with decoupled adaptive optimization. In Int. Conf. on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 10298-10322. PMLR.
Kisung Lee, Ling Liu, Yuzhe Tang, Qi Zhang, and Yang Zhou. 2013. Efficient and customizable data parti- tioning framework for distributed big rdf data pro- cessing in the cloud. In IEEE Ini. Conf. on Cloud Computing (CLOUD), pages 327-334.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 3045-3059. Association for Computational Linguistics.
Jiayin Jin, 7eru Zhang, Yang Zhou, and Lingfei Wu. 2022b. Input-agnostic certified group faimess via gaussian parameter smoothing. Inf. Conf. on Ma- chine Learning (ICML), pages 10340-10361.

---

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke 7eutlemoyrr. 2019. Bart: De- noising gqueence-to-eqqeene pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461.
Liu, Xuehai Zhou, Lei Mo, Shilei Ji, Yuan Liao, Zheng Li, Qin Gu, and Dejing Dou. 2023c. Dis- tributed and deep vertical federated learning with big data. Concurrency and Computation: Practice and Experience, 35(21):1-17.
Wei Liu, Li Chen, Yunfei Chen, and Wenyi Zhang. 2020. Accelerating federated learning via momentum gra- dient descent. IEEF Transactions on Parallel and Distributed Systems (TPDS), 31(8):1754-176.
Guanghao Li, Yue Hu, Miao Zhang. Ji Liu, Quanjun Yin, Yong Peng, and Dejing Dou. 2023. Fedhisym: hier- archical synchronous federated learning framework for resource and data heterogeneity. In Int. Conf. on Parallel Processing (ICPP), pages 1-11.
Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu, 2022c. Late prompt tuning: A late prompt could be better than many prompts. In Find- ings of the Association for Computational Linguis- tics: (EMNLP), pages 1325-1338. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Annual Meeting of the Association for Computational Linguistics and Int. Joint Conf. on Natural Language Processing (ACL/ICNLP), pages 4582 0882-4597.20 Asso- ciation for Computational Linguisties. Volume Long Papers.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022d. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Annual Meeting of the As- sociation for Computational Linguistics (ACL), pages 61-68. Volume 2: Short Papers.
Ji Liu, Daxiang Dong, Xi Wang, An Qin, Xingjian Li, Patrick Valduriez. Dejing Dou, and Dianhai Yu. 2022a. Large-scale knowledge distillation with elas- tic heterogeneous computing resources. Concurrency and Computation: Practice and Experience, pages 1-16.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint vIXiv22103.10385
Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, and Dejing Dou. 2022b. From dis- tributed machine learning to federated learning: A survey. Knowledge and Information Systems (KAIS). 64(4):885-917.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Ji Liu, Juncheng Jia, Beichen Ma, Chendi Zhou, Jingho Zhou, Yang Zhou, Huaiyu Dai, and Dejing Dou. 2023a. Multi-job intelligent scheduling with cross- device federated learning. IEEE Transactions on Parallel and Distributed Systems (TPDS), 34(2):535- 551.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-effie learning of deep networks from decentralized data. In Artificial Intelligence and Statisties (AISTATS), pages 2273-1282.
Joe Mellor. Jack Turner, Amos Storkey, and Elliot Crowley. 2021. Neural architecture search without training. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 7588-7598.
Ji Liu. Esther Pacitti, Patrick Valduriez. Daniel de Oliveira, and Marta Mattoso. 2016. Multi- objective scheduling of scientific workflows in mul- tisite clouds. Future Generation Computer Systems (FGCS), 63:76-95.
Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Int. Conf. on Machine Learning (ICML), pages 807-814.
Ji Liu, Esther Pacitti, Patrick Valduriez, and Marta Mat- toso. 2015. survey of data-intensive scientific workflow management. Journal of Grid Computing. 13(4):457-493.
Official Journal of the European Union 2016. General data protection regulation. https://eur- lex. europa. -logtettttteet// EN/TXT/PDF / /trr///rErrEEEE::- 3281 668679. Online; accessed 09/05/2022.
Ji Liu, Luis Pineda-Morales, Esther Pacitti, Alexan- dru Costan, Patrick Valduriez, Gabriel Antoniu, and Marta Mattoso. 2019a. Efficient scheduling of sci- entific workflows using hot metadata in a multisite cloud. IEEE Transactions on Knowledge and Data Engineering (TKDE), 31(10):1940-199)
Daniel C. M. de Oliveira, Ji Liu, and Esther Pacitti. 2019. Data Intan-ite Workflow Management: For Clouds and Datta-ntensive and Scalable Computing Environments. Morgan & Claypool.
Ji Liu, Zhihua Wu, Danlei Feng. Minxu Zhang, Xinx- uan Wu. Xuefeng Yao, Dianhai Yu, Yanjun Ma. Feng Zhao, and Dejing Dou. 2023b. Heterps: Distributed deep learning with reinforcement learning based scheduling heterogeneous environments. Future Generation Compuder Systems, 14661):106-117
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd

---

Annual Meeting on Association for Computational Linguistics, page 271-es.
Guangyu Sun, Matias Mendieta. Taojiannan Yang, and Chen Chen. 2022. Exploring parameter-effictee efficcient fine- tuning for improving communication efficiency in federated learning. arXiv preprint arXiv:2210.01708,
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, page 115-124.
Ilya Sutskever. James Martens, George Dahl, and Geof- frey Hinton. 2013. On the importance of initializa- tion and momentum in deep learning. In Int. Conf. on Machine Learning (ICML), pages 1139-1141 1147.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao. Hai Jin, and Lichao Sun. 2022. Fedbert: when fed- erated learning meets pre-training- ACM Transac- rions on Intelligent Systems and Technology (TIST). 13(4):1-26.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLama: Open and effi- cient foundation language models. arXiv preprint a7X.
S Reddi, Manzil Zaheer, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. 2018a. Adaptive methods for nonconvex optimization. In Annual Conf. on Neural Information Processing Systems (NeurIPS), pages 1-17.
Ellen M. Voorhees and Dawn M. Tice. 2000. Building question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, page 200-207.
Sashank J. Reddi, Satyen Kale, and Sanjiy Kumar. 2018b. On the convergence of adam and beyond. In Int. Conf. on Learning Representations (ICLR). OvenReview.net.
Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and amalysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-st zero-shot task gener- alization. In Int. Conf. on Learning Representations (ICLR), pages 1-216.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. 2020. Slowmo: Improving communication--fficeee. distributed sgd with slow momentum. In Int. Conf. on Learning Representa- tions (ICLR), pages 1-27.
Jianyu Wang, Zheng Xu, Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. 2021. Lo- cal adaptivity in federated learning: Convergence and consistency. In The Int. Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML (FL-ICML.)
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung. Iz Beltagy, Julien Lau- nay, and Colin Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization? In Int. Conf. on Machine Learning (ICMI.. pages 22964-22992 PMLR
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with auto- matically generated prompts. In Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235. Association for Computational Linguistics.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631 11642.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation. 39:165-210.

---

Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2021. Topicka: Generating common- sense knowledge-aware dialogue responses towards the recommended topic fact. Int. Joint Conf. on Artificial Intelligence (IJCAJ), pages 3766-3772.
in Neural Information Processing Systems (NeurIPS). 34.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu, 2023. Fedprompt: Communication- efficient and privacy-preservii erated learning. In JEEE Ini. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
prompt tuning in fed-
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yux- iao Dong. V.G. Vinod Vydiswaran, and Hao Ma. 2022. IDPG: An instance-dependent prompt generation method. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5507-5521.
Xin Zhao, Zeru Zhang. Zijie Zhang, Lingfei Wu, Ji- ayin Jin, Yang Zhou, Ruoming Jin, Dejing Dou, and Da Yan. 2021. Expressive 1-lipschitz neural net- works for robust multiple graph learning against ad- versarial attacks. Int. Conf. on Machine Learning (ICML), pages 12719-12735.
Da Yan, Wenwen Qu, Guimu Guo, Xiaoling Wang, and Yang Zhou. 2022a. Prefixfpm: A parallel framework for gencral-purpose mining of frequent and closed patterns. The VLDB Journal (VLDBJ), 31(2)2553- 286.
Chendi Zhou, Ji Liu, Juncheng Jia, Jingbo Zhou, Yang Zhou, Huaiyu Dai, and Dejing Dou. 2022. Efficient device scheduling with multi-job federated learning. In AAAI Conf. on Artificial Intelligence (AAAI), pages 9971-9979,
Da Yan, Yang Zhou, and Guimu Guo. 2022b. Think- like-a-task programming model. Encyclopedia of Big Data Technologies.
Yang Zhou. 2017. Innovative Mining, Processing, and Application of Big Graphs. Ph.D. thesis, Georgia Institute of Technology, Atlanta, GA, USA.
Da Yan, Yang Zhou, Guimu Guo, and Hang Liu. 2022c. Parallel graph processing. Encyclopedia of Big Data Technologies.
Yang Zhou and Ling Liu. 2013. Social influence based clustering of heterogeneous information networks. In ACM SIGKDD Conf. on Knowledge Discovery and Data Mining (KDD), pages 338-346.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Transactions Intelligent Systems and Technology (TIST), 10(2):1-9..
Yang Zhou, Ling Liu, Kisung Lee, Calton Pu, and Qi Zhang. 201 Policy-driven autonomic configu- ration management for nosql. In ACM Symposium on High- Peroorrerfe Parallel and Distributed Com- puting (HPPD)), pages 179-190.
Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney, 2020. Pyhessian: Neural networks through the lens of the hessian. In IEEE Ins. Conf. on Big Data (Big Data), pages 181-590. IEEE Computer Society.
Yang Zhou, Ling Liu, Kisung Lee, and Qi Zhang. 2015b. Graphtwist: Fast iterative graph computation with two-tier optimizations. The VLDB Journal (VLDBJ).. 8(1 8(11):122-1227.
Honglin Yuan, Manzil Zaheer. and Sashank Reddi. 2021. Federated composite optimization. In Int. Conf. on Machine Learning (ICML), volume 139, pages 12253-12266.
Yang Zhou, Sangeetha Seshadri, Lawrence Chiu, and Ling Liu. 2014. Graphlens: Mining enterprise stor- age workloads using graph analytics. In IEEE Int. Congress on Big Data (BigData), pages 1-8.
Hong Zhang, Ji Liu, Juncheng Jia. Yang Zhou, Huaiyu Dai, and Dejing Dou. 2022. FedDUAP: Federated learning with dynamic update and adaptive pruning using shared data on the server. In Int. Joint Conf. on Artificial Intelligence (ICCAT), pages 2776-2782.
Qi Zhang. Ling Liu, Kisung Lee, Yang Zhou, Aameek Singh, Nagapramod Mandagere, Sandeep Gopisetty, and Gabriel Alatorre. 2014. Improving hadoop ser- vice provisioning in a geographically distributed cloud. In IEEE Int. Conf. on Cloud Computing (CLOUD), pages 432 439.
Qi Zhang, Ling Liu, Yi Ren, Kisung Lee. Yuzhe Tang. Xu Zhao, and Yang Zhou. 2013. Residency aware imter-vm communication in virtualized cloud: Perfor- mance measurement and analysis. IEEE Int. Conf. on Cloud Computing (CLOOD), pages 204-211.
Zeru Zhang, Jayin Jin, Zijie Zhang, Yang Zhou, Xin Zhao, Jiaxiang Ren, Ji Liu, Lingfei Wu, Ruoming Jin, and Dejing Dou. 2021. Validating the lottery ticket hypothesis with inertial manifold theory. Advances

---

A Implementation Details
and P-tuning v2, 5 for LPT and IDPG, 100 for AT- TEMPT, and 20 for Prompt tuning. FedPrompt. Prompt tuning, and ATTEMPT insert prompts be- fore the transformer layers. P-tuning v2 inserts prompts to the hidden states for all transformer layers. IDPG combines these two heuristics and inserts prompts to either the input or the hidden states of all layers. LPT searches for the single best layer by training all the possible positions for each layer. Similar to P-tuning v2, FedPepTAO inserts the hidden states for all transformer layers while the prompt parameters of properly selected layers are communicated between devices and the server.
A.1 Adam Update
The original Adam update (Kingma and Ba, 2015) is shown in Algorithm 3.
Algorithm 3 Adam update
Require:
w/-1: The prompt parameters of the model at
Iteration
m The 1st momentum vector at Iteration
t 1-1
r'=1: The 2s momentum vector at Iteration
t 2-1
B Extra Experiments
g¹: The gradients corresponding to w
81, 32: Decay rates for the momentum in
B.1 Epochs Required to Achieve the Target Accuracy
Adam
a:
The step size
We conducted experiments with the number of epochs required achieve the target accu- racy and the communication overhead to demon- strate the performance of FedPepTAO on the ROBERTaJARGG model and 10 tasks. Below are the average epochs required to achieve the target accuracy.
Ensure:
w/: The prompt parameters at Iteration t
m²: The 1s momentum vector at Iteration
u²: The 2 momentum vector at Iteration t -B)gg'
1: m² < 3im
2: m t m'
T-8
3: v² t 3zu--1
+1 (1-1300) (2)
4: 0r n²
1-37
Method
Epochs
5: w (- w
the
(V6C (c)
Adapter
30.92
FedPrompt
24.50
P-tuning v2
5.99
A.2 Details for Experimental Setup
Prompt Tuning
IDPG
40.15
The number of global training epochs is set to 100 and that of local training epochs is set to 2. We utilize the Dirichlet distribution (with 1.0 as the concentration parameter alpha) to partition the data into non-IID splits and assign a certain number of samples to each device according to the Dirichlet distribution (with 5.0 as the concentration param- eter alpha). We exploit development sets for the evaluation of tasks in the GLUE benchmark since test sets are not labeled. For 4 other datasets, se- lect certain number of samples from the training set as the development set, and the number sam- ples for each label is determined according to its proportion in the original training set. For datasets in GLUE benchmark (Wang et al., 2019), we use their original data splits. For 4 other datasets with no default splits, we randomly divide the dataset into train, development, and test sets. The dataset statistics after the split are shown in Table 13
ATTEMPT
39.97
I.T
8.31
MomD
13.98
MomS+AdamD
5.99
FedPepTAO 3.88
Table The average number of epochs required to achieve the target accuracy (85% for QNLI, 92.5% for SST-2, 3% for CoLA, 77 % for MRPC, 65% for RTE, 71% for BoolQ, 85% for MPQA, 88% Subj, 78% for Trec, 11% for MR) on RoBERTaLARGE model. "/" represents that training does not achieve the target accu- racy.
From Table 6, we find that FedPepTAA requires the smallest amount of epochs to achieve the tar- get accuracy (87 45.45 84.16%, 35.26%, 90.33%, 90.29%, 53.31%, 72.22%, 35.21% % faster compared to Adapter, FedPrompt, P-tuning v2, IDPG, AT- TEMPT, LPT. MomD, and MomS+AdamD respec- tively). Prompt Tuning failed to achieve the tar- get accuracy since it only optimizes the first layer of soft prompts. FedPepTAO can also reduce the
We set prompt lengths for each method accord- ing to the original works, i.e., 128 for FedPrompt

---

communication overhead from 40% 41.55% (41.55%, 40%, 40%, 40% compared with Adapter, P-tuning v2, MomD, and MomS+AdamD) respec- tively) between devices and the server, as illustrated in Table 7. FedPrompt, Crompt-Tuning IDPG, AT- TEMPT, and LPT correspond to lower communi- cation overhead (up to 13%) since they only select one layer during tuning, which results in signifi- cantly inferior accuracy (from 17.02% 60.8% lower) compared with FedPepTAO.
B.3 Impact of heterogeneous data distribution To investigate the robustness of our approach to different degrees of non-III data, we conduct ex- periments under various levels of non-lID degrees. From Figure 3, we observe that the accuracy of FedPepTAO is the best with ranging from 1.0 to 5.0. A smaller represents higher degree of heterogeneity. This demonstrates that our approach robust to different data heterogeneity.
Method
Time
0.85
0.80
Adapter
5.80
FedPrompt
3,09
0.75
P-tuning v2
5.65
0.70
Prompt Tuning
3,00
IDPG
3,09
ATTEMPT
3.16
0.60
LPT
3,09
MomD
5.65
0.55
AdamD
5,65
0.50
FedPepTAO
3.39
60
80
1.00
Epoch
Table 7: The communication overhead (s) between de- vices and the server with RoBERTaLAKGE model.
Figure The impact of various heterogeneity degrees.
B.4 Impact of device number
B.2 Impact of server learning rate
To explore the scalability of our model, we conduct experiments with divers number of devices, i.e.. 100, 150, and 200. Figure 4 shows the correspond- ing accuracy on the RTE dataset. The accuracy gap between the best and worst is only 0.73%, which demonstrates that FedPepTAO is scalable FL settings.
Due to the high sensitivity of the hyperparame- ters NLP tasks, we investigate the impact of divers server learning rates on the RTE dataset. We analyze the accuracy with the learning rates 1e-2, 5e le-3, 5e-4. As shown in Figure 2, we find that the best performance was achieved when (r 1e which only slightly outperforms Ir= 5e 4 by 0.37%. This demonstrates that Fed- PepTAO is easy to fine-tune in practice.
0.20
0.75
0.85
0.65
0.55
0.55
0.50
20
Figure 2: The impact of server learning rate.
Figure 4: Evaluation of divers number of devices.

---

B.5
Impact of diverse bandwidth
Tables 10, 11, and 12 exhibit that FedPepTAO out- performs the random strategy by 2.91%, 4.64%, and 7.29% on RTE, MRPC, and CoLA datasets, respectively. The above experiment results indi- cate that our FedPepTTA method can achieve sub- stantial improvement compared with the random strategy.
We carry out extra experimentation on two tasks, i.c. Subj and Trec with modest network band- width (reduced to 100 times smaller) on the RoBERTaLARGE model. We find that FedPep- TAO maintains its advantages in this setting, i.e., up to 98.71%, 94.75%. 80.61%, 95%, 97.43%, 56.52%, 84.55% faster compared with Adapter, FedPrompt, P-tuning v2. IDPG, ATTEMPT, LPT. MomS+Adaml to achieve the target accuracy, as shown in Table 8.
Seed 32
Seed 35 Acc
Seed 37 Acc
Method
Acc
FedPepTTAO
84.84%
85.56%
85, 85.92%
Random 1
82.31%
82.31%
82.31%
Random 2
81.59%
81.23%
83.39%
Method
Subj
Tree
Random 3
82.67%
83.03%
83.75%
Avg Acc Gain
2.65%
3.37%
2.77%
Adapter
3591
7908
FedPrompt
1333
368
P-tuning v2
361
474
Table 10:
The performance of FedPepTTAO and random
Promtp Tuning
/
layer choosing strategy on RTE dataset under different random seeds
IDPG
1401
1345
ATTEMPT
2726
1184
LPT
161
123
MomD
/
/
AddS+D
453
496
Seed 32
Seed Acc
Seed 37
FedPepTAO
70
102
Method
Acc
FedPepTAAO
86.54%
87.09%
86.18%
Table The tuning time (s) to achieve target accuracy (88% for Subj, 78% for Trec) on RoBERTa ARGE model. "/" represents that training does not achieve the target accuracy.
Random 1
83,09%
81.86%
80.34%
Random 2
82.73%
82.03%
80.54%
Random
83.31%
81.83%
81.16%
Avg Acc Gain
3.5%
5.18%
5.5%
B.6 Parameter-efffcieet Prompt Tuning and random layer selection strategy
Table 11:
The performance of FedPepTAAO and random
layer choosing strategy MRPC dataset under differ- ent random seeds
In order clarify the impact of randomness in our experiments, we conduct three experiments with random layer selection strategy on RTE dataset. As shown in Table 9, FedPepTAAO outperforms the random strategy with the accuracy gain of 2.53%, 2.89%, and 3.25% respectively, which demon- strates the superior performance of our FedPepTAO method.
Seed 32
Seed 35
Seed 37
Method
Acc
Ace
Ace
FedPepTAA
56.94%
58.92%
56.49%
Random
46.45
52.84%
49.58%
Random 2
51.57%
49.84%
45.71%
Random 3
50.77%
53.2%
51.45%
Avg Ace Gain
7.34%
6.96%
7.58%
Method
Seed Acc
Table 12:
The performance of FedPepTTAO and random
FedPepTAAO
85.56%
layer choosing strategy on CoLA dataset under different random seeds
Random 1
83.03%
Random 2
82.67%
Random 3
12.31%
We notice an inverse correlation between the per- formance of our Parameter-cfficient Prompt Tuning (PEPT) method and the average sentence length of the three datasets. Specifically, PEPT tends to achieve a smaller performance gain on the datasets with longer average sentence length, as shown in Table 14.
Avg gain 2.89%
Table 9: The performance of FedPepTAA and random layer choosing strategy RTE dataset.
In addition, in order to further validate the im- pact of randomness on different datasets, we con- ducted additional experiments on three datasets (RTE, MRPC, and CoLAA) with three randomly se- lected seeds (32, 35, and 37) to testify the strength our snrameter-efficieen Prompt Tuning method.
A reasonable explanation is that the datasets with longer average sentence length, such as RTE, often contain more latent information. More/less latent information make them easier/more difficult to be

---

Category
Datasets
Train
Dev
Test
D
Type
Labels
SST-2
67349
872
1821
2
sentiment
positive, negative posilive, negative positive, negative subjective. objective
MPQA
7606
2000
2
opinion polarity
Single-senlence
MR
7662
1000
2000
2
sentiment
Subj
7000
1000
2000
2
subjectivity question cls. acceptability
Trec
4952
500
500
6
abbr. entity, description, human. loc., num. acceptable, unacceptable
CoLA
8551
1043
1063
2
MRPC
408
1725
2
paraphrase
equivalent, equivalent
QNLI
104743
5463
5463
2
NLI
entailment, not entailment true, false
Sentence-pair
BoulQ
9427
3270
3245
2
QA
RTE
2490
277
3000
2
NLI
entailment, not entailment
Table 13: The statistics of datasets evaluated in this work. [>]] the number for classes.
Dataset
Avg sentence length
Avg acc gain
RTE
71.91
2.91% %
MRPC
54.95
4.64%
CoLA 16.37
7.29%
Table 14: Average sentence length and accuracy im-
mmporry ohe Min
provements of FedPepTAO for three datasets.
0.65
mg
evenly distributed across all transformer layers, re- sulting in relatively equal/diverse contribution by each layer during tuning. When different lay- ers contain the similar/dissimilar amount of latent information, the impact of each unique layer is accordingly decreased/increased. Therefore, the random layer selection results in less/more accu- racy gain by our PEPT method. The above exper- iment results demonstrate that our PEPT method achieve substantial performance improvement different datasets in most experiments.
0.55
0.50
40
60
100
1pach
Figure 6: Evaluation for various optimization methods.
0.85
0.80
0.70
0.65
0.55
0.80
100
Figure 5: Evaluation of divers layer selection strategies.


Reviews:
[{"review_id": "TUm8bbFAyR", "reviewer": "Reviewer_6pQ8", "summary": "", "questions": "A. Are all the 10 Tasks used in the evaluation, encoder only tasks? \n\nB. Is there a metric to compare the communication efficiency of this technique? (Other than the number of parameters?)\n\nC. How do you evaluate and compare the client drift problem?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for valuable comments. We use RR.n and Q.n to denote the response to “Reasons To Reject,” or ``Questions For The Authors” n. For the reviewers’ convenience, we retype the review comments and suggestions in bold followed by our responses. The response will be clarified in the revised paper.\n\n**RR.1: Although the approach is generic enough, I did not see the results of applying this approach to decoder models in the paper. All the results seemed to use RoBERT-Large and would have liked to see other variations like encoder-decoder and decoder only models as well.**\n\nRR.1 Response: Thanks for the valuable comment. To demonstrate the adaptability of FedPepTAO, we carried out extra experiments with three additional decoder-based models, i.e., GPT-2 model (774M) on MRPC, MR, SST-2 dataset, LLaMA 3B model on RTE, MRPC dataset, and LLaMA 7B model on MRPC dataset.\n\nAs shown in Table 1 below, FedPepTAO significantly outperforms baseline methods in terms of the best accuracy on the decoder-based GPT-2 model (up to 32.3\\%, 18.23\\%, 43.32\\%, 15.94\\%, 4\\%, 4.34\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively). Furthermore, the efficiency of FedPepTAO is significantly higher than baseline approaches (up to 84.74\\%, 86.89\\%, and 15.27\\% faster compared to LPT, MomD, and MomS+AdamD, respectively).\n\nTable 1. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC, 81% for MR, and 92.5% for SST-2) on GPT-2 model. \"/\" represents that training does not achieve the target accuracy.\n| Method          | MRPC Acc | MRPC Time | MR Acc | MR Time | SST-2 Acc | SST-2 Time |\n|-----------------|---------|-----------|--------|---------|-----------|------------|\n| FedPrompt       | 74.98   |     /     | 57.2   |    /    | 76.49     |     /      |\n| P-tuning v2     | 74.8    |     /     | 73     |    /    | 74.77     |     /      |\n| ATTEMPT         | 37.91   |     /     | 57.3   |    /    | 85.89     |     /      |\n| LPT             | 74.98   |     /     | 84.8   |  1455   | 77.06     |     /      |\n| MomD            | 77.23   |   503     | 85.7   |  1694   | 92.55     |    3638    |\n| MomS+AdamD      | 76.89   |   291     | 88.2   |   262   | 92.55     |    2488    |\n| FedPepTAO       | 81.23   |   273     | 89.5   |   222   | 93        |    2248    |\n\nWhen the model becomes larger, i.e., LLaMA 3B with 3 billion parameters, FedPepTAO still achieves the best accuracy (up to 4.6\\%, 11.29\\%, 5.28\\%, 6.69\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and better efficiency (up to 39.81\\%, 43.04\\%, 48.16\\%, 38.86\\%, 9.72\\% faster compared to FedPrompt, ATTEMPT, LPT, MomD, and MomS+AdamD, respectively) as illustrated in Table 2.\n\nTable 2. Accuracy and tuning time (s) to achieve target accuracy (75% for RTE, 81% for MRPC) on LLaMA 3B model. \"/\" represents that training does not achieve the target accuracy.\n| Method        | RTE Acc | RTE Time | MRPC Acc | MRPC Time |\n|---------------|---------|----------|----------|----------|\n| FedPrompt     | 78.34   | 540      | 81.86    | 459      |\n| P-tuning v2   | 56.68   |   /      | 75.17    |   /      |\n| ATTEMPT       | 64.98   |   /      | 81.18    | 718      |\n| LPT           | 64.98   |   /      | 79.77    | 789      |\n| MomD          | 80.87   | 360      | 80.26    | 669      |\n| MomS+AdamD    | 80.87   | 360      | 75.58    |   /      |\n| FedPepTAO     | 83.39   | 325      | 86.46    | 409      |\n\n\nWe further verify the performance of our method with another large model, i.e., LLaMA 7B with 7 billion parameters. As presented in Table 3, FedPepTAO outperforms baseline methods in terms of accuracy (up to 5.05\\%, 26.71\\%, 18.41\\%, 18.41\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and efficiency (up to 15.77\\%, 75.14\\%, 32.06\\%, 67.04\\% faster compared to ATTEMPT, LPT, MomD, and MomS+AdamD, respectively), which demonstrates the scalability of FedPepTAO.\n\nTable 3. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC) on LLaMA 7B model. \"/\" represents that training does not achieve the target accuracy.\n| Method       | MRPC Acc | MRPC Time |\n|--------------|----------|-----------|\n| FedPrompt    | 76.18    |    /      |\n| P-tuning v2  | 74.98    |    /      |\n| ATTEMPT      | 81.52    |   317     |\n| LPT          | 81.95    |  1074     |\n| MomD         | 76.2     |   393     |\n| MomS+AdamD   | 75.75    |   810     |\n| FedPepTAO    | 82.34    |   267     |\n\nThis will be clarified in the revised version.\n\n**Q.A. Are all the 10 Tasks used in the evaluation, encoder only tasks?**\n\nQ.A. Response: Thanks for raising this straightforward question. In our experiment, we exploited the 10 tasks with an encoder-only model. However, the 10 tasks are not encoder-only tasks and can be performed with decoder models. We carried out extra experiments with three additional decoder models, i.e., GPT-2 model (774M) on MRPC, MR, SST-2 dataset, LLaMA 3B model on RTE, MRPC dataset, and LLaMA 7B model on MRPC dataset.\n\nAs shown in Table 1 below, FedPepTAO significantly outperforms baseline methods in terms of the best accuracy on the decoder-based GPT-2 model (up to 32.3\\%, 18.23\\%, 43.32\\%, 15.94\\%, 4\\%, 4.34\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively). Furthermore, the efficiency of FedPepTAO is significantly higher than baseline approaches (up to 84.74\\%, 86.89\\%, and 15.27\\% faster compared to LPT, MomD, and MomS+AdamD, respectively).\n\nTable 1. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC, 81% for MR, and 92.5% for SST-2) on GPT-2 model. \"/\" represents that training does not achieve the target accuracy.\n| Method          | MRPC Acc | MRPC Time | MR Acc | MR Time | SST-2 Acc | SST-2 Time |\n|-----------------|---------|-----------|--------|---------|-----------|------------|\n| FedPrompt       | 74.98   |     /     | 57.2   |    /    | 76.49     |     /      |\n| P-tuning v2     | 74.8    |     /     | 73     |    /    | 74.77     |     /      |\n| ATTEMPT         | 37.91   |     /     | 57.3   |    /    | 85.89     |     /      |\n| LPT             | 74.98   |     /     | 84.8   |  1455   | 77.06     |     /      |\n| MomD            | 77.23   |   503     | 85.7   |  1694   | 92.55     |    3638    |\n| MomS+AdamD      | 76.89   |   291     | 88.2   |   262   | 92.55     |    2488    |\n| FedPepTAO       | 81.23   |   273     | 89.5   |   222   | 93        |    2248    |\n\nWhen the model becomes larger, i.e., LLaMA 3B with 3 billion parameters, FedPepTAO still achieves the best accuracy (up to 4.6\\%, 11.29\\%, 5.28\\%, 6.69\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and better efficiency (up to 39.81\\%, 43.04\\%, 48.16\\%, 38.86\\%, 9.72\\% faster compared to FedPrompt, ATTEMPT, LPT, MomD, and MomS+AdamD, respectively) as illustrated in Table 2.\n\nTable 2. Accuracy and tuning time (s) to achieve target accuracy (75% for RTE, 81% for MRPC) on LLaMA 3B model. \"/\" represents that training does not achieve the target accuracy.\n| Method        | RTE Acc | RTE Time | MRPC Acc | MRPC Time |\n|---------------|---------|----------|----------|----------|\n| FedPrompt     | 78.34   | 540      | 81.86    | 459      |\n| P-tuning v2   | 56.68   |   /      | 75.17    |   /      |\n| ATTEMPT       | 64.98   |   /      | 81.18    | 718      |\n| LPT           | 64.98   |   /      | 79.77    | 789      |\n| MomD          | 80.87   | 360      | 80.26    | 669      |\n| MomS+AdamD    | 80.87   | 360      | 75.58    |   /      |\n| FedPepTAO     | 83.39   | 325      | 86.46    | 409      |\n\n\nWe further verify the performance of our method with another large model, i.e., LLaMA 7B with 7 billion parameters. As presented in Table 3, FedPepTAO outperforms baseline methods in terms of accuracy (up to 5.05\\%, 26.71\\%, 18.41\\%, 18.41\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and efficiency (up to 15.77\\%, 75.14\\%, 32.06\\%, 67.04\\% faster compared to ATTEMPT, LPT, MomD, and MomS+AdamD, respectively), which demonstrates the scalability of FedPepTAO.\n\nTable 3. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC) on LLaMA 7B model. \"/\" represents that training does not achieve the target accuracy.\n| Method       | MRPC Acc | MRPC Time |\n|--------------|----------|-----------|\n| FedPrompt    | 76.18    |    /      |\n| P-tuning v2  | 74.98    |    /      |\n| ATTEMPT      | 81.52    |   317     |\n| LPT          | 81.95    |  1074     |\n| MomD         | 76.2     |   393     |\n| MomS+AdamD   | 75.75    |   810     |\n| FedPepTAO    | 82.34    |   267     |\n\nThis will be clarified in the revised version.\n\n**Q.B. Is there a metric to compare the communication efficiency of this technique? (Other than the number of parameters?)**\n\nQ.B. Response: Thanks for the valuable comment. We utilized the tuning time to achieve target accuracy as a metric. As shown in Table 2 of the paper, FedPepTAO can outperform baseline approaches from 80.48\\% to 99\\% (95.91\\%, 95.17\\%, 92.76\\%, 99\\%, 97.28\\%, 97.59\\%, 85.8\\%, 94.23\\%, 80.48\\% compared with Adapter, FedPrompt, P-tuning v2, Ptompt Tuning, IDPG, ATTEMPT, LPT, MomD, and MomS+AdamD, respectively). In addition, we measured the time to transfer the parameters between devices and the server:\n\nTable 4. The communication overhead (s) between devices and the server with RoBERTA-Large model.\n| Method       | Time |\n|--------------|-------|\n| Adapter      | 5.80  |\n| FedPrompt    | 3.09  |\n| P-tuning v2  | 5.65  |\n| Prompt Tuning| 3.00  |\n| IDPG         | 3.09  |\n| ATTEMPT      | 3.16  |\n| LPT          | 3.09  |\n| MomD         | 5.65  |\n| MomS+AdamD   | 5.65  |\n| FedPepTAO    | 3.39  |\n\nFrom Table 4, we find that FedPepTAO can significantly reduce the communication time from 40\\% to 41.55\\% (41.55\\%, 40\\%, 40\\%, 40\\% compared with Adapter, P-tuning v2, MomD, and MomS+AdamD, respectively) between devices and the server thanks to our parameter-efficient prompt tuning method. The communication overhead of FedPrompt, Prompt Tuning, IDPG, ATTEMPT, and LPT are slightly lower (up to 13\\%), but they optimize only one prompt layer among all the layers, resulting in a significantly lower accuracy (from 17.02\\% to 60.8\\% lower) compared with FedPepTAO. This will be clarified in the revised version.\n\n**Q.C. How do you evaluate and compare the client drift problem?**\n\nQ.C. Response: Thank you for the insightful question. As a distributed optimization method, FL has a key challenge: the heterogeneity (non-iid-ness) in the data present on the diverse clients. Previous research [1] has shown that such heterogeneity introduces a drift in the updates of each client, i.e., client drift, resulting in inferior and unstable convergence. Hence, we evaluate the client drift problem based on the final accuracy with diverse approaches. As shown in Table 1 of the paper, FedPepTAO can outperform baseline approaches from 13.39\\% to 60.8\\% (25.39\\%, 23.83\\%, 14.53\\%, 60.8\\%, 51.76\\%, 51.72\\%, 17.02\\%, 54.71\\%, 13.39\\% compared with Adapter, FedPrompt, P-tuning v2, Prompt Tuning, IDPG, ATTEMPT, LPT, MomD, and MomS+AdamD, respectively). In addition, we compare the single module of our communication-efficient adaptive optimization with other adaptive optimization baseline approaches without states for each device, which demonstrates significant advantages (up to 29.24\\% compared with FedAvg, 8.3\\% compared with MomD, 29.24\\% compared with MomS, 29.24\\% compared with MomS+Con, 2.16\\% compared with AdamD, 5.05\\% compared with MomS+AdamD higher in terms of accuracy) of our method, as shown in Figure 4 of the paper. Furthermore, we carried out extra experimentation to show that the value of the state ($c_i^r$) on each device is 35.91\\% (on average) compared with that of the momentum ($m_i^r$), which reveals that the state for each device is effective while addressing the client drift problem. This will be clarified in the revised version.\n\n[1] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020b. Scaffold: Stochastic controlled averaging for federated learning. Int. Conf. on Machine Learning (ICML), pages 5132–5143. PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work presented has a strong and important motivation to improve the efficiency and reduce the communication and computation costs in FL of LLMs. It provides an initial framework to combine ideas from efficient fine tuning strategies (like prompt-tuning and prefix-tuning) with Federated Learning. The approach taken to achieve this is novel and architecture agnostic, so the ideas are scalable to FL of any model architectures.", "reasons_to_reject": "Although the approach is generic enough, I did not see the results of applying this approach to decoder models in the paper. All the results seemed to use RoBERT-Large and would have liked to see other variations like encoder-decoder and decoder only models as well.", "questions_to_authors": "A. Are all the 10 Tasks used in the evaluation, encoder only tasks? \n\nB. Is there a metric to compare the communication efficiency of this technique? (Other than the number of parameters?)\n\nC. How do you evaluate and compare the client drift problem?", "paper_topic_and_main_contributions": "The authors present an efficient prompt tuning strategy for Federated Learning of Large Language Models. The idea is to select (based on a scoring function) a subset of layers (instead of full set of prompt parameters) and exchange the prompt parameters of these layers during the tuning process. In addition to this, the authors present a communication efficient strategy to update the gradients on both server and device.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YFMMSHmewY", "reviewer": "Reviewer_nVG8", "summary": "", "questions": " Table 2, to reach a target acc, reviewer wanna know how those acc are set as target? We know the time to achieve an acc may be vary for different factors. Reviewer believe use a epoch and communication overhead will be more reasonable. How many times did the authors did this experiment? How do authors get those number, the best runtime? Avg runtime? \n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for valuable comments. We use RR.n and Q.n to denote the response to “Reasons To Reject,” or ``Questions For The Authors” n. For the reviewers’ convenience, we retype the review comments and suggestions in bold followed by our responses. The response will be clarified in the revised paper.\n\n**RR.1: Figure 1 is hard to follow. There is no caption for Figure 1, really can not understand difference between two step 5 and two step 1.**\n\nRR.1 Response: Thanks for the valuable suggestion. We apologize for the confusion. We should have put Step 5 below Step 3. In Step 1, the devices receive the corresponding updated prompt parameters of specific layers from the server. In Step 5, the prompt parameters of specific layers are sent back to the server. \n\nWithin traditional federated learning, e.g., FedAvg, each round consists of 5 steps. First, the server randomly selects a set of devices. Second, the server broadcasts the models to selected devices. Third, the selected devices perform local model updates. Fourth, the selected devices upload the updated local models. Fifth, the server performs the aggregation of received models. Step 1 in FedPepTAO is similar to the second step of FedAvg and Step 5 in FedPepTAO is similar to the fourth step of FedAvg. This will be clarified in the revised version.\n\n**RR.2 + Q: Table 2, to reach a target acc, reviewer wanna know how those acc are set as target? We know the time to achieve an acc may be vary for different factors. Reviewer believe use a epoch and communication overhead will be more reasonable. How many times did the authors did this experiment? How do authors get those number, the best runtime? Avg runtime?**\n\nRR.2 + Q Response: Thanks for the thoughtful comment. We select the highest target accuracy that at least 5 baseline approaches can achieve. We repeated the experiment with three random seeds (32, 42, 52) and calculated the average best accuracy, e.g., 22.38\\%, 7.22\\%, 24.07\\%, 5.68\\%, 13.84\\%, 1.32\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, with RoBERTa-Large model on RTE dataset.\n\n\nWe conducted experiments with the number of epochs required to achieve the target accuracy and the communication overhead to demonstrate the performance of FedPepTAO on the RoBERTA-Large model and 10 tasks. Below are the average epochs required to achieve the target accuracy.\n\nTable 1. The average number of epochs required to achieve the target accuracy (85% for QNLI, 92.5% for SST-2, 3% for CoLA, 77% for MRPC, 65% for RTE, 71% for BoolQ, 85% for MPQA, 88% for Subj, 78% for Trec, 91% for MR). \"/\" represents that training does not achieve the target accuracy.\n| Method        | Epochs      |\n|---------------|-------------|\n| Adapter       | 30.92       |\n| FedPrompt     | 24.50       |\n| P-tuning v2   | 5.99        |\n| Prompt Tuning | /           |\n| IDPG          | 40.15       |\n| ATTEMPT       | 39.97       |\n| LPT           | 8.31        |\n| MomD          | 13.98       |\n| MomS+AdamD    | 5.99        |\n| FedPepTAO     | 3.88        |\n\n\nFrom Table 1, we find that FedPepTAO requires the smallest amount of epochs to achieve the target accuracy (87.45\\%, 84.16\\%, 35.26\\%, 90.33\\%, 90.29\\%, 53.31\\%, 72.22\\%, 35.21\\% faster compared to Adapter, FedPrompt, P-tuning v2, IDPG, ATTEMPT, LPT, MomD, and MomS+AdamD respectively). Prompt Tuning failed to achieve the target accuracy since it only optimizes the first layer of soft prompts. FedPepTAO can also reduce the communication overhead from 40\\% to 41.55\\% (41.55\\%, 40\\%, 40\\%, 40\\% compared with Adapter, P-tuning v2, MomD, and MomS+AdamD, respectively) between devices and the server, as illustrated in Table 2. FedPrompt, Prompt-Tuning, IDPG, ATTEMPT, and LPT correspond to lower communication overhead (up to 13\\%) since they only select one layer during tuning, which results in significantly inferior accuracy (from 17.02\\% to 60.8\\% lower) compared with FedPepTAO. This will be clarified in the revised version.\n\nTable 2. The communication overhead (s) between devices and the server with RoBERTA-Large model.\n| Method       | Time  |\n|--------------|-------|\n| Adapter      | 5.80  |\n| FedPrompt    | 3.09  |\n| P-tuning v2  | 5.65  |\n| Prompt Tuning| 3.00  |\n| IDPG         | 3.09  |\n| ATTEMPT      | 3.16  |\n| LPT          | 3.09  |\n| MomD         | 5.65  |\n| MomS+AdamD   | 5.65  |\n| FedPepTAO    | 3.39  |\n\n**RR.3: They only has one model, which sounds not sufficient.**\n\nRR.3 Response: Thanks for your helpful comment. To demonstrate the adaptability of FedPepTAO, we carried out extra experiments with three additional decoder-based models, i.e., GPT-2 model (774M) on MRPC, MR, SST-2 dataset, LLaMA 3B model on RTE, MRPC dataset, and LLaMA 7B model on MRPC dataset. As shown in Table 3 below, FedPepTAO significantly outperforms baseline methods in terms of the best accuracy on the decoder-based GPT-2 model (up to 32.3\\%, 18.23\\%, 43.32\\%, 15.94\\%, 4\\%, 4.34\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively). Furthermore, the efficiency of FedPepTAO is significantly higher than baseline approaches (up to 84.74\\%, 86.89\\%, and 15.27\\% faster compared to LPT, MomD, and MomS+AdamD, respectively).\n\nTable 3. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC, 81% for MR, and 92.5% for SST-2) on GPT-2 model. \"/\" represents that training does not achieve the target accuracy.\n| Method          | MRPC Acc | MRPC Time | MR Acc | MR Time | SST-2 Acc | SST-2 Time |\n|-----------------|---------|-----------|--------|---------|-----------|------------|\n| FedPrompt       | 74.98   |     /     | 57.2   |    /    | 76.49     |     /      |\n| P-tuning v2     | 74.8    |     /     | 73     |    /    | 74.77     |     /      |\n| ATTEMPT         | 37.91   |     /     | 57.3   |    /    | 85.89     |     /      |\n| LPT             | 74.98   |     /     | 84.8   |  1455   | 77.06     |     /      |\n| MomD            | 77.23   |   503     | 85.7   |  1694   | 92.55     |    3638    |\n| MomS+AdamD      | 76.89   |   291     | 88.2   |   262   | 92.55     |    2488    |\n| FedPepTAO       | 81.23   |   273     | 89.5   |   222   | 93        |    2248    |\n\nWhen the model becomes larger, i.e., LLaMA 3B with 3 billion parameters, FedPepTAO still achieves the best accuracy (up to 4.6\\%, 11.29\\%, 5.28\\%, 6.69\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and better efficiency (up to 39.81\\%, 43.04\\%, 48.16\\%, 38.86\\%, 9.72\\% faster compared to FedPrompt, ATTEMPT, LPT, MomD, and MomS+AdamD, respectively) as illustrated in Table 4.\n\nTable 4. Accuracy and tuning time (s) to achieve target accuracy (75% for RTE, 81% for MRPC) on LLaMA 3B model. \"/\" represents that training does not achieve the target accuracy.\n| Method        | RTE Acc | RTE Time | MRPC Acc | MRPC Time |\n|---------------|---------|----------|----------|----------|\n| FedPrompt     | 78.34   | 540      | 81.86    | 459      |\n| P-tuning v2   | 56.68   |   /      | 75.17    |   /      |\n| ATTEMPT       | 64.98   |   /      | 81.18    | 718      |\n| LPT           | 64.98   |   /      | 79.77    | 789      |\n| MomD          | 80.87   | 360      | 80.26    | 669      |\n| MomS+AdamD    | 80.87   | 360      | 75.58    |   /      |\n| FedPepTAO     | 83.39   | 325      | 86.46    | 409      |\n\n\nWe further verify the performance of our method with another large model, i.e., LLaMA 7B with 7 billion parameters. As presented in Table 5, FedPepTAO outperforms baseline methods in terms of accuracy (up to 5.05\\%, 26.71\\%, 18.41\\%, 18.41\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and efficiency (up to 15.77\\%, 75.14\\%, 32.06\\%, 67.04\\% faster compared to ATTEMPT, LPT, MomD, and MomS+AdamD, respectively), which demonstrates the scalability of FedPepTAO.\n\nTable 5. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC) on LLaMA 7B model. \"/\" represents that training does not achieve the target accuracy.\n| Method       | MRPC Acc | MRPC Time |\n|--------------|----------|-----------|\n| FedPrompt    | 76.18    |    /      |\n| P-tuning v2  | 74.98    |    /      |\n| ATTEMPT      | 81.52    |   317     |\n| LPT          | 81.95    |  1074     |\n| MomD         | 76.2     |   393     |\n| MomS+AdamD   | 75.75    |   810     |\n| FedPepTAO    | 82.34    |   267     |\n\nThis will be clarified in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Experimental results seems good.\n2. This paper uses matrix decomposition ", "reasons_to_reject": "1. Figure 1 is hard to follow. There is no caption for Figure 1, really can not understand difference between two step 5 and two step 1.\n2. Table 2, to reach a target acc, reviewer wanna know how those acc are set as target? We know the time to achieve an acc may be vary for different factors. Reviewer believe use a epoch and communication overhead will be more reasonable. How many times did the authors did this experiment? How do authors get those number, the best runtime? Avg runtime? \n3. They only has one model, which sounds not sufficient.", "questions_to_authors": " Table 2, to reach a target acc, reviewer wanna know how those acc are set as target? We know the time to achieve an acc may be vary for different factors. Reviewer believe use a epoch and communication overhead will be more reasonable. How many times did the authors did this experiment? How do authors get those number, the best runtime? Avg runtime? \n", "paper_topic_and_main_contributions": "This paper proposes parameter-efficient and effective federated learning (FL) of Large language models (LLM). First, an efficient partial prompt tuning approach is proposed to improve performer CSE and efficiency simultaneously. Second a novel adaptive optimization method is developed to address the client drift problems. The exp shows performance improvements for both accuracy and efficiency.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "No severe grammar issue, but writing is hard to follow.", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "xK8k5f77Qt", "reviewer": "Reviewer_ZSxS", "summary": "", "questions": "\nA: Can you give the logic behind calculating the Hessian Matrix? Are there any alternative approaches? \nB: Assuming lines 3 - 14 happen in parallel, during the aggregation stage what is the expected communication overhead? You mentioned that aggregating prompt parameter ratio, but this is not defined as far as I can see. I assume this is something negligible.  \nC: Can you explain how maintaining a state for each device on the server avoids possible drift problem? Do you have any empirical data on how effective this is? \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for valuable comments. We use RR.n and Q.n to denote the response to “Reasons To Reject,” or ``Questions For The Authors” n. For the reviewers’ convenience, we retype the review comments and suggestions in bold followed by our responses. The response will be clarified in the revised paper.\n\n**RR.1: My biggest concern is how the approach scale with larger models like GLM XLarge GLM XXLarge. I can understand the limitations on the experimental side, however, it would be great to hear from authors regarding how the performance when the model gets larger. Is this approach still feasible? What are some disadvantages that you foresee?**\n\nRR.1 Response: Thanks for the valuable comment. This approach is still feasible. FedPepTAO can reduce the number of parameters to transfer with our parameter-efficient prompt tuning method and perform adaptive optimization without extra communication costs, leading to better efficiency. \n \nTo demonstrate the scalability of FedPepTAO, we carried out extra experiments with three additional decoder-based models, i.e., GPT-2 model (774M) on MRPC, MR, SST-2 dataset, LLaMA 3B model on RTE, MRPC dataset, and LLaMA 7B model on MRPC dataset. As shown in Table 1 below, FedPepTAO significantly outperforms baseline methods in terms of the best accuracy on the decoder-based GPT-2 model (up to 32.3\\%, 18.23\\%, 43.32\\%, 15.94\\%, 4\\%, 4.34\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively). Furthermore, the efficiency of FedPepTAO is significantly higher than baseline approaches (up to 84.74\\%, 86.89\\%, and 15.27\\% faster compared to LPT, MomD, and MomS+AdamD, respectively).\n\nTable 1. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC, 81% for MR, and 92.5% for SST-2) on GPT-2 model. \"/\" represents that training does not achieve the target accuracy.\n| Method          | MRPC Acc | MRPC Time | MR Acc | MR Time | SST-2 Acc | SST-2 Time |\n|-----------------|---------|-----------|--------|---------|-----------|------------|\n| FedPrompt       | 74.98   |     /     | 57.2   |    /    | 76.49     |     /      |\n| P-tuning v2     | 74.8    |     /     | 73     |    /    | 74.77     |     /      |\n| ATTEMPT         | 37.91   |     /     | 57.3   |    /    | 85.89     |     /      |\n| LPT             | 74.98   |     /     | 84.8   |  1455   | 77.06     |     /      |\n| MomD            | 77.23   |   503     | 85.7   |  1694   | 92.55     |    3638    |\n| MomS+AdamD      | 76.89   |   291     | 88.2   |   262   | 92.55     |    2488    |\n| FedPepTAO       | 81.23   |   273     | 89.5   |   222   | 93        |    2248    |\n\nWhen the model becomes larger, i.e., LLaMA 3B with 3 billion parameters, FedPepTAO still achieves the best accuracy (up to 4.6\\%, 11.29\\%, 5.28\\%, 6.69\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and better efficiency (up to 39.81\\%, 43.04\\%, 48.16\\%, 38.86\\%, 9.72\\% faster compared to FedPrompt, ATTEMPT, LPT, MomD, and MomS+AdamD, respectively) as illustrated in Table 2.\n\nTable 2. Accuracy and tuning time (s) to achieve target accuracy (75% for RTE, 81% for MRPC) on LLaMA 3B model. \"/\" represents that training does not achieve the target accuracy.\n| Method        | RTE Acc | RTE Time | MRPC Acc | MRPC Time |\n|---------------|---------|----------|----------|----------|\n| FedPrompt     | 78.34   | 540      | 81.86    | 459      |\n| P-tuning v2   | 56.68   |   /      | 75.17    |   /      |\n| ATTEMPT       | 64.98   |   /      | 81.18    | 718      |\n| LPT           | 64.98   |   /      | 79.77    | 789      |\n| MomD          | 80.87   | 360      | 80.26    | 669      |\n| MomS+AdamD    | 80.87   | 360      | 75.58    |   /      |\n| FedPepTAO     | 83.39   | 325      | 86.46    | 409      |\n\n\nWe further verify the performance of our method with another large model, i.e., LLaMA 7B with 7 billion parameters. As presented in Table 3, FedPepTAO outperforms baseline methods in terms of accuracy (up to 5.05\\%, 26.71\\%, 18.41\\%, 18.41\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and efficiency (up to 15.77\\%, 75.14\\%, 32.06\\%, 67.04\\% faster compared to ATTEMPT, LPT, MomD, and MomS+AdamD, respectively), which demonstrates the scalability of FedPepTAO.\n\nTable 3. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC) on LLaMA 7B model. \"/\" represents that training does not achieve the target accuracy.\n| Method       | MRPC Acc | MRPC Time |\n|--------------|----------|-----------|\n| FedPrompt    | 76.18    |    /      |\n| P-tuning v2  | 74.98    |    /      |\n| ATTEMPT      | 81.52    |   317     |\n| LPT          | 81.95    |  1074     |\n| MomD         | 76.2     |   393     |\n| MomS+AdamD   | 75.75    |   810     |\n| FedPepTAO    | 82.34    |   267     |\n\nThis will be clarified in the revised version.\n\n**Q.A: Can you give the logic behind calculating the Hessian Matrix? Are there any alternative approaches?**\n\nQ.A Response: Thanks for the thoughtful question. We exploit an efficient PyHessian library [1] to calculate the eigenvalues in the Hessian Matrix. With the PyHessian library, we can calculate the trace and the Eigenvalue Spectral Density (ESD) of the Hessian matrix. With the trace and ESD, we can calculate each eigenvalue of the Hessian matrix. Within PyHessian, the trace is calculated using randomized numerical linear algebra with Hutchinson's method for fast computation with Hessian matvec computations (see [1] for details). In addition, PyHessian computes the ESD with 4 steps. First, it approximates the ESD with a Gaussian kernel. Then, it simplifies the problem with Gaussian quadrature. Afterward, it exploits the stochastic Lanczos algorithm to approximate the parameters in the ESD. Finally, it approximates the expectation with multiple times of the Lanczos algorithm execution. \n\nThere are other approaches to calculate the eigenvalues of the Hessian Matrix [4, 5], or ESD [2, 3, 6]. However, there is no open-source library to exploit for [2-6] while PyHessian can be directly utilized via [7]. The Hessian matrix can be directly calculated with TORCH.AUTOGRAD.FUNCTIONAL.HESSIAN [8] in PyTorch. However, the HESSIAN function in PyTorch (180.9 seconds) incurs significantly longer time (35 times longer) compared with PyHessian (4.8 seconds), based on a simple experiment with a deep neural network of three Multi-Layer Perceptron layers and 8090 neurons. In addition, the relative error of PyHessian is less than 5\\%, which is acceptable. Furthermore, the HESSIAN function in PyTorch (2.64GB) requires 4 times bigger memory compared with PyHessian (528MB). As a result, the HESSIAN function in PyTorch does not work in our case due to unaccepted execution time and out-of-memory issues with big models, and we utilize PyHessian. This will be clarified in the revised version.\n\n\n\n[1] Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney. 2020. Pyhessian: Neural networks through the lens of the hessian. In IEEE Int. Conf. on Big Data (Big Data), pages 581–590. IEEE Computer Society.\n\n[2] Lin Lin, Yousef Saad, and Chao Yang. Approximating spectral densities of large matrices. SIAM review,\n58(1):34–65, 2016.\n\n[3] Shashanka Ubaru, Jie Chen, and Yousef Saad. Fast estimation of tr(f(a)) via stochastic Lanczos quadrature. SIAM Journal on Matrix Analysis and Applications, 38(4):1075–1099, 2017.\n\n[4] Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the Hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016.\n\n[5] Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454, 2017.\n\n[6] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. arXiv preprint arXiv:1901.10159, 2019.\n\n[7] PyHessian: https://github.com/amirgholami/PyHessian\n\n[8] https://pytorch.org/docs/stable/generated/torch.autograd.functional.hessian.html\n\n**Q.B: Assuming lines 3 - 14 happen in parallel, during the aggregation stage what is the expected communication overhead? You mentioned that aggregating prompt parameter ratio, but this is not defined as far as I can see. I assume this is something negligible.**\n\nQ.B Response: Thanks for your helpful comment. Yes, the communication overhead is negligible. The communication overhead of the aggregation stage for each device is composed of two parts. The first part is the ratio between the selected layers and the whole layer set, which is a float value with 4 bytes. The second part is the score of each layer on each device, which is a set of $L$ float values with 4*L bytes. The total communication overhead is 4 *(L + 1) bytes, which are negligible (less than 1K bytes). The prompt parameter ratio refers to the value of $r_i$ in Line 10, which is the first part of the communication overhead. This will be clarified in the revised version.\n\n**Q.C: Can you explain how maintaining a state for each device on the server avoids possible drift problem? Do you have any empirical data on how effective this is?**\n\nQ.C Response: Thank you for the insightful question. FL as a distributed optimization method has a key challenge: the heterogeneity (non-iid-ness) in the data present on the different clients. Previous research has shown that such heterogeneity introduces a drift in the updates of each client, i.e., client drift, resulting in inferior and unstable convergence [1].\n\nWhen communication budget is not a concern, the ideal momentum update on device $i$ would be: \n\n$w_i^r = w^{r - 1} - \\eta^r * \\frac{1}{|M|} \\sum_j^{|M|} m_j^r$,          \n\nwhere $\\frac{1}{|M|} \\sum_j^{|M|} m_j^r$ represents an unbiased mini-batch gradient of $f$. This update is not feasible since it requires communication with all clients for every update step [2, 3]. FedPepTAO instead exploits a state for each device on the server such that \n\n$c_i^r \\approx m_i^r$, \n\n$c_{g}^r \\approx \\frac{1}{|M|} \\sum_j^{|M|} m_j^r$, \n\n$\\beta*m^{r-1}+(1-\\beta)*g_g^r \\approx \\frac{1}{|S|} \\sum_j^{|S|} m_j^r$,\n\nwhere FedPepTAO can mimic the ideal momentum update with the following formula: \n\n$m_i^r = \\beta*m^{r-1}+(1-\\beta)*g_g^r + c_{g}^r - c_i^r \\approx \\frac{1}{|S|} \\sum_j^{|S|} m_j^r + \\frac{1}{|M|} \\sum_j^{|M|} m_j^r - m_i^r \\approx \\frac{1}{|M|} \\sum_j^{|M|} m_j^r$. \n\nAnd, we have:\n\n$w_i^r = w^{r - 1} - \\eta^r * m_i^r = w^{r - 1} - \\eta^r * \\frac{1}{|M|} \\sum_j^{|M|} m_j^r$.\n\nThen, the momentum update of FedPepTAO for each device remains synchronized and converges for the ideal momentum update. In conclusion, we maintain states for each device to overcome gradient dissimilarity, and we maintain them on the server instead of the local devices to avoid additional communication overhead, which is crucial in LLM scenarios.\n\n[1] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020b. Scaffold: Stochastic controlled averaging for federated learning. Int. Conf. on Machine Learning (ICML), pages 5132–5143. PMLR. \n\n[2] Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. 2020a. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.03606.\n\n[3] Jiayin Jin, Jiaxiang Ren, Yang Zhou, Lingjuan Lyu, Ji Liu, and Dejing Dou. 2022. Accelerated federated learning with decoupled adaptive optimization. In Int. Conf. on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 10298–10322. PMLR.\n\nThe experimental results shown in Table 1 of the paper reveal that FedPepTAO achieves significantly higher accuracy (33.21\\% for RTE) compared with baseline approaches because of the state of each device with our adaptive optimization method. In addition, we compare the single module of our communication-efficient adaptive optimization with other adaptive optimization baseline approaches without states for each device, which demonstrate significant advantages (up to 29.24\\% compared with FedAvg, 8.3\\% compared with MomD, 29.24\\% compared with MomS, 29.24\\% compared with MomS+Con, 2.16\\% compared with AdamD, 5.05\\% compared with MomS+AdamD higher in terms of accuracy) of our method, as shown in Figure 4. Furthermore, we carried out extra experimentation to show that the value of the state ($c_i^r$) on each device is 35.91\\% (on average) compared with that of the momentum ($m_i^r$), which reveals that the state for each device is effective while addressing the client drift problem. This will be clarified in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I found the both contributions useful to the community. \n\n* Efficient Partial Prompt Tuning: The paper introduces an efficient partial prompt tuning approach that improves performance and efficiency simultaneously. This approach helps to reduce the number of parameters that need to be updated during FL, making it more feasible for LLMs.\n* Adaptive Optimization for Client Drift: A novel adaptive optimization method is proposed to address client drift problems on both the device and server sides, further enhancing the performance of FL for LLMs. This addresses the challenges posed by non-IID data distribution in decentralized FL settings.\n\n The authors approach is technically sound, and the description of the approach seems implementable. \n", "reasons_to_reject": "My biggest concern is how the approach scale with larger models like GLM XLarge GLM XXLarge. I can understand the limitations on the experimental side, however, it would be great to hear from authors regarding how the performance when the model gets larger. Is this approach still feasible? What are some disadvantages that you foresee? \n", "questions_to_authors": "\nA: Can you give the logic behind calculating the Hessian Matrix? Are there any alternative approaches? \nB: Assuming lines 3 - 14 happen in parallel, during the aggregation stage what is the expected communication overhead? You mentioned that aggregating prompt parameter ratio, but this is not defined as far as I can see. I assume this is something negligible.  \nC: Can you explain how maintaining a state for each device on the server avoids possible drift problem? Do you have any empirical data on how effective this is? \n", "paper_topic_and_main_contributions": "This paper introduces a novel approach called \"FedPepTAO\" that aims to enable efficient and effective federated learning of LLMs in scenarios where data is decentralized.  This paper addresses is the difficulty of applying federated learning to Large Language Models due to the large number of parameters that need to be updated during training. \n\nInitial solution that authors propose using prompt tuning to reduce the number of parameters, but this straightforward application of prompt tuning in FL leads to high communication costs and performance degradation. Additionally, the decentralized data in FL is often non-Independent and Identically Distributed (non-IID), leading to client drift problems and poor performance.\n\nI think it’s possible to summarize the paper contributions in 2 directions: Partial prompt tuning and Adaptive Optimization for Client Drift. \n\nThe authors also provided results on 10 datasets to evaluate the performance of FedPepTAO. The results demonstrate significant improvements in terms of accuracy (up to 60.8%) and training time efficiency (up to 97.59%) compared to 9 baseline approaches. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think FedAvg should be explained in the paper. It’s hard for the reader to follow up on that. \nI think paper reads well, so thanks for your efforts. \n"}, "extra_text": {}}, {"review_id": "GHwt6Vee1S", "reviewer": "Reviewer_tTk3", "summary": "", "questions": "1) Please provide an evaluation of runtime performance on different hardware environments, such as various network bandwidth. Because network communication and on-device computing are overlapping, the metric of communication parameters simply reflects network cost and does not show system efficiency. The tuning time is heavily dependent on network bandwidth.\n\n2)  what is the step size of learning rate from $1e^{-2}$ to $5e^{-4}$?\n\n3) no Appendix in this submission version.\n\n4) Authors utilize batch data on the hidden states of each layer to select the proper layers. If randomly choosing the same number of subset layers, removing the layer scoring stage, how about the performance of FedPepTAO?\n\n5) How much the component of communication adaptive optimization contributes in the final performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for valuable comments. We use RR.n and Q.n to denote the response to “Reasons To Reject,” or ``Questions For The Authors” n. For the reviewers’ convenience, we retype the review comments and suggestions in bold followed by our responses. The response will be clarified in the revised paper.\n\n**RR.1: (If the title includes the phrase \"Large Language Models\"), the experiments were carried out using the RoBERTa-large model with only 355M parameters, which is far from LLMs, which are typically at least 7B in size.**\n\nRR.1 Response: Thanks for the valuable comment. To demonstrate the adaptability of FedPepTAO, we carried out extra experiments with three additional decoder-based models, i.e., GPT-2 model (774M) on MRPC, MR, SST-2 dataset, LLaMA 3B model on RTE, MRPC dataset, and LLaMA 7B model on MRPC dataset.\n\nAs shown in Table 1 below, FedPepTAO significantly outperforms baseline methods in terms of the best accuracy on the decoder-based GPT-2 model (up to 32.3\\%, 18.23\\%, 43.32\\%, 15.94\\%, 4\\%, 4.34\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively). Furthermore, the efficiency of FedPepTAO is significantly higher than baseline approaches (up to 84.74\\%, 86.89\\%, and 15.27\\% faster compared to LPT, MomD, and MomS+AdamD, respectively).\n\nTable 1. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC, 81% for MR, and 92.5% for SST-2) on GPT-2 model. \"/\" represents that training does not achieve the target accuracy.\n| Method          | MRPC Acc | MRPC Time | MR Acc | MR Time | SST-2 Acc | SST-2 Time |\n|-----------------|---------|-----------|--------|---------|-----------|------------|\n| FedPrompt       | 74.98   |     /     | 57.2   |    /    | 76.49     |     /      |\n| P-tuning v2     | 74.8    |     /     | 73     |    /    | 74.77     |     /      |\n| ATTEMPT         | 37.91   |     /     | 57.3   |    /    | 85.89     |     /      |\n| LPT             | 74.98   |     /     | 84.8   |  1455   | 77.06     |     /      |\n| MomD            | 77.23   |   503     | 85.7   |  1694   | 92.55     |    3638    |\n| MomS+AdamD      | 76.89   |   291     | 88.2   |   262   | 92.55     |    2488    |\n| FedPepTAO       | 81.23   |   273     | 89.5   |   222   | 93        |    2248    |\n\nWhen the model becomes larger, i.e., LLaMA 3B with 3 billion parameters, FedPepTAO still achieves the best accuracy (up to 4.6\\%, 11.29\\%, 5.28\\%, 6.69\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and better efficiency (up to 39.81\\%, 43.04\\%, 48.16\\%, 38.86\\%, 9.72\\% faster compared to FedPrompt, ATTEMPT, LPT, MomD, and MomS+AdamD, respectively) as illustrated in Table 2.\n\nTable 2. Accuracy and tuning time (s) to achieve target accuracy (75% for RTE, 81% for MRPC) on LLaMA 3B model. \"/\" represents that training does not achieve the target accuracy.\n| Method        | RTE Acc | RTE Time | MRPC Acc | MRPC Time |\n|---------------|---------|----------|----------|----------|\n| FedPrompt     | 78.34   | 540      | 81.86    | 459      |\n| P-tuning v2   | 56.68   |   /      | 75.17    |   /      |\n| ATTEMPT       | 64.98   |   /      | 81.18    | 718      |\n| LPT           | 64.98   |   /      | 79.77    | 789      |\n| MomD          | 80.87   | 360      | 80.26    | 669      |\n| MomS+AdamD    | 80.87   | 360      | 75.58    |   /      |\n| FedPepTAO     | 83.39   | 325      | 86.46    | 409      |\n\n\nWe further verify the performance of our method with another large model, i.e., LLaMA 7B with 7 billion parameters. As presented in Table 3, FedPepTAO outperforms baseline methods in terms of accuracy (up to 5.05\\%, 26.71\\%, 18.41\\%, 18.41\\%, 6.2\\%, 10.88\\% higher compared to FedPrompt, P-tuning v2, ATTEMPT, LPT, MomD and MomS+AdamD, respectively) and efficiency (up to 15.77\\%, 75.14\\%, 32.06\\%, 67.04\\% faster compared to ATTEMPT, LPT, MomD, and MomS+AdamD, respectively), which demonstrates the scalability of FedPepTAO.\n\nTable 3. Accuracy and tuning time (s) to achieve target accuracy (75% for MRPC) on LLaMA 7B model. \"/\" represents that training does not achieve the target accuracy.\n| Method       | MRPC Acc | MRPC Time |\n|--------------|----------|-----------|\n| FedPrompt    | 76.18    |    /      |\n| P-tuning v2  | 74.98    |    /      |\n| ATTEMPT      | 81.52    |   317     |\n| LPT          | 81.95    |  1074     |\n| MomD         | 76.2     |   393     |\n| MomS+AdamD   | 75.75    |   810     |\n| FedPepTAO    | 82.34    |   267     |\n\nThis will be clarified in the revised version.\n\n**Q.1: Please provide an evaluation of runtime performance on different hardware environments, such as various network bandwidth. Because network communication and on-device computing are overlapping, the metric of communication parameters simply reflects network cost and does not show system efficiency. The tuning time is heavily dependent on network bandwidth.**\n\nQ.1 Response: Thanks for the thoughtful suggestion. We carried out extra experimentation on two tasks, i.e., Subj and Trec with modest network bandwidth (reduced to 100 times smaller) on RoBERTA-Large model. We find that FedPepTAO maintains its advantages in this setting, i.e., up to 98.71\\%, 94.75\\%, 80.61\\%, 95\\%, 97.43\\%, 56.52\\%, 84.55\\% faster compared with Adapter, FedPrompt, P-tuning v2, IDPG, ATTEMPT, LPT, MomS+AdamD  to achieve the target accuracy, as shown in Table 4 below. This will be clarified in the revised version.\n\nTable 4. The tuning time (s) to achieve a target accuracy (88% for Subj, 78% for Trec) on RoBERTA-Large model. \"/\" represents that training does not achieve the target accuracy.\n| Method        | Subj   | Trec   |\n|---------------|--------|--------|\n| Adapter       | 3591   | 7908   |\n| FedPrompt     | 1333   | 368    |\n| P-tuning v2    | 361    | 474    |\n| Promtp Tuning | /      | /     |\n| IDPG          | 1401   | 1345   |\n| ATTEMPT       | 2726   | 1184   |\n| LPT           | 161    | 123    |\n| MomD          | /      | /     |\n| MomS+AdamD    | 453    | 496    |\n| FedPepTAO    | 70     | 102    |\n\n**Q.2: What is the step size of learning rate from $1e^{-2}$ to $5e^{-4}$?**\n\nQ.2 Response: We apologize for not being specific in our evaluation. The step size of learning rate from $1e^{-2}$ to $5e^{-4}$ is $1e^{-2}$, $5e^{-3}$, $1e^{-3}$, and $5e^{-4}$, as illustrated in Figure 5 from Appendix. This will be clarified in the revised version.\n\n**Q.3: no Appendix in this submission version.**\n\nQ.3 Response: Thanks for the comment. The Appendix has been included in the supplementary materials upon the submission deadline in June. Please kindly unzip the supplementary file and find our Appendix. \n\n**Q.4: Authors utilize batch data on the hidden states of each layer to select the proper layers. If randomly choosing the same number of subset layers, removing the layer scoring stage, how about the performance of FedPepTAO?**\n\nQ.4 Response: Thanks for the thoughtful comment. We experimented with various layer selection methods as shown in Figure 2 of the Appendix. From Figure 2 of the paper, we can see that FedPepTAO corresponds to significantly higher accuracy compared with other methods (up to 1.81\\% compared with random, up to 1.81\\% compared with Descending, and 5.78\\% compared with Aescending). This will be clarified in the revised version.\n\n**Q.5: How much the component of communication adaptive optimization contributes in the final performance?**\n\nQ.5 Response: Thanks for the valuable question. As we explained in Section 4.4., the component of communication adaptive optimization can outperform baseline approaches from 2.16\\% to 29.24\\% (up to 8.3\\% compared with MomD, 29.24\\% compared with MomS, 29.24\\% compared with MomS+Con, 2.16\\% compared with AdamD, 5.05\\% compared with MomS+AdamD, and 29.24\\% compared with FedAvg). In addition, we conducted extra experiments to compare FedPepTAO and FedPepTAO-opt (FedPepTAO without the communication adaptive optimization), where FedPepTAO outperforms FedPepTAO-opt up to 4.69\\% in terms of accuracy, which demonstrates the contribution of communication adaptive optimization in the final performance. This will be clarified in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Authors propose a new Prompt Tuning, instead of tuning the full set of prompts parameters, selects a part of layers depending on their importance score for trading prompt parameters during tuning process.\n2) Authors use an adaptive optimization to capture non-IID data to avoid extra data transmission and reduce communication cost.", "reasons_to_reject": "1) (If the title includes the phrase \"Large Language Models\"), the experiments were carried out using the RoBERTa-large model with only 355M parameters, which is far from LLMs, which are typically at least 7B in size.", "questions_to_authors": "1) Please provide an evaluation of runtime performance on different hardware environments, such as various network bandwidth. Because network communication and on-device computing are overlapping, the metric of communication parameters simply reflects network cost and does not show system efficiency. The tuning time is heavily dependent on network bandwidth.\n\n2)  what is the step size of learning rate from $1e^{-2}$ to $5e^{-4}$?\n\n3) no Appendix in this submission version.\n\n4) Authors utilize batch data on the hidden states of each layer to select the proper layers. If randomly choosing the same number of subset layers, removing the layer scoring stage, how about the performance of FedPepTAO?\n\n5) How much the component of communication adaptive optimization contributes in the final performance?", "paper_topic_and_main_contributions": "Federated Learning (FL) is a novel idea for collaborative model training using decentralized data. However, the training process of Large Language Models (LLMs) often involves the updating of major parameters, limiting the application of FL approaches to LLMs in real-world scenarios. Prompt tuning could result in outstanding performance while freezing the original LLMs, via searching proper prompts in the discrete space.  However, existing approaches for fine-tuning LLMs significantly degrade performance due to the massive amount of communication.\nAuthors propose FedPepTAO, a Parameter-Efficient prompt Tuning technique with adaptive optimization, to enable efficient and effective FL of LLMs. First, an effective partial prompt tuning strategy is given to increase both performance and efficiency by selecting appropriate layers of prompts for FL. A subset of layers of prompts can minimize both communication and processing costs. Second, to improve performance even further, a novel adaptive optimization strategy is created to handle client drift issues on both the device and server sides. \nExperiments show that  FedPepTAO outperforms 9 baselines on 10 benchmark datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes techniques for improving the efficiency of prompt tuning in a federated learning setting. The core of the method is around selecting a subset of most impactful layers to update, and efficiently communicating the gradients. Reviewers found the proposed methods solid and novel, the experiments to be mostly convincing, and the problem and approach to be of high interest to the community. Most of the concerns were around the scope of experiments, which were mostly addressed by the new results provided during the rebuttal and will be added to the paper. I will only add that the evaluation can be cleaner if some confidence intervals were reported, and also that I was surprised not to see a comparison to LoRA for a parameter efficient training method.
Decision: Accept-Main