Selective Aggregation for Low-Rank Adaptation in Federated Learning
Authors: Pengxin Guo
Shuang Zeng
Yanran Wang
Huijie Fan
Feifei Wang
Liangqiong Qu
Venue: ICLR 2025 Poster (2025)
Keywords: federated learning
low-rank adaptation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned $A$ and $B$ matrices. In doing so, we uncover that $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at https://github.com/Pengxin-Guo/FedSA-LoRA.

Published as a conference paper at ICLR 2025
SELECTIVE AGGREGATION FOR LOW-RANK ADAPTA- TION IN FEDERATED LEARNING
Pengxin Guo' Shuang Zeng' Yanran Wang Huijie Fan Feifei Wang Liangqiong Qu'
1 The University of Hong Kong 2 Stanford University
3 Shenyang Institute of Automation, Chinese Academy of Sciences
Materials Innovation Institute for Life Sciences and Energy (MILES), HKU-SIRI
(guopx, (engshhJecooneeee hku. hk, joycewyrestanford, edu, fanhui jieesii.eei...........e....e.i.e cn, cn, ffwangθeee waaeeee.kkuhhe hku.hk, lianggqulhku.ln
ABSTRACT
We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned A and B matrices. In doing so, we uncover that A ma- trices are responsible for learning general knowledge, while B matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Fed- erated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low- rank trainable matrices A and B to model the weight update, but only A matrices are shared with the server for aggregation. Moreover, we delve into the rela- tionship between the learned A and B matrices in other LoRA variants, such rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and AedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work subsequent LoRA variants com- bined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at https//gituub.com/Pegggg.. wwwp:///eaniiiiiioooo/SAAAAAOOO.
INTRODUCTION
Large Language Models (LLMs) trained on large amounts of text, referred to Pre-trained Lan- guage Models (PLMs), have become a cornerstone of Natural Language Processing (NLP) (Brown) 2020 Touvron al. 2023 Achiam et al. 2023 Chowdhery et al., (023). Typically, adapt PLMs for specific tasks or enhance accuracy in real-world scenarios, fine-tuning PLMs on task-specific data is often needed. However, in many world applications, data is distributed across different institutions, and data sharing between these entities is often restricted due to privacy and regula- tory concerns. Federated Learning (FL) (McMahan et al., 2017: Li et al.) 2020a Zhang et al.) 122roui et al. [202]), which utilizes collaborative and decentralized training of models across multiple institutions without sharing personal data externally, offers a promising solution to this challenge.
Despite its promise, fine-tuning PLMs in an FL system challenging due to the high computational and storage demands on local clients and the communication overhead involved. To enable fine- tuning of PLMs in an FL system with limited resources, various Srrameter-Efficiee Fine-Tume Finn-nuning (PEFT) techniques have been explored. These include adapter-tuning-bat
methods (Houlsby
et al.
2019; Zhang et al. 2022bb), prompt-tuning--ast
methods (Li & Liang. 2021 Guo er al.)
2023b
Che al. 22233 Guo al. 2023a Qiu et al., 2024 Li et al.) 2024: Deng et al. 2024 Sun 2024a
et al.)
Cui et al. 2024 Cao et al. 2024h. and LoRA-based methods (Hu et al. 2022 Yi
al.
2023
Liu et 2023 Yang al. 2224
Qi al. 2024
Cho al. 2023 Byun Lee 2024
Chen et al. 2024 Sun et al.) 2024b Lin et al. 2024 Wu et al.
2024 Wang et al. (024b). Among
these,
methods have become increasingly popular, leveraging the assumption that over-
parameterized models have low intrinsic dimension (Li et al.
2018 Aghajanyan al./ 2020). A
pre-trained model can be shared and utilized to create multiple small LoRA modules tailored for
"Corresponding author.

---

Published as a conference paper at ICLR 2025
hl
hl
+
B
+
+
frozen
Pretrained weights
B 0
Pretrained weights
8
Pretrained weights
B
trainable global
W E Raddd
A o'
W & Ruxd
A -((a.'))
W C Ruxd
A n -((a."))
trainable local
(a) LoRA
(b) FFA-LORA
(c) FedSA-LoRA (Ours)
Figure 1: The illustration of (a) LoRA, (b) FFA-LoRA, and (c) FedSA-LoRA. In LoRA, both A and B matrices are trainable and shared with the server for aggregation. In FFA-LORA, only B matrices are trainable and shared with the server for aggregation, while A matrices are fixed after initialization. In FedSA-LoRA, both and B matrices are trainable, but only A matrices are shared with the server for aggregation while B matrices are kept locally.
different tasks, making them more effective and flexible. Moreover, this simple design allows to merge the trainable matrices with the frozen weights during deployment, introducing no inference latency. Given these advantages, we focus on LoRA-based methods in this work.
However, aggregating LoRA matrices and B in FL setting poses a key problem. Directly aggre- gating the A and B matrices the server and then broadcasting them to each client may introduce aggregation errors. Specifically, in an FL task with m clients, each client's model update is rep- resented by two low-rank matrices A, and B; introduced by LoRA, After server aggregation and broadcast, the model update of each client is:
1 (A, + A2 + ..... Am); B1 + B2 + + Bm) m
m
which is different from the "ideal" model update, i.e., (B, + B2A2 + +... Bm Am).
To solve this problem, some methods have been explored (Sun et al., 2024b Wang et al. 2024b). For example, (Sun et al. (2024b) propose Federated Freeze-A LoRA (FFA-LoRA), which freezes the A matrices and only updates and aggregates the matrices, as illustrated in Figure [b). Thus, the local update of each client under FFA-LoRA is (B1 B2 + + Bm where Ao denotes the initialized and fixed weights. They point out that this term is equal to the "ideal" model update intro- duced by FFA-LORA, i.e., 1 (B, Ag + B2 +... + B, Ao). However, fixing A matrices can impair the learning ability of LoRA and result in suboptimal performance (Zhang et al.) 2023). Meanwhile, many works have demonstrated that a uniform model update for all clients not optimal, especially
under the non-lII scenario (Zhao et al., 2018: Zhu et al. 2021 [i et al. (2022). which motivates the development of personalized FL (T Dinh al.) [2200] Collins et al. [202]] Xu et al.) 2023). To this end, we aim to explore better way to combine LoRA and Fl. in this work and move beyond the constraint that the model update of each client should be the same.
To achieve this, we start by analyzing the distinct roles of the learned A and B matrices when com- bining LoRA with FL, resulting in Lemma [] This lemma suggests that when combing LoRA with FL, A matrices are responsible for learning general knowledge while B matrices focus on capturing client-specific knowledge. To verify this empirically, we locally fine-tuned a RoBERTa-large model (Liu et al. 2019) with LoRA (Hu et al. 2022) on the RTE task from the GLUE benchmark (Wang ct al. 2018) with three clients under different levels of data heterogeneity. The results, illustrated in more similar across clients than the B matrices, and with increased data heterogeneity. the similarity of matrices between different clients decreases. These results demonstrate our argument that A matrices are used learn general knowledge while B matrices focus on modeling client-specific knowledge.
Figure show that the learned A matrices
Based on findings, introduce the Federated Share-A Low-Rank Adaptation (FedSA-LoRA) method in this work. Similar to LoRA (Hu et al.) 2002), we utilize two trainable low-rank matrices, denoted as A and B, model the weight updates during local training. However, only the A matrices are shared with the server for aggregation, as illustrated in Figure [c). Then, the model
Refer to Scction A.3 Appendix for more explanations about the derived aggregation errors.
2

---

Published as a conference paper at ICLR 2025
update of client i after server aggregation and broadcast is:
Bi -A: + A m
Am).
where the first part B; is responsible for capturing client-specific knowledge while the second part is used model general knowledge. By sharing the matrices that learn general knowledge with the server for aggregation, while keeping the B matrices that capture client-specific knowledge locally during training, the learning abilities of LoRA combined with FL can be enhanced. Note that this method differs from previous works (Sun et al. 2024b; Wang et al. 2024b) that require each client to share a uniform model update. Instead, it allows for different model updates. placing it in the realm of personalized FL (T Dinh al., 2020 Collins et al. 2021 Xu et al.) 2023), which is more al. (022).
efficient under the non-IID scenario (Zhao et 20.8 et al. 2021:
Moreover, delve into the relationship between the learned A and B matrices in other LoRA vari- ants, such as rsLoRA [Klajdzzievkk] (2023) and VeRA (Kopiczko et al. 2024). The observations, illustrated in Figures S and in Appendix, demonstrate similar phenomenon ing upon these insights, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA VeRA. By extending the proposed method other LoRA variants, we establish general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL.
LoRA. Build-
We summarize our contributions as follows:
We investigate the relationship between learned and B matrices in LoRA and other LoRA variants (c.g., rsLoRA and VeRA) across different clients, delincating their distinct roles. Specifically, A matrices are responsible for learning general knowledge, while matrices focus capturing client-specific knowledge.
Building upon our findings, we establish a general paradigm for integrating LoRA with FL. Specifically, we introduce Federated Share-A LoRA (FedSA-LoRA), where both A and B matrices are trainable, but only the A matrices are shared with the server for aggre- gation. We then generalize the FedSA-LoRA framework to other LoRA variants, resulting in LodRA and FedSA-
Extensive experimental results demonstrate the superiority of the proposed FedSA-LoRA, FedRA- and FedSA- E-RAA-VeRA compared to other methods.
RELATED WORK
2.1 FEDERATED LEARNING
Federated Learning (FL) (McMahan et al. 2017 Li et al. 2020a Zhang et al. [022 Kairouz et al.)
202T), a commonly used distributed learning method for tasks requiring privacy, has gained signifi- cant attention in recent years. However, its application faces challenges due to the non-III nature of distributed datasets, resulting in accuracy discrepancies compared to centralized training. Numerous
works (Li ct al. 2020b Xu et al., 2023 Yan et al., 2023 Chan et al. 2024 Xu et al. 2024 Zeng
et al. 2024) have been proposed to mitigate this performance degradation, including optimizing lo- cal learning et 2020b), optimizing server aggregation (Zeng et al. (024)), and personalized FL
Xu et al. 2023). Recently, some studies demonstrate that fine-tuning pre-trained models, especially Pre-trained Language Models (PLMs), through FL suffers less from the non-IlI issue (Qu al.,
2022
Chen et al. 2023 Nguyen et al. 2023) Weller et al. (022). The experimental results in (Weller 2022) show that when applying PLMs, even the vanilla FedAvg can achieve performance comparable to centralized training. However, these large-scale PLMs usually introduce significant communication overheads in FL scenarios, leading slow and impractical federated training in real-world applications. Additionally, local clients are often constrained by limited computational capacity and memory, making the local fine-tuning of PLMs challenging. To enable fine-tuning of PLMs in an FL system with limited resources, various Parameter-Efficient Fine-TTuinn (PEFT)
al.
techniques have been explored, such as adapter-tuning -aaning-bbsedd methods (Houlsby et al. 2019 Zhang et al. 2024b), prompt-tuning-- based methods (Li & Liang 2021
Guo et al.) 2023b Che al. 2023 Guo et al.] 2023a Quu et al. 2024 et al.) 2024| Deng et al. 2024 Sun et al. 2024a|
u et al.
2024 Cao et al. 2024), and MoRA-basel methods (Hu et 2022 Yi et al. 2023 Liu al. 2023
3

---

Published as a conference paper at ICLR 2025
Yang et al. 2024 Qi et al. 2024 Cho et al. 2023 Byun & Lee, 2024 Chen et al. 2024 Sun al. 2024b] Lin et al. 2024 Wu et al.
2024; Wang et al. 2024b). Among these, LoRA- babed methods have become increasingly popular, leveraging the assumption that over-parameterized models have low intrinsic dimension (Li et al. 2018 Aghajanyan et al. (020). A pre-trained model can be shared and utilized to create multiple small LoRA modules tailored for different tasks, making them more effective and flexible. Moreover, this simple design allows us to merge the trainable matrices with the frozen weights during deployment, introducing no inference latency. Given these advantages, we focus on coRA-based methods in this work.
2.2 LORA IN FEDERATED LEARNING
Low-Rank Adaptation (LoRA) (Hu et al. 2022 which introduces low-rank adaptation matrices to simulate gradient updates while keeping the pre-trained model weights frozen, has recently gained significant attention due to its efficiency, effectiveness, and flexibility (Hayou et al. [024] Liu et al., 2024 Kopiezko et al., [024] Wang al.) (2024)). With this trait, LoRA can be utilized to mitigate the communication overhead in FI., which primarily relies on the size of model update parameters. Yi al. (2023) propose Fed,oRA, incorporating LoRA in FL. to increase model fine-tuning efficiency. Liu et al. (2023) introduce DP-LORA, ensuring differential privacy in FL. for LLMs with minimal communication overhead. Yang et al. (2024) propose a personalzzing adapter (FedDPA), and Qi et al. (2024) introduce FDLORA. Both adopt the similar idea where each client contains per- sonalized LoRA module and a global LoRA module to capture personalized and global knowledge,
respectively.
Another line of such work is heterogeneous LoRA. For example, Cho et al. (2023) introduce het- erogeneous LoRA, where they deploy heterogeneous ranks across clients, aggregate the heteroge- neous LoRA modules through zero-padding, and redistribute the LoRA modules heterogeneously through truncation. However, this simple zero-padding strategy can make the training process unsta- ble (Byun & Lee 2024). To solve this issue, Bynnn & Lee 2024) propose a replication-based strategy for aggregating aunk-hetrrogenouus LoRA. Chen et al. (2024) propose Rank-Based LoRA Aggrega- tion (RBLA) that performs weighted aggregation for heterogeneous LoRA structures. Wang et al. (2024b) introduce a stacking bassed aggregation method for heterogeneous LoRA.
The most related work to ours is Federated Freeze-A LoRA (FFA-LoRA) (Sun et al.) 2024b), which fixes the randomly initialized non-zero matrices and only fine-tunes the zero- pero-initialized B matrices to further halve the communication cost. However, since some matrices are fixed, the learning ability of LoRA is impaired, resulting in suboptimal performance (Zhang et al. 2023). In contrast, we propose Federated Share-A LoRA (FedSA-LoRA), where both A and B matrices are trainable and only the A matrices are shared with the server for aggregation.
3 MOTIVATING EXAMPLE
Preliminary Building upon the hypothesis that updates to the weights during the fine-tuning ex- hibit a low "intrinsic rank". LoRA (Hu et 2022) proposes using the product of two low-rank ma- trices to update the pre-trained weights incrementally. For pre-trained weight matrix Wo E #xd LoRA models the weight update An E R utilizing low-rank decomposition, expressed as BA, where B E Axr and A E Rexd represent two low-rank matrices, with r & min(k, d). Dur- ing training, Wo is frozen and does not receive gradient updates, while and contain trainable parameters. Consequently, the fine-tuned weight W' can be represented as: W = Wo + BA. The matrix A is initialized with random Gaussian distribution, while B is initially set to zero, resulting in Aw BA being zero at the start of training.
To analyze the role of learned A and B matrices, Ict's consider a simple example analogous to a single network layer with least-squares linear regression task. Specifically, suppose there is pre- trained linear model weight Wo E bxd With this model held constant, our goal is regressing (re, y4) pairs where y/ is given by:
y1 = W,Y,,
with W, = Wo + Aw. In LoRA, the target Aw is modeled by low rank update to the pre- trained Wo i.e., W Wo + BA:
ÿ (Wo + BA)
4

---

Published as a conference paper at ICLR 2025
1.60
1.00
0.75
0.75
0.75
0.28
0.25
0.25
Layer
Layer
(a) IID
(b) moderate non-lID
(c) server non-lID
0.75
0.75
0.25
0.00
0.00
(d) IID
(c) moderate non-lII
server non-IID
Figure 2: Mean of pairwise cosine similarity of the learned A and B matrices across layers of RoBERTa model locally fine-tuned with LoRA the RTE task, with different levels of data heterogeneity. (a)-(c): value matrices; (d)-(f): query matrices. The learned matrices are more similar across clients than the B matrices, and with increased data heterogeneity, the similarity of B matrices between different clients decreases.
where B E lxr and Rrxd, with r < min(k, d). Then, the least squares loss is defined on the difference between ý and yr:
E llly. (Wo + BA)x.1))
(3)
Below, we present the lemma on minimizing this loss while freezing either A or B. The proof is provided in Section A. in Appendix.
Lemma 1. Fine-tuning B while fixing A Q, with the goal of optimizing Eq. 3 yields:
B Aw B[xrat ( (QR[x;=;]]
Fine-tuning A while fixigg B = U and assuming U-1 exists, with the goal of optimizing Eq. (3) yields:
A' --=
Remark 1. From this lemma, we can conclude that the optimal solution of A is independent of the imput data distribution, while B* is related the imput data distribution captured by Mxxzat This indicates that A is responsible for learning general knowledge, while B focuses on modeling client- specific knowledge.
To verify this empirically, we locally fine-tune RoBERTa-large model (Liu et al. 2019) with LoRA (Hu et al., 2022) on the RTE task from the GLUE benchmark (Wagg et al. 2018) B using three clients. We model IID data distribution and two non-lllD data distributions. The are modeled by Dirichlet distribution with a = and a = 0.5, referred to as moderate non-llI) and severe non-IID. FigureZssoov shows the mean pairwise cosine similarity of the learned A and B matrices across clients. These results indicate that the learned A matrices are more similar across clients than the B matrices, and with increased data heterogeneity, the similarity of matrices between different clients decreases. To demonstrate that the A matrices are indeed updated, as they are similar across different clients, we further illustrate the difference between the learned and initialized matrices for each client in Figure 4 in Appendix. These results confirm that the A matrices are updated. This phenomenon is consistent with previous study about the asymmetry analysis in LoRA (Zhu al. 2024). Based these results, argue that A matrices responsible for learning general knowledge while B matrices focus on capturing client-specific knowledge.
non-IlI distributions
To demonstrate the generalizability of our findings, we further explore the relationship between the learned A and B matrices other LoRA variants, such as LoLoRA (Kalajdzievski (0023) and VeRA (Kopizzko et al. 2024). The observations, illustrated in Figures 5 5aand66in Appendix, show a similar phenomenon to LoRA. In this way, we uncover general phenomenon when combining LoRA with FL., which serves the foundation for our proposed method.
5

---

Published as a conference paper at ICLR 2025
There are some works analyzing the asymmetry of A and B within LoRA in other areas (Zhu al. 2024
Tian et al. 2024). However, our work distinctly considers the asymmetry analyses of A and B within LoRA in the context of federated learning, while these related papers (Zhu et al. 2024] Tian et al., 2024 can further serve as verification of the effectiveness of our method. In particu- lar, further analyzed different non-III scenarios, as data heterogeneity is significant issue in FL. Through our analysis, we not only found that the learned A matrices are more similar across clients than the /' matrices (consistent with previous findings), but more importantly, we discov- ered that with increased data heterogeneity, the similarity of B matrices between different clients decreases. Moreover, we further extended this asymmetry analysis to other LoRA variants, such as LsLo (Kalajdzievskij (023) and VeRA (Kopiczko et al. 2022). and found similar phenomena. This generalization was previously lacking the literature (Zhu et al. 2024
Tian et al. 2024).
OUR METHOD
4.1 FEDERATED SHARE-A LOw RANK ADAPTATION
Drawing from the insights of our findings, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), illustrated in Figure [c), which utilizes two low-rank trainable matrices A and B to model the weight update, but only A matrices are shared with the server for aggregation. Specifically, similar to LoRA (Hu al.) 2022). we employ two low-rank matrices, namely B [exr and A E 1x with r < min(k: d), to model the weight update Aw E lxx for a pre-trained weight matrix Wo Rixdd This approach allows us to represent the fine-tuned weight as Wa + BA. During the local training process, Wo is frozen and does not receive gradient updates, while A and B contain trainable parameters. Following LoRA (Hu et al. 2022). the matrix A is initialized with random Gaussian distribution, whereas B is initially set to zero, ensuring that Aw = BA is zero at the start of training. Then, for global aggregation, only the matrices are shared with the server for aggregation. Once the server averages these A matrices, they are broadcast to each client for the subsequent training round. By sharing the A matrices that learn general knowledge with the server for aggregation, while keeping the B matrices that model client-specific knowledge locally, the learning abilities of LoRA combined with FL can be enhanced.
Moreover, based on the similar phenomena observed in other LoRA variants (i.e., rsLoRA (Kala- jdzievski| (203) and VeRA (Kopiczko et al. 2)244), ), we extend the FedSA-LoRA method to these variants, resulting in FedSA-rsLoRA and FedSA-VeRA Specifical,,, rsLoRA (Kalajdzievski 2023) is similar to LoRA, differing only in the scaling factor. Thus, the difference between Led and FedSA-LoRd also lies in the scaling factor. In VeRA (Kopiczko al., 2024). the low-rank ma- trices A and B are initialized using the uniform version of Kaiming inibalization, fixed, shared across all layers, and adapted with trainable scaling vectors and b. The vectors are initialized to zero, and the d vectors are initialized with a value of 0. To make the notation consistent with our work, we rewrite the scaling vectors d and as Aa and B, to reflect the position each scaling vector. Thus, in PedSA-VeRA. only the scaling vector Aa is shared with the server for aggregation, while B, is trained locally. By extending the proposed method to other LoRA variants, we establish general paradigm for integrating LoRA with FL, offering guidance for future work subsequent LoRA variants combined with FI.
4.2 CONVERGENCE ANALYSIS
To facilitate the convergence analysis of the proposed method, make assumptions commonly en- countered in the literature (Li et al. 2020c) to characterize the smooth and non-convex optimization landscape.
Assumption 1. LI. L. are all L-smooth. For all We,5 and W.ke L (W, S (W + (W., W.,,, V. + " W -.,
Here, m is the number of clients, W.j and Wi, represent any different model weights of client i, L. is the empirical loss on client i, V (.,.W.,)) represents the gradient of L, with respect to W.., ( is the Frobenius inner product, and ||| | denotes the Frobenius norm.
Assumption 2. Let Ei,t be sampled from the i-th client's local data uniformly at random at t- th training step. The expected squared norm of stochastic gradients is uniformly bounded, iee.

---

Published as a conference paper at ICLR 2025
(., E < G2, for all i 1, m and 0,... , 1. Here T denotes the total number of every client's ' training steps.
Assumption 3. Let ! = Wy + + A!" represent the model parameters for the i-th client at the t-th step. There exist constants CB 0. CA O, cp 0, and ca > O such that:
|B S CB.
|14 | S CA.
(4 A (V. 2 call (W:
(B(") B TVC
cB (V, )|?
for all i -1, 1, and 0, ... T-
Assumptions 1 and are widely used L -ssooohhness and bounded second moment assumption in FL papers (Li et al. 2020c Yu et al. 2019: Basu et al. 2019)) For Assumption 3
according to the def-
inition of the Frobenius norm
||A|| = E Σj lo.sl* we know that the first two inequalities in Assumption hold when all parameter values in AA and B(") are finite. If we denote the eigenval-
ues of A by (A1,..., -,.,, then the eigenvalue of ) A(!)) are {Aī,-.., > Similarly, if the eigenvalues of (.W((((()) are (.I...., "K]) then the eigenvalues of (V
are (ri,....,kkk Noting that (4 A/9) VVL (W,"() = Σx Agr and (VC,(W!")) )1? = Ex the third inequality of Assumption holds when there exist con- stant сΛ such that > 2 CA, Vk. In other words, the third inequality of Assumption holds when all the eigenvalues of A are non-zero (choose Ca min(|Agl,.... (Ak/)))..,,/AIl)) Similarly, the last inequality of Assumption 3 holds when all the eigenvalues of B(")) are non-zero. Then we present the convergence rate for our method, with the proof provided in Section Appendix.
Theorem 1. Let Assumptions 7 2 and 3 hold and L, G, CA. CB. CA. be defined therein. Denote E as the number of local training deeraiion between two communication rounds. Then, for learning rate n. have:
m
T
2
DM
(6)
mf
CA
T
where 3 T.G. +
< D,Vi, and E2C G C s Mnj.
According to Theorem under smooth and non-convex conditions. This convergence rate is comparable to that of traditional FedAvg in the non-convex scenario Yu et al. 2019)
can obtain 0 VT convergence rate towards the stationary solution
EXPERIMENTS
In this section, we evaluate and compare the performance of the proposed method with other meth- ods on two types of tasks: natural language understanding and natural language generation. For the natural language understanding tasks, we use the RoBERTa model (Liu et al.] 2019) evaluated on the GLUE benchmark (Wang al., 2018), including MNLI, SST2, QNLI, QQP, and RTE. For the natural language generation tasks, employ the LLaMA model (Touvron et al. 2023) evaluated on the GSMSK dataset (Cobbe et al. 2021). Our implementation based on the FedeeraeedScope-LLM library (Kuang al. 2023). The experiments for LoRA A-baeed methods are conducted NVIDIA GeForce RTX 4090 and 3090 GPUs, while the rsLoRA based and VeRA-based methods are car- ried out NVIDIA L40S GPUs. All experiments
performed with half- precisiin enabled for efficiency. For the main results in Table [[] they are based on multiple runs to report the mean and standard deviation, while other results in our paper based single run.
7

---

Published as a conference paper at ICLR 2025
5.1 NATURAL LANGUAGE UNDERSTANDING
Model Performance. For the natural language understanding tasks, similar to FFA-LoRA (Sun et al., 2024a), we randomly split the data across three clients for federated learning. We model a non- IID data distribution using a Dirichlet distribution with a = 0.5, i.e., Dir (0.5). We use the pre-trained RoBERTa-large (355M) (Liu et al.) 2019) from the HuggingFace Transformers li- brary (Wolf' et al., 2020) as the base model. For LoRA-based methods optimization, we adopt the SGD optimizer (Ruder 2016) for all approaches. We set the batch size to 128, local update steps to 10, and total communication rounds to 1000, consistent across all experiments. Similar to Hu ct al. )2022), we only apply LoRA to W, and W, in the attention layers in our experiments. The rank = 8 and scaling factor a 16 are fixed for all algorithms. We report the best result from experiments run with learning rates 17 E {5E-3, 1E-2, 2E-2, 5E-2, 1E-1). ). The optimization of IsLoRA-based methods is similar to LoRA-based methods, only the learning rates searched from 7 [1E-3 2E-3, 5E-3, 1E-2, 2E-2, 5E-2). For VeRA-based methods, following VeRA (Kopiczko) at. 2024), we set the rank 256. We adopt the AdamW optimizer (Loshchilov & Hutter 2017), introduce separate learning rates for the classífication head and the adapted layers, and determine the learning rates through hyperparameter tuning. Other settings are the same as for LoRA-based methods. The learning rates used for each method are shown in Tables 6 7 and 8 in Appendix.
Table 1: Performance of different methods on the GLUE benchmark. MNLI-m denotes MNLI with matched test sets, and MNLI-mm denotes MNLI with mismatched test sets. For all tasks, we report accuracy evaluated across 3 runs with mean and standard deviation.
Method
MNIIIm
MNI.I-m
SS12
QNLI
QQP
RTE
Avg.
LoRA
88.71 20.0
88.19
95.16gn.0
91.1610.72
85.33_1.33
87.49 15
89.33
FFA-LoRA
88.8320.02
88.27
9-1.5520.0
91.52.0.50
86.71_0.07
86,08
89.39
LoRA
FedDPA-LORA
88. 29.0.0
88, 43.0.06
95.50+0.00
90.74
85, 85.73_1.23
87 87.4420.15
89.47
LedSA-LoRA
10.1820.2
88. 22.02
96.00_o.04
92.13.0.24
87.48.0.22
87.93
90.43
rsLoRA
88.9 18.91
88, 88.33
95. 2m.2
91.21 20.39
86.73_oo.s
85,99 34
89.36
IsLoRA
FFA-ISLORA
29.21
88, 88.4540.08
95.42+0.17
91.42+0.64
86, 86.93_1.11
85.24.0.22
89.41
FedDPA-rsLoRA
89.34.0.11
88. 13.50
95.56
90.97 10.76
86.81 =0.53
86.26
89.57
FedSA-sSLoRA
90.35go.11
89. 89.02
85.78_o.s
92.03
20.97
88.00g0.10
90.52
VeRA
05.54
85.09 20.07
93.53
91.90
82.07
86.31.0.1
87.40
FFA- VeRA
86.63
86.22
93.44
92.05
82.23
83.54.0.
87.35
VeRA
FedDPA-VeR
86.74.0.11
46.35
93.61
90.73 20.54
82. 82.11_0.41
86. 86.1220.12
87.61
VeRSA-VeRA
17.21
86.52 20.04
93.68
92.91
82.56
87.83
88.45
The experimental results are shown in Table T. From this table, we can observe that the proposed FedSA-LORA, FedS ArsLLRA, and FedSA- VeRA consistently outperform other methods across all tasks, demonstrating the effectiveness of the proposed method.
System Efficiency. We further compare the proposed method with baselines in terms of system efficiency, following Qu et al. 2022); Zhang et al. a (2024a); Lai et al. (2022). al, FL consists of communication cost and computation cost. To provide a comprehensive comparison, we detail the number of trainable parameters, the number of communication model parameters per FL round, the computation cost per FL round, and the number of communication rounds needed to reach the predefined target performance on the RTE and QNLI tasks in Table [ The target performance is defined as 95% of the prediction accuracy provided in LoRA (Hu al., (022)). Specifically, we define the target performance of the RTE and QNLI tasks as 80.94% and 90.06% respectively.
The system efficiency in
Communication cost is a critical factor in FL, as it significantly impacts the overall system effi- ciency. The communication cost can be roughly estimated by considering the number of transmitted messages required to achieve a target performance, calculated as # transmitted messages= # com- munication round x # communicated model parameter. shown in Table 2 our A-dSA-LORA requires the smallest communication cost to reach the target performance (with smallest commu- nication rounds and per-round communication parameters). Additionally, while our FedSA-LORA requires more trainable model parameters and incurs slightly more computation cost per FL round than the baseline FFA-LoRA (22s compared to 20s on RTE task), it is important to note that our model reaches the target performance with fewer communication rounds (#91 compared to #229 on RTE task). These computation and communication costs demonstrate the overall efficiency of our model.
8

---

Published as a conference paper at ICLR 2025
Time and space costs for each method the RTE and QNLI tasks. # Communication round denotes the number of communication rounds to reach the predefined target performance.
Table 2:
# Trainale Parm. # Per-round Communicated Parm.
Compatation
Communication
RTE
QNLI
RTE
QNLI
LoRA
1.83M
0.78M
35s
167
397
FFA-LoRA
1.44M
0.39M
20s
33s
229
374
FodDPA-LORA
2.62M
0.78M
37%
128
325
FedSA-LoRA
1.83M
0.39M
22s
34s
91
224
5.2
IN-DEPTH ANALYSES
In this section, utilize LoRA-based methods to perform in-depth analyses the natural language understanding tasks of QNLI, SST2, and MNLI-m to assess the impact of factors such as data heterogeneity, the number of clients, and LoRA rank model performance.
5.2.1 EFFECT OF DATA HETEROGENEITY
To investigate the effect of data heterogeneity model performance, model IID partition (Split-1) and two non-IID partitions with Dir (1) and Dir (0.5). The latter two non-IID partitions are referred to moderate non-IlD (Split-2) and severe non-IID (Split-3). The training settings the same as Section 5.1
Table 3: Performance comparison on the QNLI, SST2, and MNLI-m tasks with various degrees of data heterogeneity.
QNLI
SST2
MNL.-m
Method
Split-1
Split-2
Split-3
Split-1
Split-2
Split-3
Split-1
Split-2
Split-3
LoRA
92.92
92.44
90.60
95.30
95.53
95.26
88.52
88,35
88.80
FFA-LoRA
92.68
92.29
91.72
95.87
95.47
94.91
88.15
88.03
88.83
FedSA-LoRA
92.95
93.32
92.00
96.10
96.24
95.92
89.57
89.71
90.20
The results provided in Table 3 From these results, can observe that the proposed FedSA- LoRA consistently outperforms other basclines, demonstrating its adaptability and robustness in various heterogeneous data scenarios. Additionally, data heterogeneity increases, the improve- ment of the proposed method also increases. Specifically, FedSA-LoRA improves accuracy by 0.03%, 0.88%, and 1.84% the QNLI task from IID to severe non-IID compared with LoRA, and by 1.05%. 1.36%, and 1.4% on the MNLI-m task. This indicates that the proposed method is more effective when non-IID conditions are more severe. This phenomenon is consistent with Figure 2 which shows that with increased data heterogeneity. the similarity of B matrices between different clients decreases. Therefore, when the non-lID conditions are more severe, the advantages of keep- ing B locally become more pronounced. In this case, the learned B will be less similar, highlighting the need for personalization.
5.2.2 EFFECT OF NUMBER OF CLIENTS
In Section 5.1
we demonstrated the effectiveness of the proposed method on a small number of clients, i.e., three clients. In this section, we show the superiority of FedSA-LoRA compared to other baselines on larger number of clients, i.e. from 10 to 100 clients. Specifically, we use the same non won-IID split, i.e., Dir (0.5), to divide the data into 10, 20, and 100 clients. The training settings are the same as in Section 5. and the results are shown in Table
It can be concluded that FedSA-LORA not only outperforms other methods with small number of clients (i.e., 3 clients) but also shows superior performance with large number of clients (i.e., from 10 to 100 clients), demonstrating the adaptability and robustness of the proposed FedSA-LoRA across various client numbers.

---

Published as a conference paper at ICLR 2025
Table 4: Performance comparison the QNLI, SST2, and MNLIIm tasks with different number of clients. We apply full participation for FL system with 10 and 20 clients, and apply client sampling with rate 0.3 for FL system with 100 clients.
QNLI
SST2
MNLI-m
Method
10 clients
clients
100 clients
10 clients
20 clients
100 clients
10 clients
20 clients
100 clients
91.32
91.23
96.68
88.13
FFA-LORA
91.47
91.70
91.27
96.59
93.31
96,33
86.76
88.60
87.86
RedSA-LoRA
91.97
92.54
91.48
96.83
94.21
97.02
88.59
89.05
88.82
5.2.3 EFFECT OF LORA RANK
The adapter parameter budget (i.e., rank r) is key factor in LoRA performance. In this section, experiment with rank r e {2, 4, 8, 16) on the QNLI, SST2, and MNLI-m tasks to test its influence on model performance, keeping other settings unchanged compared Section 5.11
Table 5: Performance comparison on the QNLI, SST2, and MNLI-m tasks with different LoRA
ranks r.
Rank
Method
QNLI
SST2
MNLI-m
Rank
Method
QNLI
SST2
MNLI-m
LoRA
92.07
94.69
88,29
LoRA
92.71
94,20
88.43
FFA-LORA
91.02
93.95
87.98
FFA-LORA
92.97
94.61
88.24
FedSA-LoRA
92.69
95.67
89.14
FedSA-LoRA
93.10
95.69
88.98
LoRA
90.69
95.26
88,80
LoRA
90.59
94.98
88.78
r=
FFA-LORA
91.72
94.91
88.83
FFA-LoRA
91.62
94.13
89.25
FedSA-LoRA
92.00
95.92
90.20
FedSA-LORA
92.03
95.78
89.59
demonstrate that the proposed FedSA-LoRA outperforms other methods across various LoRA rank values, showcasing the adaptability and robustness of FedSA- LoRA in different scenarios.
The results, shown in Table 5
5.3 NATURAL LANGUAGE GENERATION
For the natural language generation tasks, we adopt the pre-trained LLaMA3-8B (Meta 2024 from the HuggingFace Transformers library (Wolf al. 2020), using the GSM8K dataset (Cobbe al. 2021 and the CodeSearchNet dataset (Husain et al. 2019) for evaluation. For the GSM8K dataset (Cobbe et al., 2021), Following (Kuang et al. 2023), we split the data into three clients under an IID distribution, and other optimization hyperparameters are the same as in that work. The results of LoRA, FFA- and FedSA-LoRA are 46.23, 46.32, and 46.63, respectively. and the generated examples are shown in Table 9 in Appendix. From the given example, it can be seen that both LoRA and FFA-LORA A have reasoning errors, but FedSA-LORA can reason accurately, demonstrating the superiority of the proposed method in complex natural language generation tasks.
For an additional dataset on the code generation task. We choose the CodeSearchNet dataset (Hu- sain et al.) 2019) and use the default non-lII partitioning provided in (Kuang et al. 2023). The performance scores for LoRA, FFA-LoRA, and our FedSA-LoRA are 58.34, 58.57, and 59.66, re- spectively, which further validates the effectiveness of our method in generation tasks.
CONCLUSION
In this work, discover that when combining LoRA with FL, A matrices responsible for learn- ing general knowledge, while B matrices focus on capturing client-specific knowledge. Building upon this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices A and B to model the weight update, but only A matrices are shared with the server for aggregation. By sharing the matrices that learn general knowledge with the server for aggregation, while keeping the B matrices that model client-specific knowledge locally, the learning abilities of LoRA combined with FL can be enhanced. Moreover, we explore the relationship between the learned A and B matrices in other LoRA variants, such rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA- VeRA.. By doing so, establish general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China (62306253, U24A201397), Early career fund (27204623), and Guangdong Natural Science Fund-General Pro- gramme (2024A1515010233).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087774, 2023.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-sgd: Distributed sgd with quantization, sparsification and local computations. Advances in Neural Information Processing Systems, 32, 2019.
Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005. 14165, 2020.
Yuji Byun and Jaeho Lee. Towards federated low-rank adaptation with rank-heterogeneous commu- nication. arXiv preprint arXiv:2406. 17477, 2024.
Linxiao Cao, Yifei Zhu, and Wei Gong. Sfprompt: Communication-cfficient split feder- ated fine-tuning for large pre-trained models over resource-limited devices. arXiv preprint CrXiv:2407.115333 2024.
Yun-Hin Chan, Rui Zhou, Running Zhao, Zhihan JIANG, and Edith CH Ngai. Internal cross- layer gradients for extending homogeneity to heterogeneity in federated learning, In The Twelfth International Conference Learning Representations, 2024.
Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou. Federated learning of large language models with parameter-efficien prompt tuning and adaptive optimization. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 7871-7888, 2023.
Hong-You You Chen, Cheng-Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao. On the importance and applicability of pre-training for federated learning. In The Eleventh International Conference Learning Representations, 2023.
Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, and Albert Y. Zomaya. Rbla: Lank-based-lon aggregation for fine-tuning heterogeneous models in flaas, 2024.
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous
lora for federated fine-tuning of on-device foundation models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International conference machine learning, pp. 2089-2099, PMLR, 2021.
Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing generalization and personalization in federated prompt learning. In Forty-first International Conference Machine Learning, 2024.

---

Published as a conference paper at ICLR 2025
Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition, pp. 6087-6097, 2024.
Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp. 1364-1374, 2023a.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptil: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing, 2023b.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning. 2024.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learming Representations, 2022.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv: (eti::/999.0940 09436, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1 2021.
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arX1v:2312.03732, 2023.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference Learning Representations, 2024.
Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-lIm: comprehensive package for fine-tuning large language models in federated learning. arXiv preprint arXiv:2309,00363 2023.
Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. Fedscale: Benchmarking model and system performance of federated learning scale. In International conference machine learning, pp. 11814-11827, PMLR, 2022.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.0 08838, 2018.
Hongxia Li, Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12151-12161 2024.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE), pp. 965-978. IEEE, 2022.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50 60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and sys- fems, 454. 2020b.
12

---

Published as a conference paper at ICLR 2025
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020c.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pp. 4582-4597, 2021.
Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, and Yue Gao. Splitlora: A split parameter-effiient fine-tuning framework for large language models. arXiv preprint arXiv:2407.00952. 2024.
Shih-yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: deight-decomposee low-rank adaptation. In Forty-first International Conference Machine Learning, 2024.
Xiao- Yang Liu, Rongyi Zhu, Daochen Zha, Jiechao Gao, Shan Zhong, Matt White, and Meikang Qiu. Differentially private low-rank adaptation of large language model using federated learning. ACM Transactions on Management Information Systems, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1907. 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effici learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1231 1282. PMLR, 2017.
Al Meta. Introducing meta llama 3: The most capable openly available Ilm date. Meta AI, 2024.
John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where begin? on the impact of pre-training and initialization in federated learning. In The Eleventh International Conference on Learning Representations, 2023.
Jiaxing Qi, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. Fdlora: Personalized federated learning of large language model via dual lora tuning. arXiv preprint 1rXiv:2406.07925 2024.
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Yi Lin. Federated text-driven prompt generation for saiiun- models. In The Twelfth International Conference Learning Representations, 2024.
Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10061-10071, 2022.
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-cfficee federated learning with sketching. In International Conference on Machine Learning, pp. 8253-8265. PMLR, 2020.
Sebastian Ruder. An overview of gradient descent optimization algorithms. 164.:1660904447,, 2016.
arXiv preprint
Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yudong Liu, Zhixu Du, Yiran Chen, and Holger R Roth. Fedbpt: Efficient federated black-box prompt tuning for large language models. In Forty-first International Conference on Machine Learning, 2024a.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. In The Twelfth International Conference Learning Representations, 2024b. 20244.
13

---

Published as a conference paper at ICLR 2025
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau velopes. Advances in neural information processing systems, 33:21394 0112119---1155, 2020.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404. 19245, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355 2018.
Xuchao Wang, Feiyang Ye, and Yu Zhang. Task-aware low-rank adaptation of segment anything model. arXiv preprint arXiv:2403.1 10971, 2024a.
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.0599 2024b.
Orion Weller, Mare Marone, Vladimir Braverman, Dawn Lawrie, and Benjamin Van Durme. Pre- trained models for multilingual federated learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1413-1421, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. Fedbiot: Llm local fine-tuning in federated learning without full model. arXiv preprint arXiv:2406. 17706, 2024.
Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In The Eleventh International Conference Learning Representa- tions, 2023.
Peiran Xu, Zeyu Wang, Jieru Mei, Liangqiong Qu, Alan Yuille, Cihang Xie, and Yuyin Zhou. Fedconv: Enhancing convolutional neural networks for handling data heterogeneity in federated learning. Transactions on Machine Learning Research, 2024.
Rui Yan, Liangqiong Qu, Qingyue Wei. Shih- Shih-Cheng Huang, Liyue Shen, Daniel L Rubin, Lei Xing, and Yuyin Zhou. Label-efficient self-supervised federated learning for tackling data heterogeneity in medical imaging. IEEE Transactions Medical Imaging, 42(7):1932 1943, 2023.
Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein. Dual- phal-personalizing adapter for federated foundation models. arXiv preprint arXiv:2403. 1921,
Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneen personalized federated learning with lora tuning. arXiv preprint arXiv:2310. 13283, 2023.
Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 5693-5700, 2019.
Shuang Zeng. Pengxin Guo, Shuai Wang, Jianbo Wang, Yuyin Zhou, and Liangqiong Qu. Tackling data heterogeneity in federated learning via loss decomposition. arXiv preprint arXiv:2408. 12300, 2024.
Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. survey on federated learning. Knowledge- Based Systems, 216: 216:106777 106775, 2021.
14

---

Published as a conference paper at ICLR 2025
Junyuan Zhang, Shuang Zeng. Miao Zhang, Runxi Wang, Feifei Wang, Yuyin Zhou, Paul Pu Liang, and Liangqiong Qu. Flhetbench: Benchmarking device and state heterogeneity in federated learn- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12098-12108 2024a.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint wrXiv:2308.0330: 2023.
Zixin Zhang, Fan Qi, and Changsheng Xu. Enhancing storage and computational efficiency in federated multimodal learning for large-scale models. In Forty-first International Conference on Machine Learning. 2024b.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:/806 1800.0088,0 2018.
Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Neurocomputiing 465:371-390, 2021.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. Forty-first International Conference Machine Learning, 2024.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
PROOF OF LEMMA
Proof. Consider fine-tuning B while freezing A = Q. The loss function in Eq.
becomes:
L (( (Wo + 11))
Then, the gradient of Eq.
w.r.t. is:
ac
d (E (lv. (Wo + BQ)=x113
aB
0B
@ (Wo + BQ) Joi [13]
aB
[E[ + Aw)r, - (Wo + BQ)
0B
[E[l|(Aw BQ) 10)
0B
]E BQ)x =Q)==
E[2(BQ Aw x Q²).
To obtain the optimal B", we set Eq. 1 to zero, which means:
E[2(BQ Aw) Q= 2BQE 10" 2 2 1Q" =
2BQE[ZaZ]00 2Aw , 'Q" = 0
BQE 1\0T= Aw
Aw (QE
Therefore, obtain 1 B(r !Q" (QE(xx=Z]Q*))
When fine-tuning A with fixed U. The loss function Eq.
becomes:
C (a (Wo + U UAr)
(10)
Then, the gradients of Eq.
w.r.t. As:
DE (E Sommmmlleee [1le1
(Wo + UA)
0A
0A
@T [1 (Wo + U AA)
0A
[ + Aww )ar,
(Wo BQ)x+13]
(11)
0A
[B[[[(Aw 1
0A
E|2UT UA)=,aa?)
To obtain the optimal 4", we set Eq.
(1I] to zero, which means:
El2 - UA) = 0
2 Aw
E.
20 UB =
(12)
U AB
U AwE(r
A U- U-
Thus, obtain 4" U-1A
16

---

Published as a conference paper at ICLR 2025
A.2 PROOF OF THEOREM 1
Proof. Let 1 = Wo + B A be the model parameters maintained in the i-th client at the I-th step. Let ZE be the set of global synchronization steps, i.e., ZE {nE' = 1,2,- ..... If I ++1 1 E Ig. which represents the time step for communication, then the one-step update of the proposed method for the i-th client can be described as follows:
B B A
B B((+t)
B ((++1) 1 4
update of (1") aand_("*)
(
it+1EIE
mn 4
(+11)
For convenience, we denote the parameters in each sub-step above as follows:
w:(") = Wo + ( A(")
0 = Wo + (( A ((+1))
v Wo B((+1)
1 (4+2)
(011) V/(0)
w((+1) =
if 1 & Ig.
if E Iz.
Here, the variable U/00 represents the immediate result of one sub-step update from the parameter of the previous sub-step w(tt), and vz(t) represents the parameter obtained after communication steps (if applicable). Furthermore, we denote the learning rate for the i-th client the 1-th step as T)i and the stochastic gradient at step / as follows:
VnC(W S.t)) VWL.WWW.... E.)A())
(t.e)) = B."' TwC,(W!",E..d VAC, .5,e)
(BC((W(")) BVV E.)).
(W(()) EIV (w, E..).. ,..))
where Ea, is the data uniformly chosen from the local data set of client at step t.
Next, apply the inequality from the smoothness Assumption T to each sub-step of the one-step
update for client i. Firstly, by the smoothness of L,, we have:
(,(U!1)) < (.( + (0 w( VwL.(W!)))
L |000
wr(t)
(13)
2
F
Since
B((+1) Bid nh. V BB, E.,)
Bud0 Th. Ww ...)).0)
and
( A Th.t A.. Ece)
= (1 B(, VwL.(W!".... EE.),
have:
0.!1) w(()
(((++1) ((+11) - -B("DAAA"
B The VwL. (
.5..)) -B( 1000
nh C. ((.)A BSST VwC,(W! ...) The WwC.W A,50
(() (3.0) Vw.C(W,
17

---

Published as a conference paper at ICLR 2025
Then, for the second term on the right side of Eq. 13 according to the law of total expectation, have:
( - W, VwC,(W,")) ]]
E
K
BOOT VW (E.)), Vw ) J
= B
F.
BDD TV VWL.WW ....) C.
BB
= n2. (VWLC((1/()) AA BLTT Vww.(W,))) Vw
- me (Vwci(w (0 )4 A vw
The
Since
(W )A!") BLOT (ww.(W, )w
$ (W, 1400 BiaT (w ||
< | B
|
I CACBB
and if we assume there exists CA > such that Jt::
(490 (0)
VV. 2 ca /WW.,(w,)) 1 r=
then we have:
VWCC(WW()444)) T A VWW.CWW.")
= Tr| -T((VCC((M;))4 440) [w
= -TT[A LW [WCC(N
= =AA T A ( )VW((())))))
2 CA Vw C. (w,")) !|=
And similarly if we assume there exists cg > 0 such that Vt:
( (B_0T Bi", VV 2 cn VWW,(WW")) |
then we have:
B )1) B VWL.(W!()) VW.((M(),,GC(((.)))
-r (1")B(' B.") (VL((W/")) Vww.(wm (
= Tr CWW, B!" D!OT VVWC((())
= TT
B BBO''
V RB0 B,OT
2 cn |
(14)
18

---

Published as a conference paper at ICLR 2025
where we use the eyclic property of the trace for the third equality above. We further get:
{ wi Vw Vw (w .1
S C - - Vw [w,C,, m | [V,
(15)
Similarly, we know:
E||li - w/' [1:] Bilmn, Vw (W, (,. BiST V Et.s
The Ww (u.)4 T A - m,, B.") Vw
< 3n C2C + 3n/ G2 + C,,
(16)
Plugging Eq.
[4). Eq. 15), and Eq. 16) into Eq. 113 we have:
[w 11 2 - MACB (w.
Li Li (
n CAC - CAACA
2 Th
3 C.BGOL
3
2 g..
2 2 LB
3 2
(17)
Secondly, by the smoothness of L,, we have:
C. (v! s Ci (1V!)) -v:"'-
- -i(h
L v.00 -1:01
(18)
2
Since
v,(0) - -1,0 (1(tt1)AA A(t
1 - B B(((t])
m
1(t+1)
m
j=1
m m
B((+1))
(1+1) - A
1=1
and
A ( = A! A4- (1E++))
Bitoo ( Entoo
1+=1-E+1
Al ((-B+1)
Blo (lu)) ; (ria))
bo=t-E E+1
we know:
((()) - 11!0 = Bi(tD)
BitoBlto) Blto) (VL,(W!"")),an))) (w,, ;sitol))
Miso
j=1 1-=1-E+1
Therefore,
21
F
nidto Biag Blto)
(VL,( :a.))))))))))))))))))))))))))))) VL,(W)(()) (V (k360)) : [
E
to to==tBi E+1
E m
02
T5.50
m
F.
j=l b+=
C+ E
VL,(W,((1)) Ei.o)) vL,((,/()))) (3:
1,.0 E
1g=t-Bi 1+1
4C1 F.G2 m
Vytto:
(19)
m
j=1 ts=t-Ba E+1
19

---

Published as a conference paper at ICLR 2025
where we use Assumption 2 to derive that:
(10); C, (W! Sstoo ; -
(W,
(t0): {(tto) 1/2]
( [12] 2 VL. (w, : 113]
s 2F
s 4G
Furthermore,
(v,0' - 0,40, VwL.(U,())
1
((t) 0 + 2 /
-
2/
20 EG2
m
T. + 1
B
2 nr.
(20)
mm
J= to ft=t-Ra E+1
Plugging Eq. 19 and Eq.
20 into Eq.
18 we have (choose constant learning rate The = 1))
C + (2C + )0) 2F
C.
(21)
Combining Eq.
17 and Eq.
(1). we have (choose constant learning rate 1/5 7))
Li W (1+1)
I Li (ww) + C2C + 6
1 2
+ (CACBB²
+ 3 2 020II0++ +
CBLLG + E 2
3
m(cA CB) [W]
which is equivalent to:
Tlca + cB) VW/C s L: (mW!!)) La (mx((t1))
1
3
C2
G
G
2
CCAC
3
CRIC²
3
ChiCce
2
Choosing M which satisfies 2/ L. + 2C (GG))'" S M1j², we get:
+ = C (C +
C2
L. ((W(")) - Li (w.(tt1)
V (W,
Mn
(22)
nCcA + CB)
CA CB
Now, by repeatedly applying Eq.
for different values of and summing up the results, we get:
S C, (ww:") L. (W?)
Σ
T
(w
2 F
+ 1)
M
T.
(23)
n(a + +c)))
CA
Dividing both side of Eq.
by T. we get:
C, (IW/")) (WF)
+
Σ
V Vw
11
+ 1]
M
(24)
7(CA + CB)T
CA +CB
Let us assume that (,(W())) L.(WT) S D, Vi, and we set
D MT.
Then, we have:
2
DM
(25)
T
CA
CA
T
Thus, we can obtain:
m
2
DM
<
(26)
CA
+
Cp
T
20

---

Published as a conference paper at ICLR 2025
A.3 FURTHER EXPLANATIONS ABOUT AGGREGATION ERRORS.
In this section, we provide more explanation about what aggregation errors are and why people do not achieve the "ideal" model update.
When introducing LoRA into Fl., the update for the i-the client is given by Aw; = B, The "ideal" model update for server aggregation should be:
1 m
1 B. m
1 (BIA B2A2 BmAm).
A W
Aw.
(27)
However, in practical LoRA training, the matrices A, and B;, not the original W,, are trainable. Thus, we cannot directly average the Aw,; instead, we can only separately average A, and B, and then combine them to obtain the update:
Aw
A:)
m
m
(B) + B₂ + ..... + B,n) (A1 + A2 + ... Am)
(28)
m (B1 A1 + B1A2 + + Am + B1211
B2A2 + ...... + B2Am + +... + Bm + A2 + .... + Bm)m),
which differs from the "ideal" model update in Eq. 27 This difference introduces aggregation errors.
Regarding why the "ideal" model update is not achieved, it is because the matrices A, and B, are trainable, not the original W, While it possible first combine A, and B, to obtain Aw, and then average them to achieve the "ideal" update, we cannot decompose W back into A and B for further federated training. This limitation prevents people from performing the "ideal" model update.
A.4 HYPERPARAMETERS
Tables 6 and 7 show the learning rates used for LoRA- Abbsed methods and rsLoRA-based methods, respectively. For the VeRA-based methods, we first tried using the SGD optimizer (Ruder) (0166) with a search learning rate from E (5E-3, 1E-2, 2E-2, 5E-2, 1E-1} adopted in LoRA-based methods, but we found the performance to be significantly worse than that LoRA-based methods. For example, the best performance among the three VeRA-based methods (i.e., VeRA, FFA- FFA-VeRA, and FedSA-VeRA) is 53.73% on the MNLI-m task, which is significantly worse than that of the LoRA-based methods (90.20%) Thus, chose the AdamW optimizer (Loshchilov & Hutter 2017) and introduced separate learning rates for the classification head and the adapted layers as used VeRA (Kopiczko al. 2024). The learning rates used for VeRA-basee methods are shown in Table
Table 6: The learning rates used for LoRA-based methods on the GLUE benchmark.
Method
MNLI-m
MNLI-mm
SST2
QNLI
QQP
RTE
LoRA
IE-2
1E-2
2B-2
1E-2
1E-2
1E-2
FFA-LoRA
5E-2
5E-2
5E-2
2E-2
5E-2
2E-2
FedDPA-LORA
1E-2
1E-2
1E-2
5E-2
5E-2
1E-2
FedSA-LoRA
2E-2
2E-2
1E-2
SE-3
2E-2
1E-2
A.5 EXAMPLES OF GENERATED ANSWER FOR GSM8K DATASETS
The results of the GSM8K dataset are shown in Table 9 demonstratigg that the proposed FedSA- LoRA outperforms other methods in complex natural language generation tasks. From the given example, it can be seen that both LoRA and FFA-LoRA have reasoning errors, but FedSA-LoRA reason accurately, demonstrating the superiority of the proposed method.
21

---

Published as a conference paper at ICLR 2025
Table 7: The learning rates used for rsLoRA-based methods the GLUE benchmark.
Method
MNLI-m
MNLI-mm
SST2
QNLI
QQP
RTE
rsLoRA
5E-3
5E-3
1E-2
2E-3
5E-3
2E-3
FFA-rSLORA
2B-2
2E-2
2B-2
1B-2
2E-2
IE-2
FedDPA-reLoRA
SE-3
5E-3
1E-2
1E-3
5E-3
IE-2
FedRA-rsLoRA
5E-3
5E-3
5E-3
1B-3
2E-3
2E-3
Table 8: The learning rates used for VeRA-based methods on the GLUE benchmark.
Method
Position
MNLI-m
MNLI-mm
SST2
QNLI
QQP
RTE
VeRA Head
IB-2
IB-2
2E-2
2E-3
2B-3
1B-2
VeRA
6E-3
6E-3
2E-3
3E-4
3E-4
2E-4
VeRA
2E-2
2E-2
1E-2
1E-2
1E-2
1E-2
FFA-VeRA
Head
2E-3
2B-3
6E-3
2F-4
6F-3
2F-4
VeRA
1E-2
IE-2
1E-2
2E-3
2E-2
IE-2
FedDPA-Vels VeRA
Head
6E-3
6E-3
6E-3
3E-4
2E-3
2F-4
VeRA
2E-3
2B-3
IE-2
IE-2
2E-3
IE-2
FedSA-VeRA
Head
3E-5
3E-5
3E-4
3E-4
3E-4
IE-4
Table Performance of different methods on the GSM8K dataset and example answers generated by each method.
LoRA
FFA-LoRA
FedSA-LoRA
Accuracy
46.24
46.32
46.63
Question:
In student council election, candidate got 20% of the votes while candidate B got 50% more than candidate A's votes. The rest of the votes was given to candidate If there were 100 voters, how many votes did candidate C get?
Output:
Candidate A got 20% of the votes. That is 20% of
Candidate A got 20% of 100 100 x 20% 100 x 20/100 = 20. Candidate got 20% more than can-
Candidate A got 20% of the votes 20% 100 = 20 votes. Then candi- date got 50% more than candidate A's 20 votes so
100 = 20% x 100= 20 x
100 2000 votes. Candi- date B got 50% more than candidate That is 50% of 2000 = 50% 2000 x 2000 = 10000 votes.
didate A's votes, which is 20% 20% 40%. 40%
50% of 20 is 20 50%
of 100 = 100 x 40% = 100
10. Then candidate B got
x 40/100 40. So the two
20 + 10 30 votes. So
The rest of the votes was
candidates together got 20
candidate C got 100 20
given to candidate C. So there are 10000 + 2000 12000 votes in total. Can- didate C got 10000 votes. The answer 10000. [X]
40 60. The rest of the
30 50 votes. The an-
votes were given to candi-
swer is 50. [V]
date C. So candidate got
100 60 40. The answer
is 40. [X]
22

---

Published as a conference paper at ICLR 2025
A.6 LEARNED MATRICES COMPARISON
A.6.1 LEARNED AND INITIALIZED LORA MATRICES COMPARISON
Since the learned LoRA A matrices are similar across different clients in Figure 2 separately plotted the relationships of A from Figure Figure show the relationships This shows they are similar but not identical. Moreover, we further illustrate the difference between learned and initialized A matrices for each client under the IID partition this section. The results, shown in Figure confirm that the A matrices updated.
more clearly.
0.9975
0.9975
0.9975
0.5950
0.5650
0.990
0.9925
0,9925
0.995
L
LORA
LARA.6
0.0900
0.0900
0.900
Lay
Layer
Layer
(a) IID
moderate non-lID
(C) server non-lID
0.0915
0.0935
0.975
0.5925
0.0915
0.9955
0.900
Layer
Layer
Layer
(d) IID
moderate non-lID
server non-IID
Figure 3: Mean of pairwise cosine similarity of the learned A matrices layers of RoBERTa model locally fine-tuned with LoRA on the RTE task, with different levels data heterogeneity. (a)-(c): value matrices; (d)-(f): query matrices. The learned A matrices across client similar but not identical.
1.000
1.000
1,000
0.990
0.990
0.990
Sel Ceeeeteeeeeeeeeeee Pe
Layers
Layers
(a) Client I & Init
(b) Client 2 & Init
(c) Client 3 & Init
1.000
1,000
1,000
0.395
0.995
0.985
0.985
0.985
0.985
Oeliw OeENii 1117--000000 -4000
Layers
Layers
Layers
Client 1 Init
Client & Init
(f) Client 3 & Init
Figure 4: Cosine similarity of learned and initialized A matrices across layers of different clients of a RoBERTa model locally fine-tuned with LoRA on the RTE task. (a)-(c): value matrices; (d)-(f); query matrices. The learned A matrices are different from the initialized A matrices, indicating that the matrices are updated.
A.6.2 LEARNED RSLORA MATRICES COMPARISON
In this section, present the mean of pairwise client relationships for the learned rsLoRA (Kala- jdzievski| 2023) matrices. These results, shown Figure demonstrate a similar phenomenon to the learned LoRA matrices. That is, the learned A matrices more similar across clients than the
23

---

Published as a conference paper at ICLR 2025
B matrices, and with increased data heterogeneity, the similarity of B matrices between different clients decreases.
1.30
1.00
1.00
0.50
0.50
0.50
"LUMAIA
SUUMAAA
"LUMAAA
"LUMA
SLIMA
"LUNA 5
0.00
0.00
Layer
Layer
Layer
(a) IID
moderate non-IID
server non-IID
1.00
1.00
1.00
0.50
0.50
0.25
0.25
0.25
Layer
Layer
Layer
(d) IID
moderate non-IID
server non-III
Figure 5: Mean of pairwise cosine similarity of the learned A and B matrices across layers of a RoBERTa model locally fine-tuned with oRR on the RTE task, with different levels of data heterogeneity. (a)-(c): value matrices; (d)-(f): query matrices. The learned matrices are more similar across clients than the / matrices, and with increased data heterogeneity, the similarity of B matrices between different clients decreases.
A.6.3 LEARNED VERA MATRICES COMPARISON
In this section, we show the mean of pairwise client relationships for the learned VeRA (Kopiczko jet al. 2024) matrices. In VeRA, the low-rank matrices A and B are initialized using the uniform version of Kaiming initialization, fixed, shared across all layers, and adapted with trainable scaling vectors d and The b vectors are initialized to zero, and the vectors are initialized with a value of 0.1. To make the notation consistent with our work, we rewrite the scaling vectors d and b as Aa and B, to reflect the position of each scaling vector. These results, illustrated in Figure 6 demonstrate similar phenomenon to the learned LoRA matrices. That is, the learned scaling vectors Ag more similar across clients than the scaling vectors B,, and with increased data heterogeneity, the similarity of scaling vectors B, between different clients decreases.
1.00
1.00
1.00
0.75
0.50
0.25
0:25
0.25
Layer
Layer
Layer
(a) IID
moderate non-III
(c) server non-III
0.75
0.75
0.75
0.25
0.2
GHAR wine Wo
Layer
(d) IID
(e) moderate non-IID
server non-IID
Figure 6: Mean of pairwise cosine similarity of the learned scaling vectors A and B, across layers of RoBERTa model locally fine-tuned with VeRA the RTE task, with different levels of data heterogeneity. (a)-(c): value matrices; (d)-(f): query matrices. The learned scaling vectors Aa are more similar across clients than the scaling vectors B,, and with increased data heterogeneity, the similarity of scaling vectors B, between different clients decreases.
24

---

Published as a conference paper at ICLR 2025
A.7 FURTHER RESULTS ON COMPRESSING A MATRICES
Based on the reviewers' comments during the review stage, we further conduct experiments on compressing the matrices. Specifically, we chose FetchSGD (Rothchild et al. 2020) to compress around 50% of the A matrices, given that the updates across the A matrices are trivial but necessary. The experimental results are shown in Table I0] which indicates that FedSA-LoRA with a compres- sion method achieves performance comparable to the original FedSA-LoRA, This confirms that the matrices can indeed be further compressed reduce communication overhead while maintaining satisfactory performance, validating the reviewer's ss hypothesis that "there must exist compression method or a sparse structure to significantly reduce the number of parameters in the A matrices that need to be updated"..
Table 10: Time and space costs for each method on the RTE and QNLI tasks. # Communication round denotes the number of communication rounds to reach the predefined target performance. denotes equipped with the compressing method FetchSGD.
Trainable Purm
Per-eound Communicated Parm.
$ Per-mound Computitio Cost
Commuication Round
Accuracy
RTF
QNLI
RTF.
QNI.
RTE
QNIJ
LoRA
1.8M
0.78M
224
35s
167
307
87 20.40
91. 16,172
FEA LoRA
1.44M
0.39M
20s
33s
229
86.08
31.52
87.44a
04.74
87
18ance
FoUSA-LoRA
1.83
0.20M
22x
34s
79
37.88
92. 12.160
25


Reviews:
[{"review_id": "BfB5jz5gLp", "reviewer": "Reviewer_KuVq", "summary": "This paper proposes a low-rank federated fine-tuning algorithm, namely FedSA-LoRA, which regards $A$ as a global adapter and $B$ as a personalized adapter. This algorithm reduces the communication cost by only aggregating matrices $A$.", "questions": "Please refer to Weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper has the interesting motivation and shows the different roles of low-rank matrices $A$ and $B$ in federated fine-tuning.", "weaknesses": "I have the following concerns:\n\n1. Lemma 1 is very interesting to me and the verified experiments show that all the matrices $A_i$ ($i$ is the client index) seems to be the same and $B_i$ differs with each other. We know that different client has independent initialization of $A$, but finally with the exactly same $A$ since the similarity of $A$ from clients is $1.0$ in Figure 2. As you mentioned in Figure 3 in Appendix that the learned A matrices are different from the initialized A matrices, so how do you guarantee that the learned A across clients are the totally same even in the server non-IID scenario.\n\n2. The implementation details for Figure 2 are not clear to me, especially how do you implement federated lora fine-tuning, do you update/aggregate both $A$ and $B$?\n\n3. In Assumption 1, what is the meaning of $\\mathcal{L}_i(W_j)$ and $\\nabla \\mathcal{L}_i(W_j)$? In your descriptions, both $i$ and $j$ are the client's index, right? Then how can you access the client-$j$'s gradient information in client $i$?  \n\n4. Assumption 2 is too strong, it becomes very easy to handle dissimilarity between clients and stochastic gradients with this assumption. This assumption doesn't appear in smoothness cases in many federated learning papers, so how do you verify this assumption? Can you provide examples to verify the rationality.\n\n5. As I mentioned in concern 4, assumption 2 is a strong assumption, which does not appear in work [1]. Even the same convergence rate as [1] is achieved, that comparison is meaningless to me. \n\n[1] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020.\n\n6. In experiments, they compare with FFA-LoRA, and show better performance than FFA-LoRA. My concern is that, is this a fair comparison? Because the trainable parameters are not the same, FFA-LoRA does not tune the matrices $A$, but FedSA-LoRA does, which means FedSA-LoRA has much more trainable parameters than FFA-LoRA. I hope to see the report of trainable parameters for each algorithm. \n\n7. Many advanced baselines are not compared, so the performance of the proposed algorithm is not convincing. For example, some related work mentioned in the paper, FedLoRA [2], FedDPA [3], and HetLoRA [4]. \n\n[2] Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneous personalized federated learning with lora tuning. arXiv preprint arXiv:2310.13283, 2023. \\\n[3] Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein. Dual-personalizing adapter for federated foundation models. arXiv preprint arXiv:2403.19211, 2024.\\\n[4] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8QETD0ur5c", "reviewer": "Reviewer_tZCA", "summary": "In this paper, the authors present FedSA-LoRA, a novel approach for federated learning that shares only the A matrices across clients to leverage the functional differences between A and B matrices in LoRA modules. The method’s effectiveness is validated through both theoretical analysis and empirical results.", "questions": "See above. I will increase my score if my concerns are well addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-motivated and easy to follow.\n2. This work introduces the shared-A LoRA technique into the FL framework, making it promising compared to previous work where the down-projection matrix (A matrix) is kept frozen.\n3. Comprehensive experimental results are provided.", "weaknesses": "1. While this work introduces a shared-A LoRA framework into FL, such a technique has already been introduced in the MoE area [1] under similar motivations. Although the novelty of introducing it into FL should be recognized, the introduction of such a framework cannot be solely credited to this work.\n\n2. On page 20, Figure 3, the authors claim that \"A matrices are different from the initialized A matrices, indicating that the A matrices are updated.\" However, the cosine similarity between the learned and initialized down-projection matrices remains quite high (greater than 0.985), which indicates that the A matrices have changed very little compared to the initialization.\n\n3. Compared with [2], the cosine similarity across A matrices for different clients reported in this work is very high. Why does this happen? Is it caused by different levels of data heterogeneity?\n\n4. Assumption 3, especially the last two inequalities, lacks verification and explanation. It appears these are purely for simplifying the proof of Theorem 1 on page 16. The authors need to elaborate more on this uncommon assumption, including under what conditions it could hold and what properties of the A and B matrices this assumption reveals.\n\n5. In the provided code, alongside the B matrices being personalized, a classifier layer is also personalized. Is this necessary? How does this personalized classifier layer influence the experimental results?\n\nReferences:\n\n[1] Tian, Chunlin, et al. \"HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning.\" arXiv preprint arXiv:2404.19245 (2024).\n\n[2] Zhu, Jiacheng, et al. \"Asymmetry in low-rank adapters of foundation models.\" arXiv preprint arXiv:2402.16842 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xseZBAFmHf", "reviewer": "Reviewer_Lpwz", "summary": "The paper begins by analyzing the roles of matrices A and B in LoRA, concluding that matrix A is responsible for general knowledge while matrix B handles domain-specific knowledge. Based on this analysis, the authors propose FedSA-LoRA, a method that aggregates only the A matrix during federated learning while retaining the edge-side B matrix. This approach aims to integrate general capabilities while preserving personalized knowledge.", "questions": "(1) In the experimental results, why does FedSA-LoRA outperform other methods under non-heterogeneous conditions? In the absence of heterogeneity, since both matrices A and B can learn shareable knowledge, sharing information from both matrices—or sharing matrix B's information as in FFA-LoRA—should intuitively achieve equal or even greater gains.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The paper's analysis of the distinct roles of matrices A and B clearly demonstrates its insights and motivations.\n\n(2) The proposed method improves upon existing solutions and achieves certain performance enhancements.", "weaknesses": "(1) One of the significant contributions of the paper is the asymmetric analysis of matrices A and B, concluding that matrix A is responsible for general knowledge and matrix B for domain-specific knowledge. However, this perspective has already been proposed in several works, such as HydraLoRA [1], which also designed an asymmetric LoRA structure based on this concept. The paper should properly cite relevant literature, and these existing findings somewhat diminish the paper's contribution.\n\n(2) The experimental results are based on a single run, lacking multiple experiments to calculate averages and perform variance analysis, which may affect the reliability of the results. It is recommended that the authors refer to previous related work [2], conduct multiple experiments, and compute means and variances to more comprehensively evaluate the method's stability and effectiveness.\n\n(3) The paper mainly considers the sharing of edge-side models but lacks a reasonable mechanism to update the server-side model to endow it with comprehensive capabilities.\n\n[1] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning\n[2] Improving LoRA in Privacy-preserving Federated Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UX6ZYab2Nh", "reviewer": "Reviewer_dqpx", "summary": "This paper introduces a new way of combining LoRA in the federated setting, called FedSA-LoRA, where the authors found that A matrics are responsible for learning general knowledge, and B matrics focus on client-specific knowledge. Only A matrics are used for server aggregation. Extensive experiment results indicate the effectiveness of the proposed FedSA-LoRA with various LoRA-related variants.", "questions": "- The authors mention in the experiment section that the proposed method is more effective when non-IID conditions are more severe. Could the authors provide some potential reasons for that?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Observing the roles that matrices A and B play is intuitive and important, and this observation is also prevalent in several LoRA-related variants.\n- The experiments are comprehensive, covering different non-iid levels and client numbers and across different tasks. Also, the authors provide many ablation studies.\n- Authors provide some theoretical justifications for the convergence of their method.", "weaknesses": "- Motivation for exploring LoRA in federated learning: Why do authors use and investigate Lora-based methods in FL? The authors can expand and explain more about why it is good compared with parameter-efficient methods. In the introduction and related works, the authors state they are efficient, effective, and flexible, which is a bit too general from my perspective. The authors can better motivate the importance of this problem.\n- Line 52-71: First, I feel that what kind of aggregation errors is unclear to me: could the authors provide a more accurate and specific illustration of the aggregation errors? Secondly, I wonder in practice, why do people not do the ideal model update? What are the difficulties in doing that?\n- Authors can provide some computational analysis (e.g. time and space cost) of their method compared with the baselines.\n- The authors could include standard deviations or confidence intervals for their main results tables, particularly for the main results.\n- On the natural language generation task, the improvement of the proposed method seems to be a little marginal. Also, it is tested under IID distribution.\n- The overall contribution seems to be marginal from my perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)