Selective Aggregation for Low-Rank Adaptation in Federated Learning

## Abstract

Abstract We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned A 𝐴 A italic_A and B 𝐵 B italic_B matrices. In doing so, we uncover that A 𝐴 A italic_A matrices are responsible for learning general knowledge, while B 𝐵 B italic_B matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices A 𝐴 A italic_A and B 𝐵 B italic_B to model the weight update, but only A 𝐴 A italic_A matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned A 𝐴 A italic_A and B 𝐵 B italic_B matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at https://github.com/Pengxin-Guo/FedSA-LoRA .

## 1 Introduction

Large Language Models (LLMs) trained on large amounts of text, referred to as Pre-trained Language Models (PLMs), have become a cornerstone of Natural Language Processing (NLP) (Brown, ; Touvron et al., ; Achiam et al., ; Chowdhery et al., ).
Typically, to adapt PLMs for specific tasks or enhance accuracy in real-world scenarios, fine-tuning PLMs on task-specific data is often needed. However, in many real-world applications, data is distributed across different institutions, and data sharing between these entities is often restricted due to privacy and regulatory concerns.
Federated Learning (FL) (McMahan et al., ; Li et al., ; Zhang et al., ; Kairouz et al., ), which utilizes collaborative and decentralized training of models across multiple institutions without sharing personal data externally, offers a promising solution to this challenge.

Despite its promise, fine-tuning PLMs in an FL system is challenging due to the high computational and storage demands on local clients and the communication overhead involved.
To enable fine-tuning of PLMs in an FL system with limited resources, various Parameter-Efficient Fine-Tuning (PEFT) techniques have been explored. These include adapter-tuning-based methods (Houlsby et al., ; Zhang et al., ), prompt-tuning-based methods (Li & Liang, ; Guo et al., ; Che et al., ; Guo et al., ; Qiu et al., ; Li et al., ; Deng et al., ; Sun et al., ; Cui et al., ; Cao et al., ), and LoRA-based methods (Hu et al., ; Yi et al., ; Liu et al., ; Yang et al., ; Qi et al., ; Cho et al., ; Byun & Lee, ; Chen et al., ; Sun et al., ; Lin et al., ; Wu et al., ; Wang et al., ). Among these, LoRA-based methods have gained significant attention due to their efficiency, effectiveness, and flexibility, which is also the focus of our work.

However, aggregating LoRA matrices $A$ and $B$ in FL setting poses a key problem. Directly aggregating the $A$ and $B$ matrices on the server and then broadcasting them to each client may introduce aggregation errors.
Specifically, in an FL task with $m$ clients, each client’s model update is represented by two low-rank matrices $A_{i}$ and $B_{i}$ introduced by LoRA.
After server aggregation and broadcast, the model update of each client is:

$$ $\frac{1}{m}(B_{1}+B_{2}+\cdots+B_{m})\frac{1}{m}(A_{1}+A_{2}+\cdots+A_{m}),$ (1) $$

which is different from the “ideal” model update, i.e., $\frac{1}{m}(B_{1}A_{1}+B_{2}A_{2}+\cdots+B_{m}A_{m})$.

To solve this problem, some methods have been explored (Sun et al., ; Wang et al., ). For example, Sun et al. () propose Federated Freeze-A LoRA (FFA-LoRA), which freezes the $A$ matrices and only updates and aggregates the $B$ matrices, as illustrated in Figure 1(b). Thus, the local update of each client under FFA-LoRA is
$\frac{1}{m}(B_{1}+B_{2}+\cdots+B_{m})A_{0},$
where $A_{0}$ denotes the initialized and fixed weights. They point out that this term is equal to the “ideal” model update introduced by FFA-LoRA, i.e., $\frac{1}{m}(B_{1}A_{0}+B_{2}A_{0}+\cdots+B_{m}A_{0})$. However, fixing $A$ matrices can impair the learning ability of LoRA and result in suboptimal performance (Zhang et al., ). Meanwhile, many works have demonstrated that a uniform model update for all clients is not optimal, especially under the non-IID scenario (Zhao et al., ; Zhu et al., ; Li et al., ). To this end, we aim to explore a better way to combine LoRA and FL in this work and move beyond the constraint that the model update of each client should be the same.

Figure: Figure 1: The illustration of (a) LoRA, (b) FFA-LoRA, and (c) FedSA-LoRA. In LoRA, both $A$ and $B$ matrices are trainable and shared with the server for aggregation. In FFA-LoRA, only $B$ matrices are trainable and shared with the server for aggregation, while $A$ matrices are fixed after initialization. In FedSA-LoRA, both $A$ and $B$ matrices are trainable, but only $A$ matrices are shared with the server for aggregation while $B$ matrices are kept locally.
Refer to caption: x1.png

To achieve this, we start by analyzing the distinct roles of the learned $A$ and $B$ matrices when combining LoRA with FL, resulting in Lemma 1. This lemma suggests that when combing LoRA with FL, $A$ matrices are responsible for learning general knowledge while $B$ matrices focus on capturing client-specific knowledge. To verify this empirically, we locally fine-tuned a RoBERTa-large model (Liu et al., ) with LoRA (Hu et al., ) on the RTE task from the GLUE benchmark (Wang et al., ) with three clients under different levels of data heterogeneity. The results, illustrated in Figure 2, show that the learned $A$ matrices are more similar across clients than the $B$ matrices, and with increased data heterogeneity, the similarity of $B$ matrices between different clients decreases. These results demonstrate our argument that $A$ matrices are used to learn general knowledge while $B$ matrices focus on modeling client-specific knowledge.

Based on our findings, we introduce the Federated Share-A Low-Rank Adaptation (FedSA-LoRA) method in this work. Similar to LoRA (Hu et al., ), we utilize two trainable low-rank matrices, denoted as $A$ and $B$, to model the weight updates during local training. However, only the $A$ matrices are shared with the server for aggregation, as illustrated in Figure 1(c). Then, the model update of client $i$ after server aggregation and broadcast is:

$$ $B_{i}\frac{1}{m}(A_{1}+A_{2}+\cdots+A_{m}),$ (2) $$

where the first part $B_{i}$ is responsible for capturing client-specific knowledge while the second part is used to model general knowledge. By sharing the $A$ matrices that learn general knowledge with the server for aggregation, while keeping the $B$ matrices that capture client-specific knowledge locally during training, the learning abilities of LoRA combined with FL can be enhanced. Note that this method differs from previous works (Sun et al., ; Wang et al., ) that require each client to share a uniform model update. Instead, it allows for different model updates, which is more efficient under the non-IID scenario (Zhao et al., ; Zhu et al., ; Li et al., ).

Moreover, we delve into the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA (Kalajdzievski, ) and VeRA (Kopiczko et al., ). The observations, illustrated in Figures 4 and 5 in Appendix, demonstrate a similar phenomenon to LoRA. Building upon these insights, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. By extending the proposed method to other LoRA variants, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL.

We summarize our contributions as follows:

- •
We investigate the relationship between learned $A$ and $B$ matrices in LoRA and other LoRA variants (e.g., rsLoRA and VeRA) across different clients, delineating their distinct roles. Specifically, $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge.
- •
Building upon our findings, we establish a general paradigm for integrating LoRA with FL. Specifically, we introduce Federated Share-A LoRA (FedSA-LoRA), where both $A$ and $B$ matrices are trainable, but only the $A$ matrices are shared with the server for aggregation. We then generalize the FedSA-LoRA framework to other LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA.
- •
Extensive experimental results demonstrate the superiority of the proposed FedSA-LoRA, FedSA-rsLoRA, and FedSA-VeRA compared to other methods.

## 2 Related work

### 2.1 Federated Learning

Federated Learning (FL) (McMahan et al., ; Li et al., ; Zhang et al., ; Kairouz et al., ), a commonly used distributed learning method for tasks requiring privacy, has gained significant attention in recent years. However, its application faces challenges due to the non-IID nature of distributed datasets, resulting in accuracy discrepancies compared to centralized training.
Numerous works (Li et al., ; Xu et al., ; Yan et al., ; Chan et al., ; Xu et al., ; Zeng et al., ) have been proposed to mitigate this performance degradation.
Recently, some studies demonstrate that fine-tuning pre-trained models, especially Pre-trained Language Models (PLMs), through FL suffers less from the non-IID issue (Qu et al., ; Chen et al., ; Nguyen et al., ; Weller et al., ).
The experimental results in (Weller et al., ) show that when applying PLMs, even the vanilla FedAvg can achieve performance comparable to centralized training.
However, these large-scale PLMs usually introduce significant communication overheads in FL scenarios, leading to slow and impractical federated training in real-world applications. Additionally, local clients are often constrained by limited computational capacity and memory, making the local fine-tuning of PLMs challenging.
To enable fine-tuning of PLMs in an FL system with limited resources, various Parameter-Efficient Fine-Tuning (PEFT) techniques have been explored, such as adapter-tuning-based methods (Houlsby et al., ; Zhang et al., ), prompt-tuning-based methods (Li & Liang, ; Guo et al., ; Che et al., ; Guo et al., ; Qiu et al., ; Li et al., ; Deng et al., ; Sun et al., ; Cui et al., ; Cao et al., ), and LoRA-based methods (Hu et al., ; Yi et al., ; Liu et al., ; Yang et al., ; Qi et al., ; Cho et al., ; Byun & Lee, ; Chen et al., ; Sun et al., ; Lin et al., ; Wu et al., ; Wang et al., ). Among these, LoRA-based methods have gained significant attention due to their efficiency, effectiveness, and flexibility, which is also the focus of our work.

### 2.2 LoRA in Federated Learning

Low-Rank Adaptation (LoRA) (Hu et al., ), which introduces low-rank adaptation matrices to simulate gradient updates while keeping the pre-trained model weights frozen, has recently gained significant attention due to its efficiency, effectiveness, and flexibility (Hayou et al., ; Liu et al., ; Kopiczko et al., ; Wang et al., ). With this trait, LoRA can be utilized to mitigate the communication overhead in FL, which primarily relies on the size of model update parameters. Yi et al. () propose FedLoRA, incorporating LoRA in FL to increase model fine-tuning efficiency. Liu et al. () introduce DP-LoRA, ensuring differential privacy in FL for LLMs with minimal communication overhead. Yang et al. () propose a dual-personalizing adapter (FedDPA), and Qi et al. () introduce FDLoRA. Both adopt the similar idea where each client contains a personalized LoRA module and a global LoRA module to capture personalized and global knowledge, respectively.

Another line of such work is heterogeneous LoRA. For example, Cho et al. () introduce heterogeneous LoRA, where they deploy heterogeneous ranks across clients, aggregate the heterogeneous LoRA modules through zero-padding, and redistribute the LoRA modules heterogeneously through truncation. However, this simple zero-padding strategy can make the training process unstable (Byun & Lee, ). To solve this issue, Byun & Lee () propose a replication-based strategy for aggregating rank-heterogeneous LoRA. Chen et al. () propose Rank-Based LoRA Aggregation (RBLA) that performs a weighted aggregation for heterogeneous LoRA structures. Wang et al. () introduce a stacking-based aggregation method for heterogeneous LoRA.

The most related work to ours is Federated Freeze-A LoRA (FFA-LoRA) (Sun et al., ), which fixes the randomly initialized non-zero $A$ matrices and only fine-tunes the zero-initialized $B$ matrices to further halve the communication cost. However, since some matrices are fixed, the learning ability of LoRA is impaired, resulting in suboptimal performance (Zhang et al., ). In contrast, we propose Federated Share-A LoRA (FedSA-LoRA), where both $A$ and $B$ matrices are trainable and only the $A$ matrices are shared with the server for aggregation.

## 3 Motivating Example

##### Preliminary

Building upon the hypothesis that updates to the weights during the fine-tuning exhibit a low “intrinsic rank”, LoRA (Hu et al., ) proposes using the product of two low-rank matrices to update the pre-trained weights incrementally. For a pre-trained weight matrix $W_{0}\in\mathbb{R}^{k\times d}$, LoRA models the weight update $\Delta_{W}\in\mathbb{R}^{k\times d}$ utilizing a low-rank decomposition, expressed as $BA$, where $B\in\mathbb{R}^{k\times r}$ and $A\in\mathbb{R}^{r\times d}$ represent two low-rank matrices, with $r\ll min(k,d)$. During training, $W_{0}$ is frozen and does not receive gradient updates, while $A$ and $B$ contain trainable parameters. Consequently, the fine-tuned weight $W^{\prime}$ can be represented as: $W^{\prime}=W_{0}+BA$. The matrix $A$ is initialized with random Gaussian distribution, while $B$ is initially set to zero, resulting in $\Delta_{W}=BA$ being zero at the start of training.

To analyze the role of learned $A$ and $B$ matrices, let’s consider a simple example analogous to a single network layer with least-squares linear regression task. Specifically, suppose there is a pre-trained linear model weight $W_{0}\in\mathbb{R}^{k\times d}$.
With this model held constant, our goal is regressing $(x_{t},y_{t})$ pairs where $y_{t}$ is given by:

$$ $y_{t}=W_{t}x_{t},$ $$

with $W_{t}=W_{0}+\Delta_{W}$. In LoRA, the target $\Delta_{W}$ is modeled by a low rank update to the pre-trained $W_{0}$, i.e., $W^{\prime}=W_{0}+BA$:

$$ $\hat{y}=(W_{0}+BA)x_{t},$ $$

where $B\in\mathbb{R}^{k\times r}$ and $A\in\mathbb{R}^{r\times d}$, with $r\ll min(k,d)$. Then, the least squares loss is defined on the difference between $\hat{y}$ and $y_{t}$:

$$ $\mathcal{L}=\mathbb{E}_{(x_{t},y_{t})}[\|y_{t}-(W_{0}+BA)x_{t}\|_{2}^{2}].$ (3) $$

Below, we present the lemma on minimizing this loss while freezing either $A$ or $B$. The proof is provided in Section A.1 in Appendix.

###### Lemma 1 .

Fine-tuning $B$ while fixing $A=Q$, with the goal of optimizing Eq. (3), yields:

$$ $B^{*}=\Delta_{W}\mathbb{E}[x_{t}x_{t}^{T}]Q^{T}(Q\mathbb{E}[x_{t}x_{t}^{T}]Q^{ T})^{-1}.$ (4) $$

Fine-tuning $A$ while fixing $B=U$ and assuming $U^{-1}$ exists, with the goal of optimizing Eq. (3), yields:

$$ $A^{*}=U^{-1}\Delta_{W}.$ (5) $$

###### Remark 1 .

From this lemma, we can conclude that the optimal solution of $A^{*}$ is independent of the input data distribution, while $B^{*}$ is related to the input data distribution captured by $\mathbb{E}[x_{t}x_{t}^{T}]$. This indicates that $A$ is responsible for learning general knowledge, while $B$ focuses on modeling client-specific knowledge.

Figure: (a) IID
Refer to caption: x2.png

To verify this empirically, we locally fine-tune a RoBERTa-large model (Liu et al., ) with LoRA (Hu et al., ) on the RTE task from the GLUE benchmark (Wang et al., ) using three clients. We model an IID data distribution and two non-IID data distributions. The two non-IID distributions are modeled by a Dirichlet distribution with $\alpha=1$ and $\alpha=0.5$, referred to as moderate non-IID and severe non-IID.
Figure 2 shows the mean pairwise cosine similarity of the learned $A$ and $B$ matrices across clients. These results indicate that the learned $A$ matrices are more similar across clients than the $B$ matrices, and with increased data heterogeneity, the similarity of $B$ matrices between different clients decreases. To demonstrate that the $A$ matrices are indeed updated, as they are similar across different clients, we further illustrate the difference between the learned and initialized $A$ matrices for each client in Figure 3 in Appendix. These results confirm that the $A$ matrices are updated. This phenomenon is consistent with previous study about the asymmetry analysis in LoRA (Zhu et al., ). Based on these results, we argue that $A$ matrices are responsible for learning general knowledge while $B$ matrices focus on capturing client-specific knowledge.

To demonstrate the generalizability of our findings, we further explore the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA (Kalajdzievski, ) and VeRA (Kopiczko et al., ). The observations, illustrated in Figures 4 and 5 in Appendix, show a similar phenomenon to LoRA. In this way, we uncover a general phenomenon when combining LoRA with FL, which serves as the foundation for our proposed method.

###### Lemma 1 .

###### Remark 1 .

## 4 Our Method

### 4.1 Federated Share-A Low Rank Adaptation

Drawing from the insights of our findings, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), illustrated in Figure 1(c), which utilizes two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. Specifically, similar to LoRA (Hu et al., ), we employ two low-rank matrices, namely $B\in\mathbb{R}^{k\times r}$ and $A\in\mathbb{R}^{r\times d}$ with $r\ll\min(k,d)$, to model the weight update $\Delta_{W}\in\mathbb{R}^{k\times d}$ for a pre-trained weight matrix $W_{0}\in\mathbb{R}^{k\times d}$. This approach allows us to represent the fine-tuned weight as $W_{0}+BA$. During the local training process, $W_{0}$ is frozen and does not receive gradient updates, while $A$ and $B$ contain trainable parameters. Following LoRA (Hu et al., ), the matrix $A$ is initialized with a random Gaussian distribution, whereas $B$ is initially set to zero, ensuring that $\Delta_{W}=BA$ is zero at the start of training. Then, for global aggregation, only the $A$ matrices are shared with the server for aggregation. Once the server averages these $A$ matrices, they are broadcast to each client for the subsequent training round. By sharing the $A$ matrices that learn general knowledge with the server for aggregation, while keeping the $B$ matrices that model client-specific knowledge locally, the learning abilities of LoRA combined with FL can be enhanced.

Moreover, based on the similar phenomena observed in other LoRA variants (i.e., rsLoRA (Kalajdzievski, ) and VeRA (Kopiczko et al., )), we extend the FedSA-LoRA method to these variants, resulting in FedSA-rsLoRA and FedSA-VeRA. Specifically, rsLoRA (Kalajdzievski, ) is similar to LoRA, differing only in the scaling factor. Thus, the difference between FedSA-rsLoRA and FedSA-LoRA also lies in the scaling factor. In VeRA (Kopiczko et al., ), the low-rank matrices $A$ and $B$ are initialized using the uniform version of Kaiming initialization, fixed, shared across all layers, and adapted with trainable scaling vectors $d$ and $b$. The $b$ vectors are initialized to zero, and the $d$ vectors are initialized with a value of 0.1. To make the notation consistent with our work, we rewrite the scaling vectors $d$ and $b$ as $A_{d}$ and $B_{b}$ to reflect the position of each scaling vector. Thus, in FedSA-VeRA, only the scaling vector $A_{d}$ is shared with the server for aggregation, while $B_{b}$ is trained locally. By extending the proposed method to other LoRA variants, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL.

### 4.2 Convergence Analysis

To facilitate the convergence analysis of the proposed method, we make assumptions commonly encountered in the literature (Li et al., ) to characterize the smooth and non-convex optimization landscape.

###### Assumption 1 .

$\mathcal{L}_{1},\cdots,\mathcal{L}_{m}$ are all L-smooth. For all $W_{j}$ and $W_{k}$:

$$ $\mathcal{L}_{i}(W_{k})\leq\mathcal{L}_{i}(W_{j})+\langle W_{k}-W_{j},\nabla \mathcal{L}_{i}(W_{j})\rangle_{F}+\frac{L}{2}\|W_{k}-W_{j}\|_{F}^{2}.$ $$

Here, $m$ is the number of clients, $W_{k}$ denotes the model weight of client $k$, $\mathcal{L}_{i}$ is the empirical loss on client $i$, $\nabla\mathcal{L}_{i}(W_{j})$ represents the gradient of $\mathcal{L}_{i}$ with respect to $W_{j}$, $\langle\cdot,\cdot\rangle_{F}$ is the Frobenius inner product, and $\|\cdot\|_{F}$ denotes the Frobenius norm.

###### Assumption 2 .

Let $\xi_{i,t}$ be sampled from the $i$-th client’s local data uniformly at random at $t$-th training step. The expected squared norm of stochastic gradients is uniformly bounded, i.e.,
$\mathbb{E}\|\nabla\mathcal{L}_{i}(W_{i}^{(t)};\xi_{i,t})\|^{2}\leq G^{2},\text
{~{}for all~{}}i=1,\cdots,m\text{ and }t=0,\cdots,T-1$. Here $T$ denotes the total number of every client’s training steps.

###### Assumption 3 .

Let $W_{i}^{(t)}=W_{0}+B_{i}^{(t)}A_{i}^{(t)}$ represent the model parameters for the $i$-th client at the $t$-th step. There exist constants $C_{B}>0$, $C_{A}>0$, $c_{B}>0$, and $c_{A}>0$ such that:

$$ $\displaystyle\|B_{i}^{(t)}\|_{F}$ $\displaystyle\leq C_{B},$ $\displaystyle\|A_{i}^{(t)}\|_{F}$ $\displaystyle\leq C_{A},$ $\displaystyle\langle{A_{i}^{(t)}}^{\top}A_{i}^{(t)},\nabla\mathcal{L}_{i}(W_{i }^{(t)})^{\top}\nabla\mathcal{L}_{i}(W_{i}^{(t)})\rangle_{F}$ $\displaystyle\geq c_{A}\|\nabla\mathcal{L}_{i}(W_{i}^{(t)})\|_{F}^{2},$ $\displaystyle\langle{B_{i}^{(t)}}^{\top}B_{i}^{(t)},\nabla\mathcal{L}_{i}(W_{i }^{(t)})^{\top}\nabla\mathcal{L}_{i}(W_{i}^{(t)})\rangle_{F}$ $\displaystyle\geq c_{B}\|\nabla\mathcal{L}_{i}(W_{i}^{(t)})\|_{F}^{2},$ $$

for all $i=1,\cdots,m\text{ and }t=0,\cdots,T-1$.

Then we present the convergence rate for our method, with the proof provided in Section A.2 in Appendix.

###### Theorem 1 .

Let Assumptions 1, 2, and 3 hold and $L$, $G$, $C_{A}$, $C_{B}$, $c_{A}$, $c_{B}$ be defined therein. Denote $E$ as the number of local training iterations between two communication rounds. Then, for a learning rate $\eta$, we have:

$$ $\frac{1}{mT}\sum_{i=1}^{m}\sum_{t=1}^{T}\mathbb{E}\left[\left\|\nabla\mathcal{ L}_{i}(W_{i}^{(t)})\right\|_{F}^{2}\right]\leq\frac{2}{c_{A}+c_{B}}\sqrt{\frac {DM}{T}},$ (6) $$

where $\mathcal{L}_{i}(W^{(0)})-\mathcal{L}_{i}(W^{*})\leq D,\forall i$, and
$(2C_{B}^{2}E^{2}G^{2}+\frac{1}{2}G^{2})\eta+\frac{3}{2}\eta^{4}C_{A}^{2}C_{B}^
{2}G^{4}L+(C_{A}C_{B}G^{2}+\frac{3}{2}C_{A}^{2}LG^{2}+\frac{3}{2}C_{B}^{2}LG^{
2}+2C_{B}^{4}E^{2}LG^{2})\eta^{2}\leq M\eta^{2}.$

According to Theorem 1, we can obtain an $O(\frac{1}{\sqrt{T}})$ convergence rate towards the stationary solution under smooth and non-convex conditions. This convergence rate is comparable to that of traditional FedAvg in the non-convex scenario (Wang et al., ).

###### Assumption 1 .

###### Assumption 2 .

###### Assumption 3 .

###### Theorem 1 .

## 5 Experiments

In this section, we evaluate and compare the performance of the proposed method with other methods on two types of tasks: natural language understanding and natural language generation. For the natural language understanding tasks, we use the RoBERTa model (Liu et al., ) evaluated on the GLUE benchmark (Wang et al., ), including MNLI, SST2, QNLI, QQP, and RTE. For the natural language generation tasks, we employ the LLaMA model (Touvron et al., ) evaluated on the GSM8K dataset (Cobbe et al., ). Our implementation is based on the FederatedScope-LLM library (Kuang et al., ). The experiments for LoRA-based methods are conducted on NVIDIA GeForce RTX 4090 and 3090 GPUs, while the rsLoRA-based and VeRA-based methods are carried out on NVIDIA L40S GPUs. All experiments are performed with half-precision enabled for efficiency.

### 5.1 Natural Language Understanding

For the natural language understanding tasks, similar to FFA-LoRA (Sun et al., ), we randomly split the data across three clients for federated learning. We model a non-IID data distribution using a Dirichlet distribution with $\alpha=0.5$, i.e., Dir (0.5). We use the pre-trained RoBERTa-large (355M) (Liu et al., ) from the HuggingFace Transformers library (Wolf et al., ) as the base model. For LoRA-based methods optimization, we adopt the SGD optimizer (Ruder, ) for all approaches. We set the batch size to 128, local update steps to 10, and total communication rounds to 1000, consistent across all experiments. Similar to Hu et al. (), we only apply LoRA to $W_{q}$ and $W_{v}$ in the attention layers in our experiments. The rank $r=8$ and scaling factor $\alpha=16$ are fixed for all algorithms. We report the best result from experiments run with learning rates $\eta\in\{5\text{E-}3,1\text{E-}2,2\text{E-}2,5\text{E-}2,1\text{E-}1\}$. The optimization of rsLoRA-based methods is similar to LoRA-based methods, only the learning rates are searched from $\eta\in\{1\text{E-}3,2\text{E-}3,5\text{E-}3,1\text{E-}2,2\text{E-}2,5\text{E-
}2\}$. For VeRA-based methods, following VeRA (Kopiczko et al., ), we set the rank $r=256$. We adopt the AdamW optimizer (Loshchilov & Hutter, ), introduce separate learning rates for the classification head and the adapted layers, and determine the learning rates through hyperparameter tuning. Other settings are the same as for LoRA-based methods. The learning rates used for each method are shown in Tables 6, 7, and 8 in Appendix.

**Table 1: Performance of different methods on the GLUE benchmark. MNLI-m denotes MNLI with matched test sets, and MNLI-mm denotes MNLI with mismatched test sets. For all tasks, we report accuracy.**
|  | Method | MNLI-m | MNLI-mm | SST2 | QNLI | QQP | RTE | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LoRA | LoRA | 88.80 | 88.21 | 95.26 | 90.69 | 86.32 | 87.34 | 89.43 |
| FFA-LoRA | 88.83 | 88.24 | 94.91 | 91.72 | 86.61 | 86.61 | 89.48 |  |
| FedSA-LoRA | 90.20 | 88.89 | 95.92 | 92.00 | 87.85 | 88.04 | 90.48 |  |
| rsLoRA | rsLoRA | 88.87 | 88.38 | 95.25 | 91.61 | 87.16 | 86.38 | 89.60 |
| FFA-rsLoRA | 89.09 | 88.66 | 95.60 | 91.93 | 87.37 | 85.45 | 89.68 |  |
| FedSA-rsLoRA | 90.13 | 89.02 | 95.80 | 92.27 | 88.02 | 87.87 | 90.51 |  |
| VeRA | VeRA | 85.58 | 85.11 | 93.48 | 91.84 | 82.33 | 86.29 | 87.43 |
| FFA-VeRA | 86.03 | 86.17 | 93.46 | 92.47 | 82.33 | 82.86 | 87.26 |  |
| FedSA-VeRA | 87.31 | 86.60 | 93.63 | 92.83 | 82.64 | 87.22 | 88.37 |  |

The experimental results are shown in Table 1. From this table, we can observe that the proposed FedSA-LoRA, FedSA-rsLoRA, and FedSA-VeRA consistently outperform other methods across all tasks, demonstrating the effectiveness of the proposed method.

### 5.2 In-depth Analyses

In this section, we utilize LoRA-based methods to perform in-depth analyses on the natural language understanding tasks of QNLI, SST2, and MNLI-m to assess the impact of factors such as data heterogeneity, the number of clients, and LoRA rank on model performance.

#### 5.2.1 Effect of Data Heterogeneity

To investigate the effect of data heterogeneity on model performance, we model an IID partition (Split-1) and two non-IID partitions with Dir (1) and Dir (0.5). The latter two non-IID partitions are referred to as moderate non-IID (Split-2) and severe non-IID (Split-3). The training settings are the same as in Section 5.1.

**Table 2: Performance comparison on the QNLI, SST2, and MNLI-m tasks with various degrees of data heterogeneity.**
| Method | QNLI | SST2 | MNLI-m |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Split-1 | Split-2 | Split-3 | Split-1 | Split-2 | Split-3 | Split-1 | Split-2 | Split-3 |  |
| LoRA | 92.92 | 92.44 | 90.69 | 95.30 | 95.53 | 95.26 | 88.52 | 88.35 | 88.80 |
| FFA-LoRA | 92.68 | 92.29 | 91.72 | 95.87 | 95.47 | 94.91 | 88.15 | 88.03 | 88.83 |
| FedSA-LoRA | 92.95 | 93.32 | 92.00 | 96.10 | 96.24 | 95.92 | 89.57 | 89.71 | 90.20 |

The results are provided in Table 2. From these results, we can observe that the proposed FedSA-LoRA consistently outperforms other baselines, demonstrating its adaptability and robustness in various heterogeneous data scenarios. Additionally, as data heterogeneity increases, the improvement of the proposed method also increases. Specifically, FedSA-LoRA improves accuracy by 0.03%, 0.88%, and 1.84% on the QNLI task from IID to severe non-IID compared with LoRA, and by 1.05%, 1.36%, and 1.4% on the MNLI-m task. This indicates that the proposed method is more effective when non-IID conditions are more severe.

#### 5.2.2 Effect of Number of Clients

In Section 5.1, we demonstrated the effectiveness of the proposed method on a small number of clients, i.e., three clients. In this section, we show the superiority of FedSA-LoRA compared to other baselines on a larger number of clients, i.e., from 10 to 100 clients. Specifically, we use the same non-IID split, i.e., Dir (0.5), to divide the data into 10, 20, and 100 clients. The training settings are the same as in Section 5.1 and the results are shown in Table 3.

**Table 3: Performance comparison on the QNLI, SST2, and MNLI-m tasks with different number of clients. We apply full participation for FL system with 10 and 20 clients, and apply client sampling with rate 0.3 for FL system with 100 clients.**
| Method | QNLI | SST2 | MNLI-m |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10 clients | 20 clients | 100 clients | 10 clients | 20 clients | 100 clients | 10 clients | 20 clients | 100 clients |  |
| LoRA | 91.32 | 91.23 | 90.32 | 96.68 | 93.16 | 96.68 | 86.94 | 88.50 | 88.13 |
| FFA-LoRA | 91.47 | 91.70 | 91.27 | 96.59 | 93.31 | 96.33 | 86.76 | 88.60 | 87.86 |
| FedSA-LoRA | 91.97 | 92.54 | 91.48 | 96.83 | 94.21 | 97.02 | 88.59 | 89.05 | 88.82 |

It can be concluded that FedSA-LoRA not only outperforms other methods with a small number of clients (i.e., 3 clients) but also shows superior performance with a large number of clients (i.e., from 10 to 100 clients), demonstrating the adaptability and robustness of the proposed FedSA-LoRA across various client numbers.

#### 5.2.3 Effect of LoRA Rank

The adapter parameter budget (i.e., rank $r$) is a key factor in LoRA performance. In this section, we experiment with rank $r\in\{2,4,8,16\}$ on the QNLI, SST2, and MNLI-m tasks to test its influence on model performance, keeping other settings unchanged compared to Section 5.1.

**Table 4: Performance comparison on the QNLI, SST2, and MNLI-m tasks with different LoRA ranks $r$.**
| Rank | Method | QNLI | SST2 | MNLI-m | Rank | Method | QNLI | SST2 | MNLI-m |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $r=2$ | LoRA | 92.07 | 94.69 | 88.29 | $r=4$ | LoRA | 92.71 | 94.20 | 88.43 |  |
| FFA-LoRA | 91.02 | 93.95 | 87.98 | FFA-LoRA | 92.97 | 94.61 | 88.24 |  |  |  |
| FedSA-LoRA | 92.69 | 95.67 | 89.14 | FedSA-LoRA | 93.10 | 95.69 | 88.98 |  |  |  |
| $r=8$ | LoRA | 90.69 | 95.26 | 88.80 | $r=16$ | LoRA | 90.59 | 94.98 | 88.78 |  |
| FFA-LoRA | 91.72 | 94.91 | 88.83 | FFA-LoRA | 91.62 | 94.13 | 89.25 |  |  |  |
| FedSA-LoRA | 92.00 | 95.92 | 90.20 | FedSA-LoRA | 92.03 | 95.78 | 89.59 |  |  |  |

The results, as shown in Table 4, demonstrate that the proposed FedSA-LoRA outperforms other methods across various LoRA rank values, showcasing the adaptability and robustness of FedSA-LoRA in different scenarios.

### 5.3 Natural Language Generation

For the natural language generation tasks, we adopt the pre-trained LLaMA3-8B (Meta, ) from the HuggingFace Transformers library (Wolf et al., ), using the GSM8K dataset (Cobbe et al., ) for evaluation. Following (Kuang et al., ), we split the data into three clients under an IID distribution, and other optimization hyperparameters are the same as in that work.

The results shown in Table 5 demonstrate that the proposed FedSA-LoRA outperforms other methods in complex natural language generation tasks. From the given example, it can be seen that both LoRA and FFA-LoRA have reasoning errors, but FedSA-LoRA can reason accurately, demonstrating the superiority of the proposed method.

**Table 5: Performance of different methods on the GSM8K dataset and example answers generated by each method.**
|  | LoRA | FFA-LoRA | FedSA-LoRA |
| --- | --- | --- | --- |
| Accuracy | 46.24 | 46.32 | 46.63 |
| Question: | In a student council election, candidate A got 20% of the votes while candidate B got 50% more than candidate A’s votes. The rest of the votes was given to candidate C. If there were 100 voters, how many votes did candidate C get? |  |  |
| Output: | Candidate A got 20% of the votes. That is 20% of 100 = 20% $\times$ 100 = 20 $\times$ 100 = 2000 votes. Candidate B got 50% more than candidate A. That is 50% of 2000 = 50% $\times$ 2000 = 50 $\times$ 2000 = 10000 votes. The rest of the votes was given to candidate C. So there are 10000 + 2000 = 12000 votes in total. Candidate C got 10000 votes. The answer is 10000. [✘] | Candidate A got 20% of 100 = 100 $\times$ 20% = 100 $\times$ 20/100 = 20. Candidate B got 20% more than candidate A’s votes, which is 20% + 20% = 40%. 40% of 100 = 100 $\times$ 40% = 100 $\times$ 40/100 = 40. So the two candidates together got 20 + 40 = 60. The rest of the votes were given to candidate C. So candidate C got 100 - 60 = 40. The answer is 40. [✘] | Candidate A got 20% of the votes or 20% $*$ 100 = 20 votes. Then candidate B got 50% more than candidate A’s 20 votes so 50% of 20 is 20 $*$ 50% = 10. Then candidate B got 20 + 10 = 30 votes. So candidate C got 100 - 20 - 30 = 50 votes. The answer is 50. [✔] |

## 6 Conclusion

In this work, we discover that when combining LoRA with FL, $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge. Building upon this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. By sharing the $A$ matrices that learn general knowledge with the server for aggregation, while keeping the $B$ matrices that model client-specific knowledge locally, the learning abilities of LoRA combined with FL can be enhanced. Moreover, we explore the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. By doing so, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL.

## Appendix A Appendix

### A.1 Proof of Lemma 1

###### Proof.

Consider fine-tuning $B$ while freezing $A=Q$. The loss function in Eq. (3) becomes:

$$ $\mathcal{L}=\mathbb{E}_{(x_{t},y_{t})}[\|y_{t}-(W_{0}+BQ)x_{t}\|_{2}^{2}].$ (7) $$

Then, the gradient of Eq. (7) w.r.t. $B$ is:

$$ $\displaystyle\frac{\partial\mathcal{L}}{\partial B}$ $\displaystyle=\frac{\partial\mathbb{E}_{(x_{t},y_{t})}[\|y_{t}-(W_{0}+BQ)x_{t} \|_{2}^{2}]}{\partial B}$ (8) $\displaystyle=\frac{\partial\mathbb{E}[\|W_{t}x_{t}-(W_{0}+BQ)x_{t}\|_{2}^{2}] }{\partial B}$ $\displaystyle=\frac{\partial\mathbb{E}[\|(W_{0}+\Delta_{W})x_{t}-(W_{0}+BQ)x_{ t}\|_{2}^{2}]}{\partial B}$ $\displaystyle=\frac{\partial\mathbb{E}[\|(\Delta_{W}-BQ)x_{t}\|_{2}^{2}]}{ \partial B}$ $\displaystyle=\mathbb{E}[2[(\Delta_{W}-BQ)x_{t}](-x_{t}^{T}Q^{T})]$ $\displaystyle=\mathbb{E}[2(BQ-\Delta_{W})x_{t}x_{t}^{T}Q^{T}].$ $$

To obtain the optimal $B^{*}$, we set Eq. (8) to zero, which means:

$$ $\displaystyle\mathbb{E}[2(BQ-\Delta_{W})x_{t}x_{t}^{T}Q^{T}]$ $\displaystyle=0$ (9) $\displaystyle 2BQ\mathbb{E}[x_{t}x_{t}^{T}]Q^{T}-2\Delta_{W}\mathbb{E}[x_{t}x_ {t}^{T}]Q^{T}$ $\displaystyle=0$ $\displaystyle 2BQ\mathbb{E}[x_{t}x_{t}^{T}]Q^{T}-2\Delta_{W}\mathbb{E}[x_{t}x_ {t}^{T}]Q^{T}$ $\displaystyle=0$ $\displaystyle BQ\mathbb{E}[x_{t}x_{t}^{T}]Q^{T}$ $\displaystyle=\Delta_{W}\mathbb{E}[x_{t}x_{t}^{T}]Q^{T}$ $\displaystyle B$ $\displaystyle=\Delta_{W}\mathbb{E}[x_{t}x_{t}^{T}]Q^{T}(Q\mathbb{E}[x_{t}x_{t} ^{T}]Q^{T})^{-1}.$ $$

Therefore, we obtain $B^{*}=\Delta_{W}\mathbb{E}[x_{t}x_{t}^{T}]Q^{T}(Q\mathbb{E}[x_{t}x_{t}^{T}]Q^{
T})^{-1}$.

When fine-tuning $A$ with fixed $B=U$. The loss function in Eq. (3) becomes:

$$ $\mathcal{L}=\mathbb{E}_{(x_{t},y_{t})}[\|y_{t}-(W_{0}+UA)x_{t}\|_{2}^{2}].$ (10) $$

Then, the gradients of Eq. (7) w.r.t. $A$ is:

$$ $\displaystyle\frac{\partial\mathcal{L}}{\partial A}$ $\displaystyle=\frac{\partial\mathbb{E}_{(x_{t},y_{t})}[\|y_{t}-(W_{0}+UA)x_{t} \|_{2}^{2}]}{\partial A}$ (11) $\displaystyle=\frac{\partial\mathbb{E}[\|W_{t}x_{t}-(W_{0}+UA)x_{t}\|_{2}^{2}] }{\partial A}$ $\displaystyle=\frac{\partial\mathbb{E}[\|(W_{0}+\Delta_{W})x_{t}-(W_{0}+BQ)x_{ t}\|_{2}^{2}]}{\partial A}$ $\displaystyle=\frac{\partial\mathbb{E}[\|(\Delta_{W}-UA)x_{t}\|_{2}^{2}]}{ \partial A}$ $\displaystyle=\mathbb{E}[2U^{T}[(\Delta_{W}-UA)x_{t}]x_{t}^{T}]$ $$

To obtain the optimal $A^{*}$, we set Eq. (11) to zero, which means:

$$ $\displaystyle\mathbb{E}[2U^{T}[(\Delta_{W}-UA)x_{t}]x_{t}^{T}]$ $\displaystyle=0$ (12) $\displaystyle 2U^{T}\Delta_{W}\mathbb{E}[x_{t}x_{t}^{T}]-2U^{T}UA\mathbb{E}[x_ {t}x_{t}^{T}]$ $\displaystyle=0$ $\displaystyle U^{T}UA\mathbb{E}[x_{t}x_{t}^{T}]$ $\displaystyle=U^{T}\Delta_{W}\mathbb{E}[x_{t}x_{t}^{T}]$ $\displaystyle A$ $\displaystyle=U^{-1}\Delta_{W}.$ $$

Thus, we obtain $A^{*}=U^{-1}\Delta_{W}$.
∎

###### Proof.

### A.2 Proof of Theorem 1

###### Proof.

Let $W_{i}^{(t)}=W_{0}+B_{i}^{(t)}A_{i}^{(t)}$ be the model parameters maintained in the $i$-th client at the $t$-th step. Let $\mathcal{I}_{E}$ be the set of global synchronization steps, i.e., $\mathcal{I}_{E}=\{nE\mid n=1,2,\cdots\}$. If $t+1\in\mathcal{I}_{E}$, which represents the time step for communication, then the one-step update of the proposed method for the $i$-th client can be described as follows:

$$ $\left(\begin{array}[]{c}B_{i}^{(t)}\\ A_{i}^{(t)}\end{array}\right)\underset{\text{update of }B_{i}^{(t)}\text{and}A _{i}^{(t)}}{\longrightarrow}\left(\begin{array}[]{c}B_{i}^{(t+1)}\\ A_{i}^{(t+1)}\end{array}\right)\underset{\text{ if }t+1\in\mathcal{I}_{E}}{ \longrightarrow}\left(\begin{array}[]{c}B_{i}^{(t+1)}\\ \frac{1}{m}\sum_{j=1}^{m}A_{j}^{(t+1)}\end{array}\right).$ $$

For convenience, we denote the parameters in each sub-step above as follows:

$$ $\displaystyle W_{i}^{(t)}$ $\displaystyle=W_{0}+B_{i}^{(t)}A_{i}^{(t)},$ $\displaystyle U_{i}^{(t)}$ $\displaystyle=W_{0}+B_{i}^{(t+1)}A_{i}^{(t+1)},$ $\displaystyle V_{i}^{(t)}$ $\displaystyle=W_{0}+B_{i}^{(t+1)}\frac{1}{m}\sum_{j=1}^{m}A_{j}^{(t+1)},$ $\displaystyle W_{i}^{(t+1)}$ $\displaystyle=\begin{cases}U_{i}^{(t)}&\text{ if }t+1\notin\mathcal{I}_{E},\\ V_{i}^{(t)}&\text{ if }t+1\in\mathcal{I}_{E}.\end{cases}$ $$

Here, the variable $U_{i}^{(t)}$ represents the immediate result of one sub-step update from the parameter of the previous sub-step $W_{i}^{(t)}$, and $V_{i}^{(t)}$ represents the parameter obtained after communication steps (if applicable).
Furthermore, we denote the learning rate for the $i$-th client at the $t$-th step as $\eta_{i,t}$, and the stochastic gradient at step $t$ as follows:

$$ $\displaystyle\nabla_{B}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t})$ $\displaystyle=\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t}){A_{i}^{(t)}}^{ \top},$ $\displaystyle\nabla_{A}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t})$ $\displaystyle={B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i ,t}),$ $\displaystyle\nabla_{B}\mathcal{L}_{i}(W_{i}^{(t)})$ $\displaystyle=\mathbb{E}[\nabla_{B}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t})],$ $\displaystyle\nabla_{A}\mathcal{L}_{i}(W_{i}^{(t)})$ $\displaystyle=\mathbb{E}[\nabla_{A}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t})],$ $$

where $\xi_{i,t}$ is the data uniformly chosen from the local data set of client $i$ at step $t$.

Next, we apply the inequality from the smoothness Assumption 1 to each sub-step of the one-step update for client $i$.
Firstly, by the smoothness of $\mathcal{L}_{i}$, we have:

$$ $\mathcal{L}_{i}(U_{i}^{(t)})\leq\mathcal{L}_{i}(W_{i}^{(t)})+\langle U_{i}^{(t )}-W_{i}^{(t)},\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\rangle_{F}+\frac{L}{2} \left\|U_{i}^{(t)}-W_{i}^{(t)}\right\|^{2}_{F}.$ (13) $$

Since

$$ $\displaystyle B_{i}^{(t+1)}$ $\displaystyle=B_{i}^{(t)}-\eta_{i,t}\nabla_{B}\mathcal{L}_{i}(W_{i}^{(t)},\xi_ {i,t})$ $\displaystyle=B_{i}^{(t)}-\eta_{i,t}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_ {i,t}){A_{i}^{(t)}}^{\top},$ $$

and

$$ $\displaystyle A_{i}^{(t+1)}$ $\displaystyle=A_{i}^{(t)}-\eta_{i,t}\nabla_{A}\mathcal{L}_{i}(W_{i}^{(t)},\xi_ {i,t})$ $\displaystyle=A_{i}^{(t)}-\eta_{i,t}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_ {i}(W_{i}^{(t)},\xi_{i,t}),$ $$

we have:

$$ $\displaystyle U_{i}^{(t)}-W_{i}^{(t)}$ $\displaystyle=$ $\displaystyle B_{i}^{(t+1)}A_{i}^{(t+1)}-B_{i}^{(t)}A_{i}^{(t)}$ $\displaystyle=$ $\displaystyle\left(B_{i}^{(t)}-\eta_{i,t}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)} ,\xi_{i,t}){A_{i}^{(t)}}^{\top}\right)\left(A_{i}^{(t)}-\eta_{i,t}{B_{i}^{(t)} }^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t})\right)-B_{i}^{(t)}A_{ i}^{(t)}$ $\displaystyle=$ $\displaystyle\eta_{i,t}^{2}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t}){A_ {i}^{(t)}}^{\top}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi _{i,t})-\eta_{i,t}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t}){A_{i}^{(t)} }^{\top}A_{i}^{(t)}$ $\displaystyle-\eta_{i,t}B_{i}^{(t)}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{ i}(W_{i}^{(t)},\xi_{i,t}).$ $$

Then, for the second term on the right side of Eq. (13), according to the law of total expectation, we have:

$$ $\displaystyle\mathbb{E}\left[\left\langle U_{i}^{(t)}-W_{i}^{(t)},\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)})\right\rangle_{F}\right]$ $\displaystyle=\$ $\displaystyle\eta_{i,t}^{2}\mathbb{E}\left[\left\langle\nabla_{W}\mathcal{L}_{ i}(W_{i}^{(t)},\xi_{i,t}){A_{i}^{(t)}}^{\top}{B_{i}^{(t)}}^{\top}\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t}),\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}) \right\rangle_{F}\right]$ $\displaystyle-\eta_{i,t}\mathbb{E}\left[\left\langle\nabla_{W}\mathcal{L}_{i}( W_{i}^{(t)},\xi_{i,t}){A_{i}^{(t)}}^{\top}A_{i}^{(t)},\nabla_{W}\mathcal{L}_{i }(W_{i}^{(t)})\right\rangle_{F}\right]$ $\displaystyle-\eta_{i,t}\mathbb{E}\left[\left\langle B_{i}^{(t)}{B_{i}^{(t)}}^ {\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t}),\nabla_{W}\mathcal{L}_{ i}(W_{i}^{(t)})\right\rangle_{F}\right]$ $\displaystyle=\$ $\displaystyle\eta_{i,t}^{2}\left\langle\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}){ A_{i}^{(t)}}^{\top}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}), \nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\rangle_{F}$ $\displaystyle-\eta_{i,t}\left\langle\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}){A_{ i}^{(t)}}^{\top}A_{i}^{(t)},\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right \rangle_{F}$ $\displaystyle-\eta_{i,t}\left\langle B_{i}^{(t)}{B_{i}^{(t)}}^{\top}\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)}),\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right \rangle_{F}.$ $$

Since

$$ $\displaystyle\left\langle\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}){A_{i}^{(t)}}^{ \top}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}),\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)})\right\rangle_{F}$ $\displaystyle\leq\$ $\displaystyle\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}){A_{i}^{(t)}}^{\top} {B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|\left\| \nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}$ $\displaystyle\leq\$ $\displaystyle\left\|{A_{i}^{(t)}}\right\|\left\|{B_{i}^{(t)}}\right\|_{F}\left \|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{3}$ $\displaystyle\leq\$ $\displaystyle C_{A}C_{B}G^{3},$ $$

and if we
assume there exists $c_{A}>0$ such that $\forall t$:

$$ $\Big{\langle}{A_{i}^{(t)}}^{\top}A_{i}^{(t)},\nabla_{W}\mathcal{L}_{i}(W_{i}^{ (t)})^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\Big{\rangle}_{F}\geq c_{A} \left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{2},$ $$

then we have:

$$ $\displaystyle\left\langle\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}){A_{i}^{(t)}}^{ \top}A_{i}^{(t)},\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\rangle_{F}=\$ $\displaystyle\text{Tr}\Big{[}\Big{(}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}){A_{ i}^{(t)}}^{\top}A_{i}^{(t)}\Big{)}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)} )\Big{]}$ $\displaystyle=\$ $\displaystyle\text{Tr}\Big{[}{A_{i}^{(t)}}^{\top}A_{i}^{(t)}\nabla_{W}\mathcal {L}_{i}(W_{i}^{(t)})^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\Big{]}$ $\displaystyle=\$ $\displaystyle\Big{\langle}{A_{i}^{(t)}}^{\top}A_{i}^{(t)},\nabla_{W}\mathcal{L }_{i}(W_{i}^{(t)})^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\Big{\rangle}_{F}$ $\displaystyle\geq\$ $\displaystyle c_{A}\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{ 2}.$ $$

And similarly if we assume there exists $c_{B}>0$ such that $\forall t$:

$$ $\Big{\langle}{B_{i}^{(t)}}^{\top}B_{i}^{(t)},\nabla_{W}\mathcal{L}_{i}(W_{i}^{ (t)})^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\Big{\rangle}_{F}\geq c_{B} \left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{2},$ $$

then we have:

$$ $\displaystyle\left\langle B_{i}^{(t)}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L} _{i}(W_{i}^{(t)}),\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\rangle_{F}=\$ $\displaystyle\text{Tr}\Big{[}\Big{(}B_{i}^{(t)}{B_{i}^{(t)}}^{\top}\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)})\Big{)}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t) })\Big{]}$ $\displaystyle=\$ $\displaystyle\text{Tr}\Big{[}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})^{\top}B_{i }^{(t)}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\Big{]}$ $\displaystyle=\$ $\displaystyle\text{Tr}\Big{[}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)})^{\top}B_{i}^{(t)}{B_{i}^{(t)}}^{\top}\Big{]}$ $\displaystyle=\$ $\displaystyle\left\langle\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)})^{\top},B_{i}^{(t)}{B_{i}^{(t)}}^{\top}\right \rangle_{F}$ $\displaystyle\geq\$ $\displaystyle c_{B}\Big{\|}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\Big{\|}_{F}^ {2},$ (14) $$

where we use the cyclic property of the trace for the third equality above.
We further get:

$$ $\displaystyle\mathbb{E}\left[\left\langle U_{i}^{(t)}-W_{i}^{(t)},\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)})\right\rangle_{F}\right]$ $\displaystyle\leq\$ $\displaystyle\eta_{i,t}^{2}C_{A}C_{B}G^{3}-\eta_{i,t}c_{A}^{2}\left\|\nabla_{W }\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{2}-\eta_{i,t}c_{B}^{2}\left\|\nabla _{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{2}.$ (15) $$

Similarly, we know:

$$ $\displaystyle\mathbb{E}[\|U_{i}^{(t)}-W_{i}^{(t)}\|_{F}^{2}]=\$ $\displaystyle\mathbb{E}[\|\eta_{i,t}^{2}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}, \xi_{i,t}){A_{i}^{(t)}}^{\top}{B_{i}^{(t)}}^{\top}\nabla_{W}\mathcal{L}_{i}(W_ {i}^{(t)},\xi_{i,t})$ $\displaystyle-\eta_{i,t}\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t}){A_{i} ^{(t)}}^{\top}A_{i}^{(t)}-\eta_{i,t}B_{i}^{(t)}{B_{i}^{(t)}}^{\top}\nabla_{W} \mathcal{L}_{i}(W_{i}^{(t)},\xi_{i,t})\|_{F}^{2}]$ $\displaystyle\leq\$ $\displaystyle 3\eta_{i,t}^{4}C_{A}^{2}C_{B}^{2}G^{4}+3\eta_{i,t}^{2}C_{A}^{2}G ^{2}+3\eta_{i,t}^{2}C_{B}^{2}G^{2}.$ (16) $$

Plugging Eq. (A.2), Eq. (A.2), and Eq. (A.2) into Eq. (13), we have:

$$ $\displaystyle\mathcal{L}_{i}\left(U_{i}^{(t)}\right)\leq\$ $\displaystyle\mathcal{L}_{i}\left(W_{i}^{(t)}\right)+\eta_{i,t}^{2}C_{A}C_{B}G ^{3}-\eta_{i,t}c_{A}\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^ {2}-\eta_{i,t}c_{B}\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{2}$ $\displaystyle+\frac{3}{2}\eta_{i,t}^{4}C_{A}^{2}C_{B}^{2}G^{4}L+\frac{3}{2} \eta_{i,t}^{2}C_{A}^{2}G^{2}L+\frac{3}{2}\eta_{i,t}^{2}C_{B}^{2}G^{2}L.$ (17) $$

Secondly, by the smoothness of $\mathcal{L}_{i}$, we have:

$$ $\mathcal{L}_{i}\left(V_{i}^{(t)}\right)\leq\mathcal{L}_{i}\left((U_{i}^{(t)} \right)+\left\langle V_{i}^{(t)}-U_{i}^{(t)},\nabla_{W}\mathcal{L}_{i}(U_{i}^{ (t)})\right\rangle_{F}+\frac{L}{2}\left\|V_{i}^{(t)}-U_{i}^{(t)}\right\|^{2}_{ F}.$ (18) $$

Since

$$ $\displaystyle V_{i}^{(t)}-U_{i}^{(t)}$ $\displaystyle=B_{i}^{(t+1)}A_{i}^{(t+1)}-\frac{1}{m}B_{i}^{(t+1)}\sum_{j=1}^{m }A_{j}^{(t+1)}$ $\displaystyle=B_{i}^{(t+1)}\frac{1}{m}\sum_{j=1}^{m}\Big{(}A_{i}^{(t+1)}-A_{j} ^{(t+1)}\Big{)},$ $$

and

$$ $\displaystyle A_{j}^{(t+1)}$ $\displaystyle=A_{j}^{(t-E+1)}-\sum_{t_{0}=t-E+1}^{t}\eta_{j,t_{0}}\nabla_{A} \mathcal{L}_{j}(W_{j}^{(t_{0})};\xi_{j,t_{0}})$ $\displaystyle=A_{j}^{(t-E+1)}-\sum_{t_{0}=t-E+1}^{t}\eta_{j,t_{0}}{B_{j}^{(t_{ 0})}}^{\top}\nabla_{W}\mathcal{L}_{j}(W_{j}^{(t_{0})};\xi_{j,t_{0}}),$ $$

we know:

$$ $\displaystyle V_{i}^{(t)}-U_{i}^{(t)}=B_{i}^{(t+1)}\Big{(}-\frac{1}{m}\sum_{j= 1}^{m}\sum_{t_{0}=t-E+1}^{t}\eta_{j,t_{0}}{B_{j}^{(t_{0})}}^{\top}\big{(} \nabla\mathcal{L}_{i}(W_{i}^{(t_{0})};\xi_{i,t_{0}})-\nabla\mathcal{L}_{j}(W_{ j}^{(t_{0})};\xi_{j,t_{0}})\big{)}\Big{)}.$ $$

Therefore,

$$ $\displaystyle\mathbb{E}\Big{[}\Big{\|}V_{i}^{(t)}-U_{i}^{(t)}\Big{\|}_{F}^{2} \Big{]}$ $\displaystyle=\$ $\displaystyle\mathbb{E}\Big{[}\Big{\|}B_{i}^{(t+1)}\Big{\|}_{F}^{2}\Big{\|}- \frac{1}{m}\sum_{j=1}^{m}\sum_{t_{0}=t-E+1}^{t}\eta_{j,t_{0}}{B_{j}^{(t_{0})}} ^{\top}\big{(}\nabla\mathcal{L}_{i}(W_{i}^{(t_{0})};\xi_{i,t_{0}})-\nabla \mathcal{L}_{j}(W_{j}^{(t_{0})};\xi_{j,t_{0}})\big{)}\Big{\|}_{F}^{2}\Big{]}$ $\displaystyle\leq\$ $\displaystyle C_{B}^{2}\frac{E}{m}\sum_{j=1}^{m}\sum_{t_{0}=t-E+1}^{t}\eta_{j, t_{0}}^{2}\mathbb{E}\Big{[}\Big{\|}{B_{j}^{(t_{0})}}^{\top}\big{(}\nabla \mathcal{L}_{i}(W_{i}^{(t_{0})};\xi_{i,t_{0}})-\nabla\mathcal{L}_{j}(W_{j}^{(t _{0})};\xi_{j,t_{0}})\big{)}\Big{\|}_{F}^{2}\Big{]}$ $\displaystyle\leq\$ $\displaystyle\frac{C_{B}^{4}E}{m}\sum_{j=1}^{m}\sum_{t_{0}=t-E+1}^{t}\eta_{j,t _{0}}^{2}\mathbb{E}\Big{[}\Big{\|}\nabla\mathcal{L}_{i}(W_{i}^{(t_{0})};\xi_{i ,t_{0}})-\nabla\mathcal{L}_{j}(W_{j}^{(t_{0})};\xi_{j,t_{0}})\Big{\|}_{F}^{2} \Big{]}$ $\displaystyle\leq\$ $\displaystyle\frac{4C_{B}^{4}EG^{2}}{m}\sum_{j=1}^{m}\sum_{t_{0}=t-E+1}^{t} \eta_{j,t_{0}}^{2},$ (19) $$

where we use Assumption 2 to derive that:

$$ $\displaystyle\mathbb{E}\Big{[}\Big{\|}\nabla\mathcal{L}_{i}(W_{i}^{(t_{0})}; \xi_{i,t_{0}})-\nabla\mathcal{L}_{j}(W_{j}^{(t_{0})};\xi_{j,t_{0}})\Big{\|}_{F }^{2}\Big{]}$ $\displaystyle\leq\$ $\displaystyle 2\mathbb{E}\Big{[}\Big{\|}\nabla\mathcal{L}_{i}(W_{i}^{(t_{0})}; \xi_{i,t_{0}})\Big{\|}_{F}^{2}\Big{]}+2\mathbb{E}\Big{[}\Big{\|}\nabla\mathcal {L}_{i}(W_{j}^{(t_{0})};\xi_{j,t_{0}})\Big{\|}_{F}^{2}\Big{]}$ $\displaystyle\leq\$ $\displaystyle 4G^{2}.$ $$

Furthermore,

$$ $\displaystyle\left\langle V_{i}^{(t)}-U_{i}^{(t)},\nabla_{W}L_{i}(U_{i}^{(t)}) \right\rangle\leq\$ $\displaystyle\frac{1}{2\eta_{i,t}}\Big{\|}V_{i}^{(t)}-U_{i}^{(t)}\Big{\|}_{F}^ {2}+\frac{1}{2}\eta_{i,t}\Big{\|}\nabla_{W}\mathcal{L}_{i}(U_{i}^{(t)})\Big{\| }_{F}^{2}$ $\displaystyle\leq\$ $\displaystyle\frac{2C_{B}^{4}EG^{2}}{\eta_{i,t}m}\sum_{j=1}^{m}\sum_{t_{0}=t-E +1}^{t}\eta_{j,t_{0}}^{2}+\frac{1}{2}\eta_{i,t}G^{2}.$ (20) $$

Plugging Eq. (A.2) and Eq. (A.2) into Eq. (18), we have (choose constant learning rate $\eta_{i,t}=\eta$):

$$ $\displaystyle\mathcal{L}_{i}\left(V_{i}^{(t)}\right)\leq\mathcal{L}_{i}\left(U _{i}^{(t)}\right)+\big{(}2C_{B}^{2}E^{2}G^{2}+\frac{1}{2}G^{2}\big{)}\eta+2 \eta^{2}C_{B}^{4}E^{2}G^{2}L.$ (21) $$

Combining Eq. (A.2) and Eq. (21), we have (choose constant learning rate $\eta_{i,t}=\eta$):

$$ $\displaystyle\mathcal{L}_{i}\left(W_{i}^{(t+1)}\right)\leq\$ $\displaystyle\mathcal{L}_{i}\left((W_{i}^{(t)}\right)+(2C_{B}^{2}E^{2}G^{2}+ \frac{1}{2}G^{2})\eta+\frac{3}{2}\eta^{4}C_{A}^{2}C_{B}^{2}G^{4}L$ $\displaystyle+(C_{A}C_{B}G^{2}+\frac{3}{2}C_{A}^{2}LG^{2}+\frac{3}{2}C_{B}^{2} LG^{2}+2C_{B}^{4}E^{2}LG^{2})\eta^{2}$ $\displaystyle-\eta(c_{A}+c_{B})\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}) \right\|^{2},$ $$

which is equivalent to:

$$ $\displaystyle\eta(c_{A}+c_{B})\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)}) \right\|_{F}^{2}\leq\$ $\displaystyle\mathcal{L}_{i}\left((W_{i}^{(t)}\right)-\mathcal{L}_{i}\left(W_{ i}^{(t+1)}\right)$ $\displaystyle+(2C_{B}^{2}E^{2}G^{2}+\frac{1}{2}G^{2})\eta+\frac{3}{2}\eta^{4}C _{A}^{2}C_{B}^{2}G^{4}L$ $\displaystyle+(C_{A}C_{B}G^{2}+\frac{3}{2}C_{A}^{2}LG^{2}+\frac{3}{2}C_{B}^{2} LG^{2}+2C_{B}^{4}E^{2}LG^{2})\eta^{2}.$ $$

Choosing $M$ which satisfies $(2C_{B}^{2}E^{2}G^{2}+\frac{1}{2}G^{2})\eta+\frac{3}{2}\eta^{4}C_{A}^{2}C_{B}^
{2}G^{4}L+(C_{A}C_{B}G^{2}+\frac{3}{2}C_{A}^{2}LG^{2}+\frac{3}{2}C_{B}^{2}LG^{
2}+2C_{B}^{4}E^{2}LG^{2})\eta^{2}\leq M\eta^{2}$, we get:

$$ $\displaystyle\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right\|_{F}^{2}\leq\$ $\displaystyle\frac{\mathcal{L}_{i}\left((W_{i}^{(t)}\right)-\mathcal{L}_{i} \left(W_{i}^{(t+1)}\right)}{\eta(c_{A}+c_{B})}+\frac{M\eta}{c_{A}+c_{B}}.$ (22) $$

Now, by repeatedly applying Eq. (22) for different values of $t$ and summing up the results, we get:

$$ $\displaystyle\sum_{t=1}^{T}\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{(t)})\right \|_{F}^{2}\leq$ $\displaystyle\frac{\mathcal{L}_{i}\left((W_{i}^{(1)}\right)-\mathcal{L}_{i} \left(W_{i}^{*}\right)}{\eta(c_{A}^{2}+c_{B}^{2})}+\eta\frac{M}{c_{A}+c_{B}}T.$ (23) $$

Dividing both side of Eq. (23) by $T$, we get:

$$ $\displaystyle\frac{1}{T}\sum_{t=1}^{T}\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{ (t)})\right\|_{F}^{2}\leq$ $\displaystyle\frac{\mathcal{L}_{i}\left((W_{i}^{(1)}\right)-\mathcal{L}_{i} \left(W_{i}^{*}\right)}{\eta(c_{A}+c_{B})T}+\eta\frac{M}{c_{A}+c_{B}}.$ (24) $$

Let us assume that $\mathcal{L}_{i}(W_{i}^{(1)})-\mathcal{L}_{i}(W_{i}^{*})\leq D,\forall i$, and we set $\eta=\sqrt{\frac{D}{MT}}$. Then, we have:

$$ $\displaystyle\frac{1}{T}\sum_{t=1}^{T}\left\|\nabla_{W}\mathcal{L}_{i}(W_{i}^{ (t)})\right\|_{F}^{2}\leq$ $\displaystyle\frac{2}{c_{A}+c_{B}}\sqrt{\frac{DM}{T}}.$ (25) $$

Thus, we can obtain:

$$ $\displaystyle\frac{1}{mT}\sum_{i=1}^{m}\sum_{t=1}^{T}\left\|\nabla_{W}\mathcal {L}_{i}(W_{i}^{(t)})\right\|_{F}^{2}\leq$ $\displaystyle\frac{2}{c_{A}+c_{B}}\sqrt{\frac{DM}{T}}.$ (26) $$

∎

###### Proof.

### A.3 Hyperparameters

Tables 6 and 7 show the learning rates used for LoRA-based methods and rsLoRA-based methods, respectively. For the VeRA-based methods, we first tried using the SGD optimizer (Ruder, ) with a search learning rate from $\eta\in\{5\text{E-}3,1\text{E-}2,2\text{E-}2,5\text{E-}2,1\text{E-}1\}$ as adopted in LoRA-based methods, but we found the performance to be significantly worse than that of LoRA-based methods. For example, the best performance among the three VeRA-based methods (i.e., VeRA, FFA-VeRA, and FedSA-VeRA) is 53.73% on the MNLI-m task, which is significantly worse than that of the LoRA-based methods (90.20%). Thus, we chose the AdamW optimizer (Loshchilov & Hutter, ) and introduced separate learning rates for the classification head and the adapted layers as used in VeRA (Kopiczko et al., ). The learning rates used for VeRA-based methods are shown in Table 8.

**Table 6: The learning rates used for LoRA-based methods on the GLUE benchmark.**
| Method | MNLI-m | MNLI-mm | SST2 | QNLI | QQP | RTE |
| --- | --- | --- | --- | --- | --- | --- |
| LoRA | 1E-2 | 1E-2 | 2E-2 | 1E-2 | 1E-2 | 1E-2 |
| FFA-LoRA | 5E-2 | 5E-2 | 5E-2 | 2E-2 | 5E-2 | 2E-2 |
| FedSA-LoRA | 2E-2 | 2E-2 | 1E-2 | 5E-3 | 2E-2 | 1E-2 |

**Table 7: The learning rates used for rsLoRA-based methods on the GLUE benchmark.**
| Method | MNLI-m | MNLI-mm | SST2 | QNLI | QQP | RTE |
| --- | --- | --- | --- | --- | --- | --- |
| rsLoRA | 5E-3 | 5E-3 | 1E-2 | 2E-3 | 5E-3 | 2E-3 |
| FFA-rsLoRA | 2E-2 | 2E-2 | 2E-2 | 1E-2 | 2E-2 | 1E-2 |
| FedSA-rsLoRA | 5E-3 | 5E-3 | 5E-3 | 1E-3 | 2E-3 | 2E-3 |

**Table 8: The learning rates used for VeRA-based methods on the GLUE benchmark.**
| Method | Position | MNLI-m | MNLI-mm | SST2 | QNLI | QQP | RTE |
| --- | --- | --- | --- | --- | --- | --- | --- |
| VeRA | VeRA | 1E-2 | 1E-2 | 2E-2 | 2E-3 | 2E-3 | 1E-2 |
| Head | 6E-3 | 6E-3 | 2E-3 | 3E-4 | 3E-4 | 2E-4 |  |
| FFA-VeRA | VeRA | 2E-2 | 2E-2 | 1E-2 | 1E-2 | 1E-2 | 1E-2 |
| Head | 2E-3 | 2E-3 | 6E-3 | 2E-4 | 6E-3 | 2E-4 |  |
| FedSA-VeRA | VeRA | 2E-3 | 2E-3 | 1E-2 | 1E-2 | 2E-3 | 1E-2 |
| Head | 3E-5 | 3E-5 | 3E-4 | 3E-4 | 3E-4 | 1E-4 |  |

### A.4 Learned Matrices Comparison

#### A.4.1 Learned and Initialized LoRA Matrices Comparison

Since the learned LoRA $A$ matrices are similar across different clients in Figure 2, we illustrate the difference between the learned and initialized $A$ matrices for each client under the IID partition in this section. The results, shown in Figure 3, confirm that the $A$ matrices are updated.

Figure: (a) Client 1 & Init
Refer to caption: x8.png

#### A.4.2 Learned rsLoRA Matrices Comparison

In this section, we present the mean of pairwise client relationships for the learned rsLoRA (Kalajdzievski, ) matrices. These results, shown in Figure 4, demonstrate a similar phenomenon to the learned LoRA matrices. That is, the learned $A$ matrices are more similar across clients than the $B$ matrices, and with increased data heterogeneity, the similarity of $B$ matrices between different clients decreases.

Figure: (a) IID
Refer to caption: x14.png

#### A.4.3 Learned VeRA Matrices Comparison

In this section, we show the mean of pairwise client relationships for the learned VeRA (Kopiczko et al., ) matrices. In VeRA, the low-rank matrices $A$ and $B$ are initialized using the uniform version of Kaiming initialization, fixed, shared across all layers, and adapted with trainable scaling vectors $d$ and $b$. The $b$ vectors are initialized to zero, and the $d$ vectors are initialized with a value of 0.1. To make the notation consistent with our work, we rewrite the scaling vectors $d$ and $b$ as $A_{d}$ and $B_{b}$ to reflect the position of each scaling vector. These results, illustrated in Figure 5, demonstrate a similar phenomenon to the learned LoRA matrices. That is, the learned scaling vectors $A_{d}$ are more similar across clients than the scaling vectors $B_{b}$, and with increased data heterogeneity, the similarity of scaling vectors $B_{b}$ between different clients decreases.

Figure: (a) IID
Refer to caption: x20.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to Weakness.
soundness: 2
strengths: This paper has the interesting motivation and shows the different roles of low-rank matrices $A$ and $B$ in federated fine-tuning.
summary: This paper proposes a low-rank federated fine-tuning algorithm, namely FedSA-LoRA, which regards $A$ as a global adapter and $B$ as a personalized adapter. This algorithm reduces the communication cost by only aggregating matrices $A$.
weaknesses: I have the following concerns:

1. Lemma 1 is very interesting to me and the verified experiments show that all the matrices $A_i$ ($i$ is the client index) seems to be the same and $B_i$ differs with each other. We know that different client has independent initialization of $A$, but finally with the exactly same $A$ since the similarity of $A$ from clients is $1.0$ in Figure 2. As you mentioned in Figure 3 in Appendix that the learned A matrices are different from the initialized A matrices, so how do you guarantee that the learned A across clients are the totally same even in the server non-IID scenario.

2. The implementation details for Figure 2 are not clear to me, especially how do you implement federated lora fine-tuning, do you update/aggregate both $A$ and $B$?

3. In Assumption 1, what is the meaning of $\mathcal{L}_i(W_j)$ and $\nabla \mathcal{L}_i(W_j)$? In your descriptions, both $i$ and $j$ are the client's index, right? Then how can you access the client-$j$'s gradient information in client $i$?  

4. Assumption 2 is too strong, it becomes very easy to handle dissimilarity between clients and stochastic gradients with this assumption. This assumption doesn't appear in smoothness cases in many federated learning papers, so how do you verify this assumption? Can you provide examples to verify the rationality.

5. As I mentioned in concern 4, assumption 2 is a strong assumption, which does not appear in work [1]. Even the same convergence rate as [1] is achieved, that comparison is meaningless to me. 

[1] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020.

6. In experiments, they compare with FFA-LoRA, and show better performance than FFA-LoRA. My concern is that, is this a fair comparison? Because the trainable parameters are not the same, FFA-LoRA does not tune the matrices $A$, but FedSA-LoRA does, which means FedSA-LoRA has much more trainable parameters than FFA-LoRA. I hope to see the report of trainable parameters for each algorithm. 

7. Many advanced baselines are not compared, so the performance of the proposed algorithm is not convincing. For example, some related work mentioned in the paper, FedLoRA [2], FedDPA [3], and HetLoRA [4]. 

[2] Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneous personalized federated learning with lora tuning. arXiv preprint arXiv:2310.13283, 2023. \
[3] Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein. Dual-personalizing adapter for federated foundation models. arXiv preprint arXiv:2403.19211, 2024.\
[4] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See above. I will increase my score if my concerns are well addressed.
soundness: 3
strengths: 1. The paper is generally well-motivated and easy to follow.
2. This work introduces the shared-A LoRA technique into the FL framework, making it promising compared to previous work where the down-projection matrix (A matrix) is kept frozen.
3. Comprehensive experimental results are provided.
summary: In this paper, the authors present FedSA-LoRA, a novel approach for federated learning that shares only the A matrices across clients to leverage the functional differences between A and B matrices in LoRA modules. The method’s effectiveness is validated through both theoretical analysis and empirical results.
weaknesses: 1. While this work introduces a shared-A LoRA framework into FL, such a technique has already been introduced in the MoE area [1] under similar motivations. Although the novelty of introducing it into FL should be recognized, the introduction of such a framework cannot be solely credited to this work.

2. On page 20, Figure 3, the authors claim that "A matrices are different from the initialized A matrices, indicating that the A matrices are updated." However, the cosine similarity between the learned and initialized down-projection matrices remains quite high (greater than 0.985), which indicates that the A matrices have changed very little compared to the initialization.

3. Compared with [2], the cosine similarity across A matrices for different clients reported in this work is very high. Why does this happen? Is it caused by different levels of data heterogeneity?

4. Assumption 3, especially the last two inequalities, lacks verification and explanation. It appears these are purely for simplifying the proof of Theorem 1 on page 16. The authors need to elaborate more on this uncommon assumption, including under what conditions it could hold and what properties of the A and B matrices this assumption reveals.

5. In the provided code, alongside the B matrices being personalized, a classifier layer is also personalized. Is this necessary? How does this personalized classifier layer influence the experimental results?

References:

[1] Tian, Chunlin, et al. "HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning." arXiv preprint arXiv:2404.19245 (2024).

[2] Zhu, Jiacheng, et al. "Asymmetry in low-rank adapters of foundation models." arXiv preprint arXiv:2402.16842 (2024).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) In the experimental results, why does FedSA-LoRA outperform other methods under non-heterogeneous conditions? In the absence of heterogeneity, since both matrices A and B can learn shareable knowledge, sharing information from both matrices—or sharing matrix B's information as in FFA-LoRA—should intuitively achieve equal or even greater gains.
soundness: 2
strengths: (1) The paper's analysis of the distinct roles of matrices A and B clearly demonstrates its insights and motivations.

(2) The proposed method improves upon existing solutions and achieves certain performance enhancements.
summary: The paper begins by analyzing the roles of matrices A and B in LoRA, concluding that matrix A is responsible for general knowledge while matrix B handles domain-specific knowledge. Based on this analysis, the authors propose FedSA-LoRA, a method that aggregates only the A matrix during federated learning while retaining the edge-side B matrix. This approach aims to integrate general capabilities while preserving personalized knowledge.
weaknesses: (1) One of the significant contributions of the paper is the asymmetric analysis of matrices A and B, concluding that matrix A is responsible for general knowledge and matrix B for domain-specific knowledge. However, this perspective has already been proposed in several works, such as HydraLoRA [1], which also designed an asymmetric LoRA structure based on this concept. The paper should properly cite relevant literature, and these existing findings somewhat diminish the paper's contribution.

(2) The experimental results are based on a single run, lacking multiple experiments to calculate averages and perform variance analysis, which may affect the reliability of the results. It is recommended that the authors refer to previous related work [2], conduct multiple experiments, and compute means and variances to more comprehensively evaluate the method's stability and effectiveness.

(3) The paper mainly considers the sharing of edge-side models but lacks a reasonable mechanism to update the server-side model to endow it with comprehensive capabilities.

[1] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
[2] Improving LoRA in Privacy-preserving Federated Learning

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - The authors mention in the experiment section that the proposed method is more effective when non-IID conditions are more severe. Could the authors provide some potential reasons for that?
soundness: 3
strengths: - Observing the roles that matrices A and B play is intuitive and important, and this observation is also prevalent in several LoRA-related variants.
- The experiments are comprehensive, covering different non-iid levels and client numbers and across different tasks. Also, the authors provide many ablation studies.
- Authors provide some theoretical justifications for the convergence of their method.
summary: This paper introduces a new way of combining LoRA in the federated setting, called FedSA-LoRA, where the authors found that A matrics are responsible for learning general knowledge, and B matrics focus on client-specific knowledge. Only A matrics are used for server aggregation. Extensive experiment results indicate the effectiveness of the proposed FedSA-LoRA with various LoRA-related variants.
weaknesses: - Motivation for exploring LoRA in federated learning: Why do authors use and investigate Lora-based methods in FL? The authors can expand and explain more about why it is good compared with parameter-efficient methods. In the introduction and related works, the authors state they are efficient, effective, and flexible, which is a bit too general from my perspective. The authors can better motivate the importance of this problem.
- Line 52-71: First, I feel that what kind of aggregation errors is unclear to me: could the authors provide a more accurate and specific illustration of the aggregation errors? Secondly, I wonder in practice, why do people not do the ideal model update? What are the difficulties in doing that?
- Authors can provide some computational analysis (e.g. time and space cost) of their method compared with the baselines.
- The authors could include standard deviations or confidence intervals for their main results tables, particularly for the main results.
- On the natural language generation task, the improvement of the proposed method seems to be a little marginal. Also, it is tested under IID distribution.
- The overall contribution seems to be marginal from my perspective.

Meta-review: **Summary:** The authors show that in an FL setting, the A and B matrices have different roles for LoRA. Specifically, the B matrices learn more client-specific features while the A matrices are responsible for general knowledge. Based on this observation, the A matrices are shared at the server for aggregation and the B matrices are client-specific. This reduces the communication cost and potentially leads to learning better features.

**Strengths:** The observation that the A and B matrices have different roles in LoRA FL is interesting. The authors perform thorough analysis and ablations. The proposed method tackles an important problem and shows improvements over the previous approaches. The authors provide some theoretical analysis as well.

**Weaknesses:** Some reviewers raise concerns about the novelty of the A and B matrices, specifically, the asymmetry has been observed before in earlier work. The reviewers also claim some of the assumptions for proving the bounds might be too strong. Some of the results lacked confidence intervals, which the authors provided during the rebuttal.

**Decision:** Initially, there were several concerns about the assumptions, novelty, and validity of the experimental results. The authors did a decent job of addressing these concerns by providing thorough feedback and additional experimental results during the discussion period. As a result of such improvements, the reviewers raised their scores and the overall response toward the paper is positive. Thus, I recommend acceptance.

Decision: Accept (Poster)