FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models

## Abstract

Abstract Low-Rank Adaptation (LoRA) is a popular technique for efficient fine-tuning of foundation models. However, applying LoRA in federated learning environments, where data is distributed across multiple clients, presents unique challenges. Existing methods rely on traditional federated averaging of LoRA adapters, resulting in inexact updates. To address this, we propose Fed erated Ex act LoRA , or FedEx-LoRA , which adds a residual error term to the pretrained frozen weight matrix. Our approach achieves exact updates with minimal computational and communication overhead, preserving LoRA’s efficiency. We evaluate the method on various Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks, showing consistent performance gains over state-of-the-art methods across multiple settings. Through extensive analysis, we quantify that the deviations in updates from the ideal solution are significant, highlighting the need for exact aggregation. Our method’s simplicity, efficiency, and broad applicability position it as a promising solution for accurate and effective federated fine-tuning of foundation models.

## 1 Introduction

The introduction of large language models (LLMs) has revolutionized natural language processing, enabling unprecedented performance across a wide range of tasks [, , , ]. While these models excel at transfer learning, their true potential is often unlocked through fine-tuning — a critical process that aligns these general-purpose models with specific tasks or domains. Moreover, the sheer size of these models presents significant challenges for fine-tuning and deployment, particularly in resource-constrained or distributed environments. To address these challenges, parameter-efficient fine-tuning (PEFT) methods have gained prominence, with Low-Rank Adaptation (LoRA) emerging as a particularly effective approach []. LoRA’s success lies in its ability to adapt LLMs to new tasks by training only a small number of parameters, while freezing rest of the parameters. This significantly reduces computational and memory requirements without compromising performance. Although good progress in training of LLMs has been realized by entities equipped with massive computational resources, there is hoards of unreachable data in verticals such as healthcare, finance, law firms, social-media and logistics. Federated learning (FL) is a popular paradigm to learn a federated model in this setting with multiple distributed entities [, , ] holding siloed data.

Federated Fine-Tuning (FFT) for foundation models addresses the challenge of leveraging distributed datasets while preserving data privacy. The current state-of-the-art, Federated Instruction Tuning (FedIT, Zhang et al. []), uses conventional federated aggregation to average the low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ individually. The resulting update matrix which is formed post aggregation is thus the product of the averaged matrices $\mathbf{A}$ and $\mathbf{B}$. However, the ideal update should be the average of the products of the low-rank adapters $\mathbf{A}$ and $\mathbf{B}$. The discrepancy results from the fact that "the average of the products is not equal to the product of the averages".
A naive adhoc intervention of modifying the aggregation to directly average the client updates is not a viable solution, since the subsequently obtained weight matrix loses its low-rank structure. The low-rank structure provides the efficiency benefits of LoRA in the first place, making this approach computationally intractable.

Figure: ((a)) FedIT
Refer to caption: extracted/5918684/figures/fig1-left.png

The aggregation process must be carefully designed for both accuracy and simplicity. We introduce FedEx-LoRA, a method that improves federated aggregation for LoRA by incorporating an error residual term, $\mathbf{\Delta W}_{res}$, into the pretrained weight matrix to address inexact aggregation, as shown in Figure 1. This adjustment preserves the low-rank efficiency of LoRA without adding computational overhead. Since the average update is inherently higher rank and cannot fit into the low-rank adapters, it is absorbed into the pretrained weight matrix, which is already high rank. This error term requires no training and is added at each aggregation step, ensuring no additional training costs.

Our key contributions are summarized as follows:

- •
We identify a critical discrepancy in traditional federated averaging of LoRA adapters and address it by explicitly assigning the error residual to the pretrained weight matrix, ensuring ideal updates.
- •
The error residual term is incorporated at each aggregation step, maintaining LoRA’s efficiency without any additional training. We propose a communication protocol that minimizes both communication and computational overhead.
- •
We empirically demonstrate the superiority of our approach through extensive experiments on both Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks, showing consistent improvements in performance over state-of-the-art federated fine-tuning methods.
- •
We provide a detailed analysis of the deviations introduced by federated averaging compared to ideal updates, and identify notable patterns. We further show that while multiple assignment strategies exist for exact aggregation, our specific assignment approach is most effective.

## 2 Related work

Parameter-efficient Fine-tuning.
PEFT methods aim to adapt foundation models while minimizing the number of trainable parameters. Input-based techniques like prefix tuning [] prepend trainable prompts, and prompt tuning [] optimizes soft prompts in the embedding space - both effective for task-specific adaptations. Architectural approaches, such as adapter layers [], add trainable components between transformer blocks [], facilitating multi-task learning. LoRA [] reduces memory overhead by representing weight updates with low-rank matrices, while AdaLoRA [] improves efficiency by dynamically adjusting the parameter budget. Optimization techniques, like QLoRA [], enable fine-tuning on consumer hardware via quantization, and LongLoRA [] targets long-context tasks. Recent advancements include combining multiple PEFT methods [] and scaling these techniques for very large models [], advancing the state of efficient fine-tuning.

Federated Fine-Tuning of Foundation Models.
Federated learning [] is a decentralized approach that allows multiple clients to collaboratively train a shared model without sharing their private data. Instead, clients perform local training on their own datasets, and only the resulting model updates are securely aggregated to update the global model []. This iterative process of local training and global aggregation continues until the model converges. FedBERT [] introduced federated pre-training for BERT, while recent efforts have focused on federated fine-tuning of foundation models [, , ]. The current state-of-the-art, FedIT []), fine-tunes LLMs by averaging LoRA parameters across clients using vanilla Federated Averaging (FedAvg, McMahan et al. []). However, averaging low-rank adapters independently introduces noise and results in inexact global updates. Federated Freeze A LoRA (FFA-LoRA) [] mitigates this by keeping one set of adapters trainable, improving aggregation stability but limiting the training flexibility of other adapters. This method is particularly advantageous in privacy-sensitive settings [, ]. Another challenge arises from heterogeneous rank settings, where clients adjust LoRA ranks based on their capacities [, ]. Some methods address this by self-pruning local LoRA modules and employing sparsity-weighted aggregation [], though this introduces substantial computational overhead.

## 3 Preliminaries and Motivation

Fine-tuning with LoRA.
LoRA [] leverages low-rank matrix factorization to efficiently represent the updates of pre-trained model weights. Specifically, the fine-tuned weights, $\mathbf{W}^{\prime}$, are expressed as a sum of the original weights $\mathbf{W_{0}}$ and a low-rank update $\mathbf{\Delta W}$:

$$ $\displaystyle\mathbf{W}^{\prime}=\mathbf{W_{0}}+\mathbf{\Delta W}=\mathbf{W_{0 }}+\mathbf{B}\mathbf{A}$ (1) $$

where $\mathbf{W_{0}},\mathbf{W}^{\prime}\in\mathbb{R}^{m\times n}$ are the pretrained and fine-tuned weight matrices, respectively, and $\mathbf{A}\in\mathbb{R}^{r\times n}$, $\mathbf{B}\in\mathbb{R}^{m\times r}$ represent the low-rank decomposition of $\mathbf{\Delta W}$. Here, the rank $r$ is significantly smaller than both $m$ and $n$, leading to a substantial reduction in the number of trainable parameters for $\mathbf{\Delta W}$. Instead of directly updating $\mathbf{W}_{0}$ during fine-tuning, LoRA optimizes the smaller matrices $\mathbf{A}$ and $\mathbf{B}$, resulting in considerable savings in memory usage. For instance, in GPT-2, LoRA reduces the number of trainable parameters from $124.44$ M to just $0.41$ M when using a rank of $r=4$, with no observed degradation in performance [].

Global Updates due to Vanilla Federated Averaging are Inexact.
The widely adopted federated learning algorithm, FedAvg [], updates the global model by performing a weighted average of local client updates in each communication round for $k$ clients:

$$ $\displaystyle\mathbf{W}^{global}=\mathbf{W_{0}}+\frac{1}{k}\sum_{i=1}^{k} \mathbf{\Delta W}_{i}=\mathbf{W_{0}}+\mathbf{\Delta W}$ (2) $$

where $\mathbf{W_{0}}$ and $\mathbf{W}^{global}$ represent the global model parameters before and after aggregation, respectively. $\mathbf{\Delta W}_{i}$ denotes the local update from the $i$-th client.
FedIT [] extends FedAvg by incorporating LoRA for federated fine-tuning, where clients fine-tune LoRA modules of a fixed rank.
The global LoRA matrices $\mathbf{A}$ and $\mathbf{B}$ are updated via weighted averaging over the client-specific LoRA parameters $\mathbf{A}_{k}$ and $\mathbf{B}_{k}$:

$$ $\displaystyle\mathbf{A}=\frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i},\quad\mathbf{B }=\frac{1}{k}\sum_{i=1}^{k}\mathbf{B}_{i}$ (3) $$

Although FedIT follows a similar aggregation process as FedAvg, only LoRA modules are updated and communicated. However, this independent averaging of $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ introduces deviation from the exact centralized LoRA updates, as the actual model updates depend on the product $\mathbf{B}_{i}\mathbf{A}_{i}$, not the individual components $\mathbf{B}$ and $\mathbf{A}$.

$$ $\displaystyle\underbrace{\tilde{\mathbf{W}}^{global}=\mathbf{W}_{0}+\frac{1}{k }\sum_{i=1}^{k}\mathbf{B}_{i}\times\frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i}}_{ \text{Parameters after aggregation with LoRA + FedAvg (FedIT)}}\neq\underbrace {\mathbf{W}_{0}+\frac{1}{k}\sum_{i=1}^{k}(\mathbf{B}_{i}\mathbf{A}_{i})= \mathbf{W}^{global}}_{\text{Ideal parameters following model-averaging}}$ (4) $$

There is No Free Lunch. A naive approach would be to directly average the client updates as $\frac{1}{k}\sum_{i=1}^{k}(\mathbf{B}_{i}\mathbf{A}_{i})$ and use the result for the global update before resuming training. However, this undermines the purpose of LoRA, as it forces subsequent training on the full-rank matrix $\mathbf{W}^{global}\in\mathbb{R}^{m\times n}$ rather than its intended low-rank adapters $\mathbf{A}\in\mathbb{R}^{r\times n}$ and $\mathbf{B}\in\mathbb{R}^{m\times r}$.

An alternative is to decompose the averaged update $\frac{1}{k}\sum_{i=1}^{k}(\mathbf{B}_{i}\mathbf{A}_{i})$ into a low-rank matrix of rank $(k\cdot r)$. However, this leads to an exponential growth in the rank with each aggregation round, as the rank increases by a factor of $k$ in every iteration, making this approach computationally intractable.

FFA-LoRA.
FFA-LoRA addresses the problem of inexact aggregation, particularly in privacy-preserving settings. Motivated from previous works [, ], it asymmetrically freezes the $\mathbf{A}$ adapters while keeping only the $\mathbf{B}$ adapters trainable. This approach mitigates the issues of non-ideal aggregation by avoiding independent updates of $\mathbf{A}$ and $\mathbf{B}$. However, the drawback is that the $\mathbf{A}$ matrix remains static, which limits expressiveness. While this method excels in privacy-sensitive scenarios where noise is amplified, it underperforms in non-private settings, even when the number of trainable parameters is equivalent.

## 4 Method: FedEx-LoRA

### 4.1 Noise-Free Exact Aggregation

To tackle the problem of inexact aggregation arising from the independent averaging of the $\mathbf{A}$ and $\mathbf{B}$ matrices across clients, we introduce a novel method called FedEx-LoRA. Instead of separately averaging the low-rank adapter matrices $\mathbf{A}$ and $\mathbf{B}$, we compute the average of their product $\mathbf{BA}$ across all clients. However, as previously noted in Section 3, we cannot keep this high-rank matrix or its lower-rank decomposition (with rank $(k\cdot r)$) trainable. Consequently, we append a high-rank error term that captures the discrepancy between the average of the products and the product of the averages. This error residual is incorporated into the global frozen weight matrix, ensuring its non-trainability. The update can be expressed as follows:

$$ $\displaystyle\mathbf{B}_{i}\leftarrow\frac{1}{k}\sum_{i=1}^{k}\mathbf{B}_{i}, \quad\mathbf{A}_{i}\leftarrow\frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i}$ (5) $\displaystyle\mathbf{W_{0}}\leftarrow\mathbf{W_{0}}+\underbrace{\frac{1}{k} \sum_{i=1}^{k}(\mathbf{B}_{i}\mathbf{A}_{i})-\frac{1}{k}\sum_{i=1}^{k}\mathbf{ B}_{i}\times\frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i}}_{\text{Residual}}$ (6) $$

We now demonstrate that our formulation results in exact aggregation for every client:

$$ $\displaystyle\mathbf{W}^{global}=\mathbf{W_{0}}+\mathbf{B}_{i}\mathbf{A}_{i}$ (7) $\displaystyle\mathbf{W}^{global}=\mathbf{W_{0}}+\frac{1}{k}\sum_{i=1}^{k}( \mathbf{B}_{i}\mathbf{A}_{i})-\frac{1}{k}\sum_{i=1}^{k}\mathbf{B}_{i}\times \frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i}+\frac{1}{k}\sum_{i=1}^{k}\mathbf{B}_{i }\times\frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i}$ (8) $\displaystyle\mathbf{W}^{global}=\underbrace{\mathbf{W_{0}}+\frac{1}{k}\sum_{i =1}^{k}(\mathbf{B}_{i}\mathbf{A}_{i})}_{\text{Ideal aggregation}}$ (9) $$

### 4.2 FedEx-LoRA: Overall Pipeline

Initially, the server distributes the global pretrained model to all $k$ clients and initializes the low-rank adapters $\mathbf{A}$ and $\mathbf{B}$ according to standard LoRA settings: $\mathbf{B}$ is initialized to zero, while $\mathbf{A}$ is initialized using a random Gaussian distribution.

$$ $\displaystyle\mathbf{B}_{i}\leftarrow\mathbf{B}_{init},\quad\mathbf{A}_{i} \leftarrow\mathbf{A}_{init},\quad\mathbf{W}_{0}\leftarrow\mathbf{W}_{pretrained}$ (10) $$

Each client then independently trains their low-rank adapters $\mathbf{A}$ and $\mathbf{B}$ using their local data for a specified number of epochs (referred to as “local epochs”). Upon completion of training, the clients send their updated low-rank adapters back to the server for aggregation. The server aggregates these low-rank adapters and incorporates the residual term into the global model:

$$ $\displaystyle\mathbf{B}=\frac{1}{k}\sum_{i=1}^{k}\mathbf{B}_{i},\quad\mathbf{A }=\frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i}$ (11) $\displaystyle\mathbf{\Delta W}_{res}=\frac{1}{k}\sum_{i=1}^{k}(\mathbf{B}_{i} \mathbf{A}_{i})-\frac{1}{k}\sum_{i=1}^{k}\mathbf{B}_{i}\times\frac{1}{k}\sum_{ i=1}^{k}\mathbf{A}_{i}$ (12) $$

The server then sends the aggregated matrices back to each client. After receiving these updates, the clients proceed to update their low-rank adapters $\mathbf{A}$ and $\mathbf{B}$, as well as the weight matrix:

$$ $\displaystyle\mathbf{B}_{i}\leftarrow\mathbf{B},\quad\mathbf{A}_{i}\leftarrow \mathbf{A}$ (13) $\displaystyle\mathbf{W}_{0}\leftarrow\mathbf{W}_{0}+\mathbf{\Delta W}_{res}$ (14) $$

Following this, clients independently resume fine-tuning for a set number of local epochs. This process repeats across multiple aggregation rounds (also referred to as communication rounds).

Multiple Assignment Strategies can Lead to Exact Aggregation.
Several methods can be used for achieving exact aggregation, with our choice of assignments for $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ being particularly pivotal. Each such assignment strategy allows us to adjust the corresponding error offset within the frozen weight matrix, facilitating precise aggregation. In Section 6, we investigate various methods and empirically show that our proposed assignments for $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ deliver the best performance.

Communication Protocol.
At first glance, it may seem necessary for the server to transmit the high-rank update matrix $\mathbf{\Delta W}_{res}$ to the clients, which could introduce substantial communication overhead. However, the rank of this update matrix is capped at $(k\cdot r)$.
Consequently, $\mathbf{\Delta W}_{res}$ can be decomposed into two low-rank matrices using methods such as Gram-Schmidt orthogonalization. This decomposition expresses the matrix as a product of the basis of its column (or row) space and the corresponding linear coefficients.
The computational overhead incurred by this operation at each aggregation step is negligible compared to the numerous matrix multiplications involved in training.
Importantly, clients are only required to transmit their low-rank adapters $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$, avoiding the need to send any high-rank update matrices.
In practice, the communication overhead is minimal compared to FedIT, and overall, the communication cost remains significantly lower than that of full federated fine-tuning. Detailed communication overhead analysis is provided in Section 6.

## 5 Experiments

Models and Datasets.
We fine-tune RoBERTa-base, RoBERTa-large [], and GPT-2 [] using FedEx-LoRA. We evaluate the RoBERTa models on natural language understanding tasks with the GLUE benchmark [] and assess GPT-2 on natural language generation tasks through the E2E NLG Challenge []. We implement all algorithms using PyTorch [], based on the widely-used HuggingFace Transformers codebase []. We run experiments on NVIDIA A100 GPUs, and present the results as average of $3$ different random runs. Dataset details are presented in Appendix A.

Implementation Details.
We apply LoRA modules only to the self-attention layers, following the setup from the original LoRA paper []. The residual and product matrices are scaled by the factor $\alpha/r$, where $\alpha$ is a constant in $r$, consistent with the approach in LoRA []. We run our experiments in a three-client cross-silo federated setting, based on the settings described in FFA-LoRA []. For data distribution among clients, we use the common method to sample data at random for each client, as implemented in standard works [, , ].

Baselines.
We primarily compare FedEx-LoRA with other federated fine-tuning versions of LoRA, but include centralized LoRA as a performance benchmark or skyline. We also include other baselines, where possible. Full Fine-Tuning (FT) refers to fine-tuning the entire pretrained model. LoRA [] represents the traditional centralized LoRA approach. FedIT [], the current state-of-the-art federated fine-tuning method, applies vanilla federated averaging (FedAvg) to LoRA []. FFA-LoRA [] freezes the $\mathbf{A}$ matrices and trains only the $\mathbf{B}$ matrices, allowing for exact aggregation in a federated setting but at the cost of losing the benefits of training $\mathbf{A}$.

### 5.1 Natural Language Understanding

Implementation Details.
RoBERTa [] is a widely used pretrained model known for its competitive performance among its size. We use the pretrained RoBERTa-base (125M parameters) and RoBERTa-large (355M parameters) from the HuggingFace Transformers library [] and evaluate them on several datasets from the GLUE benchmark: CoLA, RTE, MRPC, SST-2, QNLI, and STS-B. Models are fine-tuned at ranks $r=\{4,1\}$ over local epochs of $3$ and $10$. For RoBERTa-base, we run $50$ aggregation rounds for $3$ local epochs and $15$ rounds for $10$ local epochs. For RoBERTa-large, we perform $15$ aggregation rounds for $3$ local epochs and $5$ rounds for $10$ local epochs. Detailed experimental settings are provided in Appendix B.

Main Results.
We present results for RoBERTa-base and RoBERTa-large in Table 1, evaluated at ranks $r=\{4,1\}$. Our FedEx-LoRA method consistently outperforms state-of-the-art federated fine-tuning approaches across all datasets and settings. Notably, our method occasionally achieves performance on par with centralized LoRA. Additional results in Appendix C (Table 7) further demonstrate the robustness and superiority of our method over other federated LoRA variants across multiple settings.

**(a) Results with RoBERTa-base on the GLUE benchmark datasets**
| Method | CoLA | RTE | MRPC | SST-2 | QNLI | STS-B | All |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mcc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Corr $\uparrow$ | Avg $\uparrow$ |  |
| $\text{Centralized LoRA}_{r=4}$ | $64.31$ | $75.45$ | $87.99$ | $94.61$ | $92.75$ | $90.73$ | $84.31$ |
| $\text{FedIT}_{r=4}$ | $60.82$ | $73.64$ | $88.48$ | $94.61$ | $92.07$ | $90.91$ | $83.42$ |
| $\text{FFA-LoRA}_{r=4}$ | $59.34$ | $70.04$ | $87.50$ | $94.27$ | $91.37$ | $90.26$ | $82.13$ |
| $\text{FedEx-LoRA}_{r=4}$ | $\mathbf{62.82}$ | $\mathbf{75.09}$ | $\mathbf{89.95}$ | $\mathbf{94.84}$ | $\mathbf{92.66}$ | $\mathbf{90.95}$ | $\mathbf{84.39}$ |
| $\text{Centralized LoRA}_{r=1}$ | $62.13$ | $74.67$ | $87.75$ | $94.61$ | $92.31$ | $90.83$ | $83.72$ |
| $\text{FedIT}_{r=1}$ | $61.33$ | $71.48$ | $87.99$ | $94.52$ | $92.01$ | $90.81$ | $83.02$ |
| $\text{FFA-LoRA}_{r=1}$ | $57.52$ | $71.20$ | $87.48$ | $94.03$ | $91.78$ | $90.34$ | $82.06$ |
| $\text{FedEx-LoRA}_{r=1}$ | $\mathbf{62.07}$ | $\mathbf{73.65}$ | $\mathbf{88.73}$ | $\mathbf{94.84}$ | $\mathbf{92.21}$ | $\mathbf{90.87}$ | $\mathbf{83.73}$ |

### 5.2 Natural Language Generation

Implementation Details.
We fine-tune the pretrained GPT-2 (124M parameters) [] model on the E2E NLG Challenge dataset []. The model is fine-tuned at ranks $r=\{4,1\}$ with local epochs set to $3$ and $10$, using $6$ aggregation rounds for both settings. Detailed experimental settings are provided in Appendix B.

Main Results.
Table 2 presents the performance of GPT-2 fine-tuned with ranks $r=\{4,1\}$. FedEx-LoRA consistently outperforms leading federated fine-tuning methods, across all metrics and settings.
Additional evaluations, provided in Appendix D (Table 8), further demonstrate the reliability and strength of FedEx-LoRA across different configurations.

**Table 2: Results with GPT-2 on the E2E NLG Challenge, comparing various federated LoRA methods at ranks $r=\{4,1\}$. Centralized LoRA (in grey) sets the benchmark skyline for its federated versions. Best results among federated methods (in blue) are highlighted in bold for each setting. There are $3$ local epochs before every aggregation round. * indicates results taken from prior works. Higher is better for all metrics.**
| Method | E2E NLG Challenge |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| BLEU $\uparrow$ | NIST $\uparrow$ | MET $\uparrow$ | ROUGE-L $\uparrow$ | CIDEr $\uparrow$ |  |
| $\text{Centralized LoRA}_{\text{r}=4}$ | $68.91$ | $8.73$ | $46.78$ | $71.29$ | $2.47$ |
| $\text{FedIT}_{\text{r}=4}$ | $67.60$ | $8.67$ | $46.30$ | $68.96$ | $2.41$ |
| $\text{FFA-LoRA}_{\text{r}=4}$ | $66.79$ | $8.61$ | $45.24$ | $67.98$ | $2.39$ |
| $\text{FedEx-LoRA}_{\text{r}=4}$ | $\mathbf{68.15}$ | $\mathbf{8.72}$ | $\mathbf{46.48}$ | $\mathbf{69.49}$ | $\mathbf{2.44}$ |
| $\text{Centralized LoRA}_{\text{r}=1}$ | $67.41$ | $8.68$ | $46.01$ | $69.51$ | $2.41$ |
| $\text{FedIT}_{\text{r}=1}$ | $66.01$ | $8.56$ | $45.21$ | $68.14$ | $2.28$ |
| $\text{FFA-LoRA}_{\text{r}=4}$ | $65.87$ | $8.54$ | $45.02$ | $68.05$ | $2.27$ |
| $\text{FedEx-LoRA}_{\text{r}=1}$ | $\mathbf{67.02}$ | $\mathbf{8.61}$ | $\mathbf{45.99}$ | $\mathbf{69.52}$ | $\mathbf{2.38}$ |

## 6 Analysis

To fully understand the implications of our method, we performed several in-depth analyses, each targeting a specific aspect of FedEx-LoRA’s performance and efficiency.

Assignment Strategies for $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$.
As discussed in Section 4, we can incorporate any high-rank update matrix $\mathbf{\Delta W}_{res}$ within the frozen full-rank matrix $\mathbf{W}_{0}$. However, assignment of the low-rank adapters $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ post-aggregation is less straightforward.
Any selection of $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ can be offset by adjusting the residual update, by ensuring that $\mathbf{W}_{0}+\mathbf{B}_{i}\mathbf{A}_{i}$ remains consistent across clients.
We evaluate three strategies: (1) Reinitialize $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ reinitializes $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ after aggregation and appends the full update to the frozen weights (ensuring $\mathbf{W}_{0}+\mathbf{B}_{i}\mathbf{A}_{i}$ is identical). (2) $\mathbf{A}_{i}\leftarrow\mathbf{A}_{i}$ and $\mathbf{B}_{i}\leftarrow\mathbf{B}_{i}$ leaves $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ unchanged across clients, maintaining their pre-aggregation values. (3) FedEx-LoRA aggregates $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ using the aggregation method in FedIT (FedAvg), providing the best low-rank approximation to the aggregated update with the residual $\mathbf{\Delta W}_{res}$ stored in $\mathbf{W}_{0}$. We present results for RoBERTa-base on the GLUE benchmark in Table 3. FedEx-LoRA outperforms the other strategies, leading us to adopt $\mathbf{B}_{i}\leftarrow\frac{1}{k}\sum_{i=1}^{k}\mathbf{B}_{i}$ and $\mathbf{A}_{i}\leftarrow\frac{1}{k}\sum_{i=1}^{k}\mathbf{A}_{i}$ across all clients.

**Table 3: Results with RoBERTa-base ($r=4$) on the GLUE benchmark datasets, comparing various assignment strategies for $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$. We report Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for other datasets. Best results for each dataset are highlighted in bold.**
| Method | CoLA | RTE | MRPC | SST-2 | QNLI | STS-B | All |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mcc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Corr $\uparrow$ | Avg $\uparrow$ |  |
| Reinitialize $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ | $0.00$ | $61.37$ | $75.74$ | $76.26$ | $53.98$ | $53.38$ | $53.46$ |
| $\mathbf{A}_{i}\leftarrow\mathbf{A}_{i}$ and $\mathbf{B}_{i}\leftarrow\mathbf{B}_{i}$ | $55.54$ | $59.93$ | $84.80$ | $92.77$ | $88.98$ | $88.41$ | $78.41$ |
| FedEx-LoRA | $\mathbf{62.82}$ | $\mathbf{75.09}$ | $\mathbf{89.95}$ | $\mathbf{94.84}$ | $\mathbf{92.66}$ | $\mathbf{90.95}$ | $\mathbf{84.39}$ |

To extend our method to rank-heterogeneous settings, the assignments for $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ must also accommodate rank heterogeneity. Further investigation is required to develop an optimal assignment strategy that supports this.

Scaled Frobenius Norm of Divergence/Deviation.
We now study the deviations in updates from federated averaging (FedAvg) relative to ideal updates and analyze the findings.
To quantify this deviation, we measure the scaled Frobenius norm of the divergence between the updates produced by FedAvg and the ideal LoRA updates, revealing several notable patterns. In Figure 2, we plot this divergence for the query (Q) and value (V) matrices across model layers, computed after the first aggregation step for local epochs $=\{{\color[rgb]{0.37109375,0.62109375,0.62890625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.37109375,0.62109375,0.62890625}\textbf{3}},{\color[rgb]
{0.890625,0.4296875,0.4296875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.890625,0.4296875,0.4296875}\textbf{10}}\}$. We observe that (1) the deviations decrease as the model depth increases, (2) the deviation grows with a higher number of local epochs, and (3) the deviation is more pronounced in the query (Q) matrices compared to the value (V) matrices.
These trends hold consistently across various datasets and settings, as shown by additional plots in Appendix E.1 (see Figures 4 and 5).

Next, we examine how this deviation evolves across multiple rounds of federated aggregation. We plot the scaled Frobenius norm of the deviation between FedAvg and ideal LoRA updates over several aggregation rounds for different datasets, focusing on (a) the query matrices of the first layer, and (b) the average of the query and value matrices across all layers, as presented in Figure 3. We observe that the deviation consistently decreases as the number of aggregation rounds increases, both for the first-layer query matrix and for the average of the query and value matrices across all layers.
These findings are further supported by detailed plots across multiple datasets and settings, as shown in Appendix E.2 (see Figures 6, 7, 8, and 9).

Figure: Figure 2: Scaled Frobenius norm of divergence/deviation of updates with conventional federated aggregation (FedAvg) versus ideal LoRA updates, computed after the first aggregation step. We plot for query (Q) and value (V) matrices across model layers. Results are shown for local epochs $=\{{\color[rgb]{0.37109375,0.62109375,0.62890625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.37109375,0.62109375,0.62890625}\textbf{3}},{\color[rgb] {0.890625,0.4296875,0.4296875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.890625,0.4296875,0.4296875}\textbf{10}}\}$. (Dataset: MRPC, model: RoBERTa-large, $r=1$).
Refer to caption: extracted/5918684/figures/mrpc_r1.png

Figure: ((a)) Query matrices of first layer
Refer to caption: extracted/5918684/figures/rounds_a0_l10_r1.png

Communication Costs.
As discussed in Section 4, FedEx-LoRA transmits a higher-rank update matrix (rank = $k\cdot r$) along with the low-rank adapters, which raises concerns about potential communication overhead. Table 4 compares the communication costs of FFA-LoRA, FedIT, and full federated fine-tuning (FT), compared to FedEx-LoRA, for RoBERTa-base, RoBERTa-large, and GPT-2 models with rank $r=4$ over $5$ communication rounds. FedEx-LoRA incurs only a marginal increase in communication overhead relative to FedIT and FFA-LoRA, while FFA-LoRA has the lowest cost due to its reduced number of trainable parameters. FedEx-LoRA still maintains a substantially lower communication cost compared to federated full FT.

The practical impact of communication overhead is reduced by two factors: (1) the initial transmission of full model weights dominates communication costs, and (2) in NLU tasks, most communicated parameters come from the classification head, which requires training regardless of the aggregation method. Therefore, communication cost differences between FedEx-LoRA, FedIT, and FFA-LoRA are minimal in practice. Despite this marginal overhead, FedEx-LoRA consistently outperforms other federated LoRA approaches, making it an effective choice for federated fine-tuning.

**Table 4: Ratio of # of parameters communicated in federated LoRA variants and federated full FT to FedEx-LoRA. All results are reported with rank $r=4$ and across $5$ communication rounds.**
| Model | Federated Full FT | FedEx-LoRA | FedIT | FFA-LoRA |
| --- | --- | --- | --- | --- |
| RoBERTa-base | $7.032$ | $1$ | $0.979$ | $0.972$ |
| RoBERTa-large | $10.396$ | $1$ | $0.984$ | $0.979$ |
| GPT-2 | $9.475$ | $1$ | $0.917$ | $0.886$ |

Effect of Varying Rank.
We evaluate FedEx-LoRA against other federated fine-tuning methods on the CoLA dataset using RoBERTa-base, by varying the rank of the low-rank adapters across $r=\{1,2,4,8,16,32\}$, as presented in Table 5. Across all rank configurations, FedEx-LoRA consistently outperforms competing federated LoRA variants. In agreement with prior studies [, ], increasing the rank does not always result in performance gains. For this task, we find that the optimal performance is achieved at $r=8$, beyond which further increases in rank yield diminishing returns.

**Table 5: Matthew’s correlation on CoLA across different ranks for various federated LoRA methods. Centralized LoRA (in grey) sets the benchmark skyline for its federated versions. Best results among federated methods (in blue) are highlighted in bold for each rank. (Model: RoBERTa-base, local epochs $=3$).**
| Method | $\mathbf{r=1}$ | $\mathbf{r=2}$ | $\mathbf{r=4}$ | $\mathbf{r=8}$ | $\mathbf{r=16}$ | $\mathbf{r=32}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Centralized LoRA | $62.13$ | $62.11$ | $64.31$ | $64.44$ | $64.32$ | $63.98$ |
| FedIT | $60.05$ | $60.32$ | $60.82$ | $62.09$ | $62.15$ | $61.98$ |
| FFA-LoRA | $57.73$ | $57.78$ | $59.34$ | $57.82$ | $57.78$ | $58.24$ |
| FedEx-LoRA | $\mathbf{62.07}$ | $\mathbf{61.38}$ | $\mathbf{62.82}$ | $\mathbf{63.57}$ | $\mathbf{63.56}$ | $\mathbf{63.35}$ |

## 7 Conclusion

In our work, we identified limitations in state-of-the-art federated fine-tuning methods that struggle with inexact aggregation. We proposed a novel method, FedEx-LoRA, which appends the residual error matrix to the frozen pretrained matrix, while maintaining minimal communication and computational overhead. The strength of our approach lies in its simplicity and broad applicability. Extensive experiments demonstrate that FedEx-LoRA consistently outperforms other federated LoRA methods across various datasets and settings. Our analyses reveal that deviations in updates from federated averaging compared to the ideal solution are significant and exhibit notable patterns.

Testing in privacy-preserving scenarios is a natural extension of our work. FFA-LoRA [] demonstrated that noise in differential privacy leads to greater deviations from ideal updates. Given that our method achieves exact aggregation and outperforms FFA-LoRA in non-private settings, we anticipate similar success in privacy-sensitive applications. Our approach can be readily adapted for fine-tuning other models like Vision Transformers (ViTs) and Vision-Language models (VLMs).

## Acknowledgements and Discolsure of Funding

This work was supported by Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) and partially funded by the ADIA Lab Fellowship.

## Appendix A Dataset Details

GLUE Benchmark is a diverse suite of tasks for evaluating natural language understanding capabilities. It includes datasets such as SST-2 for sentiment analysis [], MRPC for paraphrase detection [], CoLA for linguistic acceptability [], QNLI for inference [], RTE for inference, and STS-B for semantic textual similarity []. Due to its comprehensive coverage of NLU tasks, GLUE is widely used to assess models like RoBERTa. Each dataset is released under its own license.

The E2E NLG Challenge [] dataset is widely used to evaluate systems for natural language generation, particularly for data-to-text tasks. It contains around 42,000 training examples, with an additional 4,600 each for validation and testing, all from the restaurant domain. Each input table has multiple reference outputs, where each data point $(x,y)$ includes a sequence of slot-value pairs and its corresponding reference text in natural language. The dataset is made available under the Creative Commons BY-NC-SA 4.0 license.

## Appendix B Hyperparameter Details

We run experiments on NVIDIA A100 GPUs and report the average results from $3$ independent random runs. Most hyperparameter settings follow the original LoRA paper [], with a learning rate sweep applied. All models are trained using the AdamW optimizer [] with a linear learning rate decay schedule and a weight decay of $0.01$. The detailed hyperparameter configurations for GPT-2 and RoBERTa-base/large are provided in Table 6.

**Table 6: Hyperparameter settings for GPT-2 and RoBERTa-base/large.**
| Model | GPT-2 | RoBERTa-base/large |
| --- | --- | --- |
|  | Training |  |
| Dropout Prob | $0.1$ | $0.1$ |
| Batch Size | $8$ | $128$ |
| Warmup Steps | $500$ | - |
| Warmup Ratio | - | $0.6$ |
| Label Smooth | $0.1$ | - |
| Max Seq. Len | $128$ | $512$ |
| Learning Rate | $2\cdot 10^{-3}$ | $1\cdot 10^{-3}$ |
| LoRA $\alpha$ | $32$ | $8$ |
|  | Inference |  |
| Beam Size | $10$ | - |
| Length Penalty | $0.9$ | - |
| no repeat ngram size | $4$ | - |

## Appendix C Additional Experiments for NLU

We present additonal results with the RoBERTa-base and RoBERTa-large models in Table 7, evaluated at ranks $r=\{4,1\}$, with local epochs set to $10$. These findings corroborate the results in Table 1, demonstrating that our FedEx-LoRA method consistently outperforms state-of-the-art federated fine-tuning approaches across all datasets and configurations.

**(a) Results with RoBERTa-base on the GLUE benchmark datasets**
| Method | CoLA | RTE | MRPC | SST-2 | QNLI | STS-B | All |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mcc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Corr $\uparrow$ | Avg $\uparrow$ |  |
| $\text{Centralized LoRA}_{r=4}$ | $64.31$ | $75.45$ | $87.99$ | $94.61$ | $92.75$ | $90.73$ | $84.31$ |
| $\text{FedIT}_{r=4}$ | $58.55$ | $70.75$ | $87.50$ | $94.36$ | $92.09$ | $90.58$ | $82.31$ |
| $\text{FFA-LoRA}_{r=4}$ | $57.52$ | $71.84$ | $86.76$ | $94.24$ | $91.27$ | $90.04$ | $81.95$ |
| $\text{FedEx-LoRA}_{r=4}$ | $\mathbf{61.32}$ | $\mathbf{75.81}$ | $\mathbf{87.75}$ | $\mathbf{94.57}$ | $\mathbf{92.64}$ | $\mathbf{90.62}$ | $\mathbf{83.79}$ |
| $\text{Centralized LoRA}_{r=1}$ | $62.13$ | $74.67$ | $87.75$ | $94.61$ | $92.31$ | $90.83$ | $83.72$ |
| $\text{FedIT}_{r=1}$ | $60.05$ | $71.84$ | $88.79$ | $94.62$ | $92.23$ | $90.54$ | $83.01$ |
| $\text{FFA-LoRA}_{r=1}$ | $57.73$ | $71.18$ | $87.74$ | $93.69$ | $91.41$ | $90.18$ | $81.99$ |
| $\text{FedEx-LoRA}_{r=1}$ | $\mathbf{61.31}$ | $\mathbf{73.12}$ | $\mathbf{89.21}$ | $\mathbf{94.73}$ | $\mathbf{92.40}$ | $\mathbf{90.67}$ | $\mathbf{83.57}$ |

## Appendix D Additional Experiments for NLG

**Table 8: Results with GPT-2 on the E2E NLG Challenge, comparing various federated LoRA methods at ranks $r=\{4,1\}$. Centralized LoRA (in grey) sets the benchmark skyline for its federated versions. Best results among federated methods (in blue) are highlighted in bold for each setting. There are $5$ local epochs before every aggregation round. * indicates results taken from prior works. Higher is better for all metrics.**
| Method | E2E NLG Challenge |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| BLEU $\uparrow$ | NIST $\uparrow$ | MET $\uparrow$ | ROUGE-L $\uparrow$ | CIDEr $\uparrow$ |  |
| $\text{Centralized LoRA}_{\text{r}=4}$ | $68.91$ | $8.73$ | $46.78$ | $71.29$ | $2.47$ |
| $\text{FedIT}_{\text{r}=4}$ | $67.61$ | $8.62$ | $46.45$ | $70.28$ | $2.43$ |
| $\text{FFA-LoRA}_{\text{r}=4}$ | $67.21$ | $8.57$ | $46.05$ | $69.98$ | $2.41$ |
| $\text{Exact-FedIT}_{\text{r}=4}$ | $\mathbf{68.49}$ | $\mathbf{8.72}$ | $\mathbf{46.76}$ | $\mathbf{70.71}$ | $\mathbf{2.48}$ |
| $\text{Centralized LoRA}_{\text{r}=1}$ | $67.41$ | $8.68$ | $46.01$ | $69.51$ | $2.41$ |
| $\text{FedIT}_{\text{r}=1}$ | $66.16$ | $8.56$ | $45.54$ | $68.25$ | $2.29$ |
| $\text{FFA-LoRA}_{\text{r}=1}$ | $65.78$ | $8.49$ | $45.01$ | $67.82$ | $2.26$ |
| $\text{Exact-FedIT}_{\text{r}=1}$ | $\mathbf{66.54}$ | $\mathbf{8.57}$ | $\mathbf{46.07}$ | $\mathbf{69.11}$ | $\mathbf{2.37}$ |

Table 8 presents additional experiments evaluating the performance of GPT-2 fine-tuned with ranks $r=\{4,1\}$ on the E2E Challenge dataset, with local epochs set to $5$. FedEx-LoRA consistently outperforms leading federated fine-tuning methods across all metrics and settings, consistent with the results presented in Table 2.

## Appendix E More Divergence/Deviation Plots

### E.1 Deviation/Divergence Plots Across Layers

As discussed in Section 6, we further quantify the deviation of conventional federated aggregation (FedAvg) from ideal updates by measuring the scaled Frobenius norm of the divergence the updates produced by FedAvg and the ideal LoRA updates. We present additional plots of this divergence for the query (Q) and value (V) matrices across model layers, computed after the first aggregation step for local epochs $=\{{\color[rgb]{0.37109375,0.62109375,0.62890625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.37109375,0.62109375,0.62890625}\textbf{3}},{\color[rgb]
{0.890625,0.4296875,0.4296875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.890625,0.4296875,0.4296875}\textbf{10}}\}$ across multiple datasets, in Figures 4 and 5. Figure 4 shows results for rank $r=1$, while Figure 5 presents results for rank $r=4$.

Figure: ((a)) QNLI, $r=1$
Refer to caption: extracted/5918684/figures/qnli_r1.png

Figure: ((a)) QNLI, $r=4$
Refer to caption: extracted/5918684/figures/qnli_r4.png

### E.2 Deviation/Divergence Plots Across Rounds

We now examine how the deviation evolves across multiple rounds of federated aggregation. We plot the scaled Frobenius norm of the deviation between FedAvg and ideal LoRA updates over several aggregation rounds for different datasets, focusing on (a) the query matrices of the first layer and (b) the average of the query and value matrices across all layers. This is presented in Figures 6, 7, 8, and 9. We include results for ranks $r=\{1,4\}$ and local epochs $=\{3,10\}$.

Figure: ((a)) Query matrices of first layer
Refer to caption: extracted/5918684/figures/rounds_a0_l3_r1.png

Figure: ((a)) Query matrices of first layer
Refer to caption: extracted/5918684/figures/rounds_a0_l10_r1.png

Figure: ((a)) Query matrices of first layer
Refer to caption: extracted/5918684/figures/rounds_a0_l3_r4.png

Figure: ((a)) Query matrices of first layer
Refer to caption: extracted/5918684/figures/rounds_a0_l10_r4.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the comments from the above weakness section. Two questions need to answer:

1) how to address the strong communication overhead problem between the server and clients?

2) how to address the strong computation overhead problem on the server?

These two issues will largely cause system performance degradation.
soundness: 2
strengths: This work provides a method, called FedEx-LoRA, to address the inexact aggregation problem when applying the LoRA in federated learning environment for large language model fine-tuning.
summary: This work provide a method, called FedEx-LoRA, to address the inexact aggregation problem when applying the LoRA in federated learning environment for large language model fine-tuning. FedEx-LoRA calculates a residual error matrix to the frozen pertained matrix on the server and then send it to each client to correct the error bring by the LoRA matrix aggregation on the server. This idea is straightforward. The authors also doing some experiments based on the LLM benchmarks to evaluate the model accuracy.
weaknesses: There are some concerns for the proposed methods: 

1) for the federated learning, the network bandwidth between the server and clients is often very limited. This work brings lots of extra communication overhead for each training round. The extra communication data between server and clients equals to the entire model size which brings strong communication overhead, especially for the federated learning environments.

2) the proposed methods requires to calculate n matrix multiplication, where n is the number of the layers for the LLM models. For the federated learning, the server also run on a CPU-based computing node. The n matrix multiplication also brings strong computational overhead for the server. If you want to use GPU to calculate such matrix multiplication, you need to carefully schedule the calculation process since the bandwidth between main memory and GPU memory (PCIe) also limited.

Overall, the final system performance, including the communication and computational efficiency for the proposed method is unacceptable for the federated learning environments.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1.	It would be good to see the performance comparison for different settings regarding clients e.g., the number of clients,  the portion of randomly selected clients etc. – these are standard evaluations in FL literature
2.	How the residue error term matrix affects the update of A and B? It would be good to see an analysis experiment regarding this.
soundness: 3
strengths: The paper introduces a straightforward yet effective method that ensures exact aggregation in LoRA-based fine tuning for Federated Learning. While it has a slightly higher communication cost compared to FedIT and FFA-LoRA, the increase is minimal given the significant improvement in overall performance.  
Moreover, the method's effectiveness is validated through comprehensive experiments and analysis, affirming its claim of exact aggregation and subsequently better overall performance.
Additionally, the paper has good reproducibility of the results because of its detailed presentation of the experimental settings.
summary: This paper explores the task of parameter-efficient fine-tuning under Federated Learning setting, particularly focused on the low rank adaptation approach. The work is built on the insights from the earlier FedIT model which used traditional federated aggregation methods to average low-rank matrices A and B individually. However, this approach was found to be inexact aggregation as the average of the products does not equate to the product of the averages.
To address this challenge, the authors propose an innovative method that incorporates a residual term into the pretrained frozen weight matrix. This addition aims to bridge the discrepancy between the average of the products and the product of the averages, enhancing the aggregation performance. The effectiveness of this approach is demonstrated through extensive analysis of NLU and NLG benchmark datasets using RoBERTa and GPT-2 models. The results show notable improvements in performance metrics while maintaining comparably low communication costs compared to the most benchmarked models.
weaknesses: The main issue is that the contribution may seem too incremental for ICLR conference as the method primarily focuses on a straightforward adjustment in the aggregation phase—specifically, adding back the discrepancy between the average of the products and the product of the averages. This method, while empirically effective, does not introduce a significant innovation or a novel approach that would typically be expected for ICLR.
Additionally, there are practical scalability concerns regarding the residue error term matrix, which shares dimensions with W’ and W0. The paper notes that the rank of this matrix is limited to k*r where k is the number of clients. However, in typical federated learning environments, which often involve a large number of clients (potentially hundreds), this could lead to a high-rank matrix and, consequently, significant communication costs. The experimental setup in the paper, which involved only three clients, does not adequately demonstrate the method's communication cost in larger, more realistic scenarios common in federated learning.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - What factors contribute to FedEx-LoRA outperforming centralized LoRA on certain benchmarks?

- Can FedEx-LoRA be extended to other LoRA variants, such as MoE-LoRA and QLoRA, and if so, what adaptations might be necessary?

- How does FedEx-LoRA scale and perform with larger foundation models, such as Llama or Gemma?

- How does FedEx-LoRA perform on more complex tasks, such as commonsense QA, mathematic reasoning, or coding?
soundness: 3
strengths: - Originality: The paper introduces a novel solution to the inexact aggregation problem in federated fine-tuning with LoRA by adding a residual error term directly to the pretrained weight matrix. This innovative approach ensures exact updates while preserving the low-rank efficiency of LoRA, addressing a key limitation in existing methods.

- Quality: The authors provide thorough theoretical justification and extensive empirical evaluations across multiple benchmarks. The experiments consistently demonstrate that FedEx-LoRA outperforms SOTA federated fine-tuning methods, validating the effectiveness and robustness of the proposed approach.

- Clarity: The paper is well-written and organized, clearly explaining the challenges of federated fine-tuning and the limitations of existing aggregation methods. The step-by-step presentation of FedEx-LoRA, along with illustrative diagrams and extensive formulas, makes the methodology accessible and easy to understand.

- Significance: By enabling exact aggregation in federated settings without significant computational overhead, the paper has substantial practical implications for privacy-preserving applications in NLP and beyond. The method enhances the feasibility and performance of federated fine-tuning in real-world scenarios where data privacy and efficient communication are crucial.
summary: The paper introduces FedEx-LoRA, a novel approach for federated fine-tuning of LLMs using LoRA. Existing federated fine-tuning methods face limitations in achieving exact aggregation due to the inexact averaging of LoRA adapters. The proposed FedEx-LoRA addresses this by introducing a residual error term added to the frozen weight matrix, ensuring precise updates. FedEx-LoRA is designed to be computationally and communication-efficient, preserving LoRA’s benefits without adding additional training burdens. Empirical results demonstrate improved performance on NLP tasks over SOTA federated fine-tuning methods, highlighting its applicability in distributed training settings where data privacy is a concern.
weaknesses: - Model limitations: The paper evaluates FedEx-LoRA primarily on RoBERTa and GPT-2, which are smaller foundation models. It remains uncertain how this method performs on larger models, such as Llama and Mistral, where scalability challenges might differ.

- Task scope: The evaluation focuses on standard NLP tasks. Assessing performance on more complex tasks, such as reasoning and inference, would strengthen the paper’s claims on generalizability and robustness.

- Quantization challenges: Introducing residual matrices to the original weights could complicate quantization processes for LLMs, as these residuals require re-quantization at each aggregation round. This could increase computational and memory costs, potentially limiting practical efficiency in federated settings.

- Privacy impacts: Although FedEx-LoRA is presented as beneficial for privacy-preserving federated learning, the paper lacks a detailed analysis of how the approach performs under differential privacy constraints or other mechanisms, which could be helpful for deployment in sensitive domains.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: Please see the weakness.
soundness: 2
strengths: (1) The problem raised about whether to use FedAvg on LoRA is valuable. It brings up the problem that the optimization target of LoRA federated fine-tuning is the full global model $W$ or a global LoRA $B_GA_G$.

(2) Figure 2 provides some interesting insights for FL researchers. The authors find that the deviations decrease as the model depth increases, which may lead to deeper reflections on the relationship between model architecture and fine-tuning.
summary: Given the problem that averages the A and B matrices of LoRA will result in inexact updates in federated learning, this paper proposes Federated Exact LoRA (FedEx-LoRA). FedEx-LoRA computes the aggregation error on the server, sends the error weights to the clients, and updates the local model by the weights while preserving the aggregated LoRA. The paper provides experiments on RoBERTa and GPT-2 to demonstrate the effectiveness of FedEx-LoRA compared to FedIT and FFA-LoRA.
weaknesses: **Contribution**: The authors mention in the paper that they identify a critical discrepancy in traditional federated averaging of LoRA adapters. However, this question has been raised and studied in detail. [1] and [2] identified this problem and [2] provided a solution for eliminating this error. None of these previous works are discussed in detail and listed as baselines. The reviewer believes that the author has not adequately researched this field, so the contribution is limited.

**Method**: The reviewer believes that the solution proposed in this paper is too simplistic and impractical. Here are some reasons:

(1) Increased communication cost: According to Figure 1 and Eq. (14), the server needs to send the residual error term to clients for updating the local base model. This makes the communication cost from server to clients as large as full fine-tuning (or at least the full fine-tuning on the layers that apply LoRA). It is not acceptable since we use LoRA to reduce the communication cost. Especially in federated fine-tuning LLMs, this residual error term may be as large as $kT$ times the LLM parameter size.

(2) Increased computation cost: According to Figure 1 and Eq. (11) (12), the server will iteratively compute the production of $BA$ from clients and the residual error term. This has led to a several-fold increase in computation on the server, compared to FedIT.

**Writing**: This paper has some typos, as well as unclear parts that do not conform to the federated learning paradigm.

(1) Eq. (5), (7) look confused, why the aggregation of local LoRA modules $B_i$ and $A_i$ is still the local LoRA? If here the authors mean that the new $B_i A_i$ are local modules in the next epoch, why in Eq. (7) the global model is updated by the local LoRA? These expressions may be understandable to researchers familiar with federated learning through guessing, but they are still difficult to understand.

(2) The paper use $B \cdot A$ in Figure 1. It seems incorrect to use the dot product here. However, in the equations, $BA$ and $B\times A$ are used.

(3) $W_0$, $W^{global}$, $A_{init} B_{init}$, $A_{i} B_{i}$ are used together. It is confusing which round these weights are in and whether they are the global or local weights. It could be better to use communication round numbers and uniform global and local marks.

**Experiments**: Related papers such as FedIT are using LLMs for experiments. But FedEx-LoRA is using RoBERTa and GPT-2. It is understandable that experimental resources are limited, but the research related to LoRA nowadays lacks sufficient convincing power when applied to models smaller than 1b parameters. As mentioned before, the extra communication and computation costs are not well analyzed.

**Missed Baselines**: The paper misses detailed comparison to at least two important related works that focus on the same topic:

[1] Sun, Y., Li, Z., Li, Y., & Ding, B. (2024). Improving loRA in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313.

[2] Wang, Z., Shen, Z., He, Y., Sun, G., Wang, H., Lyu, L., & Li, A. (2024). Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.05976.