Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA
Authors: Shuangyi Chen
Yuanxin Guo
Yue Ju
Hardik Dalal
Zhongwen Zhu
Ashish J Khisti
Venue: NeurIPS 2025 poster ()
Keywords: LoRA
Federated Learning
Primary area: deep_learning
TL;DR: RoLoRA improves federated fine-tuning alternating optimization of LoRA, enhancing expressiveness and robustness. It reduces communication costs by half and outperforms alternatives.

Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs.  We propose RoLoRA, a federated framework using alternating optimization to fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments to demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness of the model. We provide a theoretical analysis on a linear model to highlight the importance of learning both the down-projection and up-projection matrices in LoRA. We validate the insights on a non-linear model and separately provide a convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods.

Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA
Shuangyi Chen !M
Yuanxin Guo University of Toronto yuanxin, www.wauloommmam..o www.mmii..tooooooooo..o www.uuttoooc.oomm
University of Toronto
shuangyi.. chen@maill utorento.ca
Yue Ju
Hardik Dalal Pricsson-GAln Montréal hardik. dal @ericssonn com
Zhongwen Zhu
Ericsson-GAIA Montréal yue. juerricssnsssnn com com
Ericsson-GAIA Montréal
zhongwen. www8ericcson.cooo com
Ashish Khisti University of Toronto akhistißece. utoronto.ca
Abstract
Carameter-Effiieen Fine-Ture Thee-Tunigg (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs. We propose RoLoRA, a federated framework using alternating optimization fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness the model. We provide theoretical analysis on a lin- ear model to highlight the importance of learning both the down-projection and up-projection matrices LoRA. We validate the insights on a non-linear model and separately provide convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Lare Large, Llama-2-77 on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods.
Introduction
The remarkable performance of large language models (LLMs) stems from their ability to learn at scale. With their broad adaptability and extensive scope, LLMs depend on vast and diverse datasets to effectively generalize across a wide range of tasks and domains. Federated learning (28 offers promising solution for leveraging data from multiple sources, which could be particularly advantageous for LLMs.
Recently, Parameter-Efficiees Fine-Tuning (PEFT) has emerged an innovative training strategy that updates only a small subset of model parameters, substantially reducing computational and memory demands. A notable method in this category is LoRA (21). which utilizes low-rank matrices to approximate weight changes during fine-tuning. These matrices are integrated with pre-trained weights for inference, facilitating reduced data transfer in scenarios such as federated learning, where update size directly impacts communication efficiency. Many works integrate LoRA into federated
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Server
Agprggmton:
Cental Se Asgraaerior
83 $ AB
Σ A:80 Σ
Prozen LLM
Trainatiee Adapter
Adapter
Oddinumbere Comms Rounds
Even-numbered Comm. Rounds
food
Figure 1: (Left) Overview of the RoLoRA framework. (Right) Performance comparison with baselines on QQP in a 50-client setting, showing RoLoRA's superior convergence speed and final accuracy.
FedPETuning [45 compares various PEFT methods in a federated setting. SLoRA 1 presents a hybrid approach that combines sparse fine-tuning with LoRA to address data heterogeneity federated settings. Furthermore, FŞ-ILLM 24 presents a framework for fine-tuning LLMs in federated environments. However, these studies typically apply the FedAVG algorithm directly to LoRA modules, resulting in in-exact model updates: the server average of LoRA-A and LoRA-B does not equal the effective adapter.
setting
32
To address the issue of in-exact model updates, a few recent works have proposed modifications to the down-projp cown-projection and up-pn up-projection components in LoRA. In FlexLoRA 22, the authors propose updating these projections with matrix multiplication followed by truncated SVD. A related method is also considered in FLoRA 40). Another approach, by Sun et al., introduces federated finetuning framework named FOA-LORA [32), which builds on LoRA by freezing the down-projection matrices across all clients and updating only the up-projection matrices. They apply differential privacy [13) to provide privacy guarantees for clients' data. With a sufficient number of finetuning parameters, FFA- LoRA, using larger learning rate, can achieve performance comparable to FedAVG of LoRA while reducing communication costs by half. However, we observe that with fewer finetuning parameters, FFA-LoRA is less robust than FedAVG of LoRA, primarily due to its reduced expressiveness from freezing nown-projections. matrices and propose federated fine-tuning framework with computational and communication advantages.
this work, we explore the necessity of learning down-projection
We connect the objective of learning down-projection matrices in federated setting to multitask linear representation learning (MLRL), an approach in which a shared low-rank representation is jointly learned across multiple tasks. While, to the best of our knowledge, the alternating optimization of down- and up-rrojection matrices has not been explored within the context of LoRA, prior works on MLRL 2 34 have demonstrated the importance of alternately updating low-rank representations and task-specific heads, demonstrating the necessity of learning a shared representation. Inspired by MLRL, we tackle this challenge by employing alternating optimization for LoRA adapters. We theoretically establish that alternating updates to the two components of LoRA, while maintaining a common global model, enable effective optimization of down doojeproosstions and ensure convergence to the global minimizer in a tractable setting.
1.1 Main Contributions
RoLoRA framework. We propose RoLoRA, robust federated fine-tuning framework based on the alternating optimization of LoRA as shown in Figure RoLoRA fully leverages the expressiveness of LoRA adapters while keeping the computational and communication advantages.
Theoretical Insights. We prove that RoLoRA achieves exponential convergence to the global optimum for federated linear regression, reaching arbitrarily small errors, whereas FFA-LoRA's fixed nown-projections result in suboptimality proportional to the initialization error. Simple non-linear model experiments further validate the theoretical importance of updating the down- cown-projections in practice. For the case of smooth non-convex loss functions we also provide a proof convergence of our proposed method.
Empirical results. Through evaluations on language models (RoBERTa-Large) Llama-2- 7B) across various tasks (GLUE, HumanEval, MMLU, Commonsense reasoning tasks), we demonstrate that RoLoRA maintains robustness against reductions in fine-tuning parameters and increases in client numbers compared to prior approaches.
In summary, RoLoRA outperforms baselines with strong theoretical guarantees and robust perfor- mance across tasks and scales. It provides new insights into the role of nown-projections, advancing

---

the frontier of parameter- efffiieent federated learning, while its simple yet effective design ensures scalability and communication efficiency.
1.2 Notations
We adopt the notation that lower-case letters represent scalar variables, lower-case bold-face letters denote column vectors, and upper-case bold-face letters denote matrices. The d x d identity matrix is represented by Ig. Depending on the context, ||-I| denotes the la norm of vector or the Frobenius norm of a matrix, Il-llop denotes the operator norm of a matrix, |-| denotes the absolute value of scalar, denotes matrix or vector transpose. For number N, [N] = {1.....
2 Preliminaries and Related Works
2.1 LoRA and Its Variants
Low-Rank Adaptation (LoRA) [21] fine-tunes large language models efficiently by keeping the original weights fixed and adding small trainable matrices that apply low-rank updates. Specifically, for a pre-trained weight matrix Wo E Rdxd, the update is expressed as W = W, + aAB, where A Rax and B Ik' with < d, and only A and B are trained. Many variants of LoRA have been developed to further improve efficiency and performance. In Zhang et al. 44 the authors propose memory-efficient fine-tuning method, LoRA-FA, which keeps the projection-down weight fixed and updates the projection-up weight during fine-tuning, In Zhu et al. [46], the authors highlight the asymmetry between the projection-up and projection-down matrices and focus solely on comparing the effects of freezing either the projection-up or projection-down matrices. Hao et al. [6] introduce the idea of resampling the projection-down matrices, aligning with our observation that freezing projection-dowe downn matrices negatively impacts a model's expressiveness. Furthermore, LoRA+ [7 explore the distinct roles of projection-up up and projection- matrices, enhancing performance by assigning different learning rates to each.
2.2 LoRA in Federated Setting
In federated settings, LoRA is practical: clients fine-tune models efficiently with minimal resources, and only the small adapter matrices need be communicated, sharply reducing transmission costs compared to full model finetuning. Zhang et al. consider FedAVG of LoRA, named F43IT[43]
Zhang et al. 45 compare multiple PEFT methods in the federated setting, including Adapter 20
[21 Prompt tuning/26] and Bit-Fit[42] SLoRA which combines sparse finetuning and LoRA, is proposed address the data heterogeneity in federated setting. As discussed before, Sun al. [32] design a federated finetuning framework FFA-LoRA by freezing projection-dos -owwn matrices for all the clients and only updating projection-up matrices. FlexLoRA and FLoRA 440 consider clients with heterogeneous-rank LoRA adapters and proposes federated finetuning approaches. After we completed our work, we noticed a concurrent study, LoRA-A2 (23,, which combines alternating optimization with adaptive rank selection for federated finetuning. While their focus is mainly empirical, the use alternating optimization their algorithm similar to ours.
2.3 FedAVG of LoRA Introduces Interference
Integrating LoRA within a federated setting presents challenges. In such setup, each of the N clients is provided with the pretrained model weights Wo. which remain fixed during finetuning. Clients are required only send the updated matrices B, and A; central server for aggregation. While most current studies, such as FedIT [43] SLoRA [T] and FedPETuning [45]. commonly apply FedAVG directly to these matrices as shown in (2). this approach might not optimal. The precise update for each client's 's model, AW, should be calculated as the product of the low-rank matrices A; and B; Consequently, aggregation on the individual matrices leads inaccurate model aggregation.
1 AW,
N (A + A2B + ANBN)
N
N A1 + A2 +
AN) N B,
+B +... BBN)
(2)
There are a few options to avoid it.
Updating B and A by matrix multiplication and truncated-SVD. One approach 40 2 involves first computing the product of local matrices B, and A, to accurately recover AW, Then, the global B and A of next iteration are obtained by performing truncated SVD on the averaged set of AW,
3

---

However, this method introduces computational overhead due to the matrix multiplication and SVD operations.
Freezing A (B) during finetuning. Another method is to make clients freeze or A as in Sun et al. [32), leading to precise computation of AW. However, this method limits the expressiveness of adapter.
With these considerations, we propose a federated finetuning framework, named RoLoRA, based on alternating optimization LoRA.
3 RoLoRA Framework
In this section, we descrihe the framework design of RoLoRA and discuss its practical advantages.
Alternating Optimization and Corresponding Aggregation Motivated by the observations dis- cussed in Section 2.3 we propose applying alternating optimization to the local LoRA adapters of each client in setting with N clients. Unlike the approach in FFA-LoRA, where A is consistently frozen, we suggest an alternating update strategy. There are alternating odd and even communication rounds designated for updating, aggregating A and B, respectively.
In the odd-numbered comm. round:
N AW2++1 21+1
- (A] +... A = 1
A' A
+ ...+BB+))
(3)
the even-numbered comm. round:
N
1 Aw = N (A + +... AW B
A
B+++) = N
[1 +... +...+ AA
(4)
N
As in Algorithm in Appendix, all clients freeze A and update B' in the odd communication round. The central server then aggregates these updates compute B Σ B and distributes B++1 back the clients. In the subsequent communication round, clients freeze B++11 and update A The server aggregates these to obtain A ++1 N Σ A,+1 and returns A 1+1 to the clients. important to note that in round 2t + 1, the frozen A are identical across all clients, as they are synchronized with A' from the central server at the beginning of the round. This strategy ensures that the update and aggregation method introduces no interference, as demonstrated in (3) and ④
The parameter freezzingr nature RoLoRA enhances computational and communication efficiency. In each communication round, the number of trainable parameters in the model is effectively halved compared to FedAVG of LoRA. The only additional cost for RoLoRA compared FFA-LoRA is the alternating freezing of the corresponding parameters. We remark this additional cost is negligible because it is applied to the clients' models and can be executed concurrently during the server's aggregation.
Computation and Communication Cost.
4 Analysis
In this section, we provide an intuitive analysis of why training the down-projection in the LoRA module is essential a federated setting. We first present a thcoretical comparison between RoLoRA and FFA-LoRA examination of their solutions, clearly highlighting the limitations of FFA-LoRA in this fundamental We then empirically validate the theoretical findings using two-layer non-linear neural network. Finally, we provide a convergence analysis for RoLoRA in non-convex loss landscapes, establishing standard federated learning guarantees.
linear model. While simplified, this comparison offers direct and rigorous
4.1 Theoretical Insights into Down-Projection Learning: Linear Model Analysis
We start by analyzing RoLoRA and FFA-LoRA within simplified linear model, offering clear and rigorous comparison that reveals the inherent limitations of FFA-LoRA's approach. Consider a federated setting with N clients, each with the following local linear model fx,)) X,ab where
Y, E pmxd X, pmxd with the sample size m, IR (a unit vector) and b R are the LoRA

---

weights corresponding to rank r = 1. In this setting, we model the local data of i-th client such that
Y, = X;abb
(S)
for some ground truth LoRA weights a* Rd (a unit vector) and b* R We consider the following objective
N
min GER
((a
N
1=1
where the local loss is (a, random matrix, where each entry independently and identically distributed according a standard Gaussian distribution.
|X,a' " X,ab Each X; is assumed to be a Gaussian
()..b))- =
provides a summary of mathematical notations and also point We remind the reader that Section 1.2 to 3 in Appendix A3.1 for a summary of the symbols used throughout the theoretical analysis.
Results. In this section, we assume homogeneous clients where there single target model as in (5) In the linear model case, we modify RoLoRA from Algorithm I to Algorithm 2 employing alternating minimization for b (line 5) and gradient descent for (line 9). Details are described in Algorithm
2 in Appendix. We note that the analysis of the alternating minimizationnggaaee descent algorithm is inspired by
36 for a different setting of MLRL. See further discussion in
Appendix A2
We aim to show that the training procedure described in Algorithm 2 learns the target model (a", bb) by showing the angle distance (Definition (42) between a and a is decreasing in each iteration. Since b is solved exactly at each iteration via local minimization, it is always optimal with respect to the current a. This allows to isolate and analyze the convergence behavior of a using the angle distance, eliminating the potential impact of insufficient local updates of on the convergence of First, we make typical assumptions on the ground truth b* and formally define the angle distance.
Assumption 4.1. There exists Lmaz < oo (known priori), s.t. ||b*||
S Lmaz-
Definition 4.2. (Angle Distance) For two unit vectors a, a* E Rdd the angle distance between a and a" is defined as
sin (a, a a²)l = |[(I, aa Ja'll
where Ia- aa? is the projection operator to the direction orthogonal to a.
Let 8 [ a'a" Ja'll = |[ a'a' )a'll denote the angle distance between and a' of t-th iteration. We initialize a such that sin (a" a²)l 5u. where 0 < 5u < 1, and b is zero. All clients obtain the same initialization for parameters. Now we are ready state our main results.
Lemma 4.3, Let 8' = (LL-aaas a'a" Ja'll be the angle distance between a' and a' of t-th iteration. Assume that Assumption holds and 8t < 2......... 80. Let m be the number of samples for each updating step, let auxiliary error thresholds e' -iT [T-coX(-eij:6
_ي_ for E0, E1, 62, E3
(0, 1), if m = =[(g)) for q max
log(N)
dlog(
and auxiliary error thresholds are small
1- for any t and Li then we have,
such that C/ē<< 16
8--1 s 5¹ V 1 802
b b+
(8)
with probability at least -1
Theorem follows by recursively applying Lemma 4.3 and taking union bound over all / E [T]. Remark 4.4. The decreasing angle assumption in Lemma 4.3 is technical tool to simplify the proof. In Theorem (45)) this condition is not required: the inductive hypothesis inherently enforces the necessary bounds on angles, bypassing the need for explicit monotonicity.
Theorem 4.5. (Convergence of RoLoRA for linear regressor) Suppose are in the setting described in Section 4. and apply Algorithm 2 for optimization. Given a random initial a an initial angle distance d < (0, 1). we set step size 7 5 Dmax
and the number of iterations T T2 -(1-(8955] log( -")
5

---

for e E (0, 1). Under these conditions, if with sufficient number of samples m Qi and small auxiliary evoor thresholds é' = (I-ro)l1-a))
such that é',ē 1-
52
63 1-ro-
we achieve that
16
(Ng( with probability at least 1 -T for q max (miu(().)2))))
dloog 2
sin 0(a"; ,a") c
which we refer to as e-accurate recovery. In addition,
||a -aa-"" (1+ (1+c)c[la"b"]]
Theorem 4.5 and Lemma 4.3 show that with random initialization for the unit vector (8 E (0, 1)), RoLoRA makes the global model converge to the target model exponentially fast with large The requirement for sample complexity is well-supported, as demonstrated in 110 12]. While the proof of the above results are relegated to the Appendix, we provide a brief outline of the proof. In Appendices A3.3] first analyze the minimization step for updating b (Lemma A3.9), then establish a bound on the deviation of the gradient from its expectation with respect to a (Lemma 10).
and finally derive bound for sin 0 ((a++ a²) based on the gradient descent update rule for (Lemma 4.3) The proof of Theorem 4.5 is in Section A3.4
Intuition Freezing-A Scheme (FFA-LoRA) can Saturate. We begin by applying the FFA- LoRA scheme to a centralized setting, aiming to solve the following optimization problem:
min BERn
Xa"b* - Xa"bTI"
where a" E Rdd and b* E Rd represent the ground truth parameters, and a E Rd is the random initialization. The objective can be transformed to Ep=1 (a - abbp) T T X - a bp), with bp as the p-th entry of b, b7 as the p-th entry of b". In FFA-LORA scheme, a" remains fixed during training. a not initialized to be parallel to a", the objective can never be reduced zero. This is because optimizing b only scales the vector a ab along the direction of a", without altering the angular distance between a" and a".
Suppose are in the federated setting described in Section 4.1 apply FFA-LoRA, to optimize the objective in (6). In FFA-LORA scheme, we fix of all clients to a random unit vector a where the initial angle distance 8 sin (aa", a,)), 50 (0, 1). And only update b, by minimizing 1; and aggregate them.
Proposition 4.6. (FFA-LoRA lower bound) Suppose we are the setting described Section 4.1 For any set of ground truth parameters (a" b' ). the initialization a initial angle distance 5 E (0. 1j. we apply FFA-LoRA scheme to obtain shared global model (a FFA),, yielding an expected global loss of
N Σ X'a'''' - (F (1+ (1)6)))))((6))
Nm
where the expectation is over all the randomness in the X,, and ē = O( Nm
See Appendix A for the proof.
Remark 4.7. Proposition 4.6 holds for any unit vector a and corresponding obtained by fully minimizing the local loss. The same expected loss applies to RoLoRA by substituting RoLoRA's reduced angle into Eq. [
Comparison of RoLoRA and FFA-LORA. Proposition shows showw that for any choice of 0 E (0, 1). the global objective reached by FFA-LoRA is shown in (9) The global objective of FFA-LoRA is dominated by b (12 which due to the angular distance between a and a".
By Theorem 4.5 we demonstrate that RoLoRA achieves e-accurate recovery of the global minimizer. Specifically, the expected global loss of RoLoRA can be upper bounded by (1 + ž) I Under the same initialization and ground truth parameters for both FFA-LORA and RoLoRA, RoLoRA's ability to update a reduces the global loss caused by the angle distance between a and a" from ||b* 2 (80)² to ||b]|] 2 By increasing the number of iterations, can be made arbitrarily small.

---

we analyze the convergence of RoLoRA with single LoRA structure in federated setting with heterogeneous clients. By showing the decreasing of the angle distance between the ground truth a" and the shared down-projection a, we demonstrate that RoLoRA allows the global model to converge to global minimum while the global loss of FFA-LORA can be dominated by the term caused by the angle distance between the random initialization a and a².
Heterogeneous Case. In Appendix A3.5
4.2 Verifying Insights On a Non-Linear Model
The previous analysis considers linear model for each client. To assess the validity of the theorem in non-linear model, consider two-layer neural network model on each client given by
80
80
becurecc (%)
60
60
40
40
fx44)) (2, out (10)
20
FFA-LORA 20
0
where W out E R A Raxx and B e Brxd are weights. We train the model on MNIST [1] We consider two different ways to distribute training images to clients. The first to distribute the images to 5 clients and each client gets access to training images of two specific labels, while the second is to distribute the images 10 clients and each client only has training images of one specific label. There is no overlap in the training samples each client can access. Only weights matrices B and A are tunable, while W out are fixed. We use 10, d 784, r 16 and make each client train 5 epochs locally with batch-size 64 and aggregate clients' update following three methods: FedAVG of LoRA, referred as LoRA; FFA-LoRA (32. which freezes A during training, and RoLoRA, which alternately updates B and A.
100
25
50
Iterations
Iterationn
Figure 2: (Left) Comparison of three methods on a toy model with clients. (Right) Comparison of three methods on a toy model with 10 clients.
As shown in Figure 2 we evaluate the performance of the model in each iteration on the test set. We observe that the accuracy of FFA-LORA plateaus around 55% in both settings, which aligns with our theoretical analysis. The decline in LoRA's performance with an increasing number of clients is most likely due to less accurate model aggregation, as demonstrated in ① and (). Notably, RoLoRA demonstrates greater robustness in these settings. To study the impact of non-linearity on RoLoRA, we repeat the two-layer experiment on linear network without ReLU. As shown in Figure] in the Appendix, RoLoRA benefits more from the added expressiveness of ReLU.
4.3 Convergence in Smooth Non-Convex Settings
We follow the approach of Li et 25 to amalyze the convergence behavior RoLoRA smooth, non-convex settings, and derive TheoremA4.44 Appendix. As shown in TheoremA4.4 RoLoRA achieves an O(I/VT) convergence rate toward stationary point under smooth, non-convex condi- tions, matching the convergence rate established for FedAVG in the same regime.
Experiments on Language Models
In this section, we evaluate the performance of RoLoRA in various federated settings. Considering all clients will participate in each round, we will explore the following methods: FedAVG of LoRA (referred as LoRA) [43. FFA-LORA (32]. FlexLoRA [ FloRA [40], and RoLoRA (ours).
Implementation & Configurations. We implement all the methods based on FederatedScope-LL 24]. We use NVIDIA GeForce RTX 4090 or NVIDIA A40 for all the experiments. To make fair comparison, for each dataset, we obtain the best performance on test set and report the average over multiple seeds. Specifically, the learning rate is chosen from the set (5c 4, le 3, 2e 3, 5e 3, 1e 2, 2c 2, 5e 2, le 1}. Other pprramparameters for experiments are specified in Table in Appendix A5.2 Please note that in all tasks, we compare the performance of the three methods under the same number of communication rounds.
5.1 Language Understanding Tasks
Model and Datasets. We take the pre-trained RoBERTa-Large (355M) 27 models from the HuggingFace Transformers library, and evaluate the performance of federated finetuning methods on 5 datasets (SST-2, QNLI, MNLI, QQP, RTE) from the GLUE (38)
7

---

Rank
Clients Num
Method
SST-2
QNLI
MNLI
QQP
RTE
Avg.
LoRA
95.62-1au7
91.5
26.20
86. 16.13
81 81.46
88.20
FFA-LoRA
98.18-+00
91.35+a032
84.58+001
85.50+a25
81. 81.00-33
87 48
4
3
FlexLoRA
94
90.16:204
55.16gnm
85.69.am7
87.04
RoLoRA
95.49
91.64.000
g5
86.14inm
82.43
88.28
LoRA
86.67
78.55
83.1
51887
78.90
FFA-LaRA
93.88-+006
89. +..19
80. 10.7 1:74
83.92-6.2
57. 16-166
80.01
4
20
FlexLoRA
90.97
44.36
$3.30=11455
69.1 69.18-+.0.0
53.19
64.20
RoLoRA
94.88
90.35
85.28 11
85.83
78882117
87.03
LoRA
93.00
18.13
12.64
77.60
52.23
70.72
4
50
FFA-LaRA
93.23-4012
85.05 2030
69.97
78.444041
58,72=
76.48
FlexLoRA
54.08-55
55.4-2203
39. 144-255
72.00+264
52.71+0m
54.67
RoLoRA
94.80
90.00
82.98 1:336
85.71:00
75:57
85.81
LoRA
93.00100s
79.87
$6.96.22.
17.45
53.79
64.03
8
50
LOAA
FlexLoRA
92.74-013
50,92++.00
56,92-+1.4
83.69 20.75
64.51 1::9
37,4342200
66.40+474
79.71 42004
52.59+021
53.07 113
52.85
72.46
RoLoRA
94.53
90.1
85.17 20.1
85.25
76.3
86.27
Table 1: Results for four methods with RoBERTa-Large models across varying client numbers (3, 20, 50), maintaining constant sample count during fine-tuning.
Effect of Number of Clients. In Table 1
increased the number clients from 3 to 20, and then to 50, ensuring that there is overlap in the training samples each client can access. Consequently, each client receives smaller frac- tion of the total dataset. The configurations are presented in Table 7 in Appendix. We observe that as the number of clients increases, while maintaining the same number of fine-tuning samples, the performance of the LoRA method significantly deteriorates for most datasets. In contrast, RoLoRA maintains its accuracy lev- els. The performance of FFA-LoRA also de- clines, attributed to the limited expressiveness of the random initialization of A for clients' het- erogeneous data. Flex LoRA shows significant degradation especially under high client counts. Notably, RoLoRA achieves this accuracy while incurring only half the communication costs as- sociated with LoRA and FlcxLoRA. A comparison when using rank-2 LoRA adapter is shown in Table 8 in Appendix.
30
CNU
70
70-
LoRA
RoloMA
50
PAA
10
20
30
0
30
20
30
Round
Round
70
40-
20
Round
Round
Figure 3: Accuracies over rounds with RoBERTa- Large models on SST-2, QNLI, MNLI, and QQP. It involves 50 clients using rank 4.
Furthermore, we have provided performance comparison of FLoRA and RoLoRA in the same settings in TTblee[Tinn Appendix. RoLoRA consistently outperforms FLoRA across tasks and client counts. Additionally, Figure 3 illustrates the finetuning dynamics. highlighting that RoLoRA converges substantially faster than the other methods. An extended 100-round version provided Figure 8 in the Appendix, further demonstrating LoRA'ss superior accuracy when all baselines have converged.
Effect of NDn-IID Data Distribu- tion. Table 2 presents a perfor- mance comparison of LoRA, FFA- LoRA. FlcxLoRA, and RoLoRA on the MNLI and QQP tasks under two federated settings: (irichlet(0.55) with 10 clients and Dirichlet(1. (1.blet((1.)) 1.0) with 15 clients. Here, the Dirichlet(a) pa- rameter controls how non-iid the data across clients: smaller a pro- duces highly skewed and heteroge- neous client data distributions, while a larger yields more balanced, moderately non-iid splits. Across both tasks and settings, RoLoRA
Dir(0.5), #Clients = 10
Din'' (.0) #Clients = 15
MNLII
QQP
MNL.I
QQP
LORA
81.19
82.60
74.54 11.19
81.49 10060
FFA-LORA
75.60 11021
81.47 .10.7
74,83 10.59
78.62 1:167
FleLLORA
+000
40.9
#:448
RoLoRA
82.60 10.69
84.16 10:65
81.55 10.44
84.26 0:226
Table 2: Performance comparison of methods on MNLI and QQP under different dirichlet distributions and client settings. We report the average and std. over three seeds.
8

---

2312
74
19VA
Sunbercl
Nutbber paraTs
lanzer p8/0e
Nattbr parmm
Figure 4: Results with RoBERTa- models GLUE under different fine-tuning parameter budgets, involving three clients with rank
consistently achieves the highest performance. The configurations are presented in Table 7 in Appendix.
Effect of Number Finetuning Parameters. In Figure we compare three methods across five GLUE datasets. We apply LoRA module to every weight matrix of the selected layers, given different budgets of LoRA parameters. For each dataset, we experiment with three budgets (P1, P2. P3) ranging from low to high. The corresponding layer sets that are attached with LoRA adapters, P2,P2, P3. are detailed in Table 12 in Appendix A5.2 The figures indicates that with sufficient number of finetuning parameters, FFA-LoRA can achieve comparable best accuracy LoRA and RoLoRA, aligning with the results in [32]; as the number of LoRA parameters is reduced, the performance of the three methods deteriorates to varying degrees. However, RoLoRA, which achieves performance comparable to LoRA, demonstrates greater robustness compared to FFA-LORA, especially under conditions of limited fine-tuning parameters. It important note that with the same finetuning parameters, the communication cost of RoLoRA and FFA-LoRA is always half of that of LoRA due to their parameter freezing nature. This implies that RoLoRA not only sustains its performance but also enhances communication efficiency. Additional results of varying ranks are provided in Figure
9
T0
and 11 in Appendix A5.4
BoolQ
PIQA
SIQA
HellaSwag
LoRA FEA-LaRA
61.42aam
33.19
31.88
21.23
53.43
35 29.49
10.63
11.81
RoLoRA
6183+022
61.26+33
39.76+041
27.49
WinoGrande
ARC-2
ARC-c
OBQA
LORA LORA
31.36+51
27.36
32.03
26.07-2332
20% PRACUaIA 50% houurt
161
6.88
7,93
15.0
405 ITA LoA- 50% RoLARA
Rol oRA
47.67+077
33.19 1,29
40.13
31.67
Table 3: Results with Llama-2-7B models on common- sense reasoning tasks. This involves 50 clients using rank 8.
Figure 5: Ablation study learning vs. freezing on MNLI task.
5.2
Commonsense Reasoning Tasks
Results. We evaluate RoLoRA against FFA- LORA and LoRA Llama-2-7B for commonsense reasoning tasks. In Table 3 we compare the results of three methods with Llama-2-7B models on 8 commonsense reasoning tasks. The configurations are presented in Appendix A5.7 The performance is reported the mean accuracy with standard deviations across 3 trials. RoLoRA consistently achieves the highest accuracy across all tasks, demonstrating significant improvements over both LoRA and FFA-LORA We also highlights that FFA-LoRA exhibits large performance variances across trials, such as standard deviation of 9.55 for PIQA and 8.44 for SIQA, respectively. This significant variability is likely due to the initialization quality of parameter A, as different initializations could lead to varying optimization trajectories and final performance outcomes as discussed in Section 4 Additional results on this task are presented in Table I6 in Appendix A5.7

---

learned) for 5.3 Ablation Study
Learning vs. Freezing we conducted an experiment comparing performance of FFA FFA-LORA, RoLoRA, and different mixing strategies under the setting with 50 clients. In these strategies, for example, Ro the first 20% of communication rounds, followed by FFA-LoRA (where A is frozen) for the remaining 80%. The results are shown in the Figure We observe that finetuning with RoLoRA generally leads to faster convergence and higher final accuracy. This highlights the benefits of learning A, especially in early training.
means we finetune with RoLoRA (where
Symmetry vs. Asymmetry Update In standard LoRA implementations, LoRA-A randomly initialized while LoRA-B is set to zero, which implicitly assigns asymmet- ric roles. In our study, we view LoRA-A as a learnable basis and LoRA-B coefficients that basis. Motivated by this, we investigated whether an asymmetric update policy might be preferred. We systematically compare four strategies: (i) the symmetric alternation used in stan- dard RoLoRA, (ii) B-prioritized multi-step updates (B, B, B, A, ...,, (iii) A-prioritized multi-step updates (B, A, A, A, ting Ir = 2ir or lr = A)r As shown in Figure balanced AB alternation yields the highest accuracy and the most stable trajectory, while aggressively prioritizing either A or B degrades performance.
Test accuracy (96)
RoLoRA (symmetric alternation) Unequal LR Ur 2ir A)
80.0%
D-prioritized (808--A cycle)
A-priaritized (B (B-AAA cycle)
Unequal LR 4 A)
and (iv) unequal learning rates (e.g., set-
Rounds
Figure 6: Asymmetry vs. Symmetry in LoRA updates. Accuracy vs. round.
Effect of Local Steps on RoLoRA and FFA-LORA To evaluate the effect of local steps, we have conducted an ablation study on the number of local steps in 50-client setting with rank-4 adapters, as shown in Table 4 For a fair comparison, we kept the total computational budget (#Local Steps x # Rounds) constant across all settings. FFA-LoRA's performance consistently degrades on both datasets as the number of local steps increases. This indicates that with more local work per round, FFA-LoRA suffers severely from client drift, where local models overfit their own data. RoLoRA maintains high performance across all settings.
(Local Steps, Toral Rounds)
(1,600)
(5.120)
(10,60)
(20,30)
FFA-LORA RoLoRA
72.52 110.68
71.73 1:11
69.64 1:4.11
69.97 1:5.57
MNLI
84.39 4.034
84.96 4:0.18
84.79
4:0.3
82.98
4:3.36
QQP
FFA-LORA RoLoRA
80.51 1:3.38
80.2 =1.65
79.07
11.21
78.44 1041
85.24 ::056
85:44
84.77
4:0.77
Table 4: Results on RoBERTa-Large on MNLI and QQP with different local steps while keeping total
computational budget constant.
5.4 More Results
We provide additional experimental results in the Appendix, including: (1) evaluations of Llama-2-7B on HumanEval and MMLU tasks (Appendix A5.8); (2) comparisons of communication and time costs in Table 19 and (3) evaluations under a fixed communication budget in Figure 12
Conclusion
In this work, we introduced RoLoRA, a federated finetuning framework based on alternating opti- mization for LoRA adapters. Our approach addresses key limitations of prior methods by jointly learning both the down-pree proj-proecttion and up- up-projection matrices, thereby enhancing the expressiveness and robustness of the adapted models. Through combination of theoretical analysis on linear models and validation on nonlinear models, we established the importance of optimizing both components in LoRA. Extensive experimental evaluations across various language models tasks, and diverse federated learning settings confirmed that RoLoRA consistently outperforms existing baselines, particularly in large-scale scenarios under constrained communication and resource conditions.
10

---

Acknowledgments and Disclosure of Funding
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute www.vccooinssiuue..aaaattessssss
References
[1] Sara Babakniya, Ahmed Elkordy, Yahya Ezzeldin, Qingfeng Liu, Kee-Bong Song, MOSTAFA EL-Khamy, and Salman Avestimehr. SLoRA: Federated parameter efficient fine-tuning of language models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.
[2] Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources, 2024.
[3] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on heer-held data, 2016.
[4] Sahil Chaudhary. Code alpaca: An instruction following llama model for code generation. https://gt whw.:b...................m...... thub.
com/ sahi wahi1220114/ https///111111//oaapaa.... codealpaca 2023.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. 2021.
[6] Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish J Khisti. Robust federated finetuning of foundation models via alternating minimization of loRA. In Workshop Efficient Systems for Foundation Models II @ ICML2002, 2024.
[7] Shuangyi Chen, Yue Ju, Zhongwen Zhu, and Ashish Khisti. Secure inference for vertically parti- tioned data using multiparty homomorphic encryption. In 2024 IEEE International Symposium on Information Theory (ISIT), pages 2508-2513, 2024.
[8] Shuangyi Chen, Anuja Modi, Shweta Agrawal, and Ashish Khisti. Quadratic functional encryption for secure training in vertical federated learning. In 2023 IEEE International Symposium on Information Theory (ISIT), pages 60-65, 2023.
[9] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. Marina Mcila and Tong Zhang, edi- tors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2089-2099. PMLR, 18-24 Jul 2021.
[10] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fedavg with fine tuning: Local updates lead to representation learning. 2022.
[11] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6))141-1444 2012.
[12] Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. International Conference on Learning Representations, 2021.

---

[13] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-284. Springer, 2006.
[14] Ahmed HI Ouadrhiri and Ahmed Abdelhadi. Differential privacy for deep and federated learning: survey. IEEE access, 10:22359-2242 22380, 2022.
[15] Pengxin Guo, Shuang Zeng, Yanran Wang. Huijie Fan, Feifei Wang, and Liangqiong Qu. Selec- tive aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, 2025.
[16] Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors, 2024.
[17] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, Measuring massive multitask language understanding. 2021.
[19] Erfan Hosseini, Shuangyi Chen, and Ashish Khisti. Secure aggregation in federated learning using multiparty homomorphic eneryption. CoRR, 2025.
[20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiieet transfer learning for nlp, 2019.
[21] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[22] Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing, STOC 13, page 665 674, New York, NY, USA, 2013. Association for Computing Machinery.
[23] Jabin Koo, Minwoo Jang, and Jungseul Ok. Towards robust and efficient federated low-rank adaptation with heterogeneous clients, 2024.
[24] Weirui Kuang, Bingehen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-lInt A comprehensive package for fine-tuning large language models in federated learning, 2023.
[25] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020.
[26] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks, 2022.
[27] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
[28] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication-eeficieeen learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[29] Konstantin Mishchenko, Rustem Islamov, Eduard Gorbunov, and Samuel Horváth. Partially personalized federated learning: Breaking the curse of data heterogeneity. Transactions on Machine Learning Research, 2025.
12

---

[30] Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17716-17758 PMLR, 17-23 Jul 2022.
[31] Seyedehsara, Nayer, and Namrata Vaswani. Fast and sample-efficien federated low rank matrix recovery from column-wise linear and quadratic projections, 2022.
[32] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated learning. In The Twelfth International Conference on Learning Representations, 2024.
[33] Rohan Taori, Ishaan Guirajani. Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An inslrwition-followinng Ilama model. attps://ght thub.
wawiaatsulllassaaaeeaaeee ord alpaca 2023.
[34] Kiran Koshy Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. Sample efficient linear meta-learning by alternating minimization, 2021.
[35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[36] Namrata Vaswani. Efficient federated low rank matrix recovery via alternating gd and mini- mization: A simple proof. IEEE Transactions on Information Theory, 70(7):5162-5167, July 2024.
[37] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[38] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[39] Shuai Wang and Tsung-Hui Chang. Federated matrix factorization: Algorithm design and application to data clustering, 2020.
[40] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations, 2024.
[41] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farokhi Farhad, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis, 2019.
[42] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fune-tuning for transformer- bbssdd masked grgguage-mdeehe 2022.
[43] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedGPT: Federated instruction tuning. In International Workshop Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.
[44] Longteng Zhang. Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023.
13

---

[45] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. FedPETuning: When federated learning meets the crrameter-effiiienn tuning methods of pre- trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 9963-9977, Toronto, Canada, July 2023. Association for Computational Linguisties.
[46] Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzych Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models, 2024.
14

---

Appendix
A1 Algorithms
16
A2 Discussion
16
Theoretical Analysis on Linear Model
17
A3.1 Notation
17
A3.2 Auxiliary
18
A3.3 Homogeneous Case
19
A3,4 Proof of Theorem 5.4
25
A3.5 Heterogeneous Case
31
Convergence Analysis Non-Convex Case
36
A5 Experiments
41
A5.1 Impact of Non- Kin-arine on RoLoRA
A5.2 Hyper-parameters for GLUE task
41
A5.3 Effect of Number of Clients
41
A5,4 Effect of Number LoRA Parameters
42
A5.5 Effect Rank on RoLoRA and FFA-LoRA in the Centralized Setting
43
A5.6 Align the Communication Cost
44
A5.7
Commonsense Reasoning Tasks
44
A5.8
Language Generation Tasks
45
A3.9 Communication and Time Cost Comparison
45
A5.10
Privacy-preserving FL
45
15

---

A1
Algorithms
Algorithm 1 RoLoRA iterations
1: Input: number of iterations T, number of clients N
2: for to T do
3: for 1 to N do
4: Fix A', B B
GD-update B²)
5: Transmits B to server
6: end for
7: Server aggregates B
N E B broadcasts B
8: for = to N do
9: Fix B ++1 GD-update A', 1,B+))
10: Transmits A [+] to server
end for
12: Server aggregates A 1+1 1
N EE Att1, broadcasts A H+1
end for
Algorithm 2 RoLoRA for linear regressor, Alt-min- iterations
1: Input: GD Step size number of iterations T', number of clients N
2: for t 1 to 7 do
3: Let a < a b x bd-1
4:
for
to . do
5:
set b; + arg min, (a,
6:
end for
7: b
== b for to N do
8:
9: Compute Val(a,
10:
end for
11: a N +
Vah(a, b), ât 4
T
12: t b' <-b
end for
A2 Discussion
While Algorithm is conceptually inspired by alternating optimization techniques in matrix sensing and multi-task linear representation learning (MLRL), but it differs in the algorithmic design, and application focus.
Connection to Matrix Sensing. The problem in Eq. (6) is an instance of matrix sensing. Traditional matrix sensing methods (22] focus on recovering low-rank structures from centralized data, whereas RoLoRA is designed for a federated setting, where both data and computation are decentralized across multiple clients. related line of work is federated matrix factorization, which also applies alternating minimization techniques in distributed environments. Wang et al. [39 perform alternating minimization between local matrix factors within each communication round but do not alternate the aggregation steps. In contrast, RoLoRA alternates both the updates and aggregations of down- and up-projection matrices across rounds, fundamentally changing the communication pattern and mitigating interference between matrix components during aggregation.
Connection to MIRL. As discussed Section we connect the objective learning down- projection matrices in a federated setting to multitask linear representation learning (MLRL) 9, 34) 12]. We follow a similar derivation framework MIRL works that employ alternating optimization, such as FedRep 9 and Vaswani et al. [6) However, their focus is on multi-task setting, where the heads are kept diverse and not aggregated. As result, they perform alternating minimization and gradient descent between the local representation and head within each communication round
16

---

but do not perform the aggregation steps for the local heads. Furthermore, there are fundamental differences in model structure: in FedRep, the local head acts a foon-projection, whereas in RoLoRA, A, the corresponding component B serves as an up-projection. This distinction stems from RoLoRA's foundation in the LoRA framework, where adaptation uses down-projection followed by an up-projection, with B mapping back to the original feature space.
Connection to Personalized FL. While RoLoRA is positioned within the global model paradigm, it is worth noting that certain algorithmic structures share interesting similarities. For instance, Mishchenko et [29] and Pillutla et al. [30 fall under the category of personalized federated learning, but their update mechanism bears resemblance to Rol RoRss'ss alternating optimization. In [29]. clients compute the gradient of their local loss with respect to the global parameters, holding the personalized parameters fixed, and send only this gradient to the server for aggregation. Pillutla et al. [30] adopt an alternating update strategy in the proposed FedAlt algorithm, where clients update personal parameters while holding global parameters fixed, followed by updating the global parameters. Both resemble the alternating structure in RoLoRA However, RoLoRA operates entirely under global model setting without personalized components. Furthermore, alternating scheme motivated by the decomposition of low-rant adaptation matrices, separating the optimization and also the aggregation of up- and down-projection matrices. Moreover, the underlying proof techniques differ entirely: they employ standard FL convergence analysis, whereas our approach draws on techniques from matrix sensing to highlight the importance of learning the down-projection.
Limitations and Future Works. While RoLoRA demonstrates robust performance across various federated learning settings, our work has few limitations. The study of learning Nown-projections has primarily focused on linear models, which may not fully capture the complexities of highly non-linear language models. While empirical validation has been conducted non -inelr models, a rigorous theoretical proof is still lacking. We leave the theoretical extension to simple non non-linear models as important future work, although empirical results suggest the method remains effective. Second, the current analysis assumes full client participation in each communication round, which may not hold in real-world federated deployments with intermittent connectivity resource constraints. A theoretical guarantee for partial client participation is needed.
A3
Theoretical Analysis on Linear Model
A3.11
Notation
Table 5 provides a summary of the symbols used throughout this theoretical analysis.
Notation
Description
a²_ b;
Ground truth parameters of client
b
Average of b;
af
b
Global model parameters of t-th iteration The angle distance between a" and a², sin A(a", a²)]
6,
n
Step size
Ia
d x identity matrix norm of vector
II-1
1pop
Operator norm (12 norm) of a matrix
Absolute value of scalar
Sub-Gaussian norm of sub-Gaussian random variable
Sub-exponential norm of sub-exponential random variable
N
Total number of clients
á
Updated a by gradient descent Normalized à
b.
analytic solution for the local objective function
5
Average of b;
Table 5: Notations
17

---

A3.2 Auxiliary
Definition A3.1 Sub-Gaussian Norm). The Gaussian norm random variable ç. denoted as ag is defined as:
|5 inf 0 Elexp({ s 2}.
random variable is said to be Sub-Gaussian if its sub-Gau Gaussin norm is finite. Gaussian random variables are sub- Gaussian. The sub-Gaussian norm of standard Gaussian random variable & ~ (0, 1) is || v8/3. Definition A3.2 Sub-Exxonential Norm). The sub-exponential norm of a random variable denoted | defined as:
| inf 0 E[exp((<//tt) s 2).
A random variable is said to be sub-exponential if its sub-exponential norm is finite.
Lemma A3.3 (The product of sub-Gaussian is sub-exponential), Let and be mub-Gaussian random variables. Then çu sub-exponential. Moreover,
|lEulle & 11Ellas ||
Lemma A3.4 (Sum of independent sub- Gaussianss. Let XI, ,N be independent mean-zero Gab--aassian random variables. Then Ei= i= N2 is also sub- Gab--asssian with
N
Σ
X
SC
1=1
1/2
t=1
where C is some absolute constant.
Proof. See proof of Lemma 2.6.1 of 37
Corollary A3.5. For random vector X Rd with entries being independent standard Gaussian random variables, the inner product a x is sub-Gaussian for any fixed a E R.d, and
||a 1xTxlee, 3C'llall
where C' is absolute constant.
Proof. Note that T Σd 1 a&C. where & (0, 1) the i-th entry of the random vector x. Choose C to be the absolute constant specified in Lemma A3.4 for standard Gaussian random variables, and set C' = /8C/3. We have
a x SC Σ
|arE 12
(a)
(b) 3
Clal| *
||ax||
I
8C 3
|a||
C' [|||a
Step (a) makes use of the homogeneity of the sub- Gaussian norm, and step (b) uses the fact that || v v8/3 for & ~ (O,,
Definition A3.6 (c-net). Consider a subset A C Rdd in the d-dimensional Euclidean space. Let > 0. subset. A C A is called e-net of A if every point of A is within distance of some point in N, i.c.
Vx A. Ex' e N. ||x S
Lemma A3.7 (Computing the operator norn net). Let Bd and e [0, 1). Then, for any c-net N of the sphere 8d-1, we have
1
|a|| - - E xEN
sup (a, x)
Proof. See proof of Lemma 4.4.1 of [37].
18

---

Theorem A3,8 (Bernstein's s inequality). Let Xi...., be independent mean-zero sub-exponential random variables. Then, for every? t 2 0, we have
t
P
x
2t
exx
-emin
Σ i= ||
max; ||Xilloy
where >> O is an absolute constant.
Proof. See proof of Theorem 2.8.1 of
A3.3 Homogeneous Case
Outline of Proof. In this section, we first analyze the minimization step for updating b (Lemma A3.9), then establish bound on the deviation of the gradient from its expectation with respect to a (Lemmaa A3.10), and finally derive bound on sin 0(a aa)l based on the gradient descent update rule for a (Lemma (r semma[A3.11] The proof of Theorem i.5 provided in Section A3.4. where the result is obtained by recursively applying Lemma 4.3 over T iterations.
Lemma A3,9. Let = a'. Let 8 [[L a'a" )all [[ aa >aa denote the angle distance between a* and a. Let g' = a aaab"
>N ,b = N == b, If m O(q)) and =
max log(N) d'oog( to [m)] o then with probability 1 1-9
bb-gl gll S <'0''
(11)
where e' [I [I-eal(1-rr) for E (0, 1).
Proof. We drop superscript for simplicity. Following A we start by computing the update for b;. With g² a aa"b"" get:
6.J X, X
(12)
a X, X,a
X, X,aa b.
+ X aa ''ab""
(13)
a X X,a
[La aa )a'b'
g +
(14)
X X,a a
Therefore,
[b. gl
X X,a
-1.[|a Tx. (Ia -aaaaaa "aa"""abb" = 2. (Ia aa Ja"b"
(15)
Note that since each entry of X; independent and identically distributed according to standard Gaussian, and |||| = 1, X,a is random standard Gaussian vector. By Theorem 3.1.1 of [37], the following is true for any é E (0, 1)
cmefm K
P 2 < (1 -1--ah cmmm I exp (-
(16)
where K IKEl| for & 1o, and e1 is some large absolute constant that makes 16) holds. Next we upper bound Tx X2 aa aaab" bb ||. Note that E(a x X,
0. First we need to apply sub-exponential Berstein incquality to bound the deviation from this mean, and then apply epsilon net argument. Let.V be any en-net the unit sphere Sd in the d-dimensional real Euclidean space, then by Lemma A3.7
>a*b*
[X (T, aa )aab"* ma (I. aa aaT "'a""""""
we have
la X X, (I. aa 'aab"'
15
1
a
T X XI, aa "a"""""" w
(17)
1 eo WEN
1
max
T
X
(L aa Jaab" w
(18)
Eo WEN
19

---

Meanwhile, denote the j-th row of X; by x for every w E N, we have
X (Ia aa Daab"
Σ aa x (x, (Ia aa "aab" w)
(19)
On the right hand side of (19). a X and x (I, aa )a b" w are sub-Gaussi random variables. Thus, the summands on the right-hand side of (19) are products of
T
random variables,
making them sub- subbeexonential. Now by choosing 02 = (C")² for the C" in Corollary A3,5 we have the following chain of inequalities for all i,
[1(a (x,) (La aa aaab" w)lles
1 ||a
X 1lv ||x (La
aa |1(A2 aa )a'b b w|
Jaab a
wll
(20)
S ||a||
(21)
S 02 |||| [| 1 aa
Da'b*
||op||/||
(22)
I [[ aa "aa"""""
llop
(23)
Equation (00 is due to Lemma A3.3 21 is due to Corollary A3.5 ||w] 1.
23
is by the fact that ||a||
Furthermore, these summands are mutually independent and have zero mean. By applying sub- exponential Bernstein's 's inequality (Theorem A3.8) with & e2m] aa )a*b" llops we get
P a x. XIa - aa "aaaa"b ")""""" w 2 <2m]|(Ia - aa Jaab" 1lop
(24)
m
=P
Σ
T x (x 5 (Ia aa "a"b" w) 2 eem aa aaab"
ab llop
(25)
exp mm3333)
(26)
for any fixed w N. (2 E (0, 1), and some absolute constant C3-
26 follows because
<2m² [[T aa?
T )aa 'b
2
2m
(27)
ン== (a T x1, (x{ (Ia aa ba
m (I. aa )aab" 1lop
m
(28)
max, [1(a x,,) [x (Ia aa b w w)l
02
And 3m C
Now we apply union bound over all elements in W. By Corollary 4.2.13 in there exists an co-net N with [ < (2, + 1)a, therefore for this choice of A,,
5am eq
33
X (I. aa D'a T 2 c aa Jaab" bb 1lop
P
max
(29)
Σ
P
X
(La aa Ja'b' w c2m ([I, aa )abb
b² llop)
(30)
weN
2
+1 1
exp (-c3e3m)
(31)
60
2 exp (d (1+ + 2g c3(2m)
(32)
Combining 15 16.16 16
18 and 32 we get
IP [lb. gll S é (1 aa Jaab" 11o0 2 21- - Po
(33)
where e' [I-ea]1-e]] and Po 2xxp (d log(1 + 3)= - cze3m) - =expl (- c
Using union
R
bound over E [N], we have
N
P
b: gll <d' aa >a" 1lop 21 21-Npa Npo-
(34)
20

---

Next we bound | gll where is the average of (b; }
N
||b - glb = 11
b
ggll
(35)
N
1=1
Σ i=1
b; gl
(36)
N
(TT aa ja'b* llop
(37)
clI aa )a*l ||b
(38)
(39)
with probability Npo-
336
follows by Jensen's inequality. 38
follows since ||uv T llop =
||ul| ||v]|. (39) follows since 8²
[I( aa )a*l-
N) a log' 2
If m = Mg), where : = max( min[er
), then 1 Npo > - exp(-Co) > 11---110
for large absolute constant C. Then with probability at least 1 - 9 10
||b - gll S |'00l
(40)
Lemma A3.10. Let a'. Let 8 II[ a a
all II(I. aa Ja'll denote the angle
distance between a" and a. Then for Nm !
dlog( 2
and q = max( [min((1.e2))))
log(
dloog
with
probability least 2q-10.
[V b) - E[V b)]ll S 27((6') + + +'')8'11b'))
(41)
where غ == T-eo T and é' = (T--oi(1-a) for ED. E1, 63, E (0,
Proof. Based on the loss function I(a, b) E
4(.bb 1
Nm E "X,a'''' X,ab 1|3
we bound the expected gradient with respect to and the deviation from it. The gradient with respect and its expectation are computed as;
N
2
Val(a,b)
x
b-
(42)
Nm
2
(X] X,a - b)
(43)
Nm
i=1
N
2
x, Xab aa" }b
(44)
Nm
==1
N
V b)]
2
Σ
m(ab T - aab }b
(45)
Nm
i=1
=2(abT - -abb" )b
(46)
Next, we bound || ((a b) [VVa((a b)]l Construct net over dimensional unit spheres
1d by Lemma A3.7 we have
V b) E|V (a, b)]l
(47)
< max w Val(a, w I[, b)]l
(48)
60 WEN
2
<
wX,, -a"b' aabb'']b== w T(ab"" -a'b" }b
(49)
6D Nm WEN
21

---

where x, is the j-th row of X, Observe that X and (ab aab"") }b are sub-Gaussian variables. Thus the product of them are sub-exponentials. For the right hand side of 449 the summands are sub-exponential random variables with sub-exponential norm
[[(x w) (x: ab -a"b" 1)b w (ab
T -a'b") )b|len
(50)
- [xx (Xi, - -a'b" ))bllo, + || [| (abT - -a }bllys
(51)
< [x (x (ab - -aa )bbll + w (ab" - -a"b" }b]
(52)
log 2
w (ab a a"b"
"b* }bb
0 |||| ab aa"b" llpp ||b||
(53)
log
|w| (ab T a *b"* }bl
< 02 |||| ab =abb"
1lap ||b||
+
(54)
log 2
[w|] ||ab T
aab "b*
1lop |b]
S 02 ||w|| ab
a²b"
1lpp ||b]|
(55)
log 2
alb
a b
1lop ||b]
(56)
where C₁ = C2 +
log 2 some absolute constant greater than 1. Equation 52 due to the fact that
for a constant E R
|elles inf exp { |d
lcl log 2
Equation
(53 is derived similarly as 20 22
The summands in (49) are mutually independent and have zero mean. Applying sub-exponential Bernstein inequality (Theorem A3.8) with t e3NMmllab a²b |
N m
P
[(xx]] () - aab=")) w/ T (ab -a"b"" })bb
-abb"))bb] 21
(57)
N egNm
< 2exp
(-mmin(
(58)
= 2exp (-c3e3Nm)
(59)
for any fixed w E N, E (0, and some absolute constant
Now apply union bound over all E N using Corollary 4.2.13 of 337 We conclude that
F
max
w
T
Val(a,b) w W EV 2 2cg/|abb ab aabb" 1lop |b]]
(60)
WEN
Σ P
w E|Va b)]l 26a/ a²b" ll |b]|
(61)
WEN
< 2xx (d + 2 -3ggNm)
(62)
60
22

---

Combining
339
48
and 600, with probability at least 1 1-2 exp (dlog + 2) -csef-N)- 10
1
VV) E a b)ll
max WEN
Val(a, b) w V
(63)
263
|ab a'b" llop |
(64)
1
263
|aabb g) T (Ia aa Ja'b" llop |||]
(65)
1
263
$ 1 60
(1 (b -g) + I( aa Jaab" (lop)
(66)
263
(| gll + (1 aa Ja"ll ||b|]]]
(67)
60
2cg
(| gll + 8' | g + ggl
(68)
60
283
< 1 60
( gll + I) gll + lgll]
(69)
283
<
( gll
+
gll + |
(70)
1 60
263
(| gl|
g||| | gl ||b||| + 8'
b
|13)
co
(71)
263
((6)) ( +6') e'0' + 83)11b"12
(72)
1 60
2cg
$ (e' + 1|=10
(73)
60
25(é' + 1)²8"|
(74)
with ē ca_
664
uses
660
(66) follows by triangle inequality. (68) follows by 8 [[
aa )aaIl- 70
uses
lg
l a'a
all < b
73 follows by (8t)2 8 since 8 (0,
dog( + og
If Nm
then existing large constant C,
2 2) Nm) - 10-10 > 1 - exp(-CCp 9--10
- 2 exx (d +
(75)
Eo
> 1 d d-10
q 10
(76)
> 1 24
110
(77)
Thus with probability at least 1 2q-10 10
774 holds.
Lemma A3.11 (Lemma 4.3). Let a = a'. Let 8 = [[(Ia - a"a" )all = [[ - aa )aall denote
the angle distance between a" and a. Assume that Assumption 4. holds and 8¹ < 8z- .... < 80.
Let m be the number of samples for each updating step, let = (1-r0)(-e1)) = cs for
22
1-c0
Eo, E1, E2, (0, if
log(N) [min(e1, (1,, e2)
dlogg 2
Ω max
cp
2
and é',ē < 1-189} for any and < Th then we huve,
8c-1 s 82 V n71 - -802) | 1|2
(78)
with probability at least
29 10 for = max
log(N)
log
[min(er.c)]
23

---

Proof. Recall that à = - a b). We substract and add E V b)], obtain
à = a v b)] + (/ b)] V (al b))
(79)
Multiply both sides by the projection operator -
Pat Pa
Val b))
(80)
Pa 2,/P((al (ab a )b b)] Val(a,
(81)
Pa 2,/Pab b PPt(ePP -, aa, b))
(82)
(a - 27b (abbbb) b): [)) V. b))
(83)
where 881
uses E
V b)] 2(ab a'' bb. 82 follows by Pa" 0. Thus, we get
PP |Pa|||1 2//b b + V Val(a, b)]l
(84)
Normalizing the left hand side, we obtain
Pa +
|Pa|| 2nb Tb (E b)] Val(a,
(85)
||+1|
lat
8'11 27
[Va b)]
Val(a, b)
= 8t+11
S
(86)
[at
E1 E2
(87)
|
where (86 follows by 5++11
||Pa*1 and 5
Pall
We need to upper bound E₁ and Eg
||ā+
accordingly. Eg is upper bounded based on Lemma 13.10 10
With probability at least 1 - 24 10
Eg 7) V b)] Va b)ll
(88)
S 27j(E' + 1) |b1
(89)
To upper bound Fj, we need to lower bound |b| We can first lower bound [[b]| by:
|b]
|g - (g b)ll
(90)
z gg |gg bll
(91)
= 1 (8t)2 |b|| [g - bll
(92)
2 (8+)² |b*l 2'8'
(93)
with probability at least
992
follows by g¹ a TT a'" and aa* cos ((a, a²)
follows by Lemma A3.9
Assuming 5' S ... I 80, we choose d' <
1- to make
16
(8t)2 b*|
e'5'|
0. Hence [[b]|2 is lower bounded by:
||b] 2
2 (v/ -
(94)
(1
-2'8' -1
(95)
2 (1 (6*) (6))))))
-c'Ilb112
(96)
2 (1 [b* -b"b'="
(97)
with probability least ạ 10
(96) follows by xy < 1/2 for x² + +2² 97
follows by assuming
8t S 8t-1 ....5 80 Eg is upper bounded below.
E
8' 2//b² b)
(98)
< 8 1 27y((1 ( ( 11
(99)
with probability at least
1 17 -10 Next lower bound ||â+l] ||-
10,
| 112
=|aa- - - b)]l2
(100)
=a a + 1 aa b)]l 2na Val(a,b) b)
(101)
2 Ta 2naa" Va((a, b)
(102)
= 1-2ma Val(a, b)
(103)
= 1 - 1-77aa (V,!(a, b) b)), 27ja" E[V,,((a,b) [)]
(104)
24

---

where 102) follows by (a b)ll² 2 0, and (T03) follows by a a = 1. The first subtrahend
21ya" (V b) - E a b)]) is upper bounded such that
2na (V. - ())) - 2|||a|| (I(Val(a, b) - (l b)DII
(105)
(V, b) al b)]ll
(106)
- énē(é' + bb*1@
(107)
with probability at least 1-29 1-2--10 (107 uses Lemma A3.9 such that
The second subtrahend is upper bounded
2nja E[Val(a, b)] 4yja T (ab T
a²b" }b
(108)
4naa (ab T a"b T )g 4na (ab -a'b' [(g - -b)
(109)
where at (ab aab' 0g = a ((Ls aa 1 baab"' + aa b)" Ig = (b-g) g) g. The second
term is simplified via a (ab - a²b² (g - b) = a ((L, aa )a"b" +a(g- - T) - g) =
- (g - b)?. Both simplifications use (Ia aa = and a a = 1. (109 becomes
T
2naa T a[V b)] = 4n/b g) g + 4n - b)²
(110)
< <4vllg - ||]]| |||||b|| || + 4nllg - 2112
(111)
-
(112)
< 4n(()))² |'
(113)
with probability at least
q 9110 112 uses Lemma A3.9 Combining 107 and 113
we obtain
||á + 2| 4nyē(e' + 1)2)))bb1] -4n(())))))(())) b'
(114)
with probability at least - 24 10, Combining 99 and 114). we obtain
E1 + Eg
81(1 27 (89)2)
v1 4njē(e' + 1)2 b* b + 2mp + 1) (b
8++1
8'C
(115)
|
such that (1 (1- (69)) > 4n((e)+++')]b
160)
We can choose é',ē <<>
16
holds. Then we obtain
|b == 21y 217(11 e 11 2
(116)
v1 Anē(e' + -
1 21
21ye [10 2 (erf((e'+))))) 11)
(117)
v1 (
21/(1 [80)) n/26' + 25(e' + |1>*|
(118)
v1 (n(((c' + +12 + (c)) + c/)lb=111
1 17(1 (8")²) |+
<
(119)
V 711 (0")²) b-
- 1 - |
(120)
Assuming 7 - T <
1 b 1 - 77 ( [|b)))|[+*|1 is strictly positive. Therefore we obtain, with
probability at least 1 29 10
81+1 < 8' / 11 - (80)2)|b*11
(121)
A3.4 Proof of Theorem 5.4
Proof. In Lemma 4.3 we have shown the angle distance between a and decreasing in t-th iteration such that with probability at least 1 - 24 for g max(log(N)) d}. 5 < S for (0,1),C= C v1- 11-e(L
ca1-cc11000 - ((8)2) with proper choice of step size 17.
25

---

Proving 8'
I 8
Now we are to prove that for the first iteration, 8' S C with certain
probability.
By Lemma A3.9
we get |b gll v 5 |b*| with probability at least 1 - q--10
By Lemma A3.10
get V Val(a. (a. b) [aa b)]ll - 2ē((E')² + + 21 with probability at [|""|
least 1 2q 10
By Lemma A3.11
without assuming decreasing angles, we obtain, with probability at least 1 1-20 2-2
8² 8 m1 (
(122)
Inductive Hypothesis. Based on inductive hypothesis, by proving 8 S C, the assumption that 8' S. 188-10' S C' and proving 8'+1 s 8' C' we conclude that 8 I 8t-1C' holds for all t [T] iterations. We take union bound over all E [7] such that,
P
8t 8 8'VV -11--11-- -1-(())2)
2T4
(123)
Solve for T.
In order to achieve e-recovery of a", we need
(1 - c11 (89)2)) 3 Se
(124)
(1 - c(1 (5)) _
(125)
80
T
c gm log (1 c(1 - (b))²) log(
(126)
2
(127)
We proceed such that
log
T2
(128)
log (1 c(1 (80)2))
2 2log( 30t
(129)
c( (11 - (80)2)
80
c(1 (80)2) log(
(130)
where 129) follows by using (og(1-x) < x for |z| < 1. Thus, with probability at least 1 1 2T--T---- 10
8T sin (aa aT" a²) S c.
Convergence to the target model.
We now aim upper bound ||a ||aT (b a'(b"
Recall
that (g") TT = (a²) T* and 5 = =I((L
- - aa") )a*l. we have
(bT a 'b'
|a"(bT+i)T - a² + a (g") - -ab ||
(131)
< - || a (g") a a'b'
(132)
laT(bT+1 -g") || (aa((aa) )) -IJa'b'
(133)
|a"|||b=+|- - -g'1+ 1 a ()(a)) b*|
(134)
S e'8" <''7 ||b'|++*'|bb]
(135)
(136)
(1 (1l')ccla"""
(137)
where (155 is by Lemma
A3.9 [ ||y|] and ||a" 1.
and the fact that |a" 1, and 137) is due the fact that
Ixy || =
26

---

A3.4.1 Proof of Proposition 5.5
Proof. We start by fixing a and updating b; to minimize the objective. Let a = a". We obtain
b,_ a a'x, Xaa"""
(138)
X X,a
1 N
X
Xaab"
(b P
(139)
a X X
let b = PFAA, We aim compute the expected value of t E m the expectation is over all the randomness in the X, We define
-X,ab where
a T X,a" (X;a)T(Xaa)) ||
S a X
(140)
so that b = 1/2 Ei s,b² sb". For each i, the norm becomes
|xa'''' X,a [[X,a* 3X,a)b" + 1|
(141)
|X,a* sX,al ||
(142)
using the fact that ||uv 112 || |v| for vectors and Therefore, Ei 1/4 E [X,a'"''
X,ab [| is reduced to E Σ Th |X,aa X ||b||?
Since each entry of X, is independently and identically distributed according to standard Gaussian distribution, both and a are unit vectors, the vectors X,a" and X,a are (O, Im). The cross- covariance is aIn, where =all aa".
By linearity, sX || aXlall2 because all (X,a², X,a) are i.i.d. pairs. Let 21 N and 22 satu+sy have [X 21X1a x2 2 Let Xja", u1 21 and u2 22 Thus,
show that 1/ Em 1 "X,a" sX,a|* 2 has the expectation as 1 "x,a
I - = 22 X a = v| u u2ll
(143)
= =v T v + uf u un 2v T un 2v T u2 2u, u2 (144)
Now we compute the expectation term by term.
Expected value v Tv
We have E[y v] E|[|X [|3] m.
Expected value of u, u We have
u u = = x
(145)
87
|
(146)
N2
((xaa) (Xxxa))x)))))
|||a||
(147)
N2
|||a||
((X, (a) (Xia)))²
(148)
N2
||
Note that (Xja², ,Xja) is a correlated Gaussian pair with correlation a = T a." Without loss of generality, we assume = thus a" ae; + v1 a²ez, where and ey are standard basis vectors in Rdd So we can get X,a X X1.1, where X1,1 denotes the first column of X₁.
Accordingly X,a" = aXie + vI X = ox1,1 + vI a=x1, where X1,2 denotes the 1 |x
second column of X,, Therefore (148) can be written as N expectation of it.
Now we take
((xa) (X aa ) |X
+ 2x1,2))
(xT, + 22)
E
E
N2 E
N2
N2
|1,,
|
(149)
27

---

Let 1 || | and = 1 We have
(x ( + =)x
(ar Br2)²
(150)
|x1,1||
T1
a + B3 + 2a.3r122
(151)
71
[392
R +
[a2r]]
(152)
11
where EE [aAri] 2 ||?]] a and E [2a,Br2]
20,8 [r2]
20 [x.],1
because X1,1 and X1,2 independent standard Gaussian vectors. Then we analyze IE
71
B2E Condition on X1,1+
E rr X1,1 EE [x = -x,, [1 = 0
(153)
and Var(r2/X1,1) = |1, = -1, thus
r2 X1,1 x x1,1112 (00,ri))
(154)
Then we obtain
X1,1
(155)
Therefore E
12 X1,1
1.
We take total expectation E
E
E
X1.1
= 1.
Summarizing,
(X a)
(X)
(ar) + 3r2)
=E
(156)
|
[32r2"
E
[aarr]] +
+ E [2a32]]
(157)
a²m + 1
(158)
1
((Xaa) a) 2
E
u1
N2 E
(159)
|||||
a²m
+ (?)
(160)
N2
Expected value of u. u, We have u.j u, = 2 X a| where 22 = b is independent of pair (Xja", Xja). To compute E
X || first we condition on 22 to obtain,
[22]
2 X a
21
22m
(161)
Then we take total expectation E
[23]
X
|aal
|| = E
E
E [=5m] = mE [2]]
(s2
... + 2
(162)
N²
1
E
S
Σ
(163)
22
11j=1
[
N Σ
1
Σ
E
E
[5
(164)
22
2=2
(= 11 1
28

---

Write s; (Xaa) (X,="), Without loss generality, we assume e₁ thus a" a61 foe1+v1= a ||
where e1 and e2 standard basis vectors in R.. Thus, have X,a X,,e X;,1 where X represents the first column of X. Similarly, X,a" aX,e + v1 v1-aaX,02 0x11 vI x2x,,, where x denotes the second column X,.
Hence,
(Xja) (aa))(X;a)) (Xia") xx + a = a
V1 - a (x,,2
(165)
With
|xaa|2= ||x;, we have
8
a: x + V1 - a a2(x,, 1x,2)
|x,,1
a v -1-a²
a² x
||x;, 1
X
(166)
x Then
Let R
[ 1
a / 1 11- 0² 2
[s]]]
(167)
E a² + (1 a² R² 222 a²R
(168)
= o² + - a²) R2
20 a.
(169)
x |
x
For E [R] E
similarly in 154 x x1,2 ||x,111,,111) thus
||
X
(O0 || then
R = = [R|x,,1]]
(170)
For E [R?], since 2,, ~ (000 110,,1x111))). so E x, x ( = |x,,1 Thus, with
x R2
x
E (xx,,x,,)) (xx;111
E Rd [x,1]]
(171)
||x;1|
x 2
[R_]]
(172)
For m--dimensional standard Gaussian vector,
|x 2| follows chi-squared distribution with m
degrees freedom. Therefore, [R²] m
169
becomes
[ 2 (1 a²) ) R2 a² (1 a² m
2a
oZE
(173)
(174)
Now we compute E [sss] for * j. By independence of s; and sy. EE [s.3y] = 15 [s.] [E = E [sx]]
Take expectation of (166).
X
E ss E
a +
(175)
| 1|
x ||x 1
a
+ v1
aFE
(176)
+ v
aF
(177)
(178)
29

---

Hence, E [s,s,] = a². Summarizing,
E
[12 u2]
mB
[>]]]
(179)
m7
((N - 1)E [ + (N - 1)(N - 2)B [s;8,]
(180)
N²
m
(NV - 1)(a²
+ 1\(a (1-ca (1-a² a²)
1
+ (N - -1)(N - 2)a²)
(181)
N2
m
1- a²
(N - 1) (N-1) + (N
(182)
N2
m 2
Expected value of v u
u1 We have v u1 (X,a* T (X1a) (X (X1a). We factor
out N.
E
[
[s (X (Xia)] By (58
E ( (Xia)] = ((Xia) [a] (Xia*))
(183)
||a||
= =mm + (1 a²)
(184)
Then
a²m m (1 a²) E V u1
(185)
N
Expected value of v u2 We have v u2 22 T (Xja). Condition 22 which is indepen-
dent of (Xia") (X1a), (X1a*)(Xxaa) we obtain
E ((22(Xaa")
((X1))22) = 22E (X,a* [(xaa)
(186)
Still we assume a ei thus a* dei + VI-a where e and e are standard basis vectors in R4. With Xia = Xieı X1,1, where X1,1 denotes the first column of X1, and X,a*
axe + V1 X2 0x1,1 + vI - x²x1,2 where X1,2 denotes the second column of X1,
using 165).
E (Xiaa)((Xxss)
= E [e||x1111
V1 11--a ax
(187)
= aB
X1 |2]
+ 22 /1 a?E [x x1,2]
(188)
= am
(189)
Thus 22E
= 220m, Then we take total expectation
E
E (x2(")a)) (x]a)|=]]]
E [220m] omE [22]
(190)
(191)
where 22 txx Therefore.
amE [2g]
Σ
[ss]] (N 1)a²
(192)
N
where 192) follows by E [s:] a. Summarizing, we obtain E U2]
1) (N 1)a?.
Expected value of uf ug We have uf u2
| z1
By definition of 21 and zg, we obtain
1
21 X1a 2
N2
(Xia*) (X1a))
8
(193)
Since (X)a*) (TpP)T((X,, aX) depends only on X,
Σn
2 8 is independent of X1, we obtain
N
N
E
N2 (Xia") T (Xia))
1
Σ
=
N2 [Xia*) (Xia)] ((X(a))
E
Σ st
(194)
1=2
1=2
1
N2 (XIaa))(XXaa)) (N - 1)E [si]
(195)
(N 1)ma²
(196)
N
30

---

where (19)) follows by E (X1a")
(Xia) = am and E [s:] =
Combining (160). 182
185
192
196) and (44).
1
X aa sX,al
(v
in u1 + un U2 2v 11 2v 2vTuz + +2f(u2)
u2)
(197)
m
m
1+ N(4 11mmm - 2
= (1 - a²)
(198)
mm - 2)
= (1+)
(199)
where 8 is the angle distance between a and a". The quantity غ == N(1-m)-2 approach infinity. Therefore,
N²m(m-2) = 00 00 1
Nm as N and
E
X,a (1 (1+1))0))1
(200)
N
m
A3.5 Heterogeneous Case
Consider federated setting with N clients, each with the following local linear model
Si(X;) = X,ab
(201)
where IRd is unit vector and b IRd the LoRA weights corresponding rank 1. In this setting, we model the local data of i-th client such that Y, X, for some ground truth LoRA weights a* Rd. which unit vector, and local b E Rd. We consider the following objective
min GERd.bER
((a
(202)
1=1
We consider the local population loss (a, b) = a b - ab Tb.
We aim to learn shared model (a, b) for all the clients. It is straightforward to observe that (a', b') a global minimizer of and only if a'b' = a'b', where 5 = n Mm bj The solution is unique and satisfies a' = a* and b' = b* With this global minimizer, we obtain the corresponding minimum global error of N D - T|
We aim show that the training procedure described in Algorithm (a", b*). First, we make typical assumption and definition.
2 learns the global minimizer
Assumption A3.12. There exists Lmaz < oo (known priori), s.t. ||b*|| Lmaz-
Definition A3.13. (Client variance) For > 0, we define 72 = N Dm |b; - -b1112 where
D' N ΣN b
Theorem A3.14. (Convergence of RoLoRA for linear regressor in heterogeneous setting) 8 [[aa- aal-ata -0aaa"a"" "'al" be the angle distance between a" and a' of t-th iteration. Suppose we are in the setting described in Section A3.3 and apply Algorithm for optimization. Given random initial a an initial angle distance on E (0, 1), we set the step size S م and the number of iterations
T 2 ai (100)) log(
{) for (0, Under these conditions, achieve the following sin 0(a a²)
(laTT"a))) Se, and (bT a (b") ([laa(("=TT]
which refer e-accurate recovery of the global minimizer.
Theorem A3. follows by recursively applying Lemma A3. A3.16 18.6ffor T iterations. We start by computing
the update rule for as in Lemma A3.15 Using Lemma A3. 15 we analyze the convergence of a in
Lemma A3. 16 We also show the global error that can be achieved by FFA-LORA within this setting
in Proposition A3.17
31

---

Lemma A3.15. (Update for a) In RoLoRA for linear regressor, the update for a and b in each iteration is:
b* =b=b'a a'
(203)
a a+1 = â a' 21/(a b b. a²b² b)
(204)
|at]
where by = -E N by ||at| = |al 2ny T. a a b)ll
Proof. Minimization for b,. At the start of each iteration, each client computes the analytic solution for by by fixing a and solving their local objective arg minp, |lab; ab, 112 where a" and both unit vectors. Setting =a, obtain b, such that
baa" a b, = a ala b,a"
(205)
205
follows since a 1.
Aggregation for b;. The server simply computes the average of (b,) and gets
N
N
bi
Σ ba a b'a"
(206)
The server then sends b to clients for synchronization.
Gradient Descent for å. In this step, each client fixes b; b received from the server and update using gradient descent. With the following gradient
Va4(a, b) 2(ab a"b; b)
(207)
Thus, with step size 7) a is updated such
N
â a+=
V (a, b)
N
i=1
1
(ab b a"b;" b)
N
2n/ab b a"b' 'b)
(208)
a (n
"b a²b" b)
(209)
lât
Lemma A3.16. Let 8 sin 0(a",
a²)l "
be the angle distance belween a" and at. Assume that
Assumption A3.
A3.itiioonn...... 12 holds and 8 S 6,-1 S ... E 80 if Σ 212 max
then
sin 0(a ++1 a²)) 5.t1 S d, (1 21 (60)))11b*13)
(210)
Proof. From Lemma A3.15 A3.15 a ++1 and b computed follows:
b++ a a²
(211)
a a' 2m b - ab= 'b)
(212)
a' 2n'aab Tb a bb]l
Note that a' and +++1 are both unit vectors. Now, we multiply both sides of Equation 212 by the projection operator P = I a²aa") which is the projection to the direction orthogonal to a', We obtain:
Pa' Pm T b P* b
(213)
||a' 2mn(abb b 2 at b)ll
Pa' Pn
(214)
||at -2n(a'bTb a b b)]
32

---

The third term of the numerator is canceled since Pa" = (Ią a² aa") "aaa 0. Thus.
PP ||| 21yb
PPa +
I
(215)
a (y b b a"b b)l
Let 6, |sin O(a"((aa)))) a!)). Equation 213) becomes:
- 2 b
d++1 S 8' ||a' 221/(a/b - a b)ll
(216)
11-2mbTe
6 aaa(1 21/b + 2naa b b
(217)
6.C
(218)
Obviously C2 2 We drop the superscript t when it clear from context. Note that we have
[1
27b
C
(219)
|a(1 2//b bb) 2nja* b b
21b bb2
(220)
(1-21) b) a 417² (b* b)aa + 4m(1 bya b"b b
2n 2b22
(221)
(1 2-27bbb)2 4 (b"
b)² 4n/(1 bbj TaaTaaaaaa Ta
Recall that b bta a =b* cos 0(a" a). (2TT) becomes:
2//56/12
C²
(222)
(1 - 2m/ b)² + 47 (b. b)2 + 4m
2n/b
bb b
a
2n T b
(223)
1 4ny b b(b* b
(1 2nb b)²
(224)
(1 2/322
(225)
where (224) holds because 5² 5* - 5Tb = (1- cos² 0(a", aa)Da* b. > 0. Equation (225)
implies C < 2n lb|l if 2y/[bi| 1, which can be ensured by choosing proper step size
1) S 2D S F
Now by the assumption that 5, S 6 < ... < 50.
C' S 27l | bb2
(226)
= 2n cos 0(a", a) 1*1
(227)
= - 27/(1 - (6*)3)
(228)
S - 21y(1 | |
(229)
Summarizing, we obtain 8++1 S 8'C' 8'(( - 2 -
(
Proposition A3.17. (FFA-LORA lower bound) Suppose For any set of ground truth parameters (a", (b; 1. initialization a", initial angle distance on E
in the setting described in Section A3.3
(0, 1), we apply Freezing-A A scheme to obtain a shared global model (a.) FFAA), where b PPF
aa. b"a" T The global loss is
((a.bFFA)) 3
(230)
Proof. Through single step of minimization on b; and corresponding aggregation, the minimum of the global objective reached by FFA-LoRA. b FFA is obtained through:
by a
b
b a
(231)
a a
N
FFA
b,
b'a" a
(232)
N
33

---

Next we compute the global loss with a shared global model (a..bFFA) Note that we use Tr(.) to denote the trace of a matrix.
N
4(a L(a FFFA
(233)
N
a [b?)) 1
(234)
N
Σ a (b;)) (b") +b²( - F/
(235)
1
+1a - - Fb
N
2Tr((a"(;;) (b;)" -a((b)]]'(a((b))) -
(236)
1
Σ (b;) abb) - (bPPATT)))
N
N N
+ 2Tr((a*
Σ(b)) - (66) - -aabbFF)))
(237)
(I|a(b; - T- (1aaaaaa)) - a'a" a²(b")
(238)
N
1
b 1-
(La ala-aaa a'a" )a (b*)
(239)
N
N
1
Σ
bb 6
[[ a
Ja ||
(240)
N
1=1
72 6 11
(241)
where (238) holds since the last term is 0, 239 and (44 hold since ||uv T||
|u]|| |[vV]| for
vector u and v, (241) holds because of Definition A3.13
Proof of Theorem A.14
Proof. In order to achieve e-recovery of a", we need
80 c(1 (6")²))² Se
(242)
(1 - c(1 11-c(1(669)) (6))()()))) < f
(243)
Tlog (1 - c(1 - (8))3)) s log( e go
(244)
(245)
We proceed such that
log(
T 2 log (1 c(1 - (80)2))
(246)
log(,5)
(247)
c(1 (80)2)
c(1 (80)2) log 8
(248)
where (44) follows by using log(1 - x) < - for |z| < 1.
34

---

Now we show the convergence to the global minimizer. Recall that bT b'a" a² and 8
[[ ) |, we have
|a _ abTTt a'b'
la" ( a²b a'b"
(249)
(aa" )
Ia)aab'
(250)
[( a² a'(a") )a" [b*]]
(251)
<||
||b*|
(252)
cl bbb
(253)
where (53) is due the fact that | 1| = |x|||y| and ||a"|| 1.
Proposition A3.17 shows that for any 5n E (0, the global objective of FFA-LoRA is given by
(211), comprising two terms: 72 reflecting the heterogeneity of [b;},- and ||b* 112 due the angular distance between a and at. By Theorem A3. 14, RoLoRA achieves e-accurate recovery of
the global minimizer, with global loss upper bounded by 72 ||b* since RoLoRA reduces angular distance loss from |bb & to ||b 1/22, iterations.
We can make arbitrarily small by increasing the
35

---

A4 Convergence Analysis of Non-Convex Case
We follow the approach Li al. [25] demonstrate the convergence of RoLoRA (Algorithm in smooth, non-convex landscapes. Assumptions A4.1 and A4.2
2 are standard and commonly employed in the convergence analysis of federated learning. Assumption A4.3 is adapted from LedRA-LoRA [T]] which proposes a personalized federated fine-tuning framework that maintains local diversity in B while aggregating only A through simultaneous updates of both A and B. In contrast, our work focuses on single global model with alternating optimization of A and B.
In Assumption A4.3 bounding Frobenius norms of A and B is standard weight-regularity require- ment in LoRA fine-tuning practice, where small rank and scaling factors keep the adapters from exploding. The inner-product conditions simply posit that each adapter has enough non-degenerate singular values, requiring that the low-rank updates retain sufficient rank and alignment with local gradient directions. Formally, we require the smallest singular values of A and B be lower bounded by VCA and /CB.
Assumption A4.1 (Bounded Stochastic Gradient). Let a mini-batch x,, be drawn uniformly at random from client i's dataset, meaning Ex, Ww, (WW, x,)]= = Vw, (W.). We assume that the expected squared norm of any stochastic gradient is uniformly bounded, that is,
Ex. Vw & (W, x < G²
where the expectation is over the random draw of the mini-batch x,, and G > 0 is a constant.
Assumption A4.2 (Lipschitz smooth). Loss functions 4,, ... ,IN are all L-smooth, For all weights W and U:
4( S 4(W) + (Vw4(W),
w)F + - - w 3 Vi [N]
Assumption A4.3. Let W; = Wo + B,A, represent the model parameters for the i-th client. There exist constants CB 0, CA 0, cg 0. and сд > 0 such that:
| S CB.
| S CA.
(A,A, Vwl(W.)VW((W)) (T),))))>))))))) ) 2 V
(B, T,, Vwl(W,)) Vw4(W.)) 2 callV
for all i E [N].
Theorem A4.4 (Convergence to the stationary point). Let Assumption hold. Suppose each client runs 2T rounds, each consisting of Q local epochs, using learning rate n O(1/ (T), then obtain:
A4.
A4.2 and A4.3
A.
Dn
min BII V | 05152T
(254)
2T TTnmmin 20min
where A, = (l( H. Cmin = min(CA, CB), D is chosen such that Du2 2 DA + DB.
and DA = L o² G² G2 + +L G² + C Dr = 2nc2 + nn
2T Cf + 1 C
According to Theorem A4.4 we achieve an O
VT convergence rate toward a stationary point
under smooth, non-convex conditions, matching the convergence rate FedAVG the same setting. We follow similar derivation framework to [15) adopting similar proof techniques, such as applying Assumption| A4.2 to the global and local models (Eq. 262 and (76)). However, due to structural differences between algorithm and that of [15 in particular, use of alternating optimization rather than simultancous updates- the derivation diverges in how key gradient bounds are applied and combined. As result, even though the proof steps analogous, our final convergence bound depends on min (CA, CB) and features decoupled A- and B-related terms, in contrast to the CA + св dependence along with cross- cross-terms due to the simultaneous updates of A and B in [15). Although
convergence rate, the two algorithms address fundamentally different FL. scenarios: FedSA-LoRA optimizes personalized client models, whereas RoLA learns single shared global model.
FedSA-LoRA reports the same O
VT
36

---

Proof. Let W Wo + A'B' be the global model parameters at the 2t-th communication round. i.g be the local model parameters of client at the q-th local epoch of the 21-th communication round, where each client performs a total of Q local epochs. We define the following for convenience:
Let W Wn A'B,, i.
The 2t-th communication round:
2 Wo + A A'B'
(255)
The g-th local epoch of the 2/-th communication round:
W f. = Wo + A kg
(256)
The 2t + 1-th communication round:
1 N
w 2t+1
Wo + A'B' = W
+
Σ
A'B
(257)
N
1=1
The g-th local epoch of the 2t + 1-th communication round:
w Wo + A B
(258)
The 2t + 2-th communication round:
w Wo + l B W.
Σ
A o B
(259)
N
According to chain rule,
(B)(WW) = A (W)
(260)
VAl(W) (W)
(261)
Now we apply Assumption A4.2 to local update of B, and get
4 4(W < 4(W")) + (W?1
2 Vw2( 1)++ 2
|
w2
(262)
By 255 and 256
W W1
2 A -B))
=
(263)
=- Vwl(W (x,1
(264)
where 17 the learning rate. Then
4(W?) S 4(W2)) 1/ (A'A Vw4(W2) x(1 (w)(W)))
"''A''' " VwL(( 2
(265)
Taking expectation of 2655)
< (L(WW))) - ( T VwL(w"), Vwl(W + L 1 A'A' T
[w?,
(266)
The inner product term is lower bounded such that
(A'A* Vwl( V Tr(2 T A'A Vwl((22))
(267)
= =Tr[A'A" (W)
(268)
= =A'AAT, (W(( )
(269)
2 (W
(270)
37

---

where (70))
follows by Assumption A4.3 Moreover,
L
L
2 BIIIA'A VW
"/ 11 /1 2W2xx 1
(271)
2
L
(272)
2
L "
(V",
(273)
2
T
L
$ 2
(274)
where (271) follows by Assumption A4.3
274
follows by Assumption A4.1
Combining (266).
(70 and (244) we get
B (1 < E|4 W
)] - nCa ncal V + C2
(275)
Next apply Assumption A4.3 to aggregation step and take expectation both sides,
L E [
B[4 s Bl4( E [ W Vwl(
w 11:
2
(276)
where
N
W
W 4,
A'B SQQ - A A
(277)
N
j=1
1 N
Σ
A
A
SQQ B
(278)
We have
Q-1
Bi = B' 11 Σ
Bl, j.g x344)
(279)
9=0
B =B - V/B (W x(])
(280)
Q-1
Bi - B =
(BB4 2 xx+1) - Vnlji (W?6 3:g x3,))
(281)
7=0
Q-1
= n/
V Vwl((W2 x21 Vw (W3g x7x))
(282)
9=0
Thus,
N Q-1
2
W
A'
Vw4(W2 Vw (W J.g x
(283)
N
Therefore.
N Q
2
2
11
V (w4 xx?1)
V
Jig x22))
(284)
11
N
0=0
1
Σ
(wl((W2, x,,] Vwl,(W2
22
xxz))
(285)
22
3.q
Q-1
02
Σ
(wl(WW2 x
Vwl,
xx))) f.0
(286)
N
J=1
N Q-1 Σ
2 <
C
Vwl x(1)
Vwl,
-
21 12
(287)
N
j=1 0=0
38

---

Taking expectation, w W 2+ - -
L N
N Q-1
E
C
Σ ΣΣ E q=0
x
Vwly
j.g x
||
(288)
For any matrices A and B (or vectors), we have
-
A + B B) s | + |B|r s 2|| B
Thus.
| V x Vwlj 3.97 x
- 2E Wwl((W" xx,11)11 + 2E
(289)
4G
(290)
Leading to
21 w S 22
(291)
For the inner product term of (276). we have
E [w22+1
W
Vw4(W?) [F S 2n
E
W W
2
2 1
||
V
(W?.)
(292)
S L G
(293)
Combining (27)). 291
and (233.. we obtain
E[4 W
2
L nn +
BI4 -
(294)
Combining 275 and 294). we derive
Tcc V (w? }1
< E(4(W ET4 21 ))
1 + = +
2L + 1 (295)
Analogously, applying the same analysis to the (2t + 1)-th communication round, which fixes B and updates A. introduces key modifications steps such as (2. and Eq, (27)) which govern the weight updates. These changes propagate through the subsequent steps that depend on the updated weights. As result, we obtain
qen Vw4((2
[14 (W2) B[(((2++)))+ + C2 + 1 + + +2, + P (296)
Add the two inequalities and then sum over 0, T 1, get
Σ
cpl (w - EW + +T(D++ DB)
(297)
where
DA = = + 2 n16 + +2 + I,
(298)
DB = L2 + - + 22 Q² 2 L
(299)
Assume the per-elient loss is bounded below by 13 let
A E[[[(W)] - 1; Cmin min(cA.Cp).
(300)
39

---

Choosing D such that Drj² 2 DA + DB, then
A
Dn
E [w") 05
min
(301)
2T 2Tqmmmn
20min
We choose & O(1/VT) so that the overall convergence rate with diminishing step size is O(1/VT) which matches the canonical convergence speed of stochastic gradient methods in non-convex settings.
40

---

We show the effect of number of clients when using rank-2 LoRA modules in A5 Experiments
A5.1 Impact of Non-Linearity on RoLoRA
Across both linear and non-linear settings, all methods perform similarly, with RoLoRA showing mod- est improvement in the non-linear case, likely due to its better utilization of the added expressiveness from ReLU.
Non-linear
Linear NN
Accuracy (%)
Accuracy (%)
40-
80
LoAk-lnnar
RoleRA
RoleBh-äree
20
90
100
40
00
100
Comm. rounds
Comm. rounds
Figure 7: Comparison of RoLoRA, LoRA, and FFA-LoRA linear and non-linear networks. While overall performance similar, RoLoRA shows modest gains in the non-linear setting, likely benefiting from ReLU's added expressiveness.
A5.2 Hyper-parameters for GLUE task
SST-2
QNLI
MNLI
QQP
RTE
Total comm. rounds
500
500
500
500
200
Batch Size
64
32
32
32
32
Local Epochs
20
20
20
20
20
Table Hyper- parameters configurations. Note the total communication rounds are for the setting with 3 clients. When increasing the number of clients, we decrease the total communication rounds accordingly to maintain a constant sample count used during fine-tuning
Hyper-parameters
We show the hyper-parameter configurations for each dataset in Table 6
A5.3 Effect Number of Clients
Configurations Table 7 shows the selected layer attached with LoRA modules for Table present Table I with the results of FlexLoRA [2] added in Table
We
Layer Attributes
SST-2
QNLI W., W (15...., 23)
MNLI W.. W (15,..., 23)
QQP W., W (15....,23) 233
RTE
Type Index
W., W (18,.... 23}
W., W, {16.... 23)
P2
Table 7: The selected layer attached with LoRA modules for Table 1 and Table 2
Rank-2 Results Table g
Rank-32 Results In Table we provide additional experiments with rank-32 LoRA adapters the 20-client and 50-client setting.
41

---

Client Methods
SST-2
QNLI
MNLI
QQP
RTE
LoRA
95.64
92.04
85.85
86.16
82.19
FFA-LoRA
94.91
90.11
84.06
85.48
80.86
RoloRA
95.60
91.62
85.66
86.16
82.19
LoRA
94.27
86.91
81.22
82.07
46.21
20
FFA-LoRA
93.92
89.58
80.51
82,62
57.76
RoloRA
94.84
90.77
85.13
85,10
81.23
LoRA
93.23
82.57
58.96
76.96
49.10
50
FFA-LoRA
92.32
85.15
62.79
77.78
53.07
RoloRA
94.61
89.83
85.15
85.55
72.92
Table 8: Results with RoBERTa-Large models with varying client numbers (3, 20, 50) using rank-2 LoRA modules in federated setting, maintaining a constant sample count during fine-tuning.
Client
Methods
MNLI
QQP
LoRA
79.72
83,66
20
FFA-LoRA
80 10.47
84.08 =0.31
RoLoRA
85.91 +0:3
86.37 10000
LoRA
70.84 +-4.33
79.75 +:0.3
50
FFA-LoRA
74.47 +1.57
80.65
+00.1
RoLoRA
85.46 20.08
86.15
10.26
Table Results with RoBERTa-Large models in 20-client and 50-client setting using rank-32 LoRA adapters.
10-Client Setting In Table 10 provide results for 10-client setting with rank-4 adapter. The results show that RoLoRA still outperform other methods. In the 10-client setting, RoLoRA's performance gain other methods falls between the gains observed in the 3-client and 20-client settings.
MNLI
QQP
QNLI
LoRA
81 $1.48 #2.19
84.1 10.14
87.73 10.7
FFA-LoRA
83.19 #064
84.35
2006
89.88
RoLoRA
84.95 1:0.8
95.25
90.3 1:076
Table 10: Results with RoBERTa-Large model with 10 clients using rank LoRA adapters, running for 150 rounds in total.
FLoRA RoLoRA Table shows comparison between FLoRA and RoLoRA. In the 3-client setting, we ran 500 rounds and scaled rounds down proportionally with more clients to keep the total sample budget fixed. RoLoRA consistently outperforms FLoRA across tasks and client counts. While FLoRA eventually converges (e.g., 83.3% on MNLI after 4000 rounds), it does so much more slowly, highlighting RoLoRA's faster convergence and better scalability.
Finetuning Dynamics within 100 Rounds Figure 8 presents the 100-round extension of Figure 3, where RoLoRA consistently converges faster and achieves the highest accuracy.
We want to clarify that Figure 3 focuses on comparing convergence under a fixed sample budget rather than full convergence, and Table [ shows that this budget suffices for all methods when using clients. However, as shown in Figure its efficiency in low-resource settings.
3 with 50 clients, only RoLoRA fully converges, underscoring
A5.4 Effect of Number of LoRA Parameters
In Table [2] we include the details about layers attached with LoRA adapters for different budget of finetuning parameters, for each dataset.
42

---

Client
Method
MNLI
QQP
QNLI
3
FLORA RoLoRA
39.29
51.05
59.88
85.70
86.14
91.64
20
FLoRA
32.01
51.58
49.89
RoLoRA
85.28
85.83
90.35
50
FLoRA
31.97
50.54
38.82
RoLoRA
82.98
85.71
90.00
Table 11: Results with RoBERTa-Large models with rank-4 LoRA adapter for varying numbers of clients (3. 20, 50). comparing FLoRA with RoLoRA, maintaining a constant sample count during finetuning. In the 3-client setting, while FLoRA eventually converges (e.g.. 83.3% on MNLI after 4000 rounds), the figure shows results for only 500 rounds, within which FLoRA has not yet converged. This highlights RoLoRA's faster convergence and better scalability.
MNL
ONU
ODP
50
1oRA
75
RotoRA
LoRA
50
FloxLoRA
20
40
60
80
100
20
-10
60
D0
100
20
40
600
80
100
round
round
Figure Accuracies over 100 rounds. involves 50 clients using rank
Layer Attributes
SST-2
QNLI
MNLI
QQP
RTE
Type
W.
We, W
W., W (21...., 233
W.,, W
We, W
P
Index
(21,..., 23}
{21,.... 23]
{21,..., 23}
{21, ........ 23}
Type
W., W
We, W,
Wo, W
Wa, W
We, W
P2
Index
{18... 23}
{15.... 23}
(15,.... 23)
{15,.... 23}
{16, 23}
Type
W., W
W.. W
W. W
W., W
We. W
P3
Index
(O.....2 23}
{12,..., 23)
(12...., 23)
{12,... 233
{12, 23)
Table 12: The selected layer set attached with LoRA for the setup of Figure
:3T2
F764
34
parmm
Number
Figure 9: Results with RoBERTa-Large models on GLUE of different budget of finetuning parameters. involves clients using rank 8.
AS.5 Effeet of Rank on RoLoRA and FFA-LoRA in the Centralized Setting
We evaluated FFA-LoRA and RoLoRA on MNLI and QQP using 8 LoRA adapters attached to query and value projection of last layers, and trained for 50000 iterations to ensure full convergence. Increasing ranks can narrow the performance gap between the two schemes. Another related technique narrow the performance gap between the two schemes is by increasing the number adapters, as discussed in Section 5.I ("Effect of Number of Fine-Tuning Parameters"), With sufficient adapters,
43

---

VXT2
3011
114-44A
Marber paaarr
Mtrbbrr LoSh parmas
Mamber Lo params
Namberd of paorm
Namber .88 pe10ts
Figure 10: Results with RoBERTa-Large models on GLUE of different budget of finetuning parame- ters. It involves 3 clients using rank 2.
S672
26U
007
FIC
23
73
LobA
112
123r Marberr
35
40.2
12.35
Marber
353
12.2
3206
40.0
Figure 11: Results with RoBERTa-Larg models on GLUE of different budget of finetuning parame- ters. It involves 3 clients using rank 1.
FFA-LoRA can achieve comparable peak accuracy to RoLoRA. However, in federated settings where resources are constraiend, RoLoRA is more advantageous.
rank-1
rank-2
rank-32
rank-64 64
FFA-LoRA
80.66
81.51
83.3
83.32
MNLI
RoLoRA
83.93
84.59
85.78
85.79
Diff
3.27
+3.08
+2.51
+2.47
FFA-LoRA
69.61
74.01
75.53
75.51
QQP
RoLoRA
77.26
77.41
78.03
78.05
Diff
+7.65
+3.40
+2.5
+2.54
Table 13: Evaluation on FFA-LoRA and RoLoRA in the centralized setting using only LoRA
adapters. Increasing ranks narrow the performance gap between the two schemes.
A5.6 Align the Communication Cost
we conduct comparison of three methods under the constraint of identical communi- cation costs under the assumption that the number of clients is small. To align the communication costs across these methods, two approaches are considered. The first approach involves doubling the rank of FFA-LoRA and RoLoRA, with results presented in Appendix A5.4 The second approach requires doubling the number of layers equipped with LoRA modules. In Figure T2] the latter strategy is employed. Specifically, for both FFA-LoRA and RoLoRA. we adjust the communication costs by doubling the number of layers equipped with LoRA modules, thereby standardizing the size of the transmitted messages. The configurations are presented in Table 14] Figure 12 demonstrates that when operating within a constrained communication cost budget, the performance RoLoRA surpasses that of the other two methods for most of the tasks.
In Figure 12
In Table [4] we include the details about layers attached with LoRA adapters.
A5.7 Commonsense Reasoning Tasks
present the configurations for Table in Table T5 We show the results under the same setup but using rank-2 LoRA modules in Table 16
44

---

MINLI
QNII
10
Potelk
Accuracy
2015-10 29100
Figure 12: RoBERTa-Larg accuracies QQP, MNLI, QNLI, and RTE with specifie uplink commu- nication budget. It involves 3 clients using rank 4. Error bars reflect standard deviations.
Communication Cost
LoRA
FFA-LoRA
RoLoRA
Type
W., Wg
Wo, W
We, W {18, (18...., 23)
187.5 KB
Index
(21,...,22 23}
{18, 23}
Type
W., W {20, 23}
Wo, Wg
W. W (16...., 23)
250 KB
Index
{16, 23
Table 14: The selected layer set attached with LoRA modules for the setup of Figure 12
A5.8 Language Generation Tasks
Model, Datasets and Metrics. We evaluate the performance of three federated finetuning methods with the model Llama-2-77 2-BB [351, on two datasets: CodeAlpaca [ for coding tasks, and Alpaca mnstructionfoolloiing B3] for general instruction following tasks. Using HumanEval [ the metric for CodeAlpaca,
we assess the model's ability to generate accurate code solutions. For Alpaca, we employ MMLU (Massive Multitask Language Understanding) (18 to evaluate the model's performance across diverse domains. This provides an assessment of Llama-2-7B's coding proficiency. and general language capabilities when finetuning in the federated setting. We show the setup in Table 17
Results Table 18 presents the evaluation results the Llama-2-7 2-7B model using three methods, across tasks: HumanEval, and MMLU. The metrics reported include the average and standard deviation of performance over five seeds, with 50 clients involved. The results show that RoLoR achieves the highest scores across most metrics, demonstrating slightly improved performance compared to LoRA and FFA-LoRA. The improvements are more evident in certain subcategories of the MMLU dataset.
A5..9 Communication and Time Cost Comparison
Table 19 compares the communication cost and time cost in a 50-client setting on MNLI task. RoLoRA and FFA-LORA have the lowest communication and time costs.
A5.10
Privacy-preserving FL
Beyond robustness, practical FL often requires privacy-preserving mechanisms, most notably cryp- tographic approaches such as secure aggregation 3 homomorphic encryption [99 8 7 and statistical protections like differential privacy (DP) [1 41). While RoLoRA was not explicitly de- signed with privacy mechanisms such differential privacy, recognize that its ability to mitigate inexactness through alternating optimization and aggregation may help mitigate a key challenge for DDPaware federated learning, where inexact model updates be particularly problematic. In Table 20, we itegrated NbAFL 41 with e = 10 and = le 3-client setting. RoLoRA outperform others across tasks.
45

---

Total munds
Batch size
Local Fpochs
type attached with LoRA
Layer index attached with LoRA
Wk. Wo, We
(26, .... 31)
Table 15: Configurations for Commonsense Reasoning tasks.
BoolQ)
SIQA
HelloSwag
WinoCrrande
ARC-e
ARC-e
LoRA
14.36
42.87-+1405
19.12
26.21
47s2-tas
10.31
9.84
12.33
FFA-LORA RoLoRA
44.04
51.46_981
25,38
23,86.387
46.931151
22.25
20.65 2615
20.67
61.3_099
60.81 2335
37.97
29.02
49,59
37.05
29.09
28.93
Table 16: Results with Llama-2-78 models commonsense reasoning tasks. It involves 50 clients using rank 2.
comm.
Egochs
Tayer type attached
Layer
100
30
Wa. W., Wy. W.
{24, .2......31.
Table 17: Configurations for language generation tasks.
LoRA
FFA-LoRA
RoLoRA
HvmanEval
12.96
13.29
13.45
MML.U
45.81 10.00
45.80.1002
45.93 10000
human
43.264001
43.24.400t
43.46..0.0
other
52.67
52.72.0.0
52.83
social
51.734004
51.64
51.81 +:04
stem
37.10
37.12
37.05
Table 18: Results with Llama-2-7B model on HumanEval, and MMLU. We report the average and std. over five seeds. The number of clients The metric used across all tasks is accuracy, where higher values indicate better performance.
Method
Comm. cost
Time cost
LoRA
1500.8 KB
0.0415 sec
FFA-LoRA
750.4 KB
0.0163
FlexLoRA
1500.8 KB
2.252 see
FLoRA
193603.2
2.2179 see
RoLoRA
750.4 KB
0.0182 sec
Table 19: Communication and Time Cost Comparison
Comm. cost: Total message size sent and received by each client per communication round in the MNLI experiment shown in Table [
Time cost: Mean server aggregation time per communication round for the MNLI experiment with 50 clients in Table averaged over runs.
MNLI
QQP
LoRA
72.79 43.23
57.97 48.23
FFA-LoRA
79.65 4056
78.16 +06
RoLoRA
81.08 #081
81.64 +0.35
Table 20: Evaluation on differential privacy using
10 and
1c 6 in 3-client setting.
46

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Section 3 Section and and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A2
47

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please see Appendix A3 and Appendix A4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See Appendix
A5
Guidelines:
The answer NA means that the paper does not include experiments.
48

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The datasets are all open-source. The code is uploaded.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeuuiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
49

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See Section 5 and Appendix AS
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See standard deviation reported in Section 5 and Appendix AS
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
50

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwuuppp.ccccppbbb//bbbba
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses the broader societal implications of deploying federated learning methods enhanced with LoRA adapters. On the positive side, we emphasize that approach significantly improve the efficiency and scalability of federated learning, thereby facilitating model training on decentralized data. We did not identify any direct negative societal impacts, involve deployment or data collection.
contribution is primarily methodological and does not
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
51

---

Answer: [NA]
Justification: This paper proposes a framework for federated finetuning, and does not pose high risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: Yes]
Justification: This paper is licensed under CC-BY-NC-SA 4.0. All other codes, datasets, and references are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwpppsaittcoooeo thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We did have any released new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
52

---

14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We did involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
53


Reviews:
[{"review_id": "eSYFqoeL0v", "reviewer": "Reviewer_WEwx", "summary": "This paper proposes RoLoRA, a parameter-efficient fine-tuning method for federated fine-tuning of LLMs. RoLoRA learns the two LoRA low rank terms in an alternating minimizing style, achieving SoTA fine-tuned accuracy without increasing the communication cost.", "questions": "1. According to line 288, comparison against baselines are run under the same number of communication rounds. Does this mean roughly the same amount of communication cost like the amount of data transferred?\n2. In Table 1, the rank is 4 and 8, which seems smaller than the common values of 32 used in PEFT papers. Could the author explain the reason behind it, or show the results with higher rank?\n3. In Figure 3, could the author explain why the error band of MNLI accuracy is much wider than the other two baselines?", "limitations": "Limited by my knowledge background, I did not observe weakness/limitations of this paper.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "As an emergency reviewer, I was assigned with this paper without bidding. Sorry for the fact that I have limited knowledge in federated learning and hope the following comments & questions are reasonable.\n\n## Strength \n1. Clear method description: the method and derivations in this paper are well organized\n2.  Sufficient experiments: the paper covers both ideal toy experiments and practical problems, with additional experiments in the appendix.\n3. The improvement of the baselines are clear: the fine-tuning process of RoLoRA is more stable, and achieves much higher model performance when the number of clients is large. \n\n# Weakness\nLimited by my knowledge background, I did not observe weakness/limitations of this paper.", "was_revised": true, "final_justification": "All of my concerns have been addressed. I would like to keep my current score due to my limited knowledge in this area.", "rebuttal": "We sincerely thank the reviewer for their time and effort. We are grateful for the insightful questions, which we address below.\n\n**Q1: Clarification on Communication Rounds vs. Communication Cost**\n\nThank you for this question. It allows us to clarify a key contribution of our work: RoLoRA's ability to achieve superior performance at a significantly reduced communication cost. Our emphasis on using the same number of communication rounds is intended to ensure a **fair comparison within a typical federated learning setting**. First, we note that in our paper, unless otherwise specified (as in Figure 11 in the appendix), the communication cost per round is by default different across the methods being compared, since RoLoRA and FFA-LoRA only need to transmit a single LoRA component in each round due to their freezing nature. In contrast, standard LoRA and FlexLoRA must transmit both LoRA components (A and B). This means that for a given rank, RoLoRA has **half the communication cost per round** compared to LoRA and FlexLoRA. Second, to make the comparison even more direct, in **Figure 11 (Appendix)**, we compare all methods under an **identical total communication budget** (i.e., the same total megabytes transferred). To achieve this, we configured RoLoRA and FFA-LoRA with twice the number of LoRA adapters so their total communication cost matches that of FedAvg with LoRA. The results show that when the communication cost is equalized, RoLoRA can converge to a better accuracy.\n\n**Q2: Rank-32 results**\n\nWe chose rank-4 and rank-8 in Table 1 to reflect practical scenarios in federated settings, where communication and memory constraints on edge devices often necessitate smaller ranks.\n\nTo address the reviewer's concern, we have conducted additional experiments with rank-32 adapter in the 20-client and 50-client setting. The results, averaged over three runs, are aligned with our previous observations: RoLoRA consistently outperforms others in the setting with large number of clients at higher ranks.\n\nWe will include the rank-32 results in the revision.\n\n\n\n|Client num| Acc(std)   | MNLI        | QQP        |\n|----|----------|-------------|-------------|\n|    | LoRA     | 79.72(0.38) | 83.66(0.02) |\n| 20 | FFA-LoRA | 80(0.47) | 84.08(0.31) |\n|    | RoLoRA   | 85.91(0.63) | 86.37(0.09) |\n|    | LoRA     | 70.84(4.63) | 79.75(0.31)  |\n| 50 | FFA-LoRA | 74.47(1.57) | 80.65(0.31) |\n|    | RoLoRA   | 85.46(0.08) | 86.15(0.26) |\n\n\n\n**Q3: Error Band in Fig.3**\n\nThe observed variance arises from a combination of training dynamics and task difficulty. As shown in Fig.3 (MNLI), none of the methods have fully converged—RoLoRA is nearing convergence, while LoRA and FFA-LoRA are still early in training. High variance across runs is expected in the intermediate  phase due to different optimization paths from random initialization. This effect is also evident in SST-2 and QNLI.\n\nAdditionally, variance magnitude depends on task difficulty. MNLI's complexity leads to more diverse training trajectories and higher variance, whereas simpler tasks like SST-2, QNLI, or BoolQ show more stable results. We consistently observe greater variance on harder tasks like PIQA and SIQA."}, {"review_id": "fexe61Kswg", "reviewer": "Reviewer_jkws", "summary": "This paper proposes RoLoRA, an alternating optimization method for LoRA in LLMs, and provides a thorough theoretical analysis to demonstrate the importance of training the down-projection matrix in federated LoRA updates.", "questions": "- It seems that you used 3, 20, and 50 clients in your experiments. Could you clarify the rationale behind choosing these specific values, especially the large jump from 3 to 20?\n\n- (Discussion) In my opinion, a core issue in FFA-LoRA is that random initialization does not guarantee the subspace spanned by matrix $A$ includes the optimal subspace spanned by $A^*$. I believe that alternating optimization of $A$ and $B$ could mitigate this issue. However, it's worth noting that in standard LoRA implementations, $A$ is typically initialized with Gaussian random values while $B$ is initialized to zero. This suggests that $A$ serves as a basis while $B$ acts as a coefficient matrix. Therefore, treating $A$ and $B$ symmetrically may not be optimal. Is there a principled approach to gradually adapt matrix $A$ toward the optimal basis while ensuring matrix $B$ is updated effectively?\n\n- Is there any evaluation on how rank affects the performance gap between RoLoRA and FFA-LoRA in a centralized setting? Since angle distance may decrease as the rank increases, it would be valuable to understand whether higher ranks reduce the effectiveness gap between the two methods.", "limitations": "Yes, the paper have addressed that the theoretical analysis centers around linear models.\n- Moreover, the theoretical analysis fail to capture the **autoregressive** architecture of modern decoder-only LLMs. **As such, the theoretical analysis may not be sufficient to support the claimed benefits in LLM training**. If the authors are unable to extend the theory to autoregressive or non-linear models more representative of LLMs, I recommend placing greater emphasis on empirical evaluations or supporting the claim of effectiveness on general neural architectures rather than LLMs through theoretical analysis.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The theoretical analysis is well-grounded. The use of both linear and non-linear models to demonstrate how the initialization of matrix $A$ significantly affects the performance of FFA-LoRA via angle distance is insightful.\n\nWeaknesses:\n- The proposed method is not inherently specific to LLMs and could be applied to any LoRA-based update, reducing its uniqueness in the LLM context.\n\n- The theoretical analysis focuses on linear models, which do not capture autoregressive characteristic of modern LLMs.", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns. The experiments on alternative strategies are insightful. I hope there will be further deep investigation into these insights in the camera-ready version. I'll raise my score.", "rebuttal": "We sincerely thank the reviewer for their thorough, thoughtful, and insightful feedback. We'll address each point below.\n\n**Q1: Client Number Selection**\n\nOur choice of 3, 20, and 50 clients was intended to comprehensively showcase RoLoRA's performance across different scales of FL scenarios, and specifically to highlight RoLoRA's advantage in achieving both faster convergence (Fig.3) and higher final accuracy (Table 1) in large-scale settings. To provide a smoother transition and further strengthen our conclusions, we have now added experimental results for 10-client setting with rank-4 adapter. The results show that RoLoRA still outperform other methods. In the 10-client setting, RoLoRA's performance gain over other methods falls between the gains observed in the 3-client and 20-client settings.\n\n|  Acc(std)    | MNLI      | QQP       | QNLI      |\n|----------|-----------|-----------|-----------|\n| LoRA     |81.48(2.19)|84.11(0.14)|87.73(0.67)|\n| FFA-LoRA |83.19(0.64)|84.35(0.06)|89.88(0.13)|\n| RoLoRA   |84.95(0.8) |85.25(0.39)|90.3(0.76) |\n\n\n**Q2: Asymmetry in LoRA and Basis Adaptation Strategy**\n\nWe thank the reviewer for their deep and insightful question regarding the symmetric treatment of matrices A and B in RoLoRA. To empirically investigate whether an asymmetric update schedule might be preferred, we have conducted new experiments comparing our standard approach to two alternative strategies in a 50-client setting using rank-4 adapter on the QQP. We consider strategy 1 that prioritizes updating B, updating 3 times for every one-round update of A (e.g., B, B, B, A, ...), and strategy 2 that prioritizes updating the basis matrix A(e.g., B, A, A, A, ...). While A and B serve inherently asymmetric roles, we observe that aggressively updating either matrix without sufficiently adapting  the other leads to misalignment. Furthermore, we considered even more fine-grained approaches to handle this asymmetry, such as assigning different learning rates for A and B. We adapt LoRA+'s strategy [4] into federated setting by using a larger learning rate for B. Strategy 3 and 4 corresond to $lr_B=2lr_A$ and $lr_B=4lr_A$, respectively. Our experimental results showed that a symmetric approach with equal learning rates still yielded the best performance.\n\nIt appears that a balanced, symmetric alternation between the matrices A and B yields the most stable and effective optimization trajectory. RoLoRA's design provides this balance and provides a robust strategy to update A and B matrices. We will add a remark clarifying this in the paper.\n\n\n| Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | ... | 30 |\n|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|--------|\n| RoLoRA | B   | A   | B   | A   | B   | A   | B   | A   | ...    | A    |\n| Test Acc. | 42.48   | 63.18   | 77.77   | 79.01   | 80.75   | 82.10   | 82.52   | 83.4   | ...    | 86.30    |\n| Strategy 1 | B   | B   | B   | A   | B   | B   | B   | A   | ...   | B    |\n| Test Acc. | 42.48   | 63.18   | 77.73   | 78.82   | 78.77   | 79.42   | 81.54   | 81.86   | ...    |  85.15   |\n| Strategy 2 | B   | A   | A   | A   | B   | A   | A   | A   | ...    | A    |\n| Test Acc. | 42.48   | 63.18   | 63.18   | 63.18   | 68.96   | 74.80   | 74.88   | 75.01   | ...    | 84.77    |\n| Strategy 3 | B   | A   | B   | A   | B   | A   | B   | A   | ...    | A    |\n| Test Acc. | 47.54   | 63.18   | 65.86   | 74.92   | 79.56   | 80.56   | 81.26   | 81.99  | ...    | 85.7    |\n| Strategy 4 | B   | A   | B   | A   | B   | A   | B   | A   | ...    | A    |\n| Test Acc. | 63.18   | 63.18   | 65.23   | 71.41   | 74.66   | 76   | 77.05   | 79.38  | ...    | 84.36    |\n\n\n**Q3: Impact of Rank on RoLoRA vs. FFA-LoRA**\n\nTo investigate the reviewer's hypothesis, we've conducted a new set of experiments in a centralized setting as requested. We evaluated both methods on the MNLI and QQP using 8 LoRA adapters attached to query and value projection of last 4 layers, and trained for 50000 iterations to ensure full convergence. We note that when the rank increases from $1$ to $64$ the gap between the two schemes decreases as suggested by the reviewer. Another related technique to narrow the performance gap between the two schemes is by increasing the number of adapters, as discussed in Section 5.1 (“Effect of Number of Fine-Tuning Parameters”). With sufficient adapters, FFA-LoRA can achieve comparable peak accuracy to RoLoRA. However, in federated settings where resources are constraiend, RoLoRA is more advantageous.\n\n|        |            | rank-1 | rank-2 | rank-32 | rank-64 |\n|--------|------------|--------|--------|---------|---------|\n|        | FFA-LoRA   | 80.66  | 81.51  |  83.3  | 83.32   |\n| QQP    | RoLoRA     | 83.93  | 84.59  |  85.78  | 85.79   |\n|        | Diff       | +3.27  | +3.08  |   +2.51     | +2.47   |\n|        | FFA-LoRA   | 69.61  | 74.01  |    75.53    | 75.51   |\n| MNLI   | RoLoRA     | 77.26  | 77.41  |   78.03     | 78.05   |\n|        | Diff       | +7.65  | +3.40  |   +2.5     | +2.54   |\n\n\n**Limitation: Theoretical Analysis for Autoregressive LLMs**\n\nOur theoretical analysis deliberately focuses on simplified models to provide foundational insights. The linear model provides a tractable case where different methods can be directly compared and their limitations can be formally established. We then emperically demonstrate that similar findings also apply in more complex neural networks and large langugage models.  This approach aligns with standard practice in federated learning, where simplified yet representative linear models are used to isolate and understand the core mechanics of new methods [1,2]. Within this setting, we formally prove the mechanisms behind RoLoRA's improved performance and also provide a convergence analysis in non-convex loss landscapes.\n\nIt is also important to note that this simplification is common across the LoRA literature. To the best of our knowledge, **no existing work on LoRA or its variants has provided a theoretical analysis that fully captures the autoregressive architecture of modern decoder-only LLMs**. Prior theoretical analysis are typically limited to linear regression settings [4,6] or rely on assumptions such as rank-one shift, rank-one LoRA, or frozen B[7]. Our model is thus aligned with the current state of research [3,4,5,6], while addressing a more complex federated setting.\n\nWe view our extensive empirical evaluations on LLMs as the primary validation of our claims. The strong performance gains demonstrated in these experiments directly align with the predictions of our theory, showing that the foundational insights successfully transfer from the theoretical model to complex, practical applications.\n\n[1]Collins, Liam, et al. \"Exploiting shared representations for personalized federated learning.\" ICML21\n\n[2]Collins, Liam, et al. \"Fedavg with fine tuning: Local updates lead to representation learning.\" NeurIPS22\n\n[3]Zhu, Jiacheng, et al. \"Asymmetry in Low-Rank Adapters of Foundation Models.\" ICML24\n\n[4]Hayou, Soufiane et al. \"LoRA+: Efficient Low Rank Adaptation of Large Models.\" ICML24\n\n[5]Zhang, Yuanhe et al. \"LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large\nLanguage Models, Provably and Efficiently.\" ICML25\n\n[6]Zhang, Fangzhao et al. \"Riemannian preconditioned LoRA for Fine-Tuning Foundation Models.\" ICML24\n\n[7]Dayi, Arif Kerem et al. \"Gradient dynamics for low-rank fine-tuning beyond kernels.\"\n\n**Revisions to Paper Formatting**\nWe thank the reviewer for their careful reading and helpful suggestions for improving the paper's formatting and clarity. We will address both points in the revision. For clarity in line 30, we will revise it to succinctly state the well-known issue of applying FedAvg to LoRA and directly transition to the methods that address it."}, {"review_id": "snIS6Ph1rT", "reviewer": "Reviewer_3VmG", "summary": "This paper describes a novel method for federated learning with LoRA that improves on prior art performance.  It belongs to a class of algorithms that address the shortcoming of directly applying standard federated optimization methods such as FedAvg on LoRA modules.  LoRA's efficiency in module storage opens up impactful opportunities in federated learning applications where communication efficiency is crucial.  However, brute force combining of the two methods often leads to interference and poor performance as theoretically and empirically shown in this paper.  Hence a prior body of research has been dedicated to resolving the performance gap.  This paper builds on prior arts of novel strategies based on treatment of the adaptation matrices A and B during the federated optimization process.  The authors observe that a lot of the performance gap can be recovered by taking turn between optimizing A and B in alternating communication rounds, versus methods from prior arts that freeze A or B, or that optimize A and B in every round.", "questions": "1. Could you comment on how the number of local steps impact the performance of RoLoRA and FFA-LoRA?  \n2. Could you comment on the intuition behind why FedAvg + LoRA performs relatively well in Table 2?  How does data heterogeneity impact performance of RoLoRA?", "limitations": "No except with one mention of data privacy guarantee.\nSuggestion is to stress significance of privacy protection for client data.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Quality and clarity: well written and comprehensive, well structured with detailed analysis and easy to follow\n2. Originality: novel method that wins in its simplicity and effectiveness\n3. Significance: especially applicable today where LLMs have found mainstream and user privacy is crucial \n\nWeaknesses:\n1. Societal impact evaluation may be missing", "was_revised": true, "final_justification": "Simple, effective method.  Written in clarity.  With strong applicability.", "rebuttal": "We are deeply grateful to the reviewer for their thoughtful and encouraging reviews. Below, we address your insightful questions.\n\n**Q1: Effect of local steps**\n\nTo provide a clear, empirical answer, we have conducted an ablation study on the number of local steps in a 50-client setting with rank-4 adapter. For a fair comparison, we kept the total computational budget (Local Steps * Total Rounds) constant across all settings. FFA-LoRA's performance consistently degrades on both datasets as the number of local steps increases. This indicates that with more local work per round, FFA-LoRA suffers severely from client drift, where local models overfit to their own data.  RoLoRA maintains high performance across all settings.\n\n| Dataset | (Local steps, total rounds) | (1,600) | (5,120) | (10,60) | (20,30)|\n|----------|----------|----------|----------|----------|----------|\n| MNLI   | FFA-LoRA    | 72.52(0.68)    | 71.73(1.27)    | 69.64(4.31)    | 69.97(5.57)    |\n|      | RoLoRA    |  84.39(0.34)   | 84.96(0.18)    | 84.79(0.23)    | 82.98(3.36)    |\n| QQP   | FFA-LoRA    | 80.51(1.38)    | 80.2(1.65)    | 79.07(1.21)    | 78.44(0.41)    |\n|      | RoLoRA    | 85.24(0.56)    | 85.44(0.8)    | 84.77(0.77)    | 85.71(0.18)    |\n\n\n\n**Q2: FedAvg+LoRA Performance and Data Heterogeneity**\nWe believe the relatively good performance of FedAvg+LoRA in Table 2 is partially due to the small number of clients (10). Our findings suggest that the performance of FedAvg+LoRA is more sensitive to the number of clients than to data heterogeneity alone. With fewer clients, the model aggregation becomes more accurate. As the number of clients increases, aggregation inaccuracy becomes more pronounced.\n\nWhile RoLoRA, like all federated methods, is inevitably affected by data heterogeneity, it demonstrates greater robustness to client drift induced by non-IID data distributions. Evidence for this can be found in our ablation study on the number of local epochs, where RoLoRA maintains competitive performance under increased local update steps, which is known to exacerbate drift. By contrast, FFA-LoRA exhibits a sharper performance degradation in non-IID settings, aligned with our observation in the same ablation study. Further comparisons between i.i.d. and non-i.i.d. scenarios in the 10-client setting can be drawn from Table 2 and the additional experiments provided in response to Reviewer jkws under “Client Number Selection.” These observations highlight RoLoRA's improved resilience to heterogeneity, particularly in scenarios with significant local deviation.\n\n\n\n\n\n\n**Limitation: privacy protection for user data**\n\nThank you for this important and constructive suggestion. While RoLoRA was not explicitly designed with privacy mechanisms such as differential privacy, we recognize that its ability to mitigate inexactness through alternating optimization and aggregation may help mitigate a key challenge for DP-aware federated learning, where inexact model updates can be particularly problematic. We itegrated NbAFL [1] with $\\epsilon=10,\\delta=1e-6$ in a 3-client setting. RoLoRA outperform others across two tasks.\n\n| Method     | MNLI (std)     | QQP (std)      |\n|------------|------------------|------------------|\n| LoRA       | 72.79(5.23)     | 57.97(8.23)     |\n| FFA-LoRA   | 79.65(0.56)     | 78.16(0.60)     |\n| RoLoRA     | 81.08(0.81) | 81.64(0.35) |\n\n[1]Wei, Kang, et al. \"Federated learning with differential privacy: Algorithms and performance analysis.\" TIFS2020"}, {"review_id": "zoh33hcL9I", "reviewer": "Reviewer_5j6K", "summary": "In this paper, the authors present the robust federated finetuning of LLMs using alternating optimization of LoRAA–RoLoRA. This is a novel federated learning framework for addressing the limitations of previous work for training LoRA adapters in distributed settings.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper are summarized as follows:\n\n1. This paper is well-motivated. It gives a very rigorous mathematical analysis and proofs for showing both linear and non-convex cases. It shows why using alternating optimization is necessary. Also, the algorithm is simple and effective. It maintains computational efficiency and exact model aggregation.\n\n2. Also, this paper gives a thorough experimental result on multiple models, tasks, and federated settings. \n\nThe weaknesses of this paper are summarized as follows:\n\nAlternating optimization has been extensively studied in many other areas, such as weighted low rank approximation and low rank matrix completion.  The key contribution of this paper seems to be applying this alternating optimization to study federated LoRA. However, this paper does not make any fundamental changes to this algorithm. It would be better if the authors could compare this paper with alternating optimization in other fields and highlight the novelty of this paper.\n\nI am not an expert in this field, so I hope the AC will put more weight on the other reviewers' comments.", "was_revised": true, "final_justification": "I have read the authors’ response, and to the best of my knowledge, it is convincing. All proofs appear to be correct. However, I have chosen to maintain my original score (Borderline Accept), which reflects a relatively neutral stance, as I am not an expert in this area.", "rebuttal": "Thank you for your thoughtful feedback and for recognizing the strengths of our work, including the theory, simplicity, and thorough experiments. We appreciate your positive assessment.\n\n**W1: Need for Clearer Distinction from Prior Alternating Optimization Methods**\n\nWhile conceptually inspired by classic methods, to the best of our knowledge, **no existing algorithm both alternates the optimization of components and alternates the aggregation of the corresponding components**. This alternating strategy addresses the inexact aggregation issue unique to federated LoRA, as discussed in Section 2.3. RoLoRA differs in both its specific algorithmic design and its application focus, as discussed in **Appendix A2**.\n\nFirst, we distinguish our work from the field of **matrix sensing and matrix completion**. Unlike traditional methods [1,7] that assume centralized data, RoLoRA is explicitly designed for the decentralized federated learning setting. More importantly, our key algorithmic difference lies in the aggregation strategy itself. Prior work in **federated matrix factorization** [2] performs alternating updates only locally within each client. In contrast, RoLoRA introduces an alternating aggregation scheme at the server level, where the up- and down-projection matrices (A and B) are aggregated across communication rounds. Our algorithm is critical for mitigating destructive interference between the LoRA components, a unique challenge in this federated LoRA context.\n\nSecond, while RoLoRA shares a similar derivation framework with **Multi-Task Representation Learning (MLRL)** methods such as FedRep [3] and Vaswani et al. [4], it differs fundamentally in objective. MLRL aims to learn personalized models by keeping final layers diverse and unaggregated. In contrast, RoLoRA is designed to train a single, high-performing global model. This difference is reflected in Algorithm 2 (line 7) and in the theoretical analysis (Eq. 35-40), where RoLoRA's global aggregation on $b$ is captured through a union bound over each $b_i$, followed by transforming the bound to their average. Moreover, the architectural roles of components differ: in FedRep, the local head acts as a down-projection, while in RoLoRA—built on the LoRA framework—the matrix $B$ acts as an up-projection, mapping the adaptation back to the original feature space.\n\nFinally, we differentiate RoLoRA from **personalized federated learning** algorithms that also use an alternating structure. These methods [5,6] are inherently designed to create separate parameters for each client, whereas RoLoRA operates entirely within the global model paradigm. This distinction leads to different motivations and theoretical approaches. Our alternating scheme is motivated by the structural decomposition of LoRA matrices, not the separation of personal and global parameters. Consequently, our proof technique is entirely novel in this context, drawing from matrix sensing theory to analyze the learning process, in stark contrast to the standard FL convergence analysis used in personalized FL literature.\n\n[1]Jain et al., \"Low-rank Matrix Completion using Alternating Minimization\", STOC13\n\n[2]Wang et al. \"Federated Matrix Factorization: Algorithm Design and Application to Data Clustering\", TIP22\n\n[3]Collins et al. \"Exploiting Shared Representations for Personalized Federated Learning.\" ICML21\n\n[4]Vaswani et al. \"Efficient federated low rank matrix recovery via alternating gd and minimization: A simple proof\", TIT24\n\n[5]Mishchenko et al. \"Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity\", TMLR25\n\n[6]Pillutla et al. \"Federated Learning with Partial Model Personalization\", ICML22\n\n[7]Ward et al. \"Convergence of Alternating Gradient Descent for Matrix Factorization\", NeurIPS23"}]

Author rebuttal:

Meta-review: All reviewers suggest accepting the paper, so will be accepted. Please address the remaining concerns in the camera ready version.
Decision: Accept (poster)