FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
Authors: Chuiyang Meng
Ming Tang
Vincent W.S. Wong
Venue: ICLR 2026 Poster (2026)
Keywords: Federated fine-tuning
low-rank Gram matrix
Procrustes alignment
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a federated fine-tuning framework with a single low-rank Gram matrix and adopts Procrustes alignment on the decomposed matrix to improve the fine-tuning performance.

Abstract: Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces two types of challenges. First, aggregation error can arise from separately aggregating the two low-rank matrices.
Second, even if the server aggregates the product of two low-rank matrices, it needs to decompose the aggregated matrix back into low-rank matrices. Since the decomposition is not unique, it can lead to decomposition drift. To tackle the aforementioned challenges, we propose federated low-rank Gram-matrix aggregation (FLoRG), a federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix (i.e., the matrix of inner products of its column vectors). FLoRG can eliminate the aggregation error and reduce the communication overhead. It also minimizes the decomposition drift by introducing a Procrustes alignment approach which aligns the decomposed matrix between consecutive fine-tuning rounds for consistent updates. We theoretically analyze the convergence of FLoRG and prove that adopting the Procrustes alignment results in a tighter convergence bound. Experimental results across multiple LLM fine-tuning benchmarks demonstrate that FLoRG outperforms five state-of-the-art baseline schemes by providing higher downstream task accuracy and can reduce the communication overhead by up to 2041$\times$.

Published as a conference paper at ICLR 2026
FLORG: FEDERATED FINE-TUNING WITH LOW-RANK GRAM MATRICES AND PROCRUSTES ALIGNMENT
Chuiyang Meng
Ming Tang Southern University of Science and Technology t Langm@@susteee edu.
The University of British Columbia chu yangiinggggneeecco wwc.ca... ca
Vincent W.S. Wong
The University of British Columbia vincentweece. ubc.ca
ABSTRACT
Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning intro- duces types of challenges. First, aggregation aggregating the two low-rank matrices. Second, even the server aggregates the product of two low-rank matrices, it needs decompose the aggregated ma- trix back into low-rank matrices. Since the decomposition is not unique, it can lead decomposition drift. To tackle the aforementioned challenges, we propose federated low-rank Gram- matrix aggregation (FLoRG), federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggre- gates its Gram matrix (i.e., the matrix of inner products of its column vectors). FLoRG can eliminate the aggregation error and reduce the communication over- head. It also minimizes the decomposition drift by introducing Procrustes align- ment approach which aligns the decomposed matrix between consecutive fine- tuning rounds for consistent updates. We theoretically analyze the convergence of FLoRG and prove that adopting the Procrustes alignment results a tighter convergence bound. Experimental results across multiple LLM fine-tuning bench- marks demonstrate that FLoRG outperforms five state-of-the-an baseline schemes by providing higher downstream task accuracy and can reduce the communication overhead by up to 2041x.
arise from separately
INTRODUCTION
Large language models (LLMss) (Zhao al., 2023 Achiam al. 2023 Touvron et al. 2023 Grattafiori et al. 2024) have achieved sfate-of-the-ar performance across a wide range of natu- ral language processing tasks. However, their massive scale introduces critical challenges in terms of computation cost, memory consumption, and adaptability to downstream tasks. To address these concerns, low-rank adaptation (LoRA) (Hu et al., 2022, Hayou et al. 2024 Kopiczko et al.) 2024] effective approach. In particular, the LoRA module employs
Liu et al. 2024) has emerged a fine-tuning matrix AW with two low-rank matrices B and A into the pretrained model w as W = W AW W BA. Thus, it enables task-specific adaptation while only updating matrix AW. This approach reduces both memory usage and computation overhead significantly when compared with full-model fine-tuning. However, fine-tuning still favors domain-specifi, data at scale. Such data is typically distributed across multiple clients and thus requires collaborative fine- tuning. To resolve this issue, federated learning (FL) (McMahan et al. 2017 Liet al. 2020) provides framework for collaborative model training. where multiple clients fine-tune shared global model without exposing their raw data. Combining LoRA with FL is therefore highly
"Corresponding author: Chuiyang Meng

---

Published as a conference paper at ICLR 2026
appealing: clients can collaboratively fine-tune a model by locally performing lightweight training via LoRA modules and uploading the low-rank matrix updates for global aggregation.
The conventional works (Zhang et al., 2024 Fang et al.] 2024 Zhang et al., 2023b; Wu et al, 2024) propose federated fine-tuning with LoRA, which enables each client n to transmit its low- rank matrices B,, and A, to the central server. Afterwards, the central server aggregates B,, and A, separately and then broadcasts the two aggregated matrices back to each client for performing fine-tuning in the subsequent rounds. In this case, the updated LoRA module after model aggre- gation can be expressed as & Σ B.)( En= A,), where N is the number of clients. This approach introduces a challenge: The aggregation is fundamentally biased, because the true update should be 1/2 ( N
which is different from (/2 >> B,,B,,] Em= A... This mismatch introduces a systematic aggregation error which affects the global model update in each fine-tuning round. As the number of rounds increases, the aggregation error is accumulated, which can degrade the fine-tuning performance.
To alleviate the error induced by model aggregation, some works (Yan et al. 2024 Bai al. 2024 Yan et al. 2025) calculate the product of B, and perform aggregation at the central server. Then, the central server performs matrix decomposition to recover two low-rank matrices for the next fine-tuning round. Although this approach can eliminate the error induced by separately aggre- gating matrices B,, and A,. it introduces another non-trivial challenge: decomposition generally nof unique. In particular, the rank of matrices B,, and A. module is typically much smaller than the dimension of the input or output of the parameter matrix. The aggregated matrix is typically rank-deficient and may also exhibit repeated eigenvalues. As result, there may be multiple valid decompositions into two low-rank matrices. In addition, when the aggregated matrix has eigenvalue multiplicities, many valid decompositions exist. As a result, choosing different matrix decomposi- tions fundamentally changes two low-rank matrices. It incurs a drift in the parameter subspace and changes the direction of the model update for the subsequent fine-tuning rounds. This drift will ac- cumulate as the fine-tuning proceeds and degrade the fine-tuning performance. Furthermore, direct decomposition (e.g., eigendecomposition) may incur rank mismatch since the rank of the aggregated matrix may be different from the local low-rank matrices.
Based on the aforementioned discussions, we focus on addressing the following question: there federated fine-tuming approach which can eliminate the error induced by separate model aggrega- tion while minimizing the drift induced by matrix decomposition?
To address this question, we start by rethinking what to aggregate in federated fine-tuning As LoRA involves matrix multiplication, either separate model aggregation or matrix decomposition is unavoidable. Therefore, one of our key insights is to reparameterize the LoRA module with a single low-rank Gram matrix. We aim to propose a federated fine-tuning framework which aggregates the corresponding Gram matrices to achieve an unbiased aggregation with low communication over- head, Furthermore, another insight is to propose a Procrustes alignment approach to the decomposed matrix in order to stabilize the fine-tuning while preserving its Gram matrix, thereby mitigating the decomposition drift due to the non- -nnuueeqsss of matrix decomposition.
Designing such framework is challenging due to the following unexplored questions: (i) How can we design a low-rank parameterization which adapts to any pretrained matrix while supporting error-free aggregation? (ii) How to optimize the Procrustes alignment matrix minimize the de- composition drift? (iii) How to characterize the overall convergence rate of the proposed framework under nonconvex losses, and disentangle the impact of Procrustes alignment on the convergence? In this work, we make the following contributions to address the aforementioned questions:
We propose FLORG, federated fine-tuning framework with a single low-rank matrix and Gram-matrix aggregation. By leveraging a shared semi cemi-orthogonal basis, FLoRG adapts to parameter matrices with arbitrary dimensions. Each client only updates the single low-rank matrix, and the server aggregates the corresponding Gram matrix. FLoRG eliminates the aggregation error by turning the bilinear server-side aggregation into a linear operation. By transmitting single matrix instead of two matrices, FLoRG is communication -ffficctt when compared with some of the existing federated LoRA schemes.
We propose Procrustes alignment approach after matrix decomposition to preserve the aggregated Gram matrix while aligning the decomposed matrix across fine-tuning rounds to mitigate the decomposition drift. We formulate an optimization problem to minimize
2

---

Published as a conference paper at ICLR 2026
the decomposition drift via Frobenius-nome norm objective. The closed-form optimal solution projects the decomposed matrix onto a target r-rank subspace without changing its Gram matrix.
We theoretically analyze the convergence rate of our proposed FLoRG in the nonconvex loss setting. In particular, incorporating our proposed Procrustes alignment mitigates the decomposition drift, thereby resulting in tighter convergence bound.
We conduct extensive experiments GLUE (Wang al. 2018 with MRPC, QQP, MNLI, QNLI, WNLI, and RTE datasets. We compare our proposed FLoRG with five state-of tfte-- art baseline schemes, including FedIT (Zhang et al., 2024). FeDeRA (Yan et al. 2024). FFA-LoRA (Sun et af. (2024), FedSA-LoRA (Guo et al. (205), and FedEX-LoRA (Singhal et al., 2025). Results show that our proposed FLoRG achieves higher testing accuracy than the baseline schemes under different settings and reduces the communication overheaa by up to 2041x.
RELATED WORKS
LoRA: Low-rank adaptation (LoRA) (Hu et al., 2022)
introduces two low-rank matrices to the Multiple LoRA variants have been proposed (Zhng et al. 2023a. Dettmers et al., 2023 Hayou et al. 2024| Kopiczko et al. [2044]
pretrained model to perform parameter-efficien
Liu et al. 2024 Zhao et al. 2024 Bensaid al. 2025). For example, (Zhang al. 2023a), the authors proposed a LoRA framework to adaptively allocate the parameter budget among weight matrices based on the importance score. In (Zhao et al. 2024), the authors projected the gradient matrix into a low-rank form to perform efficient fine-tuning. In (Bensaid et al. 2025), the authors reformulated the low-rank adaptation with single matrix. While the aforementioned works have shown an improvement in the performance, they are primarily designed for centralized settings. Domain- specifc data are often possessed by number of distributed clients, which motivates the incorporation of FI..
Federated Fine-tuning with LoRA: LoRA has been incorporated into FL to enable collaborative fine-tuning across distributed clients. The conventional federated fine-tuning works (Zhang al., 2024 Fang et al. 2024 Zhhnn et al., 2023b Wu et al.; 2024, Long et al. 2024 Cho et al.) 2024
Byun Leel 2025) directly aggregate two low-rank matrices separately to obtain the global model.
Some works (aaaakniya et al. 2023 Yan et al. 2024| Bai et al. 2024 Yan et al. 2025) aggregate
the LoRA modules (i.e., the products of two matrices) and then perform matrix decomposition to
recover two low-rank matrices. In addition, the authors in (Wang et al.) 2024) proposed a stacking-
based approach to aggregate the low-rank matrices. The authors in (Sun al. 2024) proposed to
freeze matrix A and only update matrix B. The authors in (Guo et al.
2025
proposed to let the
clients locally update matrix B and only share matrix for aggregation.
3 METHODOLOGY
3.1 FLORG
In this section, we propose a federated fine-tuning framework called FLoRG. FLoRG employs single low-rank matrix and aggregates Gram matrices to eliminate the error induced by separate aggregation as in conventional LoRA. FLoRG performs Procrustes alignment to the decomposed matrix to minimize the decomposition drift. A schematic illustration is shown in Fig. |
We consider a central server and N clients. Let 21, T) and N 21, N) denote the set of T fine-tuning rounds and the set of N clients, respectively. At the beginning of the first fine-tuning round, each client n E N has the same pretrained weight matrix w E Rout xdhn Note that w is kept frozen and will not be updated during fine-tuning. We consider that each client has local dataset D,,. Let &n ~ D,, denote a mini-bate denote the local loss function of client denote the expected loss of client n with model W as fx(W) = Eg.-D. [F, (W': g.)]. Let (W.::: {.) e ddw xd, denote the local stochastic gradient of client n with model W in the 1-th fine-tuning round. The learning procedure of FLoRG can be summarized in the following steps.
training samples. Let F, (W'; E.) &n with model W in the 1-th fine-tuning round. We
3

---

Published as a conference paper at ICLR 2026
(
x
++
Alighment
Eigendecomposition
x
A
Aggregation
Q
A++1
Central Server
(
Ayy
A
min,
&f
A
Downlink
Upänk
A!
A
Downlink
Uplink
Av
L
X
(Amt)
X
A
R
L
x
(
X
A
X
R
Local
Client
Update
Client N
Update
x
X
X
x
X
X
Figure 1: System model of our proposed FL.oRG.
(i) Local Update with Low-rank Gram Matrices: Different from LoRA, FLoRG uses a single matrix A E R for fine-tuning, where k = min (din- dout and & min dout denotes the rank A', Matrices Both matrices L and R are semi-orthogonal, i.e., L I and RRT = Ir, where Ię denotes the l x & identity matrix. Note that matrices and remain unchanged during fine-tuning tooinunnng..... The fine-tuning matrix in the t-th fine-tuning round is
Rdout*k and E RR are initialized and shared across clients.
AW' LQ'R R = L(A')TARR tET,
(1)
where Q' denotes the square Gram parameter matrix in the i-th fine-tuning round. By leveraging L and R. FLoRG is compatible with parameter matrices of any dimensions. Thus, the full model can be expressed as W w + A
In the (-th fine-tuning round, the central server broadcasts A' to all clients. Client n E N updates A' using its local dataset. Let V AAF.(WT (W.: E B denote the gradient of the low-rank matrix, which is given by
VAFA 8.) A' (H) + (H)))), nEN,EET,
(2)
where H. L'VE, RT E R Client performs stochastic gradient descent to update matrix A'. We define 77 the learning rate. Let A ++} denote the locally updated low-rank matrix of client n in the t-th fine-tuning round, which is given by
A A' W NEN,EET.
(3)
(ii) Model Aggregation: When compared with the conventional federated LoRA schemes, in which clients must upload both locally-updated low-rank matrices, client in FLoRG only needs to transmit A (+7 to the central server, thereby reducing the per-round uplink communication over- head by more than half. Let Q% レ- (A 1+ A L+ denote the locally updated Gram matrix for client in the 1-th fine-tuning round. After receiving the locally updated parameter matrices from all clients, the central server performs model aggregation with respect A as
Q
Q++1_
1 N
Σ REN
Q ++
tET,
(4)
where Q--1 E k the aggregated Gram matrix at the end of the t-th fine-tuning round. Due to the Gram matrix design, aggregating (A A ++ is linear and preserves the positive semi- definite (PSD) property. Therefore, the central server obtain the true aggregated matrix. This

---

Published as a conference paper at ICLR 2026
removes the bilinear inconsistency (i.e., aggregation enror) induced by aggregating matrices B,, and An separately as in conventional federated LoRA schemes, Let r's denote the rank of matrix Q which satisfies
(x) S min {k, Nr).
rank (aik(Q - Sr' rank(Q (1+
(5)
where inequality (a) follows from the subadditivity property of rank operator.
(iii) Decomposition with Procrustes Alignment: Since Q++ is square PSD Gram matrix, the central server performs eigendecomposition to Q as follows:
Q Q (Pt+t)TA+tpIis
tET,
(6)
where p E R A+11 E R denotes the eigenvalue matrix of Q* Thus, canonical decomposition which satisfies (À ( A [+1 2+ is
A ++ (A ( TET,
where (+1 is a valid decomposition of Q++11 However, directly applying A [+1 for the subsequent fine-tuning round (i.e., A [+1 == 1++1 may yield sub-optimal performance since the decomposition yields two challenges: non-unique decomposition and rank mismatch.
First, the above expression of matrix A++ provides canonical representation of the decompo- sition of Q4+1, which non-unique. This is because for any matrix with orthogonal columns O E T x we have 10A++1)TOA guarantees that (AT A++1 preserves the aggregated Gram matrix, there exist many such decom- positions due to the non-uniqueness of decompositions (e.g., Cholesky decomposition or singular value decomposition (SVD)) when the rank is deficient the eigenvalues have multiplicities. How- ever, this -on-uniquessss affects the future fine-tuning. From the update rule eqn. (2), each client performs local fine-tuning in the (t + 1)-th fine-tuning round using the gradient which de- pends explicitly on A [+1 As result, different decompositions of Q may yield different 1+1 which result in divergent gradient paths. These paths may potentially lead to unstable fine-tuning performance across fine-tuning rounds.
Q Furthermore, although the decomposition
Second, stated in inequality (5), the rank of the decomposed matrix may be different from that of the original one (i.e., yd * r). In such cases, we need to recover a matrix of the target rank r for consistency across fine-tuning rounds.
The aforementioned challenges motivate reparameterizing matrix À 1+1 stabilize the subsequent fine-tuning process while enforcing the target rank r. To this end, we propose Procrustes alignment to project A 1++1 onto the r-rank subspace. Let St E R denote this Procrustes alignment matrix in the t-th fine-tuning round. The matrix after projection is denoted as Ä'Ä*+ 1+1 In particular, Procrustes alignment minimizes the Frobenius norm between the matrix after projection S' A ++1 and A', which minimizes the drift due to the non-uniqueness of matrix decomposition. We then formulate the following optimization problem in the 1-th fine-tuning round:
P1
minimize St
S'AA (+1+
A
(8a)
subject to
(S')T'- I
(8b)
To solve problem P1, we first convert the objective function
8a into the following form:
|''A A
Tr ((A')TA) Tr ((A (A+1))()))
-2'r 2 (111))--21-------- (A((A+I)T(S')))
||
2
A ++1
2Tr - (A((A*))T(SST))
Since ||A and | || have been determined after matrix decomposition the end of the 1-th fine-tuning round, problem Pi is equivalent to the following problem:
P2 maximize S
Tr (A)(())))))))))))))))))
(10a)
5

---

Published as a conference paper at ICLR 2026
subject to constraint (b)).
Then, we present the following theorem to obtain the optimal solution problems P2 and Pı, with the proof provided in Appendix A.I.
Theorem 1. (Optimal Procrustes Alignment Matrix) We denote the SVD of matrix (A as (((A+)) (V)T, where Ut E Brxr" and V E "x''" have orthogonal columns. Let E' diag(ot; o2. .2.-..,,00 ('me) BR''"x"" denote the diagonal matrix of singular values of A (A(ATT). The optimal solution St** to problems P2 and Pi satisfies
S - -+=U00 U'(").
(11)
The main benefits of our proposed Procrustes alignment are two-fold. First, it resolves the issue due to non-unique decomposition.
particular, the Proerustes alignment approach selects, among all valid decompositions of Q , the one which is nearest to that in the last fine-tuning round (i.e., A in Frobenius norm. Thus, it stabilizes the gradients in the subsequent fine-tuning rounds. Second, addresses the rank mismatch issue of the decomposed matrix by using semi-orthogonal Procrustes alignment matrix project A *++1 with rank rri onto the target r-rank subspace.
After the central server has determined matrix St**, it calculates the low-rank matrix for the (t (t+1) 1)- th fine-tuning round as A (+1 = S r+ Thus, the full model in the (t + 1)-th fine-tuning round given by
W W + L(A
(12)
Then, the central server broadcasts the matrix A to all clients on the downlink. Note that the proposed FL.oRG can effectively reduce the per-round communication overhead by more than half when compared with traditional federated LoRA schemes. For a model with L. LoRA layers, the total per-round computation overhead of cigendecomposition and Procrustes alignment at the server (LL**) (O (Krr'((+++ )² max(r, r))) From this complexity analysis, the per-round server computation overhead scales linearly with the number of LoRA layers and is independent of the number of clients (beyond the standard Gram aggregation), indicating that FLoRG is scalable even with many layers and many clients. The workflow of our proposed FLoRG is presented in Appendix A.2.
3.2 THEORETICAL ANALYSIS
In this section, we analyze the convergence rate of our proposed FLoRG. We define the global loss function in the t-th fine-tume tne-tuning round as f(W*) 1
N EnEN fa (W*). and its gradient as Vf(W*) = R EnEN fz (W") E Idd xd Without loss generality, we consider nonconvex loss functions in analysis. We first present the following assumptions which widely used in the literature (e.g., eL al. 2020 Wang et al.) 2020
Guo et al. 2025)).
Assumption 1. (L-Smoothness eL al. 2020
Wang et al. 20200) The loss function of each client That is, for two arbitrary matrices W and WH+1 W W F [ w/ - L LET, ,,,, N
n is continuously differentiable and L-smooth.
we have Jn (W++1) $ Jn(WW) fn (WO).
Assumption 2. (Bounded Gradient (Li et 2020 En.
Wang et al. 2020)) The local stochastic gradient v,, E T, N.
of A upper-bounded, i.e., DE,. V
2025 The Frobenius norm of model pa- Assumption 3. (Bounded Parameter Space (Guo et al. rameter matrices A' and À' are upper-bounded by two positive constants CA and Č, respectively, ie., || S CA. | I CA. TET.
In addition, present two lemmas to facilitate convergence analysis, with the proof provided in Appendices A.3 and A.4, respectively.
Lemma 1. For illustration simplicity, denote G VF, RRuuu We denote H' N EnEx H. = N EnEN LT LTG, RT. Let Amiu (X) denote the smallest positive eigenvalue of matrix X. For any matrices A and H', we have
H. (A (H)) + H H (ATTAA) 2 Amina (AA)((AA)AAA) H
TET.
F.
(13)

---

Published as a conference paper at ICLR 2026
Lemma 2. Let S' denote arbitrary Procrustes alignment matrix in the t-th fine-tuning round. Let Jmin (X) denote the smallest positive singular value of matrix X. We define X++1
St* A A
2 The difference of two Procrustes alignment ma-
F
trices bounded as follows:
a+1 proc
St S
TET.
(14)
À (A!)T)
miin
Now, we present the convergence rate of our proposed FLoRG in the following theorem, with the proof provided in Appendix A.5.
Theorem 2. (Convergence Rate of FLoRG) We denote the optimal model as W Under Assump-
tions - and Lemmas 1 - 2, if the learning rate satisfies dj 4minge (Dinn ((AF)TA*)) the convergence rate of our proposed FLoRG is bounded by
1 T
)(W))- f(W))
2 20
3Ld 3 (1p + +2
VS(W)) S
TΩ
20
Initial optimality gup
Residual bias term
1113
qpO EIET mmin ( (A9))
proc
(15)
2NT min AAmin ((A))TA)))
Procrustes alignment drift term
where 52 2ny minter (Amin ((A))TA")) S
Based on the theoretical analysis, we can observe that the convergence rate of our proposed FLoRG depends on three terms. The first term is the initial optimality gap, which diminishes as the num- ber of fine-tuning rounds T' increases. The second term is the non-diminishing residual bias term. The third term is the Procrustes alignment drift term, which captures the impact of the Procrustes alignment on the convergence rate. When the Procrustes alignment applied, A proc becomes zero. Hence, the third term becomes zero, under which achieve tighter bound and improve the convergence rate.
A++1
EXPERIMENTAL RESULTS
4.1 EXPERIMENT SETUP
Base Models, Datasets, and Baseline Schemes: We choose OPT-125M (Zhang et al. 2022 RoBERTa-lare (Liu al. 2019), and Llama--3223B (Grattafiori et al. 2024 as three base mod- els with three different scales. In particular, OPT-125M, RoBERTa- -aBERTa-large.. and Llama-3.2-3B have 125M, 355M, and 3B model parameters. We choose GLUE (Wang et al. 2018 as natural language understanding benchmark with MRPC, QQP, MNLI, QNLI, WNLI, and RTE datasets. In addition, we choose SQuAD (Rajpurkar et al. 2016) v1.1 question- question-answertie answering dataset. We compare the performance of our proposed FLoRG with the following baseline schemes:
FedIT (Zhang et al. 2024):
: Client E N transmits the locally updated LoRA matrices B, and A, to the central server. The central server aggregates B,, and A, separately.
FeDeRA (Yan et al., 2024) Client e N transmits locally updated LoRA matrices B,, and A, to the central server. The central server aggregates B, and performs SVD to obtain the updated matrices B, and A,,
2024] Client n E N freezes matrix A, and only updates matrix B, The central server performs aggregation on matrix B,,
FFA-LORA (Sun et al.
2025): Client E W locally updates matrices B,, and A, but only shares matrix A, for aggregation.
FedSA-LORA (Guo et al.
FedEx-LoRA (Singhal et al. ces B, and A, to the central server. The central server aggregates B,, and A,, separately and determines residual matrix mitigate the aggregation error.
2025): Client n / transmits locally updated LoRA matri-

---

Published as a conference paper at ICLR 2026
Implementation Details: To show the learning performance of GLUE datasets, show the test- ing accuracy. To characterize the communication overhead, we present the total number of parame- ters transmitted between all clients and the central server. For GLUE, model the data heterogene- ity across clients' local datasets, we use the Dirichlet distribution Dir(p) to create non-independent and identically distributed (non-IIID) data partitioning. In particular, p 0 controls the degree of non-IIDness across clients" local datasets, A lower value of p indicates higher degree of data het- erogeneity. To show the learning performance of the SQuAD v1.1 dataset, we show the exact match (EM) and F1 scores. For SQuAD v1.1, we consider the dataset to be independently and identically distributed (IID). In the ablation studies, we choose RoBERTa-larg. as the base model. Unless stated otherwise, we consider all clients participate in the fine-tuning and set 1) 5e - 5, 0.5, N 20, and 4. The details of the hyperparameter settings are presented in Appendix A.6.
4.2 LEARNING PERFORMANCE
In this section, we compare the testing accuracy of different schemes under different base models and datasets. Due the space limit, we present the results under MNLI, QNLI, WNLI, and RTE datasets. The experiments are repeated under 2 random seeds, and we report the average testing accuracy. Results in Table show that our proposed FLoRG outperforms the baseline schemes under those four datasets in most cases. In particular, on over the strongest baseline by 1.52 on MNLI, 1.13 on WNLI, and 0.65 on RTE. On RoBERTa-large the margins are 0.31 on MNLI, 0.45 on QNLI, 0.37 on WNLI, and 0.28 on RTE. On Llama-3.2-31 FLoRG improves the testing accuracy over the strongest baseline by 0.41 on MNLI, 0.84 on WNLI, and 0.69 RTE. Additional experimental results presented in Appendix A.7. These results validate the superiority of FLoRG.
FLoRG improves the testing accuracy
Table I: Comparison of the testing accuracy across different baseline schemes.
Base Model
Dataset
FLoRG
FedIT
FeDeRA
FFA-LORA
FedSA-LORA
FedEx-LoRA
MNLI
87.35
79.42
81.15
83.54
84.61
85.83
QNLI
89.52
84.18
86.71
87.93
88.69
89.88
OPT-125M
WNLI
65.28
58.45
59.34
62.61
62.83
64.15
RTE
68.92
61.08
64.51
66.02
67.39
68.27
MNLI
91.27
84.91
88.06
89.28
90.75
90.96
QNLI
92.58
87.49
90.07
90.96
91.54
92.13
RoBERTa-large
WNLII
66.48
59.34
61.83
63.72
63.47
66.11
RTE
71.26
64.25
67.12
68.49
69.93
70.98
MNLI
93.15
87.24
89.83
91.05
92.38
92.74
QNLI
93.12
89.17
91.45
92.53
93.27
93.05
Llama-3.2-3B
WNLI
68.73
61.52
64.19
65.97
66.81
67.89
RTE
73.84
67.08
69.75
71.33
72.56
73.15
4.3 COMPARISON OF THE COMMUNICATION OVERHEAD
In this section, we compare the communication overhead incurred under different baseline schemes. We use the QNLI dataset conduct the experiments. Results in Table 2 show that achieve the target test accuracy, our proposed FLoRG uses much lower total number of transmitted model parameters when compared with the baselines. This demonstrates that FLoRG can significantly reduce the communication overhead by up to 2041x x when compared with the baseline schemes. Additional results of the computation overhead and memory usage are presented in Appendix A.7.
4.4 ABLATION STUDIES
Impact the Procrustes alignment In this subsection, we study the impact of our proposed Pro- crustes alignment the learning performance. Results in Table show that by applying Procrustes
8

---

Published as a conference paper at ICLR 2026
Table Comparison of the total number of transmitted parameters (upload and download) achieve the target accuracy. Symbol "_" means that target accuracy cannot be achieved.
Base Model
Target Acc.
FLoRG
FedIT
FeDeRA
FFA-LORA
FedSA-LoRA
FedEx-LoRA
80.00
10
3.78 10"
2.46 107
1.59 107
2,10 10"
1.25 10
OPT-125M
85,00
1.07 10
4.2 10"
2.75 10"
3.61 10"
1.77 10
RoBERTa-large
80,00
8.7 10
4.68 x 10
3.02 x 107
1.85 107
2.59 x 10"
2.08 10
85.00
1.45 107
8.12 10"
5.17 x 10"
3.35 107
4.42 10"
2.96 10
aligmment, our proposed FLoRG yields consistent improvement in terms of the testing accuracy. On OPT- 125M. Procrustes alignment provides an improvement of 3.40 on MRPC, 2.86 on QQP, 6.27 on MNLI, 2.97 on QNLI, 5.60 on WNLI, and 4.45 on RTE. respectively. On RoBERTa-large. Procrustes alignment provides an improvement of 3.37 on MRPC, 2.84 on QQP, 2.46 on MNLI, 3.86 on QNLI, 4.34 on WNLI, and 4.31 on RTE, respectively, whereas Fl FLRRG without Procrustes align- ment can only achieve comparable testing accuracy to FeDeRA, as shown in Table It showcases the importance of our proposed Procrustes alignment to improve the fine-tuning performance.
Table 3: Comparison of the testing accuracy of FLoRG with and without Procrustes alignment.
Base Model
FL.oRG
MRPC
QQP
MNLI
QNLI
WNLI
RTE
w/ Procrustes alignment
86.54
88.71
87.20
65.41
68.77
OPT-125M
w/o Procrustes alignment
83.14
85.85
80.93
86.72
59.81
64.32
w/ Procrustes alignment
89.87
91.27
91.39
92.48
66.41
71.40
RoBERTa-large
w/o Procrustes alignment
86.50
88.43
88.93
88.62
62.07
67.09
Impact of the Rank In this subsection, we vary r to demonstrate the impact of rank on the fine- tuning performance. In partieular, conduct experiments under 2, 4, respectively. We present the results under the WNLI and RTE datasets. Results in Table 4 show that our proposed FLoRG outperforms the baseline schemes under different rank settings, demonstrating the robust- ness of our proposed FLoRG under various ranks. Additional experimental results can be found in Appendix A.7.
Table 4: Comparison of the testing accuracy under diffcrent ranks.
Rank
Dataset
FLoRG
FedIT
FeDeRA
FFA-LORA
FedSA-LORA
FedEx-LoRA
WNLI
60.55
55.57
56.30
57.50
59,14
58.32
RTE
65,82
58.41
61.19
62.88
64.30
62,79
WNLI
66.34
59.19
61.97
63.55
63.61
66.12
RTE
71.41
64.11
66.97
68.62
70.10
71.28
WNLI
68.83
61.70
63.52
65.10
66.47
68.02
RTE
72.10
64.78
66.99
68.02
70.61
71.30
Robustness the Data Heterogeneity In this subsection, study the impact of the degree of data heterogeneity across clients' local datasets on the fine-tuning performance. We present the re- sults under the WNLI and RTE datasets. It can be observed in Table 3 that under different degrees of data heterogeneity, our proposed FLoRG outperforms the baseline schemes under more hetero- geneous settings. addition, the degree of data heterogeneity increases (i.e., P decreases), the improvement over the baseline schemes also increases, showcasing the robustness and superiority of our proposed FLoRG under heterogeneous data settings. Additional experimental results are presented in Appendix A.7.
Matrix Initialization for Matrices L and R In this subsection, we show the impact of the initial- ization of matrices L and R the learning performance. In particular, compare proposed

---

Published as a conference paper at ICLR 2026
Table Comparison of the testing accuracy under different degrees of data heterogeneity.
Non-IIDness
Dataset
FLoRG
FedIT
FeDeRA
FFA-LORA
FedSA-LoRA
FedEx-LoRA
WNLI
60.12
53.07
54.21
56.14
57.74
58.84
= 0.1
RTE
65.30
55.60
59.19
61.20
60.75
61.67
WNLI
66.34
59.19
61.97
63.55
63.61
66.12
0.5
RTE
71.41
64.11
66.97
68.62
70.10
71.28
WNLI
67.83
61.70
63.52
64.33
65.61
68.31
RTE
72.21
66.90
68.71
70.40
71.78
72.55
semi-orthogonal initialization with Kaiming initialization (He et al. 2015) and SVD initialization (Boutsidis & Gallopoulos) 2007). Results in Table 6 show that the semi-orthogonal approach out- performs the other two approaches in most cases, demonstrating the effectiveness of proposed initialization approach for matrices and R.
Table Comparison of the testing accuracy under different initialization schemes.
Base Model
Initialization
MRPC
QQP
MNLI
QNLI
WNLI
RTE
Semi-orten Semi-orthogonal
86.54
88.71
87.20
89.69
65.41
68.77
OPT-125M
Kaiming
84.35
88.90
85.23
87.73
62.29
69.31
SVD
86.41
87.69
83.19
88.74
64.37
67.69
Semi-orthogonal
89.87
91.27
91.39
92.48
66.41
71.40
RoBERTa-lurge
Kaiming
87.68
92.34
89.11
91.57
64.19
70.32
SVD
88.70
90.37
91.49
91.45
65.08
71.40
Impact of the Client Availability In this subsection, we show the impact of the client availability on the learning performance. particular, we conduct experiments under varying client participa- tion ratios. Results in Table 7 show show that under different ratios of client participation, our proposed FLoRG outperforms the baseline schemes most cases, demonstrating its applicability.
Table 7: Comparison of testing accuracy under different client participation ratios.
Client participation ratio
Dataset
FLoRG
FedIT
FeDeRA
FFA-LORA
FedSA-LORA
FedEx-LORA
WNLI
58.42
54.15
55,20
56.30
57.10
56.46
0.2
RTE
63,25
57.80
59.45
60.92
61.85
61.35
WNLI
64.50
58.10
60.35
61,80
62,40
63.58
0.5
RTE
69.15
62.50
65.20
66.75
68.20
68.61
WNLI
66.34
59.19
61.97
63.55
63.61
66.12
RTE
71.41
64.11
66.97
68.62
70.10
71.28
CONCLUSION
In this work, we proposed federated fine-tuning framework with low-rank Gram matrices called FLoRG. In particular, FLoRG features single low-rank matrix instead of two low-rank matrices as in the conventional LoRA module. By transmitting this matrix and aggregating the correspond- matrices and
ing Gram matrix, FLoRG eliminates the error induced by separately aggregating significantly reduces the per-round communication overhead. Moreover, we proposed a Procrustes alignment approach reparameterize the decomposed low-rank matrix after model aggregation. We theoretically analyzed the convergence rate of our proposed FLoRG framework and character- ized the impact our proposed Procrustes alignment on the convergence. Experimental results show that our proposed FLoRG framework achieves higher testing accuracy and can reduce the communication overhead by up to 2041 x when compared with five baseline schemes.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.08/ 08774, Mar. 2023.
Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. SLoRA: Federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.06520 066222288..6.., Aug. 2023.
Jiamu Bai. Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In Proc. Advances Neural Info. Process. Syst. (NeurIPS), Vancouver, Canada, Dec. 2024.
David Bensaïd, Noam Rotstein, Roy Velich, Daniel Bensaïd, and Ron Kimmel. SingLoRA: Low rank adaptation using a single matrix. arXiv preprint arXiv22507.05500 Jul. 2025.
Christos Boutsidis and Efstratios Gallopoulos. SVD based initialization: A head start for nonnega- tive matrix factorization. Puttern Recognition, 41(4):1350-136; Sep. 2007.
Yuji Byun and Jaeho Lee. Towards federated low-rank adaptation of language models with rank het- erogeneity. In Proc. Conf. Nations Americas Assoc. Comput. Linguistics (NAACL), Albuquerque, NM, Apr. 2025.
Yac Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), Miami, FL, Nov. 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoye. QLoRA: Efficient finetuning of quantized LLMs. In Proc. Advances Neural Info. Process. Syst. (NeurIPS), New Orleans, LA, Dec. 2023.
Zihan Fang, Zheng Lin, Zhe Chen, Xianhao Chen, Yue Gao, and Yuguang Fang- Automated federated pipeline for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2404. 06448, Apr. 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The L.LMMA 3 herd of models. arXiv preprint arXiv:2407.2 Jul. 2024.
Pengxin Guo, Shuang Zeng, Yanran Wang. Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. In Proc. Int. Conf. Learn. Representa- tions (ICLR), Singapore, Apr. 2025.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Proc. Int. Conf. Mach. Learn. (ICML), Vienna, Austria, Jul. 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. IEEE Int. Conf. Compute. Vis. (ICCV), Santiago, Chile, Dec. 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In Proc. Int. Conf. Learn. Representations (ICLR), Apr. 2022.
Dawid Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. VeRA: Vector-based random matrix adaptation. In Proc. Int. Conf. Learn. Representations (ICLR), Vienna, Austria, May 2024.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of FedAvg non-lID data. In Proc. Int. Conf. Learn. Representations (ICLR), Apr. 2020.
Shih- Yang Liu, Chien Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Hung Chen. DoRA: decomt-ee low-rank adaptation. In Proc. Int. Conf. Mach. Learn. (ICML), Vienna, Austria, Jul. 2024.

---

Published as a conference paper at ICLR 2026
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint arXiv:1907.11692, Jul. 2019.
Guodong Long, Tao Shen, Jing Jiang, Michael Blumenstein, et al. pursonazsni adapter for federated foundation models. In Proc. Advances Neural Info. Process. Syst. (NeurlPS), Vancou- ver, Canada, Dec. 2024.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- effficcent learning of deep networks from decentralized data. In Proc. Int. Conf. Artif. Intell. Statist. (AISTATS), Fort Lauderdale, FI.,, Apr. 2017.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.00 00..0006600520,, 2016.
Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of foundation models. In Proc. Annu. Meeting Assoc. Com- put. Linguistics, Vienna, Austria, Jul. 2025.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving LoRA in privacy-preservin feder- ated learning. In Proc. Int. Conf. Learn. Representations (ICLR), Vienna, Austria, May 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. Feb. 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint TET4:18800.04666 Apr. 2018
Jianyu Wang, Qinghua Liu, Hao Liang Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogencous federated optimization. In Proc. Advances Neural Info. Process. Syst. (NeurIPS), Dec. 2020.
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In Proc. Advances Neural Info. Process. Syst. (NeurIPS), Vancouver, Canada, Dec. 2024.
Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. FedBiOT: LLM local fine-tuning in federated learning without full model. Proc. ACM SIGKDD Conf. on Knowl, Discovery Data Mining (KDD), Barcelona, Spain, Aug. 2024.
Yunlu Yan, Chun-Mei Feng, Wangmeng Zuo, Rick Siow Mong Goh, Yong Liu, and Lei Zhu. Fed- erated residual low-Rank adaptation of large language models, In Proc. Int. Conf. Learn. Repre- sentations (ICLR), Singapore, Apr. 2025.
Yuxuan Yan, Qianqian Yang, Shunpu Tang, and Zhiguo Shi. FeDeRA: Efficient fine-tuning of lan- guage models in federated learning leveraging weight decomposition. In Proc. Advances Neural Info. Process. Syst. (NeurIPS), Vancouver, Canada, Dec. 2024.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federated GPT: Federated instruction tuning. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), Seoul, Korea, Apr. 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive budget allocation for parameter- efficient fine-tuning. In Proc. Int. Conf. Learn. Representations (ICLR), Kigali, Rwanda, May 2023a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 May 2022.
12

---

Published as a conference paper at ICLR 2026
Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fed- PETuning: When federated learning meets the parameter-efficien tuning methods of pre-trained language models. In Proc. Annu. Meeting Assoc. of Comput. Linguistics (ACL), Toronto, Canada, Jul. 2023b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficien LLM training by gradient low-rank projection. Proc. Int. Conf. Mach. Learn. (ICML), Vienna, Austria, Jul. 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et survey of large language models. arXiv preprint arXivv2303. Mar. 2023.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A. PROOF OF THEOREM 1
During the Procrustes alignment, we define z'= =U')ss''' e "r"""""" Therefore, St satisfies
s' -''''
(16)
Note that (S')TS' = I, In addition, since matrices U and V have orthogonal columns, we have
(A'(A+)) = IF (VI(()
Tr (T((T(Z))T)
oj
(17)
Since matrix Z' satisfies (Z')TZ' = Ipss each element =/i yields =/ < 1, we have Tr ((()))) Ej of. with the equality achieved at Z' I Therefore, when the maximum objective is achieved, matrix S satisfies
S U'I
(18)
This completes the proof of Theorem 1.
A.2 ALGORITIM FOR FLORG
Algorithm FLoRG
I: Input: Local fine-tuning datasets Dr,, n. E N; learning rate y, pretrained model W initialized low-rank
matrix A
2: The central server initializes L and R with orthogonal columns and broadcasts them to all clients.
3: W := w + LLAA')AAT
4: For i do
The central server broadcasts
all clients.
6: For client n E N in parallel do
7:
A. 1
A' 1--w (W';
Transmits A ++ to the central server.
8:
9:
End for
Q4+1 1 EMEN A A!-!
10:
11: Perform rigendecomppsition to Q'+1 and obtain A 1+1 = (A (A"1)) (
12: Perform Procrustes alignment to 141 obtain g = UUVV
13: A A++1 S* Á
W = D L(A T A
15: End for
16: Output: fine-tuned model 441
A.3 PROOF OF LEMMA I
H. (A') (H')T) (H (F
(H')T (AT)TA( + (H))) (H (A'))A))
2 H' (H (A*)TA H'
2 H
H'H
(AT)TA))
(b)
Amiia Aamin((A)) (A))TA))
H +
+ Amin ((Ann((ATT))) (AA)TA))
2
F
2
14

---

Published as a conference paper at ICLR 2026
H' H(H)) 2
2
=4mii (A))
H
(19)
where equality (a) obtained due to the fact that the second factor within the inner product is symmetric. Thus, the skew-symmetric part of H is orthogonal. Inequality (b) results from (A') Amin ((A')TA')I:
This completes the proof of Lemma 1.
A.4
PROOF OF LEMMA 2
We expand S A 6+1
A'
as follows:
S'A [+1
A
A 1++1
|| 2Tr (A
(20)
Similarly, we have
Sh. A (+1
A'
F A
|| -TT (A*))+
(21)
Therefore, Attl pruc satisfies
A ALtl proc
2 Ir (th "S"AAA((A)))) -Tr (SAAAA((AA)T)))
(22)
Let M = (A') = U EM V where matrices UM E R xp't and VM E R Rx
have orthogonal columns, and ΣM TR" RR""x"" xxitt
Let P (V) and p
U (S Inse. Then, we have
Tr (S' (A'T)) = Tr = Eopph
(23)
==1
Similarly, we have
P.t
Tr (S*-AA+((AA))) = =oof of
(24)
By combining eqns.
22
23 and
24
we obtain
[+1 pruc
2 >
(1)5))))
(25)
j=1
Since matrices U and Vín have orthogonal columns, we have
||st - S' -122 = =|p' - I, 1|:
Tr -Tr((p' 1 Ipri) T(P' - Ipra)
-ppii- - Tr((P)) Tr ((P))T)
phe
2 Σ (1 - p}.))
(26)
j=1
By combining eqns.
(25 and
26) we have
r'
A proc 2 Σ
(1-1pa))
J=1
15

---

Published as a conference paper at ICLR 2026
y/1
2 20 min
(A')( 1(1 ps.))
(A++((A))T)
| Ipm
min
min
( (A'T()
|S - S 11
(27)
where inequality (a) is obtained due to the fact that muin (A*1(A))T) rearranging this inequality, we have
S oj.
,4 By
X++
||St -
proc
(28)
AAtI(A')T)
6 min
This completes the proof of Lemma 2.
A.5 PROOF OF THEOREM 2
Based on Assumption 1, we expand [[[(W++)) as
L
[f((*+))] < [f(W)] + (Vf(W') w - w') F 1
w
(29)
T2
We define GA 1 N
EDEN (AF,(W A (H')) W W satisfies
W W
=Wo L(A' I(A-I)TA*RR (Wo. +L(A))A'R)
=I(A+H)TA R -
1 Σ (L(A DA R R-AAAAR
L(A)TAR))
N
nEA
(a)
Σ (A )TA - (A')AA) R
(A
N
nEN
N Σ (A'- (A (VF. -A - (ADTA* R
nEN
-I
+ (GATA) + TIGG (A)TA)) )R. R
=L(F(GG)) - ( (GA)TA)) R
(30)
where equality (a) is obtained due to the fact that N EnEN(AA! / Q++1 (À (A+ (g A (A+I)TA+*) Let t denote the low-rank matrix obtained by applying the Procrustes alignment. We first bound Ti as follows:
Ti
7E (VS(W)) L ((A') + (G') R) + + (VIW)) E (Vf(W'), L ((A AA + (A*) - AA) R)
?E L (AAY) + (GWJA)) J
+ ,, LGGAR)))
T3 1) (HH, ((A))G) [ (H (
16

---

Published as a conference paper at ICLR 2026
T3 B (H AA (H' + (H') + (H' (A'TA") + + ( (G)) pl:
(31)
where equality (a) is obtained due to the fact that for an arbitrary matrix X, we have (VS(W"), LXR) r (H', x) T3 can be bounded as follows:
(4)
Bn //
G + (GA)T(A -A)) R| 2 ||L
T3 -
+
2
23
(b)
81)
|VS(
2
|I(A ||AA*-AA)GG
A [
2
8
(
31
|VS(W)))
)1² + 21
||
A
1l IG's |P
2
(
|V/(>)]
>1 + 2n
A - A
Σ REN
E
(W'.
2
BN2
81
+ 2mp
|| A A 1l
3N
31
[v/(w)) 11? + 2np
(S ) À |1
2
3N
8n
+ 20p
|//(>)|
| - 11 A
3N
31) V/((9))??
2mp C2 A 1+1
(32)
2
NNamin
A AA
where inequality (a) is obtained due to the fact that 2(X, Y) S +IYI1 holds for any O. Inequalities (b) and (d) result from Jensen's inequality. Inequalities (c) and (f) obtained by using the fact ||AB| F < | Inequality (e) results from Assumption 2. Inequality (g) results from Assumption 3 and Lemma 2. Then, we further bound the second term of T₁
TyLE [(H (A')TA* A (H' + (H')T) + (H' + (HH)) (A')TA)))
(a)
AmAmin ((A')TA') (W')
(33)
where inequality (a) results from Lemma 1. Then, we bound T (G,) rl as follows:
, E H (GA)
H
E
GG
H
IAS
|
+ 1
6
2
F
2
1)2
n766² 2
IA
(L)V
(34)
2
where inequality (a) follows from Cauchy-Schward inequality. Inequality (b) results from the fact
that (I( T < || I| Inequality (c) is obtained by using Assumption 2. Inequality (d) follows from Young's 's inequality.
By combining inequalities
33 and 34
we have
Ti <
2 A 1+1
A
proc
+ 4mjmmi ((A))AA))
+
n
+
3n 2
||//(>>)||
n + L
8Nomin min A (
2
2
(35)
17

---

Published as a conference paper at ICLR 2026
Now, we bound T2. In particular, it satisfies
T2
I
17 ((()))
)
(a)
3ry
[ G + +6/ [IL
(b)
3n 3n
'E
[[(GAA] | B [(AA)
(c)
S 3r 'E
||G*
+ E
|| E |
(d) S 3r/² (1/'+ 20
(36)
where inequality (a) is obtained by using Jensen's 's inequality. Inequality (b) is obtained since L and R have orthogonal columns. Inequality (c) results from the fact || S || l| [B]] Inequality (d) is obtained by using Assumption 2 and Assumption 3.
Then, we combine inequalities (29). (35). and (36) we have
+ s(W*+) S(W")
Bn
4mpmmin ((A)TA*)) (ACT))IIISSS |I()S(")))
2
2
u2
3L (1)2t= 202)
A proc
+
+
(37)
2
2
BNomin
A (At)T) (A+I(A))T)
We denote
Ω =
4n minmee (Amin ((A')TA*)
8n 2
We choose
3
2
-
9² 2. minteT { Ammin ((A))TA')) Then, we have f2 2n mimmee7 {Amin ((A))TA))}
By rear-
ranging inequality 37 and summing over all T fine-tuning rounds, we have
12662
3L 2 (122 + 2C
Ω IET
|V S -(((")-((W+) +
-
2
[+1 mp
A
proc
ET 2N minieT ( (Aiin ((A2)T } (A+I(AA)]
22262
11
3L (126 202)
/ /(W - f(W") +
2
AFLI A (AA))
C EIET
(38)
2N minier Aumin ((A))AA)))
where inequality (a) is obtained due to the fact that f(W))
f(WT+!). If
4 min (Amin (( At)]. have Ω O. Then, multiply on both sides of inequal- ity 38 and obtain
|/ f(W2) - f(W))
122
3LT (n 22
T IET
TΩ
2Ω
20
AT-L
C À ΣLet (A+T)
(39)
minRE (Amin ((At)TAP) AT)]
This completes the proof of Theorem 2.
A.6 HYPERPARAMETER SETTING
In our experiments, the batch size set to be 4 by searching over a range of {2, 4, 8, 16). Each local training epoch is set be 1. We select the optimal learning rate from the range {5e 4, le
18

---

Published as a conference paper at ICLR 2026
2.5
100
FLORG FeDERA
RedEx-LORA FeRS
FFA-LORA
FealT
90
Training Loss
Testing Accuracy
80
70
60
FLORG
FeDeRA
FedEx- LoRA
FFA-LORA
FedSA-LoRA
50
150
200
200
Fine-tuning Rounds
Fine-tuning Rounds
(a)
(b)
Figure 2: Comparison of the (a) training loss and (b) testing accuracy versus the number of fine-tuning rounds on RoBERTa-large and QNLI
Table 8: Comparison of the per-round client-side computation overhead (in FLOPs), per-round server-side computation overhead (in FLOPs), and memory usage (in MB). Symbol "-" means that the overhead is negli- gible.
Metrics
FLoRG
FedIT
FeDeRA
FFA-LoRA
FedSA-LoRA
FedEx-LoRA
Client-side FLOPs
1.18 x 10
1.24 x 10
1.24 10
8.25 x 10"h
1,24 10
1.18 x 10
Server-side FLOPs
8.50 x 1020
2.22 x 10
2.09 x 10
Memory Usage
2815
2117
2117
2110
2117
3117
5e 5, 5). The maximum sequence length is set be 128, following the common practice in fine-tuning, We choose AdamW as the optimizer. We set the scaling factor of the LoRA module be 16 for all algorithms.
A.7 ADDITIONAL EXPERIMENTAL RESULTS
Convergence curve: Fig. reports the training loss and testing accuracy versus the number of fine-tuning rounds on RoBERTa-large with the QNLI dataset. We observe that FLoRG consistently converges faster than the five baselines: it achieves lower training loss throughout the training process and reaches a higher testing accuracy with fewer rounds. This indicates that the proposed Gram-matris aggregation with the Procrustes alignment stabilizes the optimization trajectory across fine-tuning rounds, leading to both improved convergence speed and better final generalization per- formance.
Computation overhead and memory usage: Table X compares the per-round computation over- head (client-side and server-side FLOPs) and the memory usage. On the client side, FLoRG incurs 1.18 x 10 FLOPs per round, which is comparable to and matches FeAEX-LORA (1.18 1022), while FFA-LORA achieves lower client-side FLOPs (8.25 x 1) due to freezing one low-rank factor. On the server side, FLoRG introduces non- negligible overhead (8.50 1000 FLOPs) because it performs eigendecomposstion and Procrustes alignment after Gram aggregation; nevertheless, its server-side overhead is substantially lower than FeDeRA (2.22 x 10t1 FL.OPs), which relies on SVD-based decomposition. In terms of memory usage, FLoRG uses 2815 1515M MB, which is higher than most baselines that only maintain two low-rank matrices (around 2110-2117 MB), but remains comparable scale and lower than FedEx-LoRA (3117 MB).
(1.24 24 x 1022)
Question answering performance on SQuAD v1.1: Table 9 reports the exact match (EM) and Fl scores SQuAD v1.1 under three base models. FLoRG achieves the best performance across all models and datasets. For OP5-125M, FLoRG improves EM//II to 68.52/78.23, outperforming the strongest baseline (e.g., FedSA-LoRA) by 1.38 EM and 0.92 F1. For RoBERTa-large, FLoRG
19

---

Published as a conference paper at ICLR 2026
Table Comparison of the exact match (EM) and Fl scores SQuAD vl.1.
Base Model
Metric
FLoRG
FedIT
FeDeRA
FFA-LORA
FedSA-LORA
FedEx-LoRA
EM
68.52
62.31
64.78
63.19
67.14
66.85
OPT-125M
F1
78.23
72.08
74.52
73.04
77.31
76.18
EM
86.34
81.53
83.12
81.69
84.47
86.02
RoBERTa-large
F1
91.15
88.09
90.03
88.68
90.23
90.84
EM
89.83
84.21
86.58
85.37
87.12
88.25
Llama-3.2-3R
F1
92.74
89.53
91.08
90.31
91.35
91.82
Table 10: Additional comparison of the testing accuracy across different baseline schemes.
Base Model
Dataset
FLoRG
FedIT
FeDeRA
FFA-LORA
FedSA-LoRA
MRPC
88.30
82.09
85.70
84.78
86.51
OPT-125M
QQP
89.72
84.18
87.09
86.30
87.92
MRPC
90.80
85.16
88.30
87.50
89.21
RoBERTa-large
QQP
91.52
87.89
89,30
88.51
90.01
reaches 86.34 EM and 91.15 FI, slightly exceeding the best competing method (FedEX-LoRA) by 0.32 EM and 0.31 FI. For Llama-3.24 3.2-BB, FLoRG obtains 89.83 EM and 92.74 F1, improving over the strongest baseline (FedEx-LoRA) by 1.58 EM and 0.92 F1. These results demonstrate that FLoRG generalizes beyond GLUE classification tasks and is also effective for extractive question and answering.
Additional GLUE results: To complement Table T in the main content, Table 10 further reports the testing accuracy on MRPC and QQP datasets. FLoRG consistently outperforms the baselines on both OPT-125M and RoBERTa-large. In particular, on OPT-125M, FLoRG improves MRPC and QQP to 88.30 and 89.72, exceeding FedSA-LoRA by 1.79 and 1.80, respectively. On RoBERTa- large, FLoRG achieves 90.80 (MRPC) and 91.52 (QQP), exceeding FedSA-LofE by 1.59 and 1.51, respectively. This reinforces the effectiveness of FLoRG across different GLUE tasks.
Additional rank study: Table 11 provides more detailed results under different ranks (r {2, 4, (.)) on MRPC, QQP, MNLI, and QNLI (RoBERTa-large), FLoRG remains the pop-performing method under all rank settings. Moreover, increasing generally improves the testing accuracy for all meth- ods, while the performance gap between FLoRG and the baselines persists, indicating that FLoRG robust to rank choices and can better exploit higher-rank adaptation capacity when available.
Robustness to data heterogeneity: Table 12 reports additional results under different degrees of non-IIDness controlled by Under highly heterogeneous settings (e.g., P = 0.1), FLoRG consis- tently outperforms the baseline schemes MIRPC/QQPINNJ/QNLI When the data distribution becomes less heterogeneous (p = 0.5). FLoRG still maintains consistent improvements over com- peting methods. In the near-IlI case results on MRPC/QQP/MNLI, while FedSA-LORA slightly surpasses FLoRG on QNLI. Overall, these results validate that FLoRG is robust across wide range of client data heterogeneity levels, with particularly strong advantages under more heterogeneous federated settings.
1), FLoRG remains competitive and achieves the best
A.8 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (e.g., ChatGPT) for the general purpose of writing assistants to revise and polish the manuscript's prose: improving grammar, wording, clarity, and fixing mi- nor LaTeX/formatting issues. The models were not used for research ideation, designing algo- rithms/experiments, deriving proofs, selecting citations, writing substantive technical content.
20

---

Published as a conference paper at ICLR 2026
Table 11: Additional comparison of the testing accuracy under different ranks.
Rank
Dataset
FLoRG
FedTT
FeDeRA
FFA-LoRA
FedSA-LoRA
MRPC
85,91
79.66
83.17
82.32
83.00
QQP
87.62
81.88
84.90
84.09
85.65
MNLI
86.14
80.70
83.44
82.64
84.22
QNLI
89.70
86.67
87.22
86.40
88.10
MRPC
90.80
85.16
88.30
87.50
89.21
QQP
91.52
87.89
89.30
88.51
90.01
MNLI
91.39
84.76
88.20
89.12
90.90
QNLI
92.44
87.63
89.91
90.82
91.69
MRPC
91.70
85.55
87.12
88.20
89.91
QQP
91.80
86.39
89.26
88.45
90.10
==8
MNLI
91.70
86.40
89.29
90.44
90.41
QNLI
92.50
87.77
90.20
90.41
91.13
Table 12: Additional comparison of the testing accuracy under diffferent degrees of data heterogeneity.
Non-IIDness
Dataset
FLoRG
FedIT
FeDeRA
FFA-LoRA
FedSA-LoRA
MRPC
85.06
78.19
81.91
81.00
83.01
QQP
86.95
78.80
84.15
83.21
85.00
0.1
MNLI
81.35
75.44
78.42
77.57
79.36
QNLI
88.88
83.31
86.20
85.33
87.19
MRPC
90.80
85.16
88.30
87.50
89.21
QQP
91.52
87.89
89,30
88.51
90.01
0.5
MNLI
91.39
84.76
88.20
89.12
90.90
QNLI
92.44
87.63
89,91
90.82
91.69
MRPC
90.76
85.77
89.26
88.91
88.20
QQP
91.88
88.61
89.93
89.37
90.39
p=1
MNLI
91.75
86.47
90.05
91.70
91.33
QNLI
92.17
88.03
90.52
90.79
92.72
21


Reviews:
[{"review_id": "9ua8TroiqW", "reviewer": "Reviewer_srrB", "summary": "This paper addresses the issue that the federated aggregation of LoRA updates may not accurately reflect the intended global aggregation result. To this end, the authors propose FLoRG, which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix. Extensive experiments demonstrate its superiority over the existing works.", "questions": "**See weaknesses.**", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a well-structured theoretical analysis with formal proofs, offering strong theoretical soundness and clear convergence guarantees.\n2. It explores an interesting and under-studied problem—eliminating aggregation bias and decomposition drift in federated LoRA fine-tuning—introducing new insights into parameter-efficient federated learning.\n3. The paper is clearly written, correctly annotated, and provides a thorough description of the proposed FLoRG framework, making it easy to follow and reproducible.", "weaknesses": "1. The paper does not address the partial client participation scenario, which is common in practical federated learning settings. Evaluating FLoRG under varying client availability would strengthen its applicability.\n2. The experiments are conducted only on OPT-125M and RoBERTa-large, which are relatively dated compared to current state-of-the-art LLMs such as LLaMA-3 and Qwen-2.5. Using more recent backbones would better demonstrate the scalability and relevance of FLoRG.\n3. The paper reports final accuracy and communication cost but does not include convergence curves showing performance versus communication rounds. Such a figure would provide clearer insights into the training dynamics and stability of FLoRG compared with baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64TGpiJLn9", "reviewer": "Reviewer_F3oY", "summary": "This paper proposes FLoRG, a framework for federated fine-tuning of LLMs using Low-rank Adaptation. The authors point out two primary challenges with existing federated LoRA methods:  Aggregation Error (caused by naively aggregating the LoRA matrices B and A separately), Decomposition Drift (caused by the fact that there is not unique decomposition matrix) \nThe authors tackle these challenges with a two-part solution:\n\n* Gram Matrix Aggregation: Instead of LoRA's two matrices (B,A), FLoRG uses a single trainable low-rank matrix A and utilizes existing linear algebra techniques to convert the A matrix to the original dimension of the $\\Delta W$. \n\n* Procrustes Alignment: The server decomposes the aggregated weights, it performs a Procrustes alignment step. This solves an optimization problem to find an orthogonal matrix that best aligns the new A~t+1 with the matrix from the previous round At, thereby minimizing the \"decomposition drift\".\n\nThe paper provides a theoretical convergence analysis showing that the Procrustes alignment step results in a tighter convergence bound. The authors also empirically show that their method outperform four baselines on GLUE benchmarks.", "questions": "* Information about the setting is missing. For example, what is the parameter for different levels of heterogeneity? How did you do LDA for datasets without labels? \n\n* What is the federated learning setting, how many clients participate each round?\n\n* Did you do hyperparameter search? \n\n* Are the results averaged for different random seeds or they are done only for one seed?\n\nPlease also check the weakness section.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written. The authors did a good job categorising and explaining the existing problems. \n* The algorithm performs better than the mentioned baselines. \n* The authors provide a convergence analysis for FLoRG.", "weaknesses": "* Clarity on Communication Saving. I would appreciate it if the authors explained the communication saving part of their claim. Did they measure the communication compared to full matrix communication or other Federated LoRA methods?\n\n* Server-Side Computational Overhead: The paper does not discuss the server-side computational cost, which appears to be substantial, especially doing matrix decomposition and solving optimization. \n\n* The baselines are considerably basic. By just checking recent ACL and ICML conferences, I found recently accepted papers on Federated LoRA. The merits of the paper is not clear for me considering it is missing several works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCoXj5khd7", "reviewer": "Reviewer_fXbx", "summary": "FLoRG replaces the usual LoRA two-factor update BA with a single low-rank matrix A and updates the model via a Gram matrix aggregation. Clients locally SGD-update A; the server linearly aggregates n∑An⊤An, then eigendecomposes the aggregated Gram and applies a Procrustes alignment to pick a decomposition closest to the previous A, which stabilizes directions and enforces a target rank. The authors prove a non-convex convergence bound in which Procrustes alignment cancels a “drift” term, and empirically report higher GLUE accuracy vs. FedIT/FeDeRA/FFA-LoRA/FedSA-LoRA with up to 82% fewer transmitted parameters to reach target accuracy.", "questions": "1. How robust is FLoRG if L,R are learned (slowly) or adapted per layer/round? Can you provide theory/ablation for updating L,R versus keeping them fixed?\n2. What are per-round costs of eigendecomposition + Procrustes across all LoRA layers at N>100 clients?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Bias-free aggregation with one matrix.\n2. Convergence bound tightens when alignment is used; ablations show sizeable accuracy gains from Procrustes; headline comms savings to target accuracy.", "weaknesses": "1. The approach relies on semi-orthogonal L,R that never update; performance is sensitive to their initialization.\n2. Each round per layer requires eigendecomposition of Q and an SVD for Procrustes; scalability or latency with many layers or clients isn’t benchmarked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "352RLntfMi", "reviewer": "Reviewer_kKoy", "summary": "The paper proposes FLoRG, a novel federated fine-tuning framework for large language models (LLMs) that improves upon the conventional LoRA-based approach. Traditional federated LoRA uses two separate low-rank matrices, which can lead to aggregation errors and decomposition drift during global model updates. To address these issues, FLoRG employs a single low-rank matrix and aggregates its Gram matrix to eliminate aggregation errors and reduce communication costs. Additionally, the authors introduce a Procrustes alignment mechanism to align decomposed matrices between rounds, thereby mitigating decomposition drift. Theoretical analysis demonstrates that this design achieves a tighter convergence bound. Extensive experiments on multiple LLM fine-tuning benchmarks show that FLoRG achieves higher downstream task accuracy and reduces communication overhead by up to 82% compared to state-of-the-art baselines.", "questions": "1. How does the proposed method perform on more general tasks such as question answering and dialogue?\n2. The performance of the proposed method when applied to more popular and larger language models, such as LLaMA?\n3. The local training stage of the proposed method is interesting. It seems applicable to centralized learning as well. How does it perform under centralized learning? Compared with centralized learning, what advantages does it provide in the FL setting?\n4. It is unclear whether updating only module A is sufficient to learn good representations of the client data. Could the authors clarify or provide evidence?\n5. Could the authors provide details on how the dataset was used?\n\nIf the authors can adequately address my concerns, I am willing to increase my rating.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide a theoretical convergence analysis and rigorously show that the proposed Procrustes alignment leads to a tighter convergence bound, which enhances the credibility of the method.\n2. Experimental results on multiple LLM fine-tuning benchmarks demonstrate consistent improvements over several state-of-the-art baselines in both accuracy and communication efficiency.\n3. The paper is well written and easy to follow.", "weaknesses": "1. The experiments are limited to natural language understanding tasks with relatively small models. It would strengthen the paper to include evaluations on larger models or additional task types to demonstrate broader applicability.\n2. Updating only the low-rank matrix  $A$ may limit the model’s representation capability, potentially constraining its ability to adapt to more complex tasks.\n3. The paper lacks an analysis of efficiency in terms of computational cost, memory usage, or communication overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)