Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
Authors: Jin Liu
Yinbin Miao
Ning Xi
Junkang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning
Differential Privacy
LoRA.
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private federated learning (DPFL) is hindered by a fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to performance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous update of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model in the parameter space. To address these issues, we propose LA-LoRA (\textbf{L}ocal \textbf{A}lternating \textbf{LoRA}), a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA-LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example, when fine-tuning the Swin-B model on the Tiny-ImageNet dataset under a strict privacy budget ($\epsilon = 1$), LA-LoRA outperforms the best baseline, RoLoRA, by 16.83\% in test accuracy. Code is provided in the Appendix.

Published as a conference paper at ICLR 2026
RETHINKING LORA FOR PRIVACY-PRESERVING FED- ERATED LEARNING IN LARGE MODELS
Jin Liu's 1 Yinbin Miao', Ning : Junkang Liu?
School of Cyber Engineering, Xidian University
2 College of Intelligence and Computing, Tiajjin University
(jinliu9787, junkangliukk (egmakkkkkkkmmiik com, {ybmiac, nxi]exidian. edu.
ABSTRACT
Fine-tuning large vision models (LVMs) and large language models (LLMs) un- der differentially private federated learning (DPFL. is hindered by fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter- efficient fine-tuning (PEFT) method. reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to perfor- mance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous up- date of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model the parameter space. To address these issues, propose LA-LoRA (Local Alterating LoRA), a novel approach that decouples gradient interactions and aligns update directions across clients enhance robustness under stringent pri- vacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA- LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example. when fine-tuning the Swin-B -B model on the ImageNet dataset under strict privacy budget (c = 1). LA-LoRA outperforms the best. baseline, RoLoRA, by 16.83% in test accuracy. Code is provided https: / / thub. http://ccoonjnnniaio.mm
INTRODUCTION
As foundation models such as GPT (Achiam et al., 2023), BERT (Devlin et al., 2019), and VIT (Dosovitskiy al., 2020) scale in size and capacity, adapting them to downstream tasks in- creasingly relies on vast and heterogeneous datasets. However, centralized collection is becoming impractical due to data silos and rising concerns over user privacy (Liu et al., 2024b;a). Federated learning (FL, McMahan et al. (2017)) offers driaacyppreeervigg solution by enabling decentral- ized model training without sharing raw data across clients.
Despite this promise, applying large-scale models in FL remains major challenge. These models typically consist of billions of parameters, making full-model fine-tuning computationally expen- sive and communication-Iess ods, such as Low-Rank Adaptation (LoRA, Hu et al. (2021)), have been integrated into FL. Ap- proaches like FedIT (Zhang al., 2024a) freeze the majority of the model weights and only update a lightweight set of LoRA parameters, reducing communication overhead to less than 0.1% of the full model while preserving performance (Xie al., 2026; Ouyang al., 2024; Ke et al., 2025; Qiu et al., 2025; Zhou et al., 2025a;b; Wang et al., 2025; Wu et al., 2025; Qi et al., 2025a; Zhao et al., 2026; Bellavia al., 2024: Sunmola al., 2025: Yan al., 2025; Edstedt et al., 2024; Meng al., 2025; Qi et al., 2025b; Feng al., 2026; 2024a; Zhou et al., 2025c).
To mitigate this, parameter-effictent fine-tuning (PEFT) meth-
However, privacy remains a critical concern in FL. Although raw data not directly shared, transmitting gradients model updates can still expose sensitive information. Prior work has shown
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Foree
Pretrained Weights
Pretrained Weights W e 1d
Pretrained Weights
8
Pretrained Weights
==0
N10,0
Cureett step
Next step
( DP-LORA
(b) FFA-LORA
(c) RoLoRA
(d) LA-LoRA
Figure 1: The illustration of DP-LoRA, FFA-LoRA, RoLoRA, and LA-LORA. DP-LoRA updates both noisy A and B simultancously and sends them the server for aggregation. FFA-LoRA freezes A, updates only the noisy B, and sends it to the server. RoLoRA alternately updates noisy A and B across rounds. Our LA-LoRA alternately updates noisy A and B within each local round.
that adversaries may reconstruct private data from such gradients (Mu al., 2024b;a; Xiao et al., 2024). To provide formal privacy guarantees, differential privacy (DP, Dwork (2006)) is com- monly integrated into federated optimization.
While differentially private federated learning (DPFL) with PEFT methods (e.g., DP-LoRA (Liu et al., 2025e)) provides privacy guarantees, often incurs reduced performance due to federated constraints. Federated training introduces additional complexities, including non-iid data distribu- tions, noisy updates, and difficulties in aggregating LoRA modules. In practice, LoRA-based fine- tuning under privacy constraints often suffers from severe performance degradation (Feng al., 2024b; Liu et al., 2024b;a; 2025c;a;;;d; 2026). Beyond the prior finding of DP noise amplification FFA-LORA, we identify two further challenges, leading three key issues of LoRA in DPFL
Gradient coupling. Simultaneous updates to LoRA's asymmetric matrices cause gradient inter- ference, destabilizing training, especially under DP noise and non-iid distributions.
Amplified DP noise, LoRA's qumi-quadaatic update structure amplifies the impact of injected DP noise, severely degrading learning stability.
Sharp global solutions. LoRA's low-rank constraints reduce client capacity, often resulting in sharp global minima after aggregation, compromising robustness and generalization.
To overcome these, we propose LA-LoRA, a novel framework built on structural rethinking of how LoRA fits into DPFL. Its core local alternating update mechanism that decouples gradient interference and suppresses DP noise. To further enhance stability and generalization, LA-LoRA incorporates a simple yet effective low-pass smoothing filter. We provide theoretical evidence that update scheme ensures stable optimization while preserving the low-rank structure of LoRA. Extensive experiments on both image classification and language understanding tasks validate the effectiveness of LA-LoRA. Results show that LA-LoRA achieves strong performance while offering privacy protection and optimization stability. Our contributions are summarized as follows:
We identify overlooked challenges of applying LoRA in DPFL: gradient coupling and ag- gregation sharpness. Along with noise amplification, these motivate the design of our algorithm.
We propose LA-LoRA, novel algorithm that alternates local updates of LoRA components to mitigate gradient interference, noise amplification, and aggregation instability. A low-pass smoothing filter further enhances cross-client consistency and stability.
Theoretically. we prove that alternating updates yield unique closed-form solutions and ensure stable low-rank optimization, mitigating projection distortion federated settings.
We validate our algorithm both vision (Swin Transformer) and language (RoBERTa) models, covering image classification and NLP tasks. LA-LoRA outperforms SOTA privacy- preecy-prsservigg federated LoRA methods various tasks and demonstrates significant performance gains.
2 BACKGROUND AND RELATED WORK
Definition 1 (DD))) Let e > and < 1. A random mechanism M X O satisfies (c, 8)DP if. for any two neighboring datasets 'D and D' differing in sample, and for
2

---

Published as a conference paper at ICLR 2026
any measurable subset U _ O of possible ouppus,,
Pr(M(D) E E S et Prrr((DD)) E Eu] +
(I)
Differentially private federated learning (DPFL). A general FL system with central server and N clients. Each client holds a local dataset D; and performs local training on it. The server aggregates the model updates from all clients and updates the global model W. Our privacy goal is sample-level protection (changes to any single data sample should not significantly affect the output). Building Noble al. (2022), we provide (e, 8)DD guarantees. We perform privacy accounting via Rényi DP (RDP, Mironov (2017)), composing per-step privacy loss for the subsam- pled Gaussian (Wang et al., 2019) with per-sample clipping. and finally convert (e, 8)-P
To achieve DP in FL, each client i clips the per-sample gradients to fixed lg norm to bound sensitiv- ity, then adds Gaussian noise the aggregated clipped gradients locally, protecting the contribution of individual training examples. For local mini-batch B;, the privatized update is computed as:
9ig = / max (1, (i Vj B,,
9. = - E jEB, 96t + +N (O, C² /|B.). (2)
where Lúj denotes the loss for sample j on client C' the clipping norm bound, and or is the Gaussian noise multiplier determined by the privacy budget (e,
Parameter-efficient fine-tuning (PEFT). The rapid scaling of modern pre-trained models has led to an increased demand for fine-tuning in resource-constrained environments. PEFT tackles this by training a small set of parameters while freezing most backbone weights, with representative methods including adapters (Houlsby et al., 2019). prefix-tuning (Li & Liang, 2021), prompt- tuning (Lester et al., 2021), LoRA (Hu et al., 2021; Tian et al., 2024) and other methods (Shin al., 2023; Zheng et al., 2026). LoRA is partieularly popular for its simplicity and strong perfor- mance under minimal parameter overhead: it augments a pre-trained weight matrix Wo E mxx with two low-rank matrices, an up-projection B Rmxr and down-projection
E Rrx= with r & min {m, n). and reparameterizes the weights as W = Wo + +BA A while keeping Wo fixed. B is initialized a zero matrix to suppress early updates, while A uses a random Gaussian initialization.
PEFT in FL. PEFT in FL has been explored to reduce communication overhead, computational cost, and privacy risks. Accordingly, various strategies have been proposed, such as adapter-based (Cai et al., 2023; Ghiasvand et al., 2024), prompt-based (Zhao et al., 2023; Qiu et al., 2023), and selec- tive tuning approaches (Yu al., 2023). Recently, LoRA-based methods have received increasing attention in FL. FedIT (Zhang et al., 2024a) directly incorporates LoRA into the standard FedAvg framework for instruction tningg RoLoRA (Chen et al., 2025) addresses heterogeneity by alter- nately optimizing A and B across communication rounds. FedSA-LORA (Guo et al., 2025) uploads only to the server for aggregation, keeping B local to support personalized adaptation. Other works explore heterogeneous configurations (Cho et al., 2024; Wang et al., 2024) or personalized decomposition (Qi et al., 2024) to improve adaptability under system and data heterogeneity.
LoRA in DPFL. In DPFL, LoRA-based methods typ- ically apply DP mechanisms to the low-rank matrices A and B, rather than the full model parameters, which significantly reduces computational and communication overhead. DP-LoRA (Liu al., 2025e) computes per- sample gradients for A and B locally, clips them, adds Gaussian noise, and sends privatized updates for aggre- gation. FFA-LoRA (Sun et al., 2024) freezes the down- projection matrix A and trains only the up-projection B, injecting calibrated noise into its updates. This design avoids noise amplification, but at the cost of reduced model expressiveness. Despite recent efforts, differentially private federated LoRA methods still suffer from noticeable performance degradation and remain underexplored. Figure 1 shows the differences in the ideas of existing methods, while Table compares their actual characteristics.
Table I: Comparison LoRA-based meth- ods in DPFL. "V" denotes support, "X" notes not considered. "Exp." indicates the effective expression ability under DP.
Method
DP LVMs Exp. Speed
DP-LoRA FFA-LORA RoLoRA
mid slow
low
slow
X mid
mid
LA-LORA
high fast
3 LIMITATIONS OF LORA UNDER PRIVACY-PRESERVING FL
Although LoRA is widely used for efficient adaptation, its behavior under differential privacy is still poorly understood. We show that DP interacts with the low-rank parameterization and causes failure
3

---

Published as a conference paper at ICLR 2026
modes that are further amplified in federated learning due to data heterogeneity and noisy local updates. We therefore focus on DPFL and include centralized DP experiments in Appendix B.6.
3.1 GRADIENT COUPLING IN ASYMMETRIC UPDATES
We find that simultancously updating the two LoRA matrices A and B leads to intrinsic gradient coupling that destabilizes training under DP noise and data heterogeneity. In LoRA, A E R'*" projects the input into lower-dimensional subspace, while B R maps this low-rank rep- resentation back to the original output space. Their asymmetrie roles give rise to distinct gradient behaviors. More precisely, the gradients of the loss L with respect A and B are interdependent:
VaL = sB' (VwC), V = <VW
Thus, the update direction for A (resp. B) is re-parameterized by the current B (resp. A). When both matrices are updated in the same step, the latent basis defined by A can shift while B adapts to outdated directions, creating representation drift and mismatch between the two gradient path- ways. In the presence of DP perturbations and non-iid data, this coupling makes the trajectory more sensitive to perturbations and client drift, leading to oscillations and unstable convergence.
To quantify this coupling, we compute the cosine similarity between V and VgL during each training step under the experimental setup described in Section 6. As shown in Figure 2(a), DP- LoRA (simultaneous updates) maintains persistently lower similarity, reflecting strong coupling and mismatch. LA-LoRA(-fiteen (local alternating updates) achieves much higher similarity. This align- ment difference manifests in training dynamics and final performance: Figure (b) shows that the simultancous update of and B in DP-LoRA leads to a higher test loss, whereas LA-LoRA(-fltee achieves smoother convergence. Figure 2(c) further indicates that this coupling harms test accu- racy. We observe similar trends in a non-private federated setup (Appendix B.7), which suggests that gradient alignment is inherently beneficial, while DP noise further amplifies the coupling issue.
Casine Sinileity
0.6
DP-LORA
filter)
OPLLRRA
Test Rocuussa (hi)
0.5
Test Loss
40
0.4
0.3
LoRA
0.2
200
400
600
800
1000
500
1000
1500
2000
500
1000
1500
2000
Treining Steps
Traiing Staps
Traiiing Steps
(a) Cosine Similarity.
(b) Test Loss.
(c) Test Accuracy.
Figure 2: Comparison of cosine similarity between A and BLC, test loss and test accuracy for Swin-T on CIFAR-100 (e = 3). LA-LoRA(- uses local alternating updates without smoothing.
3.2 STRUCTURAL AMPLIFICATION OF DP NOISE
Differential privacy injects noise into local sample gradients to protect client data. Noise added independently to A and B. We ignore the LoRA scaling factor s. For client i, the resulting noisy low-rank term can be written as:
(B, (A; NA.) B,A, B,NA, + NN -N,
where NA. and NB. are Gaussian noises independently sampled per client i. This decomposition reveals that the update A, is perturbed by three terms: A. A BB, A, and NB.NN The first two linear in the noise, while NA, is non-Gaussian and introduces quadratic amplification effect. This structural cascade increases the variance of the aggregated updates and hinders convergence.
We consider a synthetic example. In our setting, W I 1024x 1024 with the dataset QNLI and other configurations As the Gaussian noise scale a
1500
described in Section changes, we report the Frobenius norms of the induced pertur- bations. Figure indicates that for small o, LoRA (II.BAA + BNA +NBn Allz) incurs smaller perturbations than full-model ([W updates. As a increases, the multiplicative term ([ grows quadratically and becomes the leading contribution, causing the total LoRA perturbation to exceed the full-model curve.
Noise Norm
1000
500
8.0
Figure 3: Scaling of perturbation Frobenius norms with o on QNLI.
4

---

Published as a conference paper at ICLR 2026
3.3 SHARPNESS IN GLOBAL AGGREGATION
A third distinct challenge arises after global aggregation. It is well established that the geometry of the loss landscape critically affects generalization. Convergence flat minima, regions with low curvature, has been shown to promote robustness and better generalization, while sharp minima correlate with overfitting and instability (Hochreiter & Schmidhuber, 1997: Kaddour al., 2022).
We observe that this issue is aggravated in LoRA-based DPFL. In parameter space, each client con- tributes low-rank factors A, and B, and the server aggregates them under naive FedAvg. Unlike farlmperreeeer updates, low-rank factor aggregation alters the geometry of the global update in pa- rameter space. Specifically, heterogeneous factor directions across clients do not align, and their composition produces global updates that consistently land in narrower, high-curvatrre regions of landscape. The problem exacerbated by DP noise, which injects stochastic perturbations that destabilize the already misaligned updates.
Figure visualizes the loss landscapes of Swin-T trained with DP-LoRA and LA- LoRA(-filter) under c = 1. DP-LoRA pro- duces sharper and more irregular landscape, while LA-LoRA(-flter) yields a flatter and smoother basin. This suggests that simulta- neous updates in DPFL, may hinder general- ization by increasing the curvature the ag- gregated model. Table 2 further confirms this sharpness issue via Hessian eigenvalue anal- ysis. See Section 6 for experimental setup.
(a) DP-LORA, c=1
(b) LA-LoRA(-fllen)) e=1.
Figure 4: Comparison of global loss landscapes for fine-tuning Swin-T model on CIFAR-100.
4
OUR METHOD
We address the aforementioned challenges from two complementary perspectives. At the opti- mization level, break the tight dependency between the two low-rank factors that gradients are decoupled, cross-noise terms are avoided, and the update trajectory remains smoother. At the pre-aggregation level, further suppress residual variance by filtering high-frequency com- ponents of DP perturbations on each client before aggregation, effectively smoothing noise and improving generalization. Building on these ideas, develop LA-LoRA (Local Alternating Low- Rank Adaptation), illustrated in Figure 5. It combines (i) a local alternating update strategy for the first perspective, and (ii) an optional Gaussian low-pass filter for the second, jointly improving stability and consistency under DPFL.
Local Alternating Update
Filter
Server
Client
B.
Client
A
Client
Figure 5: Our LA-LoRA framework.
4.1 LOCAL ALTERNATING UPDATE STRATEGY
Instead of simultaneously updating both A and B. LA-LoRA adopts an alternating scheme, where the two low-rank matrices are updated in turn within each local training round. For client at local step of communication round we keep one factor fixed and update the other:
Update B; if k is odd, keeping A, fixed: Bi = B nB V C + +s AB
Update A; if l: is even, keeping B; fixed: Af, Al 7I A + +B
5

---

Published as a conference paper at ICLR 2026
Here, Wo denotes the frozen backbone from the server. L((-) is the local training loss client i's private dataset D, 7,, a are learning rates, and 8 is the LoRA scaling. Gradients VAC; and V VyL, taken with respect to the corresponding low-rank matrices.
This local alternating design addresses the three challenges outlined in Section 3:
Challenge 1 (Gradient coupling). By updating one matrix at time, the gradients become condi- tionally independent, which alleviates the tightly coupled dynamics described in Eq. (3):
V VBC. = s s(V
VAL: =B' (VwL..
(5)
Challenge 2 (Amplified DP noise). Since only one matrix is updated and privatized at each step, the multiplicative noise term NA, Eq. (4) eliminated. The perturbed update reduces to:
(B,+N2, ) B,A, A,, if A, is fixed, =
(6)
if B, is fixed.
Challenge 3 (Sharp global solutions). Local alternating updates constrain each step to struc- tured lower-dimensional subspace, the column space of A; or the row space of B, This implicit regularization suppresses sensitivity to stochastic noise and client heterogeneity, yielding flatter and more stable global solutions. We report the maximum Hessian eigenvalue, which characterizes the steepest curvature of the loss surface (Sagun et al., 2018: Grosse Martens, 2016). As shown in Table 2, LA-LoRA consistently achieves smaller eigenvalues than DP-LoRA across datasets and privacy settings, indicating flatter loss landscape and more stable training dynamies.
4.2 SMOOTHING WITH A LOW-PASS FILTER
To further improve training stability under DP, LA- LoRA introduces an optional low-pass smoothing filter applied to the LoRA gradients before aggre- gation. DP noise often manifests high-frequency perturbations, which destabilize local updates and amplify sharpness in global aggregation (Zhang et al., 2024b). Our goal to attenuate these high- frequency components via lightweight operation that leaves the model architecture and the privacy mechanism unchanged.
Table 2: Maximum Hessian eigenvalue of Swin- on CIFAR-I00 and TmageemageNet.
Method
CIFAR-100 100 ImageNma
DP-LoRA c=00
42.45
44,80
LA-LORA c=
30.12
33.25
DP-LoRA c=1
101.62
115.36
LA-LORA c=1
64.77
69.53
Specifically, we adopt fixed ID Gaussian kernel G. [1, 6, 1J. i.e., the standard 5-tap binomial low-pass filter. For A E RR*x, smoothing is applied row-wise along the input feature dimension; for B E Rmxr, smoothing is applied column-wise along the output dimension. The filter acts along meaningful input/output feature axes while keeping different low-rank components decoupled. Denoting by the ID convolution with symmetric padding, the filtered gradients are
-xC.bi-1= G. VAL.ii.), E [1,r), &eCile = G. VnLil: Vj E [1,r], (7)
From an optimization perspective, filtering noisy gradients with Ge can be interpreted as approxi- mately imposing miedimensional smoothness regularizer along the filtered dimension, discourag- ing abrupt changes between neighboring entries and inducing a low-pass effect on the update.
In practice, the Gaussian kernel G, reduces DP-induced fluctuations while preserving the structural semantics encoded in A and B. The operation stabilizes local updates and alleviates sharpness in global aggregation with negligible overhead, making it suitable for federated environments.
4.3 LA-LORA FRAMEWORK
Local client update. At global round t E [Z]] each selected client i fine-tunes only the LoRA factors (A, B) top of frozen backbone Wo via the reparameterization Wo + sBA. The factors are initialized as A + 4r1 and B + 1 The client performs K local steps. At each step E [K], it samples a mini-batch B, C D; of size [bR]. b is the local data sampling rate, R is the size of DD, The client performs alternating updates of B and A across local steps.

---

Published as a conference paper at ICLR 2026
Odd steps (update B). For each example j E B, compute the per-example gradient
J& = V (Wo + BBl.o+sBB.... ,Bi dy),
(8)
Perform per-example f2-norm clipping with threshold C: gij < ty// (10.11ll11///)) Aggre- gate over the mini-batch and inject Gaussian noise to ensure DP:
9i 1/(bR) E, JEB Iú +
(9)
Smooth the noisy gradient with fixed operator Go (low-pass filter) to obtain gi G. * gi, then
B -Bi,
1J8 Di,
Al, -
Even steps (update A). Repeat the same procedure for A with learning rate 7/4-
After K alternating steps, the client returns the locally updated (and smoothed) factors (A), B,) for server-side aggregation. Throughout, Wo remains frozen and only the low-rank adaptation BA modified; gradient clipping with Gaussian noise provides privacy, while G. suppresses high- frequency perturbations and stabilizes optimization.
Server aggregation. The server averages client uploads:
= 1/
Σ sEC.
A
B' = 1/IG NIGGEE. Σ
B
(10)
SEC.
and then updates the global model as W Wo sB' After T rounds, the final model is WT = Wo + sB" Here, C, is the set of participating clients at round with |C, = [qN], is
the client sampling rate and N the total number of clients.
THEORETICAL ANALYSIS
In the following, state the necessary theorems. Full theoretical details appear in Appendix Theorem 1 (Privacy guarantee). Following the privacy analysis in Noble et al. (2022), LA-LoRA ensures that after communicution rounds with K local steps per client, the weight matrix WT satisfies (e, 8)-P for any third party:
= CO(bTK Tolo/Kl (2/0) (og(//////)))
(11)
With respect to the server, after 'I' rounds, the accumulated privacy budget satisfies (E,, 8,)-DP,
6. = 8, = (1/
(12)
Noise is added to the clipped sample gradients before forming the client update, and the Gaussian low-pass filter is deterministic function of these noisy updates. By the post-processing invariance of DP (Dwork et al., 2014), this filtering step does not weaken the guarantees in Theorem
Theorem (Closed-form projected gradients). Let Bş E mmxr and A& gg be full rank, i.e., rank( B = rank( A&) = r, and let a/r > O denote the LoRA scaling. In LoRA, updates to By and Aş can be obtained by projecting the full gradient onto the column space of Aş and the row space of Bg, respectively. Within iteration k, we update B first and then A. This projection is formulated as a least-squares problem, whose unique solution yields:
1
BB. C = s2 (E,A(A
V.
(B (B+1) V.LC,
(13)
s2
where BB,C, V V, C are the gradients defined in Eq. (3). The projected gradients V. A obtained by solving the least-squares problem.
Under the full-rank assumption, Theorem yields closed-form projected gradients that use only the local parameter gradients A. C or B, C plus r solve. This avoids the full model gradient Vw L. In practice, the computation reduces forming the small Gram matrices A& or B Be+1 and solving small x system, which is lightweight when e min(m,n).

---

Published as a conference paper at ICLR 2026
Theorem 3 (Stable feature learning.. Assume that, for the input BAr has dimension O(n). In LA-LORA, if we use the learning rate = O(1) to update B and A, it achieves stable feature learning. Moreover, the model update achieves stable feature learning as well with
Wk+1 = W - VV - mProjc(B..)) (Ww.-f)).
(14)
where Proje(As) denotes the orthogonal projection onto the row space of As. and Proje(B...) denotes the orthogonal projection onto the column space of BE+1- Besides, m(V w(Vw.C) (PProrr(A2,,PProsssssss ( L)
0(1). However, when doing joint update, the
update will introduce additional across term
(BB Bx) (V C)A (ARA)) O(1).
The across term is indeed the second order term w.rl 1/ bui it is same magnitude as (Voj,(B.)(Ww.C)) and ny(V (v,,,,) (r))))) in infinite-width NN setting.
In Theorem 3, ours achieve stable feature learning. Moreover, as the joint update would introduce the cross term with an unignorable magnitude (especially n is O(1) instead of ((1nn), simultancous update with scaled gradient descent breaks the clean interpretation of projecting the full gradient onto low-rank subspaces and degrade the performance our experiment studies show later.
Theorem 4 (Convergence rate), Assume for any E [P] the matrix C₁ = D;X satisfies the rank r-RIP with constant d, (Assumption 1) and 0 S n S 1+ then LA-LoRA without momentum
solves the over-parameterized problem leads to
Auti) ) S (1 - 11) 2 Lel Bx, An),
(15)
P Σ
1 8,
P X.
BL
-X.
(1 nc) Σ
B
(16)
1- 8p
F
where ne = 2P 6,) (1) +1++++,++1)
Explanation of Theorem 4. If each C. D,X satisfies the rank-r RIP with constant 6, and the step size obeys 0 S S (1 6, + p) p)-1, then momentum-free LA-LoRA is contractive: the objective decreases geometrically as (B 1 S (1- 11.) L. (BN. A&), with T/c =
2P(1 6,) (17 (7- +((1+6,+ Consequently, the reconstruction Xx E/ A converges
2
linearly X, in Frobenius norm: || X < 1+ (1 ) 2* 10 X, where 1+4. 1-5.
reflects the RIP conditioning (smaller 6, gives tighter bound).
EXPERIMENTS
We evaluate LA-LoRA both vision and language tasks assess its effectiveness and privacy preservation. All experiments are performed on NVIDIA A6000 GPUs.
6.1 EXPERIMENTAL SETUPS
Datasets. For image classification, we use CIFAR- 100 and Tiny-ImageNet. For language under- standing, we evaluate four GLUE benchmarks: SST-2, QNLI, QQP, and MNLI.
Models. vision tasks, we employ Swin Transformer backbones (Swin-T and Swin-B), initialized from ImageNet-22K6 pre-trained weights, which well-suited to federated environments due to their strong generalization. For language tasks, we use RoBERTa BaBERTa-Base as the backbone model.
Baselines. We compare LA-LoRA with three SOTA approaches: DP-LoRA, direct application of LoRA under DPFL constraints. FFA-LoRA, which freezes while updating B. RoLoRA, which alternates the upload of A and B in communication rounds.
Hyperparameter settings. Image classification: Federated setup with N clients (sampling rate q -.55) and default non-iid Dirichlet / 0.1. Training runs for T 100 rounds, each selected
8

---

Published as a conference paper at ICLR 2026
client performing K 20 with batch size B 16. We use LoRA fine-tuning with rank 16, scaling factor a = 16, updating both adapter and classification head. Optimization uses SGD with learning rate decay À 0.99. The learning rate 77 is selected from {le - 2, 2e 22-,,
2, 1c 1, 2e 1). The privacy budget is fixed to e E {3, 2, }.. with = le - 5 and noise smoothing 0, = 0.01. Language understanding: Federated setup with N 20, q 0.2, Dirichlet 3 0.8. Clients run
K = 20 local steps for T' = 100 rounds (B = 16). We use LoRA with = = 8, freezing the classification head. AdamW optimizer with 17 [1e-4, 2,
4e-4, 1e-3, 2e 2e-2e-3.3 4e 33- le 5 for SST-2 and QNLI, 8 = le 6 The privacy budget is also set to E {3, 2, 1), with for QQP and MNLI, and smoothing ds 0.001. We run trials for vision tasks and 15 trials for language tasks, reporting the mean test accuracy across trials. Full details see Appendix B.2, C.2.
6.2 RESULTS
Fine-tuning for image elassification. Table and Figure 6 show that LA-LoRA consistently outper- forms DP-LoRA, FFA-LORA, and RoLoRA on CIFAR- 100 and Tiny-ImageNNe using both Swin-T and Swin-B, under privacy budgets e E {3, 2, 1}. For Swin-T e = 3, LA-LoRA achieves 60.07% (CIFAR-1000 and 60.97% (Tiny-ImageNet), exceeding RoLoRA by 4.88 % and 10.10%. Even at = 1, LA-LoRA remains the top performer with 56.68% and 60.01% on the two datasets.
A similar trend holds for Swin-B. At LA-LoRA reaches 74.56% CCIAR--10) and 60.68% (Tiny- meygeeeeeNet), outperforming RoLoRA by 6.68 % and 16.83% % respectively, further confirming its consistent superiority baselines. Further experimental details provided in Appendix B.3.
Aouuuay Sy
Teal Ausuraoy 1%6 22 2 35 2t
SouPooo 200
DP LeRA
[P] LeRA
1 2011
FRA LoRA
FRA LoR4
RoLoRA
RaLoRA
LA-LRA
L6-LoRA
Communicatin Rounds
Commuricaton Rounds
Communication Rounds
Communicatin Rounds
(a) CIFAR-100, Swin-T
(b) ImageNet, Swin-T
(c) CIFAR-100, Swin-B (d) ImageNet, Swin-B
Figure 6: Test accuracy of Swin-T and Swin-B on CIFAR-100 and Tiny- liny-ImageNet with 1.
Table 3: Test accuracy of Swin-T and Swin-B CIFAR-100 and Tmag-Im for different
Privacy Budget
Swin-T Model (%)
Swin-B Model (%)
Method
CIFAR-100
Ting-emageNet
CIFAR-100
Tiny- Net
DP-LORA
15.10
32.27
56.5220.51
30.6440.30
FFA-LORA
52.09 00.38
44.62
62.10.0.39
39.8440.19
<=
RoLoRA
55.19-0.42
40..6
67.96=2.64
44.1820.66
LA-LORA
60.47
60.97 #0.44
75.29 +0.35
61.97 20.56
DP-LORA
24.82=0.57
32. 14,14.1.3
56.31
30.31
FFA-LoRA
52.05
44.31 40.44
62.02
39.54
2
RoLoRA
55. 55.0220..5
50.5640.55
67 $7.93
43.97
LA-LoRA
59.52 20.03
60.63 20.4
74.93
61.03-20.67
DP-LORA
45.58 0..47
31.00.1.44
55.98
30,20
FFA-LORA
50.75
44.3840.55
61.9441.377
39.33
6=1
RoLoRA
D4.88
50.7840.45
67 67.88
43.8540.c0
LA-LORA
56.68100 10.60
60.01 10.51
74.5610.52
60.68 20.55
Fine-tuning for language understanding. Table summarizes the results, where LA-LoRA con- sistently outperforms all baselines across datasets and privacy levels. At e 3. LA-LoRA achieves 93.12% on SST-2 and 89.83% on QNLI. At e = 1, it maintains 85.34% on QQP and 82.35% on MNLI, surpassing the best baseline RoLoRA in all cases. More results in Appendix C.3.
6.3
ABLATION STUDY
To better understand the individual contributions of the local alternating update strategy and the low-pass smoothing filter, fixed 3 for ablation studies.

---

Published as a conference paper at ICLR 2026
Table 4: Test accuracy (%) of RoBERTa- Base on SST-2, QNLI, QQP and MNLI under different
Privacy Budget
Method
SST-2
QNLI
QQP
MNLI
DP-LoRA
92.36.10.75
86.31 20.32
84.5641..8
80.98 44
FFA-LoRA
92.32+0.40
87. 87.20
85.1244..34
81.71+0.69
= 3
RoLoRA
92.70+0.52
88. 0..29
85.35+4.42
82.12++.44
LA-LoRA
93.1220067
89.83
85.83
82.9910.42
DP-LORA
92.2010064
86. 20.57
84.266.0.57
80.6210.35
FFA-LORA
92.39+0.51
87.30 $7.30=0.00
84.73.+0.47
8 01.9440..58
RoLoRA
92.55+0.40
87.08 00
85. 85.02 02.0024..3
82.01 +0044
LA-LoRA
93.00
10.188.o +0.51
85. +0.55
82.87 +o0.22
DP-LoRA
90.71+0.55
84.07
83, 83.4840..5
79.87+0.79
FFA-LoRA
91. 91.06+0.53
85. 40.08
84.30
81.1440.59
RoLoRA
92.32
86. 25.24.2
84.4944..5r
81.544...50
LA-LoRA
92.66+0.47
88. +0.442
85. 80.34
82.35+0.46
Effect of local alternating updates. We evaluate DP-LoRA with LA-LoRA(-filte)) which uses alternating updates without the filter. Across both language and vision tasks, LA-LoRA(-fiter) consistently improves performance over DP-LoRA. For example, in Table 5, accuracy Tiny- ImageNet with Swin-B improves from 30.64% to 53.07%, indicating substantial gains in deep vision models. Similarly, in Table 5, accuracy on QNLI improves from 86.31 11% to 88.92%. These results demonstrate that alternating updates improve model utility under DP, achieving a better trade-off.
Effect of low-pass smoothing filter. We compare DP-LoRA DP-LoRA(+filter) and LA-LoRA(- filter) vs. LA-LoRA. In both domains, the filter consistently delivers additional performance gains. For example, on Imay-ImageNet (Swin-B, Table 5), DP-LoRA increases the accuracy from 30.64% to 49.85% over DP-LORA, while applying the filter to LA-LoRA(-ille further improves the accuracy from 53.07% 61.97%. The filter facilitates smoother updates that bias optimization toward flatter global solutions, improving both stability and generalization under DP.
Appendix presents supplementary experiments, including more results for Gaussian low-pass smoothing filter E, computational and memory cost B.4, and other ablation studies D.
Table 5: Impact of local alternating updates and low-pass smoothing filter on federated LoRA per- formance (%) across GLUE (RoBERTa- Basee) and image classification (Swin-B) benchmarks.
GLUE tasks (RoBERTa-Base)
Image Classification (Swin-B)
Method
SST-2
QNLI
QQP
MNLI
CIFAR-100
Tiny- Tiny-ImageNet
DP-LoRA
92.36
86.31 00
80.98 84.56
06.52.0.51
30.6440.30
DP-LoRA(+filer)
92.52+0.55
87. 87.06
84.7910.42
81.4340..7
69.08-0.52
49.85 19.85
LA-LoRA(-fllee
92.74+4.67
88,9240.60
84.98
82.40 82.4044..3
70.38
53.07
LA-LoRA
93. 93.12.1067
89.83.0.41
85.83
$2.9920.40
75.29 10.35
61.97 20.58
7 DISCUSSION AND CONCLUSION
In this work, we propose LA-LoRA, privacy-preservie framework for differentially private fed- crated adaptation. By alternating local updates and applying low-pass smoothing filter, LA-LoRA addresses three key challenges in DPFL: gradient coupling, noise amplification, and sharp aggre- gation bias. Experiments on vision and language tasks show consistent improvements in accuracy, stability, and privacy efficiency. Future work will extend LA-LoRA along three axes: (i) combin- ing DP-LORA adaptation with faster and more communication-effiiinn federated optimizers (Liu al., 2024b; 2025b;c); (ii) improving robustness and generalization under stronger heterogeneity principles (Liu et al., 2024a: 2025a;d); and (iii) scaling to larger foun- updates while mitigating precon-
via averaging/faatess-aavaa dation models, potentially leveraging preconditioneed/sccond ditioner drift (Liu et al., 2026).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (No. U24A20243, No. 62232013), in part by the National Key Research and Development Program of China (No. 2024YFB3108700, and in part by the Program of China Scholarship Council (CSC).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Fabio Bellavia, Zhenjun Zhao, Luca Morelli, and Fabio Remondino. Image matching filtering and refinement by planes and beyond. arXiv preprint arXiv:241
2024.
Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. Efficient federated learning for modern nlp. In Proceedings of the 29th Annual International Conference Mobile Computing and Networking, pp. 1-16, 2023.
Shuangyi Chen, Yuanxin Guo, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust federated finetuning of Ilms via alternating optimization of lora. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for fed- crated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 12903-12913, 2024.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programmingg pp. 1-12. Springer, 2006.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and trends@) in theoretical computer science, 9(3-4):21 1-407, 2014.
Johan Edstedt, Georg Bökman, and Zhenjun Zhao. Dedode v2: Analyzing and improving the dedode keypoint detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4245 4253, 2024.
Chen Feng, Georgios Tzimiropoulos, and loannis Patras. CLIPCleaner: Cleaning Noisy Labels with CLIP. In The 32nd ACM International Conference on Multimedia (ACM MM), 10 2024a. doi: 10. 10.1145//666444733666664
Chen Feng, Georgios Tzimiropoulos, and Ioannis Patras. NoiseBox: Towards More Efficient and Effective Learning with Noisy Labels. IEEE Transactions Circuits and Systems for Video Technology, 2024b. ISSN 1558-2205. doi: 10. 00.11999TCSTT:2224444224449
Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner-Beuerle and Miguel D. Ro- drigues. Noisy but valid: Robust statistical evaluation of LLMs with imperfect judges. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. URI. https:/ http:://cpeereee wwwprevieww.eeeee//////e .nt/ wow.umiiddhhheeee...t
Sajjad Ghiasvand, Yifan Yang. Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, and Ramtin Pedarsani. Communication-efficiee and tensorized federated fine-tuning of large language mod- els. arXiv preprint arXiv:2410. 13097, 2024.
11

---

Published as a conference paper at ICLR 2026
Roger Grosse and James Martens. A layers. In International Conference on Machine Learning (ICML), pp. 573-582, 2016.
approximate fisher matrix for convolution
Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu Selec- tive aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, 2025.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1): 1997.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 000.2/0660000 066806 2021.
Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt Kusner. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577-11695, 2022.
Zong Ke, Yuging Cao, Zhenrui Chen, Yuchen Yin, Shouchao He, and Yu Cheng. Early warning of cryptocurrency reversal risks via multi- multi-source data. Finance Research Letters, pp. 107890, 2025.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Technical Report.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. http://cs231n stanford. edu/t tinutiiim
2015. Accessed: 2025-08-05.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuming. arXiv preprint arXiv:2104. 00090.......... 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101. 1010:2200..00900. 2021.
Junkang Liu, Yuanyuan Liu, Fanhua Shang. Hongying Liu, Jin Liu, and Wei Feng. Improving gen- cralization in federated learning with highly heterogeneous data via momentum-bant based stochastic controlled weight averaging. In Forty-second International Conference Machine Learning, 2024a.
Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, and YumXiang Gong. Fedbegd: Communication--lliccenn accelerated block coordinate gradient descent for federated learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 2955- 2963, 2024b.
Junkang Liu, Fanhua Shang, Yuxuan Tian, Hongying Liu, and Yuanyuan Liu. Consistency of cal and global flatness for federated learning. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 3875-3883, 2025a.
Junkang Liu, Fanhua Shang, Junchao Zhou, Hongying Liu, Yuanyuan Liu, and Jin Liu. Fedmuon: Accelerating federated learning with matrix orthogonalization. arXiv preprint arXiv:2510.27403, 2025b.
Junkang Liu, Fanhua Shang, Kewen Zhu, Hongying Liu, Yuanyuan Liu, and Jin Liu. Fedadamw: A coammunication-efficeete optimizer with convergence and generalization guarantees for federated large models. arXiv preprint arXiv:2510.27486, 2025c.
Junkang Liu, Yuxuan Tian, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Junchao Zhou, and Daorui Ding. Dp-fedpgn: Finding global flat minima for differentially private federated learning via penalizing gradient norm. arXiv preprint arXiv:2510.27504 2025d.
Junkang Liu, Fanhua Shang, Hongying Liu, Jin Liu, Weixin An, and Yuanyuan Liu. Taming pre- conditioner drift: Unlocking the potential of second-order optimizers for federated learning on non-iid data, 2026.
12

---

Published as a conference paper at ICLR 2026
Xiao-Yo Yang Liu, Rongyi Zhu, Daochen Zha, Jiechao Gao, Shan Zhong, Matt White, and Meikang Qiu. Differentially private low-rank adaptation of large language model using federated learning. ACM Transactions Management Information Systems, 16(2):1-24, 2025e.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 1907. / 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp- 10012-10022, 2021.
Zhiyu Liu, Zhi Han, Yandong Tang, Hai Zhang, Shaojie Tang, and Yao Wang. Efficient over- parameterized matrix sensing from noisy measurements via alternating preconditioned gradient descent. arXiv preprint wxXiv22502.00465 2025f.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiien learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282 PMLR, 2017.
Lei Meng. Zhuang Qi, Lei Wu, Xiaoyu Du, Zhaochuan Li, Lizhen Cui, and Xiangxu Meng. Im- proving global generalization and local personalization for federated learning. IEEE Transac- fions on Neural Networks and Learning Systems, 36(1):76-87, 2025. doi: 10.1 00.100/TNNLS.200 3417452.
Ilya Mironov. Rényi differential privacy. In Proc. IEEE computer security foundations symposium (CSF), pp. 263-275, 2017.
Xutong Mu, Ke Cheng, Teng Liu, Tao Zhang, Xueli Geng, and Yulong Shen. Fedpta: Prior-based tensor approximation for detecting malicious clients in federated learning. IEEE Transactions Information Forensics and Security, 19:9100-911 9114,,999111111 2024a.
Xutong Mu, Ke Cheng, Yulong Shen, Xiaoxiao Li, Zhao Chang, Tao Zhang, and Xindi Ma. Feddme: Efficient and robust federated learning via detecting malicious clients. IEEE Transactions on Dependable and Secure Computing, 21(6):5259-522 2024b.
Maxence Noble, Aurélien Bellet, and Aymeric Dieuleveut. Differentially private federated learning heterogeneous data. In International conference artificial intelligence and statisties, pp- 10110-10145. PMLR, 2022.
Kaichen Ouyang, Zong Ke, Shengwei Fu, Lingjie Liu, Puning Zhao, and Dayu Hu. Learn from global correlations: Enhancing evolutionary algorithm via spectral gnn. arXiv preprint arXiv:2412.11723 2024.
Mert Pilanci and Tolga Ergen. Neural networks are convex regularizers: Exact polynomial-time con- vex optimization formulations for two-layer networks. In International Conference Machine Learning, pp. 7695-7705. PMLR, 2020.
Jaxxing Qi. Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. Fdlora: Personalized federated learning of large language model via dual lora tuning. arXiv preprint arXiv:2406.0799 07925, 2024.
Luchao Qi, Jiaye Wu, Jun Myeong Choi, Cary Phillips, Roni Sengupta, and Dan B. Goldman. Over++: Generative video compositing for layer interaction effects. 2025a. doi: 10.48550/arXiv. 2512. 19661. URL https:://///....... /aaaaaivvvvrr... abs / 2512. 19661
Xin Qi, Meixuan Li, Sijin Zhou, Wei Feng, and Zhuang Qi. Federated learning for science: A survey on the path to a trustworthy collaboration ecosystem. Authorea Preprints, 2025b.
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan- Lin. Text-driven prompt generation for vision- -sagunla models federated learning. arXiv preprint arXiv:2310006/233 2023.
13

---

Published as a conference paper at ICLR 2026
Shiqing Qiu, Haoyu Wang, Yuxin Zhang, Zong Ke, and Zichao Li. Convex optimization of markov decision processes based on z transform: A theoretical framework for two-space decomposition and linear programming reconstruction. Mathematiiss, 13(11):1765, 2025.
Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rant solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471-50) 2010.
Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Léon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. In International Conference on Learning Representations (ICLR), 2018.
Jiyun Shin, Jinhyun Ahn, Honggu Kang, and Joonhyuk Kang. Fedsplitx: Federated split learning for computationaallyconnstaaa
heterogeneous clients. arXiv preprint arXiv:2310. 145779 2023.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-presseringg federated learning. The Twelfth International Conference on Learning Representations, 2024.
Idris O Sunmola, Zhenjun Zhao, Samuel Schmidgall, Yumeng Wang, Paul Maria Scheikl, and Axel Krieger. Surgical gaussian surfels: Highly accurate real-time surgical scene rendering. arXiv preprint arXiv:2503.0400 2025.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404.19245 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355 2018.
Jianyu Wang, Yu Wang, Shengjie Zhao, and Sifan Zhou, Point4bit: Post training 4-bit quantization for point cloud 3d detection. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025.
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd international conference artificial intelligence and statisticy, pp. 1236-1235 PMLR, 2019.
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint 20Xv:2240.05976. 2024.
Shuang Wu, Heng Liang, Yong Zhang, Yanlin Chen, and Ziyu Jia. A cross-modal densely guided knowledge distillation based on modality rebalancing strategy for enhanced unimodal emotion recognition. In Proceedings of the Thirty-Fourle International Joint Conference on Artificial Intelligence, IJCAI 2025, Montreal, Canada, August 16-22, 2025, pp. 4234 4244, 2025.
Hongyong Xiao, Xutong Mu, and Ke Cheng- Fedrma: A robust federated learning resistant to multiple poisoming attacks. Journal of Networking and Network Applications, 4(1):31-33 2024.
Zequn Xie, Boyun Zhang, Yuxiao Lin, and Tao Jin. Delving deeper: Hierarchical visual perception for robust video-text retrieval. arXiv preprint arXiv:2601. 12768, 2026.
Shaocheng Yan, Pengcheng Shi, Zhenjun Zhao, Kaixin Wang, Kuang Cao, Wu, and Jiayuan Li. Turboreg: Turboclique for robust and efficient point cloud registration. arXiv preprint arXiv:2507.01433 2025.
Sixing Yu, J Pablo Muñoz, and Ali Jannesari. Bridging the gap between foundation models and heterogeneous federated learning. urXiv preprint arXiv:2310.00247, 20233
Fangzhao Zhang and Ment Pilanci. Riemannian preconditioned lora for tie-ttning foundation mod- els. arXiv preprint arXiv:2402.023447 2024.
14

---

Published as a conference paper at ICLR 2026
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang. and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1915-6919 IEEE, 2024a.
Xinwei Zhang, Zhiqi Bu, Mingyi Hong, and Meisam Razaviyayn. Doppler: Differentially private optimizers with low-pass filter for privacy noise reduction. Advances neural information pro- cessing systems, 37:41826 41851, 2024b.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication- efficient and privacy-preserving prompt tuning in federated learning. ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Sichen Zhao, Zhiming Xue, Yalun Qi, Xianling Zeng, and Zihan Yu. Non-intrusive graph-based bot detection for e-commerce using inductive graph neural networks. 2026. doi: 10.48550/arXiv. 2601 12579. URL httLttss//axiiv org/
http::///aaxii..ooo///////////// 26Claa/ss.1..... 22579.
Lele Zheng, Xiang Wang, Tao Zhang, Yang Cao, Ke Cheng, and Yulong Shen. Differentially private subspace fine-tuning for large language models. arXiv preprint arXiv:2601.11113 2026.
Sifan Zhou, Jiahao Nie, Ziyu Zhao, Yichao Cao, and Xiaobo Lu. Focustrack: One-stage focus- and-suppress framework for 3d point cloud object tracking. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 7366-7375, New York, NY, USA, 2025a. Associ- ation for Computing Machinery. ISBN 9798400720352. doi: 10.1145/744027.355778) URL pettps:/ddo... ott.:/oorg//00/ 211...../37442222222 1111144/777462777777788999 3754781.
Sifan Zhou, Shuo Wang, Zhihang Yuan, Mingjia Shi, Yuzhang Shang, and Dawei Yang. GSQ- tuning: crouppshhaed exponents integer in fully quantized training for LLMs on-device fine- tuning. In Findings of the Association for Computational Linguisties: ACL 2025, pp. 22971- I- 22988, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8- 89176-256-5.
Sifan Zhou, Zhihang Yuan, Dawei Yang, Xing Hu, Jian Qian, and Ziyu Zhao. Pillarhist: A guantization-aware pillar feature encoder based on height-aware histogram. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 27336-27345, 2025c.
15

---

Published as a conference paper at ICLR 2026
8 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, no human subjects or animal experi- mentation was involved. All datasets used, including CIFAR-100, Tiny-ImageNet, SST-2, QNLI, QQP, MNLI, were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care avoid any biases discriminatory outcomes in our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concern. We are committed to maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. All code and datasets have been made publicly available in an anonymous repository (https: //gthub.. com/ junkangI (ung 1ul/ //00 LO-LORA) to facilitate replication and verification. The ex- perimental setup, including training steps, model configurations, and hardware details, is described in detail in Section 6, Appendix B.2, and C.2.
We believe these measures will enable other researchers to reproduce our work and further advance the field.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
LIST OF APPENDIX
A: More Related Work
B: Additional Details for Image Classification
B.1 Datasets and models
B.2 Experimental setup
B.3 Additional experimental results
B.4 Computational and memory overhead
B.5 Impact of different ranks performance
B.6 Centralized experiments
B.7 Non-private federated experiments
C: Additional Details for Language Understanding
C.I Datasets and models
C.2 Experimental setup
C.3 Additional experimental results
C.4 Additional experiments on Llama-2-7B
C.5 Language model results under data heterogeneity / 0.3
D: Additional Ablation Studies
D.1 Ablation results for Swin-T on CIFAR-100 and Tiny-ImageNee
D.2 Ablation smoothing strategies
D.3 Ablation on the three challenges
D.4 Effect of local steps K
D.5 Effect of different local alternating update strategies
E: More results for Gaussian Low-pass Smoothing Filter
E.1 Smoothing parameter d
E.2 Effect of different kernel widths
F: Detailed Theoretical Analysis
F.I Privacy guarantee
F.2 Closed-form projected gradients
Stable feature learning
F4 Convergence analysis
F.5 Comparison with FFA-LoRA and RoLoRA
G: Table of Notations
H: LLM Usage
17

---

Published as a conference paper at ICLR 2026
A MORE RELATED WORK
Recent advances in federated optimization and generalization provide complementary perspectives our DP federated adaptation setting. On the optimization side, FedBCGD (Liu et al., 2024b) develops an accelerated block coordinate gradient descent framework for communication-efficin FL., while FedMuon (Liu al., 2025b) further accelerates training via matrix orthogonalization. For large-model federated training. FedAdamW (Liu et al., 2025c) introduces an Adam AdamW-styl opti- mizer with improved communication efficiency and stability. Beyond optimization speed, general- ization under strong heterogeneity has been studied by FedSWA (Liu et al., 2024a), which leverages stochastic weight averaging to enhance robustness. Meanwhile, the geometry of FL objectives has attracted growing attention: FedNSAM (Liu al., 2025a) analyzes the consistency relationship be- tween local and global flatness, and the DPFL regime, DP-FedPGN (Liu et al., 2025d) explicitly penalizes gradient norms to encourage flatter minima. Finally, second-order and preconditioned methods FL. face the challenge of client-induced preconditioner drift; FedPAC (Liu et al., 2026) mitigates this drift to unlock the potential of second-order optimizers.
B ADDITIONAL DETAILS FOR IMAGE CLASSIFICATION
Section 6 outlines the main experimental configurations and results. For completeness, summa- rize only the additional settings not previously described and further results.
B.1 DATASETS AND MODELS
. CIFAR-100 (Krizhevsky. 2009) comprises 100 categories organized into 20 broader groups, containing 50,000 training images and 10,000 test images. Each image is 32x32 pixel RGB color image. Every category includes 600 images, with 500 for training and 100 for testing. Tiny-ImageNet (Iee & Yang. 2015) a subset of the ImageNet dataset contain- ing 200 object categories, totaling around 100,000 images. Each image is a 64x64 pixel RGB color image. Every category consists 500 training images, 50 validation images, and 50 test images.
Swin-T T (Liu et al., 2021) 1) is the smallest variant of the Swin Transformer, using a 4x4 patch embedding with 96-dimensional embedding size. It has stage depths of [2, 2, 6, 2], about 28M parameters. Swin-B (Liu et al., 2021) is a larger variant with a 4x4 patch embedding and 128-dimensional embedding size. It has stage depths of [2, 2, 18, 2), about 88M parameters.
B.2 EXPERIMENTAL SETUP
In image classification fine-tuning tasks, we apply LoRA with rank r = a 16, keeping the classi- fication head trainable so that it can adapt to the target dataset's label space and feature distribution, and thus remain compatible with the LoRA-updated representations during local training.
We fix 5 grid-search the Rényi order À under the subsampled Gaussian mechanism with per-sample & clipping, and choose the noise scale a that satisfies the (e, 8) constraint after converting from RDP. The resulting
and use an RDP accountant. For each target privacy budget E {3. 2, 1).
CIFAR-100: a E (0.195, 0.29, 0.56),
-inyy-ImageNet: {0.098, 0.146, 0.283).
These correspond to privacy budgets of c E {3, 2, 1} respectively. Gradient clipping is performed on per-layer basis using median clipping strategy, where each layer's clipping threshold C is set to the median of its gradient norm distribution, enabling balanced sensitivity control and stable training under differential privacy constraints.
18

---

Published as a conference paper at ICLR 2026
B.3 ADDITIONAL EXPERIMENTAL RESULTS
Table Section presents the final performance comparison between our LA-LoRA and three SOTA baselines across different values. In this section, present the complete convergence curves corresponding to these experiments. Figure presents the convergence curves for the Swin- T and Swin-B models under e=2. In both cases, LA-LoRA consistently outperforms the three SOTA baselines. The improvement is particularly pronounced on Tiny-ImageN, with Swin-B, where LA- LoRA surpasses RoLoRA's 43.97% by 17.06%. Figure illustrates the convergence of test accuracy for CIFAR- 100 and -ing-maaeeNet using Swin-T and Swin-B under = 3. LA-LoRA consistently achieves higher accuracy and faster convergence than the three baselines.
DE-LIAA ERAiLAA
MuLLrdt
Tost Accus
OPLIRA
OPLiRA TEALORA
Tont
Gommnnekknnno Roonds
22
30
100
10
(a) CIFAR-100, Swin-T
(b) ImageNet, Swin-T (c) CIFAR-100, Swin-B
(d) ImageNet, Swin-B
Figure 7: Test accuracy of Swin-T and Swin-B on CIFAR-100 and Tiny- Tingeet with e 2.
EiIt IRA
Aotumor (8)
3PLRRA
3RLRRA *AAAoA
20
130
10
CIFAR-100, Swin-T
(b) ImageNet. Swin-T (c) CIFAR-I06 Swin-B
(d) ImageNet, Swin-B
Figure 8: Test accuracy of Swin-T T' and Swin-B on CIFAR- 100 and Tiny-ImageNel with c = 3.
B.4 COMPUTATIONAL AND MEMORY OVERIIEAD
Table 6 reports the per-round computation cost, memory and test accuracy of Swin-B 1. Compared with standard DP-LORA, LA-LoRA reduces the per-round time from 30.35 to 17.44 on CIFAR-IOK 100 and from 28.02 17.23 Tiny-ImageNet, and halves the memory cost from 3524 MB to 1762 MB thanks to the low-rank parameterization.
Among LoRA-based DP baselines, LA-LoRA has per-round time that comparable the fastest method: it differs by at most about s, while consistently achieving the best accuracy. Overall, LA- LoRA offers substantial accuracy gains with negligible additional computational overhead compared to other low-rank DP methods.
Table 6: Per-round computation cost, memory cost and performance comparison of Swin-B at 1.
Time Cost (s)
Memory Cost (MB)
Test Accuracy (%)
Method
CIFAR-100 Tmny-Im
CIFAR-100 Tiny-lmageNet
CIFAR-100 Tmng-Im
DP-LoRA
30.35
28.02
3524
3524
55.98
30.20
DP-LoRA(+flin
30.72
28.51
3524
3524
67.95
48.09
FFA-LoRA
17.85
16.54
1762
1762
61.94
39.33
RoLoRA
16.64
16.32
1762
1762
67.88
43.85
LA-LoRA(-filte))
17.30
17.16
1762
1762
69.87
52.72
LA-LoRA
17.44
17.23
1762
1762
74.56
60.68
19

---

Published as a conference paper at ICLR 2026
Table 7: Impact of rank of Swin-B on CIFAR-100 and Tiny-ImageNet runs.
averaged over
Rank
Method
CIFAR-100
Any-ImmgeNee
DP-LoRA
54.30=0..2
29.57
FFA-LoRA
59.78=0.44
36.58
RoLoRA
65.74-0.61
40.89 40.59
LA-LORA
72.88 -0.07
59.03 +0.45
DP-LoRA
55.98
30.20
FFA-LoRA
61.94_0.37
39.33
r= 16
RoLoRA
67.88 =0.32
+0.60
LA-LoRA
74.56 -0.62
60. 6 +0.56
DP-LoRA
56.33=a.28
31.577-..a0
FFA-LoRA
62. 02.64_0..3
44.45
r= 32
RoLoRA
67 =0.40
44.36
LA-LORA
75. 34. 0..11
62.99 10.42
B.5 IMPACT OF DIFFERENT RANKS ON PERFORMANCE
Table summarizes the test accuracy of four methods on the Swin-B model for CIFAR-100 and Tiny-ImageNer under a privacy budget c=1 with varying rank values (r=8, 16, 32). LA-LoRA consistently achieves the highest accuracy in all configurations, attaining 75.34% CIFAR-100 and 62.99% on Tiny-ImageNet when r=32. In contrast, DP-LoRA exhibits lowest accuracy and the greatest sensitivity to rank variation, indicating that its performance is more constrained by representational capacity under strict privacy constraints. FFA-LoRA and RoLoRA demonstrate intermediate performance. benefiting steadily from increased rank. Furthermore, Imageeete complex, resulting in slower convergence within the limited number communication rounds. Consequently, its absolute accuracy is substantially lower than that of CIFAR-100, while the relative ranking of the methods remains unchanged.
These observations indicate that increasing the rank enhance representational capacity and miti- gate the adverse effects of differential privacy noise, with LA-LoRA exploiting this advantage most effectively. However, larger rank also introduces potential drawbacks, including increased commu- nication cost, heavier local computation, and a higher risk of overfitting under limited communica- tion rounds. Balancing these trade-offs, we adopt r 16 the baseline setting our experiments to achieve a favorable compromise between performance and efficiency.
B.6 CENTRALIZED EXPERIMENTS
To verify how the challenges discussed in Section manifest the centralized setting, we conduct centralized DP experiments using the same model architecture, optimizer, clipping norm, and noise multiplier as in our federated setup.
As summarized in Table 8, LA-LoRA(-filter) achieves higher test accuracy and higher "Grad. Cos. (late)" than DP-LoRA in the centralized setting. This confirms that the DP-LoRA issues we iden- tify are not specific to federated training. Moreover, the gain of LA-LoRA((filter) increases from 1.18% in centralized DP training to 11 11.22% in federated DP training, and the gradi- ent cosine gap widens from 0.032 to 00108, indicating that federated optimization exacerbates the gradient coupling inherent in centralized settings.
DP-LoRA
B.7 NON-PRIVATE FEDERATED EXPERIMENTS
To assess whether the observed gradient behavior persists the absence of DP, we run federated experiments withour DP. Table 9 shows that, even without gradient clipping and 0, LA-LoRA(- filter) achieves higher test accuracy and higher "Grad. Cos. (late)" than Fod-LoRA, indicating that gradient alignment is beneficial even in the non-private setting.
20

---

Published as a conference paper at ICLR 2026
Table 8: Centralized and federated DP training for Swin-T on CIFAR-100 with "Grad. Cos. (late)" denotes the average cosine similarity between V A and VB over the last 10% of training steps. Federated DP-LoRA has lower test accuracy and gradient cosine than centralized DP-LoRA, while LA-LoRA(-filter) improves both settings.
Setting
Method
Test Acc. (%)
Acc
Grad. Cos. (late)
ACos
Centralized
DP-LoRA
76. 10.11,m 1:.38
0.681
(A--fRR((-
77.29
t 1.18
0.711
1 0.032
Federated
DP-LoRA
45.40=+0.40
0.337
LA-LoRA(-fiter)
56.62=0.54
11.22
0.445
f 0.108
Table 9: Non-private federated training for Swin-T CIFAR-100. "Grad. Cos. (late)" denotes the average cosine similarity between V v and V n. over the last 10% of training steps.
Method
Test Acc. (%)
AAcc
Grad. Cos. (late)
ACos
Fed-Lot Fed-LoRA
90.56+0.22
0.694
Non-private
91.2511.15
10.69
0.783
0.089
Table 10 shows the performance comparison on Swin-B under non non-private federated architecture. DP-LoRA is the same Fed-LoRA. On the CIFAR-100, LA-LoRA achieves test accuracy of 84.21% %, about 1% higher than DP-LoRA (83.21%) and RoLoRA (83.25%)
Table 10: Non- Non-private federated training for Swin-B on CIFAR-100 100 and Tiny- ImageNe.
Method
CIFAR-100
Tiny-ImageNef
DP-LoRA
83.21
81.37
FFA-LoRA
81.65
80.76
Non-private
RoLoRA
83.25
81.03
LA-LoRA
84.21
82.58
C ADDITIONAL DETAILS FOR LANGUAGE UNDERSTANDING
C.I DATASETS AND MODELS
We evaluate our approach on four representative tasks from the GLUE benchmark (Wang et al., 2018): SST-2, QNLI, QQP, and MNLI. SST-2 is binary sentiment classifica- tion dataset derived from the Stanford Sentiment Treebank, containing about 67K train- ing sentences. QNLI is binary natural language inference dataset converted from the Stanford Question Answering Dataset (SQuAD), with approximately 105K training sen- tence-question pairs. QQP is binary paraphrase identification dataset from Quora, com- prising around 364K training question pairs. MNLI is three-way natural language infer- ence dataset with multi-genre coverage, containing roughly 393K training sentence pairs.
RoBERTa-Baa. (Liu al., 2019) is transformer-based language model optimized for robust pretraining. It adopts the BERT architecture with 12 transformer encoder layers, 12 self-attention heads per layer, and hidden size of 768, totaling approximately 125M parameters. Compared to the original BERT, RoBERTa removes the next-sentence predic- tion objective, uses larger batch sizes, trains on more data, and applies dynamic masking, resulting in improved performance across a variety of NLP benchmarks.
C.2 EXPERIMENTAL SETUP
For language understanding tasks, we freeze the classification head to preserve the well-trained label mapping of the language models, reduce the susceptibility of its relatively small parameter space to DP noise, and ensure stable and efficient adaptation via LoRA updates. We apply LoRA with 8 to match the language models, using maximum sequence length of hay 128.
21

---

Published as a conference paper at ICLR 2026
For privacy parameters, set ó = -e for SST-2 and QNLI, and ó = 1e - for QQP and MNLI to account for their larger dataset sizes. The noise multipliers corresponding to privacy budgets {3, 1 are:
SST-2: E {0.36, 0.53, 1.0],
QNLI: E (0.23, 0.34, 0.67),
QQP: {0.073, 0.11, 0.21}.
MNLI: (0.067, 0.10, 0.195).
Gradient clipping follows the same strategy in the image classification setup.
C.3 ADDITIONAL EXPERIMENTAL RESULTS
Teat Acstacuracy
Accurecy
(P-LAA
3 mou 1
FRVLURA
SENLORA
RoLORA
SULURA
SLURA
LALORA
LALORA
LALORA
LALORA
Communicaton Rounds
Communcation Rounds
Communicatin Rounds
Communcction Rourds
6=
(b) QNLI,
(c) QQP, c=
(d) MNLI,
Figure 9: Test accuracy of RoBERTa-Base on SST-2, QNLI, QQP, and MNLI with = =
Tor Recund (6) 31
Tost Accuncs (8) 8
OPLLAAA
Toat
LACORA
LALIFAA
LALLBA
LALIBA
Communicato Rounds
Communicat Rounds
Communication Rounds
Communication Rounds
(a) SST2,
(b) QNLI, 6 =
(c) QQP,
(d) MNLI, c =
Figure 10: Test accuracy of RoBERTa-Bars on SST-2, QNLI, QQP, and MNLI with
g
Tex Accureccy (8))
Tou 2.3...
3
JPALRA
OPALORA FRA
OALRR SALURA
FRA
20
20
53
10
10
(a) SST2, c
(b) QNLI, =3
(c) QQP. c
(d) MNLI, =
Figure 11: Test accuracy of RoBERTa-Bas Base on SST-2, QNLI, QQP, and MNLI with
Figure 9, Figure 10, and Figure 11 present the convergence curves LA-LoRA and three SOTA baselines (DP-LoRA, FFA-LORA, RoLoRA) SST-2, QNLI, QQP, and MNLI using RoBERTa- Base under different privacy budgets (e (1, 2,)). Table 4 summarizes the corresponding final test accuracies. Across all settings, LA-LoRA consistently achieves the highest accuracy while maintaining fast convergence.
When (Figure 9), where the privacy is the strongest, the performance gap between LA- LoRA and the baselines is the most pronounced. For instance, on QNLI, LA-LoRA reaches 88.73 98.73% compared with 86.25% for RoLoRA and 85.08% for FFA-LoRA, showing improvements of 2.48% and 3.65%, respectively. Similar gains are observed for QQP (0.85% over RoLoRA, 1.04% over FFA-LoRA) and MNLI (0.81% over RoLoRA, 1.21% over FFA-LoRA).
22

---

Published as a conference paper at ICLR 2026
c 2 (Figure 10), the accuracy gap narrows, but LA-LORA still leads in all datasets. For example, on QNLI, LA-LoRA obtains 89.18% exceeding RoLoRA (87.08%) and FFA-LoRA (87.30%) by roughly 2.10% and 1.88%
At e = 3 (Figure 11), where privacy constraints are weakest, all methods perform more closely, yet LA-LoRA maintains the highest accuracy across all datasets.
C.4 ADDITIONAL EXPERIMENTS ON LLAMA-2-7B
We evaluate method on the Llama-2-7B model subset of the GLUE benchmark (MNLI, QNLI, QQP, SST-2). We follow exactly the same fine-tuning hyperparameters and training pipeline in main experiments, changing only the underlying backbone model.
Table 11 shows that, compared to baseline DP-LoRA, FFA-LoRA, RoLoRA, and LA-LoRA yield consistent gains in the four GLUE tasks. Among them, LA-LoRA achieves the best average per- formance, improving the DP-LoRA baseline by 1.31%. Moreover, LA-LoRA further outperforms the strongest baseline variant, RoLoRA, by 0.7%. The improvements noticeable QQP and MNLI, indicating that our approach may also transfer to larger, more recent LLM backbone.
Table 11: Test accuracy of Llama-2-78 7B on SST-2, QNLI, QQP and MNLI with 1.
Model
Method
SST-2
QNLI
QQP
MNLI
Avg
DP-LoRA
91.56
88.22
85.56
86.86
88 88.05
FFA-LoRA
92.53
89.23
85.56
88.58 86.98
Llama-2-7B
RoLoRA
92.12
89.34
85.98
87.21
88 88.66
LA-LoRA
93.36
89.78
86.75
87.56
89.36
C.5 LANGUAGE MODEL RESULTS UNDER DATA HETEROGENEITY 0.3
We report additional results of language tasks under Dirichlet 3 0.3. Table 12 summarizes the performance of all baselines and LA-LoRA variants on GLUE tasks.
Table 12: Test accuracy(%) of RoBERTa Base on SST-2, QNLI, QQP and MNLI, Dirichlet 3 = 0.3.
Privacy
Method
SST-2
QNLI
QQP
MNLI
Avg.
DP-LoRA
92.07
86.25
84.02
81.22
85,89
FFA-LoRA
92.56
87.53
85,36
81.54
86.75
Non-private
RoLoRA
93.65
88.65
85.52
82.22
87.51
LA-LoRA
93.94
89.96
86.41
83.32
88.41
DP-LoRA
90.13
83.79
83.28
79.80
84.25
FFA-LORA
90.62
84.63
84.10
80.98
85.00
RoLoRA
91.74
85.86
84.21
81.35
85.78
LA-LoRA
92.11
87.13
85.04
82.27
86.64
GLUE, improving over DP-LoRA by 2.39% and over the best alternating baseline RoLoRA by 0.86%. In the non-private setting, LA- LoRA improves average accuracy by 2.52% over DP-LoRA and by 0.90% over the best alternating baseline, showing that LA-LORA is beneficial even without DP noise.
When 1, LA-LoRA achieves the best average
D ADDITIONAL ABLATION STUDIES
D.1 ABLATION RESULTS FOR SWIN-T ON CIFAR- AND TINY-IMAGENET
Table 13 reports the ablation results for Swin-T on CIFAR- and ImageNe,,,, examining the individual contributions of the local alternating update strategy and the low-pass smoothing filter. Consistent with the findings in the main text for Swin-B (Table 5), applying local alternating updates (DP-LoRA LA-LoR AoLLARAAflllee (-filter) substantially improves accuracy over the baseline DP-LoRA, with
23

---

Published as a conference paper at ICLR 2026
gains of 11.22% on CIFAR-100 and 19.91% Tiny- Furthermore, incorporating the low-pass smoothing filter (LA-LoRA(-fiter) - LA-LoRA) delivers additional performance boosts, reaching 60.07% and 60.97% CIFAR- -IFAR-100 and firm that both components contribute positively and complementariyy to performance, with the com- bination yielding the highest accuracy across datasets.
respectively. These results con-
Table 13: Effect of local alternating updates and low-pat smoothing filter for Swin-Tr I, c 3.
Method
CIFAR-100
Tiny- IImggeNet
DP-LoRA
45. 20.44
32.27:0.01
DP-LoRA(+
55.7520.65
50.69+0.67
LA-LoRA(-filte)
56, 56.6220.64
52.1840.35
LA-LoRA
60.07+t.41
60.97 :0.44
D.2 ABLATION ON SMOOTHING STRATEGIES
Recent work has proposed Doppler (Zhang et al., 2024b) as a generic low-pass filtering module for differentially private optimizers, processin privatized gradients in standard (non-federated DP training improve their signal-to-noiss ratio.
To place our simple Gaussian filter in context, we instantiate Doppler in our DPFL setting with LoRA and treat it baseline. Specifically, for both DP-LoRA and LA-LoRA we compare three variants under exactly the same DPFL hyperparameters: no smoothing, Doppler, and our Gaussian filter, all applied on top of the same privatized local LoRA updates. All other optimizer, model, and privacy parameters are kept fixed. For Doppler, we evaluate every configuration reported in Table 2 of Zhang et al. (2024b) and find by {1, 13/11 and a, -9/11 to perform best in our setting.
Table 15 reports the results. On GLUE with RoBERTa-Base, both low-pass filters bring small but consistent gains over DP-LoRA: LA-LoRA improves QQP and MNLI from 84.56 44.56%/80.0 /80.98% to 85.83% 117///2.988 /82.99% (+1.27% / +2.01%)) slightly outperforming Doppler 85.45% 45.45%//22 90%%//2.111 18%...% On the more challenging vision benchmarks, the effect is substantially larger. For Swin-B, our Gaussian low-pass filter raises DP-LoRA accuracy from 56.52% to 69.08% on CIFAR-100 100 (+12.56%) and from 30.64% to 49. 19.85% on Tiny-ImageNet (+19.21%), whereas Doppler reaches 66.12% 12% (+9.60%) and 48.53% (+17 7. 7.9%). respectively. Combining the filter with local alternating updates, LA-LORA further improves CIFAR- and Imag- to 75.29% 29% and 61.97%, outperforming the corre- sponding LA-LoRA(+Dopplie variant by 2.41% and 4.61%.
Table 14: Impact of different low-pass filters on federated LoRA performance (%) across GLUE and image classification benchmarks, e 3.
GLUE tasks (RoBERTa-Baae
Image Classification (Swin-B)
Method
QQP
MNLI
CIFAR-100
Tiny-ImageNet
DP-LoRA
84.56
80.98
56.52+a.01
30.6440.30
OP-LoRA(+fitee
84.7920.12 42
81
69.08
49.85
DP-LORA(++ A(+Doppler)
$1.82.2..4 49
81.01_o.57
12.12
48.5310.57
LA-LoRA(-
24.98
82.40_o.59
70.38
53.0720.60
LA-LoRA
85.
35.83
$2.99 20.42
75.29 75.29 20.5
61.97
LA LA-LoRA(+DoOm (Doppler)
85.45
82.13
72.88
37.36+0.66
Overall, our ablations indicate that both Doppler and our Gaussian filter improve DPFL with LoRA by smoothing privatized updates, but in different ways. Doppler uses a recursive filter that depends on past outputs and tuned coefficients, whereas our Gaussian filter is short window weighted average with fixed weights. In our setting, the privatized low rank updates exhibit substantial high frequency noise across local steps and clients. This simple Gaussian kernel helps suppress these high frequency fluctuations and, on our benchmarks, yields larger gains than Doppler, especially on the more challenging vision tasks.
24

---

Published as a conference paper at ICLR 2026
D.3 ABLATION ON THE THREE CHALLENGES
In this subsection, we empirically decompose how the two components of LA-LoRA (locally alter- nating updates and the low-pass filter) relate to the three challenges identified in Sec. 3: gradient coupling, noise amplification, and loss sharpness. We report the Maximum Hessian eigenvalue Amax H)) restricted to LoRA parameters.
Table 15: Ablation of LA-LoRA components and the three challenges (noise amplification, gradient coupling, and loss sharpness) on GLUE (3 0.3) and image classification (3 = 0.1), = 1. We report test accuracy (%) and the maximum Hessian eigenvalue Amax (H) on LoRA parameters.
GLUE tasks (RoBERTa-Base)
Image Classification (Swin-B)
Method
QQP
(H)
CIFAR-100
Amax
DP-LoRA
84.02
43.74
55.98
101.62
DP-LoRA(+filee
85.63
11.366(_2.8))
67.95
80.33
LA-LoRA(-filee)
85.95
(.2.2
69.87
64.77
LA-LORA 86.41
(0.22(.8.52)
74.56
55.76(145.66)
Comparing DP-LoRA with DP-LoRA(+filter) isolates the effect of the low-pass filter. The accuracy gains (55.98% to 67 17.5%% on CIFAR-100) and the drop in (Amax (H) from 101.62 to 80.33) show that suppressing noise amplification already improves utility and smooths the loss landscape.
Comparing DO-LoRA with LA-LORA instead isolates the effect of locally alternating B and at the same noise level. The larger improvement in both accuracy (55.98% 69.87%. Amax (H) from 101. 62 to 64.77), indicating that mitigating gradient coupling is particularly important for deep vision backbones and contributes strongly reducing loss sharpness.
LA-LoRA combines both components. Relative to all ablated variants, it achieves the highest accu- racy on both QQP and CIFAR-100 and the smallest Maximum Hessian eigenvalue (e.g., Amax (H) from 101.62 to 55.76), showing that jointly addressing noise amplification and gradient coupling drives the model towards significantly flatter minima.
D.4 EFFECT OF LOCAL STEPS K
We study how the local step number affects LA-LoRA under fixed clipping norm, noise multi- plier, and number of communication rounds. In this setting, increasing K makes each client perform more noisy local updates, so by standard DP composition the overall privacy loss grows with K. As shown in Table 16, larger K generally leads to higher test accuracy both CIFAR- 100 and Tiny-Iman ImageNet. At the same time, larger K also incurs higher client computation and a larger pri- vacy budget. To balance model quality, privacy, and training cost, we therefore set K 20 as the default choice all main experiments.
Table 16: LA-LoRA test accuracy (%) with different local steps K (Swin-B).
K
10
20
30
50
CIFAR- (00
0.56
72.13
74.56
75.21
75.26
(iny-ImaeeNe((5))
0.283
58.54
60.68
61.24
61.32
D.5 EFFECT OF DIFFERENT LOCAL ALTERNATING UPDATE STRATEGIES
We vary the local alternating strategies between the two LoRA factors while keeping the total local steps fixed. Table 17 shows nearly identical results across strategies: the best-worst gap is always below 0.7% (typically within 0.3%). Default I-step B 1-step A strategy attains the best accuracy in most cases. These results indicate that LA-LoRA is insensitive to the precise local alternating strategy.
25

---

Published as a conference paper at ICLR 2026
Table 17: Effect of different local alternating strategies on CIFAR-100 and Tiny-ImageNet (Swin-B)
under = 3 and different Dirichlet distributions. Strategy "k-step B/ k-step A" denotes performing local gradient steps on B followed by i steps on in each local round.
Dirichlet 0.6
Dirichlet 0.1
Strategy
CIFAR-100
CIFAR-100
1-step B/ stsp A (default)
59.62.1.33
80. 77 0.550
75.29
61.97
2-step B 2-test A
89.2940.47
80.58=0.38
75. 15.1540.35
61.36=0-42
5-step B Stsep A
89.31
80. 40.40
75.47 40.48
61.78
5-step A S-step B
89.4540.26
80.70 20.70
75.3110.44
61.7920.58
E MORE RESULTS FOR GAUSSIAN LOW-PASS SMOOTHING FILTER
E.1 SMOOTHING PARAMETER 0e
Table 18 presents the performance of federated LoRA under different smoothing parameters on GLUE (RoBERTa-Base language understanding tasks and Swin-B image classification benchmarks (E = 1). For language tasks, relatively small values of σ₂ (e.g.. a, =0=0.001) yield consistently strong and stable performance across datasets, suggesting that mild smoothing effectively suppresses local update noise while preserving task-relevan information. In contrast, for image classification tasks, moderately larger values (e.g., s., a. 00001 tend to produce more robust results, indicating that stronger smoothing can better mitigate the impact of data heterogeneity and noisy updates in vision settings.
Table 18: Impact of the smoothing parameter a, federated LoRA performance (%) across GLUE (RoBERTa--ase) language tasks and image classification (Swin-B) benchmarks (c 1).
Smoothing parametero,
GLUE tasks (RoBERTa-Base)
Image Classification (Swin-B)
SST-2
QNLI
QQP
MNLI
CIFAR-100
Tiny-ImageNet
0.000
92.514p.40
87 $7.95
84.72.0.ar
+1.83+0.50
70.34
54 64.0544.52
O, 0.001
92.66.1.447
88.73.0.42
82.35 10.34
71.80
56. 140.4
a, 0.005
92.6010.44
85.26 88.75 42
$2.42.10.55
72.67
58.7510.38
0.010
92.6210.47
87.97
85.03 48
82.03
74.56
60.68.0.55
0.050
92.50+0.40
88.0220.54
84.96
82.03
74.33
61.27
The results further indicate that the optimal range af is task-dependent. Values between 0.001 and 0.005 are generally favorable for language tasks, whereas values between 0.01 and 0.05 are more suitable for vision tasks, offering a better balance between stability and accuracy in each domain.
E.2 EFFECT OF DIFFERENT KERNEL WIDTIIS
As described in Section 4.2, LA-LoRA employs simple ID low-pass filter to smooth the LoRA gradients before aggregation. In the main experiments, we use a 5-tap binomial Gaussian kernel as the default choice. To assess the robustness of LA-LoRA, conduct sensitivity study on different kernel widths. We consider three ID binomial Gaussian kemeels with different sizes:
G( = 11, G(
16 [1,4,6,44 1,,,,,
1
G = 61
1 [1, 6, 15, 20, 15,6, 1].
These correspond to 3-, 5-, and 7-tap binomial filters, respectively, obtained from the binomial coefficients of (1 (12 (1+1)A, and (1 (1+1)) 1)6 and normalized to sum to 1. Intuitively, larger kemels apply stronger smoothing, whereas smaller kernels apply milder smoothing.
Table 19 summarizes the results for Swin-B CIFAR-100 and Imagee performance of LA-LoRA remains stable across these configurations: the test accuracy varies within most 0.97%. All kernel choices provide a large gain over the DP-LoRA baseline (Table 3). Among the three variants, the 5-tap kemne ((0)) achieves the best overall trade-off between accuracy and efficiency, and thus is used as the default in all main experiments.
1. The
26

---

Published as a conference paper at ICLR 2026
Table 19: Sensitivity of LA-LoRA the choice of ID binomial kernel size. Results are reported for Swin-B on CIFAR-100 and Tiny- Inn-ImmgeNet at 1.
Kernel
Size
Coefficients
CIFAR-100
Tiny-Iman ImageNet
G
3
11,
73.59_a.oe
69.92
((5)
5
16
[1,
4,
74.56#0.8
60.68
G(T)
7
[1,
20, 6,
73.88_0:46
60.7440.0
64
F DETAILED THEORETICAL ANALYSIS
F.I PRIVACY GUARANTEE
We report standard (e, 6)-DP guarantees using the Rényi DP (RDP) accountant. Below we present the RDP definition, composition, and the conversion from RDP to (e, 6D-DP. Further implementation details are available in our code and Noble et al. (2022).
Definition (Rényi DP). For any E (1, oc) and privacy purameter p > O, a randomized mech- anism M X + y is said to be PDDP, if for any two neighboring datasets 'D and D',
(PMDD)(W)
DA (M(D))I((D))E x-1 log Ew-M(Dr)
Sp-
(17)
Composition (additivity) in RDP. RDP composes additively: if mechanisms Mı,...,MMs Ms applied on the same dataset (possibly adaptively), then for any fixed order > 1,
Protal (A)
Σ (A)..
(18)
Conversion from RDP to (c, 8D-DP. If mechanism is (A, (()))-RDD for all 1, then for any 8 E (0, 1) it satisfies
log(1/8)
(e, 8)DD with
<(0)
min A=1
p(A)
(19)
A-1
Post-processing. If M (e, (or (A, RDDP) and f is any (possibly randomized) mapping independent of the private data, then faM enjoys the same privacy parameters (Dwork et al., 2014). In LA-LoRA, the Gaussian low-pass filter such an f applied to the noisy updates, which justifies the post-processing argument following Theorem 1.
F.2 CLOSED--ORRM PROJECTED GRADIENTS
We present projection-busee view to explain why the proposed local alternating update improves optimization stability compared standard LoRA.
As discussed in Section 3, simultaneous updates of A and suffer from gradient coupling (Eq. 3), amplified noise (Eq. 4), and sharper aggregated solutions. In contrast, LA-LoRA alternately updates A and B (Eq. 5), effectively decomposing the optimization into two sequential low-rank projections.
Let A& R and B& E Rmx denote the low-rank factors step k and a/r the LoRA scaling factor. The update to B based on solving least-squares problem that projects the full gradient onto the column space of As:
min
|s(V B. C)As Vw, C
(20)
Here, ||| 1/28 refers to the squared Frobenius norm. V.L and V are corresponding approximated gradients. Once the optimal direction is obtained, the update then proceeds as:
B < B& BV-1VB.. C.
(21)
W *++ + WW--n C)
27

---

Published as a conference paper at ICLR 2026
To maintain consistency with the simultaneous update strategy, apply the model update to intermediate state (k + 2)). In our implementation, we treat each individual update to A or B as distinct optimization step, which simplifies the analysis without ambiguity.
After performing backpropagation with respect to B, the gradient computed for no longer ac- curately reflects the full gradient at step since the model parameters have already been partially updated. Consequently, to ensure that the update to A remains faithful to the full gradient, we min- imize the discrepancy between the actual gradient the intermediate model state W++ and the low-rank approximation derived from Ag. formulated as:
min ||aB++1 ( An VW CIP A.
(22)
Then, by gradient descent, update A and the full model
Ak+1 An v/ C,
WK+1 Wk+ BB (V...).
(23)
After solving the leaae-qquares projection problems (20) and (22), we obtain their optimal solutions in closed form. Specifically, Theorem 2 states that the projected gradients can be expressed as
1 S Ba (A
BBC C
V.L = s (BL BB+1) A
where the (A+A))- and B (Bx+1) terms arise from the Gram matrix inverses in the least- squares solutions, and the fo2 factor accounts for the LoRA scaling in both A and B directions.
Proof. For (20), differentiating the least-squares objective w.r.t. VB, C and setting the derivative to zero gives
W. CA
SwaCAF.
Assuming A invertible, we have
VBC Te.==IVW.L..
Vw. CAg
(A
Using the LoRA gradient relation V VB, C V. C.J. we substitute:
1
BBC
V C (AxA
$2
The derivation for (22) is analogous, yielding
2² BK+ BR+1 VAL s Br+ Vwnc.
and hence
1 A. - (B
1 (B B841) VAcC.
BB+1) (B
B V.aC 52
F.3 STABLE FEATURE LEARNING
Proof. Under the regularized gradient formulation in Eq. 13, the update to the full model can be written two half-steps.
First half-step (updating B). Using the least-squares solution for V V.,C. C,
W *+ = We - 178 VB. Ass
We 7762 Vw. C. (ARAT)-1)A
-Wa - -V,,, L AT (ARA))-AA
-We F
(24)
28

---

Published as a conference paper at ICLR 2026
Second half-step (updating A). Using the least-squares solution for V
Wk+1 W 178 Be+1 VALC
W k++ 118 Bx+1 (B -B+1)-BBBtt BR+1 VWaizf)
W -n (B+t Bk+1) BK+t VWugh.
(25)
Pro)e(B241)
Combining the two half-steps yields
WR+1 = Wt
(Projc(B (W C).
(26)
Here,
Proje(As) (4 Proje(Be11) = Bk+1 (B Bk+1) -
denote the orthogonal projections onto the space of Aş (right- migghhlmltipliiatioo)) and the column
space of BK+1 (left-multiplication) respectively.
F.4
CONVERGENCE ANALYSIS
F.4.1 SET UP
Following the previous work Zhang & Pilanci (2024), we provide a convergence analysis of the proposed algorithm within the wver-parameterized two-layer ReLU neural network tuning problem. For a data matrix X E Raxd and any arbitrary vector u E R we consider the set of diagonal matrices (diag([Xu
Rd), which take values 1 0 along the diagonal and indicate the possible activation patterns of the ReLU units. Let the distinct elements of this set be denoted as DI..... Dp (see Zhang & Pilanci (2024) for more details). The constant corresponds to the total number of partitions of 12d by hyperplanes passing through the origin that are perpendicular the rows of X Pilanci & Ergen (2020). Intuitively, P' can be regarded the number of possible ReLU activation patterns associated with X. Pilanci & Ergen (2020) explains that a two-layer ReLU problem shares the same optimal objective with convex problem.
0))
P Σ 1=1
min FE[P]
W.WW, - Y
(27)
As we focus on fine-tuning, given a pretrained model with weights (W3) we perform low-rank adaptation and express the problem in equation 27 as
P Σ
min A., A.
D
(28)
Let X E Raxd, A; E R'xe, B; E Raxr, and Y E Raxe, We consider the response model
Y Σ
D D.X(W,
1=1
Define
X. Σ B A
where the matrices B and A (and hence X.) fixed but unknown. We use o,(-) denote the r-th largest singular value matrix. Before proceeding, we first introduce the notion of the Restricted Isometry Property (RIP).
29

---

Published as a conference paper at ICLR 2026
Definition 3. (Restricted Isometry Property, Recht et al. (2010)) The matric C' R is said to satisfy Restricted Isometry Property(RPP) with parameters (r, 5,) if there exists constants S 6, S 1, for any matrices M E Rd with rank v, the below holds
(1 - M| & |CM I (1 +1++0 6
(29)
RIP is a widely used condition in the filed compressed sensing (Recht et al. (2010)), which states that the operator C' approximately preserves distances between low-rank matrices. In the absence of noise, we can establish a direct relationship between the loss function and the recovery error. If we denote C, X, Problem (28) is equivalent the problem below up to change of labels
min CB A)
Σ C. x.)
(30)
A,,
where (Bi, Bp} and {Ai,... .Ap..
Notation Inspired by the previous work Liu et al. (2025f), we introduce two local norms and their corresponding dual norms for matrix W Rx
PAj = A((A)),
|[]]P 4
= [
Pa
P.
|W]]
IIWA P.
(31)
PBj = (BB) B
|
|
W
|p.
[
[WPE I
B
Here, we assume A and Bj are of full rank for any i. If they aren't of full rank, we can replace them with the Moore- Penrose inverse. Now we are ready to establish the convergence analysis.
F.4.2 USEFUL LEMMA
For the k-th iteration, let's denote B, = (B). B and A ....... AA.). If we apply LA-LoRA or LA-LoRA+ without momentum for Problem (30), for any E [P). the alternating update rule as proposed can be written
1 4 n7 (V Ax)
(32)
B B V/V Bi (B Ac+1)((A A
First, we will list some assumptions used in our analysis. Assumption 1. Suppose that C; D,X obeys the r-RIP with constant 8, for each i. Assumption 2. Suppose that ||C? C,Ila = x DJ D2X112 S
1+6.
P(P-1)
Assumption 1 and also adopt in Zhang Pilanci (2024) to analyze their optimizer for LoRA. For matrix X with i.i.d Gaussian entries 1/ (lo)). D satisfies RIP for constant 6, when |lo on the order of r(d + c)/(d))). Note ||X D.T D,X I || X|I2 for all (i,j)'s. Thus bounding I D, amounts to bounding the largest singular value of the empirical covariance.
Lemma 1. For a given i [P), the gradient of Problem (30) are
P
V (B),-
X.).
(33)
B ((B,)) [(G) x.)(A(1)TT.
Proof. For any given i and f, yields
P Σ
P
V n C(B. A)
C,(B, x.)
Σ (C,))T,,BBBAA -x,)
2
(34)
Similarly, derive the V Bi I.B, A) shown in (33).
n
30

---

Published as a conference paper at ICLR 2026
Lemma 2. Suppose Assumption 1 and 2 holds, then we have
Le(B:: S (Bk, max
V Ai Co
(B,, Au)
B
(35)
(o(BEt), ActI) S (B,,,.A max V Bi (
(Bx
A
where P(n +1+8,
Proof. Using the update rule in (32), we have
(B Axt2)
Σ C X.))
Σ Ci BB A/ ((BB) B LB.
(B,, Ac)) L.. -x,)
Σ C(BBA -x.)
(36)
172
Σ CB BJ) BB) IV VA Ay (Bk A
2
P Σ C. - x.). ( - + (Br, (BC A&)
7
72
For Tí, recalling Lemma 1, then we have
< 2 Ti
Σ
(BB)
B (BR A
2
1
Σ (C) B Ae Ca (Br. (,),C,B((())) B VA Co An))
+
2
$
(3)
172
(1
8x)
maxx V V Ce A #
n
+
max ifj
|IC CilpP 1) max | 13 Lo Ba. AWI mt.
2
(b) 172 (1 8, + 1 max V (B, A
<
2
(37)
where (a) uses Cauchy Inequality, Assumption and the fact that [BB]((B)] -B)) = (b) (1+6-) uses the assumption that max |IC, C5ll2 P(P-1)-
31

---

Published as a conference paper at ICLR 2026
For T2, using Lemma 1 again, we have
T2 = 17
Σ C,IB/AL- - -x.):
C.B((BB)) B (cb (B
P
= 1) Σ
Σ
BB) A Le bz (xx:
(38)
17 >
VA Ce Bk+ AA)
S rpP max V A Lo BB. Ax)
To sum up, yields
11(1+ 6, 1)
V Lal (Bx: An)
Be( A+t1) S (Bx As)
P mmax
(39)
2
rig
Similarly, we can induce
172 6r + p) p P Paxx V Lo ARt1)
(e(Bi+ Ax+1) s (Bx Axt1) 11
pr Ai-1
(40)
Lemma 3. Suppose Assumption / holds, then, for any i [P], we have
1V Ve (B Ax) 11p Dj
> 2(1 6.) BB As).
(41)
| BC C. (Br, A
> 2(1 S,) (Br, Aut1).
14+1
Proof. See Lemma 6 in Liu et al. (2025f) for the detailed proof.
Theorem 5. Assume for any i E [p] the matrix C₁ = D;X satisfies the rank r-RIP with constant
5, (Assumption 1) and 0 parameterized problem leads to
+ then LA-LoRA without momentum solves the over-
Co AR+1) (11 2o An),
(42)
and
P
P
Σ
B
B
Xx
5
+
(1 n1)
Σ
Bi Ab
X.
(43)
8,
F
where n. = 2P(1 - 5,) (1)
(1++++,++
2
Proof.
11 + 6, + +)
Le(BrtI, Axt1) S (Bk, Axt1) -
11
P Pmax V BB La ARti)
17 8 + bb
$ (B Ax+1)
11-
2P &.) Be, Arti)
$ (1 11- - 2P1PP11 - 6r)
27 + 8, + +1)
))
Lak( (Bt)AAtt)
17
2
$ <1 2 Le(BE, At),
(44)
32

---

Published as a conference paper at ICLR 2026
where we apply Lemma 2 and 3 and T = 2P(1 - -6,) 6,)(1 11(1+6++
Moreover, under As-
sumption
have
1 8r (1 na)ak
Σ B - -X,
I 8,
Σ BijAj X,
(45)
F.5 COMPARISON WITH FFA-LORA AND RoLORA
Corollary (Comparison with FFA-LORA and RoLoRA). Under the same setting Theorem 3,
consider the following two baselines.
FFA-LORA (frozen A). Let As Ao and define the fixed row-space subspace So (AW Rmx" : rrw (AW) S r( Then W* Wo E for all Hence, for any target rank-r solution W' with W Wo & So. infa [[W-W*1M 2 dist(W* Wo, So) > O.
RoLoRA (round-wis) wise alternation). For one round updating B with A fixed, followed by one round updating A with B fixed, we show
Wa+2 = Wt- 7 (V,, Proos((cci)) Projo(BR11) (V C) +)++ Ek.
where the first two terms coincide with the LA- update in Eq. (26) and the stale-block remainder satisfies ||& = O F)
In contrast, LA-LoRA alternates B and A within each local iteration and exactly matches Eq. (26). ie., it imposes no fixedroowsppac constraint as in FFA-LoRA and has Ex = O.
Next, We provide the details underlying Corollary 1.
FFA-LORA: fixed row-space subspace. FFA-LoRA freezes A at some initialization Ag and only updates B, At iteration k we can write
W = Wo + sB.An:
so
WA Wo So with So (AW R= (Ow( r(Ao)). Let W be target (e.g., optimal) rank-r solution. If W Wo & So, then by projection geometry inf IWW W which is exactly the irreducible approximation gap stated for FFA-LORA.
inf AWES.
|AW (W* - Wo)l dist(W* Wo. So) 0,
RoLoRA: stale-block remainder & RoLoRA alternates between updating B and the level of communication rounds. Consider two consecutive local steps forming one full "B-then-A" cycle. Let Wk+1/2 denote the intermediate weight after updating but before updating A. Using the projected-gradics view of Theorem 2, we have
W = W₁ Projec (Vw, W,, W 1) (V (V Proje(Au+))
Adding these relations gives
W4+2 = We Proje(Brur) (Vw, - (V L) Projr(A
We add and subtract the LA-LORA update in Eq. (26) and define remainder
E = - -n (Vw.L VW.+ G) (Vwnry V Projecmn)))
which yields the decomposition
Wk+2 = W5 - n7 (Vw. L) Proj - Proje(Ba))) (V (W C) + Ex.
33

---

Published as a conference paper at ICLR 2026
Assuming that L is L-smooth, we have
WW - Vwe S W11 WiellE.
From the first update, Wk+1/2 WR -n Proje((( (Vw, C), and the projector has operator norm at most so |Wx+1/2 W&llF I n/1V F. Using again that the projectors have norm at most 1, we obtain
|| F - 2n 1 C Vw.Clle
- 2 2 W
(46)
- 2 w.,
Under the infinite-width scaling used in Theorem 3, the Frobenius norm VV itself scales with width, so the effective contribution of & to the feature dynamics of order [l, when 1 (1).
G TABLE OF NOTATIONS
Table 20 summarizes the main notations used throughout the paper.
Table 20: Summary of the main notations.
Notation
Description
Wo E mX
Pre-trained backbone weight matrix
A, B
LoRA down-projectionnap--reee projeccioonn matrices
LoRA rank, r < min(m: n
LoRA scaling hyperparameter
= a/r
LoRA scaling factor
AAL, BB
Gradients of loss w.r.t and
V W
Gradient of loss w.r.. full weight W
2()
Local loss function Local dataset of client Total number of clients Client sampling rate per round Set of selected clients in round Number of communication rounds Number local update steps per round Local data sampling rate Size of local dataset on client i Mini-batch sampled from D; Per-sample f2 clipping norm Gaussian noise multiplier for DP Clipped gradient for sample j client Noisy averaged gradient on client Gaussian DP noise with variance o
D;
N
C,
T
K
R=
|Di]
B.
C
gij
9i
N00,C207
9
Smoothed gradient after applying G, (c, (c,0)-di -d.-)ifffretttil privacy parameters
6,6
X.
Target low-rank matrix in theory Projection onto row space of A, Projection onto column space of B Hessian of the loss and its maximum eigenvalue (sharpness)
Proje(A)- Projo(B)
H, Xmaxe (H)
8,
Rank-r RIP constant of matrices C;
H LLM USAGE
Large Language Models (LLMs) were used to aid the writing and polishing of the manuscript. Specifically, we used LLM to assist in refining the language, improving readability, and ensuring
34

---

Published as a conference paper at ICLR 2026
clarity in various sections of the paper. The model helped with tasks such sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
It is important to note that the LLM not involved in the ideation, research methodology, or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis.
35


Reviews:
[{"review_id": "yyDkndfRFB", "reviewer": "Reviewer_fjPf", "summary": "This paper revisits the role of LoRA within the context of privacy-aware FL. The method introduces a local alternating update scheme where the two LoRA matrices are updated in turn during local training to decouple gradients and reduce noise amplification. Additionally, a low-pass Gaussian smoothing filter is applied to suppress high-frequency noise before aggregation, improving stability and generalization.", "questions": "1. The word \"update\" is spelled incorrectly in the beginning of section 4.1.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Strong Empirical Performance: Extensive experiments demonstrates high accuracy while achieving better privacy–utility balance than baseline defenses.\n\n2. Incorporation of Low-Pass Smoothing Filter: Uses a Gaussian low-pass filter before aggregation to smooth high-frequency DP noise, improving stability of the algorithm.", "weaknesses": "1. Lack of studies on different local alternating strategies: The current method fixes the local alternating update pattern to update matrix A on odd iterations and matrix B on even iterations, effectively using a one-step inner loop. It would strengthen the work to explore and experiment with other strategies such as updating A for several consecutive steps before switching to update B, or vice versa. Moreover, analysis of how different alternating schedules affect performance under various degrees of data heterogeneity would provide deeper insights.\n\n2. Incomplete Theoretical Convergence Analysis: Theorems 2 and 3 provide structural insights into projected gradients and stability but fall short of delivering explicit convergence rates or optimization behavior under privacy noise. Deriving explicit convergence rates, possibly for both convex and non-convex settings, and clarifying the trade-off between differential privacy noise injection and optimization accuracy, would provide a more comprehensive understanding of algorithm performance.\n\n3. Unclear Privacy Guarantees Regarding the Smoothing Filter: The privacy guarantee in Theorem 1 appears to ignore the effect of the low-pass Gaussian smoothing filter applied to the gradients before aggregation. It raises the question of whether the differential privacy guarantee strictly holds when the smoothing filter is included. Since the filter alters the noise characteristics, a clear discussion or proof of how these impacts or preserves privacy guarantees would strengthen the claim. Without this, the smoothing filter may be viewed as a redundant or questionable design that could be alternatively addressed by carefully tuning the privacy budgets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmrD7JXLOY", "reviewer": "Reviewer_kNbJ", "summary": "This paper investigates the limitations of directly applying Low-Rank Adaptation (LoRA) in differentially private federated learning (DPFL). The authors identify three intrinsic challenges that arise when LoRA is used under DP constraints: (1) gradient coupling between asymmetric matrices, (2) structural amplification of DP noise due to multiplicative interactions between noise terms, and (3) sharpness in global aggregation that leads to unstable convergence and poor generalization.\n\nTo mitigate these issues, they propose LA-LoRA, a simple yet effective modification that alternates updates between LoRA’s two low-rank matrices within each local round and applies an optional Gaussian low-pass filter to smooth noisy gradients. Theoretical analysis establishes convergence stability and privacy guarantees. Experiments on both vision and language models show consistent improvements across multiple privacy budgets, achieving state-of-the-art accuracy while maintaining formal DP guarantees.\n\nOverall, the paper is well-written, methodologically sound, and provides both theoretical and empirical support. Its main contribution lies in identifying the underexplored failure modes of LoRA under DPFL and offering an elegant yet practical fix.", "questions": "1. Are the three limitations (gradient coupling, noise amplification, sharpness) unique to the federated setup, or do they also appear in centralized DP-LoRA? If they appear in both, could LA-LoRA also enhance centralized DP-LoRA performance?\n\n2. Figure 2 compares DP-LoRA and LA-LoRA, but not Fed-LoRA (non-DP). Does higher gradient-similarity appear in the non-DP case? This would clarify whether cosine alignment is indeed the key to improved performance.\n\n3. What is the theoretical justification for the specific kernel? Have the authors considered learned or adaptive filters, or tested sensitivity to the kernel width?\n\nI would raise my score if the above questions are addressed properly.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clear problem identification.**\nThe authors provide an intuitive and thorough analysis of why existing DP-LoRA methods fail, including gradient coupling, multiplicative noise amplification, and curvature-related instability after aggregation — each well-motivated and demonstrated through figures and equations.\n\n2. **Sound theoretical grounding.**\nThe paper presents convergence and privacy guarantees with clear mathematical derivations and ties them directly to the algorithmic design. This theoretical foundation strengthens the credibility of the proposed solution.\n\n3. **Comprehensive experiments.**\nThe work evaluates both language and vision tasks using multiple baselines under varying ε-budgets. The improvements are consistent and substantial.\n\n4. **Simple yet effective modification.**\nThe proposed alternating update and Gaussian filter are computationally lightweight, easy to integrate, and yield significant utility gains without sacrificing privacy guarantees.", "weaknesses": "1. Based on your deduction, the identified issues (gradient coupling, DP-noise amplification, sharpness) seem to arise from LoRA’s intrinsic structure rather than federated communication. Do these problems persist even in centralized DP-LoRA? The paper should clarify that. If so, the research should focus on centralized DP-LoRA, instead of the federated setting.\n\n2. While LA-LoRA achieves higher cosine similarity, it is unclear whether this causally leads to better accuracy or is merely correlated. Including a non-DP Fed-LoRA baseline could help isolate whether high similarity is inherent and thus beneficial.\n\n3. While the low-pass filter idea is interesting, the paper should better differentiate it from recent filtered gradient mechanisms such as [1], which also employ noise smoothing for DP stability.\n\n4. Although LA-LoRA is efficient, it adds alternation and filtering. A detailed breakdown of time/memory cost versus baselines (beyond Table 6) would strengthen the practical claim.\n\n**Reference**\n\n[1] *Doppler: Differentially private optimizers with low-pass filter for privacy noise reduction. Neurips 2024*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNWDKBJrpJ", "reviewer": "Reviewer_gf7V", "summary": "This paper proposes LA-LoRA (Local Alternating LoRA), a new variant of LoRA for fine-tuning models under differentially private federated learning (DPFL). The authors identify three key challenges in applying conventional Low-Rank Adaptation (LoRA) to DPFL: 1)gradient coupling 2) noise amplification, and 3) sharpness of aggregated solutions, which degrade model performance under privacy constraints. To address these, LA-LoRA introduces a local alternating update strategy, where LoRA’s two low-rank matrices are updated sequentially rather than simultaneously with a low-pass filter. Some experimental results are presented to demonstrate the effectiveness of the proposed algorithm, compared to DP-LoRA, FFA-LoRA and RoLoRA.", "questions": "1) What is the intuition and theoretical justification for the 1D Gaussian kernel filter? Why can the adjacent element row/columns-wise be used to smoothing?\n2) How can Theorems 2 and 3  explain the utility of the proposed method?\n3) Does the proposed LA-LoRA method have different utility depending on the model and data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper presents clear problem identification and the potential reasons of the problems.\n2) The proposed algorithm is elegant, clear and effective.\n3) Some analysis about the stability is presented to demonstrate the effectiveness of the proposed algorithm.", "weaknesses": "1) It is unclear how the 1D Gaussian kernel filter can be justified from the optimization perspective. A potential issue is that such a filter can be model-dependent and/or data-dependent, as shown in the ablation study that vision models benefit significantly more than the language model. \n2) Theorems 2 and 3 seem superficial. They only present an ideal case; but they do not consider DP noise, the federated learning distributed nature, or the 1D Gaussian kernel filter.\n3) The LLM-related experiments can be extended to more recent tasks with more recently published models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ln9HLDGwg0", "reviewer": "Reviewer_cWrF", "summary": "This paper addresses three key challenges in federated DP-LoRA: gradient coupling, noise amplification, and loss sharpness. The authors propose LA-LORA, which employs two components: a local alternating update strategy to tackle gradient coupling and noise amplification, and a low-pass smoothing filter applied pre-aggregation to mitigate solution sharpness. Theoretically, the paper's analysis demonstrates that this alternating update avoids a destabilizing \"across term\" inherent in standard joint updates, leading to more stable optimization. Experimentally, LA-LORA outperforms existing methods like DP-LORA and RoLoRA on both vision and language benchmarks.", "questions": "1. What explains LA-LORA's empirical gains over FFA-LORA and RoLoRA?\n2. Why is LA-LORA's performance gain dramatically larger in vision (+16.83% on Swin-B) than in language (+2.48% on QNLI)? \n3. How does the algorithm perform (the final test accuracy) against other baselines in the non-private setting?\n4. How sensitive is LA-LORA to the choice of local steps K?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper's presentation is clear and the writing flow is well-organized. Also a clear reproducibility statement is supported with codes.", "weaknesses": "1. The theoretical analysis convincingly argues for alternating updates vs. joint updates but fails to differentiate LA-LORA from other alternating baselines such as FFA-LORA, RoLoRA. \n2. The analysis lacks a deep discussion on the relative importance of the three identified challenges. It remains unclear which challenge contributes most to performance degradation or how their respective impacts differ across vision versus language tasks.\n3. The paper re-frames the aggregation challenge as \"sharpness\" but fails to address the underlying mathematical \"aggregation bias\"( $\\mathbb{E}[\\bar{B}]\\mathbb{E}[\\bar{A}] \\neq \\mathbb{E}[\\overline{BA}]$), and it is unclear how the low-pass filter resolves this fundamental discrepancy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)