Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix
Authors: Ming Wen
Jiaqi Zhu
Yuedong Xu
Yipeng Zhou
DINGDING HAN
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Low-Rank Adaptation
Differential Privacy
Primary area: deep_learning
TL;DR: The paper introduces a novel two-stage sketching framework for federated LoRA that enables differentially private and effective updating of low-rank matrices in large language models.

Abstract: Large language models (LLMs) typically require fine-tuning for domain-specific tasks, and LoRA offers a computationally efficient approach by training low-rank adaptors. LoRA is also communication-efficient for federated LLMs when multiple users collaboratively fine-tune a global LLM model without sharing their proprietary raw data. However, even the transmission of local adaptors between a server and clients risks serious privacy leakage. Applying differential privacy (DP) to federated LoRA encounters a dilemma: adding noise to both adaptors amplifies synthetic noise on the model, while fixing one adaptor impairs the learnability of fine-tuning. In this paper, we propose FedASK (Differentially Private Federated Low Rank Adaptation with Double SKetching) , a novel federated LoRA framework to enable effective updating of both low-rank adaptor matrices with robust differential privacy. Inspired by randomized SVD, our key idea is a two-stage sketching pipeline. This pipeline first aggregates carefully sketched, privacy-preserving local updates, and then reconstructs the global matrices on the server to facilitate effective updating of both adaptors. We theoretically prove FedASK's differential privacy guarantee and its exact aggregation property. Comprehensive experiments demonstrate that FedASK consistently outperforms baseline methods across a variety of privacy settings and data distributions.

Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix
Ming Wen Fudan University Shanghai Innovation Institute mwen230m. fudan. edu cn
Jiaqi Zhu Fudan University
j jiaqizhu25@m. fudan. edu. cn
Yuedong Xu i Fudan University
Yipeng Zhou
Dingding Han
Macquaric University
Fudan University ddhan@fudan edu
Shenzhen Loop Area Institue ydxu@fudan. eetlan.eddu.co edu. cn
yipeng. zhou@mq edu.
Abstract
Large language models (LLMs) typically require fine-tuning for domain-specific tasks, and LoRA offers computationally efficient approach by training low- rank adapters. LoRA is also communication-effiieee for federated LLMs when multiple users collaboratively fine-tune global LLM model without sharing their proprietary raw data. However, even the transmission of local adapters between server and clients risks serious privacy leakage. Applying differential privacy (DP) to federated LoRA encounters a dilemma: adding noise to both adapters amplifies synthetic noise on the model, while fixing one adapter impairs the learnability of fine-tuning. In this paper, propose FedASK (Differentially Private Federated Low Rank Adaptation with Double SKetching). novel federated LoRA framework to enable effecctive updating of both low-rank adapters with robust diffcrential privacy. Inspired by randomized SVD, our key idea a two-stage sketching pipeline. This pipeline first aggregates carefully sketched, privacy- preserving local updates, and then reconstructs the global matrices on the server facilitate effective updating of both adapters. We theoretically prove FedASK's differential privacy guarantee and its exact aggregation property. Comprehensive experiments demonstrate that FedASK consistently outperforms baseline methods across a variety of privacy settings and data distributions. Codes are available at https//github..oom/F wtt../......EEEE////e... HWCCHERm//Priaacyeee.... L.M.
Introduction
Federated fine-tuning of Large Language Models (LLMs) presents compelling paradigm for specializing these models on domain-specific, distributed datasets without centralizing sensitive information [41 25 (6). However, the sheer scale of LLMs, often involving hundreds of billions of parameters, renders full-parameter fine-tuning prohibitive for local clients in FL due to memory, computation, and communication constraints (46). To overcome these limitations, Parametee--fffciiet Fine-Tuning (PEFT) methods, particularly Row-Rank Adaptation (LoRA) [20], have gained promi- nence [16). LoRA facilitates efficient adaptation by freezing the pre-trained model weights Wo and training only a small set of low-rank matrices, A E x and B E g2"xr (where r & min minnm,,)) n))
"Equal contribution. Corresponnence to Yuedong Xu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A
A
col
Fix A
Vlg
AW
E
Ao
AW
A
B
W
@ Am Amplified Noise
Limited Learning Ability
Figure I: The dilemma of differential privacy with Federated LoRA: standard federated LoRA amplifies model noise, while fixing one adapter causes insufficient learnability.
The update AW- BA significantly reduces the number of trainable parameters, thereby alleviating local processing and communication overhead in federated LLM fine-tuning scenarios.
Ensuring privacy in federated learning is critical, especially when fine-tuning LLMs 29 The inherent scale and complexity of L.LSs lead to greater probability of encoding and revealing sensitive information from training data compared to smaller models. While raw data are not shared in FL settings, the model updates and gradients transmitted can still potentially leak private client information. Therefore, providing robust privacy protection with theoretical guarantees is essential. Differential Privacy (DP)
as principled framework to achieve this, typically by introducing calibrated noise during the training process.
However, applying DP federated LoRA presents fundamental trade-off. On the one hand, achieving robust privacy often requires adding noise to the gradients of low-rank matrices and B. Despite this, the intertwined nature of these gradients leads significant noise amplification when both matrices are perturbed. This significantly increases the magnitude of noise the resulting AW update, degrading the quality of the model update [34 [2]. On the other hand, existing methods commonly address this noise amplification by fixing one of the matrices (typically A) during training. Such strategies help mitigate the noise, particularly by avoiding the detrimental quadratic noise term. However, it fundamentally restricts the representational capacity of the LoRA parameters to predetermined subspace, thereby impairing the model's ability adapt effectively [13 45 Motivated by this critical dilemma. this paper addresses the following research question: Can we design federated LoRA that achieves differential privacy guarantee, learnability, and communication efficiency simultaneously?
To address this dilemma, we propose FedASK (Differentially Private Federated Low Rank Adaptation with Double SKetching). FedASK is the first federated LoRA framework designed to enable effective updates of both low-rank matrices under robust DP, overcoming the critical noise amplification and limited learnability trade-off inherent in prior methods. Our core insight is a novel two-stage projection pipeline, inspired by randomized SVD [15)
Table 1: Federated LoRA Comparison. DP (V/X): Supports DP. Agg. Type: Aggregation Precision. Client Init.: Parameter state before local training (Sync=Use Global Para, Keep B=Use Local B, Fixed A=Use Initial A, Rand A=Gaussian). "d," is the input feature dimension, and "y" is the LoRA rank.
METHOD DP
AGG. TYPE
AGG. MEMORY
COMMUNICATION
CLIENT INIT.
FEDAVG
X
IMPRECISE
(ldrr Old + Kdrr) (ldr)
O(Kdrr)
SYNC A, B 0, Ag RAND SYNC A, KEEP LOCAL FIX A. SYNC B SYNC A,
FLORA
X
PREcISE
O(K2drr)
FEDSA
X
IMPRECISE
O(Kdrr
FED-FFA
V
PRECISE
O(der)
O(Kdrr
FEDASK
V
PRECISE
O(der)
O(Kdrr
This pipeline achieves precise and resource-efficient aggregation of local LoRA update products, avoiding the issues of directly aggregating noisy matrices. Differential privacy is guaranteed by applying DP-SGD [ with noise addition and clipping to local updates. Critically, the global SVD-based aggregation step not merely an aggregation, but powerfully leverages this privatized information to influence and update both global matrices A and B, facilitating comprehensive adaptation capabilities under stringent DP constraints. Our key contributions are summarized as follows.

---

We introduce FedASK, the first privacy preserving LoRA framework updates both low-rank matri- ces and achieves robust noise suppression, privacy guarantees, and resource-efficient operations.
We theoretically prove the DP guarantee and the precise aggregation property of FedASK.
FedASK empirically achieves consistent performance gains over state-of-the-art baselines in federated LLM fine-tuning, delivering up to an 11.5% performance improvement on MMLU (7B model) and 46% improvement on GSM8K (13B model) under strong differential privacy.
2
Preliminary
This section describes the background on LoRA fine-tuning of LLMs within the context of Federated Learning. We also introduce the concept of DP and its relevance to protecting client data.
Applying DP-SGD directly to the standard LoRA update mechanism presents challenges. Specifically, when independent DP noise is added to the gradients both A and B, these noise [1] components interact quadratically during the construction of the LoRA update AW. This interaction leads to significant noise amplification, as detailed Lemma Existing strategies commonly mitigate this noise amplification by freezing one matrix and optimizing the other This approach successfully avoids the quadratic noise term. However, since the parameter update constrained to a specific subspace, it limits the model's learning capability and hinders effective adaptation to the target task. Tableunderscores FedASK's 's distinct advantages over existing federated LoRA methods. Overcoming the adaptability limitations of fixed-matte DP LoRA methods, FedASK enables dynamic and synchronized update of both LoRA matrices A and under strong DP guarantees, achieving precise aggregation with resource efficiency comparable to better than existing baselines.
2.1 Federated Learning with LoRA
The goal of Federated Learning with LoRA
40
23
minimize the global objective function:
1 F(W) K
Σ k=1
(WW,,, Dz),
where f&(WW, D&) Dx)= 1 [Dx] EGED, ((Wk: represents the local objective function client k. Here, WA denotes the weight matrix of the local model, and D2 is the local dataset at the client The function ((W computes the loss for a single data point E DR, and f& averages this loss across the entire dataset. In LoRA, instead of directly training the full weight matrix Wx. we update by adding a low-rank decomposition product to the pre-trained weights Wo. and using gradient-bar based oracles to optimize the adapters.
Øl dA
OI OW
Øl
Øl
-T
AT. aw
(2)
where A E grxx and B3 E ! are low-rank matrices with rank r (typically r < min(m, n)). and a is scaling factor. These matrices A₁ and B& are the parameters learned during fine-tuning.
straightforward approach for Federated LoRA involves clients locally training their LoRA matrices (As and B₂) and then average them the server [14 28). However, this direct averaging approach causes a misalignment with the precise global model update [38]
1 K (Wo+
W
Wx
Σ K
l B.Ar) # Wo
B
A
K2T
This discrepancy introduces undesirable cross-term noise, particularly in the presence of data het- erogeneity across clients. Prior work [3] (3 has recognized the same issue. Existing methods propose different strategies to address this misalignment. SLoRA B employs multi approach with sparse fine-tuning and SVD-based initialization, which incurs additional computational overhead. FFA-LORA 34 adopts simplified strategy by freezing matrix A and optimizing only B,limiting the model's is expressive capability. FLORA (38) relies on stacking and processing matrices on the server side, resulting in increased server computation and communication costs.

---

Client pipeline
Updatt
Random Proj.
Comm.
Align Proj.
Comm.
Update
Highiighs
B
A
A
B
B
Y
Privacy preserving
Better learnability
B
Theoredically precise
A
extra resauress
Server pipeline
Walt & Angregate
QR
Wait & Apgregate
SVD
>>
Projection
Commun
Figure 2: FedASK Pipeline.
2.2 DP in Federated LoRA
We hereby define differential privacy as the following 9
Definition 1. A randomized algorithm M : D + S satisfies (c, ddiffferrrelt privacy if. for any two adjacent datasets D, D' & 'D differing in one data point, and any output subset S C S
Pr(M(D) E S) S e Pr(M(D') E S) +
where c 2 controls the privacy loss and 0 < < is the failure probability.
DP theoretically ensures that this algorithm's 's output is nearly unaffected by the presence or absence of any single individual's data in the dataset. In the context of federated learning, DP is commonly achieved through Stochhasst Gradient Descent (DP-SGD) 24 This method adds calibrated noise to the gradients computed at each client before aggregation.
Applying DP-SGD directly the standard LoRA update mechanism presents challenges. Specifically, when independent DP noise is added to the gradients of both and B, these noise components interact quadratically during the construction of the LoRA update AW. This interaction leads to significant noise amplification, as detailed in Lemma [ Existing strategies commonly mitigate
this noise amplification by freezing one matrix and optimizing the other [34 22). This approach successfully avoids the quadratic noise term. However, since the parameter update is constrained to specific subspace, it limits the model's 's learning capability and hinders effective adaptation the target task. [able underscores FedASK's distinct ad advantages over existing federated LoRA methods. Overcoming the adaptability limitations of netrixx DP LoRA methods, FedASK enables dynamic and synchronized update of both LoRA matrices A and under strong DP guarantees, achieving precise aggregation with resource efficiency comparable to or better than existing baselines.
FedASK Framework
In this section, we presents FedASK (Differentially Private Federated Low Rank Adaptation with Double SKetching) for federated fine-tuning of large language models using LoRA. We first introduce the overall FedASK framework, which contains the two-stage projection pipeline designed for efficient and precise global updates. Subsequently, we describe the integrated DP framework, which facilitates privacy-preseree training.
3.1 Pipeline: Two-Stage Sketching
The core innovation of FedASK lies in its efficient two-stage pipeline, which employs sketching and projection techniques to accurately compute the aggregated LoRA update (E, Ag) while minimizing resource overhead. FedASK is motivated by the observation that weight updates in large neural networks [19 2). particularly with methods like LoRA, exhibit low-rank structure. This property allows their essential information to be captured efficiently in low-dimensional subspace.
Directly computing and aggregating the full product matrices Bg from each client would be computationally expensive and communication- heavy, as these matrices of the same dimension

---

Algorithm FedASK
Initial global weight matrix Wo: LoRA rank r: LoRA scaling factor Over-sketching parameter Client learning rate 7,: Set of all clients K.; Communication rounds T'; Local update steps m. Boolean DP flag use DP; Input feature dimension de Output: Final global LoRA matrices A B"
Input:
Initialization:
Initialize A E R , R Generate f R4x(++))
for ( to T do
Sample K, S K; Broadcast A 1-1 B²= clients in KG, for each client & E Kc in parallel do if use__ D___DPo is true then
A A 1-1 Bi LoocalUpdat__D([B A Du. a.)
else
A B + LocalUpdatt(A 1-1, B m,,,),
end if
Compute Y (A( Send server. end for
Aggregate Yagg = Ekek, Y! Perform QR: Q QR(QQR( ( (OR(Yog))
Broadcast Q' clients K,,.
for each client & Kc in parallel do Compute Yoros
((B)) Send server.
end for Aggregate Y = -KEE Ygo .99 Compute SVD of (Yage) U.E.V SVD((Y (T) Select leading r components U,, Er, V,
Update global LoRA matrices: B' Q'U,E A' V
end
as the original weight matrices adapted by LoRA. FedASK addresses the challenge by adopting sketching principles inspired by randomized SVD 15 Clients transmit compressed representations instead of full matrices. Since the aggregated LoRA update also maintains a low-rank structure, these sketching techniques can effectively capture its essential information, enabling the server to perform precise reconstruction. The overall process unfolds in two stages.
First stage: Randomized Subspace Sketching. Within the FedASK framework, local clients perform the standard LoRA training procedures to obtain their updated local matrices, denoted as Bf and Ag. To enable global aggregation, FedASK utilizes a shared random projection matrix Q (a+(r))) sketch these updated matrices. Specifically, each client computes the projection by:
Y (A1.))
(4)
Subsequently, this computed projection is transmitted to the server. The server aggregates these received projections and performs a QR decomposition to derive an orthonormal basis Q, which is then redistributed the participating clients. This basis serves to capture the global singular subspace relevant to the exact aggregated B' A' helping to revise the random projection of the first stage and align the local parameter the Global Space.
Second stage: Global Alignment Projection. After receiving the orthonormal basis Q, the clients project their updated matrices onto this basis, yielding updated projection:
projj = (
(5)
The server then aggregates these updated projections and proceeds to perform a Singular Value Decomposition (SVD) on the aggregated matrix to decompose into its singular components. Finally, the global parameters A' and B are updated based the results of the SVD as follows:
B = QUE+, -
(6)
This two-stage process enables FedASK to achieve precise global update that is mathematically equivalent to the average local updates K Σn B&A, within subspace rank + p. The formal thcoretical guarantee for this exact aggregation property is detailed within Section
5

---

3.2 Differentially Private Local Updates in FedASK
To ensure the privacy of individual clients' sensitive data during the training process, we integrate Differential Privacy (DP) into our FedASK framework. As detailed in Algorithm DP is applied conditionally based on the _ flag (line 6). When use _se_DP is set to true, each selected client k E Kr executes the LocalUpdate __DP function (line 7). In the DP-enabled mode, the local update conducts on the B& matrix, while Aş remains fixed for that local training phase (A) + A'-1). ).
In the LocalUpdate DP function, the LoRA matrix Bş, is updated at each step using the DP-SGD mechanism. The process, with gradient clipping and calibrated noise addition, is formulated as:
B = B
7a OI r
max (1.
(1.-
21 ll2 ! ZWE
||
+ +-N(0, (A
OWE
C
where 7 and a denote the learning rate and the LoRA scaling factor, respectively.
Although Aş is not directly perturbed by local noise in DP rounds, the information learned and privatized via updates to Bş is not confined to global B When the server aggregates the client projections and performs the SVD reconstruction the resulting global factors A and B' are both updated. The SVD step effectively decomposes the aggregated privatized information captured primarily within the B3 updates and redistributes across both newly formed global matrices A' V and B = QUE?.
Therefore, FedASK's 's unique two-stage projection and baDeb aggregation allow the knowledge gained under differential privacy influence the global representation captured by A' as well. The formal privacy analysis proving the (e, --guarante for FedASK is presented in the following section.
4 Theoretical Analysis
This section highlights the theoretical analysis of FedASK, focusing on two key guarantees: robust differential privacy and precise aggregation.
4.1 Robust Differential Privacy Integration
Applying differential privacy Low-Rank Adaptation (LoRA), particularly when simultaneously updating both low-rank matrices A, and B, through DP-SGD, presents significant challenge. Lemma I provides a formal quantitative analysis this noise amplification.
Lemma (Approximate Noise Power in LoRA Update with DP-SGD), Consider LoRA parameters A, E Rexdi and B, E ddd (where di > r). Under DP-SGD with learning rate t,, noise multiplier a, clipping C, and batch size Brizer independent Gaussian noises Ca. EB (effective per-component variance C²2 Ba are added to gradients VA, BB respectively. Let AW Wnoiee be the noise component of the resulting LoRA update.
20 E3 AW moise [2:] 17
10 B size
B dir(IA B
10
Term 1: Linear Noise
Term 2: Quadratic Noise
While Term reflects standard linear noise in DP-SGD, Term 2 arises from noise interaction and can dominate with large or scaling with d.. This causes the LoRA update's 's Signal-to-Noise Ratio (SNR) to degrade significantly faster (1/a²) than individual gradient SNRs (1/a²), revealing a critical noise amplification problem in standard DP-LoRA.
Lemma highlights critical challenge in standard DP-LoRA: simultaneously perturbing both low-rank matrices A, and B, leads to dominant quadratic noise term, severely degrading the LoRA update's 's signal-to-noise ratio and impacting model utility. FedASK involves focusing local DP-SGD updates primarily on the B, matrix, while A is primarily updated on the server through the two-stage aggregation pipeline with global SVD-based update. This strategy preempts local generation of the problematic quadratic noise term. The formal (e, diifffrrret privacy guarantee for FedASK established in Theorem[]

---

Theorem 1 (Privacy Guarantee of FedASK), Suppose the gradient sensitivity is - 1. FedASK (Algorithm guarantees that the final global LoRA matrices . are (e.f)ddifrrenttaally private with respect to the joint dataset D = - DE of K total clients, provided the variance o² of the Gaussian noise added local gradients satisfies:
4 m 9K T ln(2/6) In(2Tqx00)
8² =
K
where qK. is the client sampling ratio per communication round (total T rounds), and qo is the data sampling ratio per local update (total m local updates per client per round).
Theorem establishes the end-to-end (e, didfifferet privacy for FedASK. Noise variance o² scales consistent with established DP-SGD analyzes in federated learning [32]. The derivation rigorously tracks privacy loss using Rényi Differential Privacy (RDP), briefly with the following arguments.
P1)ve the RDP guarantee for two sketches Y and Yo released by client k within a single communication round, relative to its local data Dx; (2) convert this per round, per client RDP into an intermediate (eo, -DP guarantee; (3) applying advanced composition theorems for (e. 6)-D over communication rounds; and (4) incorporate privacy amplification from client subsampling (ratio qi) to achieve the final stated O) guarantee. The detailed proof is provided in Appendix A.2
4.2 Aggregate Precision Guarantee
Aggregating local updates in federated LoRA presents a key challenge: achieving high accuracy while minimizing communication and computational overhead. Unlike conventional methods introducing approximation errors require extensive resources (e.g., naive SVD reconstruction), FedASK's two-stage sketching mechanism enables mathematically exact aggregation as formalized in theorem 2
Theorem 2 (Aggregate Precision of FedASK), Let dg dim( HimmspannUUt Range(Bx)))
the random projection matrix Q E (nx((+)) is a standard Gaussian random matrix with over-sketching parameter p satisfying p dn r 22 2, then FedASK guarantees that the global update before
trancation AW B equals the exact average of local updates AW = >K
K
B& ie..
IWW'-WW1s -AW|E= =,
where I| denotes the Frobenius norm.
Theorem provides a theoretical guarantee: the global LoRA update (AW') computed by FedASK precisely matches the true average of all participating clients' local updates (AW), provided the specified condition for the over-sketching parameter p is met. Our empirical evaluations (detailed in Section (.4) suggest that this condition for exact, near-exact, aggregation is often satisfied with relatively small value for p. This implies that the theoretical precision highlighted by the theorem is practically achievable with minimal over-sketching overhead. Thus, FedASK delivers significant efficiency gains through its projection-base based communication without compromising aggregation fidelity in practical settings.
5 Experiment
To rigorously evaluate our proposed method, we conduct a comprehensive set of experiments across diverse tasks and models. For natural language processing and mathematical reasoning, utilize two large-scale Llama-2 models: the 7B and 13B versions [35]. The 1-ama-2-7BB model is fine-tuned on the 15K dataset [44] and assessed on general language understanding benchmarks, including MMLU [17]. DROP 77 and HumanEval [5]. Concurrently, the Llama-2 2-B3B model undergoes further fine-tuning with Chain-of-Thoughh (CoT) prompting 39] on the MetaMathQA dataset [2] with its mathematical reasoning capabilities evaluated using the GSM8K (6). GSM8K-hard and MATH (1] benchmarks. To demonstrate the versatility of our approach, FedASK, beyond NLP and standard Supervised Fine-Tuning (SFT), we also conduct experiments with Vision Language Models and Reinforcement Learning from Human Feedback (RLHF). In this setting, we perform Direct Preference Optimization (DPO) on -lava-1.5-77 using the SPA-VL. safety preference alignment dataset [47. The evaluation for this task involves MM-SSfetyBench 27 measure resilience to jailbreak attacks via an Attack Success Rate; SIUO (36] assess safety in cross modsl reasoning;
7

---

and BeaverTails-V [2]] to provide separate win-rates for harmlessness and helpfulness. Across all experiments, conditions are systematically varied encompass different privacy budget levels (Section 5.1). degrees of data heterogeneity (Section 5.2) and system robustness (Section 5.4)
All evaluations are performed on NVIDIA Tesla A100 GPUs, utilizing half- half-precision maximize computational efficiency.
Baselines. We compare FedASK with five baseline methods: (1) FedAvg [30]
Clients perform
local SGD, and the server applies weighted parameter averaging. (2) FFA-LORA (34]: Clients train
one low-rank matrix locally, mitigating noise accumulation. (3) FedSA-LoRA 13 Clients locally
update both LoRA matrices, with one matrix being transmitted. (4) FedProx [26]: Introduces
proximal term in the local client loss mitigate the effects of data heterogeneity, (5) Scaffold [33]
Employs control variates to correct client-server gradient disparities, reducing client drift.
5.1 Model Performance with Differential Privacy Guarantees
Experiments with Llama-2-7B on homogeneous data standardized settings (e.g., B 8, 10 local steps, 400 rounds) and common LoRA configurations (r = 64, a 128), detailed in the appendix. We perform grid search for learning rates and explore DP budgets Results are in Table For Llama-2-13BB, we adjust settings to B = 6, 800 total rounds and r 128, with other hyperparameters unchanged. Results are Table
{1,3, 63, following (34).
The introduction of privacy-preseringg mechanisms leads to performance degradation in several baseline methods. In contrast, FedASK algorithm consistently outperforms these methods, regardless of whether privacy protection is enabled, demonstrating its superior generalization under privacy constraints. Interestingly, find that under certain conditions, adding DP noise improves performance compared to the noiseless case. We attribute this to DP noise serving as implicit regularization, thereby enhancing model robustness.
Table 2: Performance Comparison of Different Algorithms with DP Budgets on (lama-2-77
Task
Priv. Budget
FedASK
FodAvg
FFA-LoRA
FodSA-LoRA
FedProx
Scaffold
Non- Private
46.15
45.13
45.98
45.19
44.98
45,65
45.80
42.07
42.76
42.9
41.99
43.41
MMLU
3
46.25
41.49
42.72
41.13
43.17
42.47
6
45.78
43.34
42.82
42.84
43.70
43.80
Non- Pri-Private
32.09
30,2
31.34
31.23
30.99
30.01
31.23
29.55
29.10
31.04
29.51
29.66
DROP
32.08
29.26
28.40
29.40
28.50
28.75
31.36
29.30
29.40
29.26
27.57
30.20
Non- Private
15.24
11.59
14.02
12.2
12.2
14.63
15.24
12.80
12.20
13.41
12.20
9.76
Human-Eval
15.24
10.37
10.98
10.98
13.41
11.59
15.85
11.59
12.20
12.80
10.98
12.20
Table 3: Performance Comparison of Different Algorithms with DP Budgets on Llama-2-133
Task
Priv. Budget
FedASK
FedAvg
FFA-LoRA
FedSA-Lot
FedProx
Scaffold
Non-Private
50.0
48.5
48.4
47.2
47.8
45.6
22.7
15.5
14.2
12.2
15.2
16.1
GSM8K
= 3
24.8
16.5
20.0
20.2
18.0
15.8
= 6
27.7
19.3
20.2
17.3
20.1
20.3
Non-Private
28.7
25.8
23.2
23.4
26.1
21.8
13.0
8.8
8.0
6.6
7.2
9.1
GSM8K
12.6
11.1
10.5
11.3
10.8
11.0
=
16.9
10.5
9.2
10.2
10.9
10.8
Non-Private
10.3
10.8
10.7
11.7
6.9
5.2
5.8
5.6
5.6
5.8
Math
3
6.6
6.1
6.0
5.9
6.4
5.5
7.6
6.2
6.0
5.9
6.7
7.1
8

---

5.2 Model Performance with Data Heterogeneity
To evaluate data heterogeneity's impact, we experiment with IID and three non-lID scenarios, using 10 clients with 2 selected per round. In IID settings, data is randomly partitioned. For non-IID settings, we use Dirichlet distribution Dir(a) with a E (0.1, 0.5, 1.0), following prior work [3]
To evaluate robustness under DP in these non-IlI environments, we set the privacy budget a common value for balancing privacy and utility. Other settings follow Section [.1 The experimental results in Table Ademonttrat FedASK's prominent performance. Across all tasks and data distributions (IID and non-IID), FedASK consistently outperforms baselines, underscoring its effectiveness and robustness. This superior performance stems from FedASK's 's simultaneous local updates and global aggregation of both and B matrices. While previous research [3] suggests that matrix A learns global information and matrix B captures local specifics, FedASK's design enables continuous interaction and fusion of these distinct knowledge types. This inherent information fusion capability enhances the model's adaptation to heterogeneous data distributions.
Table 4: Performance comparison across different data distributions on various tasks.
Task
Data Dist.
FedASK
FedAvg
FFA-LoRA FedSA-LoRA
FedProx
Scaffold
IID
46.25
41.49
42.72
41.13
43.17
42.47
Dir(0.1)
46.04
42.69
42.54
44.27
42,61
43.05
MMLU
Dir(0.5)
45.95
42.11
41.46
42.72
42.98
41.97
Dir(1)
46.01
42.96
43.23
41.04
42.98
41.71
IID
32.08
29.44
28.40
29.40
28.50
28.75
Dir(O.1)
31.01
28.34
30.10
28.58
28.18
28.27
DROP
Dir(0.5)
31.15
29.18
29.26
27.83
28.23
29.45
Dir(1)
31.58
30.02
28.93
29.29
29.82
30.52
IID
15.24
10.37
10.98
10.98
13.41
11.59
Dir(0.1)
13.41
7.32
10.98
12.8
13.41
7.93
Human-Eval
Dir(0.5)
14.63
10.98
13.41
12.8
14.63
9.76
Dir(1)
14.02
9.76
10.98
10.37
10.37
9.15
5.3 Model Performance with VLMs task and RLHF
To evaluate the efficacy of our method vision-language tasks and preference alignment, conduct
experiments using the Llava-1 1iiee-15-77 .,5-BB model. We employ DPO, a form of RLHF, to align the model with safety preferences using the SPA-VL dataset. The FL setup involves 10 clients (IID), selected per round for 600 rounds, and 10 local steps. We apply LoRA to q_proj, v_proj, and mm_ jim__rojeeet jertrr modules, freezing the vision_towes
The results in Table 5 demonstrate FedASK's superior performance in the VLM aligmment task under privacy constraints. At 6, FedASK consistently outperforms all baselines, achieving the highest helpfulness (56.36) and harmlessness (65.36) on BeaverTails- V, the lowest ASR (43.40) on MMsafety, and leading scores SIUO. This advantage stems from fundamentally learning process. FedASK avoids the representational bottleneck of fixed-matrix methods, which is particularly limiting in complex, multi-objective tasks like preference alignment. The global SVD reconstruction acts as a powerful information distillation step, re-decomposing the aggregated private updates into new, more optimal shared basis for both A and B. Furthermore, the inherent DP noise serves as an implicit regularizer, preventing the model from overfitting to specific client preference data and promoting more generalized alignment. This synergy preserves the model's full expressive capacity while enhancing robustness, explaining its significant margin over baselines even under stricter privacy budgets such as
expressive
5.4 Effect of the Sketching Dimension
Theorem guaraantess that FedASK achieves precise global aggregation provided the over-sketching parameter p meets a specified condition, our empirical evaluations aim to demonstrate that this precision is practically attainable with minimal p. underscoring FedASK's efficiency. To this end, we conduct two sets of experiments using the Llama-2-7B model and identical settings from Section to assess p's impact on both downstream task performance and direct aggregation fidelity.

---

Table 5: Performance comparison of DPO with LoRA across different privacy budgets.
Beavertails-V
MMsafety
SIUO
Algorithm
Helpfulness
Harmlessness
ASR ())
Effectiveness
Safety
e=6
e=1
e=6
e=1
e=6
6=1
e=6
6=1
e=6
e=1
FedASK
56.36
53.65
65.36
62.98
43.40
46.93
87.42
85.62
31.14
28.74
FedAvg
51.53
45.50
53.66
48.89
48.24
50.80
83.23
80.23
26.94
25.14
FFA-LoRA
54.66
52.38
63.32
61.96
44.39
48.45
86.23
83.83
29.34
27.54
Scaffold
51.02
48.97
52.55
46.67
50.80
51.43
83.83
76.04
27.54
23.95
First, to evaluate downstream task performance, we test FedASK with varying over-sketching param- } on the MMLU benchmark. This is performed under challenging Non- IID setting (a = 0.1) with differential privacy (c=3) 3). Second, to directly assess aggregation fidelity, compare FedASK configured with over-sketching (p 0) against FedAvg. This latter experi- ment is conducted without DP, across varying numbers of selected clients (K, E {2, 5, 10, 15, 20}) and data heterogeneity levels. After 30 local update rounds by each client, measure aggre- gation fidelity using the cosine similarity between FedASK's reconstructed global LoRA update (AW* 13 At) and the ideal average of all the local LoRA updates (AW* */
p E {0, 32, 64. 96, 128)
K, ΣkEK, B B(A)).
Feclog
(a)
Figure 3: Impact of over-sketching on FedASK. (a) MMLU score of FedASK versus p. (b) Aggregation fidelity of FedASK (with = 0) and (c) FedAvg, measured by cosine similarity with the ideal mean of local updates (1.0 indicates perfect fidelity). Subplots (b) and (c) vary selected clients (K,) and Non-IID degrees.
The empirical results, presented in Figure 3 illustrate that the over-sketching parameter p has minimal influence on the effectiveness of FedASK. Firstly, (a) shows that FedASK's ' MMLU performance remains remarkably stable across the tested p values (ranging from 45.63 at p to 46.06 at p 128), even under conditions of significant data heterogeneity and differential privacy. This stability demonstrates low sensitivity to P to achieve robust downstream utility. Secondly, FedASK's ' inherent design ensures exceptional aggregation fidelity, even with zero over-sketching (p 0). As shown (b), FedASK consistently achieves near-perfect cosine similarities (with an error the order of 10-7) across variety of selected clients (K,) and Non-IID degrees. This performance consistently outperforms FedAvg, presented (c), which typically ranges from 0.92 to 0.96. These findings affirm FedASK's 's practical efficiency and the attainability of its theoretically guaranteed precise aggregation with negligible over-sketching overhead.
Conclusion
This paper introduces FedASK, novel federated LoRA framework that successfully resolves the trade-off between noise amplification and learnability in private federated fine-tuning By employing a two-stage sketching pipeline, FedASK enables the differentially private and effective update of both LoRA adapters. Our theoretical analysis and comprehensive experiments demonstrate FedASK's superior performance in achieving precise aggregation and robust downstream utility with strong privacy guarantees and practical efficiency. Future studies will focus on extending its applicability to broader range of model architectures and complex training tasks, as well as exploring further refinements its privacy-utility trade-off.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Natural Science Foundation of China under Grant 62472103 and the National Key Research and Development Program of China under Contract 2024YFA1610900
References
[1] M. Abadi, A. Chu, 1. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Com- puter and Communications Security, CCS '16, page 308-318, New York, NY, USA, 2016. Asso-
ciation for Computing Machinery. ISBN 0781450341394. attpss://dos
doi: 291144559777749999888888
URL
[2] A. Aghajanyan, L. Zettlemoyer, and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020. URL https:ss///aaiiv..oo http:////gg/bb//////eeeee 2ttps./222..131 13255
[3] S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K.-.. Song, M. El-Khamy, and S. Avestimehr. Slora: Federated parameter efficient fine-tuning of language models, 2023. URL httpttttt////xxii..o. orgo /bs/ 7308.06522
[4] B. Balle, G. Barthe, M. Gaboardi, J. Hsu, and T. Sato. Hypothesis testing interpretations and renyi differential privacy. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 Proceedings of Machine Learning Research, pages 2496-2506. PMLR, 26-28 Aug 2020. Uttpttts:: / pttpp//iooeeg....mmom mlr grees/vv006
balle20a. html
[5] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, et al. Evaluating Large I Language Models Trained on Code. arXiv e-prints, art. arXiv:2107.03374 July 2021. doi: 10. 485500arXiw.2107.03.. 10.3744.
[6] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert. J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training Verifiers to Solve Math Word Problems. arXiv e-prints, art. arXiv:2110.1416
Oct. 2021. doi: 10.48550/arXivv211 14.16
[7] D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, Singh, and M. Gardner. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. arXiv e-prints, art. arXiv: 1903.00100 Mar. 2019. doi: 10.48550/arXiv. 11015//aaXii.111..0000 1903.0016..
[8] C. Dwork. Differential privacy. In M. Bugliesi, B. Preneel, V. Sassone, and I. Wegener, editors, Automata, Languages and Programming, pages 1-12, Berlin, Heidelberg, 2006. Springer Berlin Hcidelberg. ISBN
[9] C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science. 9(3 (334):211 1077, 2014. ISSN 1551-305X. doi: 10.1561/0400000044, URL //dx: dww.doo....oo.. org/10. 1561/0400000000000
[10] C. Dwork, G. N. Rothblum, and S. Vadhan. Boosting and differential privacy. In 2010 IEFE 5 Anmual Symposium on Foundations of Computer Science, pages 51 2010. doi: 10. 10/1000S...0000112
[11] Fu, Y. Hong, X. Ling, L. Wang, X. Ran, Z. Sun, W. H. Wang, Z. Chen, and Cao. Differentially private federated learning: A systematic review. arXiv preprint arXiv: 08299,, 2024.
[12] P. Guo, S. Zeng, Y. Wang, Fan, Wang, and L. Qu. Selective aggregation for low-rank adaptation in federated lcarning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrene
net/ net/forum?? iitimmmmmdiiiiiiiiemee
[13] P. Guo, S. Zeng. Y. Wang. Fan. Wang. L. Qu. Selective aggregation for low-rank adaptation in federated learning, 2025.
org/ abs/2410........ 01463
[14] Z. Guo, Y. Zhang. Z. Zhang, Z. Xu, and I. King. FedLFC: Towards efficient federated multilingual modeling with LoRA-based language family clustering. In Duh. H. Gomez, and S. Bethard, editors. Findings of the Association for Computational Linguistics: NAACL 2024, pages 1519-1528, Mexico City, for
Mexico, June 2024, Association for Computational Linguistics. doi: 10. 1/.1865//1/2024..00 http://22...iiiinnnmaaa..... org/ 2024. findinnnnnnnnn naacl .8/
htl.stttss: http::::///aaaaahhlloooo
[15] N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):211 288, 2011. doi: 10.1137/ 090771806. https://:: org/10. 1137/090771806
11

---

[16] Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang. Parameter-efficien fine-tuning for large models: A comprehensive survey, 2024. URL https: http:://aavvvvvr.. wwtg////////00 14608
[17] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song. and J. Steinhardt. Measuring Massive Multitask Language Understanding. arXiv e-prints, art. 2xXiv:2009033300 Sept. 2020. doi: 10.48550/ arXiv:2009.055 033000.
[18] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart Tang, D. Song, and J. Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv e-prints, art. arXiv:2103.03877 Mar. 2021. doi: 10.48550/rXX..2113.00
[19] E. Hu, Shen, P. Wallis, Allen-Zhu, Li, Wang, L. Wang, and W. Chen. LoRA: Low- Rank Adaptation of Large Language Models. arXiv e-prints, art. arXiv:2106.09885, June 2021. doi: 14485500arxi.2216....mmm
[20] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3,, 2022..
[21] J. Ji, X. Chen, R. Pan, C. Zhang, H. Zhu, J. Li, D. Hong, B. Chen, J. Zhou, K. Wang. J. Dai, C.-M. Chan, Y. Tang, S. Han, Y. Guo, and Y. Yang. Safe rlhf-v: Safe reinforcement learning from multi-modal human feedback, 2025. URL bRtpps:/://arxii wwt./orr//////ooeoooe 17682
[22] T. Kang, Z. Wang, H. He, J. Zhang, S. Song, and K. B. Letaief. Federated low-rank adaptation with differential privacy over wireless networks, 2024. TRLphtts::/ httts://arxiv org/ aa////bbb///./////////////////// 2411.07806
[23] Kuang. B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and Zhou. Federatedscope-llmm A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, page 5260 5271, New York. NY, USA, 2024, Association for Computing Machinery. ISBN 9798400704901. doi: 10.1 N0.11453367758.3666 36711533. httpss//:://do org/ www.11137/3368..../ 3671573
[24] J. Lee and D. Kifer. Concentrated differentially private gradient descent with adaptive per-iteration privacy budget. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18 page 1656-1665, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781 9781450355520. doi: 10.1 11.114//21191193220000 3219819.3222 3220076
[ttpp://:////o http///oooo.rg/...// 1145/
[25] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60 60, 2020. doi: 10.1 00.1000/MP.2222227777744
[26] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. In I. Dhillon. D. Papailiopoulos, and V. Sze, editors. Proceedings of Machine Learning and Systems, volume 2. pages 429-450, 2020. URL http::ttt/s/ttttm /ttp:ppps///prooceednngass paper wwwple/pppeee//0/////
www.mmaalooor.oo Paper. .df
[27] X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao. Mm-safetybench: A henchmark for safety evaluation of multimodal large language models. In A. Leonardis, E. Ricei, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, editors, Computer Vision ECCV 2024, pages 386-403, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72992-88
[28] X.-Y. Liu, R. Zhu, D. Zha, J. Gao, S. Zhong, M. White, and M. Qiu. Differentially private low-rank adaptation of large language model using federated learning. ACM Trans. Manage. Inf. Syst. 16(2). Mar. 2025. ISSN 2158-656X. doi: 10. 15//366668
(ttpssttt:://eat . org/10. 10. 1145/3682068
[29] N. Lukas, A. Salem, R. Sim, S. Tople, I.. Wutschitz, and S. 7anclla-Béguelin Analyzing leakage of personally identifiable information in language models. In 2023 IEFE Symposium Security and Privacy (SP). pages 346-363, 2023. doi: 10. 10.1109/SP4211... 2.222..23...22 10179300000
[30] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agliera y Arcas. Communication Effiicien Learning of Deep Networks from Decentralized Data, arXiv e-prints, art. doi: X0.48550/arXiv.. 1602.05629.
Feb. 2016.
[31] I. Mironov. Rényi differential privacy. 20/7 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263-275, 2017. doi: 10.1 109/CSF:2017.11.
[32] M. Noble, A. Bellet, and A. Dieuleveut. Differentially private federated learning on heterogeneous data. In G. Camps-Valls, J. Ruiz, and Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages PMLR, 28 30 Mar 2022. URL httpps://proceedineg mlr.. www.pppeessviii. press//v15/nnobe222
html
12

---

[33] S. Praneeth Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. Theertha Suresh. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. arXiv e-prints, art. 1911v:1910.06378, Oct. 2019. doi: 11.85550arxi.111000666
[34] Y. Sun, Z. Li, Y. Li, and B. Ding. Improving lora in https://arxiv. org/ abwg/3////4 12313
federated learning, 2024, URL
[35] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava. S. Bhosale, et al. Llama Open foundation and fine-tuned chat models, 2023. URL htttt://aaarlii org/ abs ab///3///// 2200/0988700
[36] S. Wang, X. Ye. Q. Cheng, Duan, Li, Fu, Qiu, and X. Huang. Safe inputs but unsafe output: Benchmarking cross-modality safety alignment of large vision-language model, 2025. URL https
wtapg//b////aae 2406. 15279
[37] Y.X. Wang, B. Balle, and S. P. Kasiviswanathan. Subsampled renyi differential privacy and analytical moments accountant. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty-Second International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1226-12235 1235. PMLR, 16-18 Apr 2019. URL https://proceeddin
mlr
press/v89/ hatps///99/aaag9bbbb/a wtwp/wangnlbla.......
[38] Z. Wang. Z. Shen, He, Sun, H. Wang, L. Lyu, and Li. Flora: Federated tine-tuning large language models with heterogeneous low-rank adaptations, 2024. URL MtRllattts:///rrtti org.
org/abe////a abs/ 2209/ 05976
[39] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Iehter, F. Xia, E. Chi, Q Lc, and D. Zhou. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models, arXiv e-prints, art. arXiv:2201.11900 Jan. 2022. doi: 10.48550/rXXwv22001 11903.
[40] R. Yc, W. Wang, J. Chai, D. Li, Z. Li, Y. Xu, Y. Du, Wang, and S. Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference
Knowledge Discovery and Data Mining, KDD '24, page 6137-6147, New York, NY. USA. 2024. Association for Computing Machinery. ISBN 9798400704901. doi: 10. 10.1145/3637528. 367 1582. URL https://doi. http://///ooooogg....o 1145//637///3325 3671158
[41] x. Yin, Y. Zhu, and J. Hu. A comprehensive survey of review, and future directions. ACM Comput. Surv., 54(6). July 2021. ISSN 0360-0300, doi: 10.1145/ 3460427. URL http:: //dotp org/10 1145/346042)
federated learning: A taxonomy,
[42] L. Yu, W. Jiang, Shi, Yu, Z. Liu, Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. Meta- Math: Bootstrap Your Own Mathematical Questions for Large Language Models. arXiv e-prints, art. arXiv:2309. 12284, Sept. 2023, doi: (0.48550/rxx..2309.222 122844.
[43] J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang. T. Yu, G. Wang, and Y. Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024 2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 1915-6919, 2024. doi: 10.1 10.11091CASS444455. 2024. 10447454.
[44] J. Zhang, S. Vahidian, M. Kuo. Li, R. Zhang. T. Yu, Y. Zhou, G. Wang, and Y. Chen. Towards building the federated gpt: Federated instruction tuning. 2024. httpt//s://aaa..or///aaa//////// 056444
[45] L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li. LORA-FA: Memory-efficient low rank adaptation for large language models fine- tuning, 2024. Ltticcps///ppeeevveeeeeee///oree RddkrrtM
[46] S. Zhang, L. Dong. X. Li. S. Zhang. X. Sun, S. Wang, J. Li, R. Hu, T. Zhang. F. Wu. and G. Wang. Instruction tuning for large language models: A survey, 2024. URL htttt:://sxxxiiiii h/ap:ii///////aa/aaa http://aaxvvv..r/r//.. abs/23300 10792
[47] Y. Zhang, L. Chen, G. Zheng, Y. Gao, R. Zheng, Fu, Z. Yin, S. Jin. Y. Qiao, X. Huang, Zhao, T. Gui, and J. Shao. Spa-vl: A comprehensive safety preference alignment dataset for vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19867-19878, June 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately state the paper's contributions, includ- ing the FedASK framework and its theoretical guarantees.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Limitations are mentioned in Section 6 and detailed discussion is provided in Appendix
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: Section 4 presents the theoretical analysis, with complete proofs available in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section details the experimental setup using open-source datasets, ensuring reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be open-sourced within a month and included the supple- mentary material package.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides/ Soaaad/ss//oeeSSummssssiooooee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccccuclccuuueeeeeeeeeeeee Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All necessary experimental settings and details specified in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars will be provided in the supplementary material due to main text page limitations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: System efficiency, including compute resources, is analyzed in the supplemen- tary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp/s//aarrrss.cc/ppbb//cccct cttiiiiiieeete ines?
Answer: [Yes]
Justification:
The research conforms to the NeurlPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This foundational research assessed to have no direct societal impact at this stage. We use the open access data and models that has been investigated under risk evaluation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed framework does not pose immediate high risks requiring specific safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets are properly credited, referencing their original sources.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new datasets or models assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The research utilizes existing datasets and does not involve direct human subject participation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: IRB approval is not applicable as the research does not directly involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: LLMs were utilized solely for editing and formatting purposes in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Theoretical Proof
A.1 Proof of Lemma
Proof. Let A₂ E Rrxd: and Br R4xr be the LoRA matrices. The noisy updates under DP-SGD with learning rate 1) are,
A A. 7VA. EA): B++1 B,- BV-n(VB +B),
(9)
(10)
where CA. εB are independent Gaussian noise matrices with i.d.. entries (V,0, o2/5), and de1 = O² BB, size. The noise component in the LoRA update AW = B141 B,A, is found by substituting p and (TO).
(B, n V. + (A) (VA + EA)) B,A,, AW
- VB,A) B..AA (Signal part)
mm(BBE + +BAA) + + &B VA, + (BEA)
(Noise part).
Thus, AW noise
X YY, where,
X -nn(AA + EnA.);
(11)
= (i((A + V + (BEA)-
(12)
The expected noise power is PN B||xX- + YI E|| X EII +
For the linear noise term power [[[ ] using 11
|||| P = 1 13 BBCA + {BA
" (E] A1)))=
due to E[((BEA. = 0 from noise independence and zero mean. We have [B(A = dio2, Ir and ElL = d:o2, So,
E E [2]] Tr(BBE(AAAABB - [Bll?
En [P2] = Tr(A,A B(CEER) = dioessy1Aeel
This yields,
E X| = (1 A.l + BI.
(13)
For the cross term (X, using 11 and 12
E (X,Y) (B.
V &B V. + rn
Non-zero expectations arise from,
= T = dio;; Tr(B, VB:), E[(EaA,, &B V Ar) F] = Tr(A E(E A.) dio / Tr (AA A.).
Other terms are zero since the independence of noise. Thus,
2E (X, Y) 227 2 VB,) + Tr VA)).
(14)
For the higher-order noise term power [B Y |. using 12 the dominant component is,
= " =
(15)
(16)
(17)
= =
(18)
Other terms in || are O(1' (0 1, +
21

---

Combining the dominant terms from 13 and the dominant part of E and noting the cross-term (14) often less dominant, we approximate Pv. Substituting oef o² /PC//B size! we get,
[ (derived in
(18),
0² El| AW noise [P] 2 of
C B1 size
A + |[ +
dir,
(19)
B size
which matches the expression stated Lemma I, highlighting the linear and dominant quadratic noise terms.
A.2 Proof of Theorem
To facilitate the privacy analysis of FedASK, we first recall several fundamental concepts and properties of differential privacy. We leverage the advanced composition theorem and subsampling theorem for (e, d)-differelt privacy:
Lemma 2 (Advanced Composition [IO). Let M1,..., Mg be sequence of k adaptive mechanisms, where each Mi provides (e, -.ff-eiffrrlntial privacy. For any - ddifferrential private, where
the composed mechanism
M- = (M....., MM) is (Ceotats kō 8')
22 In(1/8)c kele- 11.
Cootal
I then for any 8' > 0, the composed mechanism M is (Chouass kō - 8') ddiffeerretl private, where "oma = /2k (n1/8') + kc?
Lemma 3 (Privacy Amplification by Subsampling (37]) Let M be an (c, 8) diffferenntill private mechanism. If M run on random sample of size m drawn uniformly without replacement from dataset of size N (where m < N), let m/N be the sampling ratio. Then, the subsampled mechanism M Mabrample is (c', 8') d-dffferentially (ifiimirtatt
private, where,
è' = log(1 + 7( - 1)),
8' = 70.
While (e, 8)-DP provides a worst-case privacy guarantee, analyzing the precise privacy loss under composition, especially for Gaussian mechanisms, can be complex. Rényi Diffcrential Privacy (RDP) 31] offers convenient framework for tracking privacy loss, providing tighter bounds under composition. We define RDP as follows:
Definition (RDP (31)) A randomized mechanism M D R satisfies (a, R)-Rényi Differential Privacy (RDP) for any neighboring datasets D, D' E D (differing in one individual's 's data), and for all 1,
Da(M(D)IM(D") R
where (a(P())) = - In Q(x)
[(
P(x) [())
is the Rényi divergence of order
RDP possesses several useful properties that simplify privacy analysis.
of RDP BI). Let M: :D R be a mechanism that satisfies (a, R)- RDP. Let R + R' be an arbitrary randomized mapping (a post post-processing function). Then the mechanism g o M D > R' also satisfies (a,, R)-RDP.
Lemma 4 (Post-processing
Lemma 5 (Adaptive Sequential Composition of RDP Bil). Let M1 D - R1 be mechanism satis- fying (a, R1) RDDP. Let LetM2 R2 xD - R2 be mechanism such that any fixed output or E R2 of Mi. M2(01.)) satisfies (a, RRD- Then the mechanism M(D) = (Mi(D), M2 (D), D)), which outputs the pair (01, 02) where ~ Mi(D) and 02 ~ M2(0i, D), satisfies (o,, Ri + R2)- RDP.
Lemma 6 (Conversion from RDP to (e. DP 44.
Ifa randomized mechanism M satisfies (a, R)-
RDP, then it satisfies (R + In((a - 1)/a) (In 8 + lnaa/(a - 1): for any < <
The core mechanism for achieving privacy in FedASK involves adding Gaussian noise. The RDP of (subsampled) Gaussian mechanism is characterized as follows:
22

---

Lemma 7 (Approximate RDP for qp- po-SSubampled Gaussian Mechanism (32). Consider a qp= subsampled Gaussian mechanism with noise variance og Under the assumption that the data subsampling ratio qo is small (i.e, qD o()),, and assuming the mechanism operates a high privacy regime (Assumption 1-(üi) in (321), for any real number a > 1, the mechanism satisfies (a,, -RDP where R' = O (
The proof proceeds in several steps. We first amalyze the RDP guarantee for the information transmitted by a single client k in one communication round with respect to its local dataset Dj- Then, we analyze the RDP of the aggregated global update at the server. Finally, compose the privacy loss over T communication rounds, incorporate the amplification due to user subsampling, and convert the total RDP to an (e, 8)DP guarantee.
Proof.
Step 1: RDP of Local Updates and Transmitted Sketches by Client w.r.t. Dr.-
Privacy guarantee for first sketching Client & executes LocalUpdateDP (using m local steps, data subsampling ratio s, fixed A A and noise parameter σ²) on its private dataset Dx compute
the local LoRA matrix B By Lemma and Lemma B& satisfies (a, (( ( (a++1) RDDP w.r.t. DE. The first sketch, Y = B (A) is derived from Bi by post-processing (Lemma 4) since A and Q are independent of D in this context. Thus, y- also satisfies (a, R (a))-RDP w.r.t. De. O (vb(a+)))
o²
where R(
(a) R
Privacy guarantee for second sketching: After the client sends Yproj to the server, the server computes an orthonormal basis Q' QR(E,eK, Y! and broadcasts Q' back to the partic- ipating clients, including the client k: The client then computes its second sketch Yo (A)) ((B))) Q'). The calculation of Ypros utilizes the already privatized matrix Bi the public matrix AL. and the received matrix Q'. For client k. Q' is an external input provided by the server, and this computation does not involve fresh access to its private dataset Dj. Consequently, according to Lemma 6 Ygros also satisfies (a. R (R))-RDP with respect to De.
So we could have that, for the local uplink transmitted information, y,T and Y-roo satisfy the (a, (0) (D)) w.r.t. Dg, where
mo (a R (a) O( m + 1))
(20)
Step 2: DP guarantee of B²) w.r.t the Joint Dataset D = DR.
Dr.
The server aggregates the second sketches to form ÿz =EEEK, Apni Based on the (a, c (a))-
(2)
agge
RDP guarantee for each client's 's transmitted information w.r.t. its local data (Step 1) and the disjoint- ness of client datasets, the mechanism producing an appropriately scaled aggregate over the K2 = [C
participating clients yields (a, (a)) for Y
w.r.t. the joint data DK. Ukek, Dr. where
(a:) = (a)/K, = ( agg
(mff(a+t +
R
(21)
K2o²
Since the global LoRA matrices (A', B²) are derived via SVD of Yage (a pospoopoocesse post-processing step. Lemma
4 they inherit this RDP guarantee. Consequently, by applying the RDP DP conversion (Lemma
6), for any < 60 < 1, (A', 144,10 B³) satisfy (e0 ,6)) 5DD- w.r.t. Dx,, with 6n given by:
R(O)) (a) + In( In(a-1 a2-1 In do + In a
(a, (9 5p) Rgge
(22)
a-1
Step 3: DP guarantee of (AT BT) w.r.t the Joint Dataset D = U2 Dz. The pou-round guarantee (0 50) w.r.t. Di, is amplified by client subsampling (ratio qe from K total clients) using Lemma yielding an (gec)" (t) (a, 6o), 9x guarantee per round w.r.t. the full dataset D = k D.
23

---

(1) (ax, do), do mechanism over T adaptive communication rounds using Lemma for chosen 81 > 0, gives the final (c, 5)-P guarantee for (Az, B²
Composing this (4x+0
(m4b(e (a + +1
+ In( a
In( )-1 In 60 + In
qx 2
(23)
Kgo²
8 = qxT6o + 81.
(24)
The specific noise variance o² required in Theorem is derived by appropriately setting 6 .60
(0)
(based on RDP conversion from Step 2) and 81 to meet the target overall (e, 6), then solving for o².
Step Analysis of the RDP Order a.
Let 6n and ó1 be 6/2 and // (2 respectively, we now trying to acquire an expression of a through the following minization problem
m + + In( (-1)- In(6/2) In
min a>1
(25)
K20²
a-1
Let C1 mo K, 2 equation (25 find optimal >> 1 that minimizes is approximated by setting the
derivative of the dominant terms to
Ci(a 1)2 + In(6ga)) = 0.
(26)
Assume > 1, then, (a- - 1)² a² and the term In(6ga) can be written as In do + In
Substituting these approximations into the condition (26). we get:
Cia² + In + In
(27)
a is sufficiently large such that Cia² dominates a (i.c., the a² term grows much faster than In a),
can further simplify Eq. (27) by neglecting the in a term relative to a and In & (especially if In ool is large, which is true for small 8g). This yields:
Cia² ~ In
(28)
From Eq. 28
obtain approximate expression for the optimal
In
Cept
(29)
C1
Substituting 6g T) and Ci
mgp
K
In(2gxT/0)) K,o²
Kapp
1
(30)
Plug 30 into 24
we could conclude that to reach (c, d)DP) FedASK nessesitates noise of
T (2 In (2T
Ux
(31)
e K
This finishes the proof of theorem
A.3 Proof of Theorem 2
Lemma 8 (Expected Frobenius Norm Error of Random Projection 15).. Let A RRXB be mairix with singular values σ 02 2 Choose turget approximation rank kappmr 2 and an oversampling number Sover 2 such that the total number of random projection vectors I Kapprox + Sover minnmm, Let Ω | be standard Gaussian random matrix, and let = ASt Let Py be the orthogonal projector onto Range(Y) (Y). Then, the expected Frobenius norm of the error in approximating A by its projection Py A is bounded by,
1/2
B|[ - --y)Alll 1 (1+ Kimppros
1/2
Σ
(2(
Sover

---

Proof. Let the average of local updates be defined as:
AW
Ak
(32)
K
k=1
The first sketching follows as:
K
Y
Ba(A = AWΩ,
(33)
K R=1
where Q2 (x+x()))) is standard Gaussian. The dimension of the random subspace is r + p.
The second aggregated projection is:
Eipros _
Q))
(34)
K
1=1
Let SVD(QT AW) UEVT.
Q = UEVT
(35)
The global update AW = 1-AA=='AQU>""" =3 QUEVT.
AW QQQ AW) QQTAN
(36)
The condition p > dp - r + 2 implies + p 2 dp + 2. Since rank(AW) S dR, we have r' + p + 2 rank(AW). Apply Lemma with AW, the target rank kapprox = r + p, and Suver For kuppeox > rank(AW) the singular values a h and beyond are zero. Therefore, we could come to the conclusion of Theorem
Li
B Future Explorations
While FedASK demonstrates notable strengths, we identify the following areas for future exploration:
Local Matrix Update Strategies: FedASK currently fixes the local Aş matrix during differentially private updates one communication round. Investigating alternating local updates for both A₃ and B₁ matrices under differential privacy could reveal different learning dynamics and performance trade-offs.
Broader Model Applicability: Our validation FedASK is currently limited Large Language Models. Assessing its efficacy and potential adaptations for other architectures, such as vision transformers or diffusion models, remains an open research direction.
Advanced Training Paradigms: The current study focuses on fine-tuning for standard language and reasoning tasks. Extending FedASK to more complex federated and private training paradigms, such as model alignment (e.g., RLHF), presents valuable avenue for future work.
C More Experimets results
C.1 Training details
NLP Task, Model Performance with Privacy Guarantee. Experiments with the Llama-2-7B model focus on homogeneous data distributions. To ensure fair comparisons, standardized settings include batch size B of 8, 10 local update steps, and 4000 total communication rounds. Transformer-related hyperparameters, such as the sequence length (seq of 128, align with previous studies [19]. The LoRA rank r is fixed at 64, with the scaling factor a set to twice the rank. Optimal performance
is determined through grid search over learning rates (5e 5, le 4,2e 4, 5e 4), and in
differential privacy scenarios, privacy budgets (1,3, 6) are explored, consistent with prior work 34.. For the Llama-2 2-BBB experiments,
batch size B of 6, extend the total communication
25

---

rounds to 8000, adjust the sequence length leeg to 968, and increase the LoRA rank to 128. Other settings remain the same as in the Llama-2-7B experiments. Results are summarized in Table
NLP Task, Model Performance with Data Heterogenety. To evaluate data heterogeneity's 's impact, experiment with IID and three non-IlI scenarios, using 10 clients with 2 selected per round. In IID settings, data is randomly partitioned. For non-llI settings, we use Dirichlet distribution Dir(a) with a E {0.1, 0.5, 1.0]. following prior work [13.. To evaluate robustness under DP in these non-IID environments, we set the privacy budget = 3, a common value for balancing privacy and utility. Other settings keep the same.
VLM Task, Model Performance with RLHE To evaluate the efficacy of our method on vision- language tasks and preference alignment, we conduct experiments using the Alava-1.5-7B model. We employ DPO, a form of RLHF, to align the model with safety preferences using the SPA-VL dataset. The training dataset is curated by first filtering the source data to include only samples where the "chosen" response exceeds 450 characters, followed by a balanced sampling of 2,888 entries from each top-level category to ensure diversity. The federated learning environment consists of 10 clients with an IID data distribution, where 2 clients are selected per round for a total of 600 communication rounds, with each client performing 10 local update steps. Key hyperparameters include a LoRA rank of 256 applied to the q-proj. v_proj, and mm_projector modules while freezing the vision_ tower, a batch size of 2, and a learning rate of le-5, We evaluate performance under Mon-Private and two differential privacy budgets, E {.
Table 6: More Performance comparison of algorithms across different privacy budgets.
Benvertaik.V
MMsufety
SIUO
Algorithm
Help
Harm
ASR (1)
Effective
Safety
No DP
cmb
c=1
No DP
cm6
c=1
No DP
cm6
c=1
No DP
cmb
c=1
No DP
cmb
c=1
FedASK
55,78
56.36
53.65
65.13
65.36
62.98
43.21
43.40
46.93
87.42
85.62
30.53
31.14
28.74
FedAvg
53.98
51.53
15.50
60.61
53.66
48.89
45.63
48.21
50.80
89.82
83,23
30.23
29.94
26.91
25.14
FEA-LORA
55.68
54.66
52.38
63.43
63.32
61.96
44.93
44.39
48.45
86.83
86,23
83.83
28.74
29.34
27.54
Scaffold
52.01
51.02
52.55
46.67
48.75
50.80
83.83
76.04
28.14
27.54
C.2 Error-Bar of Current Experiments
This section presents error bar experiments, reporting the mean + standard error of the mean (SEM) over five independent runs, to substantiate the stability and performance of the FedASK framework. These evaluations cover varied differential privacy (DP) budgets and non-lID data distributions for Llama-2-7B and Llama-2-13B models on the dolly-15K and MetaMathQA datasets, respectively. Unless otherwise specified, all other parameters, such as batch size and communication rounds, align with the primary experimental configurations detailed in Section 5.
For these error-bar evaluations. specific learning rates and LoRA ranks were employed. Llama-2-7B experiments (Tables 7 and (9) used LoRA rank r = 64. For IID data with varying DP budgets (Table 7.. baseline learning rates were 2 x 10-4 (non-DP) and
10-4 (DP); FedASK and other LoRA methods used 5 x 10 (non-DP) and 4 x 10-4 (DP) with gradient clipping of 1.0. For non non-III evaluations at DP c = 3 (Table 9. baseline learning rates were 1 x 10-4, and LoRA- L-sse methods (including FedASK) used 4 x 10- with 1.0 gradient clipping. The Llama-2- 13BB experiments with IID data (Table 8 utilized a LoRA rank r = 128; FedASK learning rates were 5 x 10-4 (non-DP) and 4 x 10-1 (DP), while baselines used 2 x 10-1
The inclusion of mean + SEM from five runs in these experiments offers robust statistical validation of the delineated advantages of FedASK, reinforcing the conclusions drawn from single-run experiments in the main paper. As detailed Table 7 and Table FedASK outperforms or performs comparable to baseline methods in con-private settings and DP budgets of E {1, 3, 6), frequently producing comparable or reduced SEMs, highlighting its capacity achieve an effective equilibrium between model utility and privacy preservation. This demonstrated robustness is further evident in scenarios characterized by data heterogeneity; Table Yable@rrevat reveals FedASK's consistent maintenance of leading average performance alongside constrained variability, as indicated by the SEMs, across diverse non-III Dirichlet distributions (a E {0.1, 0.5, 1.0)), corroborating the adaptability observations presented in Section 5.2.
26

---

Table 7: Performance Comparison (Mean + SEM from five runs) Llama-2-7B with Different DP Budgets.
Task
Priv. Budget
FedASK
FodArg
FFA-LoRA
FedSA-LoRA
FedProx
Scaffold
Non-Private
40.34 = 0.19
43.13 2.01
45.72
0.27
41.63 + 0.57
43.55 = 1.44
41.49 = 1.17
45.79 0.01
42.82 0.75
44.15 1.39
43.16 + 0.25
43,00 + 1.01
43.33
MMLU
45.88 - 0.37
42.43 0.94
43.82 1.10
42.30 1.17
43.07 0.11
42.88 0.41
45.73 + 0.05
0.12 43.46
44.02
1.20
43.30 +: 0.54
43.00 + 0.71
43.43 = 0.37
Non-Private
32.01 + 0.08
30.31 + 0.11
31.65 0.31
30.86 + 0.37
30.95 + 0.04
30.87 + 0.86
31.33 +: 0.10
30 0.94
30
1.31
29.96 +: 1.08
30.18 +: 0.67
30.08 0.42
DROP
31.27 :: 0.10
29.46
0.15
30.37 = 0.97
29.91 + 0.65
28.64 = 1.07
30.19 0.01
Non-Private
14.63 + 0.61
13.42
1,83
14.02 + 0.02
12.81 + 0.61
12.81 + 0.61
14.63 = 0.00
Human-Eval
13.72 = 1.52
11.28
1.52
12.50 0.30
11.28 2.13
8.85 = 3.35
8,54 1.22
14.02 4: 1.22
7.63 2.74
11.59
8.85 2.13
10.06 +
9.76
15.85 + 0.02
9.76
1.83
11.90 0.31
10.67 + 2.13
8.85 + 2.13
9.76 + 2.44
Table Performance Comparison (Mean + SEM from five runs) on Llama-2-13B with Different
Budgets.
Task
Priv. Budget
FedASK
FodAvg
FFA-LoRA
FedSA-LoRA
FedProx
Scaffold
Non
51.40 1.40
46.25
24.95 + 2.25
16.40 = 0.90
14.25 0.05
14.50 + 2.30
16.00 = 0.80
16.45 + 0.35
GSM8K
25.35 - 0.55
19.60 +: 3.10
18.60 1.40
21.20 +: 1.00
20.20 2.20
18.30 1:00::220 2.50
24.35 -: 3.35
20.05
0.75
19.15
18.45 1.15
22.05 1.95
20.35 + 0.05
Non-Private
23.90 4.80
21.90
3.90
22.20 +: 1.00
22.10
23.05 + 3.05
20.95 + 0.85
13.65 0.65
10.25 1.45
7.85 1.1510..11
8.25 1.65
7.90
9.35 1: 0.25
GSM8K1ad
13.00
11.90 0.80
10.25 0.25
11.55 0.25
11.35 0.55
11.20 + 0.20
13.30 3.60
11.30 + 0.80
9.40 0.20
10.55 11 0.35
11.60 = 0.70
11.40 + 0.60
Non-Private
12.55
9.30 + 1.00
10.25 0.55
10.00
10.90 0.80
10.00 + 0.20
7.25::00.35
5.55 0.35
5.50 0.30
2.25
5.85 0.25
5.55 0.25
Math
0.20
6.50 0.400
0.20
6.16
6.30 0.10
5.95 0.45
6.50
6.35 0.15
6.15::0.15
6.05 0.15
6.50 0.20
7.00 +: 0.10
Table 9: Performance Comparison (Mean SEM from five runs) for DP Budget across Different Data Distributions on Llama-2-7B.
Task
Data Dist.
FedASK
FedAvg
ITA-LORA
FedSA-LoRA
FedPPrx
Scaffold
45.88 0.37
42.43
0.94
43.82 = 1.10
42.30 1.17
43.07
0.11
42.88 + 0.41
Dir(O.1)
45.21 0.84
42.85 0.116
1.09
44.11 0.16
42.99 0.39
42.33 0.73
MMLU
Dir(O.5)
45.05 + 0.90
42.71
43.22
1.76
43.26 + 0.54
42.66 0.32
42.91 + 0.94
Dir(I)
45.26 + 0.75
42.75 + 0.21
44.60 + 1.37
42.37 + 1.33
0.12 43.10
42.69 + 0.98
IID
1.04
20.78 0.34
29.87 = 1.47
29.66 + 0.26
29.07
0.57
28. 1: 0.15
Dir(O.1)
30.85+00. 0.17
0.93 29.27
30.52
0.42
28.72
0.14
28.53 0.35
28.29 + 0.02
DROP
Dir(O.5)
0.001
29.41
0.72
28.47 0.64
28.47
29.17
Dir(1)
3..19
29.77
1.23 30.16
20.40 0.11
0.03 29.85
30.20 1: 0.32
14.02
2.74
11.59 0.61
8.85 2.13
10.06 = 3.35
9.76 1.83
1)r(O.1)
13.41 0.00
9.15
11.29 = 0.31
10.67 + 2.13
9.15 4.27
8.24 0.31
Human-Eval
Din(0.5)
12.81
9.76 1.22
11.89 = 1.52
12. 0.61
14.33 0.31
8.21
1.52
Dir(1)
13.41 - 0.61
7.93 +
12.50 = 1.52
10.37 + 0.00
7.93 19932244
7.63 + 1.52
27

---

C.3 Algorithms within more DP and Non-iid Conditions
Table proviides a comparative evaluation of algorithm performance when subjected the combined effects of differential privacy and specific non-llD data distributions, namely Dirichlet distributions with a 0.1 representing higher data heterogeneity and a 1.0 indicating lower heterogeneity. This side-by-side presentation for each algorithm across various DP budgets (e E {1,3, 6) and Non-Private) allows for nuanced understanding of their robustness.
The results in Table 10 underscore FedASK's consistent ability to deliver strong performance even under these challenging compound conditions. Across the evaluated tasks (MMLU, BBH, DROP, Human-Eval), FedASK generally maintains a competitive edge or outperforms baseline methodologies for both the more heterogeneous non-IlI setting a 0.1 and the less heterogeneous setting a = 1.0. In particular, while increased DP noise (smaller c) or increased data heterogeneity (smaller a) tends to degrade performance for all algorithms. FedASK often exhibits a more graceful degradation compared to several baselines. This suggests that FedASK's s two-stage sketching and aggregation mechanism not only preserves utility under DP but also offers resilience against varying degrees of data heterogeneity. The comparative performance between the Non-III 0.1 and Non-III) 1.0 columns for FedASK within each DP budget further illustrates its capacity adapt effectively, reinforeing its suitability for practical federated learning scenarios where both privacy and non-IlI> data are prevalent concerns.
Table 10: Algorithm Performance across Varying DP Budgets for Non-IID (Dirichlet 0.1 and 1.0)
Data on Llama-2-7B
Task
DP Setting
FedASK
FedAvg
FFA-LoRA
FedSA-LoRA
FedProx
Scaffold
0.1
a
a 1.0
0.1
a 1.0
< 0.1
a 1.0
0.1
a 1.0
a 0.1
DP
46.50
45.33
45.48
43.75
44.70
MMLU
DP
45.73
45.88
42.69
42.12
41.00
43.75
41.44
41.62
41.40
43.66
43.11
43.60
DP
46.04
45.86
42.69
42.96
42.54
43.23
44.27
41.04
42.61
42.98
43.05
41.71
DP
46.24
46.42
43.97
42.82
42.24
42.94
43.93
40.07
43.56
41.06
43.83
No DP
32,15
32.55
33.50
32.03
32.16
32,86
32.25
32,11
33.10
32.29
32.99
33.08
DP
31.99
31.78
31.14
31.73
31.71
33.45
33.34
31.80
32.22
33.40
32.28
31.36
BBH
c
32.46
32.39
33.54
31.02
32.71
32.06
33.37
32.61
32.30
32.17
33.02
31.50
DP
32.12
31.91
31,98
30.99
32.42
31.56
31.00
31665
31.67
34.01
31.12
32.44
No DP
33.16
32,30
30.56
31.47
31.19
33.46
30.96
3..92
31.09
32.78
28.94
30.12
DP
31.93
31.98
28.18
29.93
29.75
31.17
96
28.81
28.16
30.71
29.45
29.20
DROP
DP
31.01
31.09
31.49
30.02
30.10
28.93
28.58
29.29
28.18
29.82
28.27
30.52
DP
32.46
30,90
28.78
28.35
29.51
29.21
30.37
26.51
29.96
28.51
28, 88
29.80
No DP
15.24
15.85
12,20
12.80
14,63
14.02
14.02
14.02
12.80
14.02
12.80
15.85
Humun-Eval
DP
14.02
11.59
10.98
9.76
12.20
10.98
10.98
12.20
12.80
14.02
12.80
DP c
12.20
13.41
12.20
9.76
10.98
10.98
12.80
10.37
13.41
10.37
7.93
9.15
DP
12.20
13.41
6.71
13.41
10.37
10.37
6.71
10.98
11.59
12.80
14.63
12.80
C.4 Sensitive Experiments
C.4.1 Choice on the lora rank
MNLU
UROP
HUMAN-EVAL
32
100 20/00
ew0
DP Setting
Setting
Setting
Figure 4: Performance of Llama 2-7B on IID data across LoRA ranks and differential privacy (DP) settings (e) for MMLU, DROP, and Human tasks.
The selection of an appropriate LoRA rank crucial to balance the performance of the model and the efficiency of the parameters, particularly when differential privacy (DP) is applied. This
28

---

MKLUI
DROP
HUMAN-EVAL
16
4.7
Lobk
0005
4+3
dmg
Nevshouldd
4+1
4+6
6+3
Setting
Setting
Figure 5: Performance of Llama 2-7B on Non-IID data (a 0.1) across LoRA ranks and differential privacy (DP) settings (c) for MMLU, DROP, and Human tasks.
02
gamak hard
math
LotA Rank
:20
2X1
X4
120
28
110
12
120
11.
Figure 6: Performance Llama 2-13B on IID data across LoRA ranks and differential privacy (DP)
settings (c) for gsm8k, gsmök hard, and math tasks.
gamak,
Lota Rank
200
1L5
111
233
291
1144
64
344
151
135
302
64
183
128
2r3
223
24
125
113
1M8
134
128
Setting
Figure 7: Performance of Llama 2-13B on Non-lID data (a 0.1) across LoRA ranks and differential
privacy (DP) settings (c) for gsm8k, gsm8k hard, and math tasks.
29

---

section details the interaction between LoRA rank, DP settings, model size, and data distribution for the FedASK framework, referencing empirical results from Llama 2-7B and Llama-2-133 2-BBB models. Although higher LoRA ranks, such as 128, often deliver superior performance in nonprivate scenarios, the introduction of DP mechanisms significantly alters these performance landscapes, often favoring intermediate ranks for a more robust uuility-privac trade-off.
key finding, illustrated in Figure 6 for the Llama 2-13B model on IID data, is the change in the optimal LoRA rank under DP for FedASK, Although rank 128 excels without privacy, for instance, on gsm8k (50.0) and gsmök hard (28.7). intermediate ranks frequently provide a superior utility-privacyy trade-off when DP is enabled. Specifically, on the gsm8k task, rank 64 consistently outperforms rank 128 under all tested DP settings; for example, with
rank 64 achieves 40.4 versus 22.7 for rank 128, and with é = 6, rank 64 achieves 40.6 versus 27.7 for rank 128. Similarly. for the math task, rank 64 shows better performance than rank 128 across all DP budgets. On gsm8k _an8k_ardd, rank 64 also remains highly competitive with, or slightly better than, rank 128 under DP conditions. This consistent strong performance of rank 64 under various DP constraints suggests that for FedASK with the 13B model on IID data, moderately sized LoRA rank can be more parameter-efficient and achieve better utility when stringent privacy guarantees are necessary. When data heterogeneity is introduced for the Llama 2-13B model, as shown in Figure 7 for Non-IID data (a = 0.1), the utility of intermediate ranks under DP persists largely. Although overall performance levels may adjust due to the non-IlD distribution, FedASK with moderately sized ranks continues demonstrate a strong balance between adaptation capability and resilience to DP noise, reinforcing the notion that maximal ranks are not universally optimal under privacy constraints in heterogencous settings.
This rank-dependent performance pattern under DP is also investigated for the Llama 2-7B model. results on IID data for MMLU, DROP, and HumanEval tasks. For FedASK, it is generally observed that while larger ranks might offer marginal gains or lead in non-private scenarios, the application of DP tends to make intermediate ranks more advantageous. These moderately sized ranks appear to strike an effective balance, providing sufficient capacity for task adaptation while mitigating the detrimental impact of DP noise that can be more pronounced with larger number of trainable parameters. The introduction of significant data heterogeneity with Non-IID data (a = 0.1), illustrated in Figure [5] further tests this dynamic. Even in these challenging conditions, FedASK with intermediate ranks often maintains robust performance relative to larger ranks under DP. This suggests that for the 7B model, an excessively large rank under combined DP and non-lll) stress may not yield proportional benefits and could be outperformed by more parameter-efficient intermediate rank configurations.
C.4.2 Choice on over-sketching rate
The precision of FedASK's aggregation mechanism is theoretically linked to the choice of the over- sketching parameter p, which, together with the LoRA rank r, defines the sketching dimension r + p. While Theorem 2 provides a condition for exact aggregation, it is crucial to empirically assess the impact of varying sketching dimensions on aggregation fidelity under practical conditions, including different degrees of data heterogeneity and client participation numbers. This appendix section details these specific evaluations for FedASK, illustrating its robustness. The experiments summarized here were conducted to determine suitable range for the sketching dimension, ensuring high fidelity without unnecessary computational overhead. All results presented pertain to the FedASK algorithm, and aggregation fidelity is quantified the cosine similarity between the global LoRA update reconstructed by FedASK and the ideal average of true local LoRA updates.
The empirical investigation involved evaluating the aggregation fidelity of FedASK across matrix of conditions, as depicted in Figure 9 The experiments systematically varied: (i) the sketching dimension (x-axis values: 6, 32. 51, 64, and 96), (ii) the degree of non-iid degree (Dirichlet distri- butions with a E {1.0, 0.8, 0.5, 0.1)), and (iii) the number of participating clients, shown in four distinct panels: (a) 5 clients, (b) 10 clients, (c) 15 clients, and (d) 20 clients. For these specific fidelity evaluations, differential privacy mechanisms were not applied to isolate the performance of the aggregation mechanism itself. The color intensity in each heatmap cell corresponds to the achieved cosine similarity, with lighter shades indicating higher fidelity.
The results consistently demonstrate FedASK's 's exceptional aggregation fidelity across the vast majority of tested scenarios. As seen in Figure combinations of sketching dimensions, non-IID degrees, and client numbers. Even with the smallest
g near-unity cosine similarity is achieved for most
30

---

clients
15 CIRRTS
1810161
AMORUS
O PHIWKC
10
SC00001
SONDIA
-033
1.0
900104
03904600
1D Deppres CDincter
0.8-
050780
33MD10
CIANNO
0.75
BON646
A0073000
3.50
Stetchvig Dimeessinn
Skanchngg Dimmruiio
(a)
[b)
10 clients
20 clients
B1915447
85738133
0993449
01999003
8.999099
0.38
6.98
ASROP30
08903422
CAMIGN
Mon HD Degree (Dinehee
6.08
Statchingg Dimension
Statching Dimennsion
(c)
Figure 8: Impact of sketching dimension (X-axis) and non- degree (y-axis, Dirichlet a / IID) on FedASK's aggregation fidelity (cosine similarity) for (a) 5, (b) 10, (c) 15, and (d) 20 clients. showing robust near-unity performance.
sketching dimensions, fidelity remains remarkably high, particularly as the number of participating clients increases (panels b, and d). While the 5-client scenario (panel a) shows slightly reduced fidelity under extreme non-IID conditions and very small sketching dimensions, the performance rapidly approaches unity with modest increases in either parameter. These findings underscore that FedASK is not highly sensitive to the over-sketching rate for maintaining precise aggregation and achieve excellent fidelity with minimal conservative sketching dimensions, confirming its practical efficiency and robustness.
C.5 System Efficiency Experiments
To quantify the system efficiency of the evaluated algorithms, we conducted experiments on NVIDIA H100 GPUs, focusing two key metrics: communication volume and end-to-end wall-clock time. The analysis uses the Llama 2-7B and Llama 2-13B models, with LoRA applied to the 'q_proj', 'v_proj", and "k_proj" modules. Our measurements do not incorporate system-level optimizations like distributed parameter servers or communication-compuaaaa
overlap.
Figure 9 illustrates the usage of resources per client. The volume of communication, seg- mented into uplink and downlink traffic. reported in millions of parameters. Peak GPU memory consumption (in MB) was meticulously monitored on the client side using the torch. cuda. max_ pexx__emrr__ale allocated (device=torch device (' cuda')) Py PyTorc func- tion, capturing the maximum memory footprint during local training operations with differential privacy mechanisms enabled. Futhermore, we analyze the communication volume per round. shown in Table [1] the choice of numerical precision (FP16, INT8, INT4) directly impacts the data transfer size for algorithms. The results highlight a clear trade-off: FedASK's 's communication volume is 25% higher than FedAvg's, s, but it remains significantly more efficient than other baselines,
31

---

Upink (Miliins)
Downlnk (Miliess
GPU Memory (MB)
Model
13B
Paramstter Count (Millions)
A
3200
Puratrater Count (Millonn)
5000
47500
91.00
45000
3000
Mermory (M1)
4000
42500
3000
40000
8000
37500
THE
8700
2000
35000
8600
1000
8500
30000
Feodels
Scatfodd
Hira
testtog
FedFa
Festakk
Scaffeld
Pirr
Feebbad
Algrrthm
Algorthh m
Figure 9: System resource utilization five federated learning algorithms when training Ilama 2-7B (left) and Llama 2-13B (right) models using 4-bit precision. The metrics, shown for a single client in a 5-client federated setup, include uplink and downlink communication volume (millions of parameters) and GPU memory consumption (MB).
Table 1:: Total communication volume client round for federated learning algorithms using Llama 2-7B and Llama 2-13B models under different numerical precisions. All values in Megabytes (MB).
Algorithm
Llama 2-7B (MB)
Llama 2-13B (MB)
FP16
INT8
INT4
FP16
INT8
INT4
FedAsk
1200
600
300
7500
3750
1875
FedAvg
960
480
240
6000
3000
1500
Scaffold
1920
960
480
12 000
6000
3000
Flora
5280
26-00
1320
33 000
16 500
8250
FedFA
480
240
120
3000
1500
750
requiring 40% less communication than Scaffold and 77% less than the communication-heavy Flora.
As demonstrated in main paper, this modest overhead unlocks substantial performance gains, with FedASK outperforming FedAvg by up to 11.5% on MMLU and 46% on GSMSK under strong privacy.
Table 12: End-to-End Wall-Clock Time per Communication Round (s)
Clients Num.
Alg.
Client Csmp.
Server ComP,(CPU) (s)
Total Comm. (s)
FedASK
1.44
0.12
1.51
2
FedAvg
1.64
0.07
1.21
FedASK
1.44
0.29
5.53
10
FedAvg
1.64
0.35
4.84
FedASK
1.44
0.69
16.61
30
FedAvg
1.64
0.71
13.29
To further assess practical performance, we empirically measured the end-to-end wall-clock time for single FL round. The experiment fine-tunes a Llama-2-7B model, with client operations an NVIDIA H100 GPU and server aggregation on a CPU, simulating a 1Gbps network across a varying number of clients (K,).
The results, detailed Table 12 confirm FedASK's practicality. Crucially, FedASK reduces client- side computation time by approximately 0.2 seconds round compared to FedAvg. This is because our local DP strategy avoids backpropagation introduces minor computational overhead on the server, this cost is minimal and does not create scalability bottleneck; the server time remains under one second on CPU even with 30 clients, as the core aggregation operations do not scale with the number of clients.
one of the LoRA matrices. While FedASK
32


Reviews:
[{"review_id": "aT3NZA2ETU", "reviewer": "Reviewer_NdtW", "summary": "The paper introduces FedASK (Federated Low Rank Adaptation with Double SKetching), a novel framework for fine-tuning large language models (LLMs) in a federated learning (FL) setting while ensuring differential privacy (DP). It addresses the trade-off between noise amplification and learnability in federated LoRA (Low-Rank Adaptation) by enabling simultaneous updates of both low-rank matrices (A and B) with strong privacy guarantees.", "questions": "The paper validates FedASK on Llama-2 models (7B and 13B) for standard language and reasoning tasks (MMLU, DROP, Human-Eval, GSM8K). How does FedASK perform on other model architectures (e.g., vision transformers or diffusion models) or more complex tasks (e.g., reinforcement learning with human feedback, RLHF)?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The strengths of this paper are as follows.\nRobust Theoretical Analysis: The paper provides strong theoretical guarantees for both differential privacy (Theorem 1) and aggregation precision (Theorem 2).\nComprehensive Experiments: The empirical evaluation is thorough, using large-scale Llama-2-7B and 13B models across multiple tasks (MMLU, DROP, Human-Eval, GSM8K) and data distributions (IID and non-IID with Dirichlet α ∈ {0.1, 0.5, 1.0}).\nAddressing a Key Challenge: FedASK tackles a critical dilemma in federated LoRA: balancing privacy (via DP) with model performance.\n\nThere are some Weaknesses in this paper:\nLimited Model Scope: The experiments are conducted only on Llama-2 models (7B and 13B).\nFixed Local Matrix Strategy: FedASK fixes the A matrix during local updates within a communication round, which may constrain learning dynamics.\nNarrow Task Focus: The evaluation focuses on standard language and reasoning tasks (MMLU, DROP, etc.).", "was_revised": false, "final_justification": "", "rebuttal": "# [W1, W3, Q1]: On Model and Task Generality\n\nWe appreciate the reviewer for pointing out the need to demonstrate FedASK's applicability beyond the initial set of models and tasks.\n\n## A. New Experiments on VLM and DPO (RLHF) \n\nThe design of FedASK is **inherently general**, making it seamlessly applicable to training pipeline that uses LoRA, including VLMs and DPO. To empirically validate this, we have conducted new experiments on more complex models and tasks as suggested. The results clearly demonstrate that **FedASK consistently and significantly outperforms the FedAvg baseline under strong DP constraints across these diverse tasks**.\n\n### 1) Experimental Setup:\n\nThe configurations for our new experiments are summarized in the table below. All experiments were run for 400 communication rounds, and other parameters align with the settings in Section 5.1 of our paper unless specified otherwise.\n\n**Table 1: Configuration of VLMs+SFT / LLM+DPO**\n\n| Experiment | Base Model | Finetuning Dataset | FL Setup | Evaluation |\n|:---|:---|:---|:---|:---|\n| VLM SFT | Llava-1.5-7B[1] | ScienceQA (12K)[2] | 10 clients, 2 selected | Science-QA from official LLaVA guidelines |\n| LLM DPO | Llama-2-7B | Anthropic/hh-rlhf (160 K)[3] | 50 clients, 10 selected | Open-ended evaluation of helpfulness (Vicuna) and harmlessness (AdvBench) |\n\n### 2) Experimental Results & Analysis:\n\nFor VLM SFT task, FedASK outperform FedAvg by a significant margin of **over 7 percentage points**. As for DPO task, FedASK achieves a score **~8-9 points higher than FedAvg**. Both indicates a significantly better capacity beyond language models.\n\n**Table 2: VLM Fine-Tuning on ScienceQA (% Accuracy)**\n\n| Privacy Budget ($\\epsilon$) | FedASK | FedAvg | Non-Private |\n|:---|:---:|:---:|:---:|\n| $\\epsilon=1$ | **71.79** | 64.12 | 69.51 |\n| $\\epsilon=3$ | **72.52** | 65.14 | 69.51 |\n| $\\epsilon=6$ | **71.83** | 65.84 | 69.51 |\n\n**Table 3: LLM DPO on Anthropic/hh-rlhf (GPT-4 Score)**\n\n| Privacy Budget ($\\epsilon$) | FedASK (Vicuna) | FedASK (AdvBench) | FedAvg (Vicuna) | FedAvg (AdvBench) |\n|:---:|:---:|:---:|:---:|:---:|\n| $\\epsilon=1$ | 7.675 | 38.46 | 7.231 | 30.76 |\n| $\\epsilon=3$ | 7.931 | 40.38 | 7.345 | 31.73 |\n| $\\epsilon=6$ | 7.836 | 40.38 | 7.506 | 32.69 |\n\nDue to time constraints, further experiments (including other baselines and additional datasets like VQAV2) are ongoing. We are confident these will further strengthen our claims and will include the full results in the revised version. \n\n\n## B. Recap: Generality of the FedASK Framework\nWe would like to re-emphasize that FedASK is designed as a general framework for **federated learning scenario that utilizes LoRA with differential privacy**.\n\n* **Our Goal is Fundamental:** Address the challenge of noise amplification and limited learnability in private federated LoRA.\n* **Our Core Method is architecture-agnostic:** A two-stage sketching pipeline theoretically enables the exact aggregation of LoRA updates and facilitates the effective update of **both** LoRA matrices at the global level under DP. This mechanism is independent of the underlying model architecture.\n\nOur training framework, developed on `opacus` and `transformers`, naturally supports a wide range of models and algorithms. To the best of our knowledge, no open-source framework currently offers out-of-the-box support for federated learning with integrated DP for LMs. We are committed to releasing our code to the community to facilitate future research in this direction. \n\n# [W2]: On the Fixed Local Matrix Strategy\n\nThis is a very valid concern regarding the learning dynamics of our proposed method. The decision to fix matrix A during local updates is **a deliberate design choice motivated by the core theoretical challenges and empirical ablations**, and FedASK's global update mechanism is designed to overcome the potential limitations of this choice. \n\n## A. Why must locally Fix one matrix?\n\nThe strategy of fixing one LoRA matrix during local updates is a direct consequence of the fundamental principles of Differential Privacy when applied to the LoRA architecture. The core issue is preventing catastrophic noise amplification, which can be understood in two facts.\n\n**Fact 1: To apply DP-SGD on LoRA, both gradients are required to add noise.**\n\nIn LoRA, the update to a weight matrix $W$ is parameterized by two smaller matrices, $A \\in \\mathbb{R}^{r \\times n}$ and $B \\in \\mathbb{R}^{m \\times r}$. The gradients of the loss $\\ell$ with respect to $A$ and $B$ are interdependent:\n$$\n\\nabla_A \\ell = \\frac{\\partial \\ell}{\\partial A} = \\frac{\\alpha}{r} B^T \\frac{\\partial \\ell}{\\partial W}\n$$\n$$\n\\nabla_B \\ell = \\frac{\\partial \\ell}{\\partial B} = \\frac{\\alpha}{r} \\frac{\\partial \\ell}{\\partial W} A^T\n$$\nA key principle of DP is composition. Since the calculation for $\\nabla_A \\ell$ depends on matrix $B$, and the calculation for $\\nabla_B \\ell$ depends on matrix $A$, a private update of the model requires that all sensitive information be properly privatized. Therefore, to update both $A$ and $B$ simultaneously in a private manner, we are required to add noise to both of their respective gradients.\n\n**Fact 2: Both gradients adding noise leads to noise amplification.**\n\nAs formally analyzed in our Lemma 1, this interaction leads to a destructive amplification. The critical issue arises from the quadratic noise term $\\eta^{4}\\frac{\\sigma^{4}C^{4}}{B_{size}^{4}}d_{l}^{2}r$ scaling with the fourth power of the noise standard deviation $\\sigma^{4}$ and the square of the model dimension $d_l^2$. Consequently, the noise quickly overwhelms the actual learning signal, which prevents the model from training effectively.\n\nBy fixing A matrix locally, we ensure that noise is only added to the other matrix's gradient. This completely **eliminates the problematic noise-on-noise interaction** and its resulting quadratic noise term, thus preserving the model's utility.\n\n## B. How FedASK Global update goes beyond fix matrix?\n\nPrior works like FFA-LORA recognized this issue and proposed mitigating it by permanently freezing matrix A throughout training. While this avoids noise amplification, it fundamentally restricts the model's expressive capability by confining the update to a predetermined, static subspace defined by the initial A.\n\nFedASK resolves this limitation. The server aggregates the privatized local updates and uses SVD to decompose this collective information into its core components. These components are then used to synthesize entirely new global A and B matrices, allowing the model's learning space to evolve each round instead of being confined to a static subspace.\n\n## C. Ablation Study\n\nTo empirically validate this design choice, we conducted an ablation study comparing our standard FedASK with two variants on the MMLU benchmark under challenging DP settings:\n\n* **FedASK-Alt:** Alternating which matrix (A or B) is frozen during local DP updates in each round. To validate the update, we first freeze matrix A and then B in the second round.\n* **FedASK-Both:** Updating both A and B locally with DP noise in every round.\n\n**Table 4: Ablation Study on Local Update Strategy (MMLU Score)**\n\n| Privacy Budget ($\\epsilon$) | Fedask | Fedask_alt | Fedask_both |\n|:---|:---:|:---:|:---:|\n| $\\epsilon=1$ | 45.58 | 42.62 | 41.55 |\n| $\\epsilon=3$ | 45.81 | 43.11 | 39.65 |\n| $\\epsilon=6$ | 45.65 | 44.31 | 40.36 |\n\nThe results clearly show that **updating both matrices locally (FedASK-Both) leads to a significant performance degradation**, empirically confirming the noise amplification issue predicted by our theory. **Alternating (FedASK-Alt) is also suboptimal**. This stems from a more nuanced theoretical conflict. Prior work[4] suggests that LoRA matrices have specialized roles: matrix $A$ captures general, global features, while matrix $B$ learns client-specific local features. FedASK-Alt creates a theoretical conflict by suboptimally tasking the \"global\" matrix $A$ with learning from a single client's data in alternating rounds, disrupting this effective division of labor and leading to a less stable learning process.\n\nTherefore, fixing A locally and update globally is a critical component for achieving a robust privacy-utility trade-off, and our two-stage global update mechanism ensures this local constraint **does not impair the model's overall learnability**.\n\nWe are grateful for the opportunity to clarify these points and believe the new evidence significantly strengthens our contribution. We hope our detailed response and additional results have addressed the reviewer's concerns.\n\n**Reference**\n\n[1] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023.\n\n[2] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.\n\n[3] R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y. Xu, Y. Du, Y. Wang, and S. Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 6137–6147, 2024.\n\n[4] P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu. Selective aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, 2025."}, {"review_id": "SBKQFBd0RR", "reviewer": "Reviewer_jeU1", "summary": "This paper investigates the trade-offs involved in applying differential privacy noise to both LoRA matrices. While doing so can lead to excessive noise amplification in the final model update, the common workaround—fixing one matrix and only training the other—significantly limits the model's learning capacity. The authors propose FedASK, a novel framework that enables the effective and differentially private updating of both LoRA matrices. The key innovation is a two-stage sketching pipeline inspired by randomized SVD. Both theoretical analysis and experimental results are presented to justify the performance and effectiveness of the proposed FedASK method, which outperforms other baseline approaches.", "questions": "- How do the method and theoretical analysis apply to models beyond dense LLMs, such as state-space models and MoE models?  \n- Are there any further optimizations that can be done to reduce the communication and computation overhead of FedASK?", "limitations": "As mentioned in the \"Weaknesses\" section, will it be possible to add end-to-end and per-iteration breakdown wall-clock time analysis of the proposed FedASK method and other baseline methods?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is overall well-written and well-motivated.\n- Enhancing differential privacy while preserving model performance for efficient federated LLM fine-tuning is of significant potential impact.\n- The proposed FedASK method is intuitively easy to follow.\n- Both theoretical and experimental results are provided to justify the performance of FedASK.\n\nWeaknesses:\n- [Major concern] The two-stage pipeline of FedASK requires two rounds of communication, which may introduce significant communication overhead and bottlenecks.\n- FedASK requires additional computational load on the server side (e.g., for matrix decompositions), and this computation appears to scale with the number of participating clients.\n- The paper does not provide end-to-end wall-clock runtime or a breakdown of the execution time, which is important for understanding the practical performance of FedASK.", "was_revised": true, "final_justification": "My concerns have been addressed by the authors. I thus raised my overall evaluation score, and I am happy to see the paper to be presented at NeurIPS 2025.", "rebuttal": "# [W1, Q2] On Communication Overhead\n\nThanks for insightful and detailed feedback! The questions raised about the practical overhead are critical for assessing real-world value. We have performed new analyses and experiments to address these points directly, and we believe the clarifications below will resolve these important concerns.\n\n## A. Clarification on Total Communication Volume\nWe understand the reviewer's concern that our two-step pipeline might imply a significant communication overhead. We would like to clarify that **FedASK makes a deliberate and highly effective trade-off: it modestly increases the total communication volume to enable a sophisticated aggregation mechanism that, in turn, yields substantial gains in model performance, especially under Differential Privacy**.\n\nTo provide a precise breakdown, we analyze the communication burden in two parts:\n\n* Uplink (Client-to-Server): To minimize the burden on resource-constrained clients, FedASK's total uplink volume is designed to be **identical to that of standard FedAvg-LoRA**. Our method achieves this by having clients transmit two low-dimensional sketches ($Y_k^{proj}$ and $\\tilde{Y}_k^{proj}$) whose combined size in our experiments exactly matches that of the full LoRA adapters ($A$ and $B$) uploaded in FedAvg.  \n\n* Downlink (Server-to-Client): The modest communication overhead is introduced on the downlink. In addition to broadcasting the updated global adapters, the server sends an additional orthonormal basis matrix, $Q$. This transmission is the cornerstone of our method, as it is the enabling mechanism for the precise aggregation and dual-matrix updates that are central to FedASK's high performance under privacy constraints.\n\nTo empirically quantify this trade-off, we conducted two new experiments. Our simulation assumes a cross-silo FL scenario with 10 selected clients and a 1Gbps network bandwidth [1, 2] for both upload and download bandwidth between server and clients.\n\n**1) Per-Round Communication Cost**\n\nFirst, we measured the communication volume and simulated transfer time per round for each algorithm. The model is trained at fp16 precision, with LoRA applied to the 'q_proj', 'v_proj', and 'k_proj' modules. The current analysis does not incorporate system-level optimizations like distributed parameter servers or the overlapping of download and upload communication. \n\n**Table 1: Communication Volume and Simual transfer time of Various Algorithms of 10 selected clients**\n| | Llama-2-7b | | Llama-2-13b | |\n|:---|:---:|:---:|:---:|:---:|\n| | Comm. Time(s) | Total Comm. Vol.(MB) | Comm. Time(s) | Total Comm. Vol.(MB) |\n| fedask | 5.53 | 1200 | 34.60 | 7500 |\n| fedavg | 4.43 | 960 | 27.68 | 6000 |\n| scaffold | 8.86 | 1920 | 55.37 | 12000 |\n| flora | 24.36 | 5280 | 152.26 | 33000 |\n\nThe results in Table 1 provide a clear picture of the trade-off. FedASK's total communication per round is 25% higher than that of FedAvg, while 40% less communication than Scaffold and 77% less than the communication-heavy Flora. This modest increase unlocks substantial performance gains, with FedASK outperforming FedAvg by up to 11.5% on MMLU and 46% on GSM8K under strong privacy, as shown in our main paper.\n\n**2) Fair comparison under an Identical Communication Budget**\n\nTo provide the most direct comparison of efficiency, we evaluated which algorithm performs best when allocated the same total communication budget. To ensure equivalent communication volume across the algorithms, we selected the results from 400 rounds of FedASK, 500 rounds of FedAvg, and 100 rounds of Scaffold.\n\n**Table 2: Performance Comparison under identical transfer volume (MMLU score)**\n| | fedask | fedavg | scaffold |\n|:---|:---:|:---:|:---:|\n| dp=1 | 45.71 | 42.74 | 38.35 |\n| dp=3 | 45.79 | 42.42 | 38.47 |\n| dp=6 | 45.56 | 42.01 | 38.13 |\n\nWhen normalized for total communication volume, FedASK consistently outperforms both FedAvg and Scaffold across all privacy settings. This demonstrates that FedASK's communication overhead is not just a cost but a high-return investment in a more powerful and accurate aggregation mechanism.\n\n## B. Further Communication Optimization Strategies\nThanks for suggestion! FedASK framework can cooperate with several methods, inheritedly or orthogonaly to further reduce overhead.\n\n**Inherent / Orthogonal Compression Techniques:** Inherently, communication volume mainly depends on LoRA rank $r$ and the over-sketching parameter $p$. Tuning these online can be formulated as an optimization problem to find the optimal rank and for a client that minimizes the trade-off between model approximation error and system costs:\n\n$$\n\\min_{r_k, p} \\quad \\mathbb{E}\\left[ || \\Delta W_k - B_k(r_k, p)A_k(r_k, p) ||_F^2 \\right] + \\lambda_C \\cdot \\text{CommCost}(r_k, p) + \\lambda_T \\cdot \\text{CompCost}(r_k, p).\n$$\n\nBesides, FedASK is compatible with standard model update compression techniques like quantization and sparsification.\n\n**Distributed Parameter Servers:** By partitioning the LoRA adapters by layers across M parallel servers, the data load on any single server is reduced. Since the main bottleneck lies in server side, this strategy can cut the total communication time to approximately $1/M$ of the original.\n\n**Overlapping Communication:** By overlapping uplink and downlink operations, the server can mask network delays. This reduces the effective round time from a sum of latencies $T_{\\text{upload}} + T_{\\text{download}}$ to the maximum of the two $\\max(T_{\\text{upload}}, T_{\\text{download}})$.\n\n# [W2 & W3] On Server Computation and Wall-Clock Runtime\n\nWe agree with the reviewer that FedASK introduces additional server-side computations compared to simpler aggregation schemes like FedAvg. However, we wish to clarify critical points regarding its practicality.\n\n## A. Server Computation Does Not Scale with Selected Clients numbers:\n\nThe server's workflow is as follows:\n\n* First, it performs a simple summation of sketches: $Y_{agg}^{t} = \\sum_{k \\in K_t} Y_k^{proj}$. This is the only step that scales linearly with $K_t$, and it is an inexpensive operation identical to the aggregation step in FedAvg.\n\n* The subsequent QR and SVD operations are performed on the aggregated matrices ($Y_{agg}^t$, $\\tilde{Y}_{agg}^t$), whose dimensions ($m \\times (r+p)$) are small and depend only on the model dimension and LoRA rank, **irrelevant to the number of clients**.\n\n## B. End-to-End Wall-Clock Runtime:\n\nWe have conducted a new experiment to empirically measure the end-to-end wall-clock time of a single federated learning round. The experiment involves fine-tuning a Llama-2-7B model, with client operations on an NVIDIA H100 GPU and server operations on a CPU, simulating networks of 1Gbps[1, 2]  across a varying number of clients.\n\n**Table 4: End-to-End Wall-Clock Time per communication Round (s)**\n\n| Selected Clients ($K_t$) | Algorithm | Client Compute (s) | Server Compute (CPU) (s) | Total Comm. Time (1Gbps) (s) | **Total Round Time (1Gbps) (s)** |\n|:---:|:---|:---:|:---:|:---:|:---:|\n| 2 | **FedASK** | 1.44 | 0.12 | 1.51 | 3.07 |\n| | FedAvg | 1.64 | 0.07 | 1.21 | 2.92 |\n| 10 | **FedASK** | 1.44 | 0.29 | 5.53 | 7.26 |\n| | FedAvg | 1.64 | 0.35 | 4.84 | 6.79 |\n| 30 | **FedASK** | 1.44| 0.69 | 16.61 | 18.74 |\n| | FedAvg | 1.64 | 0.71 | 13.29 | 15.64 |\n\n\nWall-clock time analysis confirms that FedASK is a practical and efficient framework. On the **client side**, FedASK is consistently faster, reducing local computation time by ~0.2 seconds per round because its local DP strategy avoids the backpropagation cost for one of the LoRA matrices. On the **server side**, the additional computation is minimal and does not create a scalability bottleneck since the core decomposition operations do not scale with client numbers, and the total server time remains under one second on a CPU for up to 30 clients.\n\n\n\n# [Q1] On Generalization to Other Architectures\n\nWe would first like to emphasize that the FedASK framework is designed to be architecture-agnostic. Our core method addresses the fundamental mathematical challenges of applying Differential Privacy to the LoRA. Therefore, our framework and theory are general and can be applied to federated learning scenarios that utilize LoRA with differential privacy.\n\nInspired by the reviewer's insightful question, we see that applying FedASK to these novel architectures opens up exciting and non-trivial research directions. For MoE Models, the core challenge is the asymmetric and sparse nature of local updates, as each client only trains the experts activated by its local data. This gives rise to a critical optimization question: Is the standard global aggregation optimal, or should a per-expert aggregation be performed? How to perform wise client selection within heterogeneous experts? \n\nThese are promising research directions that we will be happy to include in the future work and discussion sections of our paper to highlight the extensibility of our framework.\n\n**Reference**\n\n[1] Z. Zhang, S. Di, K. Zhao, S. Jin, D. Tao, Z. Ji, B. Liu, K. A. Alharthi, J. Cao, and F. Cappello. Fedcspc: A cross-silo federated learning system with error-bounded lossy parameter compression. IEEE Transactions on Parallel and Distributed Systems, 2025.\n\n[2] Z. Zhang, D. Cai, Y. Zhang, M. Xu, S. Wang, and A. Zhou. Fedrdma: Communication-efficient cross-silo federated llm via chunked rdma transmission. In Proceedings of the 4th Workshop on Machine Learning and Systems, pages 126–133, 2024.\n\n[3] K. Kuo, A. Raje, K. Rajesh, and V. Smith. Federated lora with sparse communication. arXiv preprint arXiv:2406.05233, 2024.\n\n[4] J. Jiang, B. Cui, C. Zhang, and L. Yu. Heterogeneity-aware distributed parameter servers. In Proceedings of the 2017 ACM International Conference on Management of Data, pages 463–478, 2017."}, {"review_id": "9ORHNCoEL6", "reviewer": "Reviewer_o7K3", "summary": "The manuscript proposed a novel federated LoRA framework to enable effective updating of both low-rank adaptor matrices with robust differential privacy. The proposed method tacked a key limitation of previous methods by enabling updates of both A and B without fixing one of them. Theoretical Analysis were provided for the guarantees of robust differential privacy and precise aggregation. The proposed method achieved SOTA performance in multiple benchmark tasks.", "questions": "1. Is there an explaination for why FedProx performs better than the proposed method in some cases reported in Table 4?\n2. While aggregation fidelity with varying client numbers is shown, the impact of client number on the overall performance of the proposed method remains unclear. Could you report how the method performs across different datasets with varying numbers of clients?", "limitations": "Althought the proposed method improved performance on benchmark datasets, the introduction of two-stage sketching and server-side SVD brings additional computation and communication cost, especially when the number of clients or model dimensions is large. More analysis on runtime or complexity will be helpful.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n    1. The proposed method enables the updates of both A and B, which is a critical limitations of existing methods.\n    2. The empirical performance is better than existing SOTA methods.\n\nWeaknesses:\n    1. The experiments are focused on LLMs for NLP. It would help to discuss whether the proposed method can be extended to other model families, such as vision transformers.\n    2. The discussion on computational and communication cost is limited. More discussion in terms of runtime or communication will be helpful to understanding the strengths of the proposed method.", "was_revised": true, "final_justification": "The rebuttal has satisfactorily addressed my primary concerns regarding computation and communication costs, although the evaluation of the model’s scope remains relatively limited. Considering the performance and the novelty of the idea, I believe the manuscript is slightly above the borderline for acceptance.", "rebuttal": "We appreciate the reviewer for providing insightful and constructive feedback on our work. We have carefully considered all the comments and have conducted new experiments and analyses to address the points raised. \n\n# [W1] On Generalization to Other Modalities and Tasks\n\nWe agree that demonstrating FedASK's applicability beyond NLP is important. To that end, we have conducted new experiments on multi-modal Vision-Language Model and Preference Optimization.\n\n**Architecture-Agnostic Design:**\nWe would first like to emphasize that the FedASK framework is designed to be architecture-agnostic. Our core method addresses the fundamental mathematical challenges of applying Differential Privacy to the LoRA. Therefore, our framework naturally extends to mainstream model, including Vision Transformers, that can be fine-tuned using LoRA.\n\n**New Experiment on a Vision-Language Model and RLHF:**\nTo empirically validate this, we conducted two new sets of experiments.\n* We performed SFT on a Vision-Language Model, specifically fine-tuning a Llava-1.5-7B model[1] on the multi-modal ScienceQA dataset[2] . This task involves 12,000 samples distributed across 10 clients (2 selected per round). Evaluation was performed following the official LLaVA guidelines.\n* We also performed Direct Preference Optimization (DPO), fine-tuning a Llama-2-7B model on the Anthropic/hh-rlhf dataset. This large-scale setup involved 160,000 preference pairs distributed across 50 clients (10 selected per round) and was evaluated using GPT-4 to score helpfulness (Vicuna-Bench) and harmlessness (AdvBench)[3].\n\n**Table 1: VLM Fine-Tuning on ScienceQA (% Accuracy)**\n\n| Privacy Budget ($\\epsilon$) | FedASK | FedAvg | Non-Private |\n|:---|:---:|:---:|:---:|\n| $\\epsilon=1$ | 71.79 | 64.12 | 69.51 |\n| $\\epsilon=3$ | 72.52 | 65.14 | 69.51 |\n| $\\epsilon=6$ | 71.83 | 65.84 | 69.51 |\n\n**Table 2: LLM DPO on Anthropic/hh-rlhf (GPT-4 Score)**\n\n| Privacy Budget ($\\epsilon$) | FedASK (Vicuna) | FedASK (AdvBench) | FedAvg (Vicuna) | FedAvg (AdvBench) |\n|:---|:---:|:---:|:---:|:---:|\n| $\\epsilon=1$ | 7.675 | 38.46 | 7.231 | 30.76 |\n| $\\epsilon=3$ | 7.931 | 40.38 | 7.345 | 31.73 |\n| $\\epsilon=6$ | 7.836 | 40.38 | 7.506 | 32.69 |\n\nFor VLM SFT task, FedASK outperform FedAvg by a significant margin of **over 7 percentage points**. As for DPO task, FedASK achieves a score **~8-9 points higher than FedAvg**. Both indicates a significantly better capacity beyond language models.\n\nOur training framework, developed on `opacus` and `transformers`, naturally supports a wide range of models and algorithms. To the best of our knowledge, no open-source framework currently offers out-of-the-box support for federated learning with integrated DP for LMs. We are committed to releasing our code to the community to facilitate future research in this direction. \n\n\n# [W2] On Computational and Communication Cost\n\nWe thank the reviewer for their valuable suggestion to provide a more detailed discussion on the computational and communication costs. To better illustrate the practical strengths of FedASK, we have performed new analyses of its communication efficiency and end-to-end runtime.\n\n## A. Detailed Analysis of Communication Efficiency\n\n**1. Per-Round Communication Cost**\n\nTo provide the most direct comparison of efficiency, we evaluated which algorithm performs best when allocated the same total communication budget on llama-2-7b. To ensure equivalent communication volume across the algorithms, we selected the results from 400 rounds of FedASK, 500 rounds of FedAvg, and 100 rounds of Scaffold.\n\n**Table 3: Performance Comparison under identical transfer volume (MMLU score)**\n\n| | fedask | fedavg | scaffold |\n|:---|:---:|:---:|:---:|\n| dp=1 | 45.56 | 42.01 | 38.13 |\n| dp=3 | 45.79 | 42.42 | 38.47 |\n| dp=6 | 45.71 | 42.74 | 38.35 |\n\nWhen normalized for total communication volume, FedASK consistently and decisively outperforms both FedAvg and Scaffold across all privacy settings. This demonstrates that FedASK's communication overhead is not just a cost but a high-return investment in a more powerful and accurate aggregation mechanism.\n\nWe also measured per-round communication cost in a simulated cross-silo FL scenario (10 clients, 1Gbps network [4,5], fp16 precision) without system-level optimizations. LoRA was applied to the 'q_proj', 'v_proj', and 'k_proj' modules.\n\n**Table 4: Communication Volume and Simual transfer time of Various Algorithms of 10 selected clients**\n\n| | **Llama-2-7b** | | **Llama-2-13b** | |\n|:---|:---:|:---:|:---:|:---:|\n| | **Comm. Time(s)** | **Total Comm. Vol.(MB)** | **Comm. Time(s)** | **Total Comm. Vol.(MB)** |\n| fedask | 5.53 | 1200 | 34.60 | 7500 |\n| fedavg | 4.43 | 960 | 27.68 | 6000 |\n| scaffold | 8.86 | 1920 | 55.37 | 12000 |\n| flora | 24.36 | 5280 | 152.26 | 33000 |\n\nThe results in Table 1 provide a clear picture of the trade-off. FedASK's total communication per round is 25% higher than that of FedAvg, while requiring 40% less communication than Scaffold and 77% less than Flora. This modest increase boosts substantial performance gains, with FedASK outperforming FedAvg by up to 11.5% on MMLU and 46% on GSM8K under strong privacy.\n\n## B. End-to-End Wall-Clock Runtime Analysis\n\nTo provide a complete picture of practical performance, we measured the end-to-end wall-clock time for a single federated learning round. \n\n**Table 5: End-to-End Wall-Clock Time per communication Round (s)**\n\n| Selected Clients ($K_t$) | Algorithm | Client Compute (s) | Server Compute (CPU) (s) | Total Comm. Time (1Gbps) (s) | Total Round Time (1Gbps) (s) |\n|:---:|:---|:---:|:---:|:---:|:---:|\n| 2 | FedASK | 1.44 | 0.12 | 1.51 | 3.07 |\n| | FedAvg | 1.64 | 0.07 | 1.21 | 2.92 |\n| 10 | FedASK | 1.44 | 0.29 | 5.53 | 7.26 |\n| | FedAvg | 1.64 | 0.35 | 4.84 | 6.79 |\n| 30 | FedASK | 1.44 | 0.69 | 16.61 | 18.74 |\n| | FedAvg | 1.64 | 0.71 | 13.29 | 15.64 |\n\nThis timing analysis confirms that FedASK is a practical and efficient framework.\n* On the client side, FedASK is consistently faster, reducing local computation time because its DP strategy avoids the backpropagation cost for one of the LoRA matrices.\n* On the server side, the additional computation is minimal and does not create a scalability bottleneck, as the core decomposition operations do not scale with the number of clients.\n\n\n# [Q1] On the Comparison with FedProx\n\nThis is an excellent and sharp observation. The performance difference stems from the fact that FedProx and FedASK are optimized for orthogonal primary challenges.\n\n* FedProx's Goal: To mitigate client drift arising from **data heterogeneity** by adding a proximal term to the local objective function. Its core strength lies in handling non-IID data, particularly in non-private settings.\n* FedASK's Goal: To enable the private and effective update of both LoRA matrices under strong **Differential Privacy**, resolving the noise amplification vs. learnability dilemma.\n\nIn the specific high-heterogeneity, moderate-privacy regime of Table 4 ($\\epsilon = 3$), FedProx's explicit non-IID handling provides a marginal edge. However, this is an edge case. As demonstrated across the majority of our results, particularly under stringent privacy ($\\epsilon = 1$), FedASK's more robust and expressive update mechanism provides a consistently and significantly superior solution.\n\n# [Q2] On the Impact of Client Numbers on Performance\n\nWe evaluated FedASK on the MMLU benchmark by varying the total client size, from 10 to 50, while keeping the selection ratio constant at 20% to ensure the same DP noise multiplier.  Our scalability analysis demonstrates FedASK's strong robustness against the increasing data heterogeneity from a larger client number. As shown in the table, even when the number of clients increases from 10 to 50, the model's performance degrades only slightly. For instance, at a privacy budget of $\\epsilon=3$, the MMLU score sees a minor drop from 45.88 to 44.72. This is an expected trade-off, as aggregating a wider diversity of local updates under the constraints of differential privacy becomes increasingly challenging.\n\n**Table6: Performance of FedASK under Different Client Pool Sizes (MMLU score)**\n\n| Clients | MMLU Score ($\\epsilon=1$) | MMLU Score ($\\epsilon=3$) | MMLU Score ($\\epsilon=6$) |\n|:---:|:---:|:---:|:---:|\n| 10 | 45.79 | 45.88 | 45.73 |\n| 20 | 45.58 | 45.81 | 45.65 |\n| 50 | 44.80 | 44.72 | 45.66 |\n\nWe will include this scalability analysis and expand upon it in the revised version. We are grateful for the detailed feedback and believe these new experiments and clarifications have thoroughly addressed the reviewer's concerns.\n\n**Reference**\n\n[1] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023.\n\n[2] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.\n\n[3] R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y. Xu, Y. Du, Y. Wang, and S. Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 6137–6147, 2024.\n\n[4] Z. Zhang, S. Di, K. Zhao, S. Jin, D. Tao, Z. Ji, B. Liu, K. A. Alharthi, J. Cao, and F. Cappello. Fedcspc: A cross-silo federated learning system with error-bounded lossy parameter compression. IEEE Transactions on Parallel and Distributed Systems, 2025.\n\n[5] Z. Zhang, D. Cai, Y. Zhang, M. Xu, S. Wang, and A. Zhou. Fedrdma: Communication-efficient cross-silo federated llm via chunked rdma transmission. In Proceedings of the 4th Workshop on Machine Learning and Systems, pages 126–133, 2024."}]

Author rebuttal:

Meta-review: This paper introduces FedASK (Differentially Private Federated Low-Rank Adaptation with Double SKetching), a novel framework designed to overcome the fundamental trade-off between privacy and model learnability in federated LoRA fine-tuning of large language models. The key challenge addressed is that applying differential privacy (DP) noise to both LoRA matrices (A and B) leads to noise amplification, while fixing one matrix to avoid this severely limits model expressiveness. FedASK resolves this dilemma through a two-stage sketching pipeline inspired by randomized SVD: it first aggregates privacy-preserving sketched updates from clients, then reconstructs and jointly optimizes both low-rank matrices on the server. This design enables effective, differentially private updates of both adaptors while maintaining high utility. The paper provides theoretical analysis establishing both the DP guarantees and precise aggregation properties of FedASK, supported by comprehensive experiments on Llama-2 models across multiple NLP tasks under various data distributions.

The paper received a range of reviewer scores, from a borderline reject (3) to an accept (5), reflecting initial concerns primarily centered on the evaluation scope and practical system overhead. Reviewers universally acknowledged the paper's strong theoretical foundation, technical novelty, and clear motivation. The main points of contention were: (1) the limited initial evaluation on only Llama-2 models and standard NLP tasks, raising questions about generalizability to other architectures (e.g., vision transformers) or training paradigms (e.g., RLHF); and (2) the practical cost of the method, including a two-round communication protocol and server-side SVD computations that scale with the number of clients.

The authors provided a thorough and effective rebuttal, which significantly strengthened the paper. To address the generalizability concern, they conducted and presented new experiments demonstrating FedASK's success on a Vision-Language Model (LLaVA-1.5 on ScienceQA) and a Direct Preference Optimization (DPO) task simulating RLHF, showing substantial performance gains over baselines. To alleviate concerns about overhead, they provided a detailed analysis of communication volume (confirming it remains O(dr)) and presented new end-to-end runtime measurements, showing the server-side computation is manageable in cross-silo settings. These new results directly and convincingly addressed the core criticisms.

As a result of the compelling rebuttal, the reviewers updated their assessments positively. The reviewer who initially scored the paper a 3 (borderline reject) had their concerns mitigated by the new experiments, and the reviewer with significant concerns about overhead explicitly stated their concerns were resolved and upgraded their score to a 5 (accept). While one reviewer noted that the evaluation scope could still be broadened (e.g., to MoE models), they acknowledged the strength of the work and supported acceptance. The consensus is that the authors have made a significant contribution by resolving a critical dilemma in private federated LLM fine-tuning. I recommend acceptance and suggest the authors incorporate the new experimental results and runtime analysis into the final version of the paper for enhanced clarity and impact.
Decision: Accept (poster)