SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA
Authors: Jianmin Liu
Li Yan
Borui Li
Lei Yu
Chao Shen
Venue: ICLR 2026 Poster (2026)
Keywords: Foundation models
LoRA
Homomorphic Encryption
Primary area: foundation or frontier models, including LLMs
TL;DR: SHE-LoRA: Privacy FL for edge devices via selective homomorphic encryption & adaptive LoRA. Matches non-private performance while cutting comms 99.7% and encryption compute 99.8%, comparing to full encryption.

Abstract: Federated fine-tuning is critical for improving the performance of large language models (LLMs) in handling domain-specific tasks while keeping training data decentralized and private.
However, prior work has shown that clients' private data can actually be recovered via gradient inversion attacks.
Existing privacy preservation techniques against such attacks typically entail performance degradation and high costs, making them ill-suited for clients with heterogeneous data distributions and device capabilities.
In this paper, we propose SHE-LoRA, which integrates selective homomorphic encryption (SHE) and low-rank adaptation (LoRA) to enable efficient and privacy-preserving federated tuning of LLMs in cross-device environments.
Based on model parameter sensitivity assessment, heterogeneous clients adaptively negotiate and select a subset of model parameters for homomorphic encryption. 
To ensure accurate model aggregation, we design a column-aware secure aggregation method and customized reparameterization techniques to align the aggregation results with the heterogeneous device capabilities of clients.
Extensive experiments demonstrate that SHE-LoRA maintains performance comparable to non-private baselines, achieves strong resistance to state-of-the-art attacks, and significantly reduces communication overhead by 99.71\% and encryption time by 99.87\%, compared to HE baselines.

Published as a conference paper at ICLR 2026
SHE-LORA: SELECTIVE HOMOMORPHIC ENCRYP-
TION FOR FEDERATED TUNING WITH HETEROGENEOUS
LORA
Jianmin Liu', Li Yan'; Borui Li', Lei Yu², Chao Shen'
ai'an Jiaotong University Rensselaer Polytechnic Institute jianmin. liuêstu. xjtu. edu. en, 11. yan. 880xjtu. edu. cn, boruili@stu. xjtu.edu.cn, cn, yul9erpi. edu, chaoshen@mail xww...tuueeee.... edu.
ABSTRACT
Federated fine-tuning is critical for improving the performance of large language models (LLMs) in handling domain-specific tasks while keeping training data decentralized and private. However, prior work has shown that clients' private data can actually be recovered via gradient inversion attacks. Existing privacy preserva- tion techniques against such attacks typically entail performance degradation and high costs, making them ill-suited for clients with heterogeneous data distributions and device capabilities. In this paper, we propose SHE-LoRd which integrates selective homomorphic encryption (SHE) and low-rank adaptation (LoRA) to en- able efficient and privacy-preserving federated tuning of LLMs in cross-device environments. Based on model parameter sensitivity assessment, heterogeneous clients adaptively negotiate and select a subset of model parameters for homomor- phic eneryption, To ensure accurate model aggregation, we design column-aware secure aggregation method and customized reparameterization techniques to align the aggregation results with the heterogeneous device capabilities of clients. Exten- sive experiments demonstrate that SHE-LoRA maintains performance comparable to non-private baselines, achieves strong resistance to state-of-the-art attacks, and significantly reduces communication overhead by 99.71 % and encryption time by 99.87 99.87% compared to HE baselines.
INTRODUCTION
Large language models (LLMs) have excelled in various tasks, but their deployment in domain- specific applications (e.g., healthcare, finance) often requires private, user-generated data (Durante al., 2024; Huang et al., 2024). However, stringent privacy preservation regulations like GDPR (Voigt & Von dem Bussche, 2017) pose significant barriers to centralized fine-tuning such data. To address this, federated learning (FL) emerged as promising solution by enabling decentralized parammterrefffcient fine-tuning (PEFT) of LLMs without exposing raw data (Zhang al., 2024).
Among various PEFT techniques, Low-Rank Adaptation (LoRA) stands out due to its high efficiency and model quality. It reparameterizes the weight matrix W E Rmxn W Wo + AW Wo + BA, where Wo E mM represents the frozen pre-trained parameters, and B E mm and A E TR"x are the two low-rank adapter matrices to be learned. Given that the rank min(m, n), LoRA significantly reduces both computation and communication costs in federated PEFT. However, recent works (Petrov al., 2024; Balunovic et al., 2022) have shown that the parameters or gradients transmitted during federated PEFT can be exploited via inversion attacks to reconstruct private training data, highlighting the need for stronger privacy protection in federated PEFT with LoRA.
techniques have been proposed to mitigate privacy leakage risks in FL, including differential privacy (DP) (Sun et al., 2024c; Yu al., 2022: Zhu et al., 2025), secure multi-party computation (MPC) (Mugunthan et al., 2019; Kanagavelu et al., 2020; Zheng et al., 2024)), and homomorphic encryption (HE) (Han & Yan, 2023; Jin et al., 2023; Hu Li, 2024)). DP ensures formal privacy guarantees by perturbing data or model updates with random noise. However, in LoRA-based settings that involve the multiplication of A and B, this noise becomes amplified through the multiplication, often hindering convergence and degrading model performance (Sun et al., 2024c). In contrast, cryptographic approaches such as MPC and HE can achieve higher
Many
"Li Yan is the corresponding author of this paper.
Code publicly available https: // //g thub. com/ wtw./2215//2H--LL.R.

---

Published as a conference paper at ICLR 2026
accuracy. MPC-based secure aggregation employs techniques like garbled circuits and secret sharing to securely compute PEFT updates. Nevertheless, it often requires intricately designed computation and synchronization protocols, making it less practical for FL with heterogeneous data and device capabilities (Kairouz et al., 2021; Li et al., 2020). Selective HE (SHE) (Han & Yan, 2023; Jin et al., 2023; Hu & Li, 2024) offers a compelling alternative by encrypting only sensitive parameters and allowing computation over ciphertexts, delivering strong privacy guarantees with low HE costs and preserving accuracy for prieecy-peserving federated PEFT.
However, existing SHE methods struggle to balance privacy and efficiency in cross-device federated PEFT with LoRA, particularly under Non-IID Non-Independently Identically Distributed) data and heterogeneous device capabilities. As discussed in Section 2.4, two observations highlight these challenges: 1) LoRA matrix multiplication makes WW denser, which may increase the number of parameters requiring encryption, and 2) heterogeneous clients produce different encrypted parameter positions, whose union during aggregation expands the enerypted set and inflates HE costs. Driven by these limitations, we aim to adaptively balance security and HE overhead per client in cross-device federated PEFT with LoRA. Achieving this goal requires addressing the following key challenges:
How adaptively apply SHE across heterogeneous clients? Algorithms like FedAvg (McMahan et al., 2017) are not directly applicable to LoRA-based PRFT. Naively applying LoRA requires all clients to use the same low-rank configuration, which is impractical for devices with heterogeneous capabilities. Furthermore, separately aggregating the adapter matrices (A and B) is not mathemat- ically equivalent to full-weight aggregation (Section 2.4). Heterogeneous hardware also prevents clients from encrypting updates of the same size, since clients with limited resources can encrypt only a small subset of parameters while stronger devices may encrypt more. This mismatch disrupts aggregation and complicates reconstruction of low-rank matrices from encrypted weights. Thus, new aggregation algorithm that is efficient, accurate, and compatible with SHE is needed.
. How to avoid expansion of encrypted subsets under SHE? In heterogeneous settings, clients may independently encrypt arbitrary positions in their model parameter matrix. inflating ciphertext size during aggregation. Moreover, mixing plaintext and ciphertext matrices introduces structural disorder, making aggregation inefficient. Without coordinated negotiation of encryption positions, both ciphertext size and aggregation overhead can increase substantially.
To address these challenges, we propose SHE-LORA, which integrates SHE and LoRA enable efficient and preeacy-pessevvigg federated PEFT in cross-device environments. Specifically,
We devise HE subset negotiation mechanism that tailors model-paramett encryption to each client's ' capabilities. Each client assesses its model parameter importance and selects an affordable subset for HE based on its resource constraints and privacy needs. This subset is encoded using order- reder-preserving encryption (OPE) and sent to server, which then negotiates a global HE subset to optimally balance privacy and HE overhead across heterogeneous clients.
We introduce a selective parameter encryption method based on column-swapping parameter obfuscation, which clusters unencrypted and encrypted parameters separately, enabling efficient matrix operations on plaintexts and batch encryption of ciphertexts. Moreover, obfuscating encrypied parameter positions increases adversarial uncertainty and mitigates privacy leakage.
We propose column-aware adaptive aggregation method, which aligns encrypted columns across clients for efficient and accurate aggregation of adapter matrices and subsequent reparameterization to recover LoRA parameters without losing meaningful model updates.
Experiments on clients and LLMs with varying scales demonstrate that SHE-LORA provides strong resistance state-of-the-art (SOTA) attacks while maintaining model performance comparable to non-private basclines. Compared to HE baselines, SHE-LoRA reduces communication overhead by up to 99.71% % and HE overhead by 99.87%.
2
PRELIMINARIES AND MOTIVATIONS
2.1 DEFINITION OF PARAMETER SENSITIVITY
Inspired by model pruning, which removes unimportant model parameters to reduce model size while maintaining performance, SHE identifies and selectively encrypts the most important parameters. Specifically, given model parameters W, let L(W) denote the loss function. For a subset of the model parameters w E W, and the model parameters with w zeroed out (denoted as W-w), the sensitivity of is defined as the change in loss when w is zeroed out:
((w) = (C(W)- L(W -
(1)
2

---

Published as conference paper at ICLR 2026
A
Matrix B
AW
Client
Client 2
Server
Encrypted
Plaintext
2
Encrypted
Plaintext
Figure I: Expansion of encryption positions.
Figure 2: Inflation of ciphertext size.
Eq. (1) implies that larger loss change upon removing indicates higher sensitivity of Thus, ((w) reflects not only importance, but also the potential privacy leakage risk associated with exposing w. However, directly computing (2()) for all parameters is computationally infeasible. Taylor approximation-basee estimation (Frantar et al., 2023) requires gradient computation, which also incurs significant overhead, especially in LLMs, where high dimensionality and outlier activations further exacerbate the cost (Sun et al., 2024a). To mitigate this, we adopt Wanda (Sun et al., 2024b) estimate the sensitivity parameter Way in the i-th row and j-th column as:
??W,)) ) = | [1zy1l2.
(2)
where - is the absolute value operator, [ız is the l2 norm of the j-th features in the input data X. Since both W and X are directly accessíble, Wanda estimates parameter sensitivity with only single forward pass. See Appendix B.1 I for details on the link between high-sensitivity parameters and privacy risk.
2.2 PRIVACY LEAKAGE QUANTIFICATION
We leverage mutual information to quantify privacy leakage caused by the selective encryption of w. Specifically, we assume that once encrypted with HE, w does not leak any privacy information. While for the accessible plaintext portion of the model parameters (i.e., W -w), measure the mutual information shared between W ww and W as:
T(W; W-w) ww-
Σ
Σ NEW
p(r. p(a, loga p())p(y))
NEW
where p(z, y) is the joint probability distribution, p(z) and p(y) are the marginal distributions of W and W W-w. respectively. I(W; W W-w) quantifies the extent of privacy leakage attributable the unencrypted model parameters. As the value of T(W; W increases, the risk of privacy leakage due to the selective encryption of also increases. See Appendix F.3 for implementation details.
2.3 THREAT MODEL.
Following prior works (Jin et al., 2023; Kiani al., 2025), we consider semi-honest adversary A that may compromise the aggregation server subset of clients. While A follows the training protocol, it passively attempts to infer client data from observed model updates. We assume that I) when A compromises subset of clients, it can only infer private information from the clients' local models; 2) when A compromises the aggregation server, it can only infer private information from unencrypted parameters; 3) when both the aggregation server and subset of clients compromised, A can access the private key (shared among all clients) to decrypt model updates sent from benign clients, which be addressed via multi-party HE techniques such multi-key HE, proxy re-encryption, etc. The multi-party HE techniques and protection against other malicious behaviors (e.g., poisoning. backdoor attacks) are not the focus this work, and we refer existing defenses (Zheng et al., 2022; Queyrut et al., 2023) and possible extensions in Appendix C as future endeavors.
2.4 MOTIVATIONS
Naive averaging of LoRAs leads to mathematical errors. Popular federated LoRA methods (Zhang al., 2024; Yan et al., 2024; Meng et al., 2024: Babakniya et al., 2023) require clients possess homogeneous LoRA ranks, and aggregate A and B separately across clients (i.e., server side = = TB). However, this introduces inconsistency aggregation of LoRA updates (i.c., (B A)) is intrinsically unequal ΣB Σ A, which will degrade model performance. Moreover, separately aggregating the LoRA matrices requires all clients to use the same rank, which is unrealistic for cross-device federated LoRA. Thus, these naive approaches are inapplicable to heterogeneous LoRA settings.
ΣB EA A, TA
global model updates, as the
Matrix multiplication expands encryption positions. Albeit the strong privacy guarantee of HE, applying HE per parameter is computationally and communicatively expensive. Although existing SHE methods like FedML-HHE (Jin et al., 2023) and MaskCrypt (Hu & Li, 2024) reduce HE overhead
3

---

Published as a conference paper at ICLR 2026
by selectively encrypting subset of model parameters with masking, the matrix multiplication of LoRA will lead to an expanded HE subset as shown in Fig. 1, which significantly impairs the cost-saving performance of SHE for federated LoRA.
Matrices from heterogeneous clients inflate ciphertext size. Our experiments show that clients with heterogeneous hardwares and Non-III data tend focus different sensitive model parameters during fine-tuning. As a result, the positions selected for encryption vary across clients. In the aggregation phase of existing SHE methods, if client encrypts specific model parameter, the parameter corresponding to the same position must remain encrypted in the global model as well, leading to inflated ciphertext size as the number of clients grows, which illustrated in Fig.
METHOD
In order to adaptively balance security and HE overhead per client in cross-device federated LoRA, encrypting A offers cost-effective solution. Since A directly operates on user data, more vulnerable to inversion attacks (Petrov et al., 2024), making its protection essential for preventing privacy leakage. Following this rationale, the diagram of SHE-LoRA is illustrated Fig.
Step 1. HE Subset Negotiation
Step 3. Adpptive Agrregation
D- griza
Get Plain Part
SVD
AW
P Plain
Df OVE aa
Server
Get HE Cipher
C Cipher
Uipheresii
Lo maao
resint
Aggregation Result
Client LoRAs
Model
Step 2. Selective E Encryption of Model Parameter Matrix
Step 4. Reparameterization
Column Swapping
Before HE Subset 1
Clennt?
B
A
Fusion Cipher& Plain to Client
Column Swapping
x
Before HE Subset
After
An
lient N
By
Fusion Cipher& to ClientN
Heterogeneous Devices
SHE-LORA After Local Updates
LoRA Updates For FL
Figure 3: The workflow of SHE-LoRA.
SHE- consists of the following components: Step 1. HE Subset Negotiation. Based on the definition model parameter sensitivity in Eq. (2), each client assesses and transmits its model param- eter importance to server. Then, the server negotiates a global HE subset and feeds it back to clients. Step 2. Selective Encryption of Model Parameter Matrix. Based on the global subset, clients perform column swapping to separately cluster unencrypted and encrypted parameters, enabling efficient matrix operations on plaintexts, batch encryption ciphertexts, and parameter position obfuscation that enhances privacy protection and HE efficiency. Step 3. Adaptive Aggregation. The server performs adaptive, column-awar aggregation of the clients' unencrypted and encrypted parameters, respectively, enabling efficient and accurate aggregation of adapter matrices. Step 4. Reparameterizatiio. Fach client reparameterizes the aggregated plaintext and ciphertext results into LoRA parameters, matching its local rank for the next round of model tuning.
3.1 HE SUBSET NEGOTIATION
3.1.1 ASSESSMENT OF MODEL PARAMETER IMPORTANCE
Fig. 4 shows the sensitivity of parameters measured by Eq. (2), where darker color indicates higher importance. We find that the sensitivity values generally differ by columns, suggest- ing strong correlation with specific data channels. Considering that encrypting gle element in column will result in the expansion of encryption positions for that entire col- due matrix multiplication (Section 2.4), and vectorized encryption by columns is bene-
sin-
4

---

Published as a conference paper at ICLR 2026
ficial improving HE efficiency (Cheon et al., 2017), let each client assess parameter im- portance by columns and determine its HE subset of columns based on its encryption budget. Specifically, the eneryption budget of Client i is de- fined as the ratio of parameters for SHE (denoted as "s E [0, 1). which is specified according its hard- ware capabilities such as CPU clock speed, etc. For an adapter matrix A E grx and input X _x we use S Σx= k= - importance of the j-th channel, where || ll2 is the l2 norm of the j-th feature 2, E RR, Thus, the proposed approach can not only provide channel-level privacy protection, but also prevent unnecessary expansion of encryption positions.
Layer 0 Rank: 16 Part: [:,3264]]
0.10
0.06
to calculate the
10
12
0.02
10
15
20
25
30
Figure 4: Sensitivity of model parameters.
3.1.2 HE SUBSET NEGOTIATION
As explained in Section 2.4, clients with heterogeneous data distributions and device capabilities will select different positions and amounts of important model parameters for encryption according to their individual budgets. As the union of HE subsets expands with the aggregation of client model updates, the ciphertext size may inflate significantly. To address this, we devise a HE subset negotiation mechanism to tailor client-specific encryption of model parameters, ensuring that the overall ciphertext size remains affordable per client.
To prevent the server from snooping on the positions of important model parameters and conducting targeted attacks, we apply OPE to hide each client's HE subset. Since SHE-LoRA is modular, OPE can be replaced by alternatives like order-revealing encryption, secure multi-party computation, or trusted third party whenever stronger order privacy is required. As OPE only preserves numerical ordering of the plaintext, the server cannot obtain any information from the cipher other than the plaintext order. Specifically, Client i first encrypts a tuple (G,, St) with OPE and sends it to the server, where G; is the set of columns that needs HE, and S, their sensitivities. Then, the server maintains two shared lists based on all the clients' tuples: the Common list, which sorts all columns in U, G2 from most to least frequently deemed as sensitive, and the Sensitivity list, which sorts all columns in U G from highest to lowest sensitivity. Finally, by taking into account both the overlap of important HE subsets across clients (reflected Common and Sensitivity) and the preferred HE subsets of individual clients (reflected in G,), the server negotiates global HE subset affordable for each client, of which algorithmic details are elaborated in Appendix D. As a result, the negotiated global HE subset optimally balances privacy and HE overhead per client.
3.2 SELECTIVE ENCRYPTION OF MODEL PARAMETER MATRIX
As illustrated in the top of Fig. 5, the selected columns for encryption may be scattered across the parameter matrix. This irregular distribution increases the complexity of matrix batch- ing and the overhead of encryption, decryption and computation. To address this, based the negotiated HE subset, we propose a column-swapping method to separately cluster the columns pending for encryption and those remain unencrypted. This approach brings three key benefits: (1) encrypted columns are clustered together, allowing for efficient batch encryption with reduced storage and communication overhead; (2) the clustered unencrypted columns can be directly used in matrix operations, improving computational efficiency; and (3) the column-wise obfuscation increases the difficulty of potential privacy attacks.
A E R rXn
Column swapping
A Aplann
AE -M
br Column chunking
Figure 5: Selective encryption of columns.
After swapping, Client i selectively encrypts its parameter matrix, and uses the last k; = nn x % columns as its HE subset, which is affordable according to the client's encryption budget. Thus, the tensor to be encrypted, denoted as A', has the shape of (r, k;)) In HE, the CKKS (Cheon-King Kiinn-Kim-Kiiii Kim- Song) scheme (Cheon et al., 2017) serves as an encryption technique that facilitates approximate floating-por -pinnt arithmetic, making it particularly advantageous for the encryption of matrices tensors. For large-scale tensors, the limited capacity of HE requires them to partitioned into blocks, each encrypted separately. As illustrated in the bottom of Fig. 5, given a block size of chunk, A' can be
5

---

Published as a conference paper at ICLR 2026
divided into Nº [k,/chunk] blocks by column (denoted A' {bi....,by bj.............................. ,j....,bn' bxy }), where each block contains a tensor that has the shape (r, chunk). For each block bj. the client applies CKKS encryption to obtain its ciphertext Cj (K pk), where pk is the public HE key. After all blocks are encrypted, the complete list of ciphertext blocks (C₂) N" sent to the server.
3.3 ADAPTIVE AGGREGATION
To prevent the inflation of ciphertext size caused by aggregating heterogeneous HE subsets as illustrated in Section 2.4, we propose am adaptive column-aware aggregation method to aggregate the unencrypted and encrypted parts separately.
Aggregation of Unenerypted Model Parameters: Upon receiving the adapter matrix B, and the unencrypted part of A; (denoted as A plain) from Client i, the server calculates the unencrypted weight update from Client as A AWPlin B, plain However, as the number of encrypted columns (i.e., k;) differs across the clients due their diverse encryption budgets, the shape of A W E (m also varies across the clients, which makes the traditional aggregation methods of weight-averaging A Willain A, plain inapplicable. Considering that the unencrypted columns of all the clients have been clustered to the left as explained in Section 3.2, we let the server apply column-win weighted averaging aggregate the unencrypted model parameters column-wise partial aggregation is consistent with standard FedAvg (McMahan et al., 2017) under client subsampling and ensures bias is introduced from non-contributing clients. The detailed algorithm is elaborated in Appendix D.2.
illustrated in the top of Fig. 6. This
a
B 17
a,
B
17
ag
B
17:
B
n
+
+
Client I:A plate
Client 2:AW plai
Client 3: AW,
Server: AW plate
Aggregation:
a - ( ta, +a,)
B ( B.)
1/
3
B
B3
17
B.
17,
a
B 17
+
Client 1: A A. eppeer
Client 2: AWW Sphpperr
Client 3: AW
Server: AW
sypar
Aggregation: a =
B= B
[n + n7))
Unenerypted Parameters
Enerypted Parameters
Zero Padding
Column-wise Weighted Averaging
Figure 6: Aggregation of unencrypted (top) and encrypted (bottom) model parameters.
Aggregation of Enerypted Model Parameters: Similarly, upon receiving B, and the encrypted part of A, (denoted as A, cipher {C}}^") from Client i, the server calculates the encrypted weight update from Client as A Weiphee B,A, cipber <Rmx: Although the difference in k; also causes wipherr to have different shapes, the cipher block order is consistent across the clients, and the encrypted columns of all the clients have been clustered to the right explained in Section 3.2. Thus, the encrypted model parameters can be similarly aggregated via column- wisse weighted averaging as illustrated in the bottom of Fig. 6. The detailed algorithm is elaborated in Appendix D.3.
To reduce communication overhead, the server sends the unencrypted and encrypted parts separately:
For the unencrypted part, since most model parameters remain unencrypted during LoRA, we let the server, which generally has more computation capability, apply singular value decomposition (SVD) to decompose the aggregation result: A AW plain
UpE.V E m where K min(k;), Up E mmm Σp E R and VT Kxk Then, given each client's rank r;, the server slices
SVD

---

Published as a conference paper at ICLR 2026
the decomposition results as: Upl:: ra] E Rmx,, Epl: (p::rl R and V ri E R Thus, each client receives the unenerypted aggregation result corresponding to its own rank. For the encrypted part, since each client only selectively encrypts small portion of model parameters, we let the server first truncate the aggregation result by the encryption budget of respective clients, and then let the clients decrypt their corresponding aggregation results as in Section 3.4.
3.4 REPARAMETERIZATION
To enable the next round of model tuning, each client needs merge the plaintext and ciphertext results returned by the server into new LoRA parameters. Suppose the client receives the plaintext
decomposition result: AW plain
SVD BpAp, where Bp Up Dop/pp> E R and Ap = EEp V Re*K and the cipher blocks from the server. For the plaintext decomposition results, the client directly zero-pads Bp and Ap to the shapes of (m,,) r) and (r, n). respectively, for further update of its model parameter matrix. For the cipher blocks, the client first decrypts the ciphertext into plaintext,
and then performs SVD and zero-padding to obtain the decomposed results as A cipher where B. V.v De and A - v V V By merging the decomposition results of the plaintext and ciphertext via Eq. (4) and restoring the position of the model parameters according to the corresponding positions in Section 3.2, the correct aggregation result can be reparameterized as:
SVD B.A..
(4) Finally, the client performs SVD on Ag and Bg again, and re-adjusts the parameter shapes according to the client's 's rank to B E Rmxxr and Á E RRxx,, As the SVD conducted by the client is all on low- rank matrices, the computation overhead is trivial. The detailed derivation of the reparameterizzation is presented in Appendix D.4. The formal proof of the losslessness of meaningful model updates in SHE-LORA is provided in Appendix D.5.
Bg [B Be] [pp Σp U.V Ee| E R=x((tt) Ag
Ap Ac
Ep
E (
V..
3.5 PRIVACY GUARANTEE OF SHE-LORA
The parameters protected by HE do not leak privacy, and privacy risks are mainly caused by un- encrypted parameters. The column swapping process, which serves as column- -wise obfuscation, increases the difficulty of privacy attacks, and can be viewed as adding an asymptotic Gaussian- distributed noise (with 22 as the equivalent variance) to the original gradient as detailed in Ap- pendix D.7.
Thus, the privacy guarantee of selective encryption can be given by the Bayesian Cramér-Rao Lower Bound (Chen et al., 2025; Huang et al., 2025). Specifically, the reconstruction error ER can be defined as the minimum expected squared reconstruction error:
ER min En-xXB-((att)) [ - -x =1 R
(5)
where R--) is any data reconstruction attack method, [-)) is the selective HE method of SHE-LoRA, and g(x) is the gradient calculated on data x. Then, the Bayesian Cramér-Rao Lower Bound can be given by (Huang et al., 2025):
d²
d
ER 2 (==)
(6)
-0- V ( + (Jp))
where JF(x) is the Fisher information matrix, tr(-) is the trace operator, Ae(Jp) is the largest eigenvalue of the prior-informed Fisher information matrix Jp, is the data dimension, n is the number of columns in G, y is the encryption ratio, s² is the variance of the equivalent noise, and [V)=g(>)] = (max,,, V (ag)(x,))) quantifies the maximum gradient exposure. which is defined as the squared maximum sensitivity of an unencrypted (exposed) gradient coordinate gn (1;;) with respect to data feature ...
Eq. (6) means that the reconstruction error Ex is lower-bounded by a quantity whose denominator depends on (x))) and (Jp). Since (Jp)) is determined solely by the data prior, the bound under fixed n, 5², and is governed exclusively by ( | By selectively encrypting the most sensitive columns, which are measured in terms of the Wanda parameter sensitivity (a proxy for their contribution input-space sensitivity via the score S, in Line 217). SHE-LoRA suppresses the dominant terms in the Fisher information matrix, thereby reducing the magnitude of the unencrypted gradients VV [2] and lowering tr( (r())
7

---

Published as a conference paper at ICLR 2026
In summary, by encrypting the most sensitive parameters, SHE-LoRA increases the minimum achievable reconstruction error and strengthens privacy against any gradient inversion attack.
PERFORMANCE EVALUATION
4.1 EXPERIMENTAL. SETUP
Table 1: Heterogeneous device types.
Model: We select the Bert- (Devlin et al., 2018) model and the OpenI_LaMA- 3B (Geng& & Liu, 2023) model for perfor- mance evaluation across diverse task scenar- ios. More evaluation settings and results larger LLMs can be found in Appendix E. 4
Type
GFlops
Rank
Budget (Bert, LLaMA)
#
105.2
8
(0.4%, 0. 0.125%
20
2
165.5
16
(0.4%, 0.125%)
15
3 216.9
16
(0.8%, 0.25%)
10
243.1
32
(1.6%, 0.50%
5
Datasets: We use the IMDB (Maas et al., 201 and natural-instructions datasets (Wang et al., 2022)
for natural language understanding and generation tasks, respectively. We use the MMLU (Hendrycks ct al., 2021) and the GLUE (Wang et al., 2019) benchmarks for evaluation on natural language generation and natural language understanding tasks, respectively. For evaluation on vision tasks, we use the MNIST (LeCun et al., 2002), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), GTSRB (Stallkamp et al., 2012), SVHN (Netzer et al., 201 1) visual classification datasets.
Hyperparameters of HE: We adopt the CKKS implementation from the TenSEAL library (Benaissa et al., 2021) for HE operations. As instructed', we set the polynomial degree to 8192, 2048, and the modules chain to [60, 40, 60], [20,20] for OpenLLaMA-3B and Bert-Large, respectively.
Implementation: SHE-LORA is implemented with PyTorcc based on the Flower framework?. We deploy federated LoRA of LLMs on 50 clients for 200 rounds. The heterogeneous data partitioning is instantiated via Dirichlet distribution with 0.3. The evaluation settings and results more clients can be found in Appendix 1. As detailed in Table 1, we configure four types of client devices with varying computing capabilities, LoRA ranks and encryption budgets. Without losing generality, we posit that weaker devices are characterized by lower ranks and encryption budgets, while stronger devices are capable of supporting higher ranks and encryption budgets.
4.2 MODEL TUNING PERFORMANCE
We compare the model tuning performance of SHE-LORA with two homogeneous LoRd-based methods (FedIT (Zhang et al., 2024) and FedSA (Guo et al., 2024)) and two heterogeneous LoRA- based methods (HeterLoRA (Cho al., 2024) and Flex-LoRA (Bai al., 2024)). The methods' performance on natural language understanding tasks and natural language generation tasks is evaluated on the GLUE benchmark (Wang al., 2019) and the MMLL benchmark (Hendrycks et al., 2021), respectively, while the methods' performance visual classification datasets. SHE-LORA achieves comparable performance to the SOTA method (Flex-LoRA) and outperforms the other baselines. Detailed results are elaborated in Appendix F.
vision tasks is evaluated on the 5
4.3
HE COST EFFICIENCY
To our best knowledge, SHE-LORA is the first integrate SHE into federated LoRA of LLMs. We implement two methods for the comparison of HE cost efficiency: (1) MaskCrypt (Hu & Li, 2024), the SOTA SHE method for securing FL, and (2) Baseline, the vanilla method with full HE of LoRA parameters. Specifically, MaskCrypt lets each client select an encryption mask and uses the union of the masks for global SHE of parameters during FL. Baseline uses the stock implementation of CKKS (Benaissa et al., 2021) to encrypt each LoRA parameter. The clients' device specifications follow Table 1. We collect the encryption time and communication overhead of all the clients under the methods per round during the federated tuning of the OpenI LaMM-3B and Bert-Large models. Fig. 7 and Fig. 8 show the collected results, where the bar represents the mean, and the lines extending upward and downward from the mean represent the maximum and minimum values, respectively.
Encryption Time: Baseline always consumes the longest encryption time as it encrypts each LoRA parameter. In comparison, MaskCrypt greatly shortens the encryption time on both models, which is primarily due to its selective encryption of partial parameters. However, the clients' encryption time in both Baseline and MaskCrypt severely fluctuates within [311s, 653s] and [1.556s, 104.63s] on OpenI OpenI_LaMAA LaMA-3B, and [12s, 60s) and [0.27s, 39.75s] on Bert-Large, respectively. This is because that these methods cannot deal with the inflation of the global HE mask caused by matrix multiplication
ai/
8

---

Published as a conference paper at ICLR 2026
OpenLLaMA-38
Bert-Large
19.0
x10 OpenLLaMA-38
Bert-Large
0.66
0.54
0.601
0.054
Comm evertead
18.5
100
400
40
75-
18.0-
Encrypton
100
50
20-
25
Bateline
SHE-CCSHEELORR SHE-LORA
Basetine
MaskCrypt SHE-LORA
Baseline
Baseline MaskCryet LoRA
Figure 7: Encryption time.
Figure 8: Communication overhead.
and mask heterogeneity (Section 2.4). Thus, the clients in these methods, whether weak or strong, have to encrypt the same amount of parameters, causing highly imbalanced encryption time. In contrast, thanks to the column swapping and clustering of encrypted columns, which enables efficient utilization of CKKS key sizes, SHE-LORA reduces the mean encryption time by 99.87%~~98. 10% and 99.81 96-99.311999 1% as compared to Baseline and MaskCrypt on the OpenLLaMA-3B and Bert-Large models, respectively. Moreover, we notice that even if the clients differ in computing capabilities, their encryption time under SHE-LoRA hardly fluctuates. The primary reason is that each client in SHE-LoRA can choose an affordable encryption budget % that matches its device capability, and hence results in no significant difference in encryption time across the clients.
Communication Overhead: The mean values under the methods follow similar rankings as those in encryption time, where SHE-LORA reduces communication overhead by 99.71 98. 19.9%%...1%.... on OpenLLaMA-3B and 98.78%~998. 18% on Bert-Large as compared to Baseline and MaskCrypt, respectively. The major difference lies in variations, where the communication overhead stays constant in Baseline and MaskCrypt, but fluctuates in SHE-LoRA. This is consistent with their designs: Baseline lets each client encrypt all the parameters (full ciphertext size) and MaskCrypt lets clients encrypt the global union of masks (partial ciphertext size), while SHE-LoRA lets clients choose encryption budgets that match their device capabilities (diverse ciphertext sizes).
4.4 RESISTANCE TO PRIVACY ATTACK
SHE-LORA is applicable both parameter and gradient updates. Although parameter- carseeter-based attacks can be transformed into gradient-based ones, they are highly inaccurate and impractical for large models, even with direct access to gradients (Wang & Li, 2024). Therefore, we evaluate the resistance of SHE-LoRA against the DAGER attack (Petrov et al., 2024), which is the SOTA gradient inversion attack method. DAGER exploits the fact that gradients are linear combinations of input embeddings (Appendix B.2). It iterates over the entire vocabulary and measures the distance between each embedding vector and the principal components of the gradient, aiming to identify tokens. More results against membership inference attacks are detailed in Appendix F.4.
We conduct the federated LoRA of the OpenLLaMA-3R model via SHE-LORA and MaskCrypt on two datasets, SST2 (Socher et al., 2013) and Rotten Tomatoes (Pang & Lee, 2005), as in DAGER, with r=256 and HE settings in Section 4. 1. For fair comparison, we let them use the same HE overhead (ciphertext size). We also implement a non-private SOTA federated LoRA method, Flex-LoRA, and its grivacy-rreservig form with DP protection, Flex-LoRA-DP, where gradients are obfuscated with o² noise, and 0 10-3 as in DAGER. We launch the DAGER attack on gradients per training round for each method over the two datasets, respectively, and collect the data reconstruction scores of DAGER under the batch sizes of 4, 8, 16. The scores are collected in terms of "ROUGE-1" (R-I in short), which measures the matching degree of unigrams, and "ROUGE-2" (R-2 in short), which measures the matching degree of bigrams. Smaller scores reflect better privacy protection.
Table 2 shows the mean and standard deviation of the scores. In practice, if SHE-LORA and MaskCrypt use the encryption budget of the weakest device in Table (i.c., 7/2 0.125%) for SHE, DAGER completely fails on both methods under all settings (i.c., scores=0). Thus, we gradually decrease 7/2 and find that DAGER begins to succeed in partially compromising SHE-LORA when %s < 0.3%,, which consumes only one ciphertext packet. In contrast, MaskCrypt is no longer secure under the same HE overhead, while DP is far less secure than SHE-LORA and may significantly degrade model accuracy (Sun et al., 2024c). The strong resistance of SHE-LORA against DAGER is primarily due to the column swapping and SHE of important columns. Although the change in the principal components of gradients caused by column swapping is trivial, which does not lead to the failure of DAGER, such change leads to a strong perturbation in the orthogonal complement of gradients in the low-rank space of LoRA parameters, which causes the failure of DAGER's 's span

---

Published as a conference paper at ICLR 2026
Table 2: Data reconstruction scores of DAGER.
B=4
B=8
B=16
Dataset
Method
R-1
R-2
R-1
R-2
R-1
R-2
Flex-LoRA
9..18.1..6
94.66.1..8
61.1411.9
52.49.1.22
10.27:11.6
5.86.11.2
Fexx-LRRA-DP
86.2511.11
86.11
80.2811.11
78.
68.62
66.44.1..7
SST2
MaskCrypt
89.16:11.3
87.93
61.4912.2
61.49:1.4
10.91
6.791114
SHE-LoRA
0.72:15.2
0.12
0.98:4..4
0.14:00.6
0.0=0.0
0.0:10.0
Flex-LoRA
38.44:1:..5
1..
3.76::1.4
2.124:2.1
0.0=0.0
0.0.:000
Rotten
Flex-LoRA-DP
36.74:1..9
31.28:12.6
3.76.11.3
2.02.1..3
0.0
0.0.10.0
Tomatoes
MaskCrypt
31.65#2.0
25.11#2.6
6.09#1.0
3.27#1.2
0.0=0.0
0.0+0.00
SHE-LORA
0.0.0..0
0.0
0.010.0
0.0
0.0
0.040.0
check. In addition, the key gradient information for reconstructing data has also been protected by SHE, DAGER completely fails when the batch size is greater than 8 even only 0.3%e parameters encrypted.
We further decrease the batch size to 1 (i.e., casiest for in-
100
version attack) with "i 0.3% in SHE-LoRA, while grad- ually increasing the ratio= MaskCrypt HE overhead SHE-LORA HE overbead and measure
SHE-LORA
80
Score
* R-2 SHE-LORA
60
R-1 MaskCrypt
the data reconstruction scores of DAGER under the two methods, which are shown in Fig. 9. Due ciphertext inflation, when MaskCrypt HE overhead =1, MaskCrypt inef- ficient at protecting sufficient parameters against DAGER. To match SHE-LoRA's security, MaskCrypt has con- sume > x00x the HE overhead of SHE-LoRA, making it unsuitable for weak clients.
40
MaskCrypt
20
SHE-LORA HE overhend
*
1
20
40
60
80
100
MaskCrypt overhead
SHE-LORA HE overhead
Figure 9: Resistance comparison.
4.5 PRIVACY LEAKAGE ANALYSIS FROM MUTUAL INFORMATION PERSPECTIVE
To delve into the root cause of SHE-LORA's effectiveness, we perform a detailed analysis on how encryption budget and SHE strategy affect privacy leakage from the perspec-
Mutual information
x10-4
Max
Min
tive of mutual information. Specifically, we implement three naive encryption strategies: (1) Max: the most im- portant parameters are prioritized for encryption: (2) Min:
Random
the least important parameters are prioritized for encryp- tion; (3) Random: parameters are randomly selected for encryption. Then, we gradually increase the encryption budget from 0.3% 80%, and measure the mutual infor- mation shared between full parameters and the selectively encrypted parameters per strategy via an efficient approach based on the kernel density estimators (Appendix F.3). Fig. 10 shows the measured results. We find that the mutual information in Max drops much faster than the others, while the mutual information in Min drops the slowest, reflecting strong correlation between parameter importance and privacy leakage risk. The cost-effective protection of important parameters is the key to the effectiveness SHE-LoRA against DAGER-IIE -like attacks.
0.3%
10%
80%
Encryption budget
Figure 10: Impact of encryption budget
and strategy on mutual information.
5 CONCLUSION
We propose SHE-LoRA, a framework integrating SHE and LoRA for efficient and privacy-preserving federated tuning of LLMs in cross-device environments. It constrains the expansion of ciphertexts through HE subset negotiation, enables tailored privacy protection via selective encryption of parame- ters based on column- column-swapping parameter obfuscation, and achieves efficient and accurate update of LoRA parameters by column-aware adaptive aggregation and subsequent reparameterization, Results show that SHE-LoRA maintains model tuning performance comparable to non-private baselines, while achieving strong resistance to SOTA attacks, and significantly reducing communication over- head by 99.71% 71% and encryption time by 99. 97%, compared to HE baselines. Our work demonstrates that SHE with a well balance between privacy and utility can secure federated LoRA of LLMs against DiGER-like attacks and membership inference attacks. We hope SHE-LoRA can foster further research into creating more reliable and cost-effective privacy protection frameworks for private collaborative learning.
10

---

Published as a conference paper at ICLR 2026
6
ACKNOWLEDGEMENTS
This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFB4500800, in part by the National Natural Science Foundation China under Grant 62103325, in part by the Shaanxi High-Level Talent Program under Grant 20210CYRC4-26.
7 REPRODUCIBILITY STATEMENT
The paper has fully disclosed all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and conclusions of the paper. We have provided a link (https: / /gi thub. htmm1lyyan211/SHEE.E explanations and annotations to support the reproducibility our work. addition, the necessary hyperparameter settings are described in Section 4.1.
to downloadable source code with detailed
REFERENCES
Rakesh Agrawal, Jerry Kiernan, Ramakrishnan Srikant, and Yirong Xu. Order preserving encryption for numeric data. In Proc. of SIGMOD, pp. 563-574, New York, NY, USA, 2004.
Asma Aloufi, Peizhao Hu, Yongsoo Song, and Kristin Lauter. Computing blindfolded on data homomorphically encrypted under multiple keys: survey, ACM CSUR., 54(9), October 2021. ISSN 0360-0300.
Giuseppe Ateniese, Kevin Fu, Matthew Green, and Susan Hohenberger. Improved proxy re-encryption schemes with applications to secure distributed storage. ACM TISSEC, 9(1):1-30, 2006.
Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.0655: 2023.
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In Proc. of NeurIPS, 2024.
Mislav Balunovic, Dimitar Dimitrov, Nikola Jovanović, and Martin Vechev. Lamp: Extracting text from gradients with language model priors. In Proc. of NeurlPS, 2022.
Ayoub Benaissa, Bilal Retiat, Bogdan Cebere, and Alaa Eddine Belfedhal. Tenseal: A library for encrypted tensor operations using homomorphic encryption. arxiv 2021. arXiv preprint 1rXi::2104.03152,1 2021.
Matt Blaze, Gerrit Bleumer, and Martin Strauss. Divertible protocols and atomic proxy cryptography. In Proc. of TACT, pp. 127-144. Springer, 1998.
Alexandra Boldyreva, Nathan Chenette, Younho Lee, and Adam O'neill Order Doder-preserving symmetric encryption. In Proc. of EUROCRYPT, pp. 224-241. Springer, 2009.
E Bolthausen. An estimate of the remainder in combinatorial central limit theorem. Probability Theory and Related Fields, 66(3):379-386, 1984.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In USENIX security, pp. 2633-2650, 2021.
Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, and Albert Y Zomaya. Rbla: Rank-based-lora- aggregation for fine-tuning heterogeneous models in flaas. In Proc. of ICWS, 2024.
Yuxiao Chen, Gamze Gursoy, and Qi Lei. Optimal defenses against data reconstruction attacks, In ICML Workshop. 2025.
Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic encryption for arithmetic of approximate numbers. In Proc. ASIACRYPT, 2017.
11

---

Published as a conference paper at ICLR 2026
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proc. of EMNLP, 2024.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proc. of CVPR, pp. 3606-3613, 2014.
Harald Cramér and Herman Wold. Some theorems on distribution functions. Journal of the London Mathematical Society, 144):290-29, 1936.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. Agent ai: Surveying the horizons of multimodal interaction. arXiv preprint arXiv:2401.03335 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot. In Proc. of ICML, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. Proc. of ICLR, 2023.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023, URL https: //gi thub. com/ reseeenmmmeeearrchooee.em.a
Craig Gentry. Fully homomorphic encryption using ideal lattices. In Proc. of STOC, pp. 169-178, 2009.
Pengxin Guo, Shuang Zeng, Yanran Wang. Hujjie Fan, Feifci Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. arXiv preprint arXiv:2410.01663, 2024.
Jaroslav Hájek. Some extensions of the wald-wollowitz-noether theorem. The Annals of Mathematical Statistics, pp. 506-523, 1961.
Junhao Han and Li Yan. Adaptive batch homomorphic encryption for joint federated learning in cross-device scenarios. IEEE loT-J, (1())333889935) 2023.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In Proc. of ICNN, pp. 293-299. IEEE, 1993.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J-STARS, 12(7): 2217-2226, 2019.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In Proc. of ICLR, 2021.
Wassily Hoeffding, A combinatorial central limit theorem. The Annals of Mathematical Statistics, pp. 558-566, 1951.
Chenghao Hu and Baochun Li. Maskcrypt: Federated learning with selective homomorphic encryp- tion. IEEE TDSC, 2024.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In Proc. of ICLR, 2022.
Qiuyuan Huang, Naoki Wake, Bidipta Sarkar, Zane Durante, Ran Gong, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Noboru Kuno, Ade Famoti, al. Position paper: Agent ai towards holistic intelligence. arXiv preprint 20t::24003008333 2024.
Ren-V Yi Huang, Dumindu Samaraweera, Prashant Shekhar, and Morris Chang. Advancing practical homomorphic encryption for federated learning: Theoretical guarantees and efficiency optimiza- tions. arXiv preprini 2509v:2509. 20476, 2025.
12

---

Published as a conference paper at ICLR 2026
Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. In Proc. of NeurlPS, 2021.
Weizhao Jin, Yuhang Yao, Shanshan Han, Jiajun Gu, Carlee Joe- Wene Srivatsan Ravi, Salman Avestimehr, and Chaoyang He. Fedml-he: An efficient woonomopphicc-ncrytiio---sss privacy- preserving federated learning system. arXiv preprint arXiv:2303. 10837, 2023.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends in Machine Learning, 14 (1-2), 2021.
Renuga Kanagavelu, Zengxiang Li, Juniarto Samsudin, Yechao Yang, Feng Yang, Rick Siow Mong Goh, Mervyn Cheah, Praewpiraya Wiwatphonthana, Khajonpong Akkarajitsakul, and Shangguang Wang- Two-phase multi-party computation enabled privacy-preserving federated learning. In Proc. of CCGrid, 2020.
Shahrzad Kiani, Nupur Kulkarni, Adam Dziedzic, Stark Draper, and Franziska Boenisch. Dif- ferentially private federated learning with time-adaptive privacy spending. Proc. of ICLR, 2025.
Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some applications. IEEE TAC, 2542):164-176, 1980.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proc. of the IEEE, 86(1 (1:))2222222222 2002.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. In Proc. of SPM, 2020.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Pruc. of ACL, pp. 142-150, Portland, Oregon, USA, June 2011.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiienn learning of deep networks from decentralized data. In Proc. of AISTATS, 2017.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In Proc. of NeurIPS, 2024.
Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52(3):2318, 1995.
Vaikkunth Mugunthan, Antigoni Polychroniadou, David Byrd, and Tucker Hybinette Balch. Smpai: Secure multi-party computation for federated learning. In Proc. of NeurIPS, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop, volume 2011, pp. Granada, 2011.
Bo Pang and Lillian Lec. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. In Proc. ACI, pp. 115-124, 2005.
Ivo Petrov, Dimitar Dimitrov, Maximilian Baader, Mark Müller, and Martin Vechev. Dager: Exact gradient inversion for large language models. In Proc. of NeurlPS, 2024.
Simon Queyrut, Valerio Schiavoni, and Pascal Felber. Mitigating adversarial attacks in federated learning with trusted execution environments. In Proc. of ICDCS, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. of ICMI., pp. 8748-8763. PmLR, 2021.
13

---

Published as a conference paper at ICLR 2026
Ronald L Rivest, Len Adleman, Michael L Dertouzos, al. On data banks and privacy homomor- phisms. Foundations of secure computation, (111))669-180) 1978.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang. Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In Proc. of ICLR, 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proc. of EMNLP, pp. 1631-1642, 2013.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
Shangchao Su, Bin Li, and Xiangyang Xue. Fedra: A random allocation strategy for federated tuning unleash the power of heterogeneous clients. In Proc. of ECCV, 2024.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In Proc. of ICLR Workshop. 2024a.
Mingje Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. In Proc. of ICLR, 2024b.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving LoRA in preacccpreseriinggg federated learning. Proc. of ICLR, 2024c.
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Proc. of NeurlPS, 26, 2013.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Proc. of NeurlPS, 2024.
Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A practical guide, 1st ed., Cham: Springer International Publishing. 10(3152676;:0-55553 5551000055555550 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Pruc. of ICLR, 2019.
Fei Wang and Baochun Li. Data reconstruction and protection in federated learning for fine-tuning large language models. IEEE TBD, pp. 2024. 2024.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap, et al. Super-naturrlinstttuttinss Generalization via declarative instructions on 1600+ nlp tasks. In Proc. of EMNLP, 2022.
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.0559 2024.
Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Gong, and Bhuwan Dhingra. Recall: Membership inference via relative conditional log-likelihoods. In Proc. of EMNLP, pp. 8671-8689, 2024.
Yuxuan Yan, Qianqian Yang, Shunpu Tang, and Zhiguo Shi. Federa: Efficient fine-tuning of language models in federated learning leveraging weight decomposition. arXiv preprint arXiv:2404. 18848, 2024.
Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, and Junbo Zhao. Data contamination calibration for black-box lmss. In Findings of ACL. pp. 10845-10861, 2024.
Liping Yi, Han Yu, Gang Wang. Xiaoguang Liu, and Xiaoxiao Li. pfedlora: model- heeelll-etrooeeeouus personalized federated learning with lora tuning. arXiv preprint arXiv:2310. 13283, 2023.
14

---

Published as a conference paper at ICLR 2026
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In Proc. of ICLR, 2022.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In Proc. of ICASSP, 2024.
Chao Zheng. Liming Wang, Zhen Xu, and Hongjia Li. Optimizing privacy in federated learning with mpe and differential privacy. In Proc. of CACML, 2024.
Yifeng Zheng, Shangqi Lai, Yi Liu, Xingliang Yuan, Xun Yi, and Cong Wang. Aggregation service for federated learning: An efficient, secure, and more resilient realization. IEEE TDSC, 20(2): 988-1001, 2022.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Proc. of NeurlPS, 2019.
Meilu Zhu, Axiu Mao, Jun Liu, and Yixuan Yuan. Deer: Deviation eliminating and noise regulating for privacy-preserving federated low-rank adaptation. IEEE TMI, 4784):1783-1795, 2025.
15

---

Published as a conference paper at ICLR 2026
Appendix
LIST OF CONTENTS
A: Related Work
B: Preliminary
B.1 Relationship between Parameter Importance and Privacy Leakage Risk
B.2 Data Reconstruction via Inversion Attacks
B.3 Federated Efficieet Fine-Tuning
B.4 Matrix Multiplication in LoRA Amplifies DP Noise
B.5 Homomorphic Encryption
B.6 Order-Preserving Encryption
B.7 Singular Value Decomposition
C: HE Key Management and Distribution
D: Additional Technical Details of SHE-LoRA
D.1 The Algorithm of HE Subset Negotiation
D.2 Aggregation of Unencrypted Model Parameters
D.3 Aggregation of Encrypted Model Parameters
D.4 Reparameterization of LoRA
D.5 Proof of the Losslessness of Meaningful Model Updates in SHE-LoRA
D.6 Distribution Shift of Model Parameter Importance Values
D.7 Proof of Asymptotic
Noise
E: Distributability and Scalability
E. Performance on More Clients
E.2 Performance Larger LLMs
E.3 Performance on Stronger Base Models and more Challenging Benchmarks
F: Additional Experimental Results
F.1 Performance Varying Tasks
F.2 Robustness under Varying Non-IID Conditions
F.3 Efficient Estimation of Mutual Information
F.4 Resistance against Membership Inference Attacks
F.5 Impact of Sensitive Parameters on Performance
G: Table of Notations
H: Limitations
I: Broader Impact
J: The Use of Large Language Models (LLMs)
16

---

Published as a conference paper at ICLR 2026
A RELATED WORK
LoRA Tuning in FL. FedIT (Zhang et al., 2024) employed FedAvg to aggregate updates from locally performed LoRA fine-tuning. FeDeRA (Yan et al., 2024) modified the initialization of matrices A and B by the SVD results of pre-trained parameters, which helps mitigate client drift caused by data heterogeneity. SLoRA (Babakniya et al., 2023) employed SVD for initializing matrices A and B, and calculated a mask to reduce the parameters for training and communication overhead. FedRA (Su et al., 2024) partitioned the pre-trained model by layer, enabling clients to fine-tune specific layers on homogeneous models. Inspired by the mixture of experts architecture, HydraLoRA (Tian et al., 2024) proposed learn multiple LoRAs corresponding different knowledge. PiSSA (Meng et al., 2024) proposed to directly fine-tune the principal component of the pre-trained model, facilitating rapid convergence and enhancing overall performance. These LoRA variants primarily foeus on reducing training costs in homogeneous settings, but neglect the heterogeneity of device capabilities and Non-IID data in cross-device federated PEFT scenarios.
Heterogeneous LoRA. FLORA (Wang et al., 2024) contended that the aggregation used in FedIT (Zhang et al., 2024) is flawed and employed stacking aggregate parameters from heterogeneous clients. HeterLoRA (Cho et al., 2024) implemented different ranks on clients and aggregates heterogeneous LoRA modules through zero-padding, which may dilute certain parameters. RBLA (Chen et al., 2024) designed rank-based LoRA aggregation method to prevent model parameter dilution caused by zero-padding. Flex-LoRA (Bai et al., 2024) synthesized complete set of LoRA weights from individual client contributions, and employed SVD for weight reparameterization, thereby fully leveraging heterogeneous client resources. Furthermore, PFedLoRA (Yi et al., 2023) aggregated adapters to facilitate personalized FL, and treated LoRA as mechanism for knowledge transfer, allowing clients to locally train heterogeneous models while maintaining a homogencous adapter. Although these works have made significant contributions to heterogeneous federated PEFT with LoRA, they still suffer from potential privacy leakage risks under inversion attacks.
Privacy Preservation Techniques. To defend against inversion attacks, Yu et al. (2022) employed the DP-SGD optimizer for fine-tuning, providing formal DP guarantees for model parameters. Considering that DP noise may be amplified by LoRA multiplication, FFA-LORA (Sun et al., 2024c)
modified the LoRA training procedure by freezing matrix A after initialization and solely applying DP on matrix B, the cost of fewer tunable model parameters. Similarly, FedSA-LoRA (Guo et al., 2024) proposed to globally train matrix A while reserving matrix B for local training without participating in aggregation. Han & Yan (2023) proposed an adaptive, precision-lossless batch HE method that transforms model parameters into non-negative values to prevent overflow errors. Inspired by model pruning techniques, FedML-HE (Jin et al., 2023) proposed to encrypt only a subset of sensitive model parameters to reduce HE overhead. MaskCrypt (Hu & Li, 2024) proposed to select a consensus mask for SHE to minimize overhead across homogeneous devices. In summary, the application of DP involves trade-off between privacy protection and model convergence, while existing HE methods struggle to balance privacy and efficiency in cross-device federated PEFT with LoRA, particularly under scenarios with NoD-lII data and heterogeneous device capabilities.
B PRELIMINARY
B. RELATIONSHIP BETWEEN PARAMETER IMPORTANCE AND PRIVACY LEAKAGE RISK
On one hand, as indicated in previous SHE methods (Hu & Li, 2024; Jin et al., 2023), the privacy leakage risk in FL mainly comes from the fact that the locally trained model weights contain private data information, and are vulnerable to attacks. The lower gradient loss a parameter results in, the higher privacy leakage risk it may cause under the attacks.
On the other hand, model pruning methods (Frantar et al., 2023; Hassibi et al., 1993; Sun et al., 2024b) have proved that by carefully screening the parameters by importance, many parameters can be removed without hurting performance. Following this rationale, SHE-LoRA lets heterogencous clients adaptively encrypt partial parameters via importance sereening and negotiate global HE subset for secure Federated PEFT without hurting privacy and efficiency.
Specifically, let W and L(W) denote parameters and the loss function, respectively. Given subset of the parameters w E W, W W_ = W w denotes the parameters with w zeroed out. According to (Hassibi et al., 1993), the loss function L(W -w) can be expanded the following Taylor series:
17

---

Published as a conference paper at ICLR 2026
C(W <(W w) L(W) + +--22// 2 Hw + O(Iwwl)),
(7)
where 9² and H are the first-order (gradient) and second-order (Hessian) partial derivatives, respec- tively. O(1) [[[[ (l ") is the higher-order infinitesimal of w. The sensitivity of w (denoted as st(w) (w) can
be denoted as:
?(w)= 1<(W w) [(W) w 2 Hw + O(l w Hw.
(8)
Considering that when W converges to the local minimum after training, g is O, while ( [ can be neglected, thus (2(w) only depends on H and w. Finally, the sensitivity of the g-th parameter
)))
2 according to OBS (Hassibi et al., 1993), where [H~ Jag the diagonal element () of the inverse Hessian matrix, H
calculated as S2( ((w)))
w
Inspired by H = X from SparseGPT (Frantar & Alistarh, 2023), Wanda (Sun et al., 2024b) modified weight importance assessment avoid the high computation cost of H and H-1 (Eq. (2)). Based on Wanda, SHE-LoRA assesses channel-wise weight importance as in Section 3.1.1.
Besides, Fig. 10 shows that the mutual information (privacy leakage) of Max decreases much sharper than the others along with the encryption of parameters, which intuitively reflects strong correlation between weight importance and privacy leakage risk.
B.2 DATA RECONSTRUCTION VIA INVERSION ATTACKS
Inversion attacks (Petrov et al., 2024; Balunovic et al., 2022; Zhu et al., 2019; Jeon et al., 2021) aim to reconstruct private training data or model parameters, such as pixel values images or sensitive information in text, by reversing the clients' updates uploaded during federated fine-tuning, such as gradients, parameter updates or prediction results. Specifically, for a model fw fw(x) = W x, a client trains it with its local data (z, y), and calculates the gradient g as the derivative of the loss function C:
ac OC 0fw dc g aw öfw aw afw -2;
and uploads it to the server. The uploaded gradient contains a linear combination of the original data. An attacker can analyze the principal components of the model update to search for the client's s data distribution space and then reconstruct the data (Petrov et al., 2024).
B.3 FEDERATED SHAAMETER-EFFICEEE
TUNING
technology designed for the efficient fine- tuning of large models within a distributed learning framework that prioritizes data privacy. For instance, LoRA (Hu al., 2022) a well-established method for PEFT. Suppose the weight matrix of the global pre-trained model is denoted as W E Rmxx LoRA introduces trainable parameters, B E mx and A E x Each client freezes the original weights W and learns only the trainable low-rank parameters B and A on its local data. Then, the updates (e.g., weights or gradients) from clients are aggregated by the server via
Federated Parameter-Efficiees Fine-Tuning (PEFT)
B.A.
(10)
where t; denotes the weight coefficient of Client i, proportional to its local data size. Federated PEFT uses Eq. (10) to guarantee heterogeneous aggregation of LoRA parameters. SVD is used reparameterize the aggregation result into heterogeneous LoRA parameters, which reduces the communication overhead of AW. The new LoRA parameters are then sent back to each device for continued local training, and the process is iterated repeatedly progressively optimize the model. is worth noting that the model parameters are transmitted in plaintext throughout this process, making them visible the server.
B.4 MATRIX MULTIPLICATION IN LORA AMPLIFIES DP NOISE
Differential privacy (DP) (Yu et al., 2022: Zhu al., 2025) is common privacy defense mechanism that adds specific noise to gradient or parameter updates, making the relationship between gradients and data non-linear, which misleads the attacker's optimization direction. Given that the LoRA
18

---

Published as a conference paper at ICLR 2026
fine-tuning parameter is AW- BA, and AW plays role during model inference, if noise satisfying the privacy budget is added to the parameters, then:
AW = (B + <B + eΛ) BA + BA + AeB + 6AEB
(11)
Thus, except for BA, all the terms in Eq. (1I) are noises that will be aggregated into the global model parameters, which may severely affect the model's performance and convergence direction.
B.5 HOMOMORPHIC ENCRYPTION
Homomorphic encryption (Rivest et al., 1978; Gentry, 2009) is cryptographic primitive that allows computations to be performed on encrypted data without revealing the underlying plaintext. is exceptionally well-suited for FL., as it enables the computation of aggregation in server without exposing clients' updates. The Cheon- 2017) supports approximate numerical computations and floating-point arithmetic. The CKKS offers relatively high computational efficiency and supports vectorized operations, making it highly suitable for LoRA tuning FL. can be used encrypt the parameters or gradients of local models, allowing the server to perform model aggregation without decrypting any data, thereby protecting user privacy, In the implementation of SHE-LORA, we employ the CKKS provided by the TenSEAL library. which supports homomorphic computations on both vectors and tensors, thereby enabling encrypted computation operations over complex model parameters.
(CKKS) encryption scheme (Cheon et al.,
B.6 ORDER-PRESERVING ENCRYPTION
Order-Preserving Encryption (OPE) (Agrawal et al., 2004; Boldyreva et al., 2009) a cryptographic technique that preserves the numerical order of plaintexts. If two plaintexts, and b. satisfy the condition b, then their encrypted ciphertexts will also satisfy Enc(a) < Enc(b). This enables comparisons, sorting, range queries to be performed on ciphertexts without decryption. In principle, OPE maps plaintexts to an interval within the ciphertext space, while ensuring that the mapping function monotonically increasing. Although OPE does not provide semantic security in the traditional sense (i.e., it is possible to infer the approximate range of the plaintext from the ciphertext), it is highly useful in applications that require sorting or range operations on encrypted data, such as encrypted databases or cloud storage queries. In SHE-LoRA, we employ OPE to hide clients' HE subset, which prevents the server from snooping on the positions of important model parameters and conducting targeted attacks.
B.7 SINGULAR VALUE DECOMPOSITION
Singular value decomposition (SVD) (Klema & Laub, 1980) is a mathematical method that decom- poses any real or complex matrix into the product of three standard matrices. It is applicable to matrices of arbitrary shapes, and hence suitable for tasks such as dimensionality reduction, data compression, and recommendation systems. For a real matrix W E Rm SVD decomposes it as follows: W = UEV where U E R xm is an orthogonal matrix, Σ Rmxx is a diagonal matrix with non-negative real values on its diagonal, known as singular values, arranged descending order, and V E x is orthogonal matrix well. To match heterogeneous LoRAs from clients, many methods (Yan et al., 2024; Babakniya et al., 2023) have employed SVD decomposition, breaking down the aggregated matrix into B = UVE and A = VEVT, which enables the projection of the space while preserving the essential features.
original parameters into
C HE KEY MANAGEMENT AND DISTRIBUTION
In the configuration outlined herein, the system necessitates an honest- bunest-but-curiuus server to execute the aggregation of models. Then, trusted third party is required to oversee the distribution of keys for both OPE and HE, with the assumption that it will not collude with the server. However, when the server colludes with compromised clients, cryptographic techniques such as threshold HE, Multi-Key HE, and proxy re-encryption can be employed to achieve distributed secure computation. Threshold Homomorphic Encryption (Aloufi et al., 2021) is cryptographic scheme that amalgamates the threshold cryptography and HE. It not only supports computations on encrypted data (homomor- phism), but also enables joint decryption by multiple participants (thresholding), thereby enhancing
19

---

Published as a conference paper at ICLR 2026
both security and fault tolerance while preserving privacy. Decryption only be accomplished with the collective participation a predetermined number of participants, known as the threshold. Proxy Re-Encryption (Blaze al., 1998; Ateniese et al., 2006) permits semi-trusted third party (the proxy) to transform ciphertext encrypted for one party into eiphertext decryptable by another, all without accessing the original plaintext either party's private keys. Multi-Key Homomorphic Encryption (Aloufi et al., 2021) enables homomorphic operations to be performed on user data encrypted under different keys, obviating the need for key sharing. The resultant ciphertext requires collaborative partial decryption by all participants, each utilizing their respective private key, to yield the final plaintext outcome. This mechanism ensures that joint computations can be performed in multi-party data collaborations while protecting the data privacy of each participant.
In SHE-LORA, we retain Multi-Key HE countermeasure against potential collusion between the server and clients, while leaving its practical integration and optimization to future endeavors.
D ADDITIONAL TECHNICAL DETAILS OF SHE-LORA
D.1 THE ALGORITHM HE SUBSET NEGOTIATION
D.1.1 OBJECTIVES OF THE NEGOTIATION
The negotiation aims select subset of columns Res U G, with cardinality |Res| (N) ki where k; denotes the number of sensitive columns that client can afford to en- crypt and {N} denotes the set of all clients, thereby achieving a principled trade-off between privacy and HE overhead. However, the notions of "privacy" and "HE overhead" inherently ambiguous without concrete metrics. To make the trade-off operational, we formalize two explicit objectives with moveCCoe and max-Risk.
We first define the Coverage, of client i as the fraction of its sensitive columns that are selected for encryption (i.e., included in Res), and hence protected from exposure: Coverage, |RenG. where
[G,]
G₁ is the set of columns that client deems sensitive, and Res is the final selected subset.
To extend HE-bbasd privacy protection to as many clients as possible, we adopt a max-min criterion and define the overall coverage as:
Sonn-overage- min Coverage, EENN}
(12)
Maximizing min-C min-Coverage ensures that the coverage of every client is at least this value, thereby guaranteeing the worst-case coverage of sensitive parameter columns: even the least-covered client receives quantifiable level of protection.
Eserrve S
Second, to quantify the privacy leakage risk of each client, define Risk;
Eseo, S, which measures the fraction of client i's total sensitivity that remains unencrypted. We further define
max-Risk- max Risk; (E(N)
(13)
to capture the worst-case privacy leakage risk across all clients, which we aim to minimize. Lower residual risk implies stronger privacy preservation. Minimizing max- Risk ensures that the privacy leakage risk of every client at most this value, thereby providing a worst-case privacy guarantee: the most exposed client suffers from more than quantifiable level of privacy leakage risk.
The negotiation thus seeks a balanced Res that jointly optimizes both Eq. (12) and Eq. (13). To formalize this goal, we define a composite objective score as Eq. (14):
|es n G
Ejec S4
score(Res) min iE(N)
(14)
|G&1
VE(N) Σjec, S
minn-Coverage
Eq. (14) balances the minimal client coverage of sensitive parameters and the maximal privacy leakage risk of unenerypted parameters. These two objectives often contradict under limited encryption budget: satisfying client- speccffc privacy leakage risk may sacrifice overall coverage of clients' sensitive parameters, and vice versa.
20

---

Published as a conference paper at ICLR 2026
Algorithm 1: HE subset negotiation
Input:
clients = fri, kis (G,, S.) a. b, c are three hyper-parameters of selection ratio.
Output: Res: the column index of global HE subset.
Res, selected_nun + 0,:;
Common + sorts columns in U, G, from most to least frequently deemed as sensitive;
3 Sensitivity + sorts columns in U, G; from highest to lowest sensitivity;
I' r- [[k, counta)) clusters clients by budget k. and sorts them in ascending order of k;; for each budger le in I do
t update current budget by k selected_ num; Clients + collect all unique columns from Uni Gi, and sort them by min Sy;
county then
P < select top À columns from G--Res of the unique client;
10
Res + (R.(Rss) J:
selected_mun k;
12
else
13
b, c < coefficients optimized via Bayesian optimization under a + + = 1, balancing min- Coverae and max-Risk as detailed in Algorithm 2;
P select top [a)| columns from Clients-Res;
15
< select top bb columns from Common-P--Re
16
S + select top
aAj [6A] columas from SensitivityP-C--ee
17
Res <- result of & columns is (S, C, P, [Re))];
selected_nee num k;
19
return Res
D.1.2 PROCEDURE OF THE NEGOTIATION
The workflow of HE subset negotiation is summarized as Algorithm 1. Note that the server keeps the rank r; and encryption budget % of all clients. The number of encrypted columns of Client i is denoted as K; = 7 n, where n is the number of columns in the parameter matrix. As described in Section 3.1.2, the server receives set of tuples (G,, S;) from clients input, where G; is Client i's set of columns that needs HE, and S, is their sensitivities.
At Lines 1-3, the server first initiates the negotiation result Res and the number of columns that have been selected selected _nmm Then, the server maintains two shared lists: the Common list, which sorts all columns in U; G, from most least frequently deemed as sensitive, and the Sensitivity list, which sorts all columns in U, G2 from highest to lowest sensitivity. The columns ranked higher in Common are more frequently deemed as sensitive by the clients, and selecting them improves min- Cove-ooee The columns ranked higher in Sensitivity have greater global sensitivity, and selecting them reduces max-Rist Risk. At Line the server clusters the clients by their budget k. and sorts them in the ascending order of k. At Lines 5-18, the server repeats the process several times, which depends on the number of unique budgets. For each process, the number of column positions to be negotiated, denoted X, is calculated by the difference between current budget k and the number of columns that have been determined. A Clients list ranks unique columns from budget-k clients by their minimum sensitivity, aiming to encrypt columns that are personally deemed as sensitive. If the current budget corresponds to a single client (i.e., count, = 1), the strategy greedily selects from that client's 's G; for optimal privacy protection. Otherwise, the server iteratively selects [aA] columns from Clients, [6A] columns from Common, and x - [aA] - [A] columns from Sensitivity without duplicate selection form the global HE subset, where b c 1 and L-J is the floor function. This hybrid selection jointly optimizes worst-case coverage and privacy risk while preserving a degree of client personalization to accommodate heterogeneous environments.
num.
Clearly, the success of this objective relies heavily on the choice of the coefficients b, and As presented in Algorithm 2, we employ the Bayesian optimization (Swersky et al., 2013) to determine their optimal values via searching different combinations of column selection from three complementary perspectives: client-specific (Clients), commonly shared (Common), and sensitivity- driven (Sensitivity). Specifically, Algorithm 2 begins by defining the feasible search space for the parameters (a, b, c) at Line 1, subject to the constraint a + b + c = 1. At Line 2, a Bayesian optimization model (i.e., Gaussian process) is initialized over this search space. At Lines 3-16, the model performs Nom (e.g., 50) iterations of optimization: at each iteration t, the current parameter triple (at, be, ce) is used to construct the negotiation result Res, at Line 5, following the same column
21

---

Published as a conference paper at ICLR 2026
Algorithm 2: Bayesian optimization for the selection of a, b and
Input: clients = fri, kis (G,,S:)) S.) Current X, Res, Clients, Common, Sensitivity.
Output: Optimal coefficients (a ,b", with a² + c² =1
X +- Define search space with {(a,b) b) E 0. 1]2 a + b+bbe S 1};
Initialize a Bayesian optimization model M over X::
for 11= == to Nap do
// Bayesian optimizer selects bi) for evaluation
(a,, by) SELECT(M));
- 1- az b2i
Pt - select top [ columns from Clients-Res;
C select top [ columns from Common-Res-Pi:
St select top À [ [6,A] columns from Sensitiviy-Res-- Pe--P
Res: the union of (Sa, C., P.):
10
for each client
Coverage, Res; n Gil.
11
G+
Risk; Lje St.
Lsea; 54
13
min- Coverage <- (N) Coverage;;
14
max-Risk + ( Risk;;
score, min- Coverag-- max-Risk;
Update Bayesian optimizer with (ar,be, score:)
16
M + UPDATE(M, (a,, be), score;);
(a", (a",b") +- arg max, score;; 18 a- -b: b": 19 return (a" (e",b",cc)
selection procedure as Algorithm 1. Then, Lines 10-12, the client-specific coverage and privacy risk are evaluated for every client. The overall score is computed Line 15 using Eq. 14, and in Line 16, this score fed back to update the Bayesian optimization model, guiding the next parameter selection. Finally, Lines 17-19 return the best best-performing coefficients (a" c²) as the outcome of the negotiation.
D.2 AGGREGATION OF UNENCRYPTED MODEL PARAMETERS
Algorithm 3 illustrates how the server aggregates unencrypted model parameters. The input of the algorithm is the set of unencrypted weight updates AW.Plain from all N clients. At Line 1, the columns of the aggregation result initialized by K = n min(k)..... ky). where is the number of columns in the frozen pre- trainen parameter matrix, and & is the number of encrypted columns of Client i. At Lines 2-3, the server initializes the aggregation result to 0 E mm K and sets a counter that records the respective contributions of the clients during the aggregation of each column. At Lines 4-7, the server incorporates Client i's parameters AW:lain into the aggregation results, and updates the counter to record the number of clients contributing to each column. At Lines 8-11, 11, the server wvight-averages the results based on the counters and returns the final aggregated result AW plain E m
D.3 AGGREGATION OF ENCRYPTED MODEL PARAMETERS
Algorithm 4 illustrates how the server aggregates encrypted model parameters. The algorithm takes the set ciphertexts AWrippeer E m as the input. At Line 1, the columns the aggregation result are initialized by K max (k1, . where k;; is the column number of encrypted model parameters of client At Lines 2-3, the server initializes the aggregation result to 0 K and sets counter to record the respective contributions of the clients. At Lines 4-6, the server incorporates encrypted model parameters eWiippeer into the aggregation results and updates the counter record the number of clients contributing to each column. Finally, at Lines 7-9, the server weight-averages the encrypted model parameters based the counters, and returns the final aggregated encrypted model parameters, denoted as AW cipher E pm K Although the columns of encrypted model parameters extends to K*, each client can receive a set of encrypted blocks matching its encryption budget.
22

---

Published as a conference paper at ICLR 2026
Algorithm 3: Aggregation of unencrypted model parameters
Input: (AWTiia), Set of N matrices, each AWTliin has shape (m, kz). Output: AW paini Aggregated matrix with shape (m, K).
K < min(k1.......s))) kx);
AW plain t- 0mxk:
Counts <- 0
for each client 1 .V do
- get column count of AWP
AW AWptnial: ; ca] < AW plain p a] + AW?"
Counts G] Counts{: di) 1;
for = 1 K
if Counts b] > 0 then
10
AW AW plann [
AW plan j. /oountsss
return AW plamn
Algorithm 4: Aggregation of encrypted model parameters
Input: (AWatder) Ji=1: Sets of N ciphertexts from clients, each AWpiiper has shape (m, kt). Output: AW cipher: Aggregated matrix with shape (m, K*).
K" (x.(kk,,,....
AW egper 0mxk*
Counts 01IK=
for client = .N
AW cipher k; AW cipher ;, k; :] + A Weepper
Counts hi 4- Counts k;
for j = to K do
if Counts b] > 0 then
AW [p + AW ciphe AWaphe
[: /ounnssss
10 return AW cipher
D.4
REPARAMETERIZATION OF LORA
The updated full- pallmearrettr for each client, termed as A AW, can be formulated as two parts, the plaintext update A W plain Ap and the ciphertext update A W cipher E Br as shown in Eq. (15). In order to reparameterize the two parts of the model parameters into the parameter matrices and Á of LoRA, we first apply SVD and zero-padding to the ciphertext update to generate two low-rank matrices (B. Rmxrr Ag Rr**), which ensures their dimension aligns with B, The final LoRA parameter matrices (B, A) are calculated as follows:
AW = A W plain + AW cipher
(15)
SVD
A, P + B.A.
(UIVE, V (U.VER)VE
(rtr)xn
[UIVE U2VE (x(+rr)
VΣ, V Σ V
SVD
(U, V.)
(U,E, E.VE (VE.V. )::
BÅ
D.5 PROOF OF THE LOSSLESSNESS OF MEANINGFUL. MODEL UPDATES IN SHE-LORA
The losslessness of meaningful model updates in the aggregation of SHE-LoRA is supported by the following theorem.
23

---

Published as a conference paper at ICLR 2026
Theorem Whether column of the parameter matrix is enerypted not, will always be inte- grated into the aggregated model, and hence results in no loss of meaningful model updates.
Proof 1 Suppose that for clients 1 to N, their encryplion bulgets are T/s (71 T22-1 .2. S (NN,, and the hidden size of the model is n (i.e. number of columns in the parameter matrix). Then the numbers of encrypted columns of the clients are k1 = n X 71 S l22 = nt X ... S kN = 11 x 7N--
From Section 3.3, all k encrypted columns integrated in AW cipher- while plaintext columns SVD
integrated in AW plain- According Section 3.4, AW plain B, and AW cipher respectively. Following Eq. (4), all meaningful updates are integrated in LoRA matrices as B [BP Bo] and A [Ap Ar] Finally, the weight update for each client can be calculated as AW AW plain +
SVD B.
AW cipher = [B, Be] [A] A. = BA. Thus, whether column is encrypted (in A AW cipher) or not (in A W plain), h, it will always be integrated into the aggregated model (AW- = BA).
D.6 DISTRIBUTION SHIFT OF MODEL PARAMETER IMPORTANCE VALUES
To determine whether the distribution of model parameter importance values will shift during training, we conduct 50 rounds of FL training on the Iatural-Instructionn (Wang et al., 2022) dataset under Non-IID conditions with the Dirichlet distribution parameter = 0.3. Fig. 11 illustrates the variation of the distribution of channel-wis importance values along with the progress of FL training. We can see that the specific importance values do change slightly, but their relative ranking remains almost unchanged as compared to Fig. 4. Considering that the parameter importance in SHE-LORA assessed via channel-wise summation of sensitivity values, the slight change of model parameter importance distribution has minor impact on performance.
Moreover, considering that extreme cases (e.g., dynamic data change) may occur, especially under Non-IID settings, the negotiation of HE subsets can executed periodically depending on the clients' tolerance to the change of model parameter importance distribution. Specifically, the theoretical costs of negotiation and training per layer on a client are listed Table 3.
Table 3: Theoretical costs of negotiation and training per layer.
Communication
Computation
Negotiation x N
Bytes hidden size
ratio N
Forward N
Training x
Bytes rank hidden size
Backpropagation
With precision=bf16, encryption ratio=1% and r=16, FL training generally takes N < 50 rounds for convergence. Even if the negotiation is executed per round, the negotiation communication overhead of 50 rounds is 4 Bytesx6663 hidden size (I (Llama-30B)x 1% 1% 50=13 13 KB, which is much smaller than the overall training communication overhead (2 Bytes 16x 666666 hidden size (Llama- 30B)=213 KB). However, although the computation cost of a "Forward" is much lower than that of a "Backpropagation" the computation cost of N rounds of "Forward" will gradually increase along with the training progress. Therefore, the clients can choose the negotiation period according to their expected balance between model parameter importance update timeliness and computation cost.
D.7 PROOF OF ASYMPTOTIC GAUSSIAN-DISTRIBUTER NOISE
Let G [g), .g1,-..:9h 9n] E R be the gradient matrix from LoRA fine-tuning, with columns ge R", and let P be the permutation matrix corresponding to a uniform random permutation T(-). We define the noise matrix ② G(P I), that the noise added each gradient element is calculated as oikk (i.m(k) Ii.k- For any fixed linear query matrix Q E "x" the query output is:
0q(0) = (Q, = Σ Σ
- - gi.)).
(16)
24

---

Published as a conference paper at ICLR 2026
Rank:16 FL Raund:1
Rank:16 FL Round:3
Rank:16 FL Round:4
Rank:16 FL Round:5
60.3
--24
02
02
0.1
Rank:16 Roundi6
Rank:16 FL Round:7
Bank:16 FL Round:8
Rank:16 FL Round:9
0.44
8.
0.2
Rank:16 FL Round:10
Rank:16 FL Round:11
Rank:16 FL Round:12
Rank:16 FL Round:13
-03
0.3
0.3
0.1
61
Rank:16 FL Round:14
Rank:16 FL Round:15
Rank:16 Round:16
Rank:16 FL Round:17
0.4
0.
8.0.
0.
62
022
0.1
Rank:16 FL Round:18
Rank:16 FL Round:19
Rank:16 Round:20
Rank:16 FL Round:21
0.3
6.
01
Rank::6 FL Round:22
Rank:16 Round:23
Rank:16 Round:24
Rank:16 FL Round:25
0.
02
02
Rank:16 FL Round:26
Rank:16 Round:27
Rank:16 Round:28
Rank:16 FL Round:29
0.4
0.4
02
62
42
02
Rank:16 FL Round:30
Rank:16 Round:31
Rank:16 Round:32
Rank:16 FL Round:33
8..4
0.4
0.4
82
62
62
Rank:16 Round:34
Rank:16 Round:35
Rank:16 Round:36
Rank:16 Round:37
0.4
6.4
0.2
42
62
02
Rani::6 Round:38
Rank:16 Round:39
Rank:16 Round:40
Rank:16 Round:41
0.2
022
02
02
01
Rank:16 Round:42
Rank:16 Round:43
Rank:16 Round:11
Round:45
0.2
82
02
0.1
011
01
Rank:16 Round:46
Rank:16 Round:47
Rank:16 Round:48
Round:49
02
0.
62
02
01
41
41
A1
Figure 11: Distribution of model parameter importance values across FL rounds.
25

---

Published as a conference paper at ICLR 2026
The randomness comes solely from the permuted sum Σi, (kk)-t,x()). which is combinatorial statistic of the form studied in (Hoeffding. 1951). Therefore, by combinatorial central limited theorem (Hoeffding, 1951) and Berry-Esseen bound (Bolthausen, 1984), we have
(+) Var Oq (
dist (N 1), sup ZER
P
Oq /ar [aa
((x)
=0
1 vn
(17)
The first statement means that the response to any fixed linear query asymptotically Gaussian, and the second quantifies the approximation error as O(1 nn). Moreover, by the classical variance formula for permuted linear statistics (Hoeffding, 1951; Hájek, 1961), we have
1 -(Qu--Q.) Σ, i=1 n1 n-1
E(OAn (os 3.)2 -B)P
Var[Oq
(18)
where 0: = = - AiA and Q2 Σ Qu,--
For fixed G and Q, we define s² as the variance of (e). Since this asymptotic normality holds for every fixed query matrix Q. the Cramér-Wold device (Cramér & Wold, 1936) implies that the noise matrix θ, viewed as random vector in m converges in distribution to zero-mean multivariate Gaussian with the same covariance structure as O Consequently, @ behaves like Gaussian noise for all linear queries.
E DISTRIBUTABILITY AND SCALABILITY
Our system implementation is predicated upon Flower, an open-source FL framework developed by team at the University of Oxford. Flower is designed streamline the construction of FL systems while affording a high degree of flexibility and scalability. It supports variety of mainstream machine learning frameworks, such PyTorch, TensorFlow, and Hugging Face Transformers, rendering suitable for researchers and engineers addressing FL requirements across diverse scenarios. Flower allows
extensively configure the framework according to their specific needs, thereby accommodating various FL scenarios while offering substantial support for Al research. Based on Flower, our SHE-LoRA supports parallelized simulation and multi-machine deployment, capable of satisfying the distributed and scalable requirements inherent in real-world applications.
E.1 PERFORMANCE ON MORE CLIENTS
We repeat the experiment of bert-large model with 100, 300, 500, 1000, 2000 clients the IMDB datasets, which takes 1.58, 4.14, 6.91, 13.8 and 25.3 hours to complete 10 rounds of FL training. Each client encrypts 0.5% of OpenLLaMA-3B with the same rank (16). The means and standard deviations of HE time and communication overhead are listed in Table 4.
Table 4: HE time and communication overhead on varying number of clients.
# of Clients
HE Time
Communication Overhead
100
53.34=0.55
93.39-t0.IMB
300
53.57=0.5s
93.39=+0.1
500
53.49-t0.6s
93.39-+0.1MP
1000
54.23
93.39:10.1
2000
14.19
93.39=10.IMB
We find that although the convergence FL training does slow down along with the increase of clients, thanks to SHE-LoRA's global control over HE subset, the clients' HE time and communication overhead do not significantly inflate even in extreme heterogeneity with >1000 clients. This means that SHE-LORA will not delay FI. training and scales well with increased number of clients.
26

---

Published as a conference paper at ICLR 2026
E.2 PERFORMANCE ON LARGER LLMS
We deploy SHE-LoRA on larger LLMs including OpenLlama-3B, Llama-3-8B, Llama-30B and Llama-3.1-7000 and analyze its scalability in comparison with the DP baseline. Specifically, as Section 4.4 has confirmed that SHE-LoRA is secure against the DAGER attack as long as more than 0.12% of the parameters are encrypted. we let each client encrypt 0.125% of the parameters in the scalability experiments with rank r=16. In the DP baseline, we let each client add DP noise to parameters with (c, σ) (10, 10- which is the same as in the DAGER (Petrov et al., 2024) experiments. The HE key size for OpenI_lama-38 and Llama-3-88 is set to 8192. However, the HE key size for Llama-30B and Llama-70B is set to 16384 as the HE key size of 8192 cannot hold a single column (minimum encryption unit in SHE-LoRA) of LLMs at this scale. Then, we measure the encryption time, time cost with DP and ciphertext size per client under varying model scales. The mean and standard deviation of the measured results shown in Table 5.
Table 5: The costs under varying model scales.
JpenLllama33B
Llama-3-8B
Llama-30B
Llama-3.1-708
of Layers Hidden Size
26
32
60
80
3200
4096
6656
8192
Encryption Budget
0.125%
0.125%
0.125%
0.125%
Encrypted Parameters
1,664
2,624
8,040
13,120
Encryption Time (s)
2.67++0.32
4.46-t:0.33
148.70+H55.41
242.32-4-8.72
HE Key
8192
8192
16384
16384
Time Cost with DP (s)
0.0548=00001
0.0878+00005
0.2693+
0.4575+00.61
Ciphertext Size (MB)
23.34:10.00
35.91=0.01
289.37=10001
385.84:10033
When HE key size is 8192, for OpenLlama-3B. encryption time per parameler=2267711666--0000
s, ciphertext size per parameter=23.34 34440/666001144 MB; for Llama-3-8B, encryption time per
paramccee==446222444.000001 ciphertext size per parameter=33311/724440000333 MB.
When HE key size 16384, for Llama- encryption time per paraameter=1488.700000-0-111
ciphertext size per
MB; for Llama-3.1-70B, encryption time per MB.
parameter=222.32/113100-00000 20=0.0184 s, ciphertext size per paramete==385.84/111200000
These observations demonstrate that when LLMs are encrypted with the same level of HE key size, the encryption time and ciphertext size scale almost linearly with the increase of LLM scale. Although the time cost with DP is much lower than that of SHE-LoRA, DP may significantly degrade model accuracy (Sun et al., 2024c), and vulnerable against inversion attacks under low-noise-level settings (as shown in Table 10 of (Petrov et al., 2024) and Table 2))).
E.3 PERFORMANCE ON STRONGER BASE MODELS AND MORE CHALLENGING BENCHMARKS
and Vanilla LoRA on the PILE dataset and evaluate the performance of the fine-tuned models on six benchmarks: MMLU-Pro GPQA MuSR MATH IFEval 10 and BBH 11 The collected results of SHE-LORA, Vanilla LoRA and the base model without fine-tuning are shown in Table 6.
We conduct the fine-tuning of Qwen3-4B-B -wwn3----Insttuet-22 Ins-Inuttttt-5000 and Llama-3.2-33 4 with
The results demonstrate that SHE-LoRA preserves the original LoRA performance while providing privacy via selective HE. This benefit generalizes across stronger base model families and diverse tasks, provided that federated LoRA is used for PEFT.
"https://huggingfaac" wwt////taall1lammm11aa---3.1..1
"https://github
27

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison of Qwen-3 and Llama-3.2 across more challenging benchmarks.
Method
Method
MMLU-Pro
GPQA
MuSR
MATH
IFEval
BBH
SHE-LoRA
47.36
43.94
30.71
78.86
67.34
63.26
Qwen3-4B
Vanilla LoRA
48.54
42.57
32.31
77.63
68.06
64.58
Base
65,36
45,00
61.67
84.00
90.17
85,93
SHE-LoRA
14.86
13.64
17.86
21.74
27.24
9.26
Llama-3.2-3B
Vanilla LoRA
14.62
13.87
18.05
20,96
27.37
9.38
Base
14.29
11.11
30.95
16.68
27.99
10.85
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 PERFORMANCE ON VARYING TASKS
We employ FedIT (Zhang et al., 2024) and FedSA (Guo et al., 2024) baseline methods under homogeneous settings (rank r=8). FedlT averages LoRA weights across clients, limiting the rank according to the capability of the weakest device. FedSA trains matrix B locally while aggregating matrix A globally, leveraging FL to enhance the representation capacity of LoRA. Moreover, we employ FLoRA (Wang al., 2024), HeterLoRA (Cho al., 2024) and Flex-LoRA (Bai al., 2024) as baselines under heterogeneous settings. FLoRA utilizes stacking to reduce full-weight computation and achieve precise averaging across heterogeneous LoRA updates, but the cost of expanded parameter space. HeterL zero-pads all LoRA matrices to the global maximum rank, applies weight-averaged aggregation similar FedAvg, and subsequently truncates the aggregated weights to align with the local rank of each client. However, zero- paoo-ppddnngggg introduces additional dilution in the aggregated parameters, which turn leads to degraded model performance. Flex-LoRA reconstructs the full parameter matrix for each client by computing B A and performs aggregation. Subsequently, the aggregated matrix is decomposed using SVD and truncated according to the client's LoRA rank, producing low-rank parameter matrix.
F.1.1 RESULTS ON NLP TASKS
According the results in Table 7, FedIT and FedSA perform the worst on the MMLU Benchmark, obtaining scores of 21.2 and 20.1, respectively. These results indicate the limitations of traditional homogeneous approaches in heterogeneous LoRA settings, where the inability to effectively utilize client-specific information hinders overall performance. While HeterLoRA integrates parameters from heterogencous devices to improve performance, its reliance on zero-patee prod-padiig leads to parameter dilution, resulting in inferior performance compared to FleX- SHE-LoRA achieves the highest scores on STEM, Social Sciences(SS) and the overall Average, and matches Flex-LoRA's performance on Humanities. Both methods outperform all other baselines by significant margin. These results indicate that SHE-LoRA better preserves informative updates in heterogeneous generative tasks, leading improved generalization and performance.
Natural Language Generation:
Table 7: Performance on the MMLU benchmark.
Method
STEM
SS
Humanities
Average
FedIT (Zhang et al., 2024)
21.5
21.3
20.4
21.2
FedSA (Guo et al., 2024)
21.8
21.4
19.7
20.1
HeterLoRA (Cho et al., 2024)
24.7
25.4
25.8
Flex-LoRA (Bai et al., 2024)
26.2
27.9
26.6
27.4
SHE-LoRA
28.1
29.2
26.5
28.2
Natural Language Understanding: Similarly, we reviewed on the six datasets of GLUE Benchmark in Table 8, and the performances of FedIT and FedSA reaffirmed the limitations of traditional aggregation methods in heterogeneous scenarios. Flex-LoRA and SHE-LoRA, on the other hand, outperform the other methods, demonstrating that SHE-LoRA can more effectively update model parameters in heterogeneous enviromments while achieving performance comparable to non non-private methods. Unsurprisingly. HeterLoRA achieves better performance than homogeneous baselines. However, lags behind Flex-LoRA and SHE-LoRA, primarily due to the performance degradation caused by parameter dilution.
28

---

Published as a conference paper at ICLR 2026
Table Performance on the GLUE benchmark.
Method
SST2
MRPC
QQP
RTE
WNLI
QNLI
FedIT (Zhang al., 2024)
47.41
31.62
64.71
43.07
46.34
48.87
FedSA (Guo al., 2024)
48.23
33.71
66.32
43.56
48.27
48.26
HeterLoRA (Cho et al., 2024)
55.73
68.38
72.17
44.72
48.86
49.14
LoRA (Bai ct al., 2024)
52.29
74.81
75.31
46.93
49.66
49,51
SHE-LoRA
57.11
70.88
72.52
50.18
57.75
59.63
SHE-LoRA demonstrates strong performance across both benchmarks, achieving SOTA results in heterogeneous settings while maintaining optimal performance despite the integration of privacy- preserving mechanisms.
Table 9; Performance comparison on 5 vision tasks.
Datasets
Method
MNIST
DTD
EuroSAT
GTSRB
SVHN
AVG
Clip- Base- Maae-Patch-aoo
==8
FedIT (Zhang al., 2024)
93.38
68.74
93.17
83.62
90.43
85.87
FedSA (Guo al., 2024)
93.13
67.51
94.23
85.12
88.49
85.69
HeterLoRA (Cho et al., 2024)
95.37
68.83
96.22
87.18
91.55
87.83
Flex- LoRA (Bai ct al., 2024)
99.28
70.32
98.48
95.74
95.37
91.84
SHE-LoRA
99.33
69.97
98.35
95.88
95.13
91.73
Clip- Vir--Vi--aa Rase-Patch-Iee 16
r= 16
FedIT (Zhang et al., 2024)
95.36
68.85
94.56
85.37
91.58
87.14
FedSA (Guo et al., 2024)
94.62
67.92
95.18
87.23
90.67
87.12
HeterLoRA (Cho et al., 2024)
94.56
68.21
96.77
89.62
92.28
88.29
Flex-LoRA (Bai et al., 2024)
99.30
70.05
98.29
95.45
95.15
91.65
SHE-LoRA
99.25
70.85
98.22
95.35
96.03
91.94
F.1.2 RESULTS ON VISION TASKS
We apply CLIP (Radford et al., 2021) the basic pre-trained model for vision tasks, a multimodal model that mixes visual model and language model. Specifically, we load the Vit--ase--atchh model from huggingface" 12 and fine-tune its visual model, and conduct experiments on five visual classification tasks, which are MNIST (LeCun et al., 2002). DTD (Cimpoi et al., 2014), AT (Helber et al., 2019), GTSRB (Stallkamp al., 2012). SVHN (Netzer al., 2011). We conduct FL. training for 10 rounds on each task, and set that each client has the same LoRA rank.
The results are shown in Table 9. The highest accuracy (%) for each task highlighted in blod. At rank 8, SHE-LoRA achieves a comparable average accuracy (91.73%) to that of Flex-LoRA (91884%), while outperforming FedIT, FedSA and HeterLoRA. At the rank of r 16, SHE-LORA can even achieve the best average accuracy (91.94%). The results indicate that the privacy protection mechanism of SHE-LoRA will not lead to significant performance degradation.
F.2 ROBUSTNESS UNDER VARYING NON-IID CONDITIONS
The results of SHE-LoRA in Tables 7 and are collected under the Dirichlet distribution with parameter 0=0.3, which confirm that SHE-LoRA achieves comparable performance SOTA non-private Federated PEFT method (Flex-LoRA) on various benchmarks under Non-IID conditions.
To further validate the robustness of SHE-LORA under varying Non-IID conditions, conduct more experiments on the natural-instructions dataset with p set to 0.1, O.5. and 10, respectively. A smaller p indicates greater Non-IID degree among clients. The experiment repeated for 10 rounds under each p value. The mean, standard deviation of model accuracies collected on the MMLU Benchmark are shown Table 10 (t means that higher accuracy is better):
12 1t "ttttps://uuggingffee.c//// httpts///heeginggee.ccc/////////
29

---

Published as a conference paper at ICLR 2026
Table 10: MMLU benchmark under varying Non-IID conditions.
P
STEM
SS
Humanitiest
Averaget
0.1
24.8.10.00
25.5
25.410.15
25.9110.09
0.5
24.7++0.15
25.6+:0.46
25.4++021
25.8++0.21
24.7+0.51
25.4+00.11
25.3++0.25
25.8++0.06
10
24.8+0.21
25.4+0.12
25.5++0.1
25.8++0.10
We can see that no matter how Non-IID the clients' data is, the models trained with SHE-LoRA can achieve stable performance across clients, which validates the robustness of SHE-LoRA under varying Non-IID conditions.
F.3 EFFICIENT ESTIMATION OF MUTUAL INFORMATION
As described in Section 2.2, mutual information measures the amount of information shared between two variables. According to Eq. (3), evaluating the mutual information requires knowledge of p(æ), ply) and the joint density p( y). yet in practice we have only samples and not the true densities. The simplest empirical approach is histogram (binning) estimator, which partitions the space and counts frequencies. However, histograms require large sample sizes and are sensitive to the choice of binning. more stable nonparametric approach is to employ kernel density estimators (KDE) (Moon et al., 1995). Concretely, flatten the parameter matrices W and W -w into one-dimensional collections and treat the corresponding elements as paired samples {(T,, 1 In this step, the marginal distribution of p(x), p(y) and the joint density p(x.3) estimated by kernel density estimation (KDE), which constructs a smooth probability density function by centering kernel functions (e.g.. Gaussian) at each sample point and aggregating them with an appropriate bandwidth. Once these probability density variables are obtained, they are substituted into Eq. (3) to compute the final mutual information. The code for the mutual information calculation is given follows:
from sklearn. sklearn..eighhors import t KernelDensity def ddd__autua__ifoo(x_llat,, Y_flat, bandwidth=0.2)
X_flat f.l.l....aatenn
Y_flat flatte)
= (=n_f_laat)
sample_num = min(10000, n)
sample_points np. www.rraaddnciceee.. sample_num, replace=False)
X_sample "__laa[ssmple_poonnsrrreeppppp
1)
Y_sample v_flat[sampleeee
1)
XY xY_sample = np. up.hsttack(X__samplee Y_sample)
kde_x = kerneDDDnssyyybmmmooiltttbboooon
fit(X__apple)
kde_y = Beneellonsstyymmmmn..dchbbbbaaia
kde_xy kennel/eestty/cmmmmmmddhhaaaaaih http://hhheooonootcocooooocooooo
log_pt kde_x. ade_x.sscree_smmplss(__ssspppp
log_py lie_y.sscore_samlee(...
log_pxy = kde_xy. wwwrssooorsaaeeeeee www.ssssssleleeX' eXxsaeesXXXX_ssmllell
return up.mean(log_pxy log_px log_py)
F.4 RESISTANCE AGAINST MEMBERSHIP INFERENCE ATTACKS
We fine-tune the base model (denoted as Vanilla LoRA in tables) and SHE-LoRA with varying encryption ratios respectively. Then, we implement seven membership inference attacks (MISs)(Loss (Carlini et al.. 2021), Lower case (Carlini et al., 2021), Zlib (Carlini et al., 2021), Min-k (0.1) (Shi et al., 2024), Min-k (0.5) (Shi et al., 2024), Recall (Xie et al., 2024) and PAC (Ye et al., 2024)) on the base model (denoted as Base in tables) and the fine-tuned models of Vanilla LoRA and SHE-LoRA. Under SHE-LoRA, attackers can only launch MIAS based on unencrypted parameters. The attack results are reported in AUROC with Table 11, FPR @95 with Table 12 and TPR@5 with Table 13.
on the PILE dataset using standard LoRA
In the AUROC results, Base performs no better than random guessing (with AUROC=50%) con- firming that the pretraining corpus does not include the evaluation data. In contrast, Vanilla LoRA achieves much higher AUROC results across all attacks, indicating substantial membership leakage after fine-tuning. Remarkably, compared with Vanilla LoRA, SHE-LORA reduces the average MIA
30

---

Published as a conference paper at ICLR 2026
Table 11: The AUROC results reported under 7 membership inference attacks.
Model
Loss
Lowercase
Zlibb
Min-k (0.1)
Min-k (0.5)
Recall
PAC
Base
50.9%
48.4%
50.2%
50.5%
50.9%
50.1%
51.2%
Vanilla LoRA
81.4%
80.5%
76.7%
80.9%
82.9%
73.8%
83,3%
1%c
62.6%
62.8%
60.3%
62.5%
63.5%
64.7%
65.0%
1%
56.8%
57.7%
55.4%
56.5%
57.3%
58.4%
58.4%
5%
54.1%
55.2%
53.1%
53.7%
54.3%
55.8%
55.3%
7 10%
56.8%
57.7%
55.4%
56.5%
57.3%
58.4%
58.4%
7 20%
52.4%
53.2%
51.7%
52.1%
52.5%
53.1%
53.3%
success rate by 21.0% with an encryption ratio as low as 7 = 1%c. With the increasing of y (c.g.. to I 1%), attack success rates further drop by 20.9% 30.9%, resulting in nearly random-guessing performance and demonstrating significantly stronger privacy protection.
Table 12: The FPR @95 results reported under membership inference attacks.
Model
Loss
Lowercase
Zlib
Min-k (0.1)
Min-k (0.5)
Recall
PAC
Base
94.1%
95.4%
96.1%
96.1%
95.2%
95.5%
95.7%
Vanilla LoRA
66.5%
63.3%
88.8%
71.4%
66.2%
79.2%
72.1%
?
89.7%
86.9%
93.9%
90.8%
90.3%
90.5%
89.9%
Y 1%
92.4%
90.7%
95.2%
93.8%
93.4%
91.6%
93.1%
7 5%
93.2%
92.1% %
95.5%
94.5%
94.1%
93.1%
93.8%
7 10%
92.4%
90.7%
95.2%
93.8%
93.4%
91.6%
93.1%
7 20%
93.2%
93.5%
95.6%
95.4%
95.1%
94.0%
94.8%
Table 13: The TPR@5 results reported under 7 membership inference attacks.
Model
Loss
Lowercase
Zlib
Min-k (0.1)
Min-k (0.5)
Recall
PAC
Base
8.6%
5.6%
7.4%
5.5%
8.0%
5.1%
9.9%
Vanilla LoRA
35.7%
35.6%
39.7%
39.4%
41.8%
25.4%
53.5%
1%
12.5%
13.0%
15.2%
14.0%
14.5%
15.4%
17.4%
7 1%
9.7%
9.3%
10.8%
9.1%
10.2%
10.3%
15.0%
7 5%
9.0%
7,4%
9.1%
7.3%
9.3%
7.7%
12.4%
7 10%
9.7%
9.3%
10.8%
9.1%
10.2%
10.3%
11.9%
y 20%
8.2%
6.7%
8.5%
6.5%
8.2%
6.2%
11.1%
Consistent with the AUROC results, even at = 1%c, SHE-LoRA achieves an average @P5 of 90.27% and average TPR @ of 14.57%, closely comparable to the base model's performance (FPR (TPR@95=95.44 94%. TPR (57..16%)
contrast, Vanilla LoRA significantly more vulnerable, with average FPR @95 of 72.50% and TPR@5 of 38.73%. These results demonstrate that SHE- LoRA preserves membership privacy during fine-tuning: even under a very small encryption ratio, attackers only achieve performance close random guessing, with negligible advantage in distinguishing members from non-members.
In summary, experiments LoRA is consistently robust. This stems from two key design features: 1) selective encryption of the most sensitive parameter columns prevents direct leakage of privacy-critical information, and 2) column-wise position obfuscation, similar to injecting structured perturbations (see Appendix D.7), increases uncertainty for attackers. These mechanisms also mitigate property inference and recon- struction attacks that leverage auxiliary priors, as they obscure the very gradients or parameters these attacks typically exploit.
-ween3-4B-Instruet-ee
across MIAs demonstrate that SHE-
F.5 IMPACT OF SENSITIVE PARAMETERS ON PERFORMANCE
As theoretically established in Appendix B.I, parameter sensitivity is closely linked to privacy risk. To empirically validate this connection, we conduct experiments on the PILE dataset using the Qwen3-4B
Instruct-2507 -iven3--BBIIstmmet--25 model. Specifically, fine-tune the model the training set and
31

---

Published as a conference paper at ICLR 2026
Table 14: Model performance comparison with perplexity.
Model
Base
Vanilla LoRA
? = 156
7 = 1%
? == 5%
Y = 10%
PPI.
74.01
21.23
38.83 21.23 50.53 / 21.23 55.21 21.23 60.57/21.55
evaluate text generation quality via perplexity (PPL) on the validation set. A higher PPL indicates poorer adaptation to the target domain. We compare three settings Table 14: (i) the original base model (denoted as Base), (ii) a raw LoRA-finetunae model (denoted as Vanilla LoRA), and (iii) SHE- LoRA with varying encryption ratios y This allows to assess whether protecting high-sensitivity parameters, rather than removing or ignoring them, preserves model utility while enhancing privacy.
SHE-LORA reports PPL. metrics: the value on the left of "/" reflects the model performance when encrypted parameters are masked (i.e., using only unencrypted columns), while the value on the right of "/" reflects the model's true performance without masking parameters. As expected, Base exhibits high PPL due to lack of domain adaptation, whereas Vanilla LoRA significantly reduces PPL, confirming effective learning. Notably, SHE-LORA with an encryption ratio of merely 1% already raises the masked PPL to 38.83, indicating that even trivial removal of the most sensitive columns substantially degrades utility. In contrast, the PPL result on the right of "f" is nearly identical that of Vanilla LoRA, demonstrating sound utility preservation under SHE-LoRA. A. Furthermore, as the encryption ratio increases, masked PPI. consistently rises, confirming that SHE-LORA prioritizes the most privacy-sensitive columns.
G TABLE OF NOTATIONS
Table 15 lists the main notations used in this paper.
Table 15: Table of Notations
Notation
Description
E Rm
Model parameters of a LLM
W w-w E RmX=
Model parameters with zeroed-out
Wo R
Frozen pre- trained parameters
A E R
Low-rank adapter matrix A of LoRA
B E R
Low-rank adapter matrix B of LoRA
X E R
Input embedding
G
Gradient Matrix
Loss function
2(-)
Sensitivity computation function
S()
Selective HE method
R()
Any data reconstruction attack method
T
Rank of LoRA adapter
L
Number of tokens in input sequence
Ti E RR.
The i-th features in the input
T(W; W-
Mutual information between W and W-w
Ratio of parameters in client i for encryption
K
Number of columns client for encryption
G
Group of indices of selected columns in client i
S.
Sensitivities of the columns in G, on client
b
Block i of tensor to be encrypted
N
Number of tensor blocks to be encrypted
pk
Public HE key
C
Ciphertext of the i-th block
K
Max columns unencrypted parameters among clients
H LIMITATIONS
As described in Section 2.3 and Appendix C, SHE-LoRA operates under the assumption of an nonest-but-curions server, where all clients share the same HE key. Although secure communication channels can be used to defend against malicious clients or collusion between the server and clients, such mechanisms incur higher encryption costs. A promising direction for future work is to explore
32

---

Published as a conference paper at ICLR 2026
more efficient distributed parameter protection using techniques such threshold homomorphic encryption, multi-key homomorphic encryption, or proxy re- re-encryption.
I BROADER IMPACT
In this work, we leverage parameter sensitivity and SHE to ensure the secure aggregation federated LoRA against inversion attacks such DAGER. Such attacks are able to recover the original client data from clients' updates uploaded during federated PEFT, exacerbating privacy concerns and hindering the possibility of FL to extract value from distributed data. Our work offers adaptive and sufficient privacy preservation, while minimizing HE overhead per client in cross-device federated PEFT with LoRA.
Importantly we find that with more sensitive model parameters being encrypted, the mutual informa- tion that can be leaked from the model updates drops dramatically, indicating that is possible to effectively reduce the risk of privacy leakage in terms of privacy information as long as the sensitive model parameters are correctly enerypted. Our work implies that critical information within the model parameters can be soundly protected against the SOTA attacks by merely encrypting less than % of the model parameters. Furthermore, we take into account the heterogeneity of the parameter sensitivity and encryption capabilities across clients, and broadly adapt the cost-effective SHE-LoRA to accommodate clients with diverse data distributions and device capabilities. With these observa- tions, we highlight the feasibility and effectiveness of applying tailored and secure privacy protection for cross-device federated PEFT at much lower overhead compared to existing off-the-shelf privacy protection techniques.
J THE USE OF LARGE LANGUAGE MODELS (LLMS)
According to the policies large language model usage ICLR 2026, state that LLMs only used to help with paper writing, including spell checking, grammar checking, and polish writing.
33


Reviews:
[{"review_id": "tkDryGNMoG", "reviewer": "Reviewer_9N3M", "summary": "The paper proposes a privacy-preserving method for fine-tuning large language models in federated learning. Traditional LoRA-based methods risk data leakage through shared gradients, while full homomorphic encryption is too costly. SHE-LoRA introduces selective homomorphic encryption and designs a column-level negotiation and aggregation mechanism to handle heterogeneous clients with different computing resources and privacy needs. The approach includes four key stages: (1) clients privately report column sensitivity to jointly select a global subset for encryption; (2) encrypted columns are reordered for efficient HE operations; (3) aggregation aligns plaintext and ciphertext parts column-wise; (4) results are re-factorized into new low-rank parameters. Experiments on NLP tasks show SHE-LoRA maintains model accuracy while reducing encryption and communication overhead by up to 99% compared with full HE, and it effectively defends against gradient inversion attacks like DAGER. The system scales to large models and non-IID data, proving that partial, structured encryption can offer strong privacy with minimal cost in federated LoRA training.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The framework explicitly supports clients with different hardware capabilities, network conditions, and privacy budgets.\n2. This paper introduces selective homomorphic encryption at the column level of LoRA matrices, encrypting only the most privacy-sensitive components.\n3. Experiments on NLP and vision datasets demonstrate that SHE-LoRA achieves accuracy comparable to or better than state-of-the-art methods (e.g., Flex-LoRA) under heterogeneous and Non-IID conditions.", "weaknesses": "1. The privacy guarantees of selective encryption, the convergence behavior of federated training under mixed plaintext/ciphertext updates, and the optimality of the HE subset negotiation are not formally proved.\n2. SHE-LoRA method can't adapt to heterogeneous LoRA approaches like FLoRA.\n3. The experiments rely on relatively small base models and simple benchmark tasks, which limits the generalizability of its results to large-scale or more complex real-world scenarios. It is recommended to evaluate the method on stronger base models such as Qwen3 and Llama 3.2, as well as on more challenging benchmark tasks including MMLU-Pro, GPQA, MuSR, MATH, IFEval, and BBH.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ijqcc93zcr", "reviewer": "Reviewer_i41f", "summary": "The proposed method SHE-LoRA integrates selective homomorphic encryption (SHE) with LoRA-based federated fine-tuning under heterogeneous client resources and Non-IID data. It aims to tackle the challenge of adaptive SHE for heterogeneous clients and the expansion of encrypted subsets on SHE. Specifically, this is done through the proposed HE subset negotiation mechanism and selective encryption and column-aware aggregation. Empirical results have shown that SHE-LoRA is resistant to privacy attacks, communication-efficient, and performant.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is empirically effective, which shows advantages in privacy-preserving, communication cost, and model performance.\n- The problem of vulnerable and heterogeneous LoRA updates is motivated well.\n- Principled design of the proposed algorithm.", "weaknesses": "- Limited novelty in the adoption of SHE methods to LLM LoRA fine-tuning.\n- Column-wise weighted averaging is proposed, but the choice of weights (e.g., proportional to client data size or sensitivity) is not formally justified or compared.\n- The negotiated global HE subset is claimed to optimally balance privacy and HE overhead per client, but lacks formal optimality guarantees or approximation bounds (e.g., submodular coverage, budgeted max coverage).\n- The paper argues that encrypting A is sufficient, but the threat model and Section 2.4 discuss expanded encrypted subsets due to BA multiplication. It is not clear whether encrypting only A’s sensitive columns is enough under all considered attacks.\n- Presentation could be improved by fixing the inline citations, defining the exact weighting scheme in column averaging, and reporting the Non-IID partitioning method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6dXAHRk5vc", "reviewer": "Reviewer_pYmU", "summary": "The paper proposes SHE-LoRA, a framework that combines selective homomorphic encryption (SHE) with low-rank adaptation (LoRA) for cross-device federated fine-tuning of LLMs. The core ideas are: (1) negotiation across heterogeneous clients using order-preserving encryption to choose columns to encrypt based on channel-wise sensitivity; (2) column swapping to cluster encrypted vs. plaintext columns for efficient batching and to obfuscate positions; (3) column-aware adaptive aggregation that aggregates plaintext and ciphertext parts separately; and (4) reparameterization via SVD so each client recovers LoRA factors at its local rank. Experiments show accuracy comparable to non-private heterogeneous LoRA while reducing communication by up to 99.71% and encryption time by up to 99.87% vs. full-HE baselines, and strong robustness to DAGER gradient inversion.", "questions": "1. How does SHE-LoRA fare against membership/property inference or reconstruction attacks that use auxiliary priors? Any reasons to expect similar robustness?\n2. How should a,b,c be chosen in practice? Could you provide an adaptive rule and an ablation on real datasets?\n3. For highly dynamic Non-IID clients, how often should renegotiation occur? Can you report end-to-end time or energy overheads for different periods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Massive reductions in HE time and bandwidth vs. full HE and MaskCrypt, with stable per-client times due to column clustering and budget control.\n2. Comparable to non-private Flex-LoRA across GLUE/MMLU and vision tasks; sometimes better on subsets.", "weaknesses": "1. OPE preserves order information; while only rankings are revealed, the paper does not quantify leakage from order disclosure or compare with order-revealing encryption alternatives.\n2. For 30B/70B models, ciphertext size/time per parameter increases notably and requires larger key sizes. While still workable at small budgets, the practicality at higher budgets or longer runs is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bo6cmcLiBY", "reviewer": "Reviewer_sB8q", "summary": "This paper introduces a new framework for privacy-preserving federated fine-tuning of large language models (LLMs) that balances efficiency, privacy, and heterogeneity among client devices. The proposed SHE-LoRA integrates SHE with LoRA to allow clients with diverse computational capacities and data distributions to securely participate in federated fine-tuning. The method adaptively encrypts only the \"most sensitive\" parameters based on a parameter sensitivity analysis, using a global negotiation mechanism to determine a subset of the parameters to be encrypted. Besides, the adaptive aggregation (on server side) and reparametrization (on client side) techniques are proposed to enable the FL cycle.", "questions": "Please refer to the question in the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The paper proposes a framework that provides a strong privacy protection guarantee for the models.\n2) The proposed algorithm can save computation cost significantly compared to the existing methods.\n3) The proposed algorithm provides some insight about how to define and select important parameters.", "weaknesses": "1) The parameter subset selection and negotiation may remain unclear for readers. For example,\n* Why is $S_j$  defined in the paper a reasonable metric to evaluate the importance? \n* What does \"features aggregated across L tokens\" mean in the context of the paper? \n* If the importance of the parameters changes in the training process, does the selection dynamically adapt to such changes? \n2) It is unclear what privacy protection strength this algorithm can provide in the worst case.\n* Do the most sensitive parameters equivalent to those that should be protected for privacy protection?\n* It seems there is no guarantee that the most sensitive parameters of a client will remain after negotiation. Does it mean the privacy of such a client, if it exists, is more vulnerable than that of others?\n* The attack experiments only consider reconstruction attack, but no membership inference attack (which is more closely related to the privacy definition people commonly believe).\n3) Given that the encrypted parameters are only a very small fraction, it is unclear whether directly removing those parameters from training can also produce similar model performance. If so, it may become arguable that the \n* Some ablation studies about the parameter may help.\n4) The writing of the paper can be further improved.\n* Some notations and definitions are missing in the paper. For example, the notation of \"batch\" uses the same notation as the matrix $B$ in LoRA.\n* Too many details are omitted in the main text, which makes readers hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)