SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC
Authors: Jinglong Luo
Zhuo Zhang
Yehong Zhang
Shiyu Liu
Ye Dong
Hui Wang
Yue Yu
Xun Zhou
Zenglin Xu
Venue: ICLR 2026 Poster (2026)
Keywords: privacy-preserving
secure multi-party computation
large-language models
prompt tuning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Privacy-preserving prompt tuning framework based on secure multi-party computation.

Abstract: Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant efficiency challenges, particularly in backward propagation, optimizer, and self-attention operations. To address these challenges, we propose SecP-Tuning, the MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs. SecP-Tuning innovatively integrates Forward-only Tuning through the ''data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization processes. Furthermore, it devises an efficient privacy-preserving Random Feature Attention, effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Experimental results demonstrate that, compared to full-Parameter Supervised Fine-Tuning and gradient-based prompt tuning, SecP-Tuning achieves approximately 12$\times$ and 16$\times$ end-to-end acceleration, as well as 17$\times$ and 20$\times$ reductions in communication overhead, respectively. Moreover, it delivers performance comparable to gradient-based methods across multiple few-shot tasks. Additionally, the ''black-box/API-style" privacy-preserving tuning paradigm of SecP-Tuning effectively avoids memory leakage risks caused by gradient/parameter transmission, thereby striking an optimal balance between privacy, efficiency, performance, and deployability.

Published as a conference paper at ICLR 2026
SECP-TUNING:
EFFICIENT PRIVACY-PRESERVING PROMPT TUNING FOR LARGE LANGUAGE MODELS VIA MPC
Jinglong Luo Zhuo Zhang Yehong Zhang"1 Shiyu Liu
Ye Dong Hui Wang! Yue Yu² Xun Zhou Zenglin Xu*.3,4
Pengcheng Laboratory, Harbin Institute of Technology, Shenzhen
Fudan University, "Shanghai Academy of AI for Science
Instiitte of Statistical Interdisciplinary Research, Southwestern University of Finance and Economies
National University of Singapore
(luojl, @ppan ac. acc. on
shouxan2028@hit.. edu. en, zeng 1 ini
ABSTRACT
Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-base srivacy-preserving machine learning provides theoretical guarantees for the pri- vacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant ef- ficiency challenges, particularly in backward propagation, optimizer, and self- attention operations. To address these challenges, we propose SecP-Tuning, the MPC-based framework designed efficient, privacy-preserving prompt tun- ing of LLMs. JecP-Tuming innovatively integrates Forward-only Tuning through the "data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization pro- cesses. Furthermore, it devises an efficient privacy- crivacy-preserving Random Feature Attention, effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Exper- imental results demonstrate that, compared to full-Parameter Supervised Fine- Tuning and gradient- gradient-based prompt tuning, SecP- Tunnng achieves approximately 12x and 16x end-to-end acceleration, well as 17x and 20 reductions in communication overhead, respectively. Moreover, it delivers performance com- parable gradient- basedd methods across multiple few-shot tasks. Additionally, the Beaack-box/API-styhee privacy-preserve
tuning paradigm of SecP-Tuning ef- fectively avoids memory leakage risks caused by gradient/parameter transmission. thereby striking optimal balance between privacy, efficiency, performance, and deployability.
INTROCTION
Large Language Models (LLMs) (Vaswani et al. 2017 Liu et al.) 2019, Hurst et al. 2024 Dubey (et al.) 22024; Guo et al. 2025) have achieved groundbreaking advancements in diverse domains, in- cluding natural language understanding, generation, reasoning, and cross-modal applications. How- ever, adapting universally pre-trained LLMs to high-sensitivity fields such as healthcare, finance, government compliance, and industrial manufacturing remains significant challenge. This diffi- culty arises from the fact that such sensitive data closely tied to the interests of data owners and is subject regulations (e.g., GDPR, HIPAA) and corporate compliance requirements, making direct access impractical. Additionally, model parameters may encapsulate statistical information from
"Corresponding author

---

Published as a conference paper at ICLR 2026
(5) Time Breskdown Badkward Opt.
0:) Time Breakdown Sofmax
Backward Opt. Conmunicctioo
uad.
Cxrmin Volumm (CO)
Comm Voture (GB)
Derera
73%
75%
Scquenoe Longth
Scauonco Langth
Figure 1: The time breakdown for SFT of RoBERTa (24 layers, 1024 dimensions) using MPC analyzed with a sequence length of 512, along with a comparison of communication volumes across different sequence lengths.
the source domain, posing potential privacy risks. Therefore, the key scientific and engineering challenge in achieving the implementation of "trustworthy intelligence" lies in efficiently adapting LLMs specific domains using effective fine-tuning methods, such Full-Parameter Supervised
Fine- Fine-Tuning (SFT) (Wei et al. 2 Devlin et al., 2019). Low-Rank Adaptation (LoRA) (Hu et al.)
2022 Dettmers et al. 2023). and Prompt Tuning (Lester al. 2021 Liu al. 2022). while ensuring
that neither the fine-tuning data nor the resulting model parameters are exposed.
Privacy-rreeerving Machine Learning (PPML) based on Secure Multi-Party Computation (MPC) Yao 1986 Goldreich et al.] 1987) offers promising solution. In this paradigm, model parameters and sensitive data are first secret-shared among participating parties. These parties then execute MPC protocols through multiple rounds of communication to complete orivacy-preserving computations for forward propagation, backward propagation, and optimization. All computations performed on secret-shared imputs and intermediate results, ensuring that parties only learn the protocol's 's explicitly permitted outputs without accessing private data or model parameters. Due to its compelling privacy guarantees, MPC-based PPML has been successfully applied to the training of linear models (Mohassel and Zhang. 2017). convolutional neural networks Wagh et al. (2019)
2021) and the inference of Transsorrmr- LLMs (Hao et al. 2022 Luo al. 2024 Pang al. 2024
Lu et al. 2023).
However, implementing Privacy-Preserving Fine-Tuning (PFT) of LLMs directly using MPC incurs prohibitive overhead. For example, performing SFT on a RoBERTa model with 24 layers and 1024 dimensions using CrypTenKKott al. (202I). with sequence length of 512, requires approx- imately 10 minutes per sample iteration and generates around 970GB of communication overhead. As illustrated in Figure [ two primary factors contribute to this overhead: a) Backward propa- gation and optimization, which account for 73% of the total runtime, far exceeding the cost of forward propagation. This is due the presence of numerous MPC-unfriendly nonlinear operations in backward propagation and optimization, such Softmax, GELU, and LayerNorm, which must undergo preracc-rreerving reverse computation. These operations cannot be directly executed in MPC environments and must be decomposed into approximations using addition, multiplication, and comparison, leading to a dramatic increase in communication rounds and volume. b) Sofimax in the self-attention, which contributes 75% of the total runtime. This is because Softmax involves large number of MPC-unfriendly nonlinear operations, including exponentiation, division, and max- imum computation. Furthermore, its computational complexity scales quadratically with the input sequence length, causing communication overhead to grow rapidly as sequence length increases. Gradient-based efficient parameter fine-tuning methods, such LoRA and gradient-based prompt tuning, effectively reduce the number of parameters requiring updates and enhance the efficiency optimization. However, they fail to resolve the fundamental communication overhead caused by backward propagation and Sofimax operations in MPC settings.
of
In this paper, we take the first step toward addressing the research question: How to perform privacy- preserving domain adaptation of LLMs in MPC environments efficiently and with high perfor- mance? Specifically, we propose SecP-Tuning, the MPC-based grivacy-preserving prompt tuning in LLMs. SecP-Tuning leverages Forward-only Tuning (FoT) (Sun et al.) 2022b a) to update prompt parameters, fundamentally eliminating the high communication overhead caused by backward propagation in gradient-busse fine-tuning methods, thereby significantly accelerating adaptation process. To address the MPC-unfriendly loss value and Gradient- Free Optimizer (GFO) (Rios and Sahinidis] 2013) computations in FOT, introduce innovative
framework for
the
2

---

Published as a conference paper at ICLR 2026
Sterver-Clienti architecture. In this architecture, MPC-unfriendly computations for loss values and GFO are offloaded to the data owner's local enviromment for efficient and precise plaintext com- putation. This approach not only significantly improves speed but also prevents the server from accessing updated prompt parameters, thereby mitigating the privacy risks of fine-tuning data leak- age caused by model memorization. Complementing this, propose Feature Attention (RFA), which avoids extensive nonlinear operations in softmax while reducing the complexity of self-attention from quadratic to linear.
Random
The experimental results systematically validate the comprehensive advantages of SecP- Tuning across multiple dimensions, including efficiency, performance, and deployability. Compared to SFT and gradient-bases prompt tuning. SecP-Tuning achieves approximately 12x and 16x end- to-end acceleration, respectively, while reducing communication volume by about 17x and x0x. Notably, these acceleration advantages are further amplified in bandwidth--oonsrriied Wide-Area Network (WAN) scenarios. terms of performance, SecP-Tuning achieved results comparable to gradient-based SFT and prompt tuning on five few-shot fine-tuning tasks, demonstrating its usability. Furthermore, the deployability comparison highlights that Tuning supports "black-box/API- style" secure tuning, effectively mitigating potential privacy risks of memory leakage caused by transmitting gradients or parameters back to the server.
2
RELATED WORK
Cryptographic techniques such as MPC and Homomorphic Encryption (HE) (Gentry 2009 Cheon
al. 2017) have been widely applied in privacy-preserving machine learning, including early works on linear networks (Mohassel and / and training and inference for convolutional neural
networks (Wagh et al. | 2019 2021] Liu et 2017 Riazi et al. 2018 Juvekar al., 2018). With the rise of Transformer- LLMs researchers have increasingly focused on srivacy-preserving
inference for LLMss (Hao et al. 2022 Li ei al. 2023 Zeng et al. 2023 Luo al. 2024 Pang al,)
2024 Yan et al.] 2025 Luo et al. 2025), aiming to protect both model parameters and inference
data. However, compared to inference, fine-tuning LLMs involves complex backward propagation and optimizer computations, which remain underexplored.
Currently, only few studies perform privacy-preserve domain adaptation of LLMs based on HE. Specifically, the first HE-based PFT framework, BlindTuner (Panzade et al. 2025). enhances practicality through pre-trained feature extraction while maintaining accuracy. Subsequently, Med- BlindTuner (Panzade et al. 2024) extended this approach to biomedical imaging and validated its effectiveness. To further reduce computational overhead, later works introduced parameter-efficien methods like LoRA: PrivTuner (Li et al. 2024b), which integrates LoRA with FHE to reduce com- putation overhead. Rho et al. (025) replaced self-attention with Gaussian Kernel Attention to miti- gate the costs of nonlinear operations. In addition, FedShield-LLLM (Mia and Amini] 2025) reduced computational overhead by combining unstructured pruning techniques.
Unlike HE, which relies on intensive unilateral encryption computations and requires costly approx- imations and re-encryption for nonlinear operations such as Softmax and GELU, making it difficult balance efficiency and accuracy, MPC enables complex nonlinear operations through multi-round communication among participants. This makes MPC more suitable for PFT. However, to the best of our knowledge, no prior work has explored MaC-based PFT of LLMs.
In addition to cryptographic techniques, Differential Privacy (DP) Dwork and Roth (014) has also fine-tuning. The primary goal of DP-based privacy-preserve fine-tuning algorithms (Wang et al., 2024] Li et al. 2024a; Charles et al.) 2024) is to ensure individual-level privacy. This is achieved by introducing mechanisms such as adding random noise and clipping during the fine-tuning process, which formally limit the influence of any single training sample on the final model. The privacy guarantee is quantified by the (c. 8) privacy budget. In con- trast, MPC-based for privacy parameters and fine-tuning data under a specified threat model, which is fundamentally different from ba-based
been applied to
fine-tuning frameworks provide theoretical privacy guarantees
frameworks.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
3.1 SOFTMAX-BASED SELF-ATTENTION & RANDOM FEATURE ATTENTION
Softmax-basee Self-Attention. The core component of each Transformer layer is the self- attention mechanism. We omit a detailed discussion of the feed-forward network and other auxiliary components, as they remain unchanged in our work. Let n. and denote the sequence length and embedding dimension, respectively. The self-attention mechanism is computed as follows:
Attention Q K. V) = Softmax ( [QK] (Va
V Rund
(1)
Here, the rows of Q, K, and V correspond to the query, key, and value vectors. The softmax function (Bridle, 1989) is applied row-wise, converting the similarity scores between each query and all key vectors into a probability distribution that weights the contribution of each value vector.
Random Feature Attention. speed up the softmax operations in attention, Peng al. (2021) has employed random feature (Rahimi and Recht, (207)) methods to approximate the dot-then- exponentiate operation using kernel tricks. The main idea to approximate the Gaussian kernel function via its Monte Carlo estimation:
exp - |x x x))12/07) Σ p0(x wx)xp(x',
where p(x,w;) 2/M cos(ws x + b,), with NNO,, 12 and b; (0,2T))
Let o(x) = (20([x (x(xxw))))))) p (xx wM)] the dol-then--xxponeniaae function can be approximated
exp (x y/o2)/ = exp 1ل2ي |x|
20 |1yll3)
exp 20 |x y113) zu x TTy)). (3)
Substituting this approximation into the softmax attention, obtain the RFA:
exp(a exp(a]k:/o")) k:/o"v/
Softmax(qe, (k:).= (vi)
Σ
Σ, exp k,
)(q)) v.
(4)
Ej ((q:) ((kk)
(q)) E, (k;) 8 v; RFA (ki) (v.)
ó q.)) Σ, o(k,)
where Q {q: 1. K | {v.)== and denotes the outer product between
vectors. Leveraging this linearized formulation, RFA achieves linear time and memory complexity with respect to the sequence length.
3.2 GRADIENT-FREE OPTIMIZATION
Gradient-Free Optimization (GFO) (Rios and Sahinidis 2013 function (fitness) evaluations, without gradients; hence, it is also called black- box or zeroth-orde optimization. These methods follow sample-evaluate evaluate update loop and are well-suited to settings where derivatives are unavailable or too expensive. Black-Box Tuning (Sun et al., 2022b) applies GFO prompt tuning for large language models (LLMss), learning continuous prompt vector E IRD that minimizes p" = arg min C(f(p: X), Y). where f is the LLM inference function, L the loss, and P the prompt space. Because GFO convergence typically degrades in high dimensions, BBT exploits the low intrinsic dimensionality of LLM prompts by optimizing a latent variable E Rd with & D and mapping via random projection E R Dxd:
optimizes an objective using only
arg min (A{ 2E2
(5)
CMA-ES (Hansen 2016) is used in this paper the gradient-free optimizer.
4

---

Published as a conference paper at ICLR 2026
3.3 2-OUT-OF-2 ARITHMETIC SECRET SHARING
For an integer ring Z. = (0,1,.... - 1,, a 2-out-of-2 arithmetic secret sharing scheme involves the following two algorithms:
The sharing algorithm Shr(z) + ([=]o, [z]1) is used to generate the shares of Specifically, value is chosen uniformly at random from 2n, such that [x]o = r, and [z)] = I - - (mod n) is computed.
The reconstruction algorithm Rec[[a]o, [a]1) + a used to reconstruct i, i.e., x = [x]] + [x]]
(mod n).
The randomness and uniformity of the share ensure that any individual share reveals no information
about the secret. We denote the arithmetic secret sharing of x as [x) = [[x)). [ar]1).
In the field of MPC, numerous MPC protocols have been developed for operating over secret shares [z). including secure addition, multiplication, comparison, and various nonlinear activation func- tions. These cryptographic primitives are summarized in Section 66.3 In this work, we treat these primitives as black-box components and utilize them without requiring additional assumptions or modifications.
SECP-TUNING
4.1 MPC-BASED PRIVACY-PRRESEVVING FINE-TUNING
The objective of crivacy-preservii fine-tuning based on MPC is to fine-tuning model while safe- guarding the privacy of both the developer's proprictary model parameters and the data owner's privacy data, ultimately producing fine-tuned parameters. This process involves two principal par- ties: the model developer and the data owner. The model developer possesses proprietary model Fe, where θ represents private parameters, while the data owner holds confidential fine-tuning data X. In this framework, both parties provide the shares of Fe and X, namely ([9)o, (9]:) and ([X) ln, [X]i), inputs. These shares processed using various two-party MPC protocols, such as privacy-preserving addition, multiplication, and GeL.U activation functions, to perform privacy- preserving inference and generate the shares of the fine-tuned parameters. Under well-defined threat models such as semi-honest and malicious models, the theoretical security is guaranteed by MPC protocols and ensures the following: 1) Confidentiality of the model developer's parameters; 2) Con- fidentiality of the data owner's s fine-tuning data; and 3) Confidentiality of the fine-tuned parameters.
For efficiency considerations, we adopt the semi-honest threat model In the semi-honest model, participants execute each step of the protocol correctly and obtain accurate results but may attempt to infer unauthorized information during execution. The semi-honest threat model is widely used in convolutional neural network training and more
PPML, including carly works recent efforts in seevacy-preservigg inference for LLMs.
Although MPC-based privacy-preservie fine-tuning provides theoretical assurances for the privacy of model parameters and fine-tuning data while achieving performance comparable to plaintext com- putation, directly employing MPC for fine-tuning faces significant efficiency challenges. These chal- lenges primarily stem from the computational costs associated with executing privacy-preserve backpropagation, optimizers, and self-attention mechanisms using MPC. To address these issues, propose Denning which leverages the intrinsic properties of MPC protocols and incorporates custom- designd, modular components to significantly enhance the efficiency of privacy-preserving fine-tuning.
4.2 PRIVACY-PRESERVIN FORWARD-ONI TUNING
During the backpropagation phase, numerous nonlinear operators, such as Softmax, GELU, and LayerNorm, must undergo privacy-preserving reverse computation. the MPC environment, these
While the malicious threat model better aligns with real-world scenarios, its computational overhead is significantly higher than that of the semi-honest model. Typically, additional cryptographic techniques such as zero-knowledge proofs required enhance the semi-honest model.
5

---

Published as a conference paper at ICLR 2026
Server
Server 1
Privacy- praccy-preeenvingg Inference
Privacy-preserving Infarence
2
Gradient-Free Optimizer
X
X
Inference Data Sharing
Prompi Embedding
X: Privacy Input
Initial Prompt
ever. <MASK>
A totally boring movie ! It was <MASK>
AZ
Fine-tuning
A R
great
Best film ever. was <MASK>
f
X
2
totally boring
<MASK>
2
v
Value
y: Inference
Data Owner
Figure 2: Workflow of SecP-Tuning. SecP-Tuning leverages MPC to protect both training data and model parameters during fine- sane-tuning. It addresses two key bottlenecks in PFT. First, it eliminates the computational overhead of backward and optimizer by adopting a FoT paradigm. Second, it improves the efficiency of privacy-preserving self attention by employing RFA.
operations cannot be executed directly and must instead be decomposed into fundamental opera- tions like addition, multiplication, and comparison for approximate computation. This decomposi- tion significantly amplifies both the number of communication rounds and the overall communica- tion volume. Furthermore, the deeply stacked architecture of Transformers exacerbates these costs. Additionally, frequent tensor transpositions, dimension rearrangements, and mask handling during gradient computation, which are mere memory operations plaintext, require explicit arithmetic- to Bo-Boolean domain conversions and additional synchronization in MPC environments, further in- creasing communication overhead.
During the optimization phase, widely used optimizers like Adam (Kingma and Ba 2015) require numerous element-win wise operations, including multiplication, division, square root computation, and bias correction, perform parameter updates. Among these, division and square root computations are particularly costly in MPC environments. Moreover, weight decay, learning rate scheduling (e.g., cosine, multi-stage, adaptive scheduling), and gradient scaling (used in mixed-precision simu- lations) introduce additional nonlinear operations and conditional branching. These complexities compel frequent domain conversions between arithmetic and boolean fields MPC environments, resulting in substantial communication overhead.
Forward-only Tuning (FoT) updates parameters via GFO, fundamentally circumventing the high communication overhead caused by crivacyprreseviigg backpropagation in gradient-based based fine- tuning methods. This presents a promising avenue for enhaneing the efficiency of privacy- prevecyg fine-tuning. However, unlike gradient- based optimizers such Adam, GFO methods, such as CMA- ES,, often involve complex operations that are unable to support in MPC-based PPMI. frameworks, such CrypTen. These operations include ranked index order, outer product of vectors, and matrix eigendecomposition. This hinder the development of an MPC-based
FoT.
To address this issue, SecP-Tuning integrates the features of MPC and FoT to design "Server- Client" architecture that ensures privacy while offloading GFO and loss computation to the client for plaintext processing. This approach not only significantly enhances efficiency but also prevents the server from accessing the updated prompt embeddings, thereby mitigating the risk of fine-tunm fine-tuning data privacy leakage caused by model memorization.

---

Published as a conference paper at ICLR 2026
As shown in Fig. 2 SecP-Tuning consists of the following seven steps: 1) The data owner locally initializes the prompt embedding p and concatenates it with the private fine-tuning token embedding to obtain the private input embedding X; 2) The data owner locally generates secret shares of X, denoted as (o 1x]),, and distributes them to the corresponding servers; 3) Two non-colluding servers take ([X)s oo (X]) and the secret shares of the private model parameters, ((+),, [0] , as inputs. They interactively execute Devacc-preserving inference using MPC protocols, producing secret shares of the inference result ([Yo, [Y];;; 4) The servers send [o, [Y]1) back to the data owner; 5) The data owner reconstructs the inference result Y using ([Y a. [Y]);; 6) The data owner takes the inference result Y and the fine-tuning data labels Y as inputs and calculates the loss value locally in plaintext; 7) The data owner inputs the loss value I, into the GFO to update the prompt embedding. By iterating this process multiple times, the data owner ultimately obtains the fine-tuned prompt embedding for
downstream task inference.
SecP-Tuning leverages the FoT framework from (Sun al. 2022b) to implement privacy-preserving fine-tuning. To guarantee fairness and reproducibility of results, it adopts the same GFO, CMA-ES. However, readers are free to select other gradient-free free optimizers, such random search, Natural Evolution Strategies, or Bayesian optimization, based on the specific requirements of their scenarios, thereby further enhancing the flexibility and adaptability of SecP-Tuning.
4.3 PRIVACY-PRESERVING RANDOM FEATURE ATTENTION
Although srivacy-rreeervin FoT based on "Server-Cliee" architecture addresses the overhead of saiacy--reservve computation in backpropagation and optimizers, SecP-Tuning still faces severe efficiency challenges stemming from the privacy-preserving implementation of softmax-based self- attention mechanisms. Specifically, for a vector x (I1, T Softmax in Transformer converts it to n-dimensional probability distribution with
(afimax((x)B
(6)
where r = max ((>A)==) is used to ensure stable numerical computations.
There are the following challenges in performing self-attention:
computation on softmax-based
Quadratic complexity with respect to sequence length. Given (Q, K, V) E Raxd, where n denotes the sequence length and the embedding dimension, the complexity of Softmax-based attention scales as (nn2d). This quadratic dependence becomes prohibitively expensive for long input sequences.
Numerous nonlinear operations incompatible with MPC. As shown in Eq. (6. computing the Softmax function involves three nonlinear operations-eepooenniiinn exponentiation, division, and maximiza- tion-all all of which are costly implement under MPC. These operations significantly inflate the attention computation (see Section 6.3.2 for details).
overhead of
To tackle these challenges, SecP-Tuning employs Random Feature Attention (RFA) to enhance the efficiency of prieeviy-reeeerving Softmax-bused self-attention mechanisms. Specifically, compared
to existing softmax approximation methods (Kitaey et al. 2020 Wang et al. 2020 Roy et al. 2021).
RFA offers the following advantages:
Theoretical Guarantee on Approximation Error. The approximation error is formally bounded,
ensuring reliable accuracy.
Reduction in Computational Complexity of Softmax. RFA reduces the complexity of softmax attention from O(n²d) to (ndr), where r represents the number of random features used.
Avoidance of Exponentiation and Maximum Operations in Softmax. By bypassing these costly nonlinear operations, RFA significantly improves efficiency in privacy-preseringg settings.
According to Eq. 4.. the computation of RFA involves multiplication, division, and cosine function operations. This implies that although RFA bypasses the exponential and maximum operations in softmax-hased attention, it introduces cosine operations that not friendly MPC.

---

Published as a conference paper at ICLR 2026
To address this challenge, SecP- SecP-Tuning design efficient MPC-based function protocol (Ilgosine) by leveraging the periodicity trigonometric functions and the sum-to- product formulas. By executing cosines MPC participants
cosine
compute the shares of the result
= (1) while preserving the privacy of the input data x. Specifically, Tassiee consists of two phases: offline phase and an online phase. In the offline phase, the computation servers S = 0, pre-generate random numbers t E Zr, and shares of sin(t), cos(t), and t, denoted as ([t), [s (cos(t)],). During the online phase, server Sj initially computes [ō]; [æ], + [fl.. Subsequently, 5 (xr + t) mod т, where T represents the periodicity of the trigonometric function, reconstructed through a single round of bidirectional communication. Finally, each server Sj computes the shares of cos(æ; using the trigonometric addition identity formulas, cos(2:) = (6) sin(() + c6) cos(t).
By executing cusine the privacy- privacy-presereee preserving computation of the cosine function can be accomplished with only single round of communication, transmitting 2t-bit elements. Building upon this result, RFA protocol, which reduces the
we further develop an efficient MPC-based privacy-preservve computational complexity of the Softmax-based attention mechanism while circumventing the need for expensive exponentiation and maximum operations. Detailed algorithmic descriptions are pro- vided in Section 6.4
5 EXPERIMENTS
5.1 SETUP
MPC-Backend & Testheds. Our implementation is based on the PPMI. framework CrypTTen while the execution of FoT and RFA relies on the open-source libraries provided in (Sun et al. 2022b) and (Peng et al. 202I). We conduct our experimental evaluations on three servers, each equipped with A100 GPU. To enable comprehensive efficiency comparison, we utilize Linux Traffic Control (TC) to simulate various network conditions. Specifically: In the LAN scenario, we set the bandwidth 3 Gbps with round trip latency of 0.8 ms. For the WAN setting, consider two different configurations: { {100 Mbps, 80 ms} and {200 Mbps, 40 ms).
2
Model and Dataset. We select ROBERTaLARGE as the backbone model to validate the effective- ness of SecP-Tuning across five representative datasets: SST-2 (Socher et al. [2013). MRPC (Dolan and Brockett, 2005), RTE Wang al.] (2018), Yelp Polarity (Zhang et al. 2015). and AG's News (Zhang et al. 2015).
Baselines. To demonstrate the effectiveness of SecP-Tuning we established the following base- lines: 1) SFT: Supervised fine-tuning of all model parameters of pre-trained model. 2) Prompt Tuning: Training only the prompt embeddings added to the input text while keeping the pre-trained model parameters frozen. For a fair comparison, we used the same prompt length, manual tem- plates, label words, and pre-trained prompt embeddings as PST/SecP-TTunng during initialization. To ensure the reproducibility of experimental results, we adopt the same hyperparameter settings as (Sun et al.) (022b) for FoT execution. For RFA, we follow the initialization settings from (Peng) cct al., 2021) and set the number of random features r to 128. Detailed configurations are provided in Section 6.6 of the appendix.
Table 1: Efficiency Comparison of RoBERTa in LAN Setting (3Gbps, 0.8ms). The input sequence length set to 512, and the number of prompt tokens is set to 50. The results are the average of ten runs.
Forward
Backward
Optimizer
Total Comm(GB)
Methods
Times(s)
Comm(GB)
Times(s)
Comm(GB)
Times(s)
Comm(GB)
Times(s)
SFT
216.184
260.41
554.512
691.150
20.902
19.159
651.598
970.720
Prompt Tuning
273.313
306.711
605.212
804.900
3.550
4.594
882.075
1116.205
FoT 173.999
205.358
0.000
0.000
0.138
0.000
174.138
205.359
SecP-Tuning
54.17
56.545
0.000
0.000
0.103
0.000
55.172
56.545
8

---

Published as a conference paper at ICLR 2026
5.2 EFFICIENCY COMPARISON
We perform an end-to-end execution of SecP-Tuning on CrypTen and compare it against base- line methods. Table [ shows the time and communication overhead of different methods in LAN environment, with additional results in a WAN environment provided in Section 6.2 Com- pared to SFT and gradient-basse prompt tuning. SecP-Tuning delivers substantial advancements in both fine-tuning speed and communication efficiency. Specifically, in LAN environment, SeeP- Tuning achieves a times speedup over SFT and 16 times speedup over gradient- prompt tuning. Additionally, it reduces communication overhead by 17 times and 20 times, respectively. This is primarily attributed to SecP-Tuning 's innovative integration of ToT through the "data owner- server interaction" paradigm, which eliminates crivacyprreservin computations for backward prop- agation and optimization. Additionally, the significantly enhances the efficiency of self-attentin computations in privacy- preserving settings.
protocol IRAA proposed in this paper
We further observed that, under MPC settings, gradient- based prompt tuning fails to bring efficiency improvements, and results in slower execution and higher communication overhead. This is because, while reduces the number of parameters requiring updates and thereby lowers the computational computations
overhead of for ward propagation and self-attention mechanisms. Furthermore, compared to model tuning, it incurs additional privacy-preserving forward and backward computations for prompt tokens.
optimization, it fails to avoid the privacy-preserve
Table 2: Comprehensive performance comparison of SecP-Tuning across various language under- standing tasks. The results in the table report the mean and standard deviation over three runs. All model with 16 samples per class.
experiments are conducted using the pretrained RoBERTa The results of the baseline are taken from (Sun et al. 2022b).
SST-2 Acc
Yelp
AG's News
MRPC
RTE Ace
Method SFT
Avg.
Acc
85.39 =: 2.81
91.82 + 0.79
86.36 1.85
77.35 : 5.70
58.00 + 6.21
79.90
Prompt Tuning Pre-trained prompt
68.23 + 3.78
61.02 6.65
84.81 +: 0.66
51.61 + 8.67
54.69 4: 3.79
64.07
/
77.48 4.85
77.13 1: 0.83
73.73
FoT
89.56 0.25
0.16 91.50
61.56
4.34
52.59
2.21
75.34
Pre-trained prompt
/
75.51
+: 5.54
77.62
+: 1.30
83.14
SeeP-Tuning
88.11 = 0.63
85.23 0:2300.0 0.46
81.27 4: 0.73
75.33 +: 5.69
52.95 1 0.33
76.58
5.3 PERFORMANCE COMPARISON
We evaluated the performance of SecP- -annig on multiple datasets and compared with baselines verify its usability. As shown in Table 2 after using pre-trained prompt embeddings (Gu al., 2.22], the FoT method achieved better performance compared to gradient-based methods. This may be due to the fact that gradient-based optimization methods are prone to overfitting in few-shot scenarios, while FoT, with its exploration mechanism, often finds better solutions (Sun et al. 20226). Without using pre-trained prompt embeddings, SecP-Tuning achieved comparable performance to gradient-based methods. Notably, in some simple sentiment classification tasks, such as SST-2 and Yelp P., SecP-Tuning even outperformed gradient-based prompt tuning, which validates the usability of SecP-Tuning.
Table 3: We evaluate the feasibility of As-A-Service (AAS), Accuracy, end-to-end time, commu- nication overhead, and the total amount of data uploaded/downloaded for completing PFT on the SST-2 and AG's News datasets.
AAS Acc
Fine-tuning Time
Communication Volume Upload (per query)
Download (per query)
SST-2 (Sequence Length: 47)
SFT
87.8
65.86 (h)
67.96 (TB)
Prompt Tuning
72.6
86.15 (h)
149.37 (TB)
SecP-Tuning
89.0
8.81 (h)
14.22 (TB)
12 KB
0.5 KB
AG's News (Sequence Length: 107)
SFT
88.4
75.37 (h)
121.27 (TB)
Prompt Tuning
84.0
80.57 (h)
153.45 (TB)
SecP
82.1
10.43
19.68 (TB)
44 KB
2 KB

---

Figure 3: Comparison of Time and Communication Overhead Between RFA and Published as a conference paper at ICLR 2026
5.4 DEPLOYABILITY COMPARISON
Beyond efficiency and performance, many other factors must be considered in practical scenarios. As shown in Table 3, we comprehensively compare SecP-Tuning with baseline methods across various dimensions, including serviceability, accuracy, fine-tuning time, communication volume, and the amount of uploaded and downloaded data. To ensure fair comparison of fine-tuning time, we employ early stopping for all methods: if no improvement in validation accuracy observed after 1000 steps, the training process is terminated. We find that only SecP-Tuning offers serviceability, allowing data owners to perform PFT directly via APIs provided by the model developer. This ensures that the model developer does not receive any information about the updated parameters. In contrast, gradient- based methods such as SFT and prompt tuning inherently require the model developer to obtain shares of the updated parameters. This introduces the risk of the model developer inferring private fine-tuning data from the updated model parameters. Thus, among all the methods considered, only SecP-Tuning achieves the best balance in terms of privacy, efficiency, performance, and deployability.
Time Speadup of Privacy--raseving RFA
(b) Communication Speedup Privacy-presaving
BPC-b Sel Att
MPC MPCebased Self Att
12
Werryyttret Frow
wnhout flow
Dera
Dex
1024
Softmax- -aamax Privacy- Preserving Self-Attention.
5.5 COMPARISON OF RFA AND ATTENTINN
We evaluated the privacy-preserving RFA protocol (ITFAA) under varying sequence lengths and compared it with both the MPC-based preserving RFA protocol without the efficient privacy-preserving cosine algorithm proposed in this study (IIF A.usshoouttToow....... As illustrated Figure 3: 1) For the MPC-based privacy-preserving self-attention, IæFA demonstrates significant improvements in execution speed and communica- tion efficiency. Moreover, as the input length increases, these advantages become increasingly pro- nounced. This is attributed to the computational complexity of the MPC-based diivacy-preservine self-attention mechanism being quadratic with respect to sequence length, whereas the RFA pro- tocol exhibits linear complexity. 2) For I w.shhott which are not MPC-friendly, results in relatively limited efficiency gains compared to the MPC- self-attention.
self-attention mechanism and the privacy-
the presence of cosine operations,
fact, for shorter sequence lengths, such as 64 and L 128, its time and communication overheads even exceed those of the MPC-based privacy- preserving self-attention. This directly highlights that the Iecosine algorithm is the critical factor in enhancing the computational efficiency of eewacc-preserving self-attention mechanisms.
based srivacy-preserving
CONCLUSION
This paper presents SecP-Tuning, the MPC-based framework designed for efficient and privacy- preserving prompt tuning of LLMs. By leveraging FoT, it eliminates secure backpropagation and optimizer computations, while introducing a prreacy-reserring random feature attention to substi- tute softmax- based self-attention, thereby cireumventing MPC-unfriendly nonlinearities and reduc- ing the computational complexity. Experimental results demonstrate that SecP- SecP-Tuning seamlessly integrates privacy, efficiency, performance, and deployability.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This study focuses on privacy-efficient fine-tuning mechanisms and does not involve ethical or moral concerns. It does not directly collect, generate, or interfere with any personally identifiable infor- mation (PII), relying solely on publicly available benchmark datasets (SST-2, MRPC, RTE, Yelp Polarity, AG's News). These datasets are widely used within the research community for English text classification and matching tasks, with licensing terms permitting their use for research pur- poses. Furthermore, no redistribution of the original data was conducted during the study, and only model performance and efficiency metries were reported.
REPRODUCIBILITY STATEMENT
This paper provides comprehensive resources to ensure the reproducibility of the experimental re- sults of the proposed SecP-Tuning Tuning algorithm. Specifically, thorough description of the theoretical foundations used in this study, along with relevant references, is included in Section 3 Detailed
steps of the proposed methodology are presented in Section 4 and pseudocode for the privacy- preserving algorithms proposed in this paper are provided in Section 6.4 Section [5.T] we present the experimental setup of SecP- BecP-nunnig, including the models, datasets, bascline configurations, de- pendency libraries, and network environment details. Detailed hyperparameter information is pro- vided in Section 6.6 These elements ensure the reproducibility of the results in this paper.
ACKNOWLEDGEMENTS
This work was supported by the Major Key Project of PCL (Grant No. PCL2025A09), the Na-
tional Natural Science Foundation of China (No. 62472125, No. 62206139. No. 72495122),
Natural Science Foundation of Guangdong Province, China (No. 2025A151501 12588,..
Key Technologies R&D Program of Guangdong Province (202680909060001),, Shenzhen Science, and Technology Programs 12XWD202312880229222222 DDSYSS00330220012000000
ZDCY20050001111000000
REFERENCES
John Bridle. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters, Advances in neural information processing systems, 2, 1989.
Ran Canetti. Universally composable security: A new paradigm for cryptographic protocols. In Proceedings 42nd IEEE Symposium on Foundations of Computer Science, pages 136-145. IEEE, 2001.
Zachary Charles, Arun Ganesh, Ryan McKenna, Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush. Fine-tuning large language models with user-level differential privacy. arXiv preprint arXiv:2407.07733, 2024.
Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic encryption for arith- metic of approximate numbers. In International conference on the theory and application of eryptology and information security, pages 409-437. Springer, 2017.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088-1000 10115, 2023.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186, 2019.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005), 2005.
11

---

Published as a conference paper at ICLR 2026
Ye Dong, Wen-jie Lu, Yancheng Zheng. Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Cheng. PUMA: Secure inference of LLaMA- ABB in five minutes. arXiv preprint arXiv:2307.12533, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prinis, pages XrXiv-2407, 2024.
C. Dwork and A. Roth. The algorithmic foundations of differential privacy. In The Algorithmic Foundations of Differential Privacy, pages 19-20, 2014.
Craig Gentry. A fully homomorphic encryption scheme. Stanford university. 2009.
Oded Goldreich, Silvio Micali, and Avi Wigderson. How to play any mental game or A completeness theorem for protocols with honest majority. In Proceedings of the 19th Annual ACM Symposium on Theory of Computing, pages 218 1-9229, ACM, 1987.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few- shot learning. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume I: long papers), pages 8410-8423, 2022.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1220 2025.
Nikolaus Hansen. The ema evolution strategy: A tutorial. arXiv preprint arXiv: 1004.0060000777 2016.
Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Pri- vate inference on transformers. Advances in Neural Information Processing Systems, 35:15718-
15731, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2 2024.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. GAZELLE: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium, pages 1651-1669, 2018
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http::/ http::///axxi..oo/abbbsss. www./bg.s.......oo... 4124111244 6980
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. CrypTen: Secure multi-party computation meets machine learning. Advances in Neural Information Processing Systems, 34:4961 4973,, 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Frrrie-Franeieee Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 3045-3059. Association for Computational Linguistics, 2021. doi: 10.18653/V 120.:3//1122222MEMPP.P.. EMNLP-MAIN.24 2433. URL Ut..ntpps:/dd...eg.1....11 /ttp://...0//00...//////////0220 emmlp- mwin.annnnn. in. 243
Dacheng Li. Rulin Shao, Hongyi Wang, Han Guo, Eric P Xing, and Hao Zhang. MPCFormer: Fast, performant and private transformer inference with MPC. In Proceedings of the Eleventh International Conference Learning Representations, ICLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Xianzhi Li, Ran Zmigrod, Zhiqiang Ma, Xiaomo Liu, and Xiaodan Zhu. Fine-tuning language mod- els with differential privacy through adaptive noise allocation. arXiv preprint arXiv:2410.02912, 2024a.
Yang Li, Wenhan Yu, and Jun Zhao. Privtuner with homomorphic encryption and lora: A p3eft scheme for srivacy-preservng parameter-efficient fine-tuning of ai foundation models. arXiv preprint arXiv:2410.00453, 2024b.
Jian Liu, Mika Juuti, Yao Lu, and Nadarajah Asokan. Oblivious neural network predictions via minionn transformations. In Proceedings of the 2017 ACM SIGSAC conference computer and communications security, pages 619-631, 2017.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pages 61 18, 2022.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 1907. 1692, 2019.
Wen-jie Lu, Zhicong Huang. Zhen Gu, Jingyu Li, Jian Liu, Cheng Hong, Kui Ren, Tao Wei, and WenGuang Chen. Bumblebee: Secure two-party inference framework for large transformers. Cryptology ePrint Archive, 2023.
Jinglong Luo, Ychong Zhang, Zhuo Zhang, Jiaqi Zhang, Xin Mu, Hui Wang, Yue Yu, and Zenglin Xu. Secformer: Fast and accurate privacy-preservee inference for transformer models via smpe. In Findings of the Association for Computational Linguistics: ACL. 2024, pages 13333-13348, 2024.
Jinglong Luo, Guanzhong Chen, Yehong Zhang, Shiyu Liu, Hui Wang, Yue Yu, Xun Zhou, Yuan Qi, and Zenglin Xu. Centaur: bridging the impossible trinity of privacy, efficiency, and performance in sinaacy-rreservng transformer inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 22751-22770, 2025.
Md Jueal Mia and M Hadi Amini. Fedshield-Ilm: secure and scalable federated fine-tuned large language model. arXiv preprini w:Xiv:2506.0564: 2025.
Payman Mohassel and Yupeng Zhang. SecureML: A system for scalable learning. In Proceedings of 2017 IEEE Symposium on Security and Privacy, pages 19-38. IEEE, 2017.
machine
Qi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng, and Thomas Schneider. Bolt: Privacy- preserving, accurate and efficient inference for transformers. In 2024 IEEE Symposium on Secu- rity and Privacy (SP), pages 4753-4771. IEEE, 2024.
Prajwal Panzade, Daniel Takabi, and Zhipeng Cai. Medblindtuner: Towards privacy-preserving fine-tuning on biomedical images with transformers and fully homomorphic encryption. In Al for Health Equity and Fairness: Leveraging Al to Address Social Determinants of Health, pages 197-208 Springer, 2024.
Prajwal Panzade, Javad Rafiei Asl, Daniel Takabi, and Zhipeng Cai. Blindtuner: On enhancement of privacy- fine-tuning of transformers based on homomorphic encryption. IEEE Internet of Things Journal, 2025.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong- Random feature attention. In 9th International Conference on Learning Representations, ICLR 0021,2021
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems, 2007.
13

---

Published as a conference paper at ICLR 2026
Donghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim, Hyunsik Chae, Ernest K. Ryu, and Jung Hee Cheon. eearyption-frinddee Ilm architecture. In The Thirteenth International Confer- ence on Learning Representations, ICLR, 2025.
M Sadegh Riazi, Christian Weinert, Oleksandr Tkachenko, Ebrahim M Songhori. Thomas Schnei- der, and Farinaz Koushanfar. Chameleon: A hybrid secure computation framework for machine learning applications. In Proceedings of the Asia conference on computer and communications security, pages 707-721, 2018.
Luis Miguel Rios and Nikolaos V Sahinidis. Derivative-free optimization: review of algorithms and comparison of software implementations. Journal of Global Optimization, 2013.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content- based sparse attention with routing transformers. Transactions of the Association for Computational Linguis- lics, 9:53-68, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pages 1631-1642, 2013.
Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuan-Jing Huang, and Xipeng Qiu. BBTv2: Towards gradient-free future with large language models. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 3916-3930, 2022a.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-s----eeee -sseevcce In International Conference on Machine Leurning, pages 20841- 20855. PMLR, 2022b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Sameer Wagh, Divya Gupta, and Nishanth Chandran. SecureNN: 3-Party secure computation for neural network training. Proceedings on Privacy Enhancing Technologies, pages 26 49,449945 2019.
Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateck Mittal, and Tal Rabin. Falcon: Honest-majority maliciously secure framework for private deep learning. Proceedings on 2021.
Privacy Enhancing Technologies, pages
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 07461, 2018
Naiyu Wang, Shen Wang, Meng Li, Longfei Wu, Zijian Zhang, Zhitao Guan, and Lichuang Zhu. Balancing differential privacy and utility: A relevance-basee adaptive private fine-tuning frame- work for language models. IEEE Transactions on Information Forensics and Security, 2024.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.do 247752000004468. 2020.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. 2021.
Guang Yan, Yuhui Zhang, Zimu Guo, Lutan Zhao, Xiaojun Chen, Chen Wang, Wenhao Wang, Dan Meng, and Rui Hou. Comet: Accelerating private inference for large language model by predicting activation sparsity. In 2025 IEEE Symposium on Security and Privacy (SP), pages 2604-2622. IEEE Computer Society, 2025.
Andrew Chi-Chih Yao. How to generate and exchange secrets. In Anmual Symposium on Founda- fions of Computer Science, pages 162-167, 1986.
14

---

Published as a conference paper at ICLR 2026
Wenxuan Zeng. Meng Li. Wenjie Xiong, Tong Tong, Wen-jie Lu, Jin Tan, Runsheng Wang, and Ru Huang. Mpevit: Searching for accurate and efficient mpe-friendly vision transformer with heterogeneous attention. In Proceedings of the IEEE/CVF international conference computer vision, pages 5052-5063, 2023.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Yu Zheng, Qizhi Zhang, Sherman SM Chow, Yuxiang Peng, Sijun Tan, Lichun Li, and Shan Yin. Secure softmax/sigmoid for machine-learning computation. In Proceedings of the 39th Annual Computer Security Applications Conference, pages 463-476, 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDICES
The appendix is organized as follows.
In Section
6.1
we report the use of large language models.
Section
6.2
We report presents additional efficiency results of SecP-Tuning.
Section
6.3
we present the underlying MPC protocols upon which SecP-Tumin SocenTuning! is built.
In Section
6.4
we detail the privacy-preserving algorithms designed for SecP-Tuning, including cosine and Random Feature Attention.
privacy-preserve
In Section 6.5
we provide comprehensive security proof of SecP-Tuning.
In Section
6.6
report all the hyperparameter settings used in this paper.
6.1
THE USE OF LARGE LANGUAGE MODELS
This work primarily utilized LLMs for academic English translation and refinement. The use of LLMs does not pertain to the significance, innovation, technical soundness of the core aspects of this work.
Table 4: Efficiency Comparison of RoBERTaLARGE WAN Setting. The input sequence length is
set to 512, and the number of prompt tokens is set to 50. The results are the average of ten runs.
Bandwidth Latency
Methods
Forward (s)
Backward (s)
Optimizer (s)
Total
SFT
605.315
718.987
48.036
2,372.338
Prompt Tuning
847.270
997 .662
7.810
2, 1.852.744
200Mbps/40ms
FoT
399.361
0.000
0.133
399.494
SecP-Tuning
102.923
0.000
0.125
103.048
SFT
1. 502.213
3, 893.772
98.822
5, 49499.00
Prompt Tuming FoT
582.691
692.951
10.975
7,2866117
100Mbps/80ms
833 433.448
0.000
0.136
833.584
SecP- SecP-Tuning
211.185
0.000
0.122
211.307
6.2 ADDITIONAL EFFICIENCY RESULTS OF SECP-TUNING
This section presents additional efficiency results of SecP-Tuning. shown the data from Ta- ble 4 under WAN setting of 100Mbps/80mss, SecP-Tuning reduces the update time per iteration from 7286.6 seconds in gradient- 34x acceleration, which significantly surpasses the 16x acceleration observed in 3Gbps/0. 8.8s LAN environment. This remarkable improvement stems from its substantial reduction commu- nication rounds and volume, enabling structural amplification advantages in bandwidth-conntrrinne and high-latency WAN scenarios.
Prompt Tuning to 211.3 seconds, achieving approximately
6.3 UNDERLYING MPC PROTOCOLS
In this section, provide brief overview of the underlying protocols used and refer to the works of Knott et al. 2022 and Zheng et al. (2023) for details. Let S, with j E (0, 1} two parties that are used to execute the MPC protocol. Each party S, will be given one additive share (lul,, (ul,) E Z of the operation inputs u and v for j E {0.1}.
6.3.1 MAIVACY-REEEEVVIEE ADDITION, MULTIPLICATION AND COMPARISON PROTOCOLS
In this section, provide detailed description of the execution processes for MPC-based addition, multiplication, and comparison protocols, along with a theoretical analysis their correctness and privacy guarantees. Other nonlinear crivacy-preserinn protocols in Section 6.3.2 and Section 6.4.2 can be constructed by invoking these three protocols, and thus their correctness and security can be directly proven based the aforementioned protocols.
16

---

Published as a conference paper at ICLR 2026
Privacy-preserving addition. Suppose participants, Alice and Bob, each possess secrets and v. By executing the addition protocol based on 2-out-of-2 arithmetic secret-sharing ((2, 2)-SS), they can compute shares of the output v. Specifically, the addition protocol based on (2, 2)-SS consists of two phases: the secret sharing phase and the computation phase.
while preserving the privacy of inputs and
In the secret sharing phase:
Alice locally generates shares of her secret u, i.e., Shr(u) - ([u)o, [u]a], and sends [u]1 to Bob.
Bob locally generates shares of his secret v, i..,, Shr(u) - ([v]o, [v]1). and sends [e] to Alice.
In the computation phase:
Alice computes [w]u [ulo + [u]o-
Bob computes [w]1 = [u] + [e]1-
Correctness Verification: [=]o+[>]: = julo [w]o [v) =
= w
Privacy Guarantee: During computation, Alice and Bob each possess only one random share of the secrets, ensuring that no information about the original secrets can be inferred.
Frivacy-preserving multiplication. Suppose two participants, Alice and Bob, each possess se- crets и and v. By executing the multiplication protocol based on 2-out-of-2 arithmetic secret-sharing ((2, 2)SS), they can compute shares of the output = u e while preserving the privacy inputs and Specifically, the addition protocol based on (2, 2)-SS consists of two phases: the secret sharing phase and the computation phase.
In the secret sharing phase:
Alice locally generates shares of her secret x, i.e., Shr(æ) ([æ]o, (e]1), and sends [x]] to Bob.
Bob locally generates shares of his secret y, i.e., Shr(y) >> ([y)o: [v]1). and sends [y] to Alice.
Alice possesses the first random shares of the Beaver triple (a, b, c), i.e., ([a)o: [b)o (clo)-
Bob possesses the second random shares of the Beaver triple (a, b, c). i... ([a]1. [6] [cl1).
Alice computes [d]o = [x)o [ajo and [e]o = [w]o [b)o-
Bob computes [d]1 [x]1 [a] and [e]1 [y]1 [b]1.
In the communication phase:
Alice sends [djo and [elo to Bob.
Bob sends [d] and [e]a to Alice.
In the computation phase:
Alice reconstructs d = [d]o + [d] = x and [e]o + [e]2
Bob reconstructs and similarly.
Alice computes [=]u = [r]oe + d'ylo + [clo-
Bob computes [z]1 = + [x]ee + d'yl + [c/1.
Correctness Verification:
[2] + [2] = |æ]oe + diu]o + |clo de + [z] e dly) + [c]
([æ] [x]]]e ([v)o [y]id de
xxy - b) + v( - (x a)(y - b) +
.y .bb + xy ag wy + ay zb ab +
.y.
17

---

Published as a conference paper at ICLR 2026
Privacy Guarantee: During computation, Alice and Bob possess only one random share each of and b. Since a and b are randomly generated and independent of the inputs z and y. no information about is revealed.
comparison. Similarly, Alice holds secret u and Bob holds secret v, and the comparison be implemented follows:
Alice and Bob first generate the shares of their respective private inputs, a.k.a., [u] and [u], as privacy-preserving addition.
Two parties locally compute
Two parties jointly invoke the Arithmetic-to-Be BoooBe conversion (Knott al., 202 to convert [w] from Arithmetic sharing to Boolean sharing (z) = A2B [w))
Two parties locally extract the most significant bit of Boolean sharing (z) as (b) (w)
(E -1
Finally, the additive shares ju < u] can be derived by converting Boolean sharing (b) to Arithmetic sharing [b] using Goolean-to-Arithmett conversion protocol (Knott et al.
2021
Correctness & Privacy. Except for sharing the inputs, the computation phase consists of loga + rounds of communication and transmits 3456 bits. The correctness is easy to follow, and the privacy guarantee is inherent from well-established 2PC basic primitives.
6.3.2 PRIVACY-PRESSERVIGG NON-LINEAR PROTOCOLS
Privacy-preserving maximum. ing log2
The maximum of the n-element vector is implemented by call- comparisons using the tree reduction algorithm (Knott et al. (021).
pressering exponential. The exponential function is complex and usually implemented using the appeated-squarnng upproximation method
l 0o (1
2"
2
which converts exponential calculations into addition and square operations. By fault, iterations set n = 8 in Knott et al. 202T).
divaacyppreervvine reciprocal. Function reciprocal 1 implemented by the Newton-Raphson method, which converts reciprocal calculations into addition and multiplication operations. The iterative formula is
Mr+1 = Vn x.
(8)
The initial value of the iteration is
y0 3e 0.003.
The number of iterations is set to 10 in (Kott al. 2021 by default.
square root. v is approximated by the Newton- Rewton-Raphson method MPC, which converts exponential calculations into addition and multiplication operations. The iterative
formula
1 Un+1 2 Mn -xy?)-
(10)
The initial value of the iteration is
yo 2.2(4+0.22 + 0. 0.199046875.
(11)
The number of iterations is set to in KKnott et al. 2021 by default.
l denotes shift bits to the right.
18

---

Published as a conference paper at ICLR 2026
Algorithm 1: Protocol for srivacy-preservngg Cosine Tossine
Input: For j E {0, S holds the shares [aly; Pscudo-Random Function (PRF), and key k,.
Output: For {0, (0,11),, S returns the shares [y]5, where y cos(x).
/* Offline Phase +/
So. [u]o [v]o <- (RF((k)
: -
4T:t= [1] [0],[[]i==iin sin(t) - [u]o, [v]i cos [e]o
/* Online Phase +/
[88, | + fll (mod
ô [ + [6] // reconstruct d by round of communicatio cation
p = sin( 9 (o)
f p|ul g
6.4 PRIVACY-PRESEERVIG PROTOCOLS
6.4.1 PRIVACY-PRESERVING COSINE
We propose efficient of the cosine function and trigonometric addition identity formulas. Here's a detailed description of the algorithm steps: In the offline phase, the protocol initiates by generating pscudo-random values. Specifically, So and the trusted third party T jointly produce [t]o. [u]o. [o]o by evaluating a
cosine protocol Tosssee by exploiting the periodicity
pseudo-random function (PRF) with a specific key ku- Similarly, S₁ and T generate [f]2 using
different key 41. Then, the trusted third party recover the actual value t = [t], [t] calculates
[u/2 sin(t) - and [o]] cos(t)-(uja This phase is crucial for preparing necessary correlated randomness that will be used in the online phase.
In the online phase, the parties compute the [sin(1:) securely. First, each party Sj computes [6]; [x], + [t] (mod r). where represents the periodicity the trigonometric function, such as 20. Then, through one round of communication, the parties reconstruct ō by exchanging [o]o and [8] Subsequently, we get
sin(ō) and cos(o). Finally, each party calculates [M]j = pl p[u], + qlolj- This effectively leverages the precomputed correlated randomness with the current input [a] to produce the [s)] in privacy- privacy-preserving preserving manner. The Iossine requires only one round of communication during the online phase, with communication cost of transmitting 28 elements.
Correctness Verification:
[y]o [v]1 plulo + g|v]o plu]] + q(v]1
p([u]o + [u]1) + 4(v]] + [e]1)
sin(5) + c8) cos(t)
cos(6- - t)
co +
cos(x).
Privacy Guarantee:
During the computation process, the server S, only obtains the information of [ ft (8]g, and 8. Since = x + mod r) and is independent x, ō also independent of Therefore, S, cannot gain any information about the private input during execution.
6.4.2 PRIVACY-PRESERVING FEATURE ATTENTION
The Privacy-preserving RFA Protocol (IIFA A) is designed to enable computation of RFA with pri- vacy preservation. The algorithm involves two parties, S, and St, and trusted third party T, to collaboratively compute the RFA while keeping the input data secure. In the offline phase, the pro- tocol begins with the generation pseudo-random values. Specifically. Sg and the trusted third party T' jointly produce [4]o, [u]o, [elo by evaluating a PRF with random seed and also generate matrix W using another random seed r. On the other hand, S₁ and the trusted third party generate [t], by evaluating the PRF with different random seed rı, and the same matrix W generated earlier. Then, the trusted third party T recovers the actual value t [t) + [t] Based on t, T computes [u]1 sin(!) (ulo and [v] cos(t) (t) [t)o- This offline phase essentially prepares some
19

---

Published as a conference paper at ICLR 2026
Algorithm 2: Privacy-preserving RFA Protocol (MRFA)
Input: For j E {0. 1).1 S holds the shares {[]e. [k]s, [u]:]:
Output: For
(0,1), S returns the shares [y]ل. where y RF ([g)( [k,a. [v].).
/* Offline Phase -/
So. f/o [u]o: < (RF( W &- (RF(r)
ST,T: t PRF(ri); W <
PRF(r)
TTtt [1/o + [0]] sin(t) - [u]o, [o]1 cos(1) |e/o
/* Online Phase +/
[d(a))] V 2 (W (::;)): [o(k)]; / Mensina (W(kss))
=]j [ō(k:)]s v;
8 (b(t)) [b{(k:)]
M) [
necessary random values and parameters, which will be used in the online phase. Although these values related trigonometric functions, they are computed in way that preserves privacy as the actual values are hidden within the shares.
Table 5: Core and auxiliary haper-parameters for Feedforward-only Tuning (FoT) and Random Feature Attention (RFA).
Name
Symbol
Default
Description
Batch size Optimizer
16 CMA-ES
Derivative-free evolutionary strategy (no backward propagation required). Number of continuous prompt tokens (controls raw dimension D L x demb)).
Prompt length
50
Initial prompt
po
NLI-pretrained
Pretrained prompt (e.g., on MNLI) for sentence-pair tasks. Dimension of the low-dimensional search subspace; trade-off between cnv- erage and GFO efficiency.
Subspace dimension
d
500
Population size
20
Number CMA-ES offspring per gen- eration (heuristic: 4 + 3 log d).
Random projection
A
Uniform
Projection matrix A R. fxd sampled from a uniform distribution (found su- perior to Gaussian).
Loss function
L(-)
Cross-Entropy
Provides dense supervisory signal un- der few-shot regime.
call budget
8000
Maximum number of model inference calls (evaluation points).
Early stopping
1000
Stop if dev accuracy shows no improve- ment for 1000 evaluations.
Kerel type
Gaussian
infft-invariant kernel approximated by random features.
Number of Random fea-
128
Number of random features per head.
tures
Random feature vectors {m
Base Gaussian samples; drawn once (fixed) for reproducibility.
In the online phase, the algorithm focuses on computing the attention mechanism. First, for each query d: at time step and key k; the corresponding feature mappings are computed. This is done by taking the shares of qe and k; (i.e.. [q]: and [k],) and applying a cosine-based transformation denoted as Iossine scaled by a factor of 22/r. The scaling factor is important to ensure proper normalization of the feature mappings.
20

---

Published as a conference paper at ICLR 2026
Next, for each key-value pair (k,, 04), the share [2], is computed as the element- Wesse product (de- noted by @) between the feature mapped key [o(k)]l and the value v,. This effectively combines the key's feature representation with its associated value.
Then, the attention score [s], is calculated as the dot product between the feature-mapped query [6( and the feature-mapped key [o(k:)] This dot product represents the similarity between the query and the key in the transformed feature space.
Finally, the output share (ul; is obtained by dividing [2], by [s]]- This step normalizes the combined key-value representation by the attention score, resulting in the weighted value that will be used as the output of the attention mechanism. The division here is crucial as it implements the attention- weighting process, where the value is scaled according to how relevant the corresponding key is to the query.
6.5 SECURITY ANALYSIS
SecP- Tuennggennes to monishonest (also known honest-but-curione assumption similar to the works Li et al. (2023) and Dong et al. (2023), where honest participants constitute the majority. Under this assumption, the security of SecP--Tuningens be formally proved against static semi- honest adversaries denoted as A, which can potentially corrupt no more than one of the servers in the hybrid model.
SecP- Tunniniss constructed from the well-established sub-protocols of Knott et al. (2021): Zheng al. 2023), and we invoke these protocols in black-box manner. Leveraging the concept of com- posable security established by CCnnetti (200T), it is easy to see that the security of SecP-Tuning is guaranteed in the sub-pr too-protoools hybrid model.
6.6 HYPERPARAMETERS
To ensure the reproducibility of the experimental results, SecP-Tuning adopted the same hyperpa- rameter settings as those used by its feedforward-only tuning and random feature attention mecha- nism plugin. The specific hyperparameter names, symbols, values, and descriptions are detailed in
21


Reviews:
[{"review_id": "Mzfh8prase", "reviewer": "Reviewer_W9X4", "summary": "This paper introduces SecP-Tuning, an efficient MPC-based framework for privacy-preserving prompt tuning of Large Language Models. It eliminates the need for privacy-preserving backpropagation and optimization by adopting a forward-only tuning strategy, where sensitive computations are done locally by the data owner. The framework further enhances efficiency through Random Feature Attention, which replaces costly softmax operations with a linear-time approximation. The approach is validated through a series of experiments that benchmark efficiency, accuracy on five language tasks, and deployability compared to prior methods, including full supervised fine-tuning (SFT) and gradient-based prompt tuning in MPC settings.", "questions": "Please see weaknesses part", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Through detailed empirical analysis (see Figure 1, Page 2) and system profiling, the authors convincingly highlight that backward propagation and softmax-based self-attention present severe efficiency barriers in MPC-based LLM fine-tuning.\n\n2. SecP-Tuning demonstrates significant improvements in speed and communication overhead.\n\n3. The framework operates under a black-box paradigm. This allows a data owner to perform tuning without the model developer ever receiving the updated parameters. This prevents the developer from potentially inferring private fine-tuning data from model updates, a risk present in gradient-based methods.", "weaknesses": "1. The optimizer used in the paper is Adam, which is different from the most popular optimizer, AdamW, in the LLM domain. Can the authors explain the reason? Besides, how do the authors choose the hyperparameters for the baselines? Does the few-shot learning with 1000 epochs cause overfitting?\n\n2. The empirical validation is performed on small-scale settings. Though it is also a valuable setting, including large-scale settings could make the reader better understand the performance of SecP-Tuning. Besides, do the authors consider comparing with In-context learning that might also be good for small-scale settings? Some ICL methods are designed for privacy [1,2].\n\n3. Experiments focus mainly on RoBERTa and text classification tasks. \n\n[1] Tang, Xinyu, et al. \"Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation.\" The Twelfth International Conference on Learning Representations.\n\n[2] Wu, Tong, et al. \"Privacy-Preserving In-Context Learning for Large Language Models.\" The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wB9sn9ul5z", "reviewer": "Reviewer_LnHB", "summary": "This paper aims to address the high communication and computational cost of fine-tuning large language models on private data using Secure MPC. Secure fine-tuning of LLMs through MPC is slow due to the complex, back-and-forth computations needed for the backward pass and the standard attention mechanism. The paper proposes SecP-Tuning, a framework that tries to solve both problems. It uses a *Forward-only Tuning* approach where the secure servers only run the model forward, while the private data owner handles the (now much simpler) tuning steps locally. This completely removes the need for a secure backward pass. It replaces the costly standard attention with a highly efficient variant called Random Feature Attention (RFA), which reduces attention's complexity from quadratic to linear.", "questions": "1. Could the authors further explain the motivation for choosing RFA over other linear attention variations?\n2. The performance of the gradient-based Prompt Tuning baseline is surprisingly low for simple tasks like SST. What are the settings here?\n3. Regarding CMA-ES, would it still be as effective when the prompt length or the model's hidden dimension increases significantly, requiring a much larger search space?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written with a clear introduction of the problem and the room for their contributions. Despite various solutions in private LLM adaptation, Secure MPC is still highly desirable due to its formal guarantees. The authors clearly identify the gaps in the communication and computational overheads in Secure MPC. The figures clearly explain the problem and proposed solutions to non-subject-matter experts.", "weaknesses": "1. The experiments are conducted exclusively on ROBERTA, an encoder-only LLM. In 2025, the field is heavily focused on autoregressive  LLMs. It is unclear if the method, and particularly its strong accuracy results, will translate to these architectures. However, I do recognize that ROBERTA is still a strong model for many downstream NLP tasks. Being able to effectively perform Secure MPC is still a significant contribution.\n2. The paper omits a comparison to non-secure, plaintext baselines in its result presentation. Including plaintext accuracy is essential to help readers understand the \"utility cost\" of privacy (if any). It would also help contextualize the gap that privacy-preserving methods still need to close.\n3. Related to the first point, the reliance on a subset of GLUE-style classification tasks (SST-2, MRPC, RTE, etc.) is quite limited. While fine for proving the concept on classification, these benchmarks do not capture the full range of capabilities and tuning challenges of modern LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ZKYWXite0", "reviewer": "Reviewer_qxro", "summary": "SecP-Tuning is an MPC-based framework for efficient and privacy-preserving prompt tuning of large language models. It integrates Forward-only Tuning to avoid costly backward computations and introduces Random Feature Attention to reduce self-attention complexity. Experiments show up to 16× faster training, 20× lower communication cost, and better few-shot performance, achieving a strong balance between efficiency, accuracy, and privacy.", "questions": "1. In this paper, the performance of SFT is shown to be worse than that of the gradient-free training method. What is the reason for this phenomenon? Does it also occur in larger models? Could SFT be overfitting? The authors are encouraged to provide the relevant hyperparameters and loss curves to support their claim. Normally, such a result is quite unlikely, as gradient-based methods generally outperform gradient-free ones due to their much smaller update variance. If this outcome is indeed caused by overfitting, the authors should re-evaluate the SFT performance. If not, could the authors explain why the gradient-free method achieves better optimization results?\n\n2. Can the authors independently evaluate the privacy protection strength of each part in the 2-out-of-2 module, which splits the original parameters into two components? If an attacker gains access to only one part and attempts to infer or guess the other, what would be the resulting degree of privacy leakage?\n\n3. Could the authors provide a detailed horizontal comparison with the four works I mentioned above, as well as several other important studies on black-box prompt-tuning methods? I would like to understand what additional advantages this approach offers beyond the attention acceleration component.\n\n4. Could the authors conduct additional experiments on larger models, such as 7B or 13B? Gradient-free methods can be efficiently supported on A100 GPUs, allowing for training even very large models.\n\nI will re-score according to the rebuttal discussions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It addresses the high computational cost of attention during fine-tuning, significantly accelerating the overall training process.\n\n2. 2-OUT-OF-2 module appears intriguing, as it separates and isolates the model’s information-capturing capability.\n\n3. It adopts gradient-free optimization, improving the efficiency of forward computation.", "weaknesses": "1. The paper devotes a large portion of the Preliminary section to background explanations, while the introduction of its own method is relatively unclear, making it difficult to follow during reading.\n\n2. Optimizing LLMs through gradient-free black-box methods is an outdated approach, and applying it to prompt-tuning tasks is not particularly novel. Its applicability within LLM scenarios remains quite limited. It also lacks horizontal comparisons with several key related works, such as the following studies.\n\nDiao, Shizhe, et al. \"Black-box prompt learning for pre-trained language models.\" arXiv preprint arXiv:2201.08531 (2022).\n\nLin, Zihao, et al. \"Efficient federated prompt tuning for black-box large pre-trained models.\" arXiv preprint arXiv:2310.03123 (2023).\n\nSun, Jingwei, et al. \"Fedbpt: Efficient federated black-box prompt tuning for large language models.\" arXiv preprint arXiv:2310.01467 (2023).\n\nZheng, Yuanhang, et al. \"Black-box prompt tuning with subspace learning.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (2024): 3002-3013.\n\nAll these approaches design and improve gradient-free prompt-tuning optimizers from different perspectives. The authors should further discuss how their proposed method compares with these works in terms of advantages and limitations.\n\n3. There is too little theoretical analysis, especially regarding the privacy protection performance of Module 2-OUT-OF-2, which lacks strong guarantees. Providing more analysis or evaluation would further strengthen the contribution of this paper.\n\n4. The experiments use only RoBERTa as the model, which is too small in scale. As a result, the stability and scalability of the experimental results are difficult to guarantee.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u5iGHHbByU", "reviewer": "Reviewer_4KeS", "summary": "This paper presents SecP-Tuning, an MPC-based framework for privacy-preserving prompt tuning of large language models (LLMs). The authors aim to address the prohibitive computation and communication overheads of privacy-preserving fine-tuning (PFT) under Secure Multi-Party Computation. They combine two main techniques: (1) Forward-only Tuning (FoT) with a gradient-free optimizer (CMA-ES) to avoid backward propagation, and (2) Random Feature Attention (RFA) with a privacy-preserving cosine protocol to replace softmax attention. Experimental results on five NLP datasets show that SecP-Tuning achieves substantial efficiency gains  while maintaining or improving performance compared to both supervised fine-tuning (SFT) and gradient-based prompt tuning.\n\nOverall, the paper proposes an interesting and timely attempt to make privacy-preserving tuning of LLMs feasible, though several technical details and justifications require clarification and deeper analysis.", "questions": "1. **On optimizer selection**: Why was CMA-ES chosen as the gradient-free optimizer? Have the authors evaluated other gradient-free methods (e.g., random search, NES, Bayesian optimization), and what is the empirical or theoretical rationale for preferring CMA-ES?\n\n2. **On performance explanation**: The results indicate that SecP-Tuning not only matches but outperforms SFT, which is counterintuitive given that gradient-free methods generally converge slower and achieve lower accuracy. Could the authors clarify the mechanism behind this improvement?\n\n3. **On abbreviation**: What does DFP in line 299 stand for? The abbreviation should be defined upon first use for clarity.\n\n4. **On the statement in line 437**: The paper claims that “gradient-based methods such as SFT and prompt tuning inherently require the model developer to obtain shares of the updated parameters.” However, prompt tuning typically updates only prefix embeddings. Could the authors clarify why such updates would require parameter sharing under MPC?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Novel integration of privacy and efficiency**. The paper successfully combines MPC with prompt tuning through forward-only optimization, reducing both computation and communication costs while maintaining data confidentiality.\n\n2. **Technical contribution in protocol design**. The development of the privacy-preserving cosine protocol and its integration into the Random Feature Attention module is well-motivated and technically sound, showing potential for broader cryptographic ML applications.\n\n3. **Comprehensive experiments**. The authors evaluate both efficiency and accuracy across multiple datasets and network conditions (LAN/WAN), offering convincing empirical evidence of the method’s efficiency gains.", "weaknesses": "1. **Limited originality in efficiency gain sources**. The claimed reduction in communication and computation largely stems from existing components—namely, gradient-free optimization (FoT) and RFA—rather than innovations specific to the MPC algorithmic design.\n\n2. **Scalability concerns**. The framework is only demonstrated under a 2-party MPC setup (semi-honest model). It remains unclear how it scales to more realistic multi-party or malicious adversarial settings, which are common in real-world PPML applications.\n\n3. **Lack of theoretical justification**. The paper provides no formal convergence or performance guarantees for the CMA-ES optimizer in this MPC context, leaving uncertainty about stability and generalization.\n\n4. **Insufficient baselines**. Comparisons exclude recent privacy-preserving fine-tuning methods based on differential privacy, homomorphic encryption, or hybrid PPML frameworks, which would help situate the contribution more clearly.\n\n5. **Opaque optimizer choice**. The paper does not analyze why CMA-ES is particularly effective, nor how it compares to other gradient-free optimizers such as NES or SPSA. This weakens the empirical claims regarding its superiority.\n\n6. **Overstated performance improvements**. The claim that a gradient-free, forward-only approach surpasses both SFT and gradient-based prompt tuning contradicts common understanding and demands stronger justification or ablation evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)