Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
Authors: Xun Wang
Jing Xu
Franziska Boenisch
Michael Backes
Christopher A. Choquette-Choo
Adam Dziedzic
Venue: ICML 2025 poster (2025)
Keywords: prompt transfer
soft prompt
privacy
distillation
confidentiality
Primary area: social_aspects->safety
TL;DR: We propose a method for efficient and privacy-preserving transfer of soft prompts tuned on a distilled small model to a larger model using public data.

Abstract: Prompting has become a dominant paradigm for adapting large language models (LLMs).
While discrete (textual) prompts are widely used for their interpretability, soft (parameter) prompts have recently gained traction in APIs. This is because they can encode information from more training samples while minimizing the user's token usage, leaving more space in the context window for task-specific input. However, soft prompts are tightly coupled to the LLM they are tuned on, limiting their generalization to other LLMs. This constraint is particularly problematic for *efficiency* and *privacy*: (1) tuning prompts on each LLM incurs high computational costs, especially as LLMs continue to grow in size. Additionally, (2) when the LLM is hosted externally, soft prompt tuning often requires sharing private data with the LLM provider. For instance, this is the case with the NVIDIA NeMo API.
To address these issues, we propose POST (**P**rivacy **O**f **S**oft prompt **T**ransfer), a framework that enables private tuning of soft prompts on a small model and subsequently transfers these prompts to a larger LLM.
POST uses knowledge distillation to derive a small model directly from the large LLM to improve prompt transferability, tunes the soft prompt locally, optionally with differential privacy guarantees, and transfers it back to the larger LLM using a small public dataset. Our experiments show that POST reduces computational costs, preserves privacy, and effectively transfers high-utility soft prompts.

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
Xun Wang' Jing Xu'
Franziska Boenisch
Michael Backes' Adam Dziedzic!
Christopher A. Choquette-Chod
Abstract
1. Introduction
Large Language Models (LLMs) are strong general-purpose language generators that can be adapted to solve various private downstream tasks (OpenAl 2023 Giemini-Team et al. 2023). One prominent paradigm for adapting LLMs t0 private tasks prompting (Devlin al. 2018 Radford ct al. 2018).. Prompting is more flexible alternative to fine- tuning, as it allows multiple tasks to be solved simultane- ously in single inference pass, leading to higher efficiency and reducing the need for storing multiple fine-tumed model copies. While discrete prompts (Schick & Schütze 2020 2021a Shin et al.) 2020 Han et al., 2022). which prepend textual tokens the LLM's input, have been shown rela- tively successful for LLM adaptations, they require large engineering efforts and lots of trials and errors. As an alter-
Prompting has become dominant paradigm for adapting large language models (LLMs). While discrete (textual) prompts are widely used for their interpretabilityy soft (parameter) prompts have re- cently gained traction in APIs This is because they can encode information from more training samples while minimizing the user's token us- age, leaving more space in the context window for task-specific input. However, soft prompts are tightly coupled to the LLM they are tuned on, limiting their generalization to other LLMs. This constraint is particularly problematic for ef- ficiency and privacy: (1) tuning prompts on each LLM incurs high computational costs, especially LLMs continue grow in size. Addition- ally, (2) when the LLM is hosted externally, soft prompt tuning often requires sharing private data with the LLM provider. For instance, this is the case with the NVIDIA NeMo API. To address these issues, we propose POST (Privacy Of Soft- prompt Transfer), a framework that enables pri- vate tuning of soft prompts on a small model and subsequently transfers these prompts a larger LLM. POST uses knowledge distillation to derive small model directly from the large L.M to improve prompt transferability, tunes the soft prompt locally- optionllly with differential privacy guarantees
native, soft prompts (Shin et al. 22200] Lester et al.
2021
Li & Liang 2021 Zhong et al. 2021 Oymak et al.
2023
prepend trainable embedding vectors to the input, which can be tuned automatically on the private downstream data using standard gradient-based approaches. Such gradient- approaches are generally known to yield higher performance at lower computational costs (Liu et al. 2022) Additionally, soft prompts can encode information from many more pri- vate downstream examples while using fewer input tokens than discrete prompts. This reduces costs-since many APIs charge per token-and leaves more of the context window for the downstream task (Varshney & Surla 2023).
Yet, soft prompt tuning has two major limitations. 1) As LLMS grow in size (Geng & Liu 2023 Chiang et al. 2023 Brown et al. 2020), it requires significantly more compu- tation to backpropagate through the entire LLM. 2) Back- propagation requires the model and data to be on the same device. For centrally hosted LLMs, this leaves two options: either the user needs to share their private data with the LLM provider, or the LLM provider needs to share their LLM with the user. The latter is impractical as users usually do not have the compute to deploy an LLM. Additionally, the model is the provider's intellectual property (Dziedzic ct al. 2022a blc, Dubiński et al. 2023) and should be pro- tected. Thus, sharing it would, at the very least, disrupt the provider's 's business model, as users would no longer be required to pay per query. Therefore, the former, i.e., users sharing their data with the LLM provider, is implemented
transfers back to the larger LLM using small public dataset. Our ex- periments show that POST reduces computational costs, preserves privacy, and effectively transfers high-utility soft prompts. Our code is available at htttp:://gituu..oom//
ICISPA Helmholtz Center for Information Security, Saarbrücken, Germany GGoogle DeepMind. Correspondence to: Franziska Bocnisch <bocnischee @ dis,, Adam Dziedzic <adam,. <addm.dziedzic @ cispa.de>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Efficient and Privacy-Preser Soft Prompt Transfer for LLMs
4
Embedding Layer
Knowledge Distillation
Embedding Layer
Transformer
Transformer
LLM Head
LLM Head
LLM Provider
Private Data Owner
Diferention Privacy
2 Prompt
Soft Prompt Transfer
+
Tuning
D private
Soft Prompt
pubbie
Figure 1: POSTR Framework. An L.L provider compresses their L.M +; into a smaller model +. through knowledge distillation. The private data owner learns a specific soft prompt P. on 4, using their private dataset (optionally with differential privacy guarantees). The LLM provider obtains the soft prompt ps for solving the user's 's task by transferring Ps to the target L.M Φ solely relying on a small public dataset and no access to the private data for transfer.
in popular APIs, such as NVIDIA's Nemo API Varshney & Surla 2023). However, this approach causes severe privacy concerns for the users when the data is private sensitive Hanke et al. 2024).
2006). Once the user has tuned the private prompt on their data, they provide this prompt the LLM provider, who (3) transfers the prompt to achieve strong performance on the large LLM. To perform this transfer without any additional leakage from the user's private data to the LLM provider, we equip POST with a novel prompt transfer method that relics purely on access to a small public dataset during transfer rather than the user's private data.
in nature (Duan et al. 2023a b
A potential solution to both problems is to tune the soft prompt locally on a smaller model and then transfer and use on the large LLM. This approach, commonly known as "prompt transfer" (Su et al.
2022 Wu et al. 2023 Xiao
al. 2023). has proven effective for discrete prompts (Rako-
Our thorough experimental evaluation both masked lan- guage models and auto-regressive language models demon- strates that our method can efficiently, effectively, and pri- vately transfer soft prompts with high utility. Thus, POST has the potential to enable more LLM providers to integrate soft prompts into their APIs, unlocking benefits such as multi-task inference, improved adaptability, extended con- text windows for users all while maintaining user privacy.
tonirina et al. 2023 Hong et al. 2023 Wen et al. 2023
However, soft prompts are highly coupled to the LLM they were tuned on, making them difficult to transfer. Therefore, existing attempts to transfer soft prompts between L.LMs face severe limitations: either they require private data ac- cess for the LLM provider during transfer (Su al., [022)) which- as we discussed- raises privacy concerns, they are ineffective, as the transferred prompts' utility on the large central LIM often underperforms even compared to the prompted small model (Wu et al. 2023), disincentivizing the transfer altogether.
In summary, we make the following contributions:
We propose POST, a framework for privacy of soft prompt transfer. POST preserves the confidentiality of users' pri- vate data and can also provide strong privacy guarantees through differential privacy.
Our proposed POST (Privacy Of Soft-prompt Transfer) framework overcomes these limitations. POST consists of three key steps shown in Figure (1) The LLM provider performs knowledge distillation (Hinton et al.) 2015) to com- press their L.M into smaller language model. Note that this is a one-time operation, and the same small model can be used by any user for their various tasks. After distillation, the L.M provider sends the small model to the user. Then, (2) the user performs local prompt tuning using their private data on this smaller model, potentially incorporating formal privacy guarantees through diffferential privacy (Dwork et al.
We design a novel method to transfer private prompts between LLMs by purely relying on public data which we integrate into POST.
We provide detailed experimental analysis using five clas- sification datasets and two open-ended generation tasks to simulate our setup and three different types of LLMs to show the effectiveness and efficiency of our method.
2

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
2. Background
more challenging scenario for prompt transfer is cross- model transfer scenario. In this scenario, one tries to tune prompt for given task on one LLM, and then use it for another LLM. The difficulty arises from the fact that soft prompts (over)fit the LLM they were tuned for and usually do not exhibit a strong performance on other LLMs. [Su et al. (2022) address transferring the soft prompt between the LLMs by using the guidance of the private data. However, this approach still exposes the data directly to the second LLM which may not be possible when this LLM is hosted centrally by a service provider (e.g., ApenAl) and the data is sensitive in nature, as the private data now needs to be shared with the external provider. |uu et al. (2023) present a zero-shot prompt transfer method, where source prompts tuned on given LLM are encoded into a relative space and used as a form of support vector when finding target prompts on the second, i.e. target model. Unfortunately, in their approach, the target model with the transferred prompt performs worse than the source model, leaving no incentive to use the target model rather than the source model with the prompt. In contrast, our method significantly improves per- formance on the target models with the transferred prompts. Additionally, their transfer requires the private data, which thereby leaks entirely to the model owner. In contrast, our method performs prompt transfer solely with public data, preserving confidentiality of the private data towards the model provider. Transferring tasks between LLMs has also been explored by Xiao et al. (2023) for transfer learning. While they focus on fine-tuning and their approach is not applicable for soft-prompt tuning, we operate in the same setup and under the same assumptions as they do.
Prompt Tuning (PT) aims at adapting a publicly pre- trained LLM to various natural language downstream tasks. There are two major types of prompts. 1) hard or discrete
prompts (Schick & Schütze 2021a b Gao et al.] 2021), which are discrete textual tokens prepended to the input text of the LLM, and 2) soft prompts (Hambardzumyan et al. 2021; Qin & Eisner [021 Zhong et al. (021). which are tunable embedding vectors prepended to the LLM's input. While discrete prompts require thorough engineering to yield good performance on downstream tasks, soft prompts can be tuned through standard gradient- based training ap- proaches (Lester et al. 2021). Formally, given input sequence with n tokens X {01, [21.F2,... .r,,..., }, labeled by y, we first prepend / randomly initialized soft prompts P= ( Pp.....P/ before X, where P. E Rd is an em- bedding vector, and d is the input dimension of the I.I.M. The training objective is to maximize the likelihood of de- coding the correct output y as L = p(y] P, X). The key is that the model itself remains frozen and only P is tunable.
Knowledge Distillation (KD) (Hinton et al. 2015 Bucilua et al. 2006) is a compression method for machine learning models. It works by transferring knowledge from a complex model, denoted as the teacher model, to a simpler, smaller model known as the student model. KD has been shown effective in compressing L.MMs during the pre-training phase while maintaining their performance (Sanh et al. 2019 Gu
eet al., 2024: Sreenivas et al. 2024). Already pre-trained
LLMs can also be compressed successfully through KD (Gu
et al. 2024 Wang et al., 2025). In prompt transfer. Zhong jet al. (2024) leverage knowledge distillation to alleviate forgetting between tasks that use transferred prompts. In contrast, our setup considers transferring prompts between models. While, in general, most of the KD approaches aim at generating a student with a similar predictive performance as the teacher, for our purpose, the student's performance is not particularly relevant. The student just needs to match the teacher's predictive behavior to a certain degree in order to facilitate the transfer of the prompt from student to teacher.
Differential Privacy (DP) (Dwork 2006 is mathemat- ical framework that provides privacy guarantees for ML by implementing the intuition that a model M : - S, trained on two neighboring datasets D. D' that differ in only one data point, will yield roughly the same output, i.e., Pr(M(D) E S] S e PrD)( E S] 9]+6 8. The privacy parameter specifies by how much the output is allowed to differ, and 8 is the probability of failure to meet that guarantee. To adapt soft prompts under differential privacy (DP) guarantees, Duan et al. (2023a) introduced the PromptDPSGD algorithm. Initially developed for text classification, PromptDPSGD was subsequently adapted to support text generation tasks as well (Hanke et al., 2024)). The algorithm builds on the widely used Differentialyy Pri- vate Stochastic Gradient Descent (DPSGD) method (Abadi et al. 2016), in which each gradient is clipped to bound its sensitivity (to limit how much each data point influences the trained parameters), and calibrated Gaussian noise is added to the aggregated gradients to ensure finite DP guarantee.
Soft Prompt Transfer can be computationally expensive, especially LLMs grow in size, because soft prompts are trained with backpropagation through the entire LLMs. This motivates the emergence of attempts to transfer, i.e., to reuse, existing soft prompts. There are two broad scenarios for prompt transfer. The first one aims at reusing a soft prompt trained for one downstream task on another (similar) down- stream task on the same LLM (cross-task transfer). This can be implemented, for example, by initializing the param- eters of the second soft prompt with the trained existing soft prompt parameters and has been shown to reduce training
Private Prompting and Text-to- Text Privatization There exist multiple DP frameworks for private prompting (Duan et al.
time for the second prompt Vuu et al.
2022 Su et af. 2022 Zhong et al. 2024 Philippy et al. 2024). The second and
2023b
Tang et al.) 2024 Wu et al. 2024). However,
3

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
they mainly operate in a different setup and only provide DP guarantees for the model ouiput, yet leak the private data to the model provider (Hanke al, 2024). In contrast, our work aims at protecting the private data against the model provider. In a similar vein to our work, DP-OPT (Hong et al. 2023) tries to avoid leakage to the model owner and tunes discrete prompts with DP guarantees on a local sur- rogate LLM and then transfers these prompts to the large LLM. Their focus on discrete prompts (in contrast our work that relies on soft prompts) leads to certain words and phrases from the training dataset leaking directly to the LLM provider, as shown in their Figure 3-which does not occur with our method. Additionally, their results show that the local surrogate model needs to be of strong performance (i.e., large) to obtain good transfer results, leading to very high compute requirements on the user side, In contrast, our method relies small surrogate models that can be used for prompt tuning with low computing on the user's end. We empirically compare against their method in Table 5
public data. To obtain a suitable small model that facilitates prompt transfer, we propose that the LLM provider performs KD from their LLM. The small model needs to meet three critical requirements: it must (i) be small enough to enable the user to perform local soft prompt tuning on their own hardware, (ii) closely match the semantics of the original LLM to facilitate an effective prompt transfer, and, from the perspective of the LLM provider, (ii) be limited in performance to ensure that users still have an incentive to use the original LLM rather than the small prompted version. Note that the KD is one-time operation for the LLM provider as the same small model can be used by any user for their various tasks. After distillation, the small model is sent to the user who tunes a soft prompt on it using their private data, potentially with DP to formally bound privacy leakage. Finally, the tuned prompt is sent to the LLM provider who performs prompt transfer step relying on public data. Eventually, the client can use the LLM using the transferred prompt. We provide an overview of this solution in Figure [ and detail its building blocks in the following section.
3. Setup and Problem Formulation
The Setup. We consider two parties: an LLM provider and a user. The LLM provider deploys general-purpose
4. Our Private Transfer of Soft Prompts Framework
LLM Provider
LLM
purpose LLM and of- fers paid query access to it. The user holds private data and wants to adapt the LLM on this data to solve their downstream tasks while ensuring the confidentiality and privacy of the data towards the LLM provider.
Our Privacy Of Soft-prompt Transfer (POST) framework consists of three main building blocks: (1) a KD from the LLM small model, (2) private prompt tuning, and (3) a prompt transfer using public data. We detail those building blocks below.
Private Data
User
4.1. Knowledge Distillation
Unfortunately, for soft prompt tuning which involves gradient balculation-both the data and the LLM are required to be on the same device, The problem that the user may not be able to share their data with the LLM provider due to privacy concerns while the LLM provider cannot share their LLM because of 1) intellectual property concerns and since 2) this would disrupt their busi- ness model, as users would no longer be required to pay for accessing model queries. Additionally, most users would lack the necessary computational resources to tune the soft prompt on the large L.MM locally, as this requires calculating gradients over the entire model. Due these limitations, the powerful LLM cannot be used for private tasks.
The Problem.
To derive small model b. from the large LLM o. we rely on KD. We discuss the advantage of KD for our POST framework over other alternatives for model compression, such as quantization and pruning, in Appendix D.2
We follow the approach of Sanh et al. (2019) to derive $2 from p.. Their distillation objective function is given by
Edistil Cce + Cim + Gccos Ccos-
(1)
This objective combines three losses: distillation loss Loes language modeling loss Lim, and embedding cosine loss Leos- The distillation loss Lee measures the Kullback- Leibler divergence between the logits of o and p,. The language modeling loss Lim corresponds to the standard pretraining objective, using cross-entropy to predict masked or next tokens. The embedding cosine loss Leos computes the cosine distance between the embeddings of D. and o;, These three losses are weighted by Oceer Ollm, and acos-
Our Goal and Solution. Our goal is to protect the private downstream data (i.e., the "training data") used to tune the [] We propose
soft prompt from leaking to the LLM provider] a solution based on tuning the soft prompt on a small local model and then transferring this prompt to the LLM by using
Protecting the queries (Le., the "test data") from leaking to the LLM provider, often referred to as private inference, is orthogonal and out of scope for this work.
In contrast to prior work (Sanh et al., 2019 Xiao et al., 2023) that relies on KD to moderately compress LLMs while maintaining high performance on t., our goal is to achieve
4

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
significant compression without focus on maintaining & performance. Thereby, we ensure that Φ, is small enough to fit into the user's hardware for local prompt tuning and is not performant enough to disincentivise the (paid) use of #, as desired by the LLM provider. We perform extensive ablations on the setup for the KD, the importance of the individual loss term in Equation (T), and the resulting trade- offss between the compressed model's 's performance and our POST's success in Section 5.6
tive update, we design a loss function
L= (1 (1) C1 + aL2,
that consists of two different loss terms. The first loss term aims at mimicking the prompted small model's predictions on the public data closely. It is defined as
C1
Σ KLDiv + +)), (P. (4)
SE DEDous
4.2. Private Prompt Tuning
where KLDiv denotes the Kullback-Leibler divergence. The second loss term aims at replicating the changes to the initial model behavior introduced by the prompt. Formally, the second loss aligns the direction change induced by the private prompt between b and o., on the public data
The goal is tune a local prompt p, on the small source model $ using the private data Pari such that ps minimizes the loss L on the private downstream task as
arg min
Σ CC + 2)
Ps
DEDprt
L2
Σ DEDpub
(())))())())))))) (0. (5) (.)). (e))),,
where x denotes a query input sequence from the task that we want to predict. This approach can be performed with standard PT. provides confidentiality for the private data since the data is not directly sent to the LLM provider. Re- cent work (Duan et al. 2023b), however, highlights that private information can leak from tuned discrete prompts. In Section (.4] we show that this also holds for soft prompts, motivating the necessity of integrating privacy guarantees.
(5) We need both loss terms, since, particularly when the small model does not have good performance, just mimicking its behavior, rather than matching the impact of the prompt, is not effective.
Based on the design of our loss function, the best choice for in Equation 3 --controlling the balance between the two loss terms depends on the zero-shot accuracy of the large LLM on the private task and the performance of the compressed LLM. If the large L.M has a good zero-shot performance, or the compressed LLM has a weak perfor- mance, a larger a, i.e., putting the emphasis L2 to mimic the prompt behavior, rather than improving the task perfor- mance, is more beneficial. We present extensive ablations and guidelines on the choice of a in Section 5.6
To formally bound privacy leakage, we propose to tune Ps with DP, for example, using the PromptDPSGD algo- rithm (Duan et al. 2023a). During optimization, Prompt- DPSGD clips the per-sample gradients of the loss to clip norm e and adds Gaussian noise drawn from (0, o²c²1) to provide (s, 6D- guarantees (see Appendix C.3 in Duan ett al. (2023a) for the privacy proof). Hence, applying their algorithm bounds the leakage of the private prompt tun- ing data Dpri from P. to (s, 5). Because of the DP post- processing guarantees, Ps can then be shared with the LLM provider without incurring further privacy loss.
5. Empirical Evaluation
5.1. Experimental Setup
Models and Distillation. We perform experiments with three LLMs of diverse scale, namely Roberta-bass (Liu) al. 2019), GPT2-XL (Radford et al., 2019), and Llama2-
4.3. Privacy-Preeerving Prompt Transfer through Public Data
The prompt Ps: tuned the small source model T.., could, in principle, be directly applied to the large target LLM Φ, However, as described above, soft prompts fit very strongly the model that they were tuned on. Hence, they do not initially perform very well on other LLMs. A naive solution to fine-tune the target prompt pt on the private data Drri- However, this would disclose the private data to the LLM provider and is, hence, not acceptable. As alternative, we propose the first privacy-preservin leverages a small public dataset Prus in an efficient transfer step to derive a high-utility prompt Pt. from p..
7b Touvron et al.) 2023). We experiment with various degrees of compression in the KD (see Appendix D.4). For the results presented in the main body of the paper, we com- press the 12-layer Roberta-base and the 32-layer Llama2-7b to 2 layers, and the 48-layer GPT2-XL to 4 layers. We use the Bookcorpus (Zuu et al. 2015) dataset for the KD.
Datasets. Following prior work (Hong et al. 2023 Wu et al. 2023), we evaluate the performance of our proposed method on five classification-aass datasets: sst2 from the GLUE benchmark (Wang et al. [211)). imdb (Maas et al.) 2011). tweet (Rosenthal et al. 2017). arisetv (Okite) 2022) and mpqa (Wiebe et al. 2005). To show the general ap- plicability of our method on open-ended generation tasks,
prompt transfer that
We start by initializing the target prompt Pt with the same initialization of Ps, then iteratively update pt. For the itera-
5

---

Efficient and Privacy-Preservingg Soft Prompt Transfer for LLMs
we go beyond what was shown in prior work, and addi- tionally perform evaluations on the MIT dataset (Liu et al. 2012), which aims to extract the genre or director of a movie given the movie description. This task requires the genera- tion of multiple tokens with varying lengths, highlighting our POST's flexibility. As public data, we also include ag- news (Zhang et al. 2015) and boolq (Clark et al., 2019) for the classification task, while for the generation task, we use AIE (Kudari 2022). We also include disaster (CrowdFlower 2019) and trec (Li & Roth 2002) for baseline comparison and ablation on the choice of public data. Further details on the datasets and their tasks are provided in Appendix [..2
two different public datasets for prompt transfer and report the respective transferred accuracy the private dataset. We first observe that the transferred performance is signif- icantly higher than the zero-shot performance of the large LLM (Full 7 This highlights the improvements that a user can gain from adapting the large LLM to their private task using our method. Additionally, after prompt transfer with POST, we outperform the small compressed prompted model (Compressed PT), giving users strong incentive to transfer their prompt back to the (paid) large LLM, rather than using the small model locally. Further, we show that our prompt transfer described in Section 4.3 is highly ef- fective as it improves over the direct transfer (Direct Trans- fer) performance by a large margin. In contrast to the soft prompt transfer method by /uu et al. (2023) which showed decrease in accuracy after transfer, our results highlight for the first time the practical applicability and the benefits of transferring soft prompts.
KD, Prompt Tuning, and Prompt Transfer. We experi- ment with various setups for the KD. not indicated oth- erwise, we use the KD hyperparameterss from |annh et al. (2019). See Appendix[B.]]]
more details. Following Su et al. 2022), we initialize our soft prompts with 100 to- kens. We include an ablation study on prompt length in Section 5.6 The hyperparameters for prompt tuning per dataset, including the 5 for the DP setup, are presented in Table [T] During the prompt transfer, the model provider has no access to the private dataset find the right moment to stop the transfer, we report the transferred accuracy after a fixed number of steps. We experiment with various numbers of steps and public data points used for transfer in Section 5.6 By default, we use 5000 steps for Roberta-base, 8000 steps for GPT2-XL, and 6000 steps for Llama2-7b.
When using DP during the prompt tuning, we observe that the improvement of the transfer performance to the large L.M over the performance on the prompted compressed model (Compressed PT) even more significant than in the non-DP setup. For example, on the sst2 dataset, using imdb public data, we observe an improvement of 18.92% for the DP case, while we only have an improvement of 11. 11.44% in the non-DP case (see first lines of Table [b and Table Ia respectively). We hypothesize that the noise added for DP during tuning acts as a regularizer that can help to prevent overfitting on the small sensitive datasets and the distilled model, hence, generalizing better to the large L.M.M Overall, our results are very close to the non-private upper bound baseline (Full PT) of tuning the prompt directly on the large LLM, which would not only leak private information from the model outputs to third parties querying the adapted model, but also leak the private dataset to the LLM provider.
Metrics and Baselines. To evaluate the success of our method, we report the accuracy on the test data split of our private datasets for the teacher LLM with the transferred prompt (Private Transfer). As baselines for comparison, we include the zero-shot performance of the teacher LLM on the private tasks' test sets (Full ZS), representing the lower bound our method should improve upon. Additionally, we provide the performance of tuning the prompt for the teacher LLM on the private training data, which, due to privacy con- cerns, is not feasible in practice (Full PT). This serves as the theoretical upper bound for potential performance. We also report the accuracy of the prompted compressed model after tuning the prompt on it (Compressed PT), as our private transfer must improve over this metric to justify using the teacher L.M instead of the small student model. Finally, we report the direct transfer accuracy (Direct Transfer), which is the accuracy achieved when the prompt tuned on the small model is directly applied to the large one. The im- provement of POST over this value shows the effectiveness of our prompt transfer step.
5.3. Transferring Prompts for Generation Tasks
Since, following Li et al. (2022). we model classification tasks as text-infilling tasks rather than adding a classifica- tion head, as detailled in Appendix [B.2) our POST method naturally extends generation tasks. To demonstrate this ability in practice we assess POST's performance on the MIT movie dataset where the task is to extract movie's director or genre from a given movie description. Instead of generating a single token, as for classification, this open- ended task requires generating multiple tokens of varying lengths. We choose AIE, which aims to extract location from a note, as public dataset and present our results in Table 2 POST outperforms Full ZS and Compressed PT, highlighting our method's 's applicability to open-ended tasks.
5.2. Effective Prompt Transfer with POST
As the main result, in Table 1 we show POST's effective- ness in transferring soft prompts in both setups, with and without DP. For each private dataset, we experiment with
6

---

Efficient and Privacy-Preserviin Soft Prompt Transfer for LLMs
Table 1: POST on Llama2-7b with and without DP. We report POST's transfer performance. The baselines are the large models' zero-shot performance on the private data (Full ZS), the accuracy of tuning the prompt with the private data on the large models (Full PT) and the small model (Compressed PT), and the performance of the prompt tuned on the small model when direly applied to the large one (Direct Transfer). POST significantly improves performance over the small prompted model and our prompt transfer yields strong improvement over the direct transfer. The same improvement can be observed for Roberta-base and GPT2-X0 1P,,-XL,&a as shown in Table 13 and Table 14 in the Appendix.
POST (ours)
Private
Full 7S
Full PT
Compressed PT
Direct Transfer
Public
Test
Public
Test
ss12
78.67
94.84
78.78
55.28
tweet
89.33
imdb
90.02
tweet
44.50
72.03
54.12
41,70
imdh
57.55
sa12
61.15
arisetv
76.57
93.47
77.92
54,23
agnews
86.71
tweel 79.59
mpga
53.11
92.36
83,82
32.96
ss12
87.37
boolg
76.18
(a) Confidential POST transfer (no DP).
POST (ours)
Private
Full ZS
Full PT
Compressed PT
Direct Transfer
Public
Test ace
Public
Test
ss12
78.67
90.60
70.99
53.55
tweet
87,50
imdb
89.91
tweet
44.50
62.40
48.16
41.65
imdb
56.60
ss12
59,55
arisety
76.57
83.73
agnews
tweet 75.24
mpga
$3.11
85.95
68.87
33,28
sst2
68.88
boolg
80.17
(b) Private POST transfer (with DP: E = 8, and ó according to the dataset size as specified in Table 10
Table 2: POST on Generation Tasks (Llama2-7b, no DP). We present POST's performance on the open-ended MIT-D and MIT-G tasks and choose the ATE dataset as the public set. We show its effectiveness over the baselines.
Table 3: LiRA Membership Inference Attack. We report the results of the LiRA attack against soft prompt trained four datasets with Roberta-base. Without DP (e x)), we observe non-trivial leakage. Our method with DP re- duces privacy leakage significantly in comparison to the non-private (confidential, E = 20) prompts.
Private
Pull PT
Compressad PT Direct Transfer
Public
Test Acc (POST)
MIL-D
70.4
92.28
21.09
43.61
AlE
75,66
GIT-G
51.28
89.35
51.92
50,51
AIE
61.41
5.4. Practical Privacy Implications of POST
00
00
AUC 0.5610
0.5258
0.5796
0.5524
0.5269
0,5006
0.5516
0.5491
We further assess the practical privacy implications of POST. A standard method for analyzing privacy risks in machine learning is membership inference attacks (MIAs) (Shokri et al. 2017). In our setup, the goal of the attack is to de- termine whether a given data point was used to train the private soft prompt p.. Such an attack could be executed by either the LLM owner or third party to gain knowl- edge on D prs. We instantiate the Likelihood Ratio Attack (LiRA) (Carlini et al. 2022) against soft prompts tuned on the four datasets for Roberta-base. As we show in Table with as few as eight shadow models, LiRA achieves a non- trivial level of data leakage, with an AUC of 0.5610 for the sst2 dataset. Applying DP with significantly decreases the AUC to 0.5258. We also report TPR @1 9FPR for the experiments. Results show that DP- tuned soft prompts reduce privacy leakage for all datasets. Beyond the privacy risk associated with the private soft prompt ps, we also analyze the potential pretraining data leakage during the KD phase, as shown in Appendix(.77
TPR FPR 0.0232
0.0137
0.0236
0.0157
0.0153
0,0128
0.0192
0.0181
through the entire model, this approach has the potential to substantially reduce computational complexity. In Table 4 we present a detailed runtime analysis of POST's individual components and compare them full PT. Note that in prac- tice, full PT on the large LLM leaks all the private prompt data to the LLM owner and is. hence, not practical. It serves only as conceptual bascline.
Our findings reveal a significant speedup, particularly for large downstream datasets, where the cost of backpropa- gating many samples through large language model is exceptionally high. For instance, on the sst2 dataset, our method achieves sixfold speedup. reducing runtime from 2,660 minutes to just 409 minutes on an A A100 GPU. These results highlight that POST not only enhances privacy but also delivers substantial computational efficiency improve- ments, especially for large datasets. Further details can be found in Appendix
and 8- = 1.5 10-3
5.5. Runtime of POST
5.6. Ablations
A notable advantage of POST is its ability to tune soft prompts on smaller models instead of significantly larger ones. Since tuning a soft prompt involves backpropagation
We perform various ablations on the inner workings of our method to provide guidance on how to best use in practice.
7

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
Table 4: Runtime of POST (GPT2-XL). We present the runtime for our method, split by its individual components, and compare against full prompt tuning on the large L.LM (PT on &t). We use arisetv and sst2 as private data and execute 5000 steps of transfer. We tune and transfer the prompts for 20 epochs (ss12) and 40 epochs (arisetv). All experiments are executed on single A100 GPU.
5.8
0.8
6.7-
6.66-
300, DP trinuter
3.5-
-46. traveter
tranafer
tmasshrr
1000
2000 2000 4000 5000
2000
6020 stege
6200
5003
stegs
(a) Roberta-base.
(b) GPT2-XL
Method
Runtime arisetv (min)
Runtime sat2 (min)
Figure 3: Effect of Number of Transfer Steps (arisetv).
PT on 4
368
2660
(1) PT on 4
310
(2) Transfer
99
99
Table 5: Baseline Comparison. We compare POST against
Ours total (1)+2)
145
409
(3 Ours total (1)+(2)+3)
1203
T20
state-of-the-art baselines, DP-OPT (Hong et al. 2023 (OPT.
1348
1612
DP-OPT) and Zero-Shot Transfer (Wu et al., 2023 (ZST).
We report test accuracies over different private datasets. We use our distilled 2-layer compressed Llama2-7b (2-Lay) or GPT2 as the source model, and Llama2-7b as the target model. For the DP versions, we use & = 8 and 8 according to Table TO] For our POST, we report the accuracies under the best public data (Table 13 and 14).
0.83
0.50
0.75
0.50
0.75
0.65
Do DP transfor
0.70
Lwb DP mavdor
DP taarder
D2 maashhrr
0.00
10
10
10
0.63
209
10
sample number
sample number
Mothod
d.
ss12
imdb
tweet
arisctv
mpga
disaster
tree
(a) Roberta-base.
(b) GPT2-XI.
OPT
2-Lay
81.31
68.44
38.35
82.00
58.50
46.00
58.00
OPT
GPT2
81.65
79.55
43,40
78.26
77.60
55.60
46.80
Figure 2: Effect of Number of Public Samples (arisetv).
DP-OPT
GPT2
72.59
69:53
24,90
30.44
61.80
48.90
32.60
ZST
2-Lay
62.38
70.57
42,80
58.33
33.31
43.55
18.22
ZST
2-Lay
53.55
59.54
32.70
11.60
POST (ours)
2-Lay
90.14
86.27
61.70
86.71
87.37
62.84
53.30
DP-POST (ours)
2-Lay
89.91
83.26
59.55
82.60
80.17
58.62
37.80
Effect of Number of Public Samples used for Transfer. We first investigate the influence of the size of the public dataset required to complete the prompt transfer. Our results in Figure [ for arisetv as the private dataset with different numbers of public samples from agnews show that we can already yield high transfer performance with less than 100 public data points. The small size of the public datasets needed makes our method highly practical.
ple, different numbers of classes. We added a more detailed discussion in Appendix D.I
Design of the KD. Building on our intuition that more simi- lar models exhibit better prompt transfer, we assess different ways of preserving this similarity during KD (see Equa- tion (.). We observe that fixing the language modeling head, i.e., causing higher output similarity, leads to slightly better transfer performance. In contrast, we do not observe a consistent improvement with fixing the embeddings. De- tailed results are shown in Appendix D.3
Effect of Number of Transfer Steps. We additionally in- vestigate how many transfer steps are required to obtain good performance. Based on the insights from the previous section, we randomly subsample 128 samples from the ag- news dataset as public data and report the achieved accuracy on arisetv as private data over different numbers of transfer steps. Our results in Figure 3 highlight that only small number of transfer steps is enough for convergence and high accuracy on the private task. We observe convergence within around 2000 steps for GPT2-XL. and around 1000 steps for Roberta-base.
Impact of Model Compression. We also investigate the trade-offs of different compression ratios. We distill the large LLM to various smaller sizes and report the perfor- mance of our POST and the distillation time. As we show in Appendix D.4] as the distilled model becomes larger, the transfer performance generally becomes better. However, it also requires more distillation time and more computational resources from the user to tune the soft prompt locally.
Choice of Public Dataset. We observe that the choice in the public dataset impacts the final test performance. To evaluate this impact of the public dataset choice on transfer performance, we conduct an ablation study using different public sets for various private datasets, see Table [6] Our findings indicate that the best transfer performance can be achieved with public datasets from the same task family, even if the datasets are not very similar and have, for exam-
Transfer Loss Design. Our transfer loss in Equation (3) has parameter a controlling the balance between the two loss terms. We observe that a good choice of depends largely on the target model's ZS performance and the compressed model's performance. We provide a detailed evaluation and heuristic for selecting the best a in Appendix D.5
8

---

Efficient and Privacy-Preseryii Soft Prompt Transfer for LLMs
Table 6: Impact of Feature Space Alignment on Transfer Performance. We compare the transfer performance of a distilled 2-layer Llama model and a non-distilled model that is fine-tuned directly on the same BookCorpus dataset. Both models have the same architecture and initialization, but only the distilled model maintains feature alignment with the large model. While the two models perform similarly before prompt transfer, the distilled model consistently achieves higher prompt transfer performance.
Dataset PT on Distilled Model
PT on non-Distilled Model
Transfer with the Distilled Model Transfer with the non-Distilled Model
sat2
78.78
78.24
90.02
85,38
imdb
79.95
79.40
87.29
82.45
tweet
54.12
54.79
61.15
45.65
arisetv
77.92
72.07
86.71
70.65
mpqu
83.82
84.16
87.37
81.52
Effect of Soft Prompt Token Length. We investigate the impact of token length on the transferabilitt of the soft prompt by conducting experiments across various lengths Appendix D.6j. Our findings suggest that an optimal range lies between 50 and 100 tokens. In contrast, both excessively short and long prompts lead to suboptimal results, with longer prompts also incurring higher computational costs.
source prompt using our compressed model as the small model, and 2) we use GPT2 as the small model. The latter is expected to have significantly higher performance and yield much better prompts. We report DP-OPT's performance on both non-DP (OPT) and DP settings. Our results in hhihhii that our POST significantly outperforms all baselines even in the DP regime. Additional results where we distill from GPT2-XL can be found in Appendix
Impact of Feature Space Alignment. In addition to the empirical results on prompt transfer, we further investigate why soft prompts trained on the distilled model remain transferable to the target model. Specifically, we conduct an ablation study to examine the role of feature space alignment induced by KD in our POST's transfer performance. We use a model with the same architecture and initial parameters as our distilled 2-layer Llama. Instead of applying knowledge distillation on the BookCorpus dataset, we directly fine-tune this model on BookCorpus to match the performance of the distilled model. This model has no feature space align- ment with the large model. We then evaluate its transfer performance and compare it to that of the distilled model, following the same transferring process in Section 4.3 The results in Table show that, although both models achieve similar prompt tuning performance, the prompt transfer performance for the distilled model is always better. This in- dicates that using knowledge distillation to preserve feature space alignment between the small model and large model is essential in our designed prompt transfer procedure.
6. Conclusions
We present POST, a framework for the private transfer of soft prompts that enables adapting LLMs with private user data while protecting both the user's privacy and the LLM provider's intellectual property. POST relies on distillation to enable an LLM provider to share a small model with limited utility to client for local prompt tuning on their private data, optionally with DP guarantees. Using our new prompt transfer method that leverages a small set of public data, the LLM provider can then transfer the prompt to their model. Our experiments highlight that the POST frame- work achieves significant improvements on the private tasks through the prompt transfer, improves the computational effi- ciency of prompt tuning and outperforms all private prompt transfer basclines. Thereby, our work paves the way for more trustworthy application of LLMs.
Impact Statement
5.7. Comparing against State-of-the-Art Prompt Transfer Approaches
We propose private transfer of soft prompts from a small language model to large language model. The primary positive societal impact of our work is that our method can protect local data privacy and also the intellectual property of the large language model provider, which encourages wider and more trustworthy applications of L.MMs. Addi- tionally, since our transfer enables more compute-efficient prompt tuning and enables the reuse of existing prompts, can have a positive environmental impact. We expect that our POST framework will also enable more LLM providers to offer the possibility of adapting their LLMs through soft prompt tuning.
We compare against two state-of-thee ohetth--aarr-- art baselines, namely the Zero-Shot transfer by Wu et al. (2023) and DP-OPT by Hong et al. (202)). Zero-Shot transfer operates in the same setup as we do and also relies on soft prompts. DP- OPT, in contrast to ours, is designed for discrete prompts. Details on their methods are presented in Appendix C,3 DP- OPT first tunes a discrete prompt locally and then directly uses it on the large model. Since their method relies on the small model having good performance, we execute their method in 2 setups for fair comparison: 1) we tune their
9

---

Efficient and Privacy-Preserving, Soft Prompt Transfer for LLMs
Acknowledgments
CrowdFlower.
Disaster response tweets, 2019. URL
https:
The project on which this paper is based was funded by the German Federal Ministry of Education and Research (BMBF) under funding number 16KIS21 14K. Responsibil- ity for the content of this publication lies with the authors.
started
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint 1rxiv:1810.04805. 2018.
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318,220 2016.
Duan, H., Dziedzic, A., Papernot, N., and Boenisch, F. Flocks of stochastic parrots: Differentially private prompt learning for large language models. Advances in Neural Information Processing Systems, 36, 2023a.
Duan, H., Dziedzic, A., Yaghini, M., Papernot, N., and Boenisch, F. On the privacy risk of in-context learning, In The 61 Annual Meeting Of The Association For Com- putational Linguistics, 2023b.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brinn, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E.,, et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp- 2397-2430. PMLR, 2023.
Dubiński, J., Pawlak, S., Boenisch, F., Trzcinski, T., and Dziedzic, A. Bucks for buckets (b4b): Active defenses against stealing encoders. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners.
Advances in neural information processing systems, 33: 1877-1901, 2020.
Dwork, C. Differential privacy. In International colloquium on automata, languages, and programming, pp. 1-12. Springer, 2006.
Buciluă, C.,, Caruana, R., and Niculescu-Minn Mizill Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Dwork, C., McSherry, F., Nissim, K., and Smith, A. Cal- ibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-284. Springer, 2006.
Carlini, N., Chien, S., NasT, M., Song, S., Terzis, A., and Tramèr, F. Membership inference attacks from first principles. In 2022 IEEE Symposium on Secu- rity and Privacy (SP), pp. 1897-1914, 2022. doi: 10. 00.119/9444214.2222.oo 9833449.
Dziedzic, A., Dhawan, N., Kaleem, M. A., Guan, J., and Pa- pernot, N. On the difficulty of defending self- self-supervised learning against model extraction. In ICML (International Conference on Machine Learning). 2022a
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, I.,, Zhuang. S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL
Dziedzic, A., Duan, H., Kaleem, M. A., Dhawan, N., Guan, J., Cattan, Y., Boenisch, F., and Papernot, N. Dataset inference for self-supervised models. In NeurIPS (Neural Information Processing Systems), 2022b.
2023-03-30-viee
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surpris- ing difficulty of natural yes/no questions. In Burstein, J.. Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pupers),
Dziedzic, A., Kaleem, M. A., Lu, Y. S., and Papernot, N. Increasing the cost of model extraction with calibrated proof of work. In International Conference on Learning Representations (ICLR) [SPOTLIGTH]] 2022c.
Gao, T., Fisch, A., and Chen, D. Making pre-trained lan- guage models better sho- learners. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 1 International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pp. 3816 2836-3830, 2021.
pp. 2924-2936, Minneapolis, Minnesota, June 2019. As- sociation for Computational Linguistics. doi: 10.18653) MIN19-1300, MBLlatpps:/aclaataaloyyy.o N1 93007
10

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
Gemini-Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.- B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Vision and Pattern Recognition (CVPR), 2018. URL
https://arxiv.org// http://////bb//11.555555
Kudari, V.
Acled information
extraction
dataset. Hugging Face, 2022. URL https
Geng, X. and Liu, H. Openllama: An open reproduction of llama, May 2023. URL https: http//ggttttubccco c open 11ama
colea--iaformatinn-eettaaar
Dataset
posted on Hugging Face.
Gu, Y., Dong, I.., Wei, F., and Huang, M. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Repre- sentations, 2024. TRL forum? 1 5= 1182Z
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter--fficient prompt tuning. arXiv preprint arXiv: 200.:22144, 2021.
Li, H. Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. Pruning filters for efficient convnets. In International Conference on Learning Representa- tions, 2017. URL
Hambardzumyan, K., Khachatrian, H., and May, J. Warp: Word-level adversarial reprogramming. In ACL-IJCNLP 2021-59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceed- ings of the Conference, pp. 4921-4933, 2021.
forum? Barum?id=JJQGTTS
Li, X. and Roth, D. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL https://
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems. 28, 2015.
www.ac..wbb.org/antho..o.
Li, X., Tramer, F., Liang, P., and Hashimoto, T. Large lan- guage models can be strong differentially private learn- ers. In International Conference on Learning Repre- sentations, 2022. URL forum?id=bVooP3tAATI
Han. S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Con- ference on Learning Representations (ICLR), 2016. URL abs 1at.://111001011
Li, X. I. and I Liang, P. Prefix-tuning: Optimizing continu- ous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), 2021.
https: / //aaaaiiivvorggr/ abtpor//////////
Han, X., Zhao, W., Ding, N., Liu. Z., and Sun, M. Pur: Prompt tuning with rules for text classification. Al Open, 3:18 2022.
pp.
Hanke, V., Blanchard, T., Boenisch, F., Olatunji, E., Backes, M., and Dziedzic, A. Open Iims are necessary for current private adaptations and outperform their closed alternatives. In Thirty-Eighth Conference on Neural In- formation Processing Systems (NeurIPS), 2024.
Liu, H., Tam, D.,, Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient fine- tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35: 1950-1965, 2022.
Hinton, G., Vinyals, O., Dean, J., et al. Distilling the knowledge in a neural network. arXiv preprini 0rXiv:1503.025311 2015.
Liu, J., Cyphers, S., Pasupat, P., Megraw, I., and Glass, J. A conversational movie search system based on con- ditional random fields. 13th Anmual Conference of the International Speech Communication Association 2012, INTERSPEECH 2012, 3, 01 2012.
Hong, J., Wang, J. T., Zhang, C., Li, Z.,, Li, B., and Wang. Z. Dp-opt: Make large language model your privacy-preserving prompt engineer. ArXiv, abs/2312.03724, 2023. URL https:
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer L., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. CoRR, hts/1907.11692 2019. URL lattp://axxiv.og//
266051675
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Conference on Computer
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng. A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
the Association for Computational Linguisties: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguis- tics. http:p//wwww.....eeeeeettttttttt
Schick, T. and Schütze, H. Few-shot text genera- tion with patter--eplooting training. arXiv preprint arXiv:2012. 2020.
P11-100 1015
Schick, T. and Schütze, H. Exploiting cloze-questions for few-shot text classification and natural language infer- ence. In Proceedings of the 16th Conference of the Eu- ropean Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, 2021a.
Okite, news-data. Hugging Face, 2022. Dataset posted on Hugging Face.
OpenAl. Gpt-4 technical report, 2023.
Schick, T. and Schütze, H. It's not just size that matters: Small language models are also few-shot learners. In Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-23552 2021b.
Oymak, S., Rawat, A. S., Soltanolkotabi, M., and Thram- poulidis, C. On the role of attention in prompt-tuning. In International Conference on Machine Learning, pp- 26724-26768. PMLR, 2023.
Philippy, F., Guo, S., Haddadan, S., Lothritz, C., Klein, J., and F. Bissyandé, T. Soft prompt tuning for cross- lingual transfer: When less is more. In Vázquez, R., Mickus, T., Tiedemann, J., Vulić, I., and Üstün, A. (eds.), Proceedings of the 1st Workshop on Modular and Open Multilingual NLP (MOOMIN 2024), pp. 7-15, St Ju- lians, Malta, March 2024. Association for Computational Linguistics. URL atttp::/aaclanttooooyy.oxx 2024 www.mmmmmmm...mm
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models, 2023.
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. Autoprompt: Eliciting knowledge from lan- guage models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP). Associa- tion for Computational Linguisties, 2020.
Qin, G. and Eisner, J. Learning how to ask: Querying Ims with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2021.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. In 2017 IEFE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Lim pruning and distillation in practice: The minitron approach, 2024. URL https //arxiv.org abs 2240b.11999
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I,, et al. Improving language understanding by generative pre-training. 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.
Su, Y., Wang, X., Qin, Y., Chan, C.-M., Lin, Y., Wang, H., Wen, K., Liu, Z., Li, P., Li, J., et al. On transferability of prompt tuning for natural language processing. In Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3949-3969, 2022.
Rakotonirina, N. C., Dessi, R., Petroni, F., Riedel, S., and Baroni, M. Can discrete information extraction prompts generalize across language models? In ICLR 2023. Proceedings of the 11th International Conference on Learning Representations (ICLR); 2023 Mai 1-5; Kigali, Rwanda. International Conference on Learning Represen- tations (ICLR), 2023.
Tang, X., Shin, R., Inan, H. A., Manoel, A., Mireshghallah, F., Lin, Z., Gopi, S., Kulkarni, J., and Sim, R. Privacy- preserving in-context learning with differentially private few-shot generation. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https //openreview.nee wttp://eeeetottooeo/ddoeetoe-eee
Rosenthal, S., Farra, N., and Nakov, P. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 1 11th international workshop on semantic evaluation (SemEvall- 2017), pp. 502-518, 2017.
Touvron, H., Martin, I.,, Stone, K. R., Albert, P., Almahairi, A., Babaci, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D. M., Blecher, L.,, Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W. Fuller, B., Gao, C., Goswami, V., Goyal,
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, 1b501910.01 108, 2019.
12

---

Efficient and Privacy-Preser Soft Prompt Transfer for LLMs
N., Hartshorn, A. S., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I. M., Korenev, A. V., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X., Tang. B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M. Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. ArXiv, 00122307.02288. 2023. http:////pp..mmmtttcccccc...aaa Corpus DD:29950929
Wiebe, J., Wilson, T., and Cardie, C. Annotating expres- sions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3): 165-2210, 2005.
Wu, T., Panda, A., Wang. T., and Mittal, Privacy- preserving in-context learning for large language mod- els. In The Twelfth International Conference on Learning Representations, 2024. URL https:// www.:e/wwwe.eee/ooouuiieee....e
Wu, Z., Wu, Y., and Mou, L. Zero-shot continuous prompt transfer: Generalizing task semantics across language models. In The Twelfth International Conference Learning Representations, 2023.
URL
Xiao, G., Lin, J., and Han, S. Offsite-tuning: Transfer learn- ing without full model. arXiv preprint arXiv:2302.04877 04870. 2023.
Varshney, T. and Surla, A. An introduction to large lan- guage models: Prompt engineering and p-tuning, 2023. URL [ttps://eevvllope...ddda.....aa.
an- a0--inoccoaccttoorr-oo-rrrggggge
Zhang, X., Zhao, J. J., and LeCun, Y. Character-level convo-
lutional networks for text classification. In NIPS. 2015.
tuning
Zhong, Q.. Ding, L., Liu, J., Du, B., and Tao, D. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. IEEE Transactions on Knowledge and Data Engineering. 2024.
Vu, T., Lester, B., Constant, N., Al-Rfou, R., and Cer, D. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5039-5059, 2022.
Zhong, Z., Friedman, D., and Chen, D. Factual probing is [mask]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 5017-5033, 2021.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benehmark and anal- ysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urta- sun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watch- ing movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
Wang, L., Xu, S., Xu, R., Wang, X., and Zhu, Q -onttransferable learning: A new approach for model ownership verification and applicability authorization. In International Conference Learning Representa- tions, 2022. URL httpss
Wang, X., Rachwan, J., Günnemann, S., and Charpen-
tier, B. Structurally prune anything: Any architec- ture, any framework, any time, 2024. URL https:
//arsiv.rgg/ab//2031121
Wang, Y., Wang, J., and Zhang, X. Parameter-efficcent on- line knowledge distillation for pretrained language mod- els. Expert Systems with Applications, 265: 165:1260 126040, 2025.
Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., and Goldstein, T. Hard prompts made easy: Gradient-baut discrete optimization for prompt tuning and discovery. ArXiv, abs/2302.03666. 2023. URL
CorpussDD:256227600
13

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
A. Limitations
Our work proposes a method to protect the privacy and confidentiality of private data during the prompt tuning phase. However, we didn't address the privacy leakage risk during the inference phase. Also, this work does not aim to address the challenge of preventing pretraining data leakage. Although our experiments suggest that the distillation process results in minimal leakage, more thorough investigation is warranted. Also, compression of the LLMs through knowledge distillation techniques may be computationally expensive for LLM providers. Additionally, in our method, the selection of public dataset will affect the transfer performance of soft prompts. While we observe, in general, that public datasets that have a similar structure to the private data work best for transfer, there is no ideal strategy for selecting the optimal public dataset.
B. Experimental Setup
B.1. Knowledge Distillation
We follow the procedure of (Sanh al.) 2019) initialize and distill compressed model. We use the first and last layers of Roberta-base, the first two and last two layers of GPT2-XL, and the first and last layers of Llama2-7b to initialize compressed Roberta-base, GPT2-XI. and Llama2-7b before knowledge distillation. We also initialize the small student model's s word embedding and language modeling head the same as their teacher model. We conduct experiments on whether to freeze the language modeling head and/or word embedding during knowledge distillation in Appendix D.3 Each model's structure and size are listed Table
Table 7: Model Size hefore and after Distillation.
Model
Layer Number
Hidden Dimension
Head Number
Parameter Num (M)
Roberta-base
12 768
12
125
Our distilled Roberta-base
2
768
12
53
GPT2-XL
48
1600
25
1560
Our distilled GPT2 XI.
4
1600
205
Llama2-7b
32
4096
32
6738
Our distilled Llama2 b
2
4096
32
667
During knowledge distillation, we use the BookCorpus (Zhu et al. 2013) dataset, and we take the checkpoint model that is distilled for 50,0000 steps. The hyperparameters used in knowledge distillation are shown in Table X
Table 8: Hyperparameters in Knowledge Distillation.
Ocee
Ctmm
Otoor
Learning Rate
Batch Size
5.0
2.0
1.0
0.00025
5
B.2. Datasets and Tasks
We use the following datasets as private sets or transfer sets in our experiments:
sst2: The Stanford Sentiment Treebank (sst2) is binary sentiment classification dataset containing sentences from movie reviews, labeled as positive or negative.
imdb: A large-scale sentiment classification dataset consisting of movie reviews from IMDb, labeled as positive or negative.
A benchmark dataset for sentiment analysis on tweets, providing real-world social media text challenges. In our experiments, we use only the sentiment classification subset which contains classes: negative, positive, and neutral.
tweeteval:
arisetv: A topic classification dataset derived from TV news transcripts, categorizing documents into 6 topic labels: business, sports, politics, health, entertainment, and tech.
agnews: A widely used topic classification dataset containing news articles labeled into four categories: World, Sports, Business, and Ciience/Technologe
14

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
mpqa: The Multi- Merspe-eerseectvve Question Answering (MPQA) dataset, designed for sentiment and subjectivity analysis with fine-grained opinion annotations.
MIT: A dataset for extracting attributes such as movie directors and genres from movie descriptions.
AIE: A dataset for extracting locations from notes.
free: The TREC Question Classification Dataset is benchmark dataset designed for question classification tasks in natural language processing (NLP). It is derived from the TREC (Text REtrieval Conference) Question Answering (QA) track, where the goal is to classify questions into predefined categories based on their topic.
disaster: A dataset containing tweets related to natural disasters, used for binary classification of whether tweet is disaster-related or not.
boolq: The BoolQ (Boolean Questions) dataset is a benchmark dataset for yes/no question answering (QA). BoolQ is widely used for training and evaluating machine reading comprehension (MRC) models and is part of the SuperGI.UF. benchmark.
We follow Li et al. 2022) to formulate our classification tasks as text-infilling task (instead of adding a classification head to output class probabilities). This means that for masked language models such as Roberta, we append "it was imask;" " to the input and let the model predict the ground truth text. The setting for GPTss and Llama2 is similar in that we append "it was" to predict the next word. To increase the robustness of this method, we use multiple ground truth text labels and compare the average probability of outputting those text labels. See Table 9 for task templates and the ground truth labels used in our experiment.
Table 9: Task Template and Ground Truth Labels used in Text-infilling. means the sentence used in the dataset.
Datasel
Task Template Roberta
Task Template GPT2/Llamu2
Ground Truth Text Lahel
8s12

, it was <mask>>>
0: I" "errible""" negative" bod"." awful""""""""
1: [" positive" " good""" great" awesome"" hrilliant" amazing"]
imdh

, it was <maas'>>
0: [" terrible""" negative"; ; bad"." poor"," awfull]]
1: I" positive"" good"" great"." awesome"." hrillian"." amazing"]
tweet

> it was <mask'>
<C
[" terrible"," negative", had"," poor"." awffil")
1: [" modcrate"." neutral", balanced"]]
2: I" positive" " good" "reat"." awesome"""" brillian"" amazing")
urisetv
<<>. it was about <-makk>>
<>>, about
0: I" busimess"] 1: sports], I" ["politics"]
3: [" health"].4: 1,4: [" "nteeraiimenn"). []5:: [" " science"]
mpqa

, it was <mask'>
<>>>
0: [" terrible"." negative"; bad"." poor"." "awful""
1: [" positive", good"." "geat"," "wesome""""" hrillian""" amazing"]
disaster
<S>, is this sentence related to disaster? <mask'> <>>, this question is about <mask>>
<S>>, sentence related to disaster?
0:1" no". " "No",, "No"". yes", "Yes"]
troe
<>>>, the topic of the question is
0:1" abbrcviation 111111 "1,2 ["description]]. 3:1" human").
4:1" location" location"|.5."" numerical value")
B.3. Prompt Tuning
Following Su et al. 2002 's setting, we use the soft prompt with a length of 100 tokens in all our experiments. We follow Duan et al. (2023a)'s 's setting to obtain DP private prompts with PromptDPSGD. Table shows the hyperparameters used in this experiment.
Table 10: Hyperparameters used for Prompt Tuning, including the d for PromptDPSGD.
Dataset
Fipachs
Learning Rate
1.5 1
0.001
imdb
4 x 10-0
20
0.001
tweet
2 x 10
20
0.001
arisetv
2 x 10-4
40
0.001
mpqu
1.16 x 10-4
20
0.001
15

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
B.4. Public Datasets for Prompt Transfer
We rely on small public datasets to perform our prompt transfer. A question is the right choice of the public dataset. We normally choose the public dataset that performs similar task as the private dataset, such as choosing imdb tweet as the public dataset of sst2 as they are all sentiment classification tasks. Transferring with public dataset that performs different task from the private dataset may lead to suboptimal performance. We tested this setting to transfer soft prompts trained on arisetv, a topic prediction dataset. The transfer performance of using tweet as public dataset is acceptable but generally worse than using agnews, another topic prediction dataset, as a public dataset. In general, we found that the public and private datasets do not need to have the same structure, such as the class number. For example, using tweet (3 classes) as a public dataset leads to better transfer performance than imdb (2 classes) on sst2 (also 2 classes). This highlights the robustness of our method and the broad selection of public datasets for the transfer.
We report the hyperparameters used in the transfer experiments as Tables 11 and 12
Table 11: Hyperparameters used during Prompt Transfer.
Model
Batch Size
Optimizer Learning Rate
Robertu base
32
Adam
0.001
GI.
8
Adam
0.001
Llama2-7b
4
Adam
0.0005
Table 12: Setting of for Different Datasets and Models during Prompt Trasnfer.
Dataset tweet
Model
sst2
imdh
arisetv
Robertu
0.2
GPT2-XL
0,7
0.4
0.0
0.6
Llama2 7b
0.9
1.0
0.6
0.6
C. Additional Experiments
C.1. Models of Different Scales
We present a full overview of our POST method's results on LLMs with different scales, i.e., Roberta-bass and GPT2-XL, Table 13 and Table [4 for confidential and DP transfer, respectively. We observe that over all models with different scales, our method significantly outperforms the baselines, both without and with DP.
C.2. Runtime
In Table 4 we analyze the runtime of our POST's individual components and compare them to the runtime of full prompt tuning on GPT2-XL. Note that in practice, full prompt tuning on the large LLM leaks all the private prompt data to the LLM owner and is, hence, not practical.
Prompt tuning time, for both large and small models, depends on the size of the private dataset when backpropagating over all private training examples. To illustrate this, we report results for both small dataset ( with 4.7k training samples) and a large dataset (sst2 with 67.3k training samples). KD is a one-time operation performed by the LLM owner, after which the distilled model can be used by multiple users. As a result, the compute cost of KD amortizes over time. To account for this, we present two sets of results for our method: (1) only the prompt tuning on the small model and the transfer step, and (2) including the non-an amortizdd KD time as a worst-case scenario. The runtime results in Table A show that, when considering only steps (1) and (2), our method significantly reduces runtime compared full prompt tuning for both small and large datasets. Notably, for the larger sst2 dataset, method achieves 6x speedup, reducing runtime from 2660 minutes to just 409 minutes an A 100 GPU. In the worst-case scenario, where the annotiztte KD time is included and only one task is tuned (a highly unrealistic assumption for an LLM owner serving multiple users), prompt tuning on the small model is faster. Yet, for the large dataset, our method still outperforms full prompt tuning with a runtime of 1612 minutes compared to 2660 minutes. These results demonstrate that beyond enhancing privacy, our POST method also provides substantial improvements in computational efficiency, especially for large datasets.
16

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
Table 13: Confidential Prompt Transfer Performance. We compress Roberta-base and GPT2-XL, tune prompts for different private datasets on the compressed models, and transfer them back using different public datasets (POST). As baselines, we present the large models' zero-shot performance on the private data (Full 7S), the accuracy of tuning the prompt with the private data on the large models (Full PT) and the small model (Compressed PT), and the performance of the prompt tuned on the small model when direcly applied to the large one (Direct Transfer). Our POST significantly improves performance over the small prompted model, and our prompt transfer yiclds strong improvement over the direct transfer.
POST (ours)
Private
Full ZS
Full PT
Compressed PT
Direct Transfer
Public
Test acc
Public
Test
ss12
72.25
91.74
(weet
87.16
imdh
imdb
72.19
89. 88
78.85
76.92
tweel
83.65
ss12
80.27
tweet
36.53
imdb
54.55
sat2
59.68
arisctv
38.80
89.81
70.98
47,82
agnows
82.97
tweet
68.48
(a) Roherta-hase.
POST (ours)
Private
Full 7S
Full PT
Compressed PT
Direct Transfer
Public
Test acc
Public
Test ace
ss12
60.78
94.84
80.94
59,06
tweet
85.89
imih
83.49
imdi
60.27
93.28
81.32
60.34
tweet
83.65
sst2
82.15 15
tweet
34.71
68.
63.13
41.50
imdb
60.55
sst2
57.70
arisctv
52.98
92.45
77.10
55.43
agnews
87,56
tweet
82.12
(b) GPT2-XL.
C.3. Baseline Comparison
Zero-Shot transfer performs prompt transfer by using the embeddings of some tokens from the vocabulary as a form of support vector to transform the source prompts into a relative space, and then search for the corresponding target prompt embeddings for the target model. To provide the optimal source model for their approach, we use compressed model that we obtained by keeping the embedding layer frozen during KD (see row Table 18). DP-OPT, in contrast to ours, is designed for discrete prompts. They first tune a discrete prompt locally and then directly use it on the large model.
We also run the baseline comparison the GPT2-XL model, as presented in Table 15 Our method consistently outperforms other methods on GPT2-XL.
D. Additional Ablations
D.1. Choice of Public Dataset
We investigate the effect of the choice of public data Table 16 The datasets fall into three categories: sentiment classification (sst2, imdb, tweet, mpqa), disaster detection (disaster), and topic classification (arisetv, agnews). The results show that using the same dataset for both private and public sets consistently achieves the best performance. Moreover, datasets with similar tasks transfer well, even if their number of classes differs. For example, imdb (2 classes) achieves 83.65 using tweet (3 classes) as the public set, and arisetv (6 classes) performs well with agnews (4 classes), i.e., 87.56 accuracy, indicating that task similarity is more crucial than class alignment. In contrast, using a public dataset from a different domain degrades performance. Sentiment classification datasets perform poorly when paired with disaster which aims to detect disaster, as seen in tweet (48.15). 15). Likewise, sst2 and imdb show weak results when using arisetv or agnews. These findings highlight that selecting public dataset with a similar task the private task is helpful for effective prompt transfer.
D.2. Other Model Compression Techniques
In this work, we focus on KD to compress the LLM. In the following, we discuss why alternative compression techniques are less suited implement our POST framework.
Quantization (Han et al. 2016 Jacob al. 2018) reduces the precision of model parameters to lower bit-widths, effectively decreasing model size. However, since our framework involves transmitting the proxy model to the user, quantization primarily reduces storage requirements without adequately protecting the intellectual property inherent in the original large model. Additionally, quantized models often require specialized hardware or software support to maintain performance, which may not be available to all users.
Pruning (Han al. 2015 et al. 2017 Wang et al. 2024 involves removing less significant weights or neurons from the
17

---

Efficient and Privacy-Preservinn Soft Prompt Transfer for LLMs
Table 14: Differentially Private and Confidential Prompt Transfer Performance. We compress Roberta- base and GPT2-XI., tune prompts for different private datasets on the compressed models with Differential Privacy guarantees = 8), and transfer them back using different public datasets (POST), As baselines, we present the large models' zero-shot performance on the private data (Full ZS), the accuracy of PromptDPSGD tuned prompt with the private data on the large models (Full PT) and the small model (Compressed PT), and the performance of the prompt tuned on the small model when direcly applied to the large one (Direct Transfer). Our POST significantly improves performance over the small prompted model and our prompt transfer yields a strong improvement over the direct transfer.
POST (ours) Public
Private
Full ZS
Full PT
Compressed PT
Direct Transfer
Public
Test acc
Test ace
ss12
72.25
90.14
67.54
77.06
tweet
84.40
imdb
81.42
imdib
72.19
88.55
72.22
74.35
tweel
79.64
ss12
80.64
tweet
36.53
62.05
imdb
55.65
59.25
arisetv
38.80
80.33
64.25
47,34
agnews
79.11
tweet
71.98
(a) Roherta-hase.
POST (ours) Public
Private
Full ZS
Full PT
Compressed PT
Direct Transfer
Public
Tost acc
Tost ace
sst2
60.78
91 1228
74.31
57.80
tweet
79.93
imdh
84.06
imdd
60.27
89.59
74.81
63.66
twoet
78.03
s812
75.16
tweet
34.71
61.47
48.60
41.50
imdb
58.05
ss12
54.75
arisetv
52.98
83.24
67.16
57.25
agnews
82.12
tweet
80.55
(b) GPT2-XL.
Table 15: Baseline Comparison. We present the performance of our method against state-of-the-art baselines on GPT2-XL. We use our compressed to represent the 4-layers small model distilled from GPT2-XI.. Our method outperforms both baselines.
Method
4
5812
imdb
tweel
arisetv
OPT (Hong stal.
2023
GPT2-XL our compressed
60.67
61.70
30.70
42,87
OPT [Tong
BUAS
GPT2-XL XL
GPT2
62.16
63.18
35,20
46.38
Zero- Shot Transier el
2023
GPT2-XI. XI. our compressed
63.65
61.27
41.60
56.64
Zero-Shot Transfer (Wu
DP
GPT2-XL our compressed
63,42
61.71
41.35
57.25
POST (ours)
EXT our compressed
85.85
8339
6..5
87.56
DP-POST (ours)
GPT2-XL our compressed
84.06
78.03
58.05
82.12
model. Unstructured pruning, which eliminates individual weights, typically does not lead to practical speedups or reduced resource consumption, making it less applicable to our approach. Structured pruning, on the other hand, removes entire neurons, filters, or layers. resulting in a smaller and more efficient model. However, many structured pruning methods are task-specific and require fine-tuning on specific tasks to regain performance.
In addition to the disadvantages of the other methods that make them less suited for our POST framework, KD also yields a desirable property for our purpose, namely, it aligns the outputs between the large L.M and the small model. This alignment in output also positively impacts the alignment of internal model behavior, supporting a better transferability of prompts tuned on the small model to the large LLM, as presented in Table 6
D.3. Knowledge Distillation Design
Knowledge Distillation Loss. The knowledge distillation loss, as shown in Equation I consists of three terms. We conducted an ablation study to analyze the impact of including or excluding each loss term and compared their end-to-end transfer performance using the Roberta-base model, as presented in Table 17 The results demonstrate that combining all three loss terms consistently yields the best transfer performance, validating our choice of loss function for KD.
Knowledge Distillation Setup. We also investigated the best way of performing KD improve prompt transferability. In particular, we analyzed the impact of keeping the word embedding or(and) language modeling heads frozen during KD on the prompt transfer performance. Our results in Table 18 highlight that keeping the language modeling head fixed performs slightly better than the alternative which mainly performs on-par. These results indicate that the successful transfer of our method is robust to the KD and independent of any specific KD setting.
When Stop Distilling, Since the L.LM provider does not have any knowledge of the users' private downstream tasks, they
18

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
Table 16: Ablation on the Choice of Public Set. We observe that best transfer performance can be observed when using same-domain data. In particular, it helps to have data from the same task familiy.
Public Set
Private
imdlb
mipqu
arisetvv
agneww
sa12
87.27
85.21
87.16
82.68
85.89
76.15
76.72
imdib
80,27
86.67
83.65
76.50
78.99
76.65
76.42
tweet
59.65
54.55
63.55
51.15
48.15
54.00
30
arisetv
82.12
77.41
78.62
50.84
57.13
86.47
87.56
Table 17: Ablation on Knowledge Distillation Loss. We compared the end-to-end prompt transfer performance of having each loss term and their combination the knowledge distillation. We first distill the model to the same checkpoint with different loss combinations, and then compare their transfer performance. We find that our choice of 3-way loss achieves best performance.
E.ce
Cim
Loos
fiee
+
Cim
fec
+ Locos
Cim +
Loco fee + Civ + fouus
Sa12
85,32
78.21
76.38
85.67
84.75
85.67
87.73
imdb
83.17
79.05
75.02
81.26
82,26
80.12
83.96
tweet
53.65
47.05
45.80
54,95
55,05
50.40
58.25
arisetv
78.99
67.87
75.60
74.52
82.49
70.65
82.73
Table 18: Analyzing the KD Setup. We perform an ablation on different designs of the KD and present their impact on the prompt transfer for the private arisetv dataset, using agnews as public data. We analze different combinations of freezing the emhedding (Fix emb) and freezing the language modeling head (Fix head).
Model
Fix emb
Fix head
Acc.
Model
Fix emb Fix bead
roberta-base
80.79 =0.885
GPT2-XL
86.51 =0,726
x
80.84 =0.360
86.81 10.732
x
80.11 =0.738
X
87.48 40.170
cannot rely on any external signal as a termination criterion for their KD, Given that they can observe the distillation loss, we find that terminating the KD when the loss starts plateauing is easy implement and effective. Hence, as the small model, we select the checkpoints of 500,000 steps, which are the checkpoints directly after the loss plateaus. Figure 4 we analyze the relationship between distillation loss and steps. We also conduct transfer experiments with different checkpoints and show the relation between distillation loss and transfer performance in Figure 5b) We observe that the performance gain becomes smaller as the distilled model starts to converge. Therefore, by finishing the KD as early as possible, we save computing time while still being effective.
D.4. Influence of Compressed Model Size
In Table 19 we also compare the transfer performance from distilled models with different compression ratios (in our case, the number of layers). These results highlight that as the distilled model becomes larger, the transfer performance generally becomes better. However, it also requires more distillation time and more computational resources from the user to tune the soft prompt locally. We observe that the gain that can achieved through soft prompt transfer becomes smaller or even negative for datasets like imdb and tweet, with 3-layer distilled model. This would incentivize the users not transfer their prompts back the large model, thereby disrupting the LLM provider's business model. Hence, found that our choice of 2-layer (4-layer) compressed model for Roberta-base (GPT2-XL) offers good balance for the LLM provider and user.
To study the relationship between transfer performance (evaluated by downstream task accuracy) and performance of the compressed model (evaluated by checkpoint loss) further, we also conducted ablations where we compress the models to different numbers of layers and with different distillation steps. We present the results in Figure 5a and Figure [b They highlight that overall better compressed models lead to better transfer accuracy.
19

---

Efficient and Privacy- Soft Prompt Transfer for LLMs
checkpoint loss
80
60
40
-0
200000
300000
400000
500000
600000
steps
distille_roberrtee roberta-blayeeeeeeeee
distilled Destilled_robrrta roberta 2layer
distilled_rofee roberta 4layer
Figure 4: Checkpoint Loss on Training Steps for Distilled RoBERTa Models with Varying Numbers of Layers. The plot illustrates the convergence behavior of I-layer, 2-layer, 3-layer, and 4-layer distilled models. The dashed vertical line represents the point at which a specific checkpoint is selected for evaluation in our experiments.
Table 19: Inffuence of Compressed Model Size (Number of Layers). We compare the confidential prompt transfer performance of compressed models based on RoBERTa-base, varying the number layers in the compressed model. We also report the compressed model performance parentheses. As the number of layers increases, transfer performance generally improves, but the distillation time also increases. Our selected number of layers in the compressed models offers good balance between transfer performance and distillation cost.
# layers in distilled version
layer
2 layors (paper)
layers
sst2
84.52 (71.67)
87.73 (78.78)
88.53 (85.55)
(71
(79.95)
88)
tweet
50.65 (41.8)
54,55 (54.12)
61.50 (66.66)
urisety
53.62 (24.55)
82.73 (77.92)
86.45 (84.77)
Distill time
6h 04min
66 45min
7h 35min
0.85
0.8
0.80
Tranafered Ascussy
0.75
0.70
0.4
0.65
0.60
02
0.55
0.50
0.45
15
20
30
40
45
50
Compressed PT Accuracy
Checkpoint Lass
joberta-llayer
distilled, 3layer
layer
disttlled_reaeee joberta-3laye
distilled robarta 2layer
distilled roberta-4layer
distilled rabertaaayea
distilled roberta-4layer
(a)
(b)
Figure 5: Analysis of Transferred Accuracy versus Compressed PT Accuracy and Checkpoint Loss at 50,000, 100,000, 200,000 and 500,000 Distillation Steps. (a) compares transferred accuracy to compressed PT accuracy for different distilled Roberta models. (b) compares transferred accuracy to checkpoint loss for different distilled Roberta models.
20

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
D.5. Influence of Transfer Loss Design
We further conduct an ablation study to examine the impact of the choice of a during the prompt transfer. As described in Section [ we should use a larger when the teacher LLM 4, already has non-trivial zero-shot performance on the private task compared to the compressed model. And we smaller when the compressed model has better performance. The intuition is that if the target model performs well, it needs to less mimic the behavior of the smaller model, but only incorporate that model's direction change induced by the prompt. On the other side, when D, has poor zero-shot performance, we put more emphasis on the output of the compressed model to provide the update.
Based on this insight, we design heuristic to find an optimal a as shown in Equation (6)
Z5 - RG mmin
RG 1.0).
(6)
where ZS represents the zero-shot performance, C represents the performance of the compressed model, and RG represents the random guess performance of the dataset (e.g., for sst2 with 2 classes, RG is 0.5). This heuristic can serve as a guideline for choosing the best alpha for transfer. In Table 20 we perform wide range of experiments using different alphas and compare the berf-perfogning alpha with the one output by our heuristic. We observe that the heuristic successfully identifies good range for and is usually not far off the empirical optimal values. Finally, this wide hyperparameter search shows that multiple a values achieve good transfer performance, highlighting our method's robustness to the choice of a.
Table 20: Performance of Roberta-base, GPT2-XL and Llama2-7B on 4 Different Datasets with Various a Values. Transferred accuracy are presented mean (standard deviation). We observe that multiple a values achieve good transfer performance, highlighting the robustness of our POST to the choice of a.
Roberts-huse
1.amm277h
Indch
6022
im3h
S522
Lmch
Dwaztt
ariectv
0.76
0.77
0.14
0.25
0,33
0,05
0.60
1.00
0.54
0,9F7
(1.23) (1.23)
(0.55)
(.88)
(0.91)
(0.53)
70.67
(2.20)
(2.64)
75.
(0.44)
60.00
(0.65)
(0.37)
0.1
$1. $1.941
(1.87)
79.20 (2 22.92
57.08 (1 (1.55)
83,09 (1.45)
76.68 (0.46)
70.23 (3.45)
18.48.0 (1 31)
85.67 (0.
68.43 (5.20)
76.27 (2.07)
59.50 (2.42)
81 31.00 (0.37)
0.2
83.33
(0 78)
80.89 (1 8.5
58.13
(2.16)
83.62 (0.77
76 80 (0.65)
78. 19 37)
57.55 00
85
(0.51)
70.76 (4.70)
76.12 (8.68)
61:33 (9)
79.83 (2.06)
80.57
83.62
(8)
86.11
61.62
81 (3.56)
0,4
$6.81
46)
79.56 (1 20
54.77
(4)
83.94 (0.79)
76.53 (0. (6)
$7.57 88)
42 11
86.
36)
70.18 (4.83)
73.29 (0.04)
59.77 (1 28)
82,81 (0.87)
0.5
58 (0.44
16 (1 55
54.10 74)
(1.75)
76.83 (0.65)
$1.74 (0.71)
50 09)
87.34 (1 50)
85.13 (4)
50.24 (2.19)
60:23 (1 34)
$3.45 (0.53)
0.6
S8.61 (0.78)
81.82 (1 (6)
52.55 (1.44)
76.57 (0.84)
83.03 (1.69)
79:82 (1 86)
53.93 (0 64)
87.72 (0_39)
89.68 (0.20)
81.90 (3.10)
55.13 (0.18) 18)
85.47 (1.61)
71.26 (0.64)
78. 18
25)
(1.72)
87.
(0.34)
82.65 (0.95)
(0 (4)
68 (0.57)
(0:93)
73.23 (2.07)
50.53 (2
86
62)
89.22 (0.53)
83.86 (0.55)
52.25 (0.57)
83
0.9
56.47 (0.64)
81.52 (1 39)
49.38 (0.55)
63.04 (1.85)
84.33 (1.26)
76. (0.91)
48.52 (1 53)
83.66
(1 42)
88.69 (0.69)
85.00 (2.35)
49.50 (0.31)
83.33 (2.06)
1.0
85.82 (0.85)
80.57 () 32)
48.42 (1.13)
58.57 (0.24)
83,37 (0.92)
73.62 33)
48.42
80. (1.35)
86,47
85.85 (1.77)
47.57 (1.1)
84:00 (0.91)
D.6. Influence of Token Number of Soft Prompt
We follow (Wang et al. 2022) use 100 tokens in our experiment. We conduct further experiments to investigate how token length influences the transferability of the soft prompt. We compare the transfer performance of soft prompts with lengths of 5. 10, 20, 50, 75, 100, and 200 as shown in Table 21 We find that a prompt length of 50-100 tokens is good range, and our method is not very sensitive to the token length. Both too few or too many tokens in the soft prompt lead to sub-optimal performance. Additionally, too many tokens need more computing resources.
Table 21: Ablation on the Token Number of Soft Prompt. We compare the prompt transfer performance of different soft prompt token lengths on four classification tasks with caberta-base.
Token Num
10
20
50
75
100
200
sa12
84.17
85.66
86.35
87.38
86.70
87.16
85.32
imdh
75.95
76.53
80.42
82.77
83.27
83.65
79.92
55.25
57.60
57.80
57.00
60.00
59.65
60.40
arisctv
79.02
79.63
80.47
82.04
82.77
82.97
83.13
D.7. Pretraining Data Leakage during Distillation
We also consider and investigate the pertaining data leakage during the KD phase. We follow the setup Sfi et al. (2023) to assess pretraining data leakage during the knowledge distillation in our paper, using the WikiMIA dataset. Specifically, we use the Pythia 2.8B model (Biderman et al.) 2023) as our base model, as pretraining data is publicly available. We
21

---

Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
apply knowledge distillation obtain smaller student models: with 3 layers and another with 4 layers. To evaluate data leakage, conduct membership inference attacks (MIA) using the Mink% method and report the AUC-ROC and TPR @ I 1%FPR for all three models. The results, summarized in Table 22 indicate that our knowledge distillation reduces the pretraining data leakage compared to the original model.
Table 22: Ablation on Pretraining Data Leakage. We compare AUC-ROC and TPR @ 1%FPR using the original Pythia 2.8B model and two distilled models on the WikiMIA dataset. The distilled models exhibit less pretraining data leakage compared to the original pretrained model.
Pythia 3-layer Distilled 4.layer Distilled
AUC-ROC
0.7103 0.0980
0.5737 0.0196
0.5750
TPR @ 1&PPR
0.0196
22


Reviews:
[{"review_id": "PGOlgJaqRb", "reviewer": "Reviewer_SqSB", "summary": "This paper proposes POST (Privacy Of Soft-prompt Transfer), a framework designed to enable efficient and privacy-preserving transfer of soft prompts between LLMs. It has three major steps:\n* Knowledge Distillation: The LLM provider first distills a smaller local model from the original large LLM.\n* Local Prompt Tuning: The user tunes soft prompts on the small model using their private data (with optional differential privacy for additional protection).\n* Privacy-Preserving Prompt Transfer: The LLM provider transfers the tuned prompt back to the large LLM using a small public dataset, avoiding any access to private data.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the Reviewer's constructive review and address each point as follows:\n\n>**The problem setting itself is unrealistic to me.**\n\nSoft prompts are exposed via public APIs, such as NVIDIA NeMo, as we highlight in the abstract and introduction of our paper.  Could the reviewer kindly clarify specific concerns regarding the “*realism of the problem setting*” in our work?\n\n>**I don't see clear contributions of the paper.**\n\nWe enumerated the contributions of our work at the end of the introduction. \n\n>**distilled model parameters can reveal pretraining details [1].**\n\nThe problem of protecting pre-training details is orthogonal to our work. However, we do agree that the LLM provider might not want to disclose such details.\n\nTo address the reviewer’s comment, we follow the setup from the paper cited by the Reviewer [1] and use mink% to perform Membership Inference Attack on the WikiMIA [1] dataset with the Pythia-2.8B model.  We test 2 models distilled from Pythia-2.8B, each with 3, and 4 layers, respectively. Our results are depicted below and show that the AUC and TPR@1%FPR of the distilled models are much smaller than the base model. This shows that the knowledge distillation in our work can even reduce the pretraining data leakage.\n\n||Pythia 2.8B|3-layer distilled|4-layer distilled|\n|-|-|-|-|\n|ROC-AUC|0.7103|0.5737|0.5750|\n|TPR@1%FPR|0.0980|0.0196|0.0196|\n\n>**The LLM provider shouldn't have knowledge of users' private data.**\n\nOur framework assumes that the model provider has knowledge of only the task domain, not the user’s actual data. This is a reasonable assumption, as task metadata is often available in practical settings. For instance, cloud providers and ML platforms (e.g., Google Vertex AI, OpenAI) typically encourage users to specify the task type or provide task demonstrations (i.e., few-shot learning) to fine-tune models [A]. Similarly, many AI service providers, such as NVIDIA’s NeMo framework, offer task-specific APIs explicitly designed for applications like chatbots and summarization, which inherently expose task-level information [B]. \nImportantly, our definition of *privacy leakage* follows DP and is measured at the individual datapoint level. Our approach follows the practical setups. We also offer a general statement regarding the privacy-utility trade-off at the end of our response.\n\n>**Additional baseline 1: Zero-shot performance of instruction-tuned models**\n\nWe appreciate the suggestion and have included the zero-shot performance of instruction-tuned Llama2-7b-chat in the following table:\n|SST2|IMDB|Tweet|arisetv|mpqa|\n|-|-|-|-|-|\n|81.42|85.01|44.55|77.29|54.37|\n\nThe instruction-tuned model has very close zero-shot performance to the non-instruction-tuned model. This is because we convert the problem of classification to a prefix infilling problem for a non-instruction-tuned model and aggregate the results with multiple ground truth text labels, see Table 7 in the Appendix. This enhances the robustness of our evaluation and allows the non-instruction-tuned model to effectively handle classification tasks. \n\n>**Advantage of prompt tuning**\n\nPrompt tuning offers advantages for LLM providers as it enables batch processing, allowing multiple tasks (from different users) to be handled in a single inference pass with prompts. In contrast, LoRA or fine-tuned models require separate inference passes through the model, significantly increasing the computational costs, and even more importantly, separate models per task [C]. The efficiency gain of soft-prompts provides a strong incentive for the LLM providers to adopt our framework. \n\n>**Additional baseline 2: Full fine-tuning on the compressed model**\n\nWe have conducted an additional experiment evaluating full fine-tuning on the compressed model:\n||SST2|IMDB|tweet|arisetv|mpqa|\n|-|-|-|-|-|-|\n|Llama2-7b-compressed|83.02|83.39|59.21|78.41|85.23|\n\nFull fine-tuning on the compressed model indeed outperforms prompt tuning on the compressed model, however, it still underperforms POST transfer. This is because the compressed model, although it maintains some similarity to the teacher model, is still not powerful enough. \n\n>**expected similarity between the private and public datasets**\n\nTable 15 in Appendix D.1 investigates the impact of public dataset selection. We categorize datasets into three groups based on task similarity and observe that public datasets with tasks similar to the private dataset yield better transfer performance. Thus, it is expected that the public dataset is from the similar task domain as the private dataset in our framework.\n\n>**Why the DP is “optional”?**\n\nOur framework is designed to be modular, allowing private data owners to balance privacy and utility trade-offs based on their specific requirements. DP is an optional component that can be integrated when strong privacy guarantees are needed, but its inclusion depends on the user’s trade-off preferences.\n\n---\n\nReferences included in answer to Reviewer Qi3Y.", "claims_and_evidence": "No obvious wrong claims found.", "methods_and_evaluation_criteria": "Methods are fine for the assumed scenario, but the problem setting itself is unrealistic to me.", "theoretical_claims": "The paper doesn't have theoretical claims.", "experimental_designs_or_analyses": "I did check the experiments in the paper. Detailed comments are in \"Strengths and Weaknesses\" below.", "relation_to_broader_scientific_literature": "I don't see clear contributions of the paper.", "essential_references_not_discussed": "No obvious missing references found.", "other_strengths_and_weaknesses": "**Strengths:**\nThe paper is well-written and easy to follow.\n\n**Weaknesses:**\n\nMy major concern is the unrealistic Problem Setting:\n* The proposed approach assumes that the LLM provider will release a distilled smaller version of their strong and close-sourced LLM. However, this is impractical in real-world scenarios because the distilled model parameters can inadvertently reveal crucial pretraining details, such as the data weighting of different training corpora [1].\n* The approach also requires the LLM provider to execute the proposed soft prompt transfer, which involves selecting a public dataset that is expected to be similar to the private dataset used by the user. This assumption is problematic because, in a proper setting, the LLM provider should not have knowledge of users' private data.\n\nImportant models and baselines are also missing in the experiments:\n* The experiments are conducted using base models such as Llama2-7B and GPT2-XL. However, real-world LLM services typically deploy instruction-tuned models (e.g., Llama2-7B-Instruct), which exhibit significantly stronger zero-shot problem-solving capabilities. The absence of instruction-tuned models makes the \"Full ZS\" baseline unconvincing.\n* Prompt tuning generally performs worse than full fine-tuning or LoRA fine-tuning while offering only marginal reductions in training cost since it still requires backpropagation through the full model. The key advantage of prompt tuning is its reduced parameter storage requirements. Therefore, a crucial missing baseline is full fine-tuning on the compressed model. Demonstrating superiority only over the prompt-tuned compressed model (\"Compressed PT\") is not sufficient; the study should compare against a fully fine-tuned small model to justify the proposed approach. For example, if the fully finetuned compressed model is better than the proposed approach, why should the users keep paying for the LLM provider's service? \n\n[1] Detecting Pretraining Data from Large Language Models, ICLR 2024.", "other_comments_or_suggestions": "no", "questions_for_authors": "* What's the expected similarity between the private and public datasets in the proposed framework?\n* I understand the soft prompt sent from the private data owner to the LLM provider should definitely have DP guarantees. Otherwise the privacy is not protected at all. Why do you say the DP in this step is just \"optional\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MrPoQ4rJH0", "reviewer": "Reviewer_f1xX", "summary": "This paper proposes POST (Privacy Of Soft-prompt Transfer), a framework designed to efficiently and privately transfer soft prompts for adapting large language models (LLMs) to private downstream tasks. The core innovation involves locally tuning soft prompts on a small, distilled model derived from a larger LLM, optionally under differential privacy constraints, and subsequently transferring these prompts back to the large LLM via a small public dataset. The method reduces computational requirements and enhances privacy, avoiding the direct sharing of sensitive data with LLM providers. Experimental evaluations using various classification and generation tasks show POST's improvements over existing prompt transfer methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the Reviewer's insightful comments! Below, we address each comment in detail:\n\n>**More rigorous exploration into why certain public datasets work better for transfer than others; Additional clarification or justification for the choice of public datasets.**\n\nWe thoroughly analyzed the effect of public dataset selection in Table 15 in the Appendix (section D.1). Our experimental results demonstrate that public datasets from the same task domain as the private dataset yield better transfer performance than others. This aligns with the intuition that task similarity enhances knowledge transfer. Based on these findings, it is suggested to select a public dataset for a similar task as the private data for effective prompt transfer.\n\n>**A clearer explanation of the hyperparameter choices (especially $\\alpha$ in the transfer loss)**\n\nWe conducted an ablation study for the parameter $\\alpha$ in Table 19 in the Appendix (section D.5). \nWe evaluated the transfer performance across various $\\alpha$ values, ranging from 0.0 to 1.0. The results indicate that multiple $\\alpha$ values lead to good transfer performance. \nFurthermore, we designed a heuristic to find an optimal $\\alpha$ as shown in Equation 6 in the Appendix (section D.5). The intuition of the heuristic is that if the target model performs well, it needs to less mimic the behavior of the smaller model, but only incorporate the directional change induced by the prompt, leading to a larger $\\alpha$. It can be observed from Table 19 that the designed heuristic can successfully identify a good range for $\\alpha$ and is usually not far off the empirical optimal values. \nThus, for each dataset and model, we selected the $\\alpha$ according to the heuristic we proposed in the same appendix section (D.5).\n\n>**Potential limitations in transfer performance to LLMs with significantly different architectures or training paradigms.**\n\nIn our setup, the LLM provider distills a small model from the LLM with the aim of guaranteeing good transfer performance. The transfer performance between different architectures is orthogonal to our problem setup.\n\n>**Practical privacy impacts beyond membership inference attacks**\n\nMembership inference attacks are widely recognized as a gold standard for evaluating privacy leakage. Notably, in its most stringent form, our POST framework incorporates DP, which, by definition, ensures that the presence or absence of any individual data point remains indistinguishable and provides theoretical protection against **every possible attack**. \n\n>**Can the authors clarify how sensitive the method is to the choice of the $\\alpha$ parameter in the prompt transfer loss? How was $\\alpha$ tuned in practice?**\n\nWe performed an ablation study on different $\\alpha$ values in Table 19, confirming that our approach is robust to a range of $\\alpha$-s. Additionally, we designed a heuristic to find an optimal alpha (see Equation 6). The underlying intuition is that if the target model exhibits strong zero-shot performance, it requires less direct imitation of the small model. Instead, it should primarily incorporate the directional changes induced by the prompt, ensuring effective transfer while maintaining the target model’s inherent capabilities.\n\n>**How would the approach scale to very large LLMs (e.g., 70B+ parameters)?**\n\nThe LLM providers are assumed to have the computational resources to distill the target models into smaller ones, as they already manage to train such large language models. The overhead on the client end should not depend on the scale of the model used by the LLM provider.", "claims_and_evidence": "The claims in the paper are generally well-supported by empirical results, including evaluations on several datasets (sst2, imdb, tweet, arisetv, mpqa, disaster, trec) across classification and open-ended generation tasks. However, more rigorous exploration into why certain public datasets work better for transfer than others would enhance the clarity of some claims about dataset suitability.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate and clearly defined for the targeted privacy and efficiency goals. The datasets selected for evaluation (sst2, imdb, etc.) are standard benchmarks, which strengthens the credibility of the results. However, additional clarification or justification for the choice of public datasets is needed.", "theoretical_claims": "The paper does not contain explicit theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound. The authors thoroughly evaluated their methods, including ablations for the impact of the number of public samples, the number of transfer steps, and model compression ratios. However, a clearer explanation of the hyperparameter choices (especially α in the transfer loss) is needed.", "relation_to_broader_scientific_literature": "The paper positions itself clearly within the broader literature on soft prompts, knowledge distillation, and differential privacy, effectively differentiating itself by combining these aspects into a cohesive framework. The distinction and improvement over previous methods like DP-OPT and zero-shot transfer (ZST) are explicitly demonstrated.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n\n1. Innovative combination of knowledge distillation, differential privacy, and prompt tuning.\n\n2. Comprehensive experimental validation demonstrating clear benefits over existing methods.\n\n3. Significant practical implications for privacy-sensitive applications of LLMs.\n\nWeaknesses:\n\n1. The choice and generality of the public dataset for prompt transfer are not fully justified.\n\n2. Potential limitations in transfer performance to LLMs with significantly different architectures or training paradigms are not fully explored.", "other_comments_or_suggestions": "An in-depth analysis or case study illustrating practical privacy impacts beyond membership inference attacks would further strengthen the narrative.", "questions_for_authors": "1. How robust is POST's performance when the public dataset significantly differs from the private dataset in distribution or domain? Would this impact the transfer accuracy significantly?\n\n2. Can the authors clarify how sensitive the method is to the choice of the α parameter in the prompt transfer loss? How was α tuned in practice?\n\n3. How would the approach scale to very large LLMs (e.g., 70B+ parameters)? Would computational constraints significantly limit the practicality of POST in such scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mWGR06WvK9", "reviewer": "Reviewer_n3JH", "summary": "This paper addresses the problem of inefficiency and privacy risks in soft prompt tuning on LLM provider-hosted models. Therefore, the authors propose a prompt transfer framework called POST, that first distill a smaller model to better match the teacher behavior, then prompt tuning on the small model using private data, and finally transfer the prompt to the larger model with a non-private dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the positive feedback! We address the main points one by one below:\n\n>**The quality of the transferred prompt heavily depends on the distilled model.**\n\nWe fully agree with the Reviewer. The influence of compressed model size (number of layers) on performance and compute time is presented in Table 18 in our paper. And we show the results in the table below for the reviewer’s convenience. Indeed, our findings show that while larger distilled models generally improve performance, they also increase computational costs for the user. \n\n|# layers in distilled version|1 layer|2 layers (paper)|3 layers|\n|-|-|-|-|\n|sst2|84.52|87.73|88.53|\n|imdb|78.01|83.96|83.64 |\n|tweet|50.65|54.55|61.50 |\n|arisetv|53.62|82.73|86.45|\n|Distill time|6h 04min|6h 45min|7h 35min|\n\n\n>**For complex tasks, where the full target LLM can effectively learn a strong prompt, a weakly tuned prompt on the small model will not transfer well, as the distilled model may fail to capture the necessary task structure.**\n\nBeyond investigating the trade-offs of different compression ratios, we also designed a heuristic approach to optimize the $\\alpha$ in Equation 6, which balances the first and second loss terms in the transfer loss function (Equation 3). Specifically, when the full target LLM has a strong zero-shot performance or the small model has a weak performance, we set a larger $\\alpha$, reducing the reliance on the small model’s behavior while primarily incorporating the directional change induced by the prompt. This adjustment enhances the effectiveness of our POST method in scenarios where the small model has weak performance.\nWe perform a wide range of experiments using different $\\alpha$ values and compare the best performing $\\alpha$ with the one output by our heuristic, as shown in Table 19 in Appendix D.5. For reviewer’s convenience, we show partial results of Table 19, i.e., the performance of Roberta-base, in the following table (transferred accuracy are presented as mean (standard deviation)):\n\n|$\\alpha$|sst2|imdb|tweet|arisetv|\n|-|-|-|-|-|\n|heuristic|0.76|0.77|0.14|0.41|\n|0.0|80.54 (1.23)|78.22 (0.55)|57.75 (1.88)|83.41 (0.91)|\n|0.1|81.94 (1.87)|79.20 (2.92)|57.68 (1.95)|83.09 (1.45)|\n|0.2|83.33 (0.78)|80.89 (1.85)|**58.13 (2.16)**|83.62 (0.77)|\n|0.3|84.67 (0.73)|80.57 (1.16)|58.08 (1.15)|83.62 (1.26)|\n|0.4|86.81 (0.46)|79.86 (1.20)|54.77 (1.04)|**83.94 (0.79)**|\n|0.5|88.61 (0.46)|82.16 (1.55)|54.10 (0.74)|80.11 (1.73)| \n|0.6|**88.61 (0.78)**|81.82 (1.06)|52.88 (0.44)|76.57 (0.84)|\n|0.7|87.88 (0.29)|80.89 (1.59)|52.33 (0.20)|71.26 (0.64)|\n|0.8|87.35 (0.24)|**82.65 (0.95)**|51.32 (0.24)|68.68 (0.57)|\n|0.9|86.47 (0.64)|81.52 (1.39)|49.38 (0.55)|63.04 (0.85)|\n|1.0|85.82 (0.85)|80.57 (1.32)|48.42 (1.13)|58.57 (0.24)|\n\nWe observe from the results that the heuristic can successfully identify a good range for $\\alpha$ and is usually not far off the empirical optimal values.", "claims_and_evidence": "Yes the claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes the proposed three-step prompt transfer method makes sense for the problem of addressing data privacy and prompt tuning efficiency.", "theoretical_claims": "No applicable.", "experimental_designs_or_analyses": "The experimental settings are generally comprehensive, the proposed POST method is compared with 1. zero-shot performance on private data 2. performance of private data prompt tuning on target model 3. performance of private data prompt tuning on source distilled model and 4. directly transfer the learned prompt to target model that not using public data. POST significantly outperforms the directly transferred prompt and achieves better performance than the distilled source model, demonstrating its practical effectiveness.", "relation_to_broader_scientific_literature": "The proposed method sits at the intersection of knowledge distillation, soft prompt tuning and privacy-preserving ML. It improves soft prompt transferability in Wu et al., 2023 and privacy issue in Su et al., 2022.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method is well-motivated\n2. The analysis is comprehensive\n\nWeaknesses:\n1. The quality of the transferred prompt heavily depends on the distilled model. For complex tasks, where the full target LLM can effectively learn a strong prompt, a weakly tuned prompt on the small model will not transfer well, as the distilled model may fail to capture the necessary task structure.", "other_comments_or_suggestions": "No", "questions_for_authors": "See weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PP3kLU6Txe", "reviewer": "Reviewer_Qi3Y", "summary": "The paper \"Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs\" introduces POST (Privacy Of Soft-prompt Transfer), a framework that enables efficient and privacy-preserving soft prompt tuning and transfer for Large Language Models (LLMs).\nKey Contributions:\n\nPrivacy-Preserving Soft Prompt Tuning: POST allows users to tune soft prompts locally on a small, distilled model instead of directly on a large LLM, avoiding the need to share private data with external providers.\n\nKnowledge Distillation for Transferability: The LLM provider distills a small version of their model, which the user uses for tuning. The tuned prompt is then transferred back to the large LLM using only a small public dataset.\n\nDifferential Privacy Integration: POST incorporates differential privacy (DP) to ensure that sensitive user data remains protected even during prompt tuning.\n\n Efficient and Effective Transfer: The method significantly reduces computational costs while maintaining strong task performance, outperforming existing soft prompt transfer approaches.\n\nExperiments & Findings:\n\nEvaluations on multiple LLM architectures (RoBERTa, GPT2-XL, LLaMA2-7B) show that POST achieves high-utility prompt transfer while preserving privacy.\n\nIt significantly reduces computational overhead compared to direct prompt tuning on large models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the constructive feedback! We address the main points one by one below:\n\n>**A more detailed discussion on public dataset selection risks**\n\nWe investigate the effect of the choice of public data in Table 15 in Appendix (section D.1), where we consider both public datasets from similar task domains as well as those from different task domains compared to the private dataset. Our experimental results show that selecting a public dataset with a similar task to the private dataset is helpful for effective prompt transfer. \n\nTo limit the privacy leakage and obtain good performance, we showed that only the target task should be considered when selecting the public dataset. In Table 15 (Appendix D.1), we find that the best transfer performance can be achieved with public datasets from the same task family, even if the datasets are not very similar and have, e.g., different numbers of classes. Our framework assumes that the model provider has knowledge of only the task domain, not the user’s actual data. This is a reasonable assumption, as task metadata is often available in practical settings. \nFor instance, cloud providers and ML platforms (e.g., Google Vertex AI, OpenAI’s API) typically encourage users to specify the task type or provide task demonstrations (i.e., few-shot learning) when fine-tuning models to optimize performance [A]. Similarly, many AI service providers, such as NVIDIA’s NeMo framework, offer task-specific APIs explicitly designed for applications like chatbots, summarization, and generation, which inherently expose task-level information [B]. \n\n>**why soft prompts trained on a distilled model remain transferable？Theoretical insights or at least empirical ablations on feature space alignment between the small and large models**\n\nWe conducted an ablation study to investigate the impact of feature alignment. We use a model with the same architecture and initial parameters as our distilled 2-layer LLaMA. Instead of applying knowledge distillation on the BookCorpus dataset, we directly fine-tune this model on BookCorpus to achieve comparable performance to the distilled smaller model. This model has no feature space alignment with the large model. We report this model’s transfer performance.\n | Dataset | PT on Distilled model | PT on non-distilled model | Transfer with the distilled model | Transfer with non-distilled model |\n|-|-|-|-|-|\n| SST2| 78.78 | 78.24  | 90.02| 85.38|\n| IMDB| 79.95 | 79.40 | 87.29| 82.45|\n| tweet   | 54.12| 54.79| 61.15  | 45.65 |\n| arisetv | 77.92| 72.07| 86.71 | 70.65 |\n| mpqa    | 83.82 | 84.16 | 87.37 | 81.52|\n\nTransfer results for the distilled model are always higher. This indicates that using knowledge distillation to preserve alignment between small model and big model is essential in our designed prompt transfer procedure. \n\n>**How to select $\\alpha$ in the prompt transfer loss (Equation 3)? Have you explored dynamic or task-adaptive weighting schemes for $\\alpha$? Could tuning $\\alpha$ differently for various LLMs improve performance?**\n\nWe conducted a thorough ablation study on $\\alpha$ (see Table 19), evaluating transfer performance across $\\alpha$ values ranging from 0.0 to 1.0. The results indicate that multiple $\\alpha$ values lead to good transfer performance, indicating the robustness of our POST to the choice of $\\alpha$. \n\nAdditionally, we clarify the intuition behind the transfer loss design in Appendix D.5 and propose a heuristic method (Equation 6) for finding an optimal $\\alpha$, which supports tuning $\\alpha$ differently for different LLMs to improve performance in practice.\n\n>**How dataset properties (e.g., domain similarity, label space overlap, dataset size) impact transfer success?**\n\nWe investigated the influence of the domain similarity of the public dataset on the transfer performance (see Appendix D.1). The findings demonstrate that selecting a public dataset with a task domain similar to the private task is helpful for effective prompt transfer.\n\n>**What if the public dataset is too different from the private task dataset?**\n\nTable 15 presents the transfer performance using public datasets from different task families. It is observed that the best transfer performance can be achieved when using same-domain data, while datasets with different tasks can lead to performance degradation.\n \n---\n\nWe provide the reference for all the responses.\n\n**References:**\n\n[A] OpenAI Fine-Tuning Guide – https://platform.openai.com/docs/guides/fine-tuning\n\n[B] NVIDIA NeMo Framework – https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html\n\n[C] Duan, H., Dziedzic, A., Papernot, N., and Boenisch, F. “Flocks of stochastic parrots: Differentially private prompt learning for large language models.” NeurIPS 2023.", "claims_and_evidence": "Yes, most claims are well-supported by empirical evidence, though some could benefit from clearer acknowledgment of limitations.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are appropriate for the problem. The paper uses knowledge distillation, differential privacy, and public dataset-based transfer, which align well with the goal of efficient and privacy-preserving prompt tuning. The evaluation includes multiple LLMs (RoBERTa, GPT2-XL, LLaMA2-7B) and diverse classification and generation tasks, making the results robust. However, the impact of public dataset choice on privacy and transferability could be further analyzed.", "theoretical_claims": "The paper does not have any theoretical claims.", "experimental_designs_or_analyses": "Yes, the experimental design is generally sound. The authors use multiple LLMs, diverse classification and generation datasets, and compare POST against relevant baselines (Zero-Shot Transfer, DP-OPT). The LiRA attack for privacy evaluation and runtime analysis for efficiency are well-executed.", "relation_to_broader_scientific_literature": "The paper's contributions relate to broader literature in several key ways:\n\nSoft Prompt Tuning & Transfer: It builds on prior work on soft prompt tuning (Shin et al., 2020; Lester et al., 2021) and addresses the challenge of cross-model prompt transfer, improving on methods that require private data sharing (Su et al., 2022) or suffer from poor performance (Wu et al., 2023).\n\nPrivacy-Preserving Adaptation: Unlike prior differential privacy (DP) methods that protect model outputs (Duan et al., 2023), POST ensures privacy during prompt tuning without leaking private data.\n\nKnowledge Distillation for Transfer: It repurposes knowledge distillation (Hinton et al., 2015; Sanh et al., 2019) to enhance soft prompt transfer, rather than just compress models.\n\nEfficiency & Benchmarking: The framework reduces computational costs compared to full fine-tuning and outperforms existing private prompt transfer methods (Hong et al., 2023) across multiple tasks.", "essential_references_not_discussed": "The paper provides a thorough review of related work, covering soft prompt tuning, knowledge distillation, and privacy-preserving techniques. It cites key contributions in these areas, including prior work on soft prompt transfer (Su et al., 2022; Wu et al., 2023), differential privacy (Dwork et al., 2006; Duan et al., 2023), and knowledge distillation (Hinton et al., 2015; Sanh et al., 2019).", "other_strengths_and_weaknesses": "Strengths:\n\n1) Innovative Transfer Method: The paper proposes a novel framework, POST, which introduces a structured approach to soft prompt transfer via knowledge distillation. This is a significant technical improvement over prior works that either required private data access or suffered from severe transfer degradation.\n\n2) Efficient Soft Prompt Tuning: By using a distilled smaller model for local tuning, POST significantly reduces the computational cost of optimizing prompts compared to full LLM fine-tuning.\n\n3) Strong Empirical Support: The framework is rigorously tested across multiple model architectures (RoBERTa, GPT-2 XL, LLaMA-2 7B) and five classification plus two open-ended generation tasks, showing strong transfer performance while preserving privacy.\n\n\nWeaknesses:\n\n1) Limited Theoretical Analysis of Transferability: While the empirical results support the effectiveness of POST, the paper lacks a theoretical discussion on why soft prompts trained on a distilled model remain transferable. The connection between knowledge distillation’s preservation of model decision boundaries and its impact on prompt representation is not analyzed in depth. Including theoretical insights or at least empirical ablations on feature space alignment between the small and large models would strengthen the argument.\n\n2) Prompt Transfer Loss Function Justification: The prompt transfer objective consists of two KL-divergence terms (Equation 3), but there is no principled justification for the chosen balance parameter αα. The weighting scheme appears to be chosen heuristically based on downstream task performance. A more rigorous exploration of how αα interacts with factors such as the student model's generalization error and the zero-shot capabilities of the target LLM is needed.\n\n3) Prompt Transfer Loss Function Is Not Well-Justified: The prompt transfer objective (Equation 3) consists of two KL-divergence terms, but the balance between them is controlled by an empirically chosen hyperparameter αα. There is no discussion on how to select αα beyond empirical tuning, nor any exploration of its effect on convergence or transfer quality.", "other_comments_or_suggestions": "Section 4.3 (Privacy-Preserving Prompt Transfer): The explanation of the loss function (Equation 3) would be clearer with a brief discussion on the intuition behind the two KL-divergence terms and why they are necessary for effective transfer.", "questions_for_authors": "1) How does the choice of public dataset affect the quality of prompt transfer?\n\n    1) The paper mentions that a small public dataset is used for transferring the prompt but does not provide a systematic analysis of how dataset properties (e.g., domain similarity, label space overlap, dataset size) impact transfer success.\n    2) Could the method fail if the public dataset is too different from the private task dataset? Have you experimented with mismatched public datasets?\n\n2) Why does the KL-divergence transfer loss (Equation 3) use a fixed weighting factor αα, and how was it selected?\n\n    1) The transfer objective combines two KL-divergence terms with a fixed αα, but there is no theoretical justification for how αα should be set.\n    2) Have you explored dynamic or task-adaptive weighting schemes for αα? Could tuning αα differently for various LLMs improve performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper submission got initially a Reject, two Accept, and one Weak Accept ratings. While reading the discussion in the rebuttal phase, I acknowledge that the problem setting can be challenging to achieve (Reviewer SqSB). However, as other Reviewers state there are some innovations: combination of knowledge distillation, differential privacy, and prompt tuning (Reviewers f1xX and Qi3Y). After an internal discussion w/ the Reviewers, all of them kept their ratings. Given the innovations, I can recommend the acceptance of the paper. However, I think the submission has to discuss the limitations of the proposed approach raised by Reviewer SqSB and include the additional results posted in the rebuttal discussion.
Decision: Accept (poster)