DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer
Authors: Junyuan Hong
Jiachen T. Wang
Chenhui Zhang
Zhangheng LI
Bo Li
Zhangyang Wang
Venue: ICLR 2024 spotlight (2024)
Keywords: large language model
privacy
prompt tuing
Primary area: societal considerations including fairness, safety, privacy
TL;DR: DP-OPT is the first end-to-end solution that makes local LLMs use local private data to engineer privacy-preserving prompts.

Abstract: Large Language Models (LLMs) have emerged as dominant tools for various tasks, particularly when tailored for a specific target by prompt tuning. Nevertheless, concerns surrounding data privacy present obstacles due to the tuned prompts' dependency on sensitive private information. A practical solution is to host a local LLM and optimize a soft prompt privately using data. Yet, hosting a local model becomes problematic when model ownership is protected. Alternative methods, like sending data to the model's provider for training, intensify these privacy issues facing an untrusted provider. In this paper, we present a novel solution called Differentially-Private Offsite Prompt Tuning (DP-OPT) to address this challenge. Our approach involves tuning a discrete prompt on the client side and then applying it to the desired cloud models. We demonstrate that prompts suggested by LLMs themselves can be transferred without compromising performance significantly. To ensure that the prompts do not leak private information, we introduce the first private prompt generation mechanism, by a differentially-private (DP) ensemble of in-context learning with private demonstrations.  With DP-OPT, generating privacy-preserving prompts by Vicuna-7b can yield competitive performance compared to non-private in-context learning on GPT3.5 or local private prompt tuning.
Codes are available at https://github.com/VITA-Group/DP-OPT.

Published as a conference paper at ICLR 2024
DP-OPT: MAKE LARGE LANGUAGE MODEL YOUR PRIVACY-PRESERVING PROMPT ENGINEER
Junyuan Hong'. Jiachen T. Wang', Chenhui Zhang', Zhangheng Li', Bo Li", Zhangyang Wang' UUiierrsity of Texas at Austin, "Princeton University, 'MIT, *University of Chicago (jyhong, zoharli, at atlaswang @utexas edu, tianhaooanggggrineeeee.ee. edu chenhui ttriitttt edu, bol wolleuucccgg.... edu
ABSTRACT
Large Language Models (LLMs) have emerged as dominant tools for various tasks, particularly when tailored for a specific target by prompt tuning. Nevertheless, concerns surrounding data privacy present obstacles due to the tuned prompts' dependency sensitive private information. A practical solution is to host a local LLM and optimize a soft prompt privately using data. Yet, hosting a local model becomes problematic when model ownership protected. Alternative methods, like sending data to the model's 'ss provider for training. intensify these privacy issues facing an untrusted provider. In this paper, we present novel solution called CetrenniiallPPrivattt Offsite Prompt Tuning (DP-OPT) to address this challenge. Our approach involves tuning discrete prompt on the client side and then applying it to the desired cloud models. We demonstrate that prompts suggested by LLMs themselves can transferred without compromising performance significantly. To ensure that the prompts do not leak private information, we introduce the first private prompt generation mechanism, by a differentially-rrive in-context learning with private demonstrations, With DP-OPT, generating privacy- preserving prompts by Vicuna-7b can yield competitive performance compared to non-private in-context learning GPT3.5 local private prompt tuning. Codes are available at https: //tpt/gghtttttttt thub. com /VIIAAAA- TALTTA-Group/Ae Group/DP-OPP
(DP) ensemble
INTRODUCTION
When Large Language Models gain vast knowledge and versatile ability from large- pre-training. prompt engineering has surfaced as the most effective, cost-efficient, and adaptable method to tailor LLMs for a range of downstream applications. In contrast to the resource-heay optimization of model parameters, prompt engineering merely necessitates API access and iteratively refines prompts based on the validation of training instances. Though manual prompt engineering has achieved impressive performance in various tasks (Petroni et al., 2019: Zhou et al., 2022), it often requires decent human experience in prompt designing and domain knowledge for downstream tasks, including legal judgement (Trautmann et al., 2022), healthcare (Wang al., 2023b) and art (Oppenlaender et al., 2023). To mitigate the high costs, data-driven prompt tuning was proposed to automate the process. The most prominent example of this is soft prompt tuning, where prompts are characterized as trainable embedding vectors and are refined using a collection of training instances (Houlsby et al., 2019; Roberts al., 2019; Brown et al., 2020; Chen al., 2022).
However, one major barrier to the applications of prompt tuning is data privacy. When searching for validate prompt for an LLM API, such as ChatGPT, there is need to upload a multitude of training samples for evaluation queries. In -eisiic-eensitive scenarios, the operation could be prohibited due to two concerns. 1) Data Confidentiality. Certain data, like medical histories, proprietary system logs, and personal messages, are inherently confidential and should not be transmitted beyond local devices, internal computing systems, and mobile phones. 2) Information Leakage. Resources derived from private data might inadvertently contain personally identifiable information. For instance, personal identifiers like names, residential addresses, and contact numbers present in the fine-tuning data (Lukas et al., 2023) during the pre-traie pre-training phase (Carlini et al., 2021) might be retrievable from the adjusted parameters. Even with a limited parameter set, such as prompts, the potential for data breaches remains significant (Duan et al., 2023a).

---

Published as a conference paper at ICLR 2024
Local Private Prompt Tuning
Cloud Inference
Private data
Trusted Environeent
Private Instruction
Query
Local model
DP-OPT
Figure Differentially-Prine Privvate Offsite Prompt Tuning (DP-OPT) works as an intermediate layer between local data and cloud models. Leveraging local model, OP-OPT can fine-tune a differentially- private prompt that can transfer to the target model.
straightforward approach would be manage the entire prompt process on the local device and offer services via an API. However, this becomes impractical when there's a preference for sophisticated closed-source model, not to mention the substantial costs involved in hosting and overseeing an LLM locally. For example, there is high demand for serving prompts with the most powerful LLMs, e.g.. GPT-3.5, to leverage the state-of-the-arr generation ability. Yet, the specifics and structure of GPT-3355 remain proprietary and undisclosed for protecting Intelligent Property (IP). Even if GPT proprietors are willing to support local prompt tuning by dispatching compressed models (Xiao et al., 2023), they could be confronted with the potential peril of losing their model's ownership.
In this paper, we propose Differentially-Privaaa Private Offsite Prompt Tuning (DP-OPT) to make LLM engineer private and transferable prompts for cloud-hosted LLMss, which is illustrated in Fig. 1. The crux of privacy protection is that DP-OPT operates exclusively on the client. Given a confidential training dataset, DP-OPT uses a few samples as demonstrations to guide local LLM to generate prompts. The local assistant LLM may be significantly smaller than the intended cloud-based LLMss. Such a prompt generation process is facilitated by a iifferentially-Privatt (DP) ensemble of in-context learning with disjoint private demonstration subsets. Our contributions are summarized as follows:
We proposed the first end-to-end framework where DP-OPT operates on client devices and data and yields front-end prompt for inference on the privacy-untrusted cloud. Our framework is the first solution that simultaneously protects (i) data confidentiality by keeping data local; (ii) information privacy by the DP noise mechanism; and (iii) cloud model ownership and IP by eliminating the parameter exposure of cloud models from local training.
We first show that discrete prompts automatically tuned by LLMs are transferable across models with favorable performance cloud models. The finding motivates the offsite prompt tuning (OPT) framework with local prompt tuning and cloud inference.
We then provide the first Differentially--rivaal mechanism for generating private prompts without gradients or public data. The privacy costs can be tightly bounded during prompt tuming.
Empirically, our method presents an outstanding performance on multiple language tasks. Prompts tuned open-source Vicuna-7b (Chiang al., 2023) can achieve significant perfor- mance gains across 4 tasks after transfer to closed-source deterogeneous-architeee. models (GPT3.5) or open-source models (Llama-2 (Touvron et al., 2023) Vicuna-33b).
RELATED WORK
Discrete Prompt Tuning. On the rise of pre-trained generative language models, discrete prompt tuning (Shin et al., 2020) was introduced to amplify the inference capability of LLMs through demonstrations (Brown al., 2020) or informative instructions (Prasad al., 2022: Shin al., 2020). The method is orthogonal to the soft prompt tuning (Lester et al., 2021; Liu et al., 2021) that optimizes continuous prepended embeddings instead of discrete tokens and is therefore not favored for API-only models. One line of the discrete prompt tuning aims to improve the search efficiency by gradient-free phrase editing (Prasad al., 2022), reinforcement learning (Deng et al., 2022), embedding optimization (Shi et al., 2022), and senjicted-graddeen optimization (Wen et al., 2023). Another line of work does not rely on search or optimization algorithms and was initiated by Automatic Prompt Engineering (APB) (Zhou et al., 2022) that employed LLM to prompt themselves. Later, Deep Language Network (DLN) (Sordoni et al., 2023) improves APE by backward updates and stacked language models. Though these methods achieve great progress in discrete prompt tuning approaching the soft prompt results, the privacy risks or protection of the generated prompts have not been studied yet. In our work, we first highlight the advantage of DL.N prompts in transfer learning.
2

---

Published as a conference paper at ICLR 2024
Essentially different from the negative transfer effect presented in recent work (Wen et al., 2023), we show that DLN prompts can transfer to and work better on larger models than on the source models, namely positive transfer. Mcanwhile, provide the first solution to ensure the privacy of the gradient-free algorithms that demonstrate strong empirical performance compared to in-context learning and previous private gradient-based competitors.
Privacy Risks in Prompt Tuning. LLMs have been shown to possess the ability to memorize
data, not just from extensive pre-training datasets (Carlini et al., 2021; 2022b; Wang et al., 2023a), but also from more concise private prompt-tuning datasets (Duan et al., 2023a). Consequently, it becomes imperative to shield private data from potential exposure in released prompts. To illustrate this vulnerability, Duan et al. (2023a) employed membership infferncce attack (Shokri et al., 2017; Carlini et al., 2022a) to prohe the susceptibilities of tuned soft prompts, revealing significant success rate for the attacks. Though the risk of soft prompt tuning has been highlighted, it is unclear how the discrete prompts exemplified by DLN will leak private information. Our research uncovers direct data leakage via prompts crafted by DLN, underscoring the need for novel discrete defense mechanisms.
Mitigating Privacy Leakage in Prompt Tuning. As a golden standard for bounding privacy risks, there is increasing interest to incorporate differential privacy (DP) (Dwork, 2006) into prompt tuning for privacy protection. Closely related to our work, PromptPATE utilized a DP ensemble approach to label public data. Using these as in-context examples, they devised a discrete prompt tailored for few-shot learning on designated models (Duan et al., 2023a). However, the work assumes a set of non-private data which may not hold in practice. In the absence of public datasets, Duan et al. (2023a) also showed the viability of DP-SGD (Abadi et al., 2016) in the realm of soft prompt tuning. In parallel, DP In-Context Learning (ICL) (Wu et al., 2023) advocated for ensembling multiple in-context samples to predict classification labels directly, which limits the query times by the number of available private samples. Instead of prediction, Tang et al. (2023) proposed ensemble generation by prompting LLMs with examples. The generated samples can be used for ICL and enable infinitely many queries. Yet, these approaches were used with the full exposure of private samples to the target model, rendering it infeasible when model vendor is untrustworthy. From a technical standpoint, our strategy differs from these methods on ensembling outputs to produce instructional prompts. Specifically, we ensemble in-context examples to produce instructional prompts rather than labels. This design facilitates the effortless migration of trained prompts to many models, eliminating the need for extra training cloud models.
Another focal point among the community is the sanitization of texts (Feyisetan et al., 2020; Xu et al., 2020; Carvalho et al., 2023; Du al., 2023; Utpala et al., 2023). These works introduce randomness at the word level and are able to achieve the privacy guarantee in terms of local differential privacy (LDP) (Kasiviswanathan et al., 201 1) or similar definition called metric differential privacy. Recent advancements in this field (Mattern et al., 2022; Utpala et al., 2023) combine the idea of perturbation with paraphrasing based on fine-tuning or zero-shot prompts.
Prompt Ensemble. In our work, we use a prompt ensemble together with a noise mechanism for privatization. Previously, ensembling multiple prompts has been explored for improving inference quality with large language models. Pitis et al. (2023) proposed to boost the in-context learning by ensembling multiple few-shot prompts, which is effective on classification tasks. Similarly, Hou et al. (2023) query an LLM by samples enveloped in diverse prompt templates, exhibiting ofte-off-the-ar classification performance. Yet the line of works focuses on few-token prediction instead of long-context generation. As pioneering effort, the ensemble technique was harnessed for linguistic generation, leveraging long short-term memory networks equipped with attention mechanisms (Juraska et al., 2018). Subsequently, this ensemble paradigm found its application in large language transformers. Hewitt et al. (2021) orchestrated an ensemble of token logits sourced from lightweight--inetuned language model and fully-fintuned one generate texts, thereby bolstering the robustness of the generated content. Different from (Hewitt et al., 2021), our method leans on voting-based ensemble of prompts from multiple prompts, which facilitates the application of differential privacy. Novel to this work, we show that the ensemble of hundreds of prompts can be effective in generating a long context with DP noise.
PRELIMINARIES
Large Language Models (LLMs) and Prompt Tuning, LLMs like GPT(Radford et al., 2018;
OpenAI, 2023), Llama (Touvron et al., 2023), and OPT (Zhang et al., 2022) pre-trained to generate
3

---

Published as a conference paper at ICLR 2024
tokens conditioned on previous context. Generally, the language generation can be represented as a conditional probability Pím(y)a) where I' is a prompt and y is the corresponding output. The temperature t 2 0 can be increased to generate more diverse responses. We use to represent front-end prompt, e.g., a task instruction that guides the LLM to think and conclude a response. Prompt funing optimizes a prompt that can be wrapped with the input query x in template (* and improves the response quality of LLM (1 (TT, x)), e.g., the accuracy on text classification.
Differential Privacy (Dwork et al., 2006) stands as the gold standard for assessing the privacy guarantee of machine learning algorithms. DP has gained significant attention among the privacy community as a robust, quantifiable privacy notion, thereby becoming the de-facto choice in privacy protection. Formally, we use D' E N to denote two datasets with an unspecified size over space X. We call two datasets D and D' adjacent (denoted as D D¹) if we can construct one by adding/removing one data point from the other, e.g.. D = D' U {z) for some E X.
Definition 3.1 (Differential Privacy (Dwork et al., 2006)), For e, 5 2 O, a mechanism M N + y is (e, -i-iiffrrntially private if for every pair of adjacent datasets D ~ D' and for every subset of possible outputs E' S V. we have PrM(M(D) E E] s e PIM(M(D') E E] + here the randomness is over the coin flips of M.
The above definition indicates that for an arbitrary pair of neighboring datasets, a DP algorithm should yield statistically indistinguishable output distribution, preventing adversaries from distinguishing between the outcomes from the datasets. In our study, the mechanism M being considered is the prompt generation algorithm.
4 METHOD
Assumptions. Due to the convenience and high performance of cloud models, a common interest for a client to tune prompt that can be served on the cloud. We assume that client has set of data D that will be used for prompt tuning but has strict constraints on the data usage as follows. 1) Data Confidentiality. The client data cannot be shared with the cloud-model vendor. 2) Information Privacy. The tuned prompt should not leak private information about the client data, including but not limited to enclosing private contents, and inferrable private information. 3) Model Ownership. On the cloud, model ownership could be a concern and therefore parameters should not be shared with the client.
Threat Model. We assume an adversary on the cloud-model vendor side which aims to gain private information (e.g., membership information) from the private dataset stored in the client device. The adversary can only get tuned prompt provided by the client but can leverage any available LLMs for attacking. The real-world consequence of privacy leakage through released prompts could result in violation of privacy regulation, e.g., GDPR (2016). Concretely, private identifiable information (e.g.. names) could be exposed in prompts. Empirically, the privacy risks have been identified in existing works using viable attacks (Wang et al., 2023a; Duan et al., 2023b). Especially, Liu (2023) shows that private instructions behind Bing can be extracted merely by adversarial prompts.
Main Idea. To preserve the data confidentiality and privacy, we propose iiiierenttilll-Privaat Offsite Prompt Tuning (DP-OPT) which isolates the prompt tuning and data from the cloud model. The general idea of DP-OPT includes two steps: 1) Private Prompt Engineering: Engineer private prompt ı by fully localized model and datasets, i.c., π ~ DP-OPT(D, (LM(-)); 2) Prompt Transfer: Deploy prompts cloud model for public inference, i.e., y t- PalonaLL (t) F(I, x)), where F() is forward template.
To achieve the goal, the two major technical challenges are: (1) How to engineer model-transferable prompt? (2) How to guarantee that the prompts do not leak private information? We will answer the two questions sequentially in the following two sections.
4.1 TRANSFERABLE DISCRETE PROMPTS ENABLE OFFSITE PROMPT TUNING
To make a prompt transferable across models, it is necessary to have a discrete prompt that is not bonded with any model-specific embeddings tokenization strategies. Importantly, recent advances show that discrete prompts are naturally transferable (to some extent) across domains. Wen et al. (2023) demonstrated that the projected soft prompts tuned by their method, PEZ, on GPT-2 (755M) can be used on larger GPT-2 variants (1.3B) or different architectures, like OPT (6.7B). However, such transfer was shown suffer from significant loss of performance. As reported in their paper,
4

---

Published as a conference paper at ICLR 2024
the prompt tuned on GPT-2 (755M) would lose 10.9% absolute accuracy on SST-2 if transferred to OPT (2.7B) and 15.7% to OPT (6.7B). We extend the same experiment to training on Vicuna-7b and testing Da VaVinii-003. Similarly, we observe a 6.9% loss of accuracy upon transfer. The situation could worsen on smaller datasets according to our extended experiments in Appendix B.2.
As discussed in (Wen et al., 2023), the key reason for the poor transferability of projected prompt tuning is the incoherence of the tuned prompts. For example, the method for SST-2 with the fluency constraint produced "negative vibeThis immatureollywood MondarinolywwodThis energetic screenplay." " This means that the method does not generate a semantically transferable but might still heavily hinge on the embedding space to prompt the training model.
Observing the limitation of projected prompt tuning, we intend to find a semantically transferable prompt. To avoid the pitfalls of the embedding space, we look for method that does not backward the signal through the embeddings but produces fluent and coherent prompt. Inspired by the automatic prompt engineering (APE) (Zhou et al., 2022; Sordoni et al., 2023), we conjecture that the LLM itself is an ideal tool for this purpose. Given a well-trained LLM, APE uses samples as context and prompts the LLM to generate task instruction, which is naturally fluent, coherent, and perhaps transferable. As the task instruction is not optimized in the embedding space but in the output space, it barely relies on hidden neural connections for enhancing inference accuracy. Instead, it captures the explicit and generalizable task semantics that may be reused and strengthened by stronger LLMs. Therefore, we hypothesize that discrete prompts crafted by one LLM may transfer to another with wuget-modeeldependee performance on the same task.
Make LLM Prompt Engineer. To gain the best performance, we consider the state-of-the-art APE method, Deep Lan- guage Network (DLN) (Sordoni al., 2023), that mimics gradient-baste -based opti- mization to use forward and backward train prompts on a dataset D = {(x,y)} with input-output pairs (x. 1) Prompt Generation. In the forward pass, an LLM is prompted via a forward template F(z, x) to predict labels
Source
Target
Task
Vicuna-70 7b
Llama-2-70b
A
DaVinci 103
A
SST-2
92.8(0.2)
93.3(1.8)
0.5
92.7(0.3)
-0.1
Trec
59.9(5.7)
65.2(7.5)
5.3
70.7(3.9)
11.2
Mpqa
75.8(6.2)
78.0(2.3)
2.2
81.4(1.6)
5.6
Disaster
61.7(3.2)
73.1
11.4
77.0(1.9)
15.3
Table 1: DLN-1 can produce transferable prompts, bring- ing non-trivial gains (A). Accuracy (%) on test sets is reported with standard deviation in the brackets.
small batch of training samples S < {(I,)) ~ D),
i.e., ÿ ~ (M Far, m)). Then in the backward pass, the correct and incorrect predictions will be used as in-context examples for L.LM to generate a task instruction æ. Formally, π is sampled from (TT BBg (( y, 5)}, ))) where B2 is backward template. 2) Prompt Selection. With set of candidate prompts, DLN-1 yields the best prompt with the highest log probability on the training set.
LLM-Engineered Prompts Are Transferrable. To verify our hypothesis that the prompts generated by DLN-I are transferable across models and gain better accuracy, we let DLN-1 train prompts using a relatively small LLM, Vicuna-7b (Chiang et al., 2023). Then, the generated prompt is then model, 1lamm--2-706 and heterogeneous--rchhin architecture and closed-source model, Da 003. Experiments are carried out on four sentiment classification tasks. In Table 1, DIN-1 demonstrates competitive performance on the target model. Different from traditional observation, D even attains 8% accuracy gains after transfer to Da Vinci-003 on average and 4.9% to Llama-2-70b. Not surprisingly, the transferrable prompts generated by DLN-1 are also coherent and fluent as exemplified in Fig. 2.
applied to larger camoogeeeus-arcchieettee
4.2 DIFFERENTIALLY--PRINE OFFSITE PROMPT TUNING (DP-OPT)
Though leveraging DIN-1 can keep data confidential against the cloud model by transferring prompts, it does not provide any provable guarantee for privacy protection. Notice that DLN-1 feeds private samples to LLM to generate instruction prompts, which may leak private information. To unveil the risk, we present an example prompt from DLN-1 in Fig. 2, where three private examples are verbatim copies from the training set. The only difference between the generated examples and private examples is the capitalization,
LLM tends to make the sentence grammatically correct. Since LLMs are known to repeat words from prompts, it is not surprising that LLMs copy private in-content examples into generated prompts.
5

---

Published as a conference paper at ICLR 2024
D SST-2 prompt
Classify the input text as positive or negative. Use the correct output for cach input. Avoid phrases like
"might" or "probably", "carnage and". "i recommend" or words like "barely". Input: "(I) Actor Michel Serrault" Correct Output: positive Input: "(2) Unique residences" Correct Output: positive Input: "(3) Buy the movie milk when the TV cow is free" Correct Ouiput: negative Input: A
Leaked training samples:
(1) Eocror micche vermant)
]
( > (unigne renieence) ] (3) i Eflyy the movie mikk when the tv cow iree)
Figure 2: A DLN- I potential leakage in the prompt and semantically-nearess >> [leaked sample] from the training set.
-generated prompt is coherent but suffers from privacy leakage. We highlight the
Algorithm 1 DP-OPT (cp < oc) or OPT (co = 00)
Input: Training datasets D {(x, 3)} and Dval {(z, y)). LLM pim(() with generation temperature t, number of prompts N, and privacy parameters eo, do-
1: Initialize To with task description empty and II <- 0
2: D' + ++ {x, y, 0)ly 0 = (xx, v(z,y) y) E D}
Forward pass
3: for ni [1,.... .,,,,................. N} do
4: π ~ A.P-EnGen(T,, D'.
> Private Prompt Generation
5: II 1U (T*)
6: â + DP-Arrgax 50 EE Accuracy(T; Dval) DDall
b Private Prompt Selection
7: Output â
To defend the risk, we develop a DP variant of DLN-1, termed DP-OPT, which provides provable privacy protection through the DP noise mechanism. In DP-OPT, we privatize the two core operations in DLN-1: prompt generation and prompt selection, that directly access private data.
Private Prompt Generation. As demonstrated above, the main privacy leakage comes from non- private prompt proposals. In Algorithm 2, we develop a privatized version of the prompt generation. Specifically, we leverage the classic sampleaandaggregaul paradigm (Nissim al., 2007), where we partition the full batch of data into disjoint subsets. We then generate each token based on the voting results formed by querying the language model with each disjoint subset. While we can simply apply the commonly used Exponential Mechanism (EM) (McSherry & Talwar, 2007) to privately release the token with the maximum count, the naive application of EM may result high variance and poor performance as the token space can be as large as 30,000 Chiang et al. (2023). Fortunately, extending EM on large domain space has been studied in the DP community. In this work, we leverage the LimitedDomain mechanism (Durfee & Rogers, 2019) which reduces the domain space to only those tokens with top-k vote counts (with some privacy budget). We note that LimitedDomain has a small failure probability that will not output any token for the scenario where the highest vote count is not too high compared with the kth highest vote count. In this case, we retry to generate using the next batch of data. If we run into more than one failure case for generating single token, it means that the disjoint partitions do not have a majority agreement on single token choice and we terminate the token generation for this prompt.
Private Selection among Generated Prompts. With the generated prompt candidates, DLN-1 selects the best one by contradicting their performance on training samples. This may leak private information about the validation set when some private samples significantly affect the evaluation. To defend against such risks, we use the exponential mechanism to select the best- geseerenddd prompt that achieves the highest count of correct predictions on the validation set in a differentially private manner. Formally, given a histogram h, we define DP-Argmax* as Pr PrrDa-Argmax (h) = j] xx exp (ch;). Note that this part protects the privacy of the validation set, which is disjoint with the training set. Hence, the privacy cost of this part does not add up to the privacy cost of prompt generation.
We adopt the commonly used Rényi differential privacy (RDP) (Mironov, 2017) to track the privacy cost of Algorithm for generating prompts and ensure that the total privacy cost is within a prespeci- fied budget. To understand the asymptotic behavior of the number of generated tokens m, we provide Theorem A.3 in the appendix showing that the growth of e of the order of Vmeg. We defer the detailed privacy analysis to Appendix A.2.

---

Published as a conference paper at ICLR 2024
Algorithm 2 DP-EnsGen: Differentially-Privanl Ensemble Generation
Input: Max number of new tokens L. privacy parameters: 6o, dg. subsampling rate q, and predicted dataset D'.
1: Initialize output
and / +-
2: for < L do
3:
if < 1 0 then
4:
Sample a batch S - {(x, y. 9) ~ D'} by Poisson sampling with probability
5:
Partition S' into disjoint subsets of equal size (U)=15} S*)
6:
h = -Histogram((1))
(S n,, =)) for E [J]} Histogram of the next token
7:
i < oo then
8:
LimitedDomain(tan 10, eo. 6b) (Algorithm 4)
Select private top-1 token
9:
else
10:
arg max; h;
11:
if is EOS then Break
b Append generation
12:
if T = then
13:
if and T--1 then Break
14:
else
15:
+ 1+1
Output
[20-
Table 2: Test accuracy (%) with standard deviation the brackets. All trainable methods are trained Vicuna-7b. Bold methods are model-transferable and therefore tested Vinci-0003 PromptSGD and PromptDPSGI are not transferable and, therehy are tested on Vicuna-7b. The non-confidential baseline uses private data in prompts. Confidential and confidential-ande and-private prompts are trained local Vicuna-7b. We highlight the best and the second-best results in each setting as bold and underlined numbers, respectively.
Method
SST-2
Trec
Mpqa
Disaster
Average
ICL
94.7(0.4)
79.1(0.5)
88.8(0.1)
69.0(5.9)
82.9
PromptSGD (Vicuna-7b)
93.7(1.1)
56.1(6.7)
88.1(0.8)
79.4(1.2)
79.3
DLN-1
92.7(0.3)
70.7(3.9)
81.4(1.6)
77.0(1.9)
80.4
OPT (ours)
92.4(0.5)
71.5(0.8)
85.8(1.2)
79.0(0.9)
82.2
PrompiDPSGD (Vicuna-7b)
90.4(1.7)
32.3(3.1)
84.2(4.0)
78.5(0.4)
70.5
0-shot
92.4(0.0)
51.8(0.2)
84.5(0.1)
76.4(0.2)
76.3
DP-OPT (ours)
92.2(0.8)
68.7(6.5)
85.8(0.7)
78.9(0.3)
81.4
EXPERIMENTS
Tasks. Our study focuses on sentiment classification tasks. We use SST-2 from the GLUE bench- mark (Wang al., 2018) which includes 6.7 10 samples. Trec and Mpqa (Lu al., 2021) and Disaster (Bansal et al., 2019) are smaller datasets consisting of fewer training samples. Both SST2 and Mpqa are sentiment classification tasks for positive negative reviews. Trec is predict 6-option label for question types. Disaster analyzes if the sentence is relevant to disaster.
Setup. We use Vicuna-7b as the local model to train prompts by default. For DP algorithms, we follow the common practice to set the privacy budget as 8 and 5 = D/ where [D] is the training size (De et al., 2022; Sander et al., 2023; Duan et al., 2023a). More experiment details are deferred to Appendix B.I.
5.1 PRIVATE OFFSITE PROMPT TUNING
Table 2, we evaluate the effectiveness of DP-OPT in generating private prompts for Da Vinci-003, Our private baseline is the PrompiDPSGD which uses DPSGD (Abadi et al., 2016) to tune soft prompts (Duan et al., 2023a). We also include the non-private variant PronptDPSGD, PromptSGD, for comparison. As non-private baseline, we follow (Sordoni et al., 2023) to include the In-Context Learning (ICL) with class-balanced demonstrations that have secondary best performance compared to DLN-1 in the sentiment classification. To show the improvement of training, evaluate the

---

Published as a conference paper at ICLR 2024
Table 3: Transfer test accuracy (%) on different models with standard deviation in brackets. Trainable methods (bold) are executed on Vicuna-7b. ICL is represented as an upper bound without confi- dentiality, We highlight the best and the second-best confidential methods as bold and underlined numbers, respectively.
Task
Method
Vicuna-7b 7b
Vicuna-33B
Llama-2-13b
Llama-2-70b
DaVinci-003
Average
ICI
90.1(1.5)
94.2((0.)
94.9(0.4)
95.5(0.6)
94.7(0.4)
93.9
DLN-1
92.8(0.2)
92.5(2.2)
93.5(1.0)
98.3(1.8)
92.7(0.3)
93.1
SST-2
OPT
89.7(2.7)
91.8(1.7)
92.1(0.9)
94.2(1.3)
92.4(0.5)
92.0
DP-OPT
89.5(2.6)
92.5(0.3)
92.7(1.5)
93.0(1.6)
92.2(0.8)
92.0
ICL
49.8(18.9)
66.9(3.9)
77.8(7.8)
72.7(8.7)
79.1(5.1)
69.2
DLN-1
39.9(5.7)
55.2(8.2)
38.6(0.5)
65.2(7.5)
70.7(3.9)
57.9
Trec
OPT
62.1(1.2)
72.5(10.2)
53.7(14..)
52.3(1.6)
70.4(2.7)
62.2
DP-OPT
65.3(4.3)
(9.7(15.1)
61.9(2.2)
53.6(3.8)
68.7(6.5)
63.8
ICL
85.4(2.0)
85.6(0.9)
86.0(0.9)
87.8(0.2)
88.8(0.1)
86.7
DLN-1
75.8(6.2)
64.8(8.7)
66.3(10.3)
78.0(2.3)
81.((6.)
73.2
Mpqa
OPT
80.8(1.2)
75.6(1.6)
69.5(9.8)
84.5(1.1)
85.8(1.2)
79.2
DP-OPT
80.7(3.3)
67.8(6.7)
82.8(1.6)
81.7(3.1)
85.8(0.7)
79.8
ICI
58.9(1.3)
52.0(4.2)
61.2(3.4)
69.6(4.5)
69.0(5.9)
62.1
DLN-1
61.7(3.2)
62.4(4.3)
66.3(8.3)
73.1(1.6)
77.0(1.)
68.1
Disaster
OPT
67. 67.4(5.1)
60.6(9.1)
57.8(10.0)
49.1(12.0)
79.0(0.9)
DP-OPT
65.6(0.3)
53.7(0.0)
67.9(1.5)
42.9(0.3)
78.9(0.3)
61.8
initial instruction (0-shot) wrapped in the forward template. DLN-1 serves as the state-of-the-art LLM-driven tuning method for offsite transfer.
We demonstrate that offsite prompt tuning via OPT and DP-OPT can significantly enhance prompt efficacy compared to the initial instruction (O-shot). For three tasks (SST-2, Mpqa, and Disaster), OPT and DP-OPT approach the performance the non-private baseline, ICL. In the absence of DP, OPT boosts performance for these three tasks relative to DLN-1, likely due the ensemble's ability to bolster model generalization.
While automatic discrete prompts often fall short compared soft prompts in the literature (Wen al., 2023), our research indicates that using transfer prompts with Da Vinci-000 can somewhat alleviate this discrepancy. When operating within the same DP budget, both OPT and DP-OPT demonstrate superior results compared to PrompIDPSGD and are nearly on par with the non-private PromptSGD. On the SST-2 dataset, DP-OPT matches the accuracy of PromptSGD. For Trec, discrete prompts consistently surpass the performance of soft prompts. The underwhelming results of PromptDPSGD can be traced back to weak zero-shot performance on Trec and the noisy gradient descent. However, when the zero-shot approach excels, as seen in our other three datasets, PromptDPSGD will yield better accuracy.
In Table 3, we assess the transferability of the prompts produced by Vicuan-7b on various larger models including Vicuna-33b, Llama-2-13b. 13b,, Llama-2-70b (Touvron et al., 2023) and DaVinci-003 (text generation version of GPT3.5) (Ouyang et al., 2022). The experiment yields several intriguing implications. 1) The closed- -ooseedd-urceeeeeee model, Da Vinci-003, exhibits greater stability in transfer compared to its apen-sourced counterparts, where DP-OPT presents competitive performance compared to non-private baselines. Such stability offers more reliable predictions in various applications and therefore encourages clients to pair DP-OPT with the closed-source DaVinei-003. Without the DP noise mechanism, the ensemble method (OPT) itself enhances prompt quality relative to DLN-1 on Vicuna-33b and Llama-2-13b. 2) We observe discrepancy in DLN-1's performance on Trec, which is considerably lower than the figures presented in (Sordoni et al., 2023). It seems that Vicuna-7b struggles with the complexities of the 5-way classification task present in the Trec dataset when engineering prompts. This limitation could be result of architectural constraints training nuances specific to Vicuna-7b.
5.2 ABLATION STUDIES
Privacy-utility Trade-off. order to examine the privacy-utility trade-off of DP-OPT, we con- duct an ablation study with varying privacy parameters and report the test accuracy of DP-
8

---

Published as a conference paper at ICLR 2024
OPT SST2 prompt
Classify the input text as positive or negative. Student successes Input: "(I) the movie was a masterpiece"
Correct Output: positive Input: "(2) the movie was a disaster" Correct Output
Leaked training samples:
hurdly musterpiecee
amateurisk)
fiie disustes
OP-OPT SST2 prompt with budget =
Classify the input text as positive negative. Student successes Input: "(1) movie that a masterpiece"
Correct Output: positive Input: "(2) the movie was disaster" Correct Output: negative Input
Leaked training somples. (3)
Johe movie hundly muntepiieee]
(2)
tte movie ia amatvarisk)
(thk movie is dinunees
DP-OPT SST2 prompt with budget =
Classify the input text positive negative. For positive text: + "(1) The film masterpiece." " "(2) a collectively stellar performance" + "(3) The book was a delight"
Leaked training samples:
(1) An the movir is hurdly a mosterpiice]
(2) i fthr comundy was finnyy (3)
Eis kearffed stryy
B [/wor a guiliy plasouue
DP-OP OP-OPT SST2 prompt with budget 2
Classify the input text as positive or negative.
Figure Examples of Generated Prompts. OPT/DP-OPT tends to generate pseudo examples (blue text) which do not belong to the training set. We highlight potentially-leakdd samples and semantically-neares retrieved examples are in Table 10.
> |[raining samples] (there might be multiple such samples). More
OPT the SST-2 dataset with different test models, shown in Fig. 4. In the smallest model, Vicuna-7b, we observe a trade-off between accuracy and privacy: when the privacy budget (c) reduces to the accuracy drops 86% approximately. The accuracy drops because when the budget is limited, the LimitedDomain will prohibit DP-OPT from generating any new content. Interestingly, such a trade-off is greatly mitigated when scaling up model sizes. We notice that the LLM can still repeat the initial instructions even if under very limited budget. Thus. the mitigation can be attained when larger LLMs present strong zero-shot ability.
jama-2113
Viruns Durieci
Vicum 53b
Examples of Privacy Leakage in Generated Prompts. In Fig. and Fig. 3, we present examples of generated prompts that potentially leak private samples from the training dataset. To find such examples, we perform semantic similarity search to retrieve the training sentence closest to each forged demon- stration (dctails in Appendix B. In the prompts generated by DLN-1 (Fig. 2), we observe verbatim copies of training examples. In contrast, OPT and OP-OPT will modify a few words when generation, potentially due to the randomness of the ensemble. In Fig. 3, The word "is" in training example "the movie a disaster" is replaced with "was" in an OPT-generated prompt. Actually, "the movie is" is a common format in the training set, for example, "the movie amateurish". On the other hand, IP-OPT-generrde privacy-preserving preserving behaviors due to their formal privacy guarantee. Especially, when get smaller, there are fewer word copies. Our observations the generated prompts imply that verbatim copies of training examples are diminished by reducing in DP-OPT.
2.5 5.0
75
10.0
12.5
35.0
Figure 4: Privacy-utility trade-off of DP-OPT. Smaller indicates stricter privacy protection.
prompts exhibit privacy-
6 DISCUSSION AND CONCLUSION
With the rising popularity of prompt tuning, our research endeavors to extend this tool to applications with heightened privacy concerns. We introduce the pioneering end-to-end system designed to derive differentially-prin -private prompts from confidential training datasets and deploy these prompts on cloud models. Our approach is underpinned by theoretical validations of its privacy assurances, and through empirical analysis, we highlight the advantageous balance it strikes between utility and data privacy caused by the strong performance of scaled L.Mss.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The work of Z. Wang is in part supported by the National Science Foundation under Grant IIS- 2212176. This work is partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, No. 2229876, DARPA GARD, the National Aeronautics and Space Admin- istration (NASA) under grant No. 80NSSC20M0229, the Alfred P. Sloan Fellowship, the Michael Hammer Fellowship MIT, and Princeton's 's Gordon S. Wu Fellowship. Portions of this research conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing', a composable computing cluster (He et al., 2023). We also want to thank anonymous reviewers for their constructive suggestions and comments.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. In CCS: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '16 pp. 308-318, New York, NY, USA, 2016. ACM. ISBN 978- 978-1-4503-4139-4 doi: 10. N1111///776749,2988111
Trapit Bansal, Rishikesh Jha, and Andrew McCallum. Learning to few-shot learn across diverse natural language classification tasks. arXiv preprint arXiv:1911.03865 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,
Mark Bun and Thomas Steinke. Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds. In Martin Hirt and Adam Smith (eds.), Theory of Cryptography, volume 9985, pp. 635-658. Springer Berlin Heidelberg, Berlin, Heidelherg, 2016. ISBN 978-3-662-53640-7 978-3-662-55641--2 doi: 10. 0-11778---662--5-4--
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/), pp- 2633 2633-2650, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022a.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint Ttl1:2202.076466 2022b.
Ricardo Silva Carvalho, Theodore Vasiloudis, Oluwaseyi Feyisetan, and Ke Wang, Tem: High utility metric differential privacy on text. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pp. 883-890. SIAM, 2023.
Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, and Yue Zhang Adaprompt: Adaptive model training for prompt-based nlp. arXiv preprint arXiv:2202.0444 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https:
//1mays..../////////////// org/ hhtgg//lg////////o wal.og//22200222
2023-00 -3-00-3--0 30-3000-0 viovviunaa.....
Soham De, Leonard Berrada, Jamie Hayes, Samuel I. Smith, and Borja Balle. Unlocking high- accuracy differentially private image classification through scale, arXiv preprint arX1v:2204.13650, 2022.
https:://prr.. tamu. edu / /
10

---

Published as a conference paper at ICLR 2024
Mingkai Deng, Jianyu Wang. Cheng- Bieng-Ping Hsich, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12545 2022.
Minxin Du, Xiang Yue, Sherman SM Chow, and Huan Sun. Sanitizing sentence embeddings (and labels) for local differential privacy. In Proceedings of the ACM Web Conference 2023, pp. 2349-2359, 2023.
Haonan Duan, Adam Dziedzie, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. Conference on Neural Information Processing Systems, 2023a.
Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, and Franziska Boenisch. On the privacy risk of in-context learning. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023b.
David Durfee and Ryan M Rogers. Practical differentially private top-k selection with pay-what-you- get composition. Advances in Neural Information Processing Systems, 32, 2019.
Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pp. 1-12. Springer, 2006.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Shai Halevi and Tal Rabin (eds.). Theory of Cryptography, Lecture Notes in Computer Science, pp. 265-284. Springer Berlin Heidelberg, 2006. ISBN 5-773---40--
Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pp. 51-60. IEEE, 2010.
Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. Privacy-and utility- preeervipge textual analysis via calibrated multivariate perturbations. In Proceedings of the 13th international conference on web search and data mining, pp. 178-186, 2020.
GDPR. Gdpr, 2016. URL
Zhenhua He, Aditi Saluja, Richard Lawrence, Dhruva Chakravorty, Francis Dang, Lisa Perez, and Honggao Liu. Performance of distributed deep learning workloads on composable cyberinfras- tructure. In Practice and Experience in Advanced Research Computing, pp. 60-67. 2023.
John Hewitt, Xiang Lisa Li, Sang Michael Xie, Benjamin Newman, and Percy Liang. Ensembles and cocktails: Robust finetuning for natural language generation. 2021.
Bairu Hou, Joe O' d'connor, Jacob Andreas, Shiyu Chang, and Yang Zhang, Promptboosting: Black-box text classification with ten forward passes. In International Conference on Machine Learning, pp- 15309-13324 PMLR, 2023.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):335-547, 2019.
Juraj Juraska, Panagiotis Karagiannis, Kevin K Bowden, and Marilyn A Walker. A deep ensemble model with slot alignment for soguence-to-sqquene natural language generation. arXiv preprint arXiv:1805.066 06555, 2018
Shiva Prasad Kasiviswanathan, Homin K Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What learn privately? SIAM Journal on Computing, 40(3):793-826, 2011.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for pfffmettr-efffiient prompt tuning. arXiv preprint arXiv: arXiv:2104.08661 104.08691, 2021.
11

---

Published as a conference paper at ICLR 2024
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to gequence-to-eqqueee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. 2019.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in in-context examples for gpt-3? arXiv preprint arXiv:2101.06804 2021.
Kevin Liu. The entire prompt of microsoft bing chat?! (hi, sydney.). 2023. URL https:// twitter.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv22104.08886 2021.
Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella- Béguelin. Analyzing leakage of personally identifiable information in language models. arXiv preprint arXiv:2302.00533 2023.
Justus Mattern, Benjamin Weggenmann, and Florian Kerschbaum. The limits of word level differential privacy. arXiv preprini arXiv:2205.02130 2022.
Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS 07), pp. 94-103. IEEE, 2007.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg- Kirkpatik, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929 2022.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp. 263-275. IEEE, 2017.
Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing,
pp. 75-84, 2007.
OpenAI. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
Jonas Oppenlaender, Rhema Linder, and Johanna Silvennoinen. Prompting ai art: An investigation into the creative skill of prompt engineering. arXiv preprint arXiv:2303.135.. 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 1909.0 0666, 2019.
Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304.0599 2023.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, adit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.0728/ 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018.
Adam Roberts, Colin Raffel, Katherine Lee, Michael Matena, Noam Shazeer, Peter Liu, Sharan Narang, Wei Li, and Yanqi Zhou. Exploring the limits of transfer learning with unified text-to-text transformer. 2019.
Tom Sander, Pierre Stock, and Alexandre Sablayrolles. Tan without burn: Scaling laws of dp-sgd. In International Conference on Machine Learning, pp. 19937-29949) PMLR, 2023.
12

---

Published as a conference paper at ICLR 2024
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. Toward human readable prompt tuning: Kubrick's 's the shining is a good movie, and good prompt too? arXiv preprint arXiv:2212. 2022.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium security and privacy (SP), pp. 3-18. IEEE, 2017.
Alessandro Sordoni, Xingdi Yuan, Marc-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Deep language networks: Joint prompt training of stacked Ilms using variational inference. arXiv preprint wxXi:2336.112509 2023.
Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Ziman Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. preseering in-context learning with differentially private few-shot generation. arXiv preprint arXiv:2309. 1765, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Dictrich Trautmann, Alina Petrova, and Frank Schilder. Legal prompt engineering for multilingual legal judgement prediction. arXiv preprint arXiv:2212.02119, 2022.
Saiteja Utpala, Sara Hooker, and Pin Yu Chen. Locally differentially private document generation using zero shot prompting. arXiv preprint arXiv:2310.16111 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 1rX.v:1804.07461, 2018
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306. 1698, 2023a.
Jiaqi Wang, Enze Shi, Sigang Yu, Zihao Wu, Chong Ma, Haixing Dai, Qiushi Yang, Yanqing Kang, Jinru Wu, Huawen Hu, et al. Prompt engineering for healthcare: Methodologies and applications. arXiv preprint arXiv:2304.14677 2023b.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Conference on Neural Information Processing Systems, 2023.
Tong Wu, Ashwinee Panda, Jiachen Wang, and Prateek Mittal. learning for large language models. In The Twelfth International Conference on Learning Repre- sentations, 2023.
in-context
Lukas Wutschitz, Huseyin A. Inan, and Andre Manoel. dp-transformers: Training transformer models with differential privacy. https:/ / //wwww .icrooooft. com/en us/ ceen-uu/resseen research// project/dp- http/ddpprranssee transformerss August 2022.
Guangxuan Xiao, Ji Lin, and Song Han. offsite-tuning: Transfer learning without full model. arXiv preprint arXiv:2302.0488 2023.
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier. A differentially private
text perturbation method using a regularized mahalanobis metric. arXiv preprint arXiv:2010.11947 2020.
13

---

Published as a conference paper at ICLR 2024
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- trained transformer language models. arXiv preprint arXiv:2205.01000 01068, 2022.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. International Conference on Learning Representations. 2022.
Yuqing Zhu and Yu-Xiang Wang. Adaptive private-k-selectin with adaptive k and application to multi-label pate. In International Conference on Artificial Intelligence and Statistics, pp. 5622-5635, PMLR, 2022.
14

---

Published as a conference paper at ICLR 2024
A METHOD
A.1 DEEP LANGUAGE NETWORK
In Algorithm 3, we present the DLN-1 algorithm from (Sordoni et al., 2023) where highlight the potential privacy leakage.
Algorithm 3 Deep Language Network (DLN-1) (Sordoni al., 2023) and potential privacy leakage
Input: Client datasets D = {(x, v) number of prompts N, number of iterations T
I: Initialize ı with a task description or empty
2: for E [1...., T do
3:
8 x (x.1) 1) ~ D}
Sample minibatch Foward pass
4:
ÿ - PL F Fxx x)) for all (z. y) in S
5:
II t PL IT Br ((z, v. 0) 1
b> Sample N candidate prompts
6:
- arg m E (z Po
> Select the best prompt
7:
Output π
A.2 PRIVACY ANALYSIS
Rényi differential privacy (RDP) (Mironov, 2017) is variant of the standard (c, 6D- that Rényi-divergence as distance metric between the output distributions of MMD) and M(D'), which particularly useful in training differentially private machine learning models.
Definition A.1 (Rényi Differential Privacy (Mironov, 2017)). We say that mechanism M is (a, (am) RDD with order a (1, oo) if for every neighboring dataset ~ D', we have:
[UMMDOO) )1 S <m(())
Da (M(D)II (D)) a
log BOwMiDs
(1)
where (-M(-) denotes the density function of M's distribution.
Next, we introduce strict relaxation of RDP which allows for small failure probability 6. It is analog to the term in the standard DP definition.
Definition A.2 (Approximate RDP (Bun & Steinke, 2016; Zhu & Wang, 2022)). We say randomized algorithm M is 6-approximately (a,&M(( (M:))) with order a 2 1, if for all neighboring dataset D. D', there exist events F. (depending M(D) and F' (depending on M(D')) such that Pr(E] >
1 1- and Pr[E"] 2 - 8. and Va 2 1, we have
Da (M(D)) II I (M') (M(a)
(2)
In this work, we use RDP and approximate RDP for tighter measure of the privacy cost, as the composition for both privacy notions is trivial. After we obtain the approximate RDP guarantee for the overall mechanism, we can then convert the privacy guarantee back into the standard DP definition (see Bun & Steinke (2016) and Mironov (2017) for the composition and conversion formula for RDP and approximate RDP). the following, we state the privacy guarantee of individual building blocks for private prompt generation and selection in terms of (approximate) RDP.
A.2.1 PRIVACY ANALYSIS FOR PRIVATE PROMPT GENERATION
The exponential mechanism takes utility function N x y - IR and can be thought of as evaluating how good g(D,y) 3) is for an outcome y y on dataset D. In our context, y E y is potential token to be released and g(D, v) is the vote count for the token y from the disjoint partitions.
Definition A.3 (Exponential Mechanism (McSherry & Talwar, 2007)). Let EM, N + y mechanism where for all outputs y E y' we have
E Pr(EM, = y] x exp
((D,))
2A(q)
15

---

Published as a conference paper at ICLR 2024
where (g)) is the sensitivity of the quality score, ie. for all neighboring inputs D, D' we have sup (g(D.) y) - (D', y)I Σ (4))
Theorem A.1 (Bun Steinke (2016)). The exponential mechanism is e-DP, and (L)) s.1. EEM (a) ge²..
We now state the privacy guarantee for LimitedDomain algorithm.
Theorem A.2. Algorithm satisfy (c. 6)PP and 6-approximately (a, (EM(a))-RDP.
Proof. This immediately follows from the proof of Theorem from Durfee Rogers (2019), as LimitedDomain algorithm is essentially an exponential mechanism with an extra step of reducing the domain size, whose privacy cost is being added to the å term.
Algorithm 4 LimitedDomainnkk; k, k, co: 66) from (Durfee Rogers, 2019), where Dg the to sensitivity of the utility function, and A is the foo sensitivity of the utility function.
Input: Histogram h, top-k (by default, k = 1) from the E [k, d) limited domain, privacy parameters co, 60
1: Sort h such that h())) 2 h(2) 2...
2: h + h5+1 +1+ 2 2nimmin ( k d 60
3: en h Gumbel(2A.o./e))
4: for j < do
5: "()) + ho) + Gumbel(2A
6: Sort (m()) Uvi and let Viass Vi(z) U1 be the sorted list up until
7: Output (1)))..... (0) 13 if k, otherwise {i(a)....,
Asymptotic Privacy Analysis. Below, we provide an asymptotic analysis for the overall privacy bound that may facilitate the understanding of the capability of DP-OPT. Theorem A.3 shows the number of generated tokens is essentially limited by the privacy budget.
Theorem A.3. Suppose we set the privacy parameters of LimitedDomain as eo, 8g, then the total privacy bound of our private prompt generation algorithm for generating tokens (e, mdo 8 -DP with any > and
== ==0(vm log(L/8)))
Proof. This immediately follows from the advanced composition theorem of differential privacy (Dwork al., 2010).
We stress that Theorem A.3 is only used for illustrating the asymptotic growth rate of the e with the number of tokens being generated. We use the RDP-based privacy accountant numerically calculate in the actual implementation, which leads to a much tighter bound and therefore allows generating more tokens.
A.2.2 PRIVATE SELECTION AMONG GENERATED PROMPTS
While private selection algorithm is simply exponential mechanism. we can actually obtain a tighter privacy bound than what is stated in Theorem A.1. We first state the definition of the monotonic utility function.
Definition A.4 (Monotonic Utility Function). We say that a utility function q(-,) :) is monotonic in the dataset if the addition of data record either increase (decrease) remain the same with any outcome, e.g. (D,,)) y) & ((D) U (x), y) for any input and outcome y.
Clearly, in context, since the utility function is defined the count of correct predictions on the validation set, the addition of validation point will not decrease the utility function, and hence our utility function monotonie in this case. We now have the following tighter privacy guarantee.
Theorem A.4 (Durfee Rogers (2019)). The exponential mechanism is e/2-DP, and (a, (a))))))
RDP s.I. EM (a) = ge2 the utility function g is monotonic.
16

---

Published as a conference paper at ICLR 2024
A.3 DISCUSSION OF COMPUTATIONAL EFFICIENCY
Our method is quite efficient and feasible for black-box models. Since we do not require gradients but only the forward pass which mimies zeroth order gradients, method is much efficient for any gradient-based bottleneck comes from the ensemble. Here, we provide detailed memory and computation analysis of the ensemble. 1) Computation efficiency. Our method will do ensemble prediction per token which has a similar complexity as inference. Parallelizing the process could speed up the training. 2) Memory efficiency of training. As we only do inference, the complexity only depends on the context length. We use demonstrations meta-prompts, whose complexity close to k-shot in-context learning. 3) Memory efficiency of inference. Compared to ICL, our generated prompts are short resulting in low overhead for memory at inference time.
memory
method, including soft prompt tuning. The main computation
17

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL SUPPLEMENTARIES
B.1 EXMPERIMENT DETAILS
We present the detailed statistics of all four tasks in Table 4. Disaster has the smallest volume of training data. But Trec is short on samples per class, making it harder task.
Table 4: Task statistics. Note that different from (Sordoni et al., 2023), we use the full set rather than small fixed 250-sample subset of the original dataset for testing. The validation set is selected from the training set per random seed. The ratio of validation with respect to the training set is included in brackets.
Task
# Train
Valid
# Test
# Class
Description
SST-2
66, 674
673 (1%)
1,8
Sentiment analysis movie reviews
Trec
452
272 (5%)
500
Question type classification
Mpqa
603
430 (5%)
0.000
Sentiment analysis
Disaster
4, 430
221 (5%)
1,000
2
Determine whether a sentence is relevant to a disaster.
Hyperparametes. For DP algorithms, we limit the DP budget parameters for DP-OPT are given in Table 5. As the total is determined by sample size, we mainly tune the cy and óu for each dataset such that enough tokens can be generated in DP-OPT, For DP-OPT and OPT on Mpqa, we set the repetition penalty to be 1.1 to avoid repeated words. For DPSGD, we adopt dp-trans formorreee rs package to reduce the memory overhead caused by gradient clipping (Wutschitz et al., 2022) and tune the parameerraetterss for each dataset. We follow (Duan et al., 2023a) to use the same templates for soft prompt tuning. For all our experiments, report average and standard deviation from three repetitions with seeds, {1, 3). For ICL, the randomness is on the selection of in-context examples. We notice the most influential factor in ICL is the balance of examples in experiment. We follow DLN-1 implement the ICL where sample 5 balanced examples. Our results are comparable to those reported in (Sordoni et al., 2023) in Trec and Mpqa. ICL results Disaster are different because used much larger test (using the standard test set of Disaster) while Sordoni el al. selected small 100-sample subset for the test. For all trainable methods, we hold 5% of training data for validation and report accuracy on the original test set.
8. and D/ Detailed
Table 5: Hyperparamters for DP-OPT and OPT. For batch size, 5 x 205 means 205 5-demo meta- prompts. en and d the parameters for LimitedDomain in Algorithm
Experiment
Max new tokens
Batch size
6p
temperature
SST-2
50
5 205
1.8
5 x 10
1.1
Tree
3 102
0.8
x 10-6
Mpqa Disaster 50
50
3 102
1.6
x 10-6
1.1
3 102
0.8
4x 10
1.1
In Table 6, we list the initial instructions that we used in OPT, DP-OPT, and DLN-1 following (Sordoni al., 2023).
Table 6: Initial instructions for tasks.
Task
Classes
Instruction
SST-2
[negative, positive]
Classify input text as positive or negative.
Tree
[description, entity, expression,
Read the following question, then choose whether it is about a description, entity, expression, human, location number.
human, location, number) [negative, positive]
Mpqa
Read the following review, then choose whether negative or positive.
Disaster
nnot relevant, relevant)
Read the following sentence, then choose whether it is relevant
a disaster.
18

---

Published as a conference paper at ICLR 2024
Classification Forward Template
[PROMPT]
[ENPUT]
Backward Template B
student is cong etiing a task that requires productig text outpot fron text input. The student receives iastruction that descriaes hov to prodice the output giver. each ingut. The student errors, Your
ingrove the instruction suca tast student
This wae the instruction.
[CURRENT INSTRUCTIONN]
Student successes
Ingut: (INFUT]
Cutput: [OUTPUT
Input: [INPUT]
Correct Cotput: [OUTPUT
Student aErors
Inpat: [IRFOT]
Student
[OUTPUT]
Cupat: (TARGET)
Inprove the inst ructicn to fix the student crrors, [NSC] Improved Instructicn:
where [MSG] is randomly chosen from
Clarify the rnst ion by addingg few words or whort wen sent eenee concise
Inpeove the snsizuction by providing examples on how Lo solve Lhe Lask, Be concias.
Shorten the instruction by zenoving super2lous words sencences. providing detwiled information
embiguiit. concise
Figure 5: Templates for DLN-1, OPT and DP-OPT.
In Fig. 5, we list the templates used for DLN-1, OPT and DP-OPT. The templates are slightly different from those in (Sordoni et al., 2023) since we use Vicuna-7b instead of GPT to handle the instructions.
Finding privacy leakage prompts. find strings leaked from the training set, we perform semantic similarity search to retrieve the training sentence closest each forged demonstration. Concretely, we perform mean-pooling of the hidden states of the last layer of the BART-large decoder (Lewis et al., 2019) over the token dimension to obtain sentence embeddings for both training examples and the demonstrations generated by prompt tuning. We use Faiss (Johnson et al., 2019) retrieve the top-5 training examples closest the demonstrations that appeared in the generated prompts in terms of the & distance between their embeddings. We then manually examine the retrieved examples identify training example leakage.
19

---

Published as a conference paper at ICLR 2024
B.2 ADDITIONAL EXPERIMENTS
Source Vicuna-7b
Target
Task
Llama-2-70b
A
DaVinci-003
A
SST-22
80.4(4.0)
83.9(1.5)
3.5
73.5
6.9
Trec
(6.1(0.8)
23.1
-21
46.0(1.4)
-0.1
Mpqa
74.2(5.7)
60.8(5.6)
-3..4
71.8(7.6)
-2.4
Disaster
59.9(2.0)
45.9(4.1)
-14
55.4(4.0)
-4.5
Table 7: PEZ cannot produce transferable prompts, bringing non-trivin triviaivil losses (A) on multiple tasks. Average accuracy (%) on test sets reported with standard deviation in the brackets.
Negative Transfer of Projected Prompt Tuning. In Table 7, we evaluate the transferability of projected prompts, dubbed PEZ, proposed by Wen et al. (2023). We did not report standard deviation on Da Vinci-003 and SST-2 since two prompts are not tokenizable by Davinei-003. On both Da Vinci- 003 and Llama-2-70b, the PEZ-engineered prompts loses great portion of accuracy after transfer.
Liama-2-1
Vicuna-TB
DaVinc-003
- Vicuna-33b
aama-2-706
Test Accuracc (%)
85
75
Erisemble
Figure 6: Ablation of the number of ensemble prompts.
Sensitivity to Ensemble Number. We ablate the number of ensemble prompts in Fig. For larger models, the number of ensembles seems less influential when the number is larger than 30. Only for Vicuan-7b, the parameter is critical and is essential to have more prompts in the ensemble. We attribute the stability to the robustness of larger models. Larger models could be less sensitive to words replacement.
Empirical Evaluation of Privacy Risks. We evaluate the privacy risks empirically by membership inference attack (MIA) using Likelihood Ratio test (LiRA) (Mireshghallah et al., 2022). Results are reported in Table 8. For SST-2, because of the distributional bias between the training and test sets, we subsample the training set to include samples with more than 20 tokens, in which case only 15 test samples are eliminated. The data filtering can avoid undesired high MIA AUC due to lack of short samples in test sets. We observe non-trivial AUCs for DLN-I on Mpqa and SST-2. In comparison, both DP-OPT and OPT has very low AUC. OPT has slightly higher risks than OP-OPT when applied on the Trec dataset.
Comparison to Text Sanitization. Distinct from our work, text sanitization, for example, Mattern et al. (2022); Utpala et al. (2023), sanitizes text samples before input into LLMSs. In Table 9, we compare our method to DLN-1 using sanitized data (Mattern al., 2022), denoted as Private DLN-I
Method
Disaster
Mpqa
SST2
Trec
DLN-1
0.498
0.772
0.773
0.446
OPT
0.511
0.443
0.510
0.494
DP-OPT
0.497
0.456
0.518
0.468
Table 8: Average MIA AUC of Likelihood Ratio attacks. Bold numbers indicate the highest leakage risks (over 0.5).
20

---

Published as a conference paper at ICLR 2024
Data
Private DLN-1 (E = 8)
DP-OPT (E = 8)
SST-2
0.868
0.895
Trec
0,311
0.653
Mpqa Disaster
0.690
0.807
0.657
0.656
Table 9: Comparison between DP-OPT and DLN-1 using sanitized data (Mattern et al., 2022). Test accuracy averaged over 3 repetitions are reported on SST-2 using Vicuna-7B.
The implementation includes three steps: (1) First, the embedding of each token will be perturbed and projected into the original embedding space. This step can be extended to other sanitization methods (Utpala et al.. 2023; Feyisetan et al., 2020). (2) We use DLN-1 1 to tune prompts on these samples. DLN-1 selected here due to its similarity to our method but can be replaced by other prompt-tuning algorithms in practice. (3) We use the generated prompts for inference. Note that the text sanitization is measured under the metric Differential Privacy instead standard DP. Because of the different privacy assumptions, emphasize that the meaning of is different. We show that our method can outperform Private DLN-1 significantly on three datasets and has similar performance as Private DLN-1 on Disaster.
Generated Prompts. In Tables 10 and 11, we give more examples of prompts generated by LLMs. D able generate a very semantic prompt (e.g., seed in SST-2 task) but may fail transfer (with a 3.5% drop). Consistent with the conclusions in the main content, the DLN-1 tends to leak more private information and DP-OPT presents much less visible leakage. In the hardest task, Tree, DLN-1 extensively overfits the source model with semantically favored prompts but transfers poorly to DaVinci where two prompts suffer from negative transfer.
Interestingly, we see that "# Student successes Input:" does not provide useful task information but often occurs enjoy positive transfer, e.g., DP-OPT on SST-2 and Disaster. We conjecture that the prompt induces the LLM to produce "success" output.
We notice that the prompt engineering degrades to generating dummy samples sometimes. However, our method can create prompts without samples and promote the performance, well. For example, DP-OPT may only slightly change the prompt by appending # Student successes Input:" to an initial instruction. Intuitively, the modification prompts LLMs to generate "successful" responses. We notice such minor modifications can improve the accuracy of the Disaster dataset from 76.4% (0-shot) 78.9% (DP-OPT) tested by Da Vinci-0033 It even outperforms generated by DLN-1 (77%).
complicated prompts
Avoid Privacy Leakage via Instruction. When notice the direct breach of training samples in generated prompts, a straightforward fixture could be to instruct LLMs to keep secrets. We tried two of such instructions in experiments: (1) Instruction 1: Do not provide examples in the prompt. (2) Instruction 2: Do not use existing samples but create durmy samples as examples in the prompt. For DLN-1 and (DP-)OPT, we all ap- pend the privatization instruction to the instruction in the backward template. We test the instructions on SST-2 following the same setting in the main experiment and report the generated prompts in Table 12. We notice Instruction 1 still leaks private samples and the AUC measured by Likelihood Ratio MIA attack is as high as 73%. Interestingly, the second instruction only generates dummy ex- amples that have no similar examples in the training set. However, the prompt will present non-trivial risks (69% AUC) measured by Likelihood Ratio MIA attack.
In conclusion, though privatization instruction could remove private examples, it still suffers from information leakage. The method is orthogonal to our method that provides theoretical guarantees and can be combined with our DP-OPT to reduce the chance of explicit leakage.
21

---

Published as a conference paper at ICLR 2024
Table 10: Generated prompts. We present test accuracy on Vicuna-7b (Sre Val Acc) and Da Vancii-003
(Trg Test Acc). Samples found in training sets that best match the generated samples are marked as red text.
Method
Test
Tig Test Ace
Generated Promppts
DLN-1
92.7%
89.2%
Classidy the text as posétive ur negative, 2. The correct cutput for positive text is positive. 3. The correct oulput for negaiive text negative. 4, Do not classily teals that are a mix of posilive and negative either posiive negative. S. Pay amention the
of words used the
some words can
both positive and negative contests. Examples: Isput The
92.7%
93.5%
Film with small budgel Input: small, personal film emotional wallop (turn out small personal film with emotional wallop) Output: posilive Input: nightmare sbout had cinera (nightmare about had cineoa ) Outpuz: negurivel I Inpur: film with their charismt (the film with their churismm Ouiput pusilive! Input: who feels acting is heart and soul of cinema (who foels acting and
cinems Outuut positive Tnputt easily one the and exctting movies the year (cusily une of the best and must exciting movies of the year . Output: positive Input:
93.0%
93.0% Clussify the input teet an positive or negative Use the coneet output for each input. Avcid phnnes like "might" or "poobabll": "carnage and". "I recommend" or words like "barely" Input: Actor Michel serroutt
Oulput pocitive Ioput: Unique residencess cunique residences >- Comrect Dutput pasitive Tegut Ray the movic mill whas the TV cow free (buy the movie milk when the tv cow is free) - Correct Output: negutive liput: A
Sernall
OPT
91.7%
92.0% Clissity the input text as positive or negutive. i Student successes lnput: movie that is movie that is a mavse" movie more than a movie. Correct Output negutive Iaput: "a movie that is a movie that
86.6%
92.3%
Clussify input pastiive negative.
90.7%
93.0%
Classify the input positive negutive. Stadent (the movie handly a masterpiece Cornect Output: positive Input: "the movie was disaster" (the movic a disaster Correct Output
Input: "the muvie masterpiece"
DP-OPT
90.1%
91.4%
Clussify the ingut text as positive negative. Student anccesses Inpur:
85.0%
9X2.4%
Classity the ingut text as positive negative.
91.7%
92.9%
Classify input as positive neguzive. Student successes Input: "a movie that masterpiece" (a great movie it is not J Correct Ouiput: positive Inpul: the movie was a disaster (the movie is a disister. Conrect Ourput negative Inpur
Tree
DLN-1
57.6%
58.0%
Read the question carefully. 2. Determine the vorrect output type based the correl uulput catepory from the list options, proviee brief description. S. Avoid ambiguity in the question procept.
Select the quesion doess' have single corret culput.
65.8%
51.6%
Read question Choose correct output from following options: a Description (D) 4 Earity (E) + Expression (EX) + Human (11 + Locotion (1.) + Number (N) Correct Outpuis: Homm (1f) Human (H) Human (H) Number (N) Human (Ht)
2
66.4%
58.0%
1. Read the text input carefully. 2. Ideniity the type of question from the options provided. 3. Choose the outpur type from the list of options. Compare your choice with
output
provided. S. If your choice correct, proceed the next input. not, go back step
OPT
60.4%
70.6%
Read fellowing question. choose correct type of output: description, entlty, expression. haman, location, or number. " For each question, you will be given a text inpur. Your task to produce the conrect
2
60.4%
72.0%
Read the following quesiion, then choose the correct type of output description, eniily, expression. hemsan location, number. For each question, there is only one correct output. Thee student most
read question
62.8%
72.0%
Read following question. choose correct type of output: description, enilty expression.
hamon, location, number. For example, if the question is "What cspital France? (What the capital of Italy 7) (What is the capital of California 7) (Whur is the lirgest city in Germany T), the correct oulpui
DP-OPT
63.0%
72.2%
Rend the following question, then choose the cornect type of curput deseription, entiyy, expression, hamie location, numberr For example, the question "What the eapital the capital of Italy 31 (What capital Califortia 7) (What lureest city in Germany Ti, the cornect output
(Whar
2
70.2%
61.2%
Read the folkowing question, then choose the correct type of ourput
63.0%
72.4%
Read the following question. then choose the cornect type of output: description, enity, expression.
human, location, number. " For example, if the question is "What is the cupital of France? (What is the capinal of lialy ") (Whot capital California ?) (What the largest city Germany the correct oulpui
22

---

Published as a conference paper at ICLR 2024
Table 11: Generated prompts. We present test accuracy Vicuna-7b (Sre Val Acc) and Da V0n (Trg Test Acc).
Mathol
Seal
Sie Toxt Ave
The Test Acc
Generated Prompts
Mpga
DLN-1
82.8%
79.8%
Read the following review, then choose whether it negative postive by identitying the conrect output each input based
following esamples: Displayed unrelenting resolve and confi- dence (his displayed unselesting resalve coufidence): positive Protests (protests): negutive Constractive and cooperative lies (constractive and cooperative ciss:: positive Increasingly angry (incressingly anpry): negotive Pasitive and optimistic views: positive * Denial (donioll: negative * Advanced: negative United Stares threstening
73.7%
82.9% Read the following review. Identify each semeence that requires the student choose correct idenilledd semiene,, write the correct oulput based
output based given input. Compare your outpul with the correct uulpui provided in the instructions and make sure they musch. 5. the student's cuiput incorrect. revise based on the correct output provided in the instractions.
given input.
70.9%
81.6%
Read the following review. Chouse whether is negative posilive. Comect Ouiput: positive
4. Output: positive Cornect Dutput segative Cornet Ouputt regutive
Duiput:
negasive Correct Ourput: negative 9. Corrccl Oulpui: negative J0. Correct Ouiput: negative 1l.
Correct Output negalive 12. Correct Output: negative 13, Correct Output
OPT
62.3%
87.3%
Read the following reviow, then choose whether is negative or positive, For each statement.
detenmine it negative positive.
80.1%
80.2%
Read the following review, then choose whether it is negative or posutive. For each statemeent.
detemmin positive nogative
80.1%
85.1%
Read following review, then choose whether negative positive. * For each statement. detenmine it is positive negutive sestiment.
DP-OPT
84.6%
85.0%
Read following review and determine if positive or negative based the words text.
78.8%
86.3%
Read the following roview and determine it is pasitive or negative.
78.8%
86.1%
Read the following review and determine it is pasitive negative.
DLN-1
58.9%
76.0%
Read the sentence and determine the information relevant a disaster, The relevant ancormation is when the sensence memions disaster or its effects. Please choose "yes" the sentence discusses a disaster its effects, and otherwise.
65.2%
75.8%
Read each sentence carefully. and ensure it relates to disaster not. Choose "yes" "mo" the correct oulpui. Produce correct culpui each sentence.
60.9%
79.2%
Read given sentencee 2. Determine the sentienee relevant to disaster, select "yes"" IIf select "no."
to disauder. 3. If sentence
OPT
66.8%
79.2%
the following sentence, choose whether relevast a disnster. Studeit sucoesses Eaput:
6n syyada, khan Wow! I'm so glad found this! Correct Outpur:
47.0%
78.1%
Read the sollowing senience, then choose whether it relevan: a disustor. Student successes Ixput:
@ Airbab is great way to make money. Correct Ourput: no Input: The world
59.8%
79.8%
Read the following senience, then choose whether it relevam, to disanterr & Student sucvesses Irput:
@r5%&"0 ==]] sool 1.2 123
DP-OPT
65.4%
78.6%
Read the following sentence, than choose whather is relevant a disastar. # Studest sucoesses Isput:
65.4%
78.9%
Read the following sentence, then choose whether is relevant disaster, # Student successes laput:
65.9%
79.2%
Read the following sentence and determine whether is relevant to a disaster. Student successes
Input:
23

---

Published as a conference paper at ICLR 2024
Table 12: Generated prompts. We present test accuracy on Vicuna-7b (Sre Val Acc) and Da Vanii-O0
(Trg Test Acc).
Method
Seed
Sie Test Ace
Trg Test Ace
Ganented Prampts
Instraction 1
DLN-1
91.8%
91.8%
I. For each inpul, produce sporesponding
outpur. 2. The imput is given firs, followed by the statemsent the sentimea expeessed
ourput. correct outpur simple input, for evangle "posttive" "negutive"" "incrretting"" esc. 4, Label student erroes red and provide an alternate correct ourpur in blue. S. Provide examples prrentheses Examples: Suudent successes
2
86.1%
93.9%
Instruction: Classily the input text as positive or nepative. For example: real-tife personu (real- life pensonal: positive
of dogs who
than him fby
dogs who
archly funny deeply anthentic (candid, archly funny deoply authentick: positive bran tufano handsome widescrees photography and paul grahowskcy excellent music tum this faity purochial meledrama into someching really ruther special, (brian tufuno handsome widescreen photography and paul grabowsky 's exvellent music turn this fairly parochual melodrama something really rather special pounive
smarter
negative
91.9%
93.9%
Classify the input
negotive.
Output: (choose positive, regative
Inpui: (provide the text input) Correct Ourput: (choose one' pusitive, negative. Example: Input: "The movie is mosterpicce. Output: positive" Input: "hiis movie is terrible." Ouiput: megative Input: "I hated the movie
Instraction 2
DIN-1
90.3%
93.1%
Classify input the inpui test as pasitive negative. Improvoed Instriction: Clussify each input positive negative bused on the given ouspu. Instraction: Classify the input lexi as positive or negative and provide the correct
positive nogative bused given output, Instraction:
Instraction: " Classify each inpux text positive
2
92.2%
94.7%
1. Classity the text posiiive or negative by identifying the sontiment expressed in the text. Mentify indicate the sontimeat. For iopur, provide clear example of the correct output, For example:: Coneect Ouiput: negative 1. "I stand this and poorly made;" (so poorly ploted and scripedd J Correct Ourgut: positive 2. find the concapt of infinity tase
bocing
91.0%
91.8%
Classify the input test positive or negative.
24


Reviews:
[{"review_id": "QmWRglDLz7", "reviewer": "Reviewer_W4ZT", "summary": "This paper introduces Differentially-Private Offsite Prompt Tuning (DP-OPT) as a solution for data privacy concerns when utilizing Large Language Models (LLMs). DP-OPT operates on the client side and offers an end-to-end framework to generate private and transferable prompts for cloud-hosted LLMs. It ensures data confidentiality, information privacy, and model ownership protection. The paper demonstrates that prompts tuned by LLMs can be effectively transferred across models and introduces a novel differentially-private mechanism for generating private prompts.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) DP-OPT offers a new end-to-end solution for addressing data privacy in the context of prompt tuning for Large Language Models (LLMs).\n2) Paper is well witten and is easy to understand.", "weaknesses": "1) This paper ignores whole body of related work where document is converted to private documents and then used for downstream tasks, check[1,2, 4]. This approaches are task-agnostic and latest work [4] shows significantly better privacy-utility tradeoffs and obtain SOTA results.  It is recommended that these approaches be discussed and ideally compared in experiments, as the setups and datasets are similar, and the methods are simpler than the proposed mechanism.\n\n2) No comparison with real-world threat models has been provided.  Epsilon-utility trade-offs can be misleading without testing them against actual attacks, as epsilon guarantees are built upon numerous assumptions, as indicated in [3, 4]. For a comprehensive evaluation, it is recommended to conduct experiments that demonstrate trade-offs between *empirical* privacy and utility. As simple attack framework as  Membership inference attacks greatly improves rigorousness of experiments. \n\n\n\n\nRefs:\n----\n\n[1] Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations, 2020. (https://arxiv.org/abs/1910.08902) \n\n[2] The Limits of Word Level Differential Privacy, 2022 (https://arxiv.org/abs/2205.02130)\n\n[3] A Critical Review on the Use (and Misuse) of Differential Privacy in Machine Learning, 2022 (https://arxiv.org/abs/2206.04621)\n\n[4] Locally Differentially Document Generating Using Zero Shot Prompting, 2023 (https://arxiv.org/abs/2310.16111)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FuuR7y2Vst", "reviewer": "Reviewer_rsb4", "summary": "The paper proposed an approach, DP-OPT, of generating privacy-preserving prompts in LLM, which can yield competitive performance compared to non-private in-context learning.", "questions": "1. Please describe the threat model with more clear details.\n2. Please evaluate the privacy performance of the proposed approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The study focuses on an interesting and important topic, the privacy protection in generated prompts of LLM. \n+ The introduction of related work is comprehensive and covers most of the recent studies in prompt engineering.", "weaknesses": "- The threat model is unclear\n\nMy first concern is related to the threat model. It would be better and necessary to provide more details and more specific description on the adversary's goals, capabilities, and knowledge in a threat model. In addition, please provide more description and real-world cases of the consequences of a privacy leakage from the prompt (e.g., how much information can be breached).\n\n- Lack of evaluation on privacy leakage \n\nConsidering that one of the key motivations of the study is to address the threats related to \"data confidentiality\" and \"information leakage\", it would be expected that an evaluation on the privacy performance of the proposed approach is conducted. There are only several examples of prompts provided, rather than evaluate the privacy protection in a quantitative manner. It would be necessary to involve some privacy metrics in the evaluation. It would be even better if some privacy attacks are involved in the privacy performance evaluation, instead of only comparing the model utility performance. In addition, the presentations of prompt examples are misleading and confusing - the \"semantically-nearest retrieved\" should not be in-line with other prompt messages.\n\n- Setting a constraint in prompt generation\n\nIMHO, to avoid leak privacy information from generated prompts, one intuitive method could be adding some constraints during the prompt generation, such as \"do not provide examples in the prompt\", or \"do not use existing samples but create dummy samples as examples in the prompt\". I would suggest having a discussion whether this would be feasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d9nEl0TMO2", "reviewer": "Reviewer_pHUr", "summary": "This paper introduces DP-OPT, a solution for adapting Large Language Models (LLMs) on sensitive data while ensuring data privacy. The paper discusses the challenges of hosting a local LLM and optimizing a soft prompt using private data, and how DP-OPT can help overcome these challenges. DP-OPT uses differential privacy to protect the privacy of the data while optimizing the prompt, and it also allows for the transfer of prompts suggested by LLMs without compromising performance.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Motivation:** The paper solves a very important yet larger overlooked problem, i.e., the two-fold concerns surrounding data privacy when adapting LLMs on sensitive data\n-\tData Confidentiality: Certain data, like medical histories, proprietary system logs, and personal messages, are inherently confidential and should not be transmitted beyond local devices, internal computing systems, and mobile phones.\n-\t Information Leakage: Resources derived from private data might inadvertently contain personally identifiable information. Even with a limited parameter set, such as prompts, the potential for data breaches remains significant.\n\n**Method:** DP-OPT operates exclusively on the client device and uses demonstrations to guide a local LLM to generate prompts. The local assistant model may be significantly smaller than the intended cloud-based LLMs. DP-OPT uses differential privacy to protect the privacy of the data while optimizing the prompt. It also allows for the transfer of prompts suggested by LLMs without compromising performance. DP-OPT is the first end-to-end framework where the entire prompt process is managed on the local device and offers services via an API, thus ensuring data confidentiality, information privacy, and cloud model ownership and IP. \n\nPrior work shows that prompts suggested by LLMs can be transferred without compromising performance. The paper shows that DLN prompts can transfer to and work better on larger models than on the source models, which is called positive transfer. DP-OPT allows for the transfer of prompts suggested by LLMs without compromising performance. The authors also provide the first solution to ensure the privacy of the gradient-free algorithms that demonstrate strong empirical performance compared to in-context learning and previous private gradient-based competitors.\n\n**Experiments:** The authors conducted experiments on four different tasks, including sentiment analysis, question type classification, sentiment analysis on news articles, and disaster relevance classification. They compared the performance of DP-OPT with other state-of-the-art methods, including PEZ and GPT-3, and showed that DP-OPT outperforms them in terms of accuracy and privacy. Impressively, DPOPT generates privacy-preserving prompts by Vicuna-7b, that can yield competitive performance compared to non-private in-context learning on GPT3.5 or local private prompt tuning.", "weaknesses": "It’s known that in-context learning performance is unstable across the choice and even order of examples. How the authors ensure their ICL performance is reliable?\n\nTable 2 compares methods on different models. Why this comparison is valid/fair?\n\nIn Figure 4, the provided examples seems to suggested that the method won’t vary the task description while only generating few-shot prompts?\n\nOverall, I appreciate the contribution of this paper. I'd love to increase my score if these problems could be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vAj5MtAJ0u", "reviewer": "Reviewer_D5hy", "summary": "This paper proposed DP-OPT to address the challenge of optimizing prompts using private data while ensuring data confidentiality and information privacy. The authors provide a comprehensive evaluation of the proposed framework and show that it outperforms other state-of-the-art methods in terms of accuracy and privacy. Additionally, the authors show that DLN prompts can “positive” transfer to and work better on larger models than on the source models. The paper also provides the first solution to ensure the privacy of the gradient-free algorithms that demonstrate strong empirical performance compared to in-context learning and previous private gradient-based competitors.", "questions": "No particular", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed method is highlighted for its efficiency, especially for black-box models. The method does not rely on gradients but only on the forward pass, which is likened to zeroth-order gradients. This approach makes the method more memory-efficient compared to any gradient-based method, including soft prompt tuning.\n-\tThe method emphasizes the memory efficiency of the method in both training and inference. During training, since only inference is performed, the complexity depends solely on the context length. The paper mentions the use of k demonstrations in meta-prompts, which has a complexity similar to k-shot in-context learning. For inference, the generated prompts are short, resulting in low memory overhead.\n-\tThe main computational bottleneck is identified as the ensemble. However, the method performs ensemble prediction per token, which has a complexity similar to inference. The potential for parallelizing the process is mentioned, suggesting that this could further speed up the training.\n\nIt is further novel and a bit surprising to see the “positive transfer” DLN prompts can work on larger models than on the source models, because it challenges the traditional assumption that prompts generated by a smaller model would only work well on similar or smaller models. This finding suggests that the prompts generated by DLN can be used to improve the performance of larger models, which can be beneficial in real-world applications. It might potentially open new possibilities for progressively improving the performance of newer large language models using prompts generated by smaller “old” models.\n\nThe paper compares the proposed method with existing methods, such as DLN-1, highlighting the advantages of the proposed method. The paper also provides insights into the potential limitations of other methods, such as Vicuna-7b's struggles with certain datasets.", "weaknesses": "Overall, the paper presents a unique and effective solution for adapting Large Language Models on sensitive data while ensuring data privacy. It is strong work and clearly written, and I have not spotted particular weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)