Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models
Authors: Haonan Duan
Adam Dziedzic
Nicolas Papernot
Franziska Boenisch
Venue: NeurIPS 2023 poster (2023)
Keywords: differential privacy
in-context learning
trustworthy ML
TL;DR: We propose differential learning algorithms for soft and discrete prompts.

Abstract: Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock's knowledge into a single public prompt. We show that LLMs prompted with our private algorithms closely match the non-private baselines. For example, using GPT3 as the base model, we achieve a downstream accuracy of 92.7% on the sst2 dataset with $(\varepsilon=0.147, \delta=10^{-6})$-differential privacy vs. 95.2% for the non-private baseline. Through our experiments, we also show that our prompt-based approach is easily deployed with existing commercial~APIs.

Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models
Haonan Duan' Adam Dziedzici, Nicolas Papernot, Franziska Boenisch University of Toronto and Vector Institute
Abstract
Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose privately learn prompt. We first show that soft prompts privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock's 's knowledge into single public prompt. We show that LLMs prompted with our private algorithms closely match the non non-private baselines. For example, using GPT3 as the base model, we achieve a downstream accuracy of 92.7% on the sst2 dataset with (s = 0.147, = 10 "--differentill privacy vs. 95.2% for the non-private baseline. Through our experiments. we also show that our prompt-based approach is easily deployed with existing commercial APIs.
be obtained
Introduction
Large language models (LLMs) exhibit strong capabilities for in-context learning 6 40). By prepending the adequate prompt to LLM's input, the model can perform myriad of natural language downstream tasks without any modifications to its parameters [41] While the data used to train an LLM is usually assumed to be public, downstream data used in the prompt is often more sensitive. This can elicit confidentiality issues, for instance, prompts contain information that represents valuable intellectual property [34]. At the same time, also raises privacy concerns when the data involves personal information about individuals.
In this paper, our first contribution is to show that these concerns are valid. We are the first to instantiate a highly effective membership inference attack (MIA) 45 against prompts. Our attack is able to determine if a given data point was used within the prompt of the LLM. The only existing solution to mitigate this privacy risk would be to forego prompting and instead fine-tune the LLM with privacy-preservee
training algorithm (25 544 Yet, fine-tuning lacks the efficiency and practicality of prompting. Indeed, fine-tuning requires significantly more data 42], computational resources (55,, and storage space [26). Additionally, fine-tuning requires access to the LLM parameters. However, many the state-of- the-art LLMs are proprictary models deployed behind an API which only allows its users to query the LLMs
B
6
10
17 35).
"Coneesooniigg and leading author: haonand @ cs. toronto.edu Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Private Labeled Data
Not Accessible
Publicly Accessible
Instruction
Instruction
Noisy Labeling
Embed
LIM
Soft Prompt
Loss
Embeddings
L.M
Private Labeled
Instruction
Instruction
Update
Clip Noise
Unlabeled Public Data
Student
Privatized
Soft Prompt
Private Prompts
Prompt
Gradients
Gradients
(a) PromptDPSGD
(b) PromptPATE
Our methods for private prompt learning. Left: PromptDPSGD obtains the input gradients from the LLM, and performs DPSGD to update the soft prompt embedding while keeping the LLM frozen. Right: PromptPATE creates noisy ensemble of private discrete prompts, and then transfers knowledge by selecting a student prompt that can be publicly released. PromptPATE only needs black-box access of the LLM and, thus, can be easily deployed with commercial APIs.
Figure 1:
To leverage the benefits of prompting while at the same time protecting the data contained in prompts, propose the first algorithms for prompt learning with privacy. Our algorithms offer rigorous guarantees expressed using differential privacy [.5]. Perhaps closest to existing work on fine-tuning, we propose leverage the canonical DPSGD algorithm [[] to learn soft promptswith differential privacy guarantees. Our PromptDPSGD algorithm performs private gradient descent on the soft prompt embeddings prepended to the LLM's private input. Since these embeddings have very few parameters in comparison to LLMs, our PromptDPSGD is efficient and yields competitive privacy utility trade-offs a fraction of the training complexity of private fine-tuning.
However, learning soft prompts with DPSGD may not always be possible because it requires computing gradients with respect the prompt input. As mentioned previously, current APIs 3 6
10
usually do not provide these gradients. We thus turn to discrete prompts which consist of natural language tokens. Discrete prompts address the aforementioned limitations while being more data-efficient. Our insight is to observe that LLMs with discrete prompts naturally lend themselves another canonical approach of differentially private learning known as the private aggregation of teacher ensembles (PATE) 337. We introduce PromptPATE, which creates an ensemble of LLMs with different discrete prompts from the private dataset which we refer to as a flock of stochastic parrots [5). Since interacting with the flock directly can leak private information about the prompts, as we demonstrate with our MIA, PromptPATE additionally performs a knowledge transfer. Therefore, each model in the flock generates a next token prediction for a short input sequence of some public data. By performing a noisy majority vote over all models' token output, we generate a single output that, due to the noise addition, implements differential privacy guarantees while incorporating knowledge from the flock. The public input together with the noisy aggregated output form new single example for the discrete student prompt that can be prepended to the LLM in lieu of the individual prompts which contain private information. guarantees, our PromptPATE is highly efficient, since, instead of having query every model from the flock at inference time, it suffices to query the LLM prepended with the student prompt once.
17 355
addition to providing rigorous privacy
We perform extensive experiments against multiple popular LLMs, such as GPT3 O and Claude [3) that are deployed behind commercial black- box APIs. Our results highlight that PromptPATE provides high downstream performance that matches the one of non-private prompting even at very strong privacy guarantees. On the sst2 dataset with GPT3, for instance, we reach an accuracy of 92.7% with privacy costs as little as (5 0.147, 8 = 10- -00-)-diiiie --iffeerenff privacy, even when the public data used during PromptPATE's knowledge transfer stem from a different distribution than sst2. Our results closely matches the non-private baseline accuracy (95.2%). Thus, we conclude that prompt learning for LLMs is not only more efficient and practical than fine-tuning but can also achieve high utility even with strong and practical privacy protection in place.
In summary, we make the following contributions:

---

We instantiate the first MIA on prompted LLMs and show that we can effectively infer membership of the prompted data points with high success.
We propose a lightweight alternative to DP fine-tuning, namely PromptDPSGD, which optimizes orders of magnitude fewer parameters while keeping the original LLM frozen.
We propose PromptPATE, the first method for DP learning with LLMs that requires only black-box access to the model-making it easily deployable for commercial LLM APIs.
Our experiments on multiple state-of-the-art commercial APIs 6 highlight that our
methods achieve both high utility and strong privacy protections in various setups.
2
Background and Related Work
The success of LLMs, such as BERT, Claude, OPT, or different versions of GPT and their exceptional in-context learning capacities gave rise to prompt-based learning [14]
Prompts for LLMs.
39 40
35 [6]. Prompts serve as demonstrations the downstream task, which the model can then generalize from. There are two paradigms for LLM prompting, namely discrete and soft prompts.
Discrete prompts 6 16 18 27 44
natural-language instructions that contain examples from the downstream task in a well-crafted template. Tuning discrete prompts is often done by prompting the model with different combination of examples, assessing their performance the downstream task, and choosing the combination that yields the highest performance as the final prompt.
In contrast to discrete prompts, soft prompts [24 27] prepend trainable continuous embeddings to the inputs of LLMs. These embeddings are initialized either at random or with embedding vectors that correspond to tokens from the dictionary. During tuning, the embeddings are updated through gradient descent to minimize the loss of the prompted model on the private downstream task. To increase performance further, trainable embeddings be prepended not only to the input but also to every LLM layer, a technique known as prefix [26] 28 29].
Both soft prompts and prefix train end-to-end without any human involvement through backpropaga- tion over the I.I.M. On the other hand, discrete prompts have to be designed manually through careful prompt engineering. Yet, prompt engineering only needs inference passes over the LLM which makes discrete prompt more computationally lightweight. Our work provides privacy protection for all of these paradigms: discrete prompts, well as for soft prompts, and prefix.
Privacy Leakage in LLMs.. L.Ms have been shown to memorize data both from their original large training corpora 8 20 23
48 55] and from smaller private datasets used to fine-tune them for downstream tasks 33 The only prior work around privacy leakage in prompt-based
32
learning utilizes prompts to extract knowledge from trained LLMs [3] 22 38] In contrast, we study the privacy of the prompting data itself. To do so, we investigate the canonical privacy attack known as membership inference attacks (MIA) 7 Its use as a practical means to demonstrate leakage of private information in ML recently popularized by a line of work
quantifying memorization ( 43 47 While prior work utilizes MIAs to assess whether a given data point was used train LLM, we instantiate MIA to assess whether given data point was used within the prompt prepended to the inputs of a trained L.M..
Defending Against Privacy Leakage in LLMs. Prior work either focuses on training 2 19 or
fine-tuning 25 54 LLMSs with privacy guarantees. These approaches rely on the mathematical
framework of differential privacy (DP) [15) and in particular the DPSGD algorithm for private stochastic gradient descent (.] Here, DPSGD is applied to guarantee that one outputs approximately the same model parameters whether or not any given data point was used to train or fine-tune the model. To achieve this, DPSGD clips the per-example gradients that are computed during training and adds well-calibrated noise to each model update. These two operations typically increase the computational complexity of training and decrease the utility of the resulting model 4 49). To counteract these effects, -fate-of-the-a methods for full DP-fine tuning in LLMs require extensive hyperparameter tuning and vast computational resources (25). Alternative approaches refrain from updating the large number of model parameters and instead introduce additional layers into the model architecture and only fine-tune these layers with DPSGD (44 To the best of our knowledge, no prior work attempted to provide DP guarantees for prompt data in LLMs.
3

---

6-
Member Noa member
Tree Postive Rate
0.88
Density
0.4
Average AUC 0.64
0.0
0.0
0.2
0.6
0.8
1.0
Target Prediction Probability
Fullec Positive Rate
(a) Probability Distribution.
(b) AUC-ROC Curve.
Figure 2: MIA Risk. We study GPT3 prompted with 100 different one-shot examples (dbpedia). left: We present the prediction probabilities at the correct class for members (the one-shot example) and non-members (50 randomly sampled private points). The output probability for members is significantly higher than for non-member data points. right: We present the AUC-ROC curves of our MIA against the 100 prompts (gray lines) and the blue line as an average over all attacks. Given that each prompt has only one member, the resulting TPRs can only be 0% or 100% which leads to the step-shape of the gray curves. The result indicates that our attack is significantly more successful than random guessing (the red dashed line).
Setup and Notation. We denote by the soft or discrete prompt that is prepended to any input
sequence 2; when querying the language model L. For brevity, we denote L([P, x.l) by Lp (2x))) The output ys of (x²)) is an M-dimensional probability vector, with M being the size of the model's 's vocabulary. Each component of Y corrssponds to the probability that the Lp assigns the respective token for being the next token in the sequence r;- The semantic meaning of the next token varies depending on the given downstream task. For instance, for classification, the index with the highest probability indicates the token of the class that Lpp assigns to Tí.
Private Information about Prompt Data Leaks from Prompted LLMs
By instantiating a MIA against prompted LLMs, we want to highlight that the private data used within a prompt (which we refer to as prompt data from hereon) can be subject substantial privacy risk. We showcase this risk at the example of LLMs that are prompted with discrete prompts P containing tuples of demonstrations from classification downstream tasks as prompt data = {(2z. Py)]. For example, in prompt with demonstration (one-shot learning), the prompt data p may be specified as p [("The movie was great.", " "positive")), Our prompts are provided in a consistent template multiple demonstrations combined with instructions P [Instruction, (text sequence Pa, class-label token py). .....
where
For our MIA, we consider an adversary who aims at inferring whether given private demonstration (Pr,Py) was used within the prompt data p. The adversary holds n candidate demonstrations of text sequences and corresponding labels I; and queries the text sequences (x1,'.. ,.n)) to Tpp with black-bor access. The prompted model Lp then returns the output probability vectors (V1. ... yn)- Following prior work 21 (33, analyze the model's 's output probability at token ytJ1 that corre- sponds to the correct target class label of every Tū. The intuition to distinguish between members and non-members is that the output probabilities at the correct class 4 will be significantly higher for demonstrations that were used within the prompt, i.e., members with (P., Py) = (xt, 4). We show that even with this simple MIA, we can reliably determine membership for the prompt data.
Experimental Setup. We prompt GPT3-Babbage 6 with multiple one-shot examples to solve four standard downstream text classification tasks, namely dbpedia [57]. sx/2 [46), agnews [57] and trec [50]. The template of our prompts follows (58) To evaluate our MIAs, we consider the single data point used within the prompt a members and 50 other randomly selected data points from the respective task's 's training dataset as non-members. This skewed distribution between members and non-members (1 vs 50) corresponds to realistic scenario where only small proportion of the candidate data targeted by the adversary are members (2T]. To quantify the success of our attack, we report the AUC-ROC curves of 100 random trials.
'In the prefix method, Lp denotes prepending trainable parameters to every layer's input, and not only to the model input ii.

---

Results. Before evaluating the success of the MIA, we analyze the probability output from GPT3 for the correct target class between member and non-member data points. Figure FigureZajshin Saareeaasswww shows for the dbpedia dataset that the prediction probabilities for non-mem me-meerbrr are significantly lower than for
members. Figure 2blshows that this leads to a high MIA risk in terms of an average AUC score of 0.84 for the prompt data. Similar results for other datasets and models are presented in Appendix [ These results highlight that private information can leak from prompt data easily and thus motivate the urgent need for defenses which we develop in the rest of this paper.
Methods for Privacy Preserving Prompts
As of now, if we want to protect the private downstream data, we have to forego prompting altogether because, to the best of our knowledge, no algorithms for private prompt learning exist. The only alternative to privately adapt the LLM would be to perform DP fine-tuning [25] [54]. However, this
approach only feasible when we have direct access to the L.L.M to update its parameters with
fine-tuned with DPSGD [54]. This is prohibitively expensive and mostly impossible with the commercial API, thus we propose the first algorithms that enable differentially private prompt learning.
DPSGD [5 or to even change the model architecture to insert additional
We consider two main paradigms of prompting: soft prompts and discrete prompts. To learn private soft prompts, we introduce PromptDPSGD. PromptDPSGD is parameter-efficier alternative to DP fine-tuning that does not need modifying the parameters or architectures of the LLM. However, many popular APIs B 6 I0 17 33] do not support soft prompts yet as it requires gradients with respect to the input. Therefore, we propose PromptPATE for discrete prompts. PromptPATE requires only black-box access to an LLM without any knowledge of the LLM's 's architecture or mode of operation. Instead, the algorithm only needs the next-token prediction of the LLM. This, to our knowledge represents the first solution for privately adapting LLMs in restricted API setups.
4.1 PromptDPSGD: DPSGD for Private Soft Prompt Learning
In general, all discrete input tokens to LLMs are internally transformed into continuous input embeddings that the LLM then operates on. Soft prompts are just additional continuous input embeddings that can be prepended the original input embeddings before passing them through the LLM. To train (or tune) soft prompts, we require training data from potentially private downstream task. After prepending the continuous soft prompt embeddings to input examples from the training data, we can calculate the gradients for the loss of the prompted LLM with respect to these soft prompt embeddings. The gradients provide information about how the soft prompt should be updated in order to minimize the loss on the training data.
If we can obtain the gradients for soft prompts, we can learn these prompts with privacy guarantees by applying the canonical DPSGD algorithm M.]. The same applies to prefix. therefore, when we talk about soft prompts in the following, we implicitly also include prefix. We call this approach PromptDPSGD. The algorithm yields soft prompts with DP guarantees that can be deployed with the LLM to solve the respective downstream task. The privacy analysis of PromptDPSGD follows the one of the standard DPSGD. Note, however, that while conceptually similar to fine-tuning the LLM's parameters with DPSGD (54 25), PromptDPSGD differs in a crucial aspect. In DP-SGD fine-tuning, we require the gradients with respect to all or subset of the model parameters and update these parameters to minimize the loss. contrast, in PromptDPSGD, we use the gradients with respect to the soft prompt embeddings and only alter these. We highlight this difference in our PromptDPSGD-algome
that we present in Appendix C
While this difference seems subtle, it has far-reaching consequences, First, there are orders of magni- tude fewer parameters that need to be updated which increases training efficiency. Second, and most importantly, it allows us to keep operating on the original LLM. We discuss the resulting advantages, such as storage efficiency, and the ability to process multiple different tasks simultaneously at the end of this section (in 4.3) These advantages make PromptDPSGD conceptually superior to private fine-tuning. At the same time, as we show our evaluation, despite the small number of trainable parameters, ProMp for simpler tasks, matches the performance of private fine-tuning. Yet, do not support soft prompting, prefix, or private fine-tuning and only provide black-box access through discrete prompts. For these setups, we propose PromptPATE.
current APIs
3 6 10
17 35
5

---

4.2 PromptPATE: PATE for Privacy Preserving Discrete Prompts
PATE 36 37 enables learning classifiers with DP guarantees. It first trains an ensemble of teacher models on disjoint subsets of the private data. Second, through a noisy labeling process, the ensemble privately transfers its knowledge to an unlabeled public dataset. Finally, separate student model is trained on this labeled public dataset for release. The noisy knowledge transfer in the second step relies on the Confident GNMAX algorithm 37 that we detail in Appendix C It consists of three main parts: for any input data point from the public unlabeled dataset, each teacher votes for the most likely class. Then, the consensus over the teachers' votes is determined and queries with low consensus are rejected to avoid revealing too much information about the private decision boundary. Finally, the returned class label for any non-rejected data point is determined as a noisy argmax over all teachers' vote counts- the added noise is sampled from Gaussian distribution to implement the DP guarantees. For each rejected or labeled data point from the public dataset, privacy costs are accumulated and the ensemble stops labeling once target privacy budget is reached.
Our PromptPATE follows the general flow of standard PATE: training the teacher models, private knowledge transfer, and training a student model. However, due to the significant differences between in-context learning for LLMs and supervised learning in the original PATE and how these different paradigms leverage private and public data, we had to redesign each of these building blocks. This allows to leverage both the data-efficiency of prompts and the rigorous privacy protection from PATE. In the following, we present the building blocks in our PromptPATE.
Teacher Models (Flock of Stochastic Parrots). Instead of training teacher models on disjoint partitions of the private data, we use the private data to create disjoint prompts for the LLM. More specifically, we use examples, for instance [("The movie was great." ", "posifive"),..) from the private training data to create prompts that can then be deployed with the LLM teachers.
During the private knowledge transfer, the teachers label public data sequences, such as ("T did enjoy it." _). Each teacher votes with the most likely class labels for the private downstream task. In Appendix D we show that PromptPATE can also operate directly on pure next token predictions from Claude 3 without access to per-token probabiliiee--enabiin full black-bor -ox private prompts. By performing the private voting process according to standard PATE with the Confident GNMAX algorithm, we turn our per-teacher predictions into final class label token that will be appended to the sequence, e.g., ("I did enjoy it", "positive"). The privacy accounting and analysis of our PromptPATE exactly follows the one of standard PATE (37].
Private Knowledge Transfer.
Student. The most naive way to obtain a student model following standard PATE would be to label many public sequences and train a language classifier using supervised learning on this data. However, due to the relatively high number of data needed for supervised learning, and the fact that each query to the private teachers consumes privacy, this process would incur high privacy costs. We propose a better approach building on the data-efficiency of prompting (42) by using labeled public sequences to create new discrete student prompts. The selected prompt can then be deployed with the L.M as the PromptPATE student model.
In theory, labeling one public sequence by the ensemble would be sufficient create such prompt. This approach yields negligible privacy costs, but the resulting prompt might not have good utility due to the high variance in the performance of prompts [58). Therefore, we generate multiple prompts based on different labeled public sequences and perform prompt tuning to select the best student prompt. Care must be taken during selection: utility cannot be evaluated on the private data anymore given that the prompt will be publicly deployed and selecting based on the private data would incur additional privacy costs. We solve this tension by using parts of the newly-labeled public data as validation data to assess utility of the student prompts. By selecting the prompt with the highest validation accuracy, we deploy the student prompt that most resembles the private teachers.
4.3 Advantages of (Private) Prompting over (Private) Fine- Sine-Tuning
Our private prompt learning enables us to leverage the general advantages of prompting over fine- tuning while preserving privacy. Private prompting requires significantly less storage than private fine-tuning. While fine-tuning requires storing a separate copy of the LLM model for each downstream task (24), prompts operate only on the input level of LLMS without adapting model parameters, such that only a small task-specific prompt needs to be stored for each downstream task. For example, each copy of the fine-tuned RoBER BERTa base model requires 125M parameters (~500MB). This becomes

---

M
Soft- Prompt (Our)
Prefix (Our)
Full-Tuning 22
LoRA-Tuniin Tuning
Dataset
<10K
<100K
125M
1.2M
8 =
E = 20
E= 8
00
E
5 = 00
5 =
E = 00
sst2
92.31
95.64
91.97
96.33
85.89
96.40
92.97
96.60
qnli
84.11
89.48
87.17
94.84
84.81
94.70
88.59
94.70
पчР 81.52
86.56
82.58
91.42
86.15
92.20
86.26
92.20
mnli 75.15
82.49
80.57
90.34
83.30
90.20
82.92
90.20
Table Performance of PromptDPSGD. We report the accuracy values (%) for each dataset. All values are reported as standard DP guarantees. We run the experiment RoBERTa [30]. The first
row M: the type of the private Method, the second row the number of Parameters tuned for the method, and the third row G: DP Guarantee. We also present results for 3 in Appendix D
prohibitively expensive, especially as the number of parameters for state-of-thh-ane LLMs rapidly increases. In contrast, soft- soft-prompts and prefix, the generated by PromptDPSGD (using implementation from [29]) with the standard prompt length of 10 tokens require less than 10K parameters (40KB) for the soft-prompt and 100K parameters (400KB) for the prefix. A discrete prompt, such as the one generated in PromptPATE, requires less than 1 KB of prepended text. Prompts also enable processing many examples from different tasks in a single batch [26), called mixed-tas tixeed-taskaa inference. This allows more efficient use of LLMs since we do not have to wait for sufficient number of requests for a single task before processing them. This is not possible with any form of fine-tuning, where the fine-tuned model can serve solely a single task.
Experimental Evaluation
We evaluate both PromptDPSGD and PromptPATE and show that they match the performance of non-private prompting while providing strong privacy guarantees.
5.1 PromptDPSGD
Experimental Setup. To train soft-prompt and prefix, we follow the experimental setup from prior work on DP fine-tuning. Specifically, we use differentially-privaal optimization engines for transformers, such models from the BERT family for the language understanding tasks. The experimental results for classification were performed on the RoBERTa models [30), using the standard NLP datasets, namely sst2, qnli, qタp. and mnli, from the GLUE benchmark (51). Our imple- mentation for soft-prompt and prefix is based on P-Tuning v2 29). To tune the (hyper)parameters for PromptDPSGD, the default value of 10, which commonly yields good performance). For the privacy parameters, we set the 5 1/N, where N is the number of data points in given dataset, The clipping threshold of per-example gradients is set to 0.1 in most cases. We use a batch size of 1024, The detailed selection of (hyaee-)parameters presented in Appendix E
adjust the length of the soft soft-prompt prefix in the private setting (with
Results. We compare our PromptDPSGD against state-of-the-art approaches for private fine- tuning on multiple private downstream datasets. Our results are shown in Table [ We highlight that both soft prompts and prefix provide competitive privacy utility trade-offs. For example, the difference in accuracy values between the non-private baseline and the private soft prompt ranges from 3% (for the simplest sst2 dataset) and up to 7% (for the most difficult mnli dataset). This mirrors results for other private methods, such as the private fine-tuning of LoRA (54). We also observe that, similarly, for simple tasks, such sst2 or qnli, the performance of soft prompt or prefix matches the one of fine-tuning. For the more difficult tasks, namely qqp and mnli, the performance of prefix and soft prompts is also relatively close to fine-tuning. The results obtained for these methods are highly influenced by the number of optimized parameters. For example, for the SST2 model, the prefix requires 19970 additional parameters while soft prompt adds solely 2306 parameters. On the other hand, the number of privately tuned parameters is a few orders of magnitude bigger for fine-tuning and equal to the size of the trained model, namely 125M for the method proposed in (25), while the fine-tuning approach from [54) optimizes around 1.2M
task and the RoBERTa-
7

---

Lower Bound
Ens.
Upper Bound
Our PromptPATE
Acc.
IID Transfer
OOD Transfer
Private
E 0
Public
Test acc
Public
Test acc
sst2
76.3
90.0
93.8
sst2
0.178
88.8+2.3
imdb
0.187
87.221.
agnews
62.0
72.8
78.2
agnews
0.248
71.7
arisetv
0.258
67.9
trec
40.7
57.6
58.7
trec
0.281
52.8 11.5
qqp 0.293
50.9
dbpedia
44.2
81.6
85.6
dbpedia
0.194
80.3 +1.3
agnews 0.203
74.6214
sst2 (C)
82.0
94.0
95.2
sst2
00.17
92.3
#1.1
imdb
0.154
92.7 #0.8
agnews (4)
62.0
75.8
81.0
agnews
0.145
73.5
11.2
arisetv 0.145
69.6 11.8
Table 2: Performance of PromptPATE. We compare PromptPATE with three baselines: zero-shot (Lower Bound), the ensemble's accuracy (Ens. Acc), and the non-private baseline (Upper Bound) on four classification benchmarks. We study two settings, (IID Transfer) when the public dataset is from the same and (OOD Transfer) different distribution than the private data. We find that PromptPATE achieves strong privacy protection (s < 0.3 at 5 = 10-6) and utility close to the non-private and significantly higher than the zero-shot. Unless otherwise specified, the experiments are performed on GPT3- BrT3-Babbage with one-shot prompts. Additionally, we also run experiments on GPT3-Curie for sst2 (C) and 4-shot prompts for agnews (4).
parameters. Our results reflect general trend, where prompts are suited for small downstream tasks while fine-tuning with its bigger number of parameters can also cater to more complex tasks with larger training data sets.
5.2 PromptPATE
Experimental Setup. Teachers: Unless otherwise specified, we rely GPT3-Babbage as the base LLM and select one-shot examples randomly without replacement from the private downstream task as prompt data. Our prompt template follows Zhao et al. [58]. For each setting, we deploy 200 teacher prompts. Private knowledge transfer: We use the implementation of PATE's Confident GNMAX algorithm and the privacy accounting from [12] and report
algorithm's 's hyperparameters in Appendix [ Student: For each private downstream task, we experiment with two setups (1) selecting public input sequences from the same (IID) and (2) from different distribution (OOD) as the private data. We introduce three new datasets for the OOD setup: imdh [31], arisetv [1] and qqp [52]. The details of preprocessing these datasets can be found in Appendix [E] In both the III) and OOD setup, we limit the size of the public dataset to 500 input sequences from the respective datasets. After the ensemble finishes labelling, we select the best labeled public sequence as prompt data based on the validation accuracy on the labeled public set. We repeat the process three times and report average and standard deviation of the test accuracy for the selected student prompt on the private test set. To improve utility. both teachers' and students' output probabilities from GPT3 are recalibrated using contexual calibration (58).
Results. We compare PromptPATE against three baselines: the lower bound baseline represented by zero-shot prediction (s = 0), i.e., when the LLM is only prompted with an instruction, the private ensemble accuracy (5 = oo), and the upper bound as non-private one-shot prediction (e oo) using the best example from the private data as prompt data. (To save costs, we select from 200 candidates.) Table shows that, over all setups, PromptPATE achieves similar utility to the non-private baseline and significantly improves over zero-shot predictions- privacy protection (e < 0.3, 5 = 10-6). Our results also highlight that the distribution of the public data does not need to be very close to the distribution of the private data to yield high-utility student prompts. For example, they can be collected from different domains (dbpedia holds extracts from wikipedia while its public data agnews contains news articles) and for different tasks (tree aims to classify the topic of given answer while qqp serves to measure the similarity of two questions). Still, with dbpedia being the private downstream data and agnews as public, we achieve an accuracy of 74.6%, which is significantly higher than the zero-shot baseline with 44.2%.
at very strong
We also provide further insights into the privacy-utility trade-offs that can be achieved with Prompt- PATE in Figure 3b) Our results highlight that with more public sequences queried to the ensemble, the privacy consumption increases while, after roughly 100 queries, with even < 0.2, the student model's 's test accuracy saturates. This yields very favorable privacy-utility trade-offs which we attribute
8

---

Student Test Accuracy
Numbor Queuriis
150-
-70
30
65"
0.2
0.8
1.0
60
Proportion Teachers Agreed
Public Dutaset Size
(a) Teacher Consensus
(b) Utility-privacy tradeoff
Figure 3: Additional Insights PromptPATE. We perform ablation studies on GPT3-Babbage and use dbpedia as private and agnews as public data. Left: Teacher consensus as the fraction of teachers who vote for the correct class over 500 public input sequences. PromptPATE achieves overall high consensus. Right: Student accuracy as a function of the public query set's 's size. Already with as few as 100 queries, we observe a plateau in accuracy which highlights PromptPATE's data efficiency.
mainly to the data efficiency of discrete prompts: Even from within as little as 100 labeled examples, high-performing student prompt can be derived. Additionally, we observe that the per-query privacy costs of PromptPATE are relatively low, further benefiting the privacy-utility trade-off. The small privacy costs result from the high consensus between the teacher predictions" see Figure 3a that might result from all teachers relying on the same underlying LLM, just with different prompts.
Scalability. Finally, we also study how PromptPATE scales with larger LLMs and more examples in the prompt. We experiment with a more performant LLM (GPT3 (PPT--Curie) for sst2. Due to the higher per-query costs, we are not able to repeat this experiment for all datasets. Our results show that performance our private prompt increases together with performance of the public prompt (92.7% accuracy on Curie vs. 87.2% on Babbage) while the privacy budget decreases (from 0. 78 to 0.147). To investigate flexibility in terms of numbers of private examples provided as prompt data, also experiment for agnews with 4-shot teachers. Similar to the non-private study [58] that reports improvements for agnews in the 4-shot setting over 1-shot, we observe that this improvement also translates the private prompt. Our results indicate that with increasingly more powerful LLMs and larger context windows, private prompting will increase further terms of privacy-utility trade-offs.
Conclusions and Outlook
By instantiating the first simple yet effective membership inference attack against prompted LLMs, we show that they leak private information about their prompt data. We propose private prompt learning holistic and broadly applicable PromptDPSGD that enables to train soft- peft-prompts with privacy guarantees. In contrast to fine- tuning, soft prompts optimize significantly fewer parameters and do not require any update of LLM parameters or changes to its architecture. As the first solution to private downstream learning with L.MMs in black-box access scenarios, we propose PromptPATE. PromptPATE builds the highly data-efficient discrete prompts and implements privacy through a noisy knowledge transfer. Through our evaluation against two popular LLMs deployed behind commercial black-box APIs (GPT3 and Claude) 6 3, we highlight that this method yields downstream performance that matches the one of non-private prompting at very strong privacy guarantees. As LLMs rapidly improve and increase in size, prompts are achieving consistently higher performance while fine-tuning becomes more challenging this scale. This suggests that privacy protections for prompts will become even more important, especially context sizes expand.
approach to mitigate this risk. We first introduce
Acknowledgments
We would like acknowledge our sponsors, who support our research with financial and in innkind contributions: Amazon, Apple, CIFAR through the Canada CIFAR AI Chair, DARPA through the
As motivated in Section 4.2 high consensus reveals less information about the private decision boundary. and hence incurs smaller privacy costs.

---

GARD project, Intel, Meta, NSERC through Discovery Grant, the Ontario Early Researcher Award, and the Sloan Foundation. Resources used in preparing this research were provided. part. by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We also thank members of the CleverHans Lab for their feedback.
References
[1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pages 308-318, 2016.
[2] R. Anil, B. Ghazi, V. Gupta, R. Kumar, and P. Manurangsi. Large-scale differentially private bert. arXiv preprint arXiv: 208. 102.0/44,, 2021.
[3] Antropic. Introducing claude. Antropic Website. 2023-03-14, https:// https:///wwaaatthoii..
claude!
[4] R. Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464447. IEEE, 2014.
[5] E. M. Bender, T. Gebru, A. McMillan-Maii and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
[8] N. Carlini, D. Ippolito. M. Jagielski. K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07647 2022.
[9] N. Carlini, F. Tramer, F. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/), pages 2633-2650) 2021.
[10] H.-T. Cheng and R. Thoppilan. Lamda: Towards safe, grounded, and high-quality dialog models for everything. Google Blog Post. 2023-05-09, ltttps:/ : wiggoogeeblog.goooooooooooooooo. www.comm////// 2022/01/
1 tosiaattoorddd-ssaa grounded- high. html
[11] O. chimaobi Samuel. news-data. Huggingface, 2022.
[12] C. A. Choquette-Choo, N. Dullerud, A. Dziedzic, Y. Zhang, S. Jha, N. Papernot, and X. Wang. Cape learning: Confidential and private collaborative learning. In International Conference on Learning Representations, 2021.
[13] Davison, J. Feldman, and A. M. Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), pages 1173-1178, 2019.
[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.00 0405, 2018.
[15] C. Dwork. Differential privacy. In Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14 2006, Proceedings, Part II 33, pages 1-12. Springer, 2006.
10

---

[16] T. Gao, A. Fisch, and D. Chen. Making pre- pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
[17] Google. Lamda: Towards safe, grounded. and high-quality dialog models for everything. Website, 2023. 2023, https: google.co
[18] H. Guo, B. Tan, Z.. Liu, F. Xing, and 7. Hu, Efficient (soft) q-learning for text generation with limited good data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 199996991, 2022.
[19] S. Hoory, A. Feder, A. Tendler, S. Erell, A. Peled-Cohen, I. Laish, H. Nakhost, U. Stemmer, A. Benjamini, A. Hassidim, al. Learning and evaluating a differentially private pre-trained language model. In Findings of the Association for Computational Linguistics: EMNLP 2021,
pages 1178-1189 2021.
[20] D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette- Choo, and N. Carlini. Preventing verbatim memorization in language models gives a false sense of privacy, arXiv preprini arXiv:2210. 17546. 2022.
[21] B. Jayaraman, L. Wang, K. Knipmeyer, Q. Gu, and D. Evans. Revisiting membership inference under realistic assumptions. Proceedings on Privacy Enhancing Technologies, 2021 2021.
[22] Z. Jiang, F. F. Xu, J. Araki, and G. Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[23] E. Kharitonov, M. Baroni, and D. Hupkes. How bpe affects memorization in transformers. arXiv preprint u 2021.
[24] B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, Nov. 2021.
[25] X. Li. Tramer, P. Liang, and T. Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2022.
[26] X. L. Li and P. Liang. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, 2021.
[27] Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06800 2021.
[28] X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68, Dublin, Ireland, May 2022. Association for Computational Linguisties.
[29] X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, 7. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2 10,07660 2021.
[30] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis. L. Zettlemoyer, and V. Stoyanov. Ro(bert) ja: A robustly optimized (bert) pretraining approach, 2020.
[31] A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng. and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
[32] R. T. McCoy, P. Smolensky, T. Linzen, J. Gao, and A. Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. arXiv preprint arXiv:2111.09550 2021.
[33] Mireshghallah, A. Uniyal, T. Wang, D. Evans, and Berg-Kirkpatrice Memorization in nlp fine-tuning methods. arXiv preprint arXiv:2205. 2022.

---

[34] R. Mitchell. Samsung fab data leak: How chatgpt exposed sensitive information. electropages.
[35] OpenAl. Gpt-4 technical report, 2023.
[36] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar. Semi-supervised knowl- edge transfer for deep learning from private training data. In International Conference Learning Representations, 2017.
[37] N. Papernot, S. Song, I. Mironov, A. Raghumathan, K. Talwar, and U. Erlingsson. Scalable private learning with pate, In International Conference on Learning Representations, 2022.
[38] Petroni, Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and Riedel. Language models as knowledge bases? arXiv preprint 1rXiv: 2019.
[39] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training 2018.
[40] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[41] C. Raffel, N. Shazeer, Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(5)55485-5551 2020.
[42] T. L. Scao and A. M. Rush. How many data points is a prompt worth? arXiv preprint arXiv:2/03.08493, 2021.
[43] V. Shejwalkar, H. A. Inan, A. Houmansadr, and Sim. Membership inference attacks against NLP classification models. In NeurIPS 2021 Workshop Privacy in Machine Learning, 2021.
[44] T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
[45] R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEFE, 2017.
[46] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pages 1631-1642, 2013.
[47] C. Song and V. Shmatikov. Auditing data provenance text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 196-206 2019.
[48] K. Tirumala, A. H. Markosyan, L. Zettlemoyer, and A. Aghajanyan. Memorization with- overfitting: Analyzing the training dynamics of large language models. arXiv preprint 10770 2022.
[49] F. Tramer and D. Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations, 2021.
[50] E. M. Voorhees and D. M. Tice. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200-207, 2000.
[51] A. Wang, A. Singh, J. Michael, Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task bench- mark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
[52] Z. Wang. W. Hamza, and R. Florian. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 1144-4150 2017.
12

---

[53] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfiting. 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[54] D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, S. Yekhanin, and H. Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022.
[55] C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini. Counterfactual memo- rization in neural language models. arXiv preprint arXiv:2112. 2021.
[56] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205. 000668 2022.
[57] X. Zhang. J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances neural information processing systems, 28, 2015.
[58] Z. Zhao, E. Wallace, S. Feng, D. Klein, and Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference Machine Learning, pages 12697-12706. PMLR, 2021.
13

---

A Broader Impacts
The growing importance of in-context learning a paradigm for leveraging LLMs private downstream tasks has significant implications for privacy. We present the first approaches for obtaining prompts with privacy guarantees, thereby enabling the use of this learning paradigm sensitive data. This advancement has the potential to increase trust and acceptance of JLM--ased systems for private applications. Our approach PromptPATE is the first viable technique for private downstream adaptation of black-box LLMs, which enables integrations into the state-of-the-an commercial LLM APIs. We acknowledge that-as as with any application that relies DP-care must be taken when choosing the privacy parameters and since setting these incorrectly can lead to false sense of privacy. Therefore, our work orientates at the privacy parameters that have been shown to provide reasonable protection in prior work. Thereby, we also ensure consistency and comparability in evaluations between the different appraoches.
B
Limitations
Tuning Instructions and Templates. For our discrete prompts, we did not tune the instructions or templates but instead relied template from prior work (58). The effectiveness and performance of our PromptPATE could potentially be further improved by tuning the instructions and templates.
We build on pretrained LLMs to learn and deploy our private prompts. Our methods solely target the protection of the private data used for these prompts. However, is also important acknowledge the inherent privacy risks for data used to pretrain the LLM. We leave the pretrainig of LLMs with privacy guarantees to orthogonal line of work.
Privacy Risk of Pretrained LLM:
Limited Monetary Budget for our Experiments. Due to cost limitations, we were unable to experiment with the latest and best available model, GPT4. Our experiments with GPT3-Curie in comparison to less powerful GPT3 BaTbbBe however indicate the clear trend the our private prompts improve in performance again due to the cost limitation, we were not able to incorporate a larger number of teachers in our experiments for PromptPATE. Therefore, the best non-private teacher baseline that report might not be the best achievable one had more teachers to choose from. We chose from 200 and note that with more (and potentially better teachers), not only the baseline but also the teacher ensemble's performance would get better.
the non- non-private baseline improves due to better models. Furthermore,
Hyperparameter Tuning. To save computation costs, did exhaustively tune all hyperpa- rameters in our experiments. While our approach still achieves high utility and good arivacy-utilig trade-offs, we acknowledge that with more hyperparameter tuning the performance together with the understanding of optimal configurations for private prompt learning could increase.
Assumption of a Trusted LLM API Provider. our work, the API provider gets to interact with the private data, for example, through the teachers' prompts in PromptPATE. Therefore, we have to assume trust in the API provider. The privacy guarantees through our private prompt learning protect the privacy of the prompt data against users that interact with the prompted LLM. In practice, companies that are concerned about the privacy of their data with respect to the API provider could make contracts with the API providers the use of their data or buy access plans that guarantee that data queried to the API is treated privately. We leave implementing cryptographic approaches that trusting the API provider entirely, for example, by enabling the LLM
could relief the assumption to run inference on encrypted private data to future work.
C Additional Insights into our Methods
C.1 PrOMptDPSGD
We present the full PromptDPSGD algorithm in Algorithm I
14

---

Algorithm 1: PromptDPSGD, In contrast to the standard DPSGD algorithm that updates model parameters during private training or fine-tuning, our PrompiDPSGD privately updates the soft prompt parameters. We highlight these changes with respect to standard DPSGD training fine-tuning blue.
Require: Private downstream data D = [[x;, l E [N]), prompi sequence length embedding dimensionality trained LLM L with frozen parameters, loss function ((L,, 2)) for prompted LLM, Params: learning rate ft. noise scale a, sampling rate q, max gradient norm c, training iterations T.
1: Initialize Po E IR xe at random
2: for f E [7] do
3: Sample mini-batch B, according to sampling rate q from D ( (Poisson sampling)
4: For each E [Be), compute (x(),, < Vr, ((Tp (Compute per sample gradient w.r.t. pc]
5: (x.), - (x.))/ max (1, [Is:(2))2) (Clip gradient)
6: g:
- P2 [2, (zi) + + (0, r)) (Add noise)
7:
Pa+1 P. n: {Update soft prompt)
8:
end for
9: Output pr and compute overall privacy cost (e, 8).
C.2 PromptPATE
Extended Background on PATE. We include the standard Confident-GNNMax Aggregator Algo- rithm from (37
Aggregaaaa
below.
Algorithm 2: GNNf Aggregator by [37]
Require: input z, threshold T, noise parameters σ1 and
if max, {(iige) Ri., (z)) NV(0, of) 2 T then
2: Output arg max, [EETE (x, ) + N(0,o²))
3: else
4: Output
5: end if
C.3 Privacy Analysis
PromptDPSGD. Our PromptDPSGD can be seen as repeated sampled Gaussian mechanism M with sampling performed over the entirety of the private prompt dataset. The difference to standard DPSGD for training or fine-tuning is that we do not update the model parameters, but the trainable embeddings for the soft prompts. This is conceptually different from standard DPSGD in terms of which parameters are updated. The privacy guarantees of the training mechanism still follow Abadi et al. (1. but with respect to the soft prompt embeddings: whether or not particular data point will be included the private training set used for tuning the prompt, the resulting soft prompt embeddings after training will be roughly the same. Especially by applying the clipping operation at every step, each mechanism's 's sensitivity is bounded by c. Privacy is then implemented as the trainable soft prompt embeddings are updated while adding noise noise drawn from (0,, c2
Theorem 1 (Privacy of PromptDPSGD). Let T be the total number of repetitions (training itera- tions) of our PromptDPSGD and the sampling rate be denoted by q. Then, there exist two constants en and C22 such that for any < T our PromptDPSGD guarantees (s, (e,8)-DP, if for any > 0,
qev Tey/Toog1/
we choose the noise according to 2 02
Proof. The proof follows the one by Abadi et al. M. using their moments accountant that models the privacy loss as a random variable dependent on the stochastic noise added.
PromptPATE. Our PromptPATE relies entirely on the Confident GNMAX algorithm from Pa-
pernot el al. 337 We preserve the assumption underlying the algorithm and the respective privacy analysis that the sensitivity during the voting mechanism equals one. This is done in PromptPATE by assigning disjoint data points from the private prompt downstream dataset to all teachers. As consequence, the privacy analysis of our PromptPATE entirely follows Papernot al. [37].
15

---

Both our PromptDPSGD and PromptPATE experience the post-processing properties of DP, i.e., once trained, the privacy guarantee (E, 6) sets an upper bound on privacy leakage for the prompt data, independent the number and type of queries that will be posed to the final prompted L.M.
D Additional Results
D.I Membership Inference Attacks
We present the full results of MIA against GPT3 with one-shot prompts on 4 datasets SCW; Section 4
1.
BERKE
New
02
Pree Prababilyy
Torget
Prakabilky
Turget Pentiction Probabilky
Torget Precirtio Probabilty
(a) agnews
(b) dbpedia
(c) sst2
(d) tree
1.0
Austage
0.4 Faloor Pondice Bate
Jake Pastire
Palle Pancicce Rate
Pake Pastiie
(c) agnews
(f) dhpedia
(g) sst2
(h) tree
Figure 4: MIA Risk Multiple Datasets GPT3. We study GPT3-babbage prompted with 100 different one-shot examples on four datasets. Top: We present the prediction probabilities at the correct class for members (the one-shot example) and non-members (50 randomly sampled private points). The output probability for members is significantly higher than for non-member data points. Botiom: We present the AUC-ROC curves of MIA against the 100 prompts (gray lines) and the blue line as an average over all attacks. Given that each prompt has only one member, the resulting TPRs can only be 0% or 100% which leads the step-shape of the gray curves. The result indicates that our attack is significantly more successful than random guessing (the red dashed line).
In addition, we also perform similar experiments on GPT2-xl with four-shot examples, with results presented in Figure 5 We replace dbpedia with eb because the input dbpedia is usually longer than the context length of GPT2.
D.2 PromptPATE on Claude
We present the experiment results of PromptPATE Claude (3. Different from GPT3 that outputs logits over the whole vocabulary, Claude only gives us access to the next most likely token.
Experimental Setup. Teachers: We rely on Claude-v1 as the base LLM. We use 2-shot prompts for sst2 and agnews, 4-shot for trec and 1-shot for dhpedia. We set the maximum generated tokens to 1 and temperatures to O. We also create an "other" category in case the moel's output does not fall under any specified categories. For each setting, we deploy 400 teacher prompts. Private knowledge transfer: We use the implementation of PATE's Confident GNMAX algorithm and the privacy accounting from [2] and report our algorithm's 's hyperparameters Appendix D Student: We limit the size of the public dataset to 200 input sequences from the respective datasets. The number of shots for students corresponds with the teachers.
D.3 More results for PrOMptDPSGD
We present the additional results for PromptDPSGD with the classification tasks in Table
16

---

Sea scaler
15
28-
67 0. BA
6.2
0.4
96
02 6.4
Ed
16
03
(a) agnews
(b) cb
(c) sst2
(d) trec
2021
10
14
54
63
10
62
644 0.5 03
63
6.2
644
03
02
0.4
04
(c) agnews
(f) cb
(g) SSE2
(h) trec
Figure 5: MIA Risk over Multiple Datasets on GPT2-xl (4 shot). We study GPT2-xl prompted
with 100 different four-shot examples four datasets. top: We present the prediction probabilities at the correct class for members (the one-shot example) and non-members (50 randomly sampled private points). The output probability for members is significantly higher than for non-men non-member data points. bottom: We present the AUC-ROC curves of our MIA against the 100 prompts (gray lines) and the blue line an average over all attacks. Given that each prompt has only one member, the resulting TPRs can only be 0%, 25%, 50%, 75% 100% which leads to the step-shape of the gray curves. The result indicates that our attack significantly more successful than random guessing (the red dashed line).
Lower Bound
Ens. Acc.
Upper Bound
Our PromptPATE
Private
=
0o
00
Public
6
Test acc
sst2
92.7
96.0
98.0
sst2
0.048
95.7 1.4
agnews
72.4
79.1
82.7
agnews
0.056
74.6 + 1.5
trec
69.0
79.9
82.2
trec
0.068
79.3 + 1.2
dbpedia 88.0
92.4
93.5
dbpedia
0.042
90.9 + 0.6
Table 3: Performance of PromptPATE Claude. We compare PromptPATE with three baselines: zero-shot (I (Lower Bound), the ensemble's s accuracy (Ens. Acc), and the non-private baseline (Upper Bound) on four classification benchmarks. We find that PromptPATE achieves strong privacy protection (s < 0.1 at 8 = 10-6) and utility close to the non-private and significantly higher than the zero-shot.
E Additional Setup
E.1
PromptDPSGD
We train PromptDPSGD on NVIDIA A100 GPUs. We execute into account learning rate (LR), max grad norm (GRAD), number of epochs (Epochs), the token length of prefix and prompt. In general, we find that the prompt and prefix token length of 10 is close to the optimal value in most cases. For the private (hyper--parameeers, in most cases tune for E = 8 and use similar (or even the same) parameters for other e values. We set the max grad norm to 0.1 most cases and then adjust the number epochs (the more the better, for example, 100), and the learning rate [54 The batch size is set by default to 1024.
search that takes
We show the specific parameters chosen for PromptDPSGD in Table 6
'ee would like to thank the authors of (54 for their help, especially for the very useful and practical pieces of advice how tune the parameters for differential privacy from Huseyin A. Inan.
17

---

Soft-Pro Poft-Prompt (Our)
Prefix (Our)
Full-Tuning 22
LoRA-Tumee Tuning [54]
Dataset
<10K
<100K
125M
1.2M
8
E = 20
== 3
00
E 3
5 00
5
E 00
SST2
90,48
95.64
90.37
96.33
91.86
96.40
92.60
96.60
QNLI
83.62
89.48
86.05
94.84
87.42
94.70
86.97
94.70
QQP
80.29
86.56
80.89
91.42
85.56
92.20
85.12
92.20
MNLI
73.97
82.49
80.10
90.34
82.99
90.20
82.08
90.20
Table
Private classification with soft prompts and prefix for {3, and the
RoBERTaBASE model. We use the same setup and notation as in Table
M
Soft-Prompt (Our)
Prefix (Our)
Full-Tuning 25
Dataset
P
<10K
<100K
125M
SST2
91.05
93.58
90.94
QNLI
87.62
89.45
89.42
QQP
82.29
83.50
87.49
MNLI
76.05
86.71
86.28
Table 5: Private classification with soft prompts and prefix for E = 8 and the ROBERTaLARGE model. We use the same setup and notation in Table T
E.2 PromptPATE
E.2.1
Hyperparameters for GoNfi
We present our hyperparameters for Confident-GNMan in Table
E.2.2 Dataset Preprocessing
sst2, trec, agnews, dbpedia and cb are taken from the repo of (58] All other public datasets are downloaded from huggingface. To reduce the cost of quering APIs, we randomly sample 300 points from the test set to report the test accuracy. For imdb, we random select one sentence from each entry and also remove the <br/> tag. For qqp, we only take the column of "question 1 in the public set.
18

---

Dataset
Method
RoBERTa
BS
LR
E
GRAD
Epochs
P-Length
Accuracy (%)
SST2
Prompt
Base
1024
0.005
N/A
60
100
93.23
SST2
Prompt
Base
900
0.05
8
0.01
9
92.32
SST2
Prompt
Base
1024
0.005
0.05
100
10
86.35
SST2
Prompt
Large
2048
0.005
8
4
100
10
91.05
SST2
Prefix
Base
32
0.01
8
N/A
60
20
94.61
SST2
Prefix
Base
1000
0.05
8
4
22
1
91.97
SST2
Prefix
Base
1024
0.01
0.2
100
50
90.37
SST2
Prefix
Large
2048
0.05
8
4
22
93.58
QNLI Prompt
Base
1024
0.005
N/A
60
128
89,48
QNLI Prompt
Base
1024
0.005
0.05
100
84.11
QNLI Prompt
Base
1024
0.005
3
0.1
100
50
83.62
QNLI Prompt
Large
2048
0.01
0.05
100
87.62
QNLI Prefix
Base
1024
0.005
N/A
60
20
94.84
QNLI
Prefix
Base
1000
0.03
8
0.07
22
10
88.77
QNLI
Prefix
Base
1024
0.01
3
0.2
100
50
85.78
QNLI
Prefix
Large
2048
0.03
8
0.07
22
10
89,45
QQP
Prompt
Base
1024
0.005
DC
N/A
60
50
86.64
QQP
Prompt
Base
1024
0.05
8
0.1
10
7
82.58
QQP
Prompt
Base
1024
0.001
3
0.01
100
15
80.29
QQP
Prompt
Large
2048
0.005
8
0.05
100
10
82.29
QQP
Prefix
Base
1024
0.005
X
N/A
60
20
91,42
QQP
Prefix
Base
1024
0.05
8
0.1
10
7
82.59
QQP
Prefix
Base
1024
0.05
3
I
15
2
80.89
QQP
Prefix
Large
2048
0.05
8
0.1
10
7
83.50
MNLI
Prompt
Base
32
0.001
0C
N/A
60
20
82.49
MNLI
Prompt
Base
1024
0.005
8
0.05
60
10
75.01
MNLI
Prompt
Base
1024
0.005
0.05
100
10
73.97
MNLI
Prompt
Large
2048
0.005
8
0.2
60
10
76.05
MNLI
Prefix
Base
32
0.001
0c
N/A
20
82.49
MNLI
Prefix
Base
1024
0.005
8
0.05
60
50
80.42
MNLI
Prefix
Base
1024
0.005
0.2
100
50
80.10
MNLI
Prefix
Large
2048
0.01
8
0.1
100
10
86.71
Table 6: Detailed parameters for soft prompts and prefix. Type is the type of training, BS
represents batch size, LR denotes the learning rate, is the DP guarantee, P-Length the token length of soft- soft-prompt prefix.
LLM
Dataset
T
σi
02
GPT3
sst2
180
20
GPT3
agnews
180
5
20
GPT3
trec
180
20
GPT3
dbpedia
170
1
20
Claude
sst2
390
1
50
Claude
agnews
360
Claude
trec
320
50
Claude
dbpedia
320
Table Detailed parameters for Confi
19

---

agnews
Sst2
1.0
1.0
True Positive Rate
0.8
0.8
True Positive
0.6
0.6
0.4
0.4
0.2
0.2
Average AUC
0.48
Average AUC
0.49
0.0
0.0
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.8
1.0
False Positive Rate
False Positive Rate
(a) agnews
(b) sst2
dbpedia
trec
1.0
1.0
True Positive Rate
True Positive Rate
0.8
0.8
0.6
0.6
0.4
0.1
0.2
0.2
Average AUC
0.52
Average AUC
0.51
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(c) dbpedia
(d) tree
Figure 6: MIA against the public prompts of PromptPATE. We depict the AUC-ROC curve of MIA against the public prompts of PrompIPATE. The member data is the examples from the prompts of all private teachers, and the non-members are randomly-selected data from the training set. Each blue curve corresponds to a different public prompt selected in one random trail. All curves are very close to the red dash line (random guess), which show that our PromptPATE is effective against MIA.
20


Reviews:
[{"review_id": "xF4q66momN", "reviewer": "Reviewer_p7La", "summary": "This paper presents differentially private techniques for optimizing continuous and discrete prompts for solving text classification tasks using Large Language Models (LLMs). The need for differentially private prompt learning is motivated by showing that examples used in few shot prompts for classification tasks are highly susceptible to simple membership inference attacks\n\n- For optimizing continuous prompts, the paper uses DP-SGD in combination with existing parameter efficient prompt tuning techniques. When used in combination with either soft prompt or prefix tuning, the method is generically called **PromptDPSGD** in the paper. This method requires access to gradients for tuning and the ability to adapt the model at inference time with tuned parameters, neither of which is typically supported in consumer APIs.\n\n- For optimizing discrete prompts, the paper adapts the Private Aggregation of Teacher Ensembles (PATE) approach to label public examples using an ensemble of LLMs prompted with private few shot examples in order to learn a few shot prompt. This method is called **PromptPATE** in the paper. This method can be implemented on top of existing LLM completion APIs, even when they do not expose token prediction probabilities.\n\nThe paper evaluates the utility of using **PromptDPSGD** to fine-tune RoBERTa-Base on the SST-2, QNLI, QQP, and MNLI tasks from the GLUE benchmark with $\\varepsilon \\in$ {3,8}. It compares it to parameter efficient LoRA-tuning and full fine-tuning using DP-SGD. The results show that the accuracy of **PromptDPSGD** tuned models is competitive despite having orders of magnitude fewer tunable parameters than full fine-tuning and LoRA-tuning.\n\nThe paper evaluates **PromptPATE** using GPT-3 (Babbage and Curie) on SST-2, TREC, AG News, DBPedia. The results show that for $\\varepsilon < 0.3$, it achieves similar accuracy to few-shot (1- and 4-shot) classification and to a non-private majority vote using the same teacher ensemble.\n\nThe supplemental material contains an Appendix describing hyperparameter choices, additional experimental results, and a discussion of broader implications and limitations. It also includes implementations of **PromptDPSGD** and **PromptPATE** built on existing libraries.", "questions": "a) Table 2 reports for AG News and IID public data 1-shot $\\varepsilon = 0.248$ and 4-shot $\\varepsilon = 0.145$. How can the privacy budget be lower for 4-shot than 1-shot?\n\nb) I tried to wrap my head about how you produced the ROC curves in Figure 2b (and in the Appendix), but after reading the description in the paper many times I still do not fully understand it. \n\nDo I get it right that your attack infers that $(x, l)$ is a member if and only if $\\arg \\max_j L_P(x)_j = l$?\n\nOr does the attack take 51 different examples $(x_i, l_i)$ and infers that the example with index $\\arg \\max_i L_P(x_i)_{l\\_i}$ is a member and the rest are not? Can you provide a pseudocode description of how the points in the ROC curve are computed?", "limitations": "Yes, both broader societal impact and limitations.\n\nAppendix A discusses societal impacts. It highlights the risk of overreliance on DP guarantees and the need to correctly select the privacy hyperparameters $\\varepsilon$ and $\\delta$.\n\nAppendix B discusses limitations. It highlights that privacy concerns are limited to private data in few-shot examples (as opposed to the training data of the LLM itself), the limited scale of experiments driven by cost constraints, and that private data used to construct prompts is exposed to the provider of the LLM API.\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Demonstrates for the first time that examples used in LLM prompts for text classification are susceptible to membership inference attacks.\n\n- Evaluates the use of DP-SGD together with 3 different parameter efficient fine-tuning techniques for text classification tasks and compares it to full fine-tuning.\n\n- Proposes a novel adaptation of PATE to optimize discrete prompts for solving text classification tasks using LLMs that is implementable using available commercial LLM APIs using a much lower privacy budget than fine-tuning techniques.", "weaknesses": "- Oversells the novelty of **PromptDPSGD**, which is just standard DP-SGD applied to existing parameter efficient fine-tuning techniques.\n\n- The susceptibility of few shot learning to membership inference attacks has been studied before at least for image classification (https://openreview.net/forum?id=39kvovk9ju7). This diminishes the novelty of the observation that the same phenomenon holds for text classification.\n\n- **PromptPATE** is applicable to classification tasks, but it does not appear generalizable to generative tasks where LLMs excel.\n\n- **PromptPATE** performs worse than a non-private few-shot baseline in all scenarios evaluated. Table 2 shows that even a 4-shot **PromptPATE** has worse accuracy than the 1-shot non-private baseline. This casts doubt on the necessity of learning differentially private discrete prompts. It would be enough to declassify a single private prompt to get better utility with perfect privacy for the rest of the prompts.\n\n- The choice to evaluate **PromptDPSGD** and **PromptPATE** on model classes with hugely different capabilities (RoBERTa, and GPT-3 and Claude, respectively), and on mostly disjoint tasks (with the exception of SST-2), makes it hard to compare the two approaches. Despite the difference in model capabilities likely making **PromptPATE** look better than **PromptDPSGD**, the results still show a large gap to a non-private baseline: e.g., for AG News, Table 2 reports 71.7 $\\pm$ 0.8% with $\\varepsilon = 0.248$ vs. 81% using a 4-shot prompt (which I think is a more appropriate baseline than a 1-shot prompt). The utility of **PromptPATE** cannot be significantly improved because it saturates at a lower privacy budget (cf. Figure 3b) compared to parameter efficient fine-tuning.\n\n**Comments**\n\n- In line 120, \"To the best of our knowledge, no prior work attempted to provide DP guarantees for prompt data in LLMs.\" This peer-reviewed paper published shortly before NeurIPS 2023 deadline explores the use of differentially private prompt tuning in federated learning: https://doi.org/10.1109/ICASSP49357.2023.10095356.\n\n- In line 322, The reference should be to Table 2 instead of Table 5.\n\n- In line 539 in the Appendix: \"trend the our private prompts\" should read \"trend that our private prompts\".\n\n- In Algorithm 1 in the Appendix: $D$ is specified first as a set of labeled examples $(x_i,y_i)$ but later only the features $x_i$ are used. In line 4, $L_P$ should read $L_{P_t}$ and $p_t$ should read $P_t$. In line 9, $p_T$ should read $P_T$.\n\n- In Algorithm 2 in the Appendix: $\\mathbf{x}$ should read $x$ and the parameter $E$ is not described.\n\n- In the caption of Figure 5, \"each prompt has only one member\" should read \"each prompt has only four members\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Please find our detailed response in the following:\n\n>**1. The susceptibility of few shot learning to membership inference attacks has been studied before at least for image classification**\n\nThank you for pointing out this paper. We will add it to our related work section. However, we think our observations are novel since the few-shot learning algorithm studied in the paper is the traditional gradient-based fine-tuning, while ours is in-context learning. These two learning paradigms are vastly different. The main distinction is that in-context learning does not modify the model’s parameters whereas fine-tuning does. Given that fine-tuning modifies the model parameters according to the sensitive data, it is expected that the model then exposes membership information on that data. For our settings where the model's parameters are unchanged, the previous conclusion does not simply transfer. \n\n>**2.  It would be enough to declassify a single private prompt to get better utility with perfect privacy for the rest of the prompts.**\n\nWe want to clarify that the 1-shot non-private baseline is with regards to the best 1-shot prompt selected based on the validation accuracy instead of a randomly-selected 1-shot prompt. [58] shows that the performance of in-context learning has huge variance, so a validation set is very important for utility. The privacy implications of this difference are also crucial and publishing a model with a 1-shot prompt cannot be seen as leaking only privacy of the chosen example. During the process of selecting the best 1-shot prompt, the model is prepended with multiple examples from the validation set, and the best performing one is chosen. As a consequence, the final prompt implicitly also contains private information on the non-chosen examples.\n\n>**3. Different models and datasets for PromptDPSGD and PromptPATE make it hard to compare.**\n\nThe main reason why we use different models and datasets for soft and discrete prompts is to choose the most suitable set-up for each paradigm and keep consistent with each of its own previous works. Also, we want to emphasize that our paper does not aim to compare the performance of PromptDPSGD and PromptPATE. Instead, we want to provide alternatives for people to use in different scenarios.  Please see a more detailed response in the first point of the global response. \n\n>**4. Why is the privacy budget lower for 4-shot than 1-shot?**\n\nOur PromptPATE’s privacy analysis follows the standard PATE where the privacy budget depends on the number of teachers, the consensus among teachers and the size of the public set. The number of examples in the prompt does not directly influence the privacy budget. This is similar to standard PATE, where the privacy costs are not influenced by the private training set size \n\n>**5. How is the ROC curve produced?**\n\nWe thank the reviewer for their question and are happy to clarify. We perform a confidence-based membership inference attack. The intuition is that for labeled samples (x,y), if (x,y) was used as a member of the prompt, when queried to predict on x, the prompted model will have a higher confidence on y than when (x,y) was not a member. \n\nIn our case, we assume that the attacker has a pool of potential member-candidates that consists of 1 member and 50 non-members. This skewed distribution is realistic given that the attacker has access to large amounts of data, out of which only a small fraction will be members of the concrete task. The attacker then queries x_1, … x_51 to the model and obtains the confidence scores z_1, … z_51  for the prompted model’s predictions at the right class belonging to x_i.\n\nOut of this vector of confidence values, we generate the ROC-curve (Figure 2b). We calculate the true positive rate (TPR) at different false positive rates (FPR). Given that only one of the 51 data points is a member point, there is a jump in the curve, where at a given threshold, the member data point is correctly classified as a member.\n\nWe repeat this experiment over 100 different 1-shot prompts which yields 100 gray curves. We then average over these curves and plot the average as the blue curve. This then expresses the privacy risk of the prompted model for the private downstream dataset. \n\n\n>**6. PromptPATE does not appear generalizable to generative tasks.**\n\nThanks for your question. Please see the third point of our global response, where we explain how PromptPATE can be extended to generative tasks.\n\n>**7. 4-shot PromptPATE has worse accuracy than the 1-shot non-private baseline.**\n\nFor agnews in Table 2, the improvement of 4-shot compared with 1-shot is not significant (< 3%). Thus, it’s not surprising that 4-shot private is worse than 1-shot non-private. Indeed, previous work [58] shows that having more examples in the prompts does not always increase the accuracy. For example, in Table 1 of [58], on the Trec dataset, 4-shot (69.7) gives worse accuracy than 1-shot (75.7). Therefore, we do not think it’s suitable to compare public and private setups with different number of shots. \n\n\n>**8. The utility of PromptPATE  saturates at a lower privacy budget compared to parameter efficient fine-tuning.**\n\nAs shown in Table 2, PromptPATE is already very close to the teacher ensemble's performance, which is the upper bound of the student's performance. Therefore, if the teacher ensemble's performance can be improved, the utility of PromptPATE will improve as well.\n\n\n>**9. Related work of differentially private prompt tuning in federated learning**\n\nThank you for pointing out this concurrent paper. We will add this work to the related work and change the sentence on line 120 “To the best of our knowledge, no prior work attempted to provide DP guarantees for **discrete** prompts in LLMs.”\n\n>**10 Comments 2-6**\n\nThank you very much for pointing out the typos in the paper. We will update all of them in the new version of our paper.\n"}, {"review_id": "uQH6g1eItK", "reviewer": "Reviewer_RZeY", "summary": "This paper discusses the potential privacy risks associated with prompting data in large language models, which can be exposed through a membership inference attack. To address this issue, the authors propose two methods, PromptDPSGD and PromptPATE, for achieving private prompt learning. PromptDPSGD involves obtaining input gradients from the LLM and using FedAvg to update the soft prompt embedding while keeping the LLM frozen. On the other hand, PromptPATE creates a noisy ensemble of private discrete prompts and transfers knowledge by selecting a student prompt that can be publicly released. The experiments demonstrate that LLMs prompted with these private algorithms closely match the non-private baselines.", "questions": "1. Could you provide more details of the membership inference attack? How to discover the member and non-member data points?\n2. Would the proposed methods be applicable to NLG tasks? Additionally, have you tested the effectiveness of these methods on other NLU tasks?\n3. I am not entirely convinced that prompting data poses significant privacy risks. From my perspective, the user input itself may present a more substantial potential privacy risk. What is your perspective on this? Are these methods also applicable to addressing privacy concerns related to user input?", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors explore the potential privacy risks that may arise from prompting data in large language models, which can be exploited through a membership inference attack.\n- The proposed approach for privately learning to prompt is novel and effective in preserving privacy while maintaining high accuracy.\n- The experimental results demonstrate the effectiveness of the proposed approach in preserving privacy, which is important for real-world applications of large language models.\n", "weaknesses": "- The authors only verify the effectiveness of the proposed approach on NLU tasks. It is necessary to conduct experiments to verify the performance of the proposed methods on NLG tasks.\n- I am not entirely convinced that prompting data poses significant privacy risks. In my view, the user input itself may present a more significant potential privacy risk.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Please find our detailed response in the following:\n\n>**1.  Could you provide more details of the membership inference attack?**\n\nWe thank the reviewer for their question and are happy to provide more details on the membership inference attack. As we describe in Section 3, we are interested in the question if a given demonstration (private data point example) was used in a prompt. \n\nA prompt consists of a template that tells the model what to do, e.g. “Tell us if the following sentence is “positive” or “negative”, and example is “The movie was good”, “positive”. The data-label pair from the private data (“The movie was good”, “positive”) is the demonstration whose membership we are interested in. The question which data points *are* members, is therefore, determined by the chosen prompt. All other data points that are not used in the prompt are non-members.\n\nIn standard MIA literature (e.g. [45]), the attacker who wants to find out which data points are the members  always has some candidate pairs of potential members. For example they would have the data points (“The movie was good”, “positive”), (“I liked the movie”, “positive”), (“What a terrible movie”, “negative”). Then, they would query all these candidates (only the sentences) to the model and observe the model’s confidence scores  on the labels (“positive” or “negative”) for the attack. Member data points have a higher confidence for the correct label. \n\nIn our case, we assume that the attacker has a pool of potential member-candidates that consists of 1 member and 50 non-members. This skewed distribution is realistic given that the attacker has access to large amounts of data, out of which only a small fraction will be members of the concrete task. The attacker then queries x_1, … x_51 to the model and obtains the confidence scores z_1, … z_51  for the prompted model’s predictions at the right class belonging to x_i.\n\nOut of this vector of confidence values, we generate the ROC-curve (Figure 2b). We calculate the true positive rate (TPR) at different false positive rates (FPR). Given that only one of the 51 data points is a member point, there is a jump in the curve, where at a given threshold, the member data point is correctly classified as a member.\n\nWe repeat this experiment over 100 different 1-shot prompts which yields 100 gray curves. We then average over these curves and plot the average as the blue curve. This then expresses the privacy risk of the prompted model for the private downstream dataset. \n\n>**2.  Would the proposed methods be applicable to NLG tasks?**\n\nIn principle, PromptDPSGD can be directly applied to generative tasks. For PromptPATE, one way to extend it to generative tasks is by performing a private teacher vote for next-token predictions. Generating each next token can be thought of as a classification problem over the whole vocabulary. In fact, this idea has been used in SeqPATE (Tian et al. NeurIPS 2022)  for fine-tuned models. We perform some preliminary experiments on one-shot prompts with Claude, which shows that 86% of the private teachers generate the identical first token on the e2e dataset. This high consensus among private teachers implies a promising signal for PromptPATE to succeed. However, we also find that many details and design choices in this extension require a deeper analysis and extensive engineering efforts. For example, it is not immediately clear what vocabulary to vote on when that information is not given, which is the case for Calude. Due to the constraints of time and resources, we admit this to be a limitation to our work and leave it as our future work. \n\n\n>**3. I am not entirely convinced that prompting data poses significant privacy risks.**\n\nWe would like to clarify that the threat-model for leakage of  private prompt data and private user inputs differs substantially. While the user input is leaked to the company who uses the prompted model and the API provider who deploys the LLM, the prompt data is already known to the company and leaks to the API provider, but **additionally**, when the company deploys the prompted model, it will leak to all other users of the prompted LLM. While we assume that the user shared their data with a company that they trust (or at least a company that has to follow the current privacy regulations) and while the API provider offers a service where the company can make contracts with them to not leverage the sent data for training with opt-out accounts (https://openai.com/blog/new-ways-to-manage-your-data-in-chatgpt), no guarantees can be given about other users (that are uncontrolled and potentially malicious). This significantly increases the risk for the private prompt data.\n\nAdditionally, with pre-trained LLMs becoming more powerful, companies can leverage their users' private data more easily by deploying the model prompted with some examples of a task they want to solve. We identified privacy risk for data included in the prompts using the membership inference attack. Thus, the prompt provider can potentially expose private data via the textual prompts which is a severe privacy leakage.\n\nCitation:\n- Tian, Zhiliang, et al. \"SeqPATE: Differentially private text generation via knowledge distillation.\" Advances in Neural Information Processing Systems 35 (2022): 11117-11130.\n"}, {"review_id": "njmKwUSO4j", "reviewer": "Reviewer_96HV", "summary": "This paper studies privacy preservation in the context of prompt learning for large language models (LLMs). It highlights the vulnerability of prompting data to membership inference attacks (MIAs) and proposes differential privacy (DP)-based defense methods for both soft and hard prompt learning. For soft prompts, the DP-SGD algorithm is utilized, while the PATE algorithm is adapted for hard prompts. Experiments on several common datasets and LM architectures demonstrate that the proposed algorithms can achieve good utility with a relatively small privacy budget.", "questions": "I noticed that different language model architectures and datasets are used for PromptDPSGD and PromptPATE. Could you provide further discussion on this difference?", "limitations": "The authors have adequately discussed the limitations and potential negative societal impact of their work in the appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1. This paper presents a timely study on privacy preservation for prompt learning.\n\nS2. The paper addresses both soft and hard prompt settings and proposes suitable differential privacy (DP) algorithms for each. The PromptPATE algorithm includes in-depth adaptations to the data-efficient characteristics of prompt learning.\n\nS3. It presents MIA to motivate the DP-based defense algorithms.\n\nS4. This paper provides systematic experiments on several common datasets and LM architectures to demonstrate the effectiveness of the proposed methods.\n\nS5. Very well-written and easy to follow.", "weaknesses": "W1. PromptDPSGD is somewhat of a direct application of the original DPSGD to the soft prompt learning setting.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Please find our detailed response in the following:\n\n>**1. PromptDPSGD is somewhat of a direct application of the original DPSGD to the soft prompt learning setting.**\n\nOur work is the first one to show the good utility of soft prompt tuning with DP-SGD. Compared with the [25] and [54], our method optimizes orders of magnitude fewer parameters while keeping the original LLM frozen.  We leveraged libraries for DPSGD from full [25] and parameter-efficient [54] fine-tuning (LoRA) and combined them with the P-tuning V2 to be able to apply DP-SGD to the continuous prompts. Then, we carefully tuned the standard and privacy (hyper-)parameters, which resulted in good performance on many downstream tasks. \n\n\n>**2. I noticed that different language model architectures and datasets are used for PromptDPSGD and PromptPATE.**\n\nThe main reason why we use different models and datasets for soft and discrete prompts is to choose the most suitable set-up for each paradigm and keep consistent with each of its own previous works. As soft prompts can be applied to white-box LLMs, in order to provide a fair comparison with the previous DP fine-tuning methods [25,54], we use the same architecture (RoBERTa) and datasets as these two previous works. On the other hand, discrete prompts are mostly used with the decoder-only architecture like GPT3. In order to study the most suitable set-up for discrete prompts, our dataset choice and experiment design follows one of the pioneer works on in-context learning for GPT3 [58]. \n\nAlso, we want to emphasize that our paper does not aim to compare the performance of PromptDPSGD and PromptPATE. Instead, we want to provide alternatives for people to use in different scenarios. For example, one should use PromptDPSGD with the smaller and open-source LLM that offers gradient access and the downstream dataset size is large enough. On the other hand, PromptPATE should be used for larger and commercial LLMs (such as, GPT3 and Claude) has only the input-output access and the sensitive dataset is small. \n"}, {"review_id": "1Y4LXw9zB1", "reviewer": "Reviewer_Hn4c", "summary": "This paper investigated the privacy leakage in prompted large language models (LLMs), and have proposed methods to protect the privacy of potentially sensitive data used for prompt engineering. The authors first demonstrated high membership inference leakage in existing prompted LLMs and then proposed PromptDPSGD and PromptPATE for privately learning soft prompts and discrete prompts with DP guarantees. PromptPATE specifically works with existing commercial API, making it ideal for deployment in the real world.  The authors have validated their approaches on LLMs with extensive experiment setup and have shown a reasonably good privacy-utility balance.   \n", "questions": "* Why restrict PromptPate with much smaller epsilon < 0.2 while using larger epsilon (8) for PromptDPSGD? Do you not observe an increase in utility when increasing the privacy budget for PromptPate?\n* Have you run membership inference on the DP protected models as a sanity check?\n", "limitations": "This paper has raised a privacy concern about real world deployment of LLMs. The authors have acknowledged limitations in the appendix of the paper.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-motivated as LLMs and prompt engineering are increasingly popular, the privacy concerns of sensitive data used in prompted LLMs are less studied. The privacy leakage analysis in this paper also validated the concern. \n* The paper is organized and easy to follow. The literature review is comprehensive and the methods are clearly described.\n* The experiments are conducted on LLMs with real world deployment scale and on black-box commercial API, showing the applicability of this research on existing real world LLMs systems. The setup of the experiment is extensive, including different scenarios such as private and public data from different domains.\n", "weaknesses": "Compared to the parameter-efficient fine-tuning approach (LoRA), prompt learning with DP has inferior performance.  Although the authors argued that the storage is cheaper with prompt learning, one might still prefer parameter-efficient LLMs based on their needs on the utility (also storage is typically considered cheap compared to other resources).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback! Please find our detailed response in the following:\n\n>**1. One might still prefer parameter-efficient LLMs based on their needs on the utility.**\n\nWe want to emphasize that the main advantage of prompt learning with DP is its flexibility and applicability even when the LLM is deployed behind an API, which is nowadays a standard (see for example the GPT family or Claude). DP with LoRA requires users to modify the model architectures, which almost all LLM APIs do not provide access to. In contrast, our PromptDPSGD only requires gradient access and PromptPATE only needs input-output access. These make our methods more practical and applicable without full access to LLMs. \n\n>**2. Why restrict PromptPate with much smaller epsilon < 0.2 while using larger epsilon (8) for PromptDPSGD?**\n\nTable 2 shows that our PromptPATE is very close to the accuracy of ensemble teachers, which is the upper bound of the student's performance. Figure 3.b further shows that the performance plateaus after $\\epsilon=0.2$ and labeling more data points is not helpful. To further improve the utility of PromptPATE, it's crucial to improve the performance of the ensemble teachers, which is a separate question than privacy. \n\n>**3. Have you run membership inference on the DP protected models as a sanity check?**\n\nWe thank the reviewer for this suggestions. During the rebuttal period, we ran our MIA on the public prompts over 3 random trials for each dataset. The average ROC scores over 3 trials are shown below. The AUC-ROC curves (each blue line corresponds with one student prompt for one trial) are in the pdf. The ROC scores are close to 0.5 and the curves are close to the random-guessing line, which show that PromptPATE effectively prevents the leakage from MIA. \n\n\n|     | sst2 | agnews | trec | dbpedia |\n|-----|------|--------|------|---------|\n| ROC | 0.49 ± 0.02 | 0.48 ± 0.03   | 0.51 ± 0.02 | 0.52 ± 0.04    |\n\n"}, {"review_id": "iHVegBkivw", "reviewer": "Reviewer_k8ve", "summary": "The paper studies the problem of differentially private prompting, i.e. the scenario where a prompt-augmented LLM is exposed to users, which should be able to interact with the model, but should not be capable of extracting the private prompt prepended to their queries. The authors first show, that a membership inference attack against example data used in a prompt is feasible and easy to carry out (given access to model logprobs). They then show how to adapt the DPSGD algorithm to enable differentially private soft prompt learning (PromptDPSGD) and devise a new method for differentially private prompt learning in the discrete setting (PromptPATE) based on teacher-student knowledge transfer. Their evaluation shows that both methods are effective in protecting prompt privacy without sacrificing too much model performance under reasonable \\epsilon values. ", "questions": "\n* Can you comment on the effectiveness of your MIA attack against larger OpenAI models like `text-davinci-003`. From my own experience (which is anecdotal), logit noise is added to the API response for these larger models, which may make the attack less effective. \n\n* Can you comment on MIA effectiveness with models that do not expose model logits at all, as is the case with ChatGPT and GPT-4 (see above)? Is prompt privacy still an issue in this case?\n\n* Can you discuss the novelty of PromptDPSGD and PromptPATE compared to existing work like the standard DPSGD algorithm or existing discrete prompt optimisation approaches, to highlight the concrete contribution of the paper?\n\n* Does PromptPATE and PromptDPSGD generalize to more free-form text generation tasks, e.g. when the model is used to generate longer text given a (private) prompt or exposed as a fully interactive chatbot? Can you protect against instruction extraction in these cases?", "limitations": "I think the paper should clarify early on, that it focuses on the restricted classification setting. This would help readers to better understand the specific problem the paper is trying to solve, where the more general prompt extraction problem should be considered as a separate problem.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and easy to follow, even though it covers a lot of ground. It motives the problem well by first demonstrating an effective MIA.\n\n* The paper considers both the soft and discrete prompt learning setting. I think this is a good and pragmatic choice, as the most popular LLMs are API-guarded and typically do not provide access to gradients or hidden states, which are required for soft prompt learning. \n\n* The paper explicitly considers efficiency and scalability of the proposed methods. This is important as state-of-the-art LLMs are very large and training them is expensive or even infeasible when accessible only via an API. Both the discrete and soft prompt learning methods are efficient and scalable, which is a major advantage over more data-hungry methods such as full model fine-tuning.\n", "weaknesses": "* The shown membership-inference-attack relies on model logits (log probabilities), which are not always available to users (or developers). This means the attack will not work with e.g. the latest OpenAI models like ChatGPT or GPT-4 [1]. Further, even with access to logits via API, model vendors often add noise to the logits to prevent distillation attacks.\n\n* While stated later on, the paper initially does not clarify that PromptPATE is specifically designed for the relatively simple few-shot LLM classification setting. It is not clear how well the methods would work in other settings, e.g. when the model is used for free-form text generation. For PromptDPSGD this is less of an issue, as it seems to be more of a general method for prompt learning.\n\n* I rated the contribution only as \"fair\", because after reading the paper it seems like PromptDPSGD is a rather simple adaptation of DPSGD and PromptPATE also is a relatively simple idea. It may be helpful if the authors could highlight the novelty of PromptDPSGD and PromptPATE compared with existing work.\n\n* The paper does not cover the case where attackers want to extract instructional information from a prompt, e.g. when a prompt not only contains example data, but also a description of the task and the desired LLM output. From what I understand, PromptPATE would not work very well in this case, since even with teacher-student transfer, the resulting student prompt would still have to contain the same (likely human-readable) instructional information as the teacher prompt. I recognize this is as a different problem, but it may be worth mentioning in the paper, as common LLM use has shifted heavily from example-driven prompting to instruction-driven prompting, with the advent of instruction-tuned and RLHF models like ChatGPT.\n\n[1] Chat API documentation, OpenAI, URL: https://platform.openai.com/docs/api-reference/chat/create", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Please find our detailed response in the following:\n\n>**1. The novelty of PromptDPSGD and PromptPATE compared with existing work.**\n\n- PromptDP-SGD: Our work is the first one to show the good utility of soft prompt tuning with DP-SGD. Compared with [25] and [54], our method optimizes orders of magnitude fewer parameters while keeping the original LLM frozen.  We leveraged libraries for DPSGD from full [25] and parameter-efficient [54] fine-tuning (LoRA) and combined them with the P-tuning V2 to be able to apply DP-SGD to the continuous prompts. Then, we carefully tuned the standard and privacy (hyper-)parameters, which resulted in good performance on many downstream tasks. \n\n- PromptPATE: This is the first method for DP learning with LLMs that requires only input-output access to the model. Also, our instantiation of each building block of PATE is novel and original. In the following, we list the novelty w.r.t each building block of PATE:\n\n  - Teachers: We are the first to observe how to leverage the effectiveness of in-context learning for the design of teachers. Instead of training teachers from scratch, we notice that the same LLM (with different prompts) can be instantiated as a teacher ensemble. This does not only make obtaining the teachers more efficient but also vastly decreases the required number of private data points.\n\n  - Student: The naive way of training a student for PATE would be to obtain many labels from the ensemble and then train a model in a supervised way. This would consume a large privacy budget due to a large public dataset needed for supervised learning. Therefore, instead of distilling teacher knowledge into a student model, we distill it into a single prompt, which significantly differs from the original paradigm of PATE. It enables us to obtain high-performance prompts with a small number of public labeled data points, making PromptPATE significantly better in terms of privacy-utility trade-offs than the naive adaptation of PATE.\n\n\n>**2. The effectiveness of MIA for the models that do not expose model logits at all,**\n\nThank you for the comment. First, we would like to highlight that the main purpose of our MIA attack is not to present a powerful attack against modern LLMs but to serve as a motivational example that demonstrates private information from the prompts leaks through predictions of the prompted model. Our MIA shows that the predictions of the prompted LLM can be measurably influenced by the prompt data, which motivates the need for privacy-preserving prompt learning algorithms (the main focus of our paper). Previous works on label-only MIA (Choquette-Choo 2021, Li 2021) show that membership signals cannot be removed by publishing only labels. These attacks approximate the data’s distance to the decision boundary by making multiple queries on the perturbed inputs, which are shown to be as effective as the confidence scores. It is feasible to apply these member-only MIA to our prompt set-up. However, since it is not the focus of our paper, we’d like to leave it to future work.\n\n>**3. The effectiveness of MIA attack against larger OpenAI models that might add noise to the logits**\n\nAdding noises to the logits can decrease the success rate of MIA (in the limit, we can add enough noise to overshadow the membership signal) but this comes at the cost of lowering performance for legitimate users. Label-only MIA would be an effective attack against this defense strategy. \n\n\n>**4. Does PromptPATE and PromptDPSGD generalize to more free-form text generation tasks?**\n\nFor PromptPATE, one way to extend it to generative tasks is by performing a private teacher vote for next-token predictions. Generating each next token can be thought of as a classification problem over the whole vocabulary. In fact, this idea has been used in SeqPATE (Tian et al. NeurIPS 2022)  for fine-tuned models. We perform some preliminary experiments on one-shot prompts with Claude, which shows that 86% of the private teachers generate the same first token on the e2e dataset. This high consensus among private teachers implies a promising signal for PromptPATE to succeed. However, we find that many details and design choices in this extension requires a deeper analysis and extensive engineering efforts. For example, it’s not immediately clear what vocabulary to vote on when that information is not given, which is the case for Calude. Due to the constraints of time and resources, we admit this to be a limitation to our work and leave it to the future work.\n\n\n>**5. Can you protect against instruction extraction in these cases?**\n\nThat's a great point. However, we are not aware of any attack that could extract the private instructions from prompts effectively. On the other hand, our MIA against few-shot examples is shown to be successful. Therefore, we think that defending against privacy leakage of few-shot examples is better-motivated. \n\n>**6. I think the paper should clarify early on, that it focuses on the restricted classification setting.**\n\nThank you for your suggestions. We agree with this. In our updated introduction, we will emphasize that \"our paper focuses on classification\" and \"PromptPATE protects private examples in the few-shot prompts\". \n\nCitation:\n- Tian, Zhiliang, et al. \"SeqPATE: Differentially private text generation via knowledge distillation.\" Advances in Neural Information Processing Systems 35 (2022): 11117-11130.\n- Choquette-Choo, Christopher A., et al. \"Label-only membership inference attacks.\" International conference on machine learning. ICML, 2021\n- Li, Zheng, and Yang Zhang. \"Membership leakage in label-only exposures.\" Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. 2021.\n"}]

Author rebuttal:
We want to thank all reviewers for their feedback which has greatly helped us improve the paper. We are glad that the reviewers recognize our work to “present a timely study” (reviewer 96HV) on the private adaptation of LLMs “with real world deployment scale and on black-box commercial APIs” (reviewer Hn4c). Our proposed defense methods are “efficient and scalable” (reviewer k8ve), “novel and effective” (reviewer RZeY) with “a much lower privacy budget than fine-tuning techniques” (reviewer p7La). We hope that our work will contribute to a more trustworthy deployment of LLMs. Below we offer clarifications to some common questions reviewers have. 
 
>**1. Why did you choose to compare PromptPATE and PromptDPSGD on different model architectures and datasets? (Reviewers: p7La, 96HV, Hn4c)**

The main reason why we use different models and datasets for soft and discrete prompts is to choose the most suitable set-up for each paradigm and keep consistent with each of its own previous works. As soft prompts can be applied to white-box LLMs, in order to provide a fair comparison with the previous DP fine-tuning methods [25,54], we use the same architecture (RoBERTa) and datasets as these two previous works. On the other hand, discrete prompts are mostly used with the decoder-only architecture like GPT3. In order to study the most suitable set-up for discrete prompts, our dataset choice and experiment design follows one of the pioneer works on in-context learning for GPT3 [58]. 

Also, we want to emphasize that our paper does not intend to compare the performance of PromptDPSGD and PromptPATE. Instead, we want to provide alternatives for people to use in different scenarios. For example, one should use PromptDPSGD with smaller and open-source LLMs that offers gradient access. On the other hand, PromptPATE should be used for larger and commercial LLMs (such as, GPT3 and Claude) that provide only input-output access and when the sensitive dataset is small. 


>**2. Discuss the novelty of PromptDPSGD and PromptPATE compared to existing work (Reviewers: p7La, 96HV, and k8ve)**

- PromptDP-SGD: Our work is the first one to show the good utility of soft prompt tuning with DP-SGD. Compared with [25] and [54], our method optimizes orders of magnitude fewer parameters while keeping the original LLM frozen.  We leveraged libraries for DPSGD from full [25] and parameter-efficient [54] fine-tuning (LoRA) and combined them with the P-tuning V2 to be able to apply DP-SGD to the continuous prompts. Then, we carefully tuned the standard and privacy (hyper-)parameters, which resulted in good performance on many downstream tasks. 

- PromptPATE: This is the first method for DP learning with LLMs that requires only input-output access to the model. Also, our instantiation of each building block of PATE is novel and original. In the following, we list the novelty w.r.t each building block of PATE:

    - Teachers: We are the first to observe how to leverage the effectiveness of in-context learning for the design of teachers. Instead of training teachers from scratch, we notice that the same LLM (with different prompts) can be instantiated as a teacher ensemble. This does not only make obtaining the teachers more efficient but also vastly decreases the required number of private data points.

   - Student: The naive way of training a student for PATE would be to obtain many labels from the ensemble and then train a model in a supervised way. This would consume a large privacy budget due to a large public dataset needed for supervised learning. Therefore, instead of distilling teacher knowledge into a student model, we distill it into a single prompt, which significantly differs from the original paradigm of PATE. It enables us to obtain high-performance prompts with a small number of public labeled data points, making PromptPATE significantly better in terms of privacy-utility trade-offs than the naive adaptation of PATE.

>**3. How do the presented methods PromptPATE and PromptDPSGD extend to other tasks, for example, to the text generation tasks? (Reviewers: p7La, RZeY, k8ve)**

In principle, PromptDPSGD can be directly applied to generative tasks. For PromptPATE, one way to extend it to generative tasks is by performing a private teacher vote for next-token predictions. Generating each next token can be thought of as a classification problem over the whole vocabulary. In fact, this idea has been used in SeqPATE (Tian et al. NeurIPS 2022)  for fine-tuned models. We perform some preliminary experiments on one-shot prompts with Claude, which shows that 86% of the private teachers generate the identical first token on the e2e dataset. This high consensus among private teachers implies a promising signal for PromptPATE to succeed. However, we also find that many details and design choices in this extension require more extensive analysis and engineering efforts. For example, it is not immediately clear what vocabulary to vote on when that information is not given, which is the case for Claude. Due to the constraints of time and resources, we admit this to be a limitation to our work and leave it as our future work. 

We thank all reviewers again for their encouragements, feedback and comments. Please find the individual response to each reviewer below.  

Citation:
- Tian, Zhiliang, et al. "SeqPATE: Differentially private text generation via knowledge distillation." Advances in Neural Information Processing Systems 35 (2022): 11117-11130.

Meta-review: The paper contributes to the timely topic of privacy concerns in large language models (LLMs), particularly focusing on text classification tasks. It introduces novel variations of Differential Privacy via Stochastic Gradient Descent (DP-SGD) and Private Aggregation of Teacher Ensembles (PATE) to engineer privacy-preserving prompts. Despite some shortcomings, the paper is well-received by reviewers for its clear presentation, rigorous evaluation, and consideration of scalability and efficiency.

The paper presents one of the initial studies investigating the susceptibility of LLM-based text classification to membership inference attacks. The proposed PromptDPSGD and PromptPATE methods demonstrate a meaningful effort to bridge the gap between differential privacy and prompt engineering in the LLM setting. These methods' scalability and efficiency make them particularly relevant, given the computational constraints inherent in large-scale language models.

While the reviewers found the paper's contributions noteworthy, they also raised valid concerns about its limitations. The paper oversells the novelty in certain aspects, especially regarding PromptDPSGD, which is a somewhat incremental idea over existing DP-SGD techniques. I suggest this should be revised and contributions should be made more precise in the final version. Additionally, the paper mainly focuses on classification and leaves out other applications such as generation. This can also be clearly noted.

Overall, the paper provides a valuable contribution to the privacy challenges posed by LLMs. I believe it will make a nice addition to NeurIPS program. I encourage authors to refine their manuscript based on reviewer feedback for the camera-ready version.
Decision: Accept (poster)