SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haonan Duan\*, Adam Dziedzic†, Nicolas Papernot, Franziska Boenisch University of Toronto and Vector Institute
Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock's knowledge into a single public prompt. We show that LLMs prompted with our private algorithms closely match the non-private baselines. For example, using GPT3 as the base model, we achieve a downstream accuracy of  $92.7\%$  on the sst2 dataset with  $(\varepsilon = 0.147, \delta = 10^{-6})$ -differential privacy vs.  $95.2\%$  for the non-private baseline. Through our experiments, we also show that our prompt-based approach is easily deployed with existing commercial APIs.
Large language models (LLMs) exhibit strong capabilities for in-context learning [6, 40]. By preponding the adequate prompt to an LLM's input, the model can perform a myriad of natural language downstream tasks without any modifications to its parameters [41]. While the data used to train an LLM is usually assumed to be public, downstream data used in the prompt is often more sensitive. This can elicit confidentiality issues, for instance, if prompts contain information that represents valuable intellectual property [34]. At the same time, it also raises privacy concerns when the data involves personal information about individuals.
In this paper, our first contribution is to show that these concerns are valid. We are the first to instantiate a highly effective membership inference attack (MIA) [7, 45] against prompts. Our attack is able to determine if a given data point was used within the prompt of the LLM. The only existing solution to mitigate this privacy risk would be to forego prompting and instead fine-tune the LLM with a privacy-preserving training algorithm [25, 54]. Yet, fine-tuning lacks the efficiency and practicality of prompting. Indeed, fine-tuning requires significantly more data [42], computational resources [25], and storage space [26]. Additionally, fine-tuning requires access to the LLM parameters. However, many of the state-of-the-art LLMs are proprietary models deployed behind an API which only allows its users to query the LLMs [3, 6, 10, 17, 35].
*Corresponding and leading author: haonand@cs.toronto.edu
(a) PromptDPSGD
Figure 1: Our methods for private prompt learning. Left: PromptDPSGD obtains the input gradients from the LLM, and performs DPSGD to update the soft prompt embedding while keeping the LLM frozen. Right: PromptPATE creates a noisy ensemble of private discrete prompts, and then transfers knowledge by selecting a student prompt that can be publicly released. PromptPATE only needs black-box access of the LLM and, thus, can be easily deployed with commercial APIs.
(b) PromptPATE
To leverage the benefits of prompting while at the same time protecting the data contained in prompts, we propose the first algorithms for prompt learning with privacy. Our algorithms offer rigorous guarantees expressed using differential privacy [15]. Perhaps closest to existing work on fine-tuning, we propose to leverage the canonical DPSGD algorithm [1] to learn soft promptswith differential privacy guarantees. Our PromptDPSGD algorithm performs a private gradient descent on the soft prompt embeddings prepended to the LLM's private input. Since these embeddings have very few parameters in comparison to LLMs, our PromptDPSGD is efficient and yields competitive privacy utility trade-offs at a fraction of the training complexity of private fine-tuning.
However, learning soft prompts with DPSGD may not always be possible because it requires computing gradients with respect to the prompt input. As mentioned previously, current APIs [3, 6, 10, 17, 35] usually do not provide these gradients. We thus turn to discrete prompts which consist of natural language tokens. Discrete prompts address the aforementioned limitations while being more data-efficient. Our insight is to observe that LLMs with discrete prompts naturally lend themselves to another canonical approach of differentially private learning known as the private aggregation of teacher ensembles (PATE) [37]. We introduce PromptPATE, which creates an ensemble of LLMs with different discrete prompts from the private dataset which we refer to as a flock of stochastic parrots [5]. Since interacting with the flock directly can leak private information about the prompts, as we demonstrate with our MIA, PromptPATE additionally performs a knowledge transfer. Therefore, each model in the flock generates a next token prediction for a short input sequence of some public data. By performing a noisy majority vote over all models' token output, we generate a single output that, due to the noise addition, implements differential privacy guarantees while incorporating knowledge from the flock. The public input together with the noisy aggregated output form a new single example for the discrete student prompt that can be prepended to the LLM in lieu of the individual prompts which contain private information. In addition to providing rigorous privacy guarantees, our PromptPATE is highly efficient, since, instead of having to query every model from the flock at inference time, it suffices to query the LLM prepended with the student prompt once.
We perform extensive experiments against multiple popular LLMs, such as GPT3 [6] and Claude [3], that are deployed behind commercial black-box APIs. Our results highlight that PromptPATE provides high downstream performance that matches the one of non-private prompting even at very strong privacy guarantees. On the sst2 dataset with GPT3, for instance, we reach an accuracy of  $92.7\%$  with privacy costs as little as  $(\varepsilon = 0.147, \delta = 10^{-6})$ -differential privacy, even when the public data used during PromptPATE's knowledge transfer stem from a different distribution than sst2. Our results closely matches the non-private baseline accuracy  $(95.2\%)$ . Thus, we conclude that prompt learning for LLMs is not only more efficient and practical than fine-tuning but can also achieve high utility even with strong and practical privacy protection in place.
- We instantiate the first MIA on prompted LLMs and show that we can effectively infer membership of the prompted data points with high success.
- We propose a lightweight alternative to DP fine-tuning, namely PromptDPSGD, which optimizes orders of magnitude fewer parameters while keeping the original LLM frozen.
- We propose PromptPATE, the first method for DP learning with LLMs that requires only black-box access to the model—making it easily deployable for commercial LLM APIs.
- Our experiments on multiple state-of-the-art commercial APIs [6, 3] highlight that our methods achieve both high utility and strong privacy protections in various setups.
Prompts for LLMs. The success of LLMs, such as BERT, Claude, OPT, or different versions of GPT and their exceptional in-context learning capacities gave rise to prompt-based learning [14, 6, 39, 40, 35, 56]. Prompts serve as demonstrations of the downstream task, which the model can then generalize from. There are two paradigms for LLM prompting, namely discrete and soft prompts.
Discrete prompts [6, 16, 18, 27, 44] are natural-language instructions that contain examples from the downstream task in a well-crafted template. Tuning discrete prompts is often done by prompting the model with different combination of examples, assessing their performance on the downstream task, and choosing the combination that yields the highest performance as the final prompt.
In contrast to discrete prompts, soft prompts [24, 27] prepend trainable continuous embeddings to the inputs of LLMs. These embeddings are initialized either at random or with embedding vectors that correspond to tokens from the dictionary. During tuning, the embeddings are updated through gradient descent to minimize the loss of the prompted model on the private downstream task. To increase performance further, trainable embeddings can be prepended not only to the input but also to every LLM layer, a technique known as prefix [26, 28, 29].
Both soft prompts and prefix train end-to-end without any human involvement through backpropagation over the LLM. On the other hand, discrete prompts have to be designed manually through careful prompt engineering. Yet, prompt engineering only needs inference passes over the LLM which makes discrete prompt more computationally lightweight. Our work provides privacy protection for all of these paradigms: discrete prompts, as well as for soft prompts, and prefix.
Privacy Leakage in LLMs. LLMs have been shown to memorize data both from their original large training corpora [8, 20, 23, 32, 48, 55] and from smaller private datasets used to fine-tune them for downstream tasks [33]. The only prior work around privacy leakage in prompt-based learning utilizes prompts to extract knowledge from trained LLMs [13, 22, 38]. In contrast, we study the privacy of the prompting data itself. To do so, we investigate the canonical privacy attack known as membership inference attacks (MIA) [7, 45]. Its use as a practical means to demonstrate leakage of private information in ML was recently popularized by a line of work on quantifying memorization [9, 43, 47]. While prior work utilizes MIAs to assess whether a given data point was used to train an LLM, we instantiate a MIA to assess whether a given data point was used within the prompt prepended to the inputs of a trained LLM.
Defending Against Privacy Leakage in LLMs. Prior work either focuses on training [2, 19] or fine-tuning [25, 54] LLMs with privacy guarantees. These approaches rely on the mathematical framework of differential privacy (DP) [15] and in particular the DPSGD algorithm for private stochastic gradient descent [1]. Here, DPSGD is applied to guarantee that one outputs approximately the same model parameters whether or not any given data point was used to train or fine-tune the model. To achieve this, DPSGD clips the per-example gradients that are computed during training and adds well-calibrated noise to each model update. These two operations typically increase the computational complexity of training and decrease the utility of the resulting model [1, 4, 49]. To counteract these effects, state-of-the-art methods for full DP-fine tuning in LLMs require extensive hyperparameter tuning and vast computational resources [25]. Alternative approaches refrain from updating the large number of model parameters and instead introduce additional layers into the model architecture and only fine-tune these layers with DPSGD [54]. To the best of our knowledge, no prior work attempted to provide DP guarantees for prompt data in LLMs.
(a) Probability Distribution.
(b) AUC-ROC Curve.
Figure 2: MIA Risk. We study GPT3 prompted with 100 different one-shot examples (dbpedia). left: We present the prediction probabilities at the correct class for members (the one-shot example) and non-members (50 randomly sampled private points). The output probability for members is significantly higher than for non-member data points. right: We present the AUC-ROC curves of our MIA against the 100 prompts (gray lines) and the blue line as an average over all attacks. Given that each prompt has only one member, the resulting TPRs can only be  $0\%$  or  $100\%$  which leads to the step-shape of the gray curves. The result indicates that our attack is significantly more successful than random guessing (the red dashed line).
Setup and Notation. We denote by  $P$  the soft or discrete prompt that is prepended to any input sequence  $x_{i}$  when querying the language model  $L$ . For brevity, we denote  $L([P,x_i])$  by  $L_{P}(x_{i})$ . The output  $y_{i}$  of  $L_{P}(x_{i})$  is an  $M$ -dimensional probability vector, with  $M$  being the size of the model's vocabulary. Each component of  $y_{i}$  corresponds to the probability that the  $L_{P}$  assigns to the respective token for being the next token in the sequence  $x_{i}$ . The semantic meaning of the next token varies depending on the given downstream task. For instance, for classification, the index with the highest probability indicates the token of the class that  $L_{P}$  assigns to  $x_{i}$ .
3 Private Information about Prompt Data Leaks from Prompted LLMs
By instantiating a MIA against prompted LLMs, we want to highlight that the private data used within a prompt (which we refer to as prompt data from hereon) can be subject to a substantial privacy risk. We showcase this risk at the example of LLMs that are prompted with discrete prompts  $P$  containing tuples of demonstrations from classification downstream tasks as prompt data  $p = \{(p_x, p_y)\}$ . For example, in a prompt with one demonstration (one-shot learning), the prompt data  $p$  may be specified as  $p = \{("The movie was great.", "positive")\}$ . Our prompts are provided in a consistent template where one or multiple demonstrations are combined with instructions as  $P = [Instruction, (text sequence p_x, class-label token p_y), \ldots]$ .
For our MIA, we consider an adversary who aims at inferring whether a given private demonstration  $(p_x,p_y)$  was used within the prompt data  $p$ . The adversary holds  $n$  candidate demonstrations of text sequences and corresponding labels  $l_{i}$  and queries the text sequences  $(x_{1},\dots ,x_{n})$  to  $L_{P}$  with black-box access. The prompted model  $L_{P}$  then returns the output probability vectors  $(y_{1},\dots ,y_{n})$ . Following prior work [21, 53], we analyze the model's output probability at token  $y_{i,l_i}$  that corresponds to the correct target class label of every  $x_{i}$ . The intuition to distinguish between members and non-members is that the output probabilities at the correct class  $l_{i}$  will be significantly higher for demonstrations that were used within the prompt, i.e., members with  $(p_x,p_y) = (x_i,l_i)$ . We show that even with this simple MIA, we can reliably determine membership for the prompt data.
Experimental Setup. We prompt GPT3-Babbage [6] with multiple one-shot examples to solve four standard downstream text classification tasks, namely dbpedia [57], sst2 [46], agnews [57] and trec [50]. The template of our prompts follows [58]. To evaluate our MIAs, we consider the single data point used within the prompt as a members and 50 other randomly selected data points from the respective task's training dataset as non-members. This skewed distribution between members and non-members (1 vs 50) corresponds to a realistic scenario where only a small proportion of the candidate data targeted by the adversary are members [21]. To quantify the success of our attack, we report the AUC-ROC curves of 100 random trials.
3In the prefix method,  $L_{P}$  denotes preponding trainable parameters to every layer's input, and not only to the model input  $x_{i}$ .
Results. Before evaluating the success of the MIA, we analyze the probability output from GPT3 for the correct target class between member and non-member data points. Figure 2a shows for the dbpedia dataset that the prediction probabilities for non-members are significantly lower than for members. Figure 2b shows that this leads to a high MIA risk in terms of an average AUC score of 0.84 for the prompt data. Similar results for other datasets and models are presented in Appendix D. These results highlight that private information can leak from prompt data easily and thus motivate the urgent need for defenses which we develop in the rest of this paper.
4 Methods for Privacy Preserving Prompts
As of now, if we want to protect the private downstream data, we have to forego prompting altogether because, to the best of our knowledge, no algorithms for private prompt learning exist. The only alternative to privately adapt the LLM would be to perform DP fine-tuning [25, 54]. However, this approach is only feasible when we have direct access to the LLM to update its parameters with DPSGD [25] or to even change the model architecture to insert additional parameters—fine-tuned with DPSGD [54]. This is prohibitively expensive and mostly impossible with the commercial API, thus we propose the first algorithms that enable differentially private prompt learning.
We consider two main paradigms of prompting: soft prompts and discrete prompts. To learn private soft prompts, we introduce PromptDPSGD. PromptDPSGD is a parameter-efficient alternative to DP fine-tuning that does not need modifying the parameters or architectures of the LLM. However, many popular APIs [3, 6, 10, 17, 35] do not support soft prompts yet as it requires gradients with respect to the input. Therefore, we propose PromptPATE for discrete prompts. PromptPATE requires only black-box access to an LLM without any knowledge of the LLM's architecture or mode of operation. Instead, the algorithm only needs the next-token prediction of the LLM. This, to our knowledge represents the first solution for privately adapting LLMs in restricted API setups.
4.1 PromptDPSGD: DPSGD for Private Soft Prompt Learning
In general, all discrete input tokens to LLMs are internally transformed into continuous input embeddings that the LLM then operates on. Soft prompts are just additional continuous input embeddings that can be prepended to the original input embeddings before passing them through the LLM. To train (or tune) soft prompts, we require training data from a potentially private downstream task. After prepending the continuous soft prompt embeddings to input examples from the training data, we can calculate the gradients for the loss of the prompted LLM with respect to these soft prompt embeddings. The gradients provide information about how the soft prompt should be updated in order to minimize the loss on the training data.
If we can obtain the gradients for soft prompts, we can learn these prompts with privacy guarantees by applying the canonical DPSGD algorithm [1]. The same applies to prefix, therefore, when we talk about soft prompts in the following, we implicitly also include prefix. We call this approach PromptDPSGD. The algorithm yields soft prompts with DP guarantees that can be deployed with the LLM to solve the respective downstream task. The privacy analysis of PromptDPSGD follows the one of the standard DPSGD. Note, however, that while conceptually similar to fine-tuning the LLM's parameters with DPSGD [54, 25], PromptDPSGD differs in a crucial aspect. In DP-  $\mathbf{SGD}$  fine-tuning, we require the gradients with respect to all or a subset of the model parameters and update these parameters to minimize the loss. In contrast, in PromptDPSGD, we use the gradients with respect to the soft prompt embeddings and only alter these. We highlight this difference in our PromptDPSGD-algorithm that we present in Appendix C.
While this difference seems subtle, it has far-reaching consequences. First, there are orders of magnitude fewer parameters that need to be updated which increases training efficiency. Second, and most importantly, it allows us to keep operating on the original LLM. We discuss the resulting advantages, such as storage efficiency, and the ability to process multiple different tasks simultaneously at the end of this section (in 4.3). These advantages make PromptDPSGD conceptually superior to private fine-tuning. At the same time, as we show in our evaluation, despite the small number of trainable parameters, PromptDPSGD, for simpler tasks, matches the performance of private fine-tuning. Yet, current APIs [3, 6, 10, 17, 35] do not support soft prompting, prefix, or private fine-tuning and only provide black-box access through discrete prompts. For these setups, we propose PromptPATE.
4.2 PromptPATE: PATE for Privacy Preserving Discrete Prompts
PATE [36, 37] enables learning classifiers with DP guarantees. It first trains an ensemble of teacher models on disjoint subsets of the private data. Second, through a noisy labeling process, the ensemble privately transfers its knowledge to an unlabeled public dataset. Finally, a separate student model is trained on this labeled public dataset for release. The noisy knowledge transfer in the second step relies on the Confident GNMAX algorithm [37] that we detail in Appendix C. It consists of three main parts: for any input data point from the public unlabeled dataset, each teacher votes for the most likely class. Then, the consensus over the teachers' votes is determined and queries with low consensus are rejected to avoid revealing too much information about the private decision boundary. Finally, the returned class label for any non-rejected data point is determined as a noisy argmax over all teachers' vote counts—where the added noise is sampled from a Gaussian distribution to implement the DP guarantees. For each rejected or labeled data point from the public dataset, privacy costs are accumulated and the ensemble stops labeling once a target privacy budget is reached.
Our PromptPATE follows the general flow of standard PATE: training the teacher models, private knowledge transfer, and training a student model. However, due to the significant differences between in-context learning for LLMs and supervised learning in the original PATE and how these different paradigms leverage private and public data, we had to redesign each of these building blocks. This allows to leverage both the data-efficiency of prompts and the rigorous privacy protection from PATE. In the following, we present the building blocks in our PromptPATE.
Teacher Models (Flock of Stochastic Parrots). Instead of training teacher models on disjoint partitions of the private data, we use the private data to create disjoint prompts for the LLM. More specifically, we use examples, for instance  $\{("The movie was great.", "positive"), \ldots\}$ , from the private training data to create prompts that can then be deployed with the LLM as teachers.
Private Knowledge Transfer. During the private knowledge transfer, the teachers label public data sequences, such as ("I did enjoy it.", _.). Each teacher votes with the most likely class labels for the private downstream task. In Appendix D, we show that PromptPATE can also operate directly on pure next token predictions from Claude [3] without access to per-token probabilities—enabling full black-box private prompts. By performing the private voting process according to standard PATE with the Confident GNMAX algorithm, we turn our per-teacher predictions into a final class label token that will be appended to the sequence, e.g., ("I did enjoy it", "positive"). The privacy accounting and analysis of our PromptPATE exactly follows the one of standard PATE [37].
Student. The most naive way to obtain a student model following standard PATE would be to label many public sequences and train a language classifier using supervised learning on this data. However, due to the relatively high number of data needed for supervised learning, and the fact that each query to the private teachers consumes privacy, this process would incur high privacy costs. We propose a better approach building on the data-efficiency of prompting [42] by using labeled public sequences to create new discrete student prompts. The selected prompt can then be deployed with the LLM as the PromptPATE student model.