SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In theory, labeling one public sequence by the ensemble would be sufficient to create such a prompt. This approach yields negligible privacy costs, but the resulting prompt might not have good utility due to the high variance in the performance of prompts [58]. Therefore, we generate multiple prompts based on different labeled public sequences and perform prompt tuning to select the best student prompt. Care must be taken during selection: utility cannot be evaluated on the private data anymore given that the prompt will be publicly deployed and selecting based on the private data would incur additional privacy costs. We solve this tension by using parts of the newly-labeled public data as validation data to assess utility of the student prompts. By selecting the prompt with the highest validation accuracy, we deploy the student prompt that most resembles the private teachers.
4.3 Advantages of (Private) Prompting over (Private) Fine-Tuning
Our private prompt learning enables us to leverage the general advantages of prompting over fine-tuning while preserving privacy. Private prompting requires significantly less storage than private fine-tuning. While fine-tuning requires storing a separate copy of the LLM model for each downstream task [24], prompts operate only on the input level of LLMs without adapting model parameters, such that only a small task-specific prompt needs to be stored for each downstream task. For example, each copy of the fine-tuned RoBERTa base model requires 125M parameters ( $\sim$ 500MB). This becomes
Dataset	M	Soft-Prompt (Our)	Prefix (Our)	Full-Tuning [25]	LoRA-Tuning [54]
P	<10K	<100K	125M	1.2M
G	ε = 8	ε = ∞	ε = 8	ε = ∞	ε = 8	ε = ∞	ε = 8	ε = ∞
sst2	92.31	95.64	91.97	96.33	85.89	96.40	92.97	96.60
qnli	84.11	89.48	87.17	94.84	84.81	94.70	88.59	94.70
qqp	81.52	86.56	82.58	91.42	86.15	92.20	86.26	92.20
mnli	75.15	82.49	80.57	90.34	83.30	90.20	82.92	90.20
Table 1: Performance of PromptDPSGD. We report the accuracy values (\%) for each dataset. All  $\varepsilon$  values are reported as standard DP guarantees. We run the experiment on RoBERTa [30]. The first row M: the type of the private Method, the second row P: the number of Parameters tuned for the method, and the third row G: DP Guarantee. We also present results for  $\varepsilon = 3$  in Appendix D.
prohibitively expensive, especially as the number of parameters for state-of-the-art LLMs rapidly increases. In contrast, soft-prompts and prefix, as the one generated by PromptDPSGD (using implementation from [29]) with the standard prompt length of 10 tokens require less than  $10\mathrm{K}$  parameters (40KB) for the soft-prompt and  $100\mathrm{K}$  parameters (400KB) for the prefix. A discrete prompt, such as the one generated in PromptPATE, requires less than  $1\mathrm{KB}$  of prepended text. Prompts also enable processing many examples from different tasks in a single batch [26], called mixed-task inference. This allows more efficient use of LLMs since we do not have to wait for a sufficient number of requests for a single task before processing them. This is not possible with any form of fine-tuning, where the fine-tuned model can serve solely a single task.
We evaluate both PromptDPSGD and PromptPATE and show that they match the performance of non-private prompting while providing strong privacy guarantees.
5.1 PromptDPSGD
Experimental Setup. To train soft-prompts and prefix, we follow the experimental setup from prior work on DP fine-tuning. Specifically, we use differentially-private optimization engines for transformers, such as models from the BERT family for the language understanding tasks. The experimental results for classification were performed on the RoBERTa models [30], using the standard NLP datasets, namely sst2, qnli, qqp, and mnli, from the GLUE benchmark [51]. Our implementation for soft-prompt and prefix is based on P-Tuning v2 [29]. To tune the (hyper)parameters for PromptDPSGD, we adjust the length of the soft-prompt or prefix in the private setting (with the default value of 10, which commonly yields good performance). For the privacy parameters, we set the  $\delta = 1 / N$ , where  $N$  is the number of data points in a given dataset. The clipping threshold of per-example gradients is set to 0.1 in most cases. We use a batch size of 1024. The detailed selection of (hyper-)parameters is presented in Appendix E.
Results. We compare our PromptDPSGD against state-of-the-art approaches for private fine-tuning on multiple private downstream datasets. Our results are shown in Table 1. We highlight that both soft prompts and prefix provide competitive privacy utility trade-offs. For example, the difference in accuracy values between the non-private baseline and the private soft prompt ranges from  $3\%$  (for the simplest sst2 dataset) and up to  $7\%$  (for the most difficult mnli dataset). This mirrors results for other private methods, such as the private fine-tuning of LoRA [54]. We also observe that, similarly, for simple tasks, such as sst2 or qnli, the performance of soft prompt or prefix matches the one of fine-tuning. For the more difficult tasks, namely qqp and mnli, the performance of prefix and soft prompts is also relatively close to fine-tuning. The results obtained for these methods are highly influenced by the number of optimized parameters. For example, for the SST2 task and the RoBERTa-Base model, the prefix requires 19970 additional parameters while soft prompt adds solely 2306 parameters. On the other hand, the number of privately tuned parameters is a few orders of magnitude bigger for fine-tuning and equal to the size of the trained model, namely 125M for the method proposed in [25], while the fine-tuning approach from [54] optimizes around 1.2M
Private	Lower Bound	Ens. Acc.	Upper Bound	Our PromptPATE
IID Transfer	OOD Transfer
ε = 0	ε = ∞	ε = ∞	Public	ε	Test acc	Public	ε	Test acc
sst2	76.3	90.0	93.8	sst2	0.178	88.8±2.3	imdb	0.187	87.2±1.9
agnews	62.0	72.8	78.2	agnews	0.248	71.7±0.8	arisetv	0.258	67.9±1.7
trec	40.7	57.6	58.7	trec	0.281	52.8 ±1.5	qqp	0.293	50.9±3.5
dbpedia	44.2	81.6	85.6	dbpedia	0.194	80.3 ±1.3	agnews	0.203	74.6±1.4
sst2 (C)	82.0	94.0	95.2	sst2	0.147	92.3 ±1.1	imdb	0.154	92.7 ±0.8
agnews (4)	62.0	75.8	81.0	agnews	0.145	73.5 ±1.2	arisetv	0.145	69.6 ±1.8
Table 2: Performance of PromptPATE. We compare PromptPATE with three baselines: zero-shot (Lower Bound), the ensemble's accuracy (Ens. Acc), and the non-private baseline (Upper Bound) on four classification benchmarks. We study two settings, (IID Transfer) when the public dataset is from the same and (OOD Transfer) different distribution than the private data. We find that PromptPATE achieves strong privacy protection ( $\varepsilon < 0.3$  at  $\delta = 10^{-6}$ ) and utility close to the non-private and significantly higher than the zero-shot. Unless otherwise specified, the experiments are performed on GPT3-Babbage with one-shot prompts. Additionally, we also run experiments on GPT3-Curie for sst2 (C) and 4-shot prompts for agnews (4).
parameters. Our results reflect a general trend, where prompts are suited for small downstream tasks while fine-tuning with its bigger number of parameters can also cater to more complex tasks with larger training data sets.
5.2 PromptPATE
Experimental Setup. Teachers: Unless otherwise specified, we rely on GPT3-Babbage as the base LLM and select one-shot examples randomly without replacement from the private downstream task as prompt data. Our prompt template follows Zhao et al. [58]. For each setting, we deploy 200 teacher prompts. Private knowledge transfer: We use the implementation of PATE's Confident GNMAX algorithm and the privacy accounting from [12] and report our algorithm's hyperparameters in Appendix E. Student: For each private downstream task, we experiment with two setups (1) selecting public input sequences from the same (IID) and (2) from a different distribution (OOD) as the private data. We introduce three new datasets for the OOD setup: imdb [31], arisetv [11] and qqp [52]. The details of preprocessing these datasets can be found in Appendix E. In both the IID and OOD setup, we limit the size of the public dataset to 500 input sequences from the respective datasets. After the ensemble finishes labelling, we select the best labeled public sequence as prompt data based on the validation accuracy on the labeled public set. We repeat the process three times and report average and standard deviation of the test accuracy for the selected student prompt on the private test set. To improve utility, both teachers' and students' output probabilities from GPT3 are recalibrated using contextual calibration [58].
Results. We compare PromptPATE against three baselines: the lower bound baseline represented by a zero-shot prediction  $(\varepsilon = 0)$ , i.e., when the LLM is only prompted with an instruction, the private ensemble accuracy  $(\varepsilon = \infty)$ , and the upper bound as a non-private one-shot prediction  $(\varepsilon = \infty)$  using the best example from the private data as prompt data. (To save costs, we select from 200 candidates.) Table 2 shows that, over all setups, PromptPATE achieves similar utility to the non-private baseline and significantly improves over zero-shot predictions—even at very strong privacy protection  $(\varepsilon < 0.3, \delta = 10^{-6})$ . Our results also highlight that the distribution of the public data does not need to be very close to the distribution of the private data to yield high-utility student prompts. For example, they can be collected from different domains (dbpedia holds extracts from wikipedia while its public data agnews contains news articles) and for different tasks (trec aims to classify the topic of a given answer while qqp serves to measure the similarity of two questions). Still, with dbpedia being the private downstream data and agnews as public, we achieve an accuracy of  $74.6\%$ , which is significantly higher than the zero-shot baseline with  $44.2\%$ .
We also provide further insights into the privacy-utility trade-offs that can be achieved with Prompt-PATE in Figure 3b. Our results highlight that with more public sequences queried to the ensemble, the privacy consumption increases while, after roughly 100 queries, with even  $\varepsilon < 0.2$ , the student model's test accuracy saturates. This yields very favorable privacy-utility trade-offs which we attribute
(a) Teacher Consensus
(b) Utility-privacy tradeoff
Figure 3: Additional Insights of PromptPATE. We perform ablation studies on GPT3-Babbage and use dbpedia as private and agnews as public data. Left: Teacher consensus as the fraction of teachers who vote for the correct class over 500 public input sequences. PromptPATE achieves overall high consensus. Right: Student accuracy as a function of the public query set's size. Already with as few as 100 queries, we observe a plateau in accuracy which highlights PromptPATE's data efficiency.
mainly to the data efficiency of discrete prompts: Even from within as little as 100 labeled examples, a high-performing student prompt can be derived. Additionally, we observe that the per-query privacy costs of PromptPATE are relatively low, further benefiting the privacy-utility trade-off. The small privacy costs result from the high consensus between the teacher predictions<sup>4</sup>, see Figure 3a—that might result from all teachers relying on the same underlying LLM, just with different prompts.
Scalability. Finally, we also study how PromptPATE scales with larger LLMs and more examples in the prompt. We experiment with a more performant LLM (GPT3-Currie) for sst2. Due to the higher per-query costs, we are not able to repeat this experiment for all datasets. Our results show that the performance of our private prompt increases together with the performance of the public prompt (92.7% accuracy on Curie vs. 87.2% on Babbage) while the privacy budget  $\epsilon$  decreases (from 0.178 to 0.147). To investigate flexibility in terms of numbers of private examples provided as prompt data, we also experiment for agnews with 4-shot teachers. Similar to the non-private study [58] that reports improvements for agnews in the 4-shot setting over 1-shot, we observe that this improvement also translates to the private prompt. Our results indicate that with increasingly more powerful LLMs and larger context windows, private prompting will increase further in terms of privacy-utility trade-offs.
6 Conclusions and Outlook
By instantiating the first simple yet effective membership inference attack against prompted LLMs, we show that they leak private information about their prompt data. We propose private prompt learning as a holistic and broadly applicable new approach to mitigate this risk. We first introduce PromptDPSGD that enables to train soft-prompts with privacy guarantees. In contrast to fine-tuning, soft prompts optimize significantly fewer parameters and do not require any update of LLM parameters or changes to its architecture. As the first solution to private downstream learning with LLMs in black-box access scenarios, we propose PromptPATE. PromptPATE builds on the highly data-efficient discrete prompts and implements privacy through a noisy knowledge transfer. Through our evaluation against two popular LLMs deployed behind commercial black-box APIs (GPT3 and Claude) [6, 3], we highlight that this method yields downstream performance that matches the one of non-private prompting at very strong privacy guarantees. As LLMs rapidly improve and increase in size, prompts are achieving consistently higher performance while fine-tuning becomes more challenging at this scale. This suggests that privacy protections for prompts will become even more important, especially as context sizes expand.
We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: Amazon, Apple, CIFAR through the Canada CIFAR AI Chair, DARPA through the
<sup>4</sup>As motivated in Section 4.2, high consensus reveals less information about the private decision boundary, and hence incurs smaller privacy costs.
GARD project, Intel, Meta, NSERC through a Discovery Grant, the Ontario Early Researcher Award, and the Sloan Foundation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We also thank members of the CleverHans Lab for their feedback.
[1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] R. Anil, B. Ghazi, V. Gupta, R. Kumar, and P. Manurangsi. Large-scale differentially private bert. arXiv preprint arXiv:2108.01624, 2021.
[3] Antropic. Introducing claude. Antropic Website. 2023-03-14, https://www.anthropic.com/index/introducing-claude.
[4] R. Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473. IEEE, 2014.
[5] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
[8] N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022.
[9] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
[10] H.-T. Cheng and R. Thoppilan. Lamda: Towards safe, grounded, and high-quality dialog models for everything. Google Blog Post. 2023-05-09, https://ai.googleblog.com/2022/01/ lamda-towards-safe-grounded-and-high.html.
[11] O. chimaobi Samuel. news-data. Huggingface, 2022.
[12] C. A. Choquette-Choo, N. Dullerud, A. Dziedzic, Y. Zhang, S. Jha, N. Papernot, and X. Wang. Capc learning: Confidential and private collaborative learning. In International Conference on Learning Representations, 2021.
[13] J. Davison, J. Feldman, and A. M. Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 1173-1178, 2019.
[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[15] C. Dwork. Differential privacy. In Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33, pages 1-12. Springer, 2006.
[53] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[54] D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, S. Yekhanin, and H. Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022.
[55] C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramér, and N. Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938, 2021.
[56] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[57] X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[58] Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
The growing importance of in-context learning as a paradigm for leveraging LLMs on private downstream tasks has significant implications for privacy. We present the first approaches for obtaining prompts with privacy guarantees, thereby enabling the use of this learning paradigm on sensitive data. This advancement has the potential to increase trust and acceptance of LLM-based systems for private applications. Our approach PromptPATE is the first viable technique for private downstream adaptation of black-box LLMs, which enables integrations into the state-of-the-art commercial LLM APIs. We acknowledge that—as with any application that relies on DP—care must be taken when choosing the privacy parameters  $\varepsilon$  and  $\delta$  since setting these incorrectly can lead to a false sense of privacy. Therefore, our work orientates at the privacy parameters that have been shown to provide reasonable protection in prior work. Thereby, we also ensure consistency and comparability in evaluations between the different approoches.
Tuning Instructions and Templates. For our discrete prompts, we did not tune the instructions or templates but instead relied on a template from prior work [58]. The effectiveness and performance of our PromptPATE could potentially be further improved by tuning the instructions and templates.
Privacy Risk of Pretrained LLM: We build on pretrained LLMs to learn and deploy our private prompts. Our methods solely target the protection of the private data used for these prompts. However, it is also important to acknowledge the inherent privacy risks for data used to pretrain the LLM. We leave the pretraining of LLMs with privacy guarantees to an orthogonal line of work.
Limited Monetary Budget for our Experiments. Due to cost limitations, we were unable to experiment with the latest and best available model, GPT4. Our experiments with GPT3-Curie in comparison to less powerful GPT3-Babbage however indicate the clear trend the our private prompts improve in performance as the non-private baseline improves due to better models. Furthermore, again due to the cost limitation, we were not able to incorporate a larger number of teachers in our experiments for PromptPATE. Therefore, the best non-private teacher baseline that we report might not be the best achievable if one had more teachers to choose from. We chose from 200 and note that with more (and potentially better teachers), not only the baseline but also the teacher ensemble's performance would get better.
Hyperparameter Tuning. To save computation costs, we did not exhaustively tune all hyperparameters in our experiments. While our approach still achieves high utility and good privacy-utility trade-offs, we acknowledge that with more hyperparameter tuning the performance together with the understanding of optimal configurations for private prompt learning could increase.
Assumption of a Trusted LLM API Provider. In our work, the API provider gets to interact with the private data, for example, through the teachers' prompts in PromptPATE. Therefore, we have to assume trust in the API provider. The privacy guarantees through our private prompt learning protect the privacy of the prompt data against users that interact with the prompted LLM. In practice, companies that are concerned about the privacy of their data with respect to the API provider could make contracts with the API providers on the use of their data or buy access plans that guarantee that data queried to the API is treated privately. We leave implementing cryptographic approaches that could relief the assumption on trusting the API provider entirely, for example, by enabling the LLM to run inference on encrypted private data to future work.
C Additional Insights into our Methods
C.1 PromptDPSGD
We present the full PromptDPSGD algorithm in Algorithm 1.
Algorithm 1: PromptDPSGD. In contrast to the standard DPSGD algorithm that updates model parameters during private training or fine-tuning, our PromptDPSGD privately updates the soft prompt parameters. We highlight these changes with respect to standard DPSGD training or fine-tuning in blue.
Require: Private downstream data  $D = \{(x_{i},y_{i})\mid i\in [N]\}$ , prompt sequence length  $s$ , embedding dimensionality  $e$ , trained LLM  $L$  with frozen parameters, loss function  $\ell (L_p,x)$  for prompted LLM, Params: learning rate  $\eta_t$ , noise scale  $\sigma$ , sampling rate  $q$ , max gradient norm  $c$ , training iterations  $T$ .
1: Initialize  $P_0 \in \mathbb{R}^{s \times e}$  at random
2: for  $t \in [T]$  do
3: Sample mini-batch  $B_{t}$  according to sampling rate  $q$  from  $D$  {Poisson sampling}
4: For each  $i\in |B_t|$ , compute  $\mathbf{g}_t(x_i)\gets \nabla_{P_t}\bar{\ell} (L_P,\bar{x}_i)$  {Compute per sample gradient w.r.t.  $p_t$  }
5:  $\bar{\mathbf{g}}_t(x_i)\gets \mathbf{g}_t(x_i) / \max \left(1,\frac{\|\mathbf{g}_t(x_i)\|_2}{c}\right)$  {Clip gradient}
6:  $\tilde{\mathbf{g}}_t\leftarrow \frac{1}{|B_t|}\left(\sum_i\bar{\mathbf{g}}_t(x_i) + \mathcal{N}\left(0,\sigma^2 c^2\mathbf{I}\right)\right)\{\mathrm{Add~noise}\}$
7:  $P_{t + 1}\gets P_t - \eta_t\tilde{\mathbf{g}}_t$  {Update soft prompt}
8: end for
9: Output  $p_T$  and compute the overall privacy cost  $(\varepsilon, \delta)$ .
C.2 PromptPATE
Extended Background on PATE. We include the standard Confident-GNMax Aggregator Algorithm from [37] below.