Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives
Authors: Vincent Hanke
Tom Blanchard
Franziska Boenisch
Iyiola Emmanuel Olatunji
Michael Backes
Adam Dziedzic
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
privacy
differential privacy
dpsgd
LoRA
private fine-tuning
PromptPATE
PromptDPSGD
Adaptations
soft prompt
prefix tuning
hard prompts
Primary area: safety_in_machine_learning
TL;DR: We demonstrate that privately adapting open LLMs outperforms the recently proposed private adaptations for closed LLMs across all privacy regimes while using smaller model architectures and incurring lower financial costs.

Abstract: While open Large Language Models (LLMs) have made significant progress, they still fall short of matching the performance of their closed, proprietary counterparts, making the latter attractive even for the use on highly *private* data. 
Recently, various new methods have been proposed to adapt closed LLMs to private data without leaking private information to third parties and/or the LLM provider. 
In this work, we analyze the privacy protection and performance of the four most recent methods for private adaptation of closed LLMs. 
By examining their threat models and thoroughly comparing their performance under different privacy levels according to differential privacy (DP), various LLM architectures, and multiple datasets for classification and generation tasks, we find that: (1) all the methods leak query data, i.e., the (potentially sensitive) user data that is queried at inference time, to the LLM provider, (2) three out of four methods also leak large fractions of private training data to the LLM provider while the method that protects private data requires a local open LLM, (3) all the methods exhibit lower performance compared to three private gradient-based adaptation methods for *local open LLMs*, and (4) the private adaptation methods for closed LLMs incur higher monetary training and query costs than running the alternative methods on local open LLMs.
This yields the conclusion that, to achieve truly *privacy-preserving LLM adaptations* that yield high performance and more privacy at lower costs, taking into account current methods and models, one should use open LLMs.

Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives
Vincent Hanke, Tom Blanchard, Franziska Boenisch, Iyiola E. Olatunji, Michael Backes, Adam Dziedzic* CISPA Helmholtz Center for Information Security
Abstract
While open Large Language Models (LLMs)) have made significant progress, they still fall short of matching the performance of their closed, proprietary counterparts, making the latter attractive even for the use on highly private data. Recently, various new methods have been proposed to adapt closed LLMs to private data without leaking private information to third parties and/or the LLM provider. this work, we analyze the privacy protection and performance of the four most recent methods for private adaptation of closed LLMs. By examining their threat models and thoroughly comparing their performance under different privacy levels according to differential privacy (DP), various LLM architectures, and multiple datasets for classification and generation tasks, find that: (1) all the methods leak query data, i.e., the (potentially sensitive) user data that is queried at inference time, the LLM provider, (2) three out of four methods also leak large fractions of private training data to the LLM provider while the method that protects private data requires local open LLM, (3) all the methods exhibit lower performance compared to three private gradient- based adaptation methods for local open LLMs, and (4) the private adaptation methods for closed LLMs incur higher monetary training and query costs than running the alternative methods on local open LLMs. This yields the conclusion that, achieve truly privacy-preservic high performance and more privacy at lower costs, taking into account current methods and models, one should use open LLMs.
LLM adaptations that yield
1 Introduction
Recently, there has been the trend of releasing open Large Language Models (LLMs), such as LLama 22 57,, Vicuna 10. or Mistral [27] as an alternative their proprictary closed counterparts, such as GPT from OpenAI (2). Claude from Anthropic [4) or Gemini from Google (54). Despite the significant progress improving open LLMs, they are still outperformed in multiple tasks by closed LLMs [1I making the latter attractive even for learning tasks from highly private data.
Since it was shown that private data can leak from the adaptations of LLMS 15 16], in the last few months alone, an array of new methods for grivacy-preserving adaptation of closed LLMSs has been proposed by the machine learning community multiple conferences (NeurIPS'23 15 and ICLR'24 (25] [53] 60)). Given the lack of access to the closed L.LMs parameters which renders parameter-tuning based adaptations infeasible- they all rely on the generation of privacy-preserving discrete prompts. We detail their operational setup in Figure (let).
In this work, ask the simple yet impactful question of whether these efforts actually lead into the right direction towards the goal of achieving truly privacy-preseriing LLM adaptations. Therefore, we thoroughly analyze the proposed methods both conceptually and empirically and compare them to
"Corresonndence to adam wea..zieezziiecippe.e
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LLM Provider
Melhod
Open LLM
Prompt
Closed LLM
Query
DP-CCL
60
Not Noeded
PromptATE
15
Not Needed
Answer
Private Queries
DP-Fow ShotGen( 1)
531
X
Not Nooded
DF-Few ShotSl))))n(())
Needed
Open
23 Private Local Adapiation
OPT
X
Needed
LLM
Query
Needed
Data Curator [Company)
Querying Party
Answver
[Customer)
Figure I: Setup for Privacy Protection with Open vs Closed LLMs. The three parties involved are (1) an LLM provider who hosts the proprietary LLM, (2) data curator, such company that curated private data, for example, of their customers' previous transactions, and (3) a querying party, Lee, customer of the company who wants to perform a new private transaction. There are three steps where privacy leaks: A During the creation of the discrete prompt, the data curator's 's private data leaks to the LLM provider.
The private query the querying party leaks to the LLM provider.
Private information from the data curator leaks to the querying party through the returned answers of the prompted LLM [6]. Prior methods for closed LLMs [15 53 60) only provide protection against To prevent leakage through
they require access to (powerful) local open LLM. As an alternative (dashed purple lines), the data curator could privately adapt the open LLM locally and let the querying party interact with this LLM, protecting against
None of them protects against
alternatives that rely on privately adapting open local LLMs. In partieular, we study each approach's 's threat space, assumptions, and methodological limitations and perform extensive experiments using state-of-the-art open and closed LLMs of various sizes, including Vicuna, Llama 3, Open LLaMa, BERT, RoBERTa, the Pythia suite of models, Claude, two versions of GPT3 (Babbage and Davinci), and GPT4 Turbo applied to multiple datasets both for classification and generation tasks. Our analyses cover the axes of privacy protection, performance in terms of privacy-utility trade-offs, and monetary costs for training and queries.
Our results provide the following insights: (1) All current methods for adapting closed LLMs leak private query data (intended for the data owner) at inference time to the LLM provider. (2) Three out of the four methods studied also leak large fractions of the private training data to the LLM provider.
The approaches that do not, require an additional locally deployed open LLM for prompt engineering. (3) All methods for closed L.Mss yield lower final downstream performance than privacy-preserving local adaptations
open L.MS- even when the local methods rely on significantly smaller L.LMs than their closed counterparts, (4) The training and query costs of the private adaptations of closed LLMs (API access costs imposed by the LLM provider) are significantly higher than the costs for private open LLM adaptations (estimated as the costs of training and querying on cloud-based hardware). We provide a condensed summary of our results in Figure I (right Table above), and Table
Overall, results indicate that, from the perspective of effective privacy peeaccyyrreeevaation, current adaptations of open LLMs are strictly preferable over their closed LLM alternatives, since they are more private, more performant, and less expensive. Going beyond the concrete existing methods studied [5 25 53 60), we then analyze the reasons behind the underwhelming results of privacy- preserving closed LLM adaptations and discuss potential directions for improvements.
On the way, to further strengthen private adaptations for open LLMs, we demonstrate how to locally apply privacy-preserving prompt-based methods to train generation tasks with high-performance claimed impossible by prior work (35). In particular, we show for the first time that private prompt tuning for text generation tasks PromptDPSGDGen vate (full) fine-tuning and private low-rank adaptations (LoRA). Additionally, we demonstrate that ensemble-base based fhw-shot prompts PromptPATEGen can privately generate high-quality text at low privacy cost.
achieve comparable performance to pri-
In summary, we make the following contributions:

---

Table Comparison of privacy protection, performance, and cost between private adaptations for closed vs open LLMs. We select the top- poo-performing adaptations. For closed LLMs. we use DP-ICI. 60) and leverage PrivateL.oRA [61] on open LLMSs for both tasks. We consider sentiment classification on SST2 and the dialog summarization on SAMSum. The training data is denoted by Dr and the test queries by Q. Reveals represents which data are exposed to the L.LM provider. The methods were trained with DP guarantees: E = 8 and 6 = I/N, where N is the number of examples in Dr. We report the Performance (higher is better) on test data (where Acc denotes the classification accuracy). The cost (in S) is computed separately for training (Train) and for answering 10k test queries (Query). Note, the (estimated) number of parameters for closed LLMs is 1.76T for GPT4 Turbo and 175B for GPT3 Davinei, while Llama3 has only 8B and BART-Large is significantly smaller with 355M parameters. The adaptation of the open LLMs is more expensive on SST2 than on SAMSum due to the larger training data size for SST2 and larger model. DP-ICL's query cost is high due to the usage of an ensemble of 100 prompts to answer each query. In summary, open local I.M adaptations are more private, more performant, and less expensive.
query
Adaptation
LLM Type
Model
Task Reveals Performance? Train(S) Query(S)
DP-ICL (60
Closed
GPT4 Turbo
SST2 Dr+Q Acc=95,9 2200 %
138.00
Privatel.oRA
61
Open
Llama3- (ains-t)(instruet)
SST2
None Acc=96.0 %
27.60
0.78
DP-ICL 60
Closed
GPT3 Davinci
SAMSum Dr+Q RougeL=31
0
665.91
PrivateLoRA 61
Open
BART-Large
SAMSum None RougeL=39. 3.63
0.80
1. We perform a thorough conceptual and experimental study on existing privacy-presentee closed and open LLM adaptations, analyzing their threat space, assumptions, and achieved results.
2. Our extensive experiments various open and closed LLMs and on multiple classification and generation tasks show that the local (gradient- (gradient-based) adaptations outperform their current closed (discrete prompt-based) counterparts in terms of privacy, performance, and cost efficiency.
3. We propose differentially private prompts for text generation tasks that, for the first time, reach performance comparable to private LoRA or private fine-tuning.
2 Background and Related Work
Differential Privacy. Differential Privacy (DP) (88 is a mathematical framework that provides privacy guarantees by implementing the intuition that an algorithm A : R, executed on two neighboring datasets D. D' that differ in only one data point (we adopted the definition of neighboring based addition/reeooal. [35 44), will yield approximately the same output, i.e. Pr[A(D) E R] < e Pr[A[DD) E R + 8. While specifies by how much the output can differ, ō specifies the probability of failure. There are two prevalent DP algorithms for training machine learning models. The first one is the differential private stochastic gradient descent algorithm (DPSGD) 3 where the impact of each private training data point is limited during training through gradient clipping, and privacy guarantees are integrated through the addition of calibrated amounts of stochastic noise. The second algorithm is the private aggregation of teacher ensembles (PATE) [48] where first, an ensemble of teacher models is trained on disjoint subsets of the private data, and then noisy knowledge distillation is performed a student model using public data. Another general mechanism for implementing DP is the exponential mechanism (EM) 43 The EM selects an output r from a set of possible outputs based on scoring function (DD r) that measures the quality of for dataset D. Let Ag be the sensitivity of the scoring function. The EM chooses r with probability
proportional to exp
(g((Dr)
2Aq
LLM Adaptations. LLMs are pre-trained on large amounts of public data and then adapted to downstream tasks using private data [24]. We divide existing methods for private LLM adaptations into private tuning methods that rely on access to the LLM gradients, and private in-context learning (ICL) which requires only API (black-box) access to the LLM. While private tuning is only applicable to open LLMs, private ICL can, in principle, be applied to both open and closed LLMs. We note that all private LLM adaptations rely in their core on the three DP algorithms introduced above and summarize existing methods, their setup, and their assumptions in Table
Private Tuning for Open LLMs. There exist three main ways for private tuning. 1) Prompt-based adaptations adds small number of parameters (usually <1% % of the total number of parameters)
3

---

Table 2: Comparison of properties between private LLM adaptations. The in in-context learning (ICL) optimizes instructions and shots (demonstrations). Many privacy techniques include the ones designed for multi-label PATE (denoted as MLPATE) [64, exponential mechanism (EM) 4 , joint
exponential mechanism (JEM) [2I] Gaussian Mechanism (GM), Report-Noisy-Max Mechanism
(RNM), Propose-Teste Test-Release (PTR) 19], sample-and-aggregar (SAA) 46
Limited Domain
Algorithm (LDA) 17.
Property
Privacy
Optimization
Inference
Aduptation
Algorithms
Strategy
Privatize
Type
Require
PrompDDPSGDD
15
DPSGD
Gradient- Bradient-bused Soll Promp//reeix
Multi-task
Open LLM
PrivateLoRRA
3
DPSGD
Gradient hased
Added parameters
Single -aakk
Open L.IMM
DP-VineTune
DPSGD
Gradient- based
all LLM parameters
Single-task
Open LLM
DPIICL (0)
ICL
Answers
Limited Queries
None
PromptPATE 15
PATE
ICL
Shots
Multi-tikk
Public Data.
Fe-FeehhuiGen
53
GM.RNM.EM
ICL
Shots
Multi task
Public Labels, Open LLM
DP-OPT
SAA,LDA
ICI.
Instractions+-hhos
Multi-task
Validation Optn,Open I.I.M
only in the model input space, either on the level of token embeddings (soft prompts 39 40 ) or also to every LLM layer (prefix-tuning [30 33)) Duan et al. [15) presented PromptDPSGD, which adapts the DPSGD algorithm to soft prompts. The main advantage of prompt- adaptations is that they enable multi-task batch processing, i.e., many soft prompts for different users and tasks can be processed in the same mini- during LLM training or inference. 2) Parameter efficient fine-tuning- basee adaptations such as LoRA (26 add a relatively small number of parameters (<10% of total number of parameters) within the model, usually in each block of transformer architecture (58). These added parameters are then tuned while the pre-trained original parameters remain frozen. PrivateLoRA [6] extends LoRA with DP guarantees by building on the DPSGD algorithm. 3) Full fine-tuning-basee
adaptations either fine-tune the whole model or only a few last layers. The DP-FineTTune (35,, again based on the DPSGD algorithm, shows that full fine- tuning with DP optimization can provide strong privacy guarantees and good performance. The general trend, when choosing an adequate method, suggests that the more difficult the task, the higher the number of adaptation parameters required [15]. Thus, for simple downstream tasks, PromptDPSGD [5] is sufficient, while DP-LoRA 61 is recommended for medium- credium-difficulty tasks, and the full fine-tuning (35] for complex tasks.
Private ICL for Closed LLMs. Recently, many new methods were proposed for private in-context learning with closed LLMs. All of them leverage discrete (hard) prompts and rely
voting mechanism for privacy protection, similar to PATE (47 48 and CaPC [2]. We divide the existing methods into the following four categories: (1) Private Question Answering: The work on DP- ICL [60) proposed to answer queries based on the private dataset. Following the PATE setup, the private data is divided into non-overlapping partitions and then each partition is prepended with an instruction to form a private teacher prompt. The prompts form an ensemble of private teachers (prompted LLMs). Since DP-ICL does not implement the idea of student model from PATE, all the teachers (usually 100) are required to answer each query, rendering the method expensive when executed on a closed L.M. Moreover, each query incurs additional privacy cost, such that the method can answer only a limited number of queries for a given privacy budget. (2) Private Student Prompt: PromptPATE [5] tackles the problem of the high costs and the limited number of answered queries in DP-ICL by creating a student prompt. PromptPATE uses an ensemble of teacher prompts (usually around 200) label public data. Then it selects the most performant shots for the student prompt from these newly labeled examples. (3) Private Prompt Generation: DP-FewShotGen (53) is similar to PromptPATE but eschews the assumption about the public data for labeling and, in turn, starting from a public label, generates each output token privately to obtain a private shot. (4) Private Prompt Engineering: Finally, DP-OPT 25 privatizes prompt engineering based on the Deep Language Network (DLN) method [51. While DP-ICL. PrompiPATE, and FP-FeeShotGen assume generic instruction and emphasize the protection of the direct leakage from the shots only, DP-OPT (25] proposed to privately generate shots and instructions since either can leak information about the private training set. To overcome the problem that PATE-basee approaches face with large output spaces (here equal to the vocabulary size of around 50k), DP-ICL [60] and OP-OPT 25 incorporate the EM and its improved versions [17 21 to privately release a token with the maximum count based on the voting from teacher prompts.

---

3 Prompt-based Private Adaptations for Text Generation
While PrompIDPSGD and PromptPATE [.5] were designed for classification tasks only, we further extend them text generation tasks. Having pomppt-aseed generation holds the advantage that, in contrast to fine-tuning based approaches, they support mixed-task inference [30] 33 37], i.e., they require one frozen model for multiple tasks rather than separate model copy for each of them. This reduces storage and offers greater flexibility and efficiency.
PromptDPSGDGen. We observe that adequate choice of hyperparameters is sufficient for adjusting PrompIDPSGD [5] to generation tasks. This in line with prior work highlighting that the challenge of prompt tuning is that it requires experimenting with various hyperparameter choices to achieve good performance [37]. In particular, we observe that increasing the number of parameters in the soft prompt from 0.1% of the total LLM parameterss, as done for classification [15), 10% of total model parameters, by enabling prefix projection, yields a significant increase in generation performance. Additionally, we observe the need for an increased learning rate, compared to other tuning methods, to generate more precise outputs. Otherwise, the hyperparameters are dependent on the data the model is trained on.
PromptPATEGen. Adjusting PromptPATE [15 to generation tasks (where more than one output token is generated) is challenging due to 1) the large output space (equivalent to the number of tokens in the vocabulary) and 2) the privacy costs incurred by generating multiple tokens through the teacher ensemble. To overcome this challenge and support generation tasks with unlimited number of queries, we extended PromptPATE by combining the training of the student prompt from [15] with the privacy techniques used in [60] and call the result PromptPATEGen In particular, TromptPATEGen uses the private generation in DP-ICL to obtain longer output sequences for some public data inputs. The outputs sequences then be treated "label" for the public data and can be deployed as a form of student prompt, just like in PromptPATE [1]
4 Comparing Open and Closed LLM Adaptations
We perform a thorough conceptual and empirical study to compare the adaptation of both open LLMs with private tuning (PrOmpDDPSGD 115. PrivateLoRA [61]. and OP-FineTume (35)) and closed LLMs with private ICL (DP-ICL 660, PromptPATE [5), DP-FewShotGen [53], and DP-OPT (55).
Our comparison spans the axes of privacy protection, performance, and cost.
4.1 Comparing Privacy Protection
All the considered methods offer privacy guarantees according DP. Thereby, they ensure that the final prompted LLM's predictions will not leak more than the specified tolerated privacy budget to any party who queries the LLM or gets access to the final private prompt. Yet, the threat model of multiple private ICL methods for closed LLMs does not include providing privacy against the LLM provider. Those methods that do might still occasionally experience leakage. We analyze the result of this lack of consideration for the goal of truly prreecypprss LLM adaptations. In our analysis, we distinguish between the leakage of private training data and the leakage of test data queried at inference time, which might also be sensitive.
(without using an open LLM) disclose (large parts of) their private training set to the LLM provider in the form of shots in their teacher prompts and their engineering. This leakage is inherent in their design. To avoid such leakage, DP-OPT (25) tunes the prompt locally with DP guarantees and then exposes it to the LLM provider. Thereby, the data that the prompt was generated from is protected towards the LLM provider with the DP guarantees that also protect against leakage to querying party. While the experimental evaluation in 255 suggests that at higher E, the locally generated DP prompts might still contain generated data close to the private training data, this is a step towards the right direction. However, to generate the private prompt, OP-OPT 255 requires a powerful open LLM deployed locally. Looking at Figure it becomes obvious that any private tuning method executed on that open L.M would, conceptually, improve privacy protection since the L.LM provider would neither be involved in the adaptation nor the use of the adapted LLM, yielding absolute privacy against them.
Private Training Data. PromptPATE (15), DP-ICL 60), and FeeeeStn
53
5

---

Private Query Data. DP does not aim at protecting query data. Hence, none of the studied private ICL methods attempt to protect that data against the LLM provider. While the protection of query data is often considered as an orthogonal research direction, we note that all the private tuning-based adaptations of the open local L.MMs do naturally prevent leakage of the query data to the L.MM provider. This is because the querying party directly interacts with the data owner (see Figure [ making the use of open models inherently more suited for truly savacc-preservvng application than relying on closed models.
4.2 Comparing Performance
We look at privacy-utility trade-off's to compare the performance of private tuning on open LLMs vs. private ICL on their closed counterparts. We depart from analyzing the trade-off' and required assumptions conceptually and then present our thorough experimental evaluation.
Private Tuning Outperforms Private ICL Conceptually. Previous work B7 has shown for the non-private settings that gradient based tuning methods (used for open LLMs) offer better accuracy and significanly lower computational costs than ICL (used for closed LLMss since the adaptations can leverage the internal behavior of the L.M. This benefit holds also in the privacy regime. Moreover, the tuning based methods do not make additional assumptions, such the availability of public data (required by PATE-based methods, such as PromptPATE [15 ) making them inherently more practical.
Private Tuning outperforms Private ICL Experimentally. To assess the performance of private tuning vs. private ICL, we perform extensive experimental evaluation. We use various LLM architectures and multiple datasets for classification and text generation tasks.
4.2.1 Experimental Setup
Text Classification. We follow the setup from 25 and use four datasets for the evaluation: SST2 from the GLUE benchmark (59), Trec (34). Mpqa 42] and Disaster [5]. SST2 and Mpqa are two-class sentiment analysis datasets. SST2 includes 67.3k training samples and 872 test samples, while Mpqa contains 8.6k training samples and 2k test samples. Trec is six-class question-ty type classification dataset with 5.4k training samples and 500 test samples. Finally, the Disaster dataset involves determining whether a sentence is relevant to disaster scenario not and includes 4.4k training and 1000 test samples.
Text Generation. We use three different datasets: SAMSum, a dialog summarization 22 (14732 train, 818 val, and 819 test samples), PFL.-Doc question answering (56) (85k train and 10k test samples), and MIT Movies trivia10k13, movie extraction on directors (MIT-D with 1561 train and 415 test samples) and genre (MIT-G with 2953 train and 780 test samples) (38]
Closed Models. We follow the setup and choice of models originally proposed in the respective previous papers evaluate the four private ICL methods for closed LLMs 1 25 53 60 The GPT3- Gab3-Babbage and GPT3-Davinci models cited in [53] 60 were discontinued in early 2024 and
replaced by their second versions (babbage-00x and davinci-002). Therefore, we use the newer versions here. The (estimated) number of parameters for the closed models is: 1.3B for GPT3 Babbage, 175B for GPT3 Davinci, 1. 1.76T for GPT4 Turbo, and 200B for Claude 2.1.
Open Models. We consider various open L.MMs with differing pre-training sets and numbers of parameters to simulate the choices a data owner can make for their local LLM. We select the following models: Pythia (6), OpenLLaMA 200. Vicuna [10), Mixtral 28
Bart 31
and RoBERTA [41].
whose sizes vary from 160M to 45B parameters.
4.2.2 Performance of Private Adaptations for Classification
We show that the private adaptations local open LLMs outperform the private methods for closed LLMs for classification tasks. In 3 we analyze the performance differences. We follow the evaluation in 25 (Table 2) and average the accuracy across the tasks (denoted as Average). Our analysis follows the standard practice and sets the privacy budget as é = 8 and 5 = D) where [D] is the training size [15 25 Among the methods for closed LLMs. OP-OPT was tested on

---

Table 3: Private local adaptations on open LLMs outperform their closed alternatives for classification tasks. The default privacy budget is set to E = 8, except for PromptPATE [I3], where the performance plateaus after e = 0.3. The best result for a given task bolded, and the 2nd best underlined. T(S) is training cost while Q($) is query cost for 10k queries (SST2), All($) is total cost.
Method
LLM Type
Model
SST2
Iree
Mpga
Disaster
Average
T($)
Q15)
ABIS)
D-abor ie
Closel GPT3 Davinci
92.4+0.3
51.8+0.2
84.5+0.1
76.4+0.2
76.3
6.00
6.00
DP.OPT (orginal)
Closed
GPT3 Disinci
92.240.5
68. 68.746.6
86.840.7
78.940.3
81.4
2.10
6:000
8.10
ICL
oo]
25
Clenn
GPT3 Dundeci
94.7+0.4
79.1
88.840.11
69.045.9
$2.9
0
6.00
6.00
PromiPATE
Closel
GPT3 Rabiuge
93.8
68.7
83.0
64.3
8666
1.72
10.38
PrompiPATE
0.3)
Closed
GPT3 Bahlage
88.842.3
52.8+1.5
79.0
58.040.5
69.6
9.72
1.72
11.44
PrompPPT:: 15 < 0.3)
Closed
Clande 2.1
96.741.4
79.341.2
92.1-0.6
71.040.8
84.5
48.24
5,36
53.6
DP-FowShottaat))
1
Closed
GPT3 Rabbuge
72.847.7
51.345.8
78.448.5
59.242.5
64.2
0.86
1.10
1.96
DP-ICL
Closed
GPT3 Babtuge
02.840.9
26.345.a
80.640.9
50.64
62.6
17.2
17.2
DP-ICL
Closed
GPI4 Turbo
96.940.1
16.241.7
90.440.1
70.3+0.4
68.2
138.00
138.00
PronpDPSGDD
15
Open
RaDERTA Large
92.340.5
54.5
50.0
17.810.6
68.6
7.59
0.40
7.99
DP-HineThne
15
Open
RoBERIA
98.010.3
93.710.
88.210.4
82.210.3
89.4
6.15
PrivateLoRA
Open
RoBERTA Large
98.640.3
93.9+0.6
87.740.9
81.840.2
89.3
3:45
PrivateLoRA
Open
Vicuna
94.8
97.3.10.1
87.8
81.342..5
13.80
14:58
DP-OPT (local) [4
Open
Vicuna
80.512.6
65.314.3
80.713.3
65.610.3
75.3
2.10
0.78
2.88
Privace
Open
Pyrhia 6.98
02.240.5
96.32...
87.2
82.1
89.4
13.80
0.78
14.58
PrivateLoRA
Open
Pythis 160M
80.4+0.7
82.5+3.2
77.0+0.3
73.6+0.2
1.60
PrivatLLRAA
S1
Open
Liama3 (IClnstruct)
96.0
96.840.2
87.3+0.2
80.8
90.2
27.60
0.78
28.38
the strongest Davinci model (with 175B parameters) from the GPT3 family. Across all the tasks. DP-OPT is outperformed by both DD-FineeTunn Tune and PrivateLoRA by large margin (even >26% absolute on Trec), even though DP-FineTune and PrivateLoRA were trained on RoBERTa Large with only 355M parameters (500X fewer than for GPT3 Davinci). Furthermore, we show that PrivateLoRA outperforms DP-OPT even when using (rthia-6.9,, which guarantees that the open LLM for PrivateLoRA was not pre-trained on any of the downstream datasets. For fair comparison, we also train Pri on Vicuna 7B, which was used in DP-OPT as the local model to find the transferable prompts and show that Private LoRA A is also significantly better than DP-OPT applied either directly to Vicuna 7B or when run on GPT3 Davinei. This suggests that the data owners, rather than using their local LLM to tune prompts for DP-OPT, should privately tune it with PrivateLoRA (in this case on ROBERTA Large) since it yields stronger performance and privacy at a lower cost.
For PromptPATE, the performance plateaus after around & = 0.3, since it creates public prompt using only few shots, and the selection of the demonstrations from a large pool of publicly labeled examples has a negligible gain on the final performance. In the limit, we also show that PromptPATE even with an infinite privacy budget (s = 00) for GPT3 Babbage (with 1.3B parameters) performs worse than PrivateLoRA or DP-Fine Tune on RoBERTA Large (3.6X fewer parameters). In the same setup of models, Privatel.of and Tu-FineTune on RoBERTA Large also outperform DP-ICL tested on GPT3 Babbage on all tasks. Additionally, PrivateLoRA adapted Pythia- 160M (with even fewer parameters) performs much better than DP-FewShotGen on BaP3-eaabage (8X more parameters).
We also run DP-ICL with GPT4 Turbo. with SST-2 and Mpqa. However, it has the lowest accuracy on Tree (with 6 classes), caused by a small number of output probability tokens released for a query (only 20 vs 100 for GPT3, which might not contain the correct class label token) while being the most expensive option. Similar trends are observed for PromptPATE on Claude, however, it has more consistent performance and emerges as the most performant closed model on the tested tasks (while being the 2nd most expensive one). In contrast, Private LoRA with Vicuna 7B performs the best on Trec and on average. It is the best of all tested adaptations while incurring around 3.7 and 9.5 times lower costs than Claude and GPT4 Turbo, respectively. In general, the open models have the highest average performance at much lower cost.
The resulting accuracies are high for sentiment classification
We further analyze the privacy-utility trade-off for classification tasks across different privacy budgets (e E [0, 8) in Figure 2 We show that even under tight privacy constraints (e < 1.0), the privacy- preserving adaptation for open LLMs performs significantly better than the one for closed LLMs. Specifically, we analyze the differences between PrivateLoRA for open LLMs vs PromptPATE for closed LLMs. The performance for PromptPATE plateaus after around & = 0.3 and only for one out of four datasets, namely for MPQA, we observe that the crossover point between PromptPATE and Privatel (PromptPATE performs better than Privatel.oR until = 0.6). For the smallest
e = 0.1 values that analyzed, the performance of Poi is better by 0.6% on SST2, by
7

---

320
Tut A2LAAC (h)
40
Prosstingg
Ce-Redhaatiin
Thps Accurey
onf XC.
Lefta
Zein
Pramettic Dresedhaldee
Trp
DIECI
Lokk
Privacy Cost (el
troaay Costro
Proscy Cout
Privay Cost
(a) SST2
(b) Trec
(c) MPQA
(d) Disaster
Figure 2: Privacy-utility trade-off for classifications tasks. We use Privatel to adapt Vicuna-7b
to the downstream tasks, PromptPATE, DP-ICL, and DP-FowShotGer with GPT3 Babbage. We analyze the privacy costs in the range [0, 8] (sce corresponding Figure 3 for text generation tasks).
Table 4: Evaluation on Dialog Summarization with SAMSum for & = 8. T(S) is training cost
while Q($) is query cost for 10k queries, All($) is total cost.
Type
Model
Ronge-1
Ronge2
Rouge-L
All(S)
DP-ICL
Closed
GPT3 Davinci
41.2=0.6
16.3
31.840.3
665.91
665.91
DP-ICL
Closed
GPT35 Turbo
42.610.2
18.910.3
33.8 33.810.5
449.16
449.18
Closed
GPT4 Turbo
41.8+0.2
17.340.3
38.4+0.2
3419,42
3419.42
PrompIPATEGem
Open
Vicuma 70
41.320.3
18.020.4
32.840.2
3.29
2.74
6.03
PromptPATEGen
Open
LaM 13B
43.440.3
19.710.5
34.210.4
18.63
0.80
19:43
PromptOPSGDGen
Open
RART-Farge
46.120.4
21.340.1
37.4.0..0
2.13
PrivateLoRA
Open
BART-Large
48.820.0
23.510.5
39.1.0.2
2.90
0.69
3.59
PrivaieLoRA
Open
Pythaa 410M
40.4.0.11
16.6.0.5
38.0+0.44
3.45
1.34
4.79
PrompIDPSGDGen
Open
Pythia
41.240.2
17.840.1
33. 740..
4.85
0:95
5.75
DP-FincTune 13
Open
Pythis 1B
42.520.7
18.410.3
33. 33.910-3
9.84
1.08
10.92
PrivateLeRA
Open
Pythis
42.344.8
18.440.7
34.7-t0.5
4.24
1,00
5,24
PivateLoRA
Opent
Pythia
45.620.3
21.410.3
37.420.5
16.75
PrivateLeRA
01
Open
Vicuna 7M
48.643.5
24.842.6
40.2+33.4
11.28
6.19
17.47
PrivateLoRA
G
Open
OpenL 138
48.5
24.210.8
40.140.9
19.46
8.05
27.51
PovateLoRA
Open
Mistral 8C7B
62.8+0.4
29.640.2
44.7
67.55
Table 5: Evaluation on Question Answering with PFL-DocVQL for 8.
LLM Type
Model
Rouge-1
BLEU
Levenshtein
T($)
O15)
AIMS)
DF 60
Open
OpenLLaMA 13B
60.7-6.6
23. 23.9
1
0
641.32
611.32
PrompiPAT Peenn
EGen
Open
Vicata
31.7
26.6+0.7
35.7+0.4
2:28
0.37
2.85
PrompiDPSGDGen
Open
Pythia IB
57.3
40.1
66.8
8...7
37.26
0.96
38.22
FinTine 155
Open
Pythia IB
70.2=0.2
58.
58.7+0.3
78. 78.340.3
137.06
1.30
138.38
PrivateL.oRA [1
Open
Pythia 1B
64.2_o.7
4
73.411.5
44.16
1.26
45.44
PrivaseLoRA
Open
Pythia 6.9
64.4-0.1
47.920.2
73.310.2
293.25
299.05
PrivateLoRA
[1
Open
Openl.aMAA 13B
63.1-1.1
22.2+1.3
70.7+2.1
358.80
9.00
367.82
Table 6: Evaluation on information extraction with MIT-D and MIT-G for E = 8.
Method
LLM Type
Model
MIT-D
MIT-G
T(5)
Q:5) AIKS)
DP Sen
53
Closed
GPT3 Davinci
80.6
64.1
0.42
2.36
2.78
PromplPATEGen
Open
Vicuna
74.1+0.0
41.7+166
0.52
0.75
1.25
PromptPATEGon
Open
Openl.I 138
70.940.5
33.44-1.
3.11
0.80
1.91
FromeLoRA
Ogen
Pythit 410M
74.316.3
64.312.5
0.56
PromptDPSGDGen
Open
Pythia
89.8
69.1
0.17
0.25
0.42
FineTume
15
Open
Pyshia IB
92.2
71.611.1
0.91
0.50
1.44
PrivaeeLoRA
11
Open
Pythia IB
90.240.11
68 68.84.0.5
0.08
0,31
0.39
ProaieLoRA
Open
Vicuna
95.0
74.44:12
0.2
6,44
PrivaeeLoRA
Ogen
OpenLLaMA 138
94.0.0.s
78.410.
7.25
PrivateLoRA
61
Open
Mixtral 8x7B
93.00
69.7
1.52
9.47 10.99
4.4% on Trec, and by 3.5% on Disaster. Overall, the private adaptations for open LLMs outperform the ones for closed LLMs in most privacy regimes.
4.2.3 Performance of Private Adaptations for Text Generation
The evaluation of the three text generation tasks demonstrates superior performance of private adaptations on open vs closed LLMs. We consider the privacy-preserving ICL methods of DP-ICL and Fe-SeShh on closed LLMs, since only these methods were executed for generative tasks. For the SAMSum datasets in Table 4 the first three adaptations (including our PromptPATEGen) are based on few-shot in-context learning (using discrete prompts), while the remaining results are for the private gradient- based adaptations. For the discrete prompts, our PromptPATEGen runs on local open Vicuna 7B and outperforms other discrete prompt-based methods from closed LLMs. Our PromptDPSGDGen performs on par with the other private tuning method (PrivateL.oRA) run on Pythia IB. Note that only PrompIDPSGDGen and ICI. adaptations (PromptPATEGen and DP-ICL) support multi-task inference.
8

---

We additionally leverage BART-Large (with 355M parameters) [ that was fine-tuned on the XSum summarization task [45] (which does not include SAMSum). This specialized open model out- performs other LLMs apart from Vicuna with 7B parameters, OpAAL_LMMA with 13B parameters, and Mixtral with 45B parameters. Crucially, Privatel.oRA on BART-Large outperforms DP-ICL run on GPT3 Davinci, despite using the model with around 500X fewer parameters. This further indicates that we can leverage a large selection of open models to solve a specific task at lower cost and with better privacy protection without resorting to general-purpose closed LLMs. We also use PrivateLoRA on larger models from different families (Vicuna 7B, OpenLLama 13B, and Mixtral 8x7B) and observe that its performance and cost steadily increase with more parameters.
The evaluation on PFL-DocVQA in Table B shows that PrivateLoRA on open LLMs outperforms DP-ICL (which was run also only on OpenLLaMA 13B in the original paper 60] due to the cost constraints). We also evaluate both MIT-D and MIT-G in Table (6 on the accuracy of predicted vs target labels following the metrics in De-FeeShot.en. 'he adaptations of open LLMs with privacy-preserving gradient-based methods outperform De-FewShotGen on the significantly larger GPT3 Davinci, for example, on MIT-D by 13.4% and on MIT-G by 22.3% absolute, respectively by Pr on LaA 13B.
We also present the privacy-utility trade-off for the SAMSum, MIT-G, and MIT-D datasets with varying values of across the PrivateLoRA, PromptPATEGen, and DP-FewShotGen methods in Figure 3] We use the Pythia IB model for MIT-D and MIT-G and the BART-Large model for SAMSum. The graphs clearly demonstrate similar trend that shown previously in Figure 2 PrivateL.RRA A for open LLMs significantly surpasses the performance of both DP-ICI and DP- FewShotGien, which rely on GPT-3 Davinci.
60
Sal Accuracy (%6
80
80
Tbo1 Accuracy
600
40
35
40
DP-ICL
Dr-fewhhhtGen
Di-fewShatGGn
30
PromptPateGen
20
PrompeAATEGen
PromprAATEGen
LoRA
LoRA
20
25
Zoro Shat
Zero
Privacy Cost
Privacy Cost
Privacy Cost
(a) SAMSum
(b) MIT-G
(c) MIT-D
Figure 3: Privacy-utility trade-off for generation tasks. We amalyze the privacy costs E in the range
[0, 8) for the three generation tasks. PRivateLoRRA for open LLMs substantially outperforms DP-ICL and Se--eeShh,,Gen which both utilize GPT3 Davinci. PrivateLoRA for MIT-D and MIT-G is trained on the Pythia 1B model, and for SAMSum on the BART-Large Model. PromptPATEGGS EGen uses Vicuna 7B.
4.3 Comparing Costs
We compare the costs of obtaining a private predictor for given downstream task using open vs closed LLMs. We use the wall clock time to capture the running time of methods for local open LLMs, which we then translate to the monetary cost that would be incurred if we ran the method on cloud-based hardware. For the adaptations of closed LLMs, we count the number of tokens used in the queries and obtained outputs from the APIs. The pricing from cloud providers and OpenAl forms the basis for the cost estimations, and we show the selected values in Table 22 in the Appendix. Further details on how the costs were calculated for each private ICL methods are presented in methods
Appendix D Based on the estimated costs in Tables 3545 and 6 the for open LLMs require much lower costs (and perform better) than for closed LLMs in the considered scenarios. The costs for classification tasks are relatively low, especially for closed LLMss since the tasks are simple and the number of tokens (particularly for outputs) is small. However, the costs increase substantially for generation tasks, especially for the closed LLMs, where DP-ICL is around 150X more expensive than PrivateLoRA for dialog summarization. While larger models often incur higher costs, they do not necessarily imply higher performance. For example, smaller models like RoBERTA Large for classification or BART-Large for dialog summarization can obtain one of the highest performances at the lowest prices.

---

5 Discussion and Future Work
In summary, our results highlight that from the perspective of providing truly preecyypreservation adaptations, open LLMs are strictly preferable over closed LLMs, since their adaptations are more private, more performant, and more cost-effective. Going beyond the concrete existing methods studied in this work [ 25 53 60], in the following, we analyze the general reasons behind the underwhelming results of Srrvacy-rreeervigg closed L.M adaptations.
Privacy Leakage. The enhanced privacy protection from adapting open L.MMs is a major benefit: users' private training data and queries to adapted open LLMs are never revealed to third parties. On the contrary, the leakage of private query data the LLM provider is to date an inherent problem with closed LLMs, since no methods to provide formal guarantees for the query data are currently known. Potential solutions might involve private inference for LLMs, where model performs inference on encrypted queries, however, it is still in its nascency 9 23 32
for the scale of closed LLMs [7].
Performance. We argue that the lower performance of closed LLM adaptations stems from the fact that they have to rely on discrete prompts and that engineering such prompts for the closed LLMs is highly challenging. This is because 1) prompts, in general, have been shown to exhibit an unstable performance and to require a large number of trials and errors or discrete optimization while still underperforming gradient-based approaches [37] Additionally, 2) when the prompts (for privacy reasons) are not tuned on the closed LLM but on an open LLM surrogate model, additional performance decrease is incurred through the prompt transfer, since it has been shown that transferred prompts cannot reach the performance of prompts directly tuned on a given LLM (52). While the latter problem might be mitigated through the design of more performant prompt transfer techniques. the former one seems to be a more fundamental limitation [37].
Costs. The high costs incurred by some closed LLM adaptations result from the fact that they rely on ensemble- based approaches to yield DP guarantees and the fact that they incur continuous query costs at inference time. The former one could be solvable by designing more efficient DP schemes for discrete prompts, however, the latter is inherent to the nature of closed LLMs.
We hope that implementing the above-mentioned solutions will shrink the gap between private adap- tations of open and closed LLMs. However, it remains unclear whether it is worth the community's effort, given the effectiveness of private adaptations for open LLMs.
Broader Impacts
Our comparative study of open and closed LLMs has significant implications for private adaptations: our research advocates for the use of open LLMs for private adaptations.
We stress that our goal is not to discredit closed LLMs, but to highlight the potential privacy and performance benefits as well as cost-effectiveness associated with the use of open LLMs. Through thorough evaluations, we demonstrated in our paper that adapting open LLMs with private parameter efficient fine-tuning methods results in higher performance and mitigates open privacy risks of in-context learning with closed models. This not only leads to better performance but also reduces costs, making privacy adaptation on open L.MS a more viable option for many applications.
Moreover, our work can serve as a baseline for future private learning methods for LLMs. We believe that an open dialogue about the strengths and weaknesses of both open and closed LLMs is crucial for the advancement of privacy-preserving LLMs. We hope that our research will serve as a catalyst for further investigations into private adaptations of LLMs, ultimately leading to the development of models that effectively balance the need for both openness and privacy, all while ensuring that user privacy remains uncompromised.
Acknowledgments
The project on which this paper is based was funded by the German Federal Ministry of Education and Research (BMBF) under funding number 16KIS21 14K. This work was also supported by the German Research Foundation (DFG) within the framework of the Weave Programme under the project titled "Protecting Creativity: On the Way to Safe Generative Models" with number 545047250. Responsibility for the content of this publication lies with the authors.
10

---

References
[1] Bart large xsum. URL http////ee///nggaae.eeecco///ee/
large-araee xsum
[2] Openai, https://opemai.com. URL htttps:///ppaa..... com/
[3] Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[4] Antropic. Introducing claude. Antropic Website. URL https::/ww...ndtttt www.
com/ Linddee//trrouuning--g claude! 2023-03-14, https: http://wwwaattrpicc.oooooom///aa introducing-clann
[5] Trapit Bansal, Rishikesh Jha, and Andrew McCallum. Learning to few-shot learn across diverse natural language classification tasks. arXiv preprint arXiv: 1et.e111111111 1113666.3, 2019.
[6] Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2430. PMLR, 2023.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[8] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of production language model. In International Conference on Machine Learning. 2024.
[9] Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. THE-X: Privacy-prese preeeerirrs transformer inference with homomorphic encryption. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 3510-3520, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v www.....iiningsaee... URL
.orooog.oooo.oooooooooooo org 20wgg//222.2222 findings-ncclll acl. 277
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, lon Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL
https://lmsys.. org/ httg//llgg///// 20-2200-2202222200 30----0----0-vvcc
[11] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
[12] Christopher A Choquette- Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, and Xiao Wang. Cape learning: Confidential and private collaborative learning. In International Conference on Learning Representations, 2021.
[13] Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency misnomer. arXiv preprint arXiv:21 10. 12894, 2021.
[14] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. arXiv preprint arXiv:2305, 143144 2023.
[15] Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[16] Haonan Duan, Adam Dziedzic Mohammad Yaghini, Nicolas Papernot, and Franziska Boenisch. On the privacy risk of in-context learning. The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.

---

[17] David Durfee and Ryan M Rogers. Practical differentially private top-k selection with pay- what-you- composition. Advances in Neural Information Processing Systems, 32, 2019.
[18] Cynthia Dwork. Differential privacy. In Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 13,, pages 1-12. Springer, 2006.
[19] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3-4):211-477, 2014.
[20] Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023. URL https://gi thips.. com/ http//ppellmmeeeeee heapmmmmeeeeerrcoppee..ee 1lama
thub.
[21] Jennifer Gillenwater, Matthew Joseph, Andres Munoz, and Monica Ribero Diaz. A joint exponential mechanism for differentially private top-k. International Conference on Machine Learning, pages 7570-7582. PMLR, 2022.
[22] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18553v1/D11--54000 URL https
//wwwaclweb..
aclweb. org/ www.annngoooi------ 5409
[23] Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Private inference on transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openrevie:: wwt.://oorrriidddeeeiioo.ooooo.
[24] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. In ACL (Association for Computational Linguistics), pages 2744-2751, Online, July 2020. ACL (Association for Computational Linguisties). doi: 10. 1111665/1112222..a--mmmmm
[25] Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang. DP-OPT: Make large language model your differentially--rivee prompt engineer. In The Twelfth International Conference Learning Representations, 2024. URL https
7/ // wtt.::towoommmi:111133llgg...a
[26] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09663 2021.
[27] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 1rX0v:2310.06825, 2023.
[28] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxxx org/ aaa/a/as///000 /aoss///411.m 221./2211.004 04088
[29] Simon Lermen and Charlie Rogers-Smith LoRA fine-tuning efficiently undoes safety training in Ilama 2-chat 70b. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL httpa///w..///////////o//ttottta
[30] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter- efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, November 2021. URL htttps://cclanhhooog..ooo
emnlp emnlp-main... 243
12

---

[31] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension. CoRR, 0e//1910.1346.. 2019. httttt/tp///xii.//////
wtwb/////... 13461
[32] Dacheng Li, Hongyi Wang, Rulin Shao, Han Guo, Eric Xing, and Hao Zhang. MPCFORMER:
FAST. PERFORMANT AND PRIVATE TRANSFORMER INFERENCE WITH MPC. In The Eleventh International Conference on Learning Representations, 2023. URL https://
openreview net/ Sidiummiiidii Cw= jOEhgit-
[33] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582- 4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021. acI- acl-long.353. URL
org/ 2021 aalaccllllol 353
[34] Xin and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL
aclweb
org/anthology/Co2-110 CC2-
[35] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language mod- els can be strong differentially private learners. In International Conference on Learning Representations, 2022. URL htttpss://opeereeeeert http:/oorro..eee--VP333ttt.t.
[36] Chin- Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL httpt::///wwtt www.ww.wlwbbboo.. org/ wttgg/annhhology/04-000000
[37] Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficiet fine-tuning is better and cheaper than in-context learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL httts://openreviiee net/forum?1d==rBCoWCC
[38] Jingjing Liu, Scott Cyphers, Panupong Pasupat, Ian Mcgraw, and Jim Glass. A conversational movie search system based on conditional random fields. 13th Annual Conference of the International Speech Communication Association 2012, INTERSPEECH 2012, 3, 01 2012.
[39] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprint arXiv:21 10.07602, 2021.
[40] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang. and Jie Tang. tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 12.188653/1/2022.acll 8
URL https://aclanthollgyy 202g/2222.ooo acl short
[41] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Ro[bert]a: A robustly optimized [bert) pretraining approach, 2020. URL htlllhttpp:///eeeevvieee http:eeeewe/ooooom/o/esyyyoo///
[42] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786 2021.
[43] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07), pages 94-103. IEEE, 2007.
13

---

[44] Ily Mironov, Kunal Talwar, and Li Zhang. R enyi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv:1908.10533 2019.
[45] Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797- 1807, Brussels, Belgium, October-Novemtee 2018. Association for Computational Linguistics. doi: 10. 10111665//1/DD1---- 1206. URL https: http://aaaaaattotoocoeo org/ 1206
[46] Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 75-84, 84, 2007.
[47] Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[48] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with PATE. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
[49] Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu, Bleu: method for automatic evaluation of machine translation. pages 311-318 2002.
[50] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation; Research Papers, pages 186-191, Belgium, Brussels. October 201 8. Association for Computational Linguistics. URL https:///vii www.ww.wwwwwwaaa.aae...... /nghhlogy
VI8-6319
[51] Alessandro Sordoni, Xingdi Yuan, Marc-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Deep language networks: Joint prompt training of stacked Ilms using variational inference. arXiv preprint arXiv:2306. 12509, 2023.
[52] Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3949- 3969, 2022.
[53] Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. context learning with differentially private few-shot generation. In The Twelfth International Conference on Learning Representations, 2024. URL httph:tt////o htttp::///peereviee. Selppenreriie.eee//oorumr
in-
id=oZtt0pRnD1
[54] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint ar 1805, 2023.
[55] Zhiliang Tian, Yingxiu Zhao, Ziyue Huang, Yu-Xiang Wang. Nevin L Zhang, and He He. Seqpate: Differentially private text generation via knowledge distillation. Advances Neural Information Processing Systems, 35:11117-11130 2022.
[56) Rubèn Tito, Khanh Nguyen, Marlon Tobaben, Raouf Kerkouche, Mohamed Ali Souibgui, Kangsoo Jung, Lei Kang, Ernest Valveny, Antri Honkela, Mario Fritz, and Dimosthenis Karatzas. Privacy-aware document visual question answering. arXiv preprint arXiv:2312 10108. 2023.
[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
14

---

[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[59] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/1/WW18-550 5446. URL BtLhttts::/ /tttps::/aclaannooooo..
org/W18-5446
[60] Tong Wu, Ashwinee Panda, Jiachen T. Wang, and Prateek Mittal. Privacy-preserving in-context learning for large language models. In The Twelfth International Conference on Learning
Representations, 2024. URL htttps://openrevieeet http:///wwrreeeea.otoorrrdddddd/ met:/forum??d==x4PP/1111
[61] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022. URL jd=Q42f0dfEECo)
[62] Jiachen ZHAO, Zhun Deng, David Madras, James Zou, and Mengye Ren. Learning and forgetting unsafe examples in large language models, 2024. URL
net/forum?id hd-hhkQYyUchhhhi
[63] Chujic Zheng, Fan Yin, Hao Zhou, Fandong Meng. Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL /tLLhttts://pporrrriii http:s///pppaaooiiooeeeeerororro http://rrrumiiduuuupppEEEEE
[64] Yuqing Zhu and Yu-Xiang Wang. Adaptive private-k-seledteee with adaptive k and application to multi-label pate. In International Conference on Artificial Intelligence and Statistics, pages 5622-5635. PMLR, 2022.
A Limitations
Hyperparameter tuning. Due to limited computational resources, we were not able to tune the hyperparameters for all open LLM private tuning methods. This is especially true for the generation tasks with longer input length, PFL-DocVQA for example, due to their long training time. Therefore, still the potential for an increase in performance. But, as our results for private tuning already show that these methods outperform the private in-context learning approaches, the used hyperparameters suffice for these comparisons.
Pretraining set as privacy risk. Closed models do not provide much information about their pretraining process. This also includes the non-disclosure of the pretraining set. For this reason, purposefully chose open LLMs where the pretraining set is known and can be easily downloaded. But, even though the pretraining set is available, and user could check it directly against their own data for any privacy leakage, we recognize that this process is also costly. This is why, despite having potential access to the pretraining set, user might not be able cover the privacy risk fully.
Cost estimation. To show the potential cost of running the training of privately tuning an open LLM, we used the cost of $0.69 per compute hour from RunPod. This represents the average cost of running such training, and the cost might vary for different users on machines of different cloud providers. Additionally, we ran our training same server setup as the one given by RunPod. This could lead to additional variations in training time and therefore training costs.
own machines, therefore, do not have the exact
this paper we looked at different closed models (GPT3 Davinci, GPT3 Babbage, GPT4 Turbo, and Claude 2. 1) and methods to add private in-context learning. Due to limitations to our budget, we were not able to look at a higher variety of closed models and also extend the experiments with GPT4 Turbo.
Budgetary limits.
15

---

Limited model sizes. In our experiments for the private tuning of open LLMs, we tried a variety of different models and model sizes. To handle the models with a higher amount of parameters, we needed to use 4-bit loading to be able to run the training on our GPUs. Unfortunately, due to some type-inconsistencies with Opacus, the Python library we use to run DP-SGD, and 4-bit loading from bitsandbytes, we were not able to do training on our machines for some of the models. This included 13B models for PrivateLoRA for the classification tasks and 7B+ models for PromptDPSGDGen and DP-FineTune for the text generation tasks.
B Further Details on the Related Work
B.1 Private Adaptations for Closed LLMs
We present the methods for the private adaptations of closed LLMs with a few more details.
PromptPATE [15 prompts an LLM with different prompts containing disjoint examples from the private training dataset, each prompt corresponding to a teacher. To label the public data for the knowledge transfer, PromptPATE [15] infers the next token prediction of each teacher on public text sequences and interprets them labels. Instead of training student model from scratch, PromptPATE [15 creates a student prompt. It utilizes the data efficiency of discrete prompts by selecting examples for the student prompt from the labeled public sequences.
DP-ICL 60). For the generation tasks, it proposes the Embedding Space Aggregation(ESAA) which involves mapping each sentence produced by the LLM for a given exemplar-query pair onto the embedding space and then reconstructing a sentence from the noisy mean of these embeddings. This process depends on the quality of the rext-to-embediing models and the zero-shot examples employed to map the noisy mean embedding back to the sentence, potentially leading to suboptimal outputs. The other approach proposed in DP-ICL is keyword space aggregation (KSA). It creates keyword space by segmenting each output sentence into keywords to form a histogram. The keywords with the highest counts selected privately using either the Propose-Test Exponential Mechanism (JointEM) 21). The selected private keywords are then used to create a prompt and query the L.M.
(PTR) or Joint
FehheGenn [53] introduces a method for text generation of public prompts. In this method, tokens are individually generated using disjoint subsets of the private data and then noisily aggregated based on the frequency of the generated tokens to predict the next token. The drawback of this approach is that the generation process is conditioned on the label. Consequently, despite being text generation task, necessitates the assignment of public label to the private data.
DP-OPT 25 is currently the only private ICL method that uses discrete prompt tranferrability to create private prompt on local open model, which can be used infer a closed model. Based on the approach of deep language networks [51), multiple initial prompts with different private examples are optimized through separate back- and forward passes such that a prompt is created that gives good performance the downstream task. To add privacy, they use the exponential mechanism to sample each generated token from all different initial prompts. Currently, their proposed method is only shown to work with classification tasks.
B.2 Private Text Generation based on PATE
SeqPATE (55) safeguards the privacy of individual training samples and sensitive phrases in the training data of language model. To adapt PATE for text generation, SeqPATE creates pscudo- contexts, simplifying the sequence generation task to a next-word prediction problem. To manage the extensive output space, SeqPATE introduces a candidate filtering strategy that dynamically narrows the output space and enhances the teacher aggregation in PATE to avoid low agreement caused by voting among large number of candidates. Additionally, further minimize privacy losses, employs knowledge distillation to reduce the number of teacher queries.
C Additional Details on our Setup
In this section, we present the detailed (hyper- pparameess used to evaluate the tasks that were used for the different Open and Closed LLMs privacy-preservigg training methods.
16

---

C.1 Text classification
Detailed information about the datasets. We expose the different statistics of each dataset used for text classification evaluation in Table For SST2, the validation set was used as the test set, as the original test set is only provided with unknown labels for each sample.
Table 7: Stastistics of the 4 evaluated tasks related to text classification.
Task
#Train
#Test
#Class
Task description
SST2
66,674
872
2
Sentiment analysis movie reviews
Tree
5,452
500
Question type classification
Mpqa 8,603
2,000
2
Sentiment analysis on short ensembles
Disaster
4,430
1,000
2
Relevance of sentence to a disaster
Private Tuning. We detail the hyperparameters used to fine-tune the models with private LoRA in T for DP-Fine Tune in Table for PromptDPSGID in Table TO) All the experiments were conducted on different seeds. Note that unlike LoRA or Full-Finetune, PromptDPSGID requires a precise tuning of hyperparameters. A total of 50 trials over 100 epochs were necessary to tuned them. For the Mpqa sentiment analysis task, no converging set of hyperparameters was found.
Table 8: Hyperparameters for PrivateLoRA 66 on evaluated classification datasets for = 8.
Hyperparameters
Datasets
SST2
Trec
Mpqa
Disaster
bs
128
128
128
128
le-3
le-3
le-3
le-3
max grad clip
0.1
0.1
0.1
0.1
epochs
10
40
20
20
lora rank
4
4
4
[21
D
GradClip
0.1
0.1
0.1
0.1
Table Hyperparameters for DP-FineTune Large for ε = 8.
on evaluated classification tasks with Roberta-
Hyperparameters
SST2
Trec
Mpqa
Disaster
LR
le-4
le-4
le-4
le-4
BS
128
128
128
128
Epoch
10
40
40
50
D
[D]
[7]
[D
GradClip
0.1
0.1
0.1
0.1
17

---

Table 10: ayperparameters for PromptDPSGD [5]. The hyperparameters for SST2 datasets are directly extracted from the paper and are evaluated on Roberta-Large for 8. LR = learning rate, BS = batch size, GRAD = per sample gradient clipping. P-length = length the prepended prompt in number of tokens. The trainings are all performed with prefix-tuning and not soft-prompt. Those are the hyperparameters of the best performing prompt on the test set of each dataset, and the accuracy of this prompt is reported in the table.
Hyperparameters
SST2
Trec
Mpqa
Disaster
LR
0.01
0.001
0.01
BS
32
32
32
32
GRAD
0.3
1.0
Epochs
22
100
100
100
P-length
10
10
10
Best accuracy
92.8
58.0
50.0
78.6
Private in-context learning. The respective set of hyperparameters for DP-FewShotGen, Prompt- PATE and DP-ICL are listed in Table 11] Table [2]and Table [3 For the used hyperparameters for DP-OPT, see 25 since the results of Table are directly extracted from the paper. The accuracy results for DP- De- were computed for 5 different generated prompts following the method from the paper. For the PromptPATE method, experiments were only conducted for MPQA and Disaster datasets as we used already made evaluation from the original paper PromptPATE [15] for SST2 and Trec datasets on using GPT3-Babbage. All hyperparameters here are extracted directly from the previous paper.
Table 11: Hyperparameters for DP-FewShotGen 53 for the evaluation of new datasets with E on GPT3-Babbage M = Number of private prompts used for meta prompt generation. N = number of private shots per prompt. o noise relative wanted g using the Gumbel mecanism. Tmaz max number of tokens of the generate prompt.
Hyperparameters
SST2
Tree
Mpqa
Disaster
[1.0.0.61, 0.48,0.34]
[3.0.0.83, 0.59,0.44]
[2.0,0.77. 0.57,0.41]
[3.5,0.93,
o (6 (0.1, 1,3, 8)
0.64,0.46]
MN
80
80
80
80
M
20
20
20
20
Tmaz
50
50
50
50
Table 12: Hyperparameters for PromptPATE 15 for the evaluation of new datasets with & = on GPT3-Babbage.
Ba..aaabbae.hheeee Those parameters common to all 4 tasks.
Hyperparameters
Claude
GPT3-babbage
train set
400
400
student set
200
300
num shots
2
18

---

Table 13: Hyperparameters for DP-ICL 60 for the evaluation of the text classification datasets with = on GPT3- -aTbbBBa
Hyperparameters
SST2
Trec
Mpqa
Disaster
num shots
4
4
Ensemble
10
10
10
Queries
872
500
1000
1000
C.2 Text Generation
We analyze the following generative downstream tasks: SAMSum, PFL-DocVQA, and MIT Movies trivial0k13, 13. As we did for classification tasks, compare the methods on closed LLMs against PrivateLoR A [6], PromptDPSGD [15), and DP-FineTune (35) that are run on open LLMs. For the PrivateLoR A [6] training, we use 4-bit quantization with QLoRA 14 to reduce the occupied GPU memory, which was implemented for the adaptations of open LLMs with more than IB parameters on PFL-Doc VVAA and SAMSum datasets due to their long input sequences.
Detailed information about the datasets. We show the amount of data that we utilized in the experiments in Table 14
Table 14: Overview of the text ge tasks related to text generation.
Task
#Train
#Test
Task description
SAMSum
14,732
819
Dialogue summarization
PFL-DocVQA
85,000
10,000
Question and answering
MIT-G 2,953
780
Extracting genres from movie reviews
MITD 1,561
415
Extracting directors from movie reviews
Private Tuning. In Table 15 Table 16 and Table 17 to train the open models with PrivateL.RRA PromptDPSGDDGenn, DP-Fine'Tume respectively. For PrivateLoRA, we were able use the same hyperparameters for all models for each task. In the tables, the Max Seq Length refers to the maximum amount of tokens of the sequence the model trains on. For Schedulers, we chose different options, constant scheduler that does not change the learning rate during training, and linear scheduler. The linear scheduler is the default scheduler of the Hugging Face implementation of the Trainer class. It linearly decreases the learning rate over the whole training. For PrompiDPSGDGen, we additionally have Prefix Projection. If enabled, prefix projection adds two additional linear layers trainable parameters, which in turn also increases the capability of the prefix represent tasks. The evaluations for MIT-D, MIT-G, and SAMSum were done for different seeds, whereas we used different seeds for PFL-DocVQA.
show the hyperparameters used
prefix encoder. This increases the amount of
19

---

Table 15: Hyperparameters for PrivateLoRA [6] on evaluated generation tasks for 8. The hyperparameters are the same for the used models. The tested schedulers for MIT-G and MIT-D does make a difference during training
Hyperparameters
SAMSum
PFL-DocVQA
MIT-G
MIT-D
LR
8e-4
8e-4
8e-4
8e-4
BS
256
256
256
256
LoRA Rank
8
8
8
Max Seq Length
650
1500
128
128
Epoch
20
15
20
20
Scheduler
Linear
Linear
[P]
[P]
[|]
[D]
GradClip
0.1
0.1
0.1
0.1
Table 16: Hyperparameters for PrompiDPSGDGen evaluated generation tasks for a = 8. The hyperparameters are the same for the used models. The tested schedulers for MIT-G and MIT-D do not result in difference in performance.
Hyperparameters
SAMSum
PFL-DocVQA
MIT-G
MIT-D
LR
le-3
1e-3
1e-3
3e-3
BS
256
256
256
256
P-Length
10
25
5
5
Prefix Projection
True
True
True
True
Max Seq Length
650
1500
128
128
Epoch
20
40
40
Scheduler
Linear
Linear
[b]
1
[D]
[D]
GradClip
0.1
0.1
0.1
1
Table 17: Hyperparameters for DP-FineTune (35] on evaluated generation tasks for = 8.
Hyperparameters
SAMSum
PFL.-DoVQQA
MIT-G
MIT-D
LR
8e-4
2e-4
2e-4
2e-4
BS
256
256
256
256
Max Seq Length
650
1500
128
128
Epoch
20
15
20
20
Scheduler
Linear
Linear
Constant
Linear
8
D
D
D
GradClip
0.1
0.1
0.1
0.1
Frivacy-preservin prompt tuning. In the following, we provide the used hyperparameters for the methods for Private ICI. for Closed L.L.s. In detail, for DP-FewShotGien in Table T8, for DP-ICI. in Table 19 and for PromptPATEGen in Table 20
20

---

Table 18: Hyperparameters for DP-FewShotGen [33 on evaluated generation tasks for 8. We used the hyperparameters given in the original paper for MIT-G and MIT-D.
Hyperparameters
SAMSum
MIT-G
MIT-D
0.384
0.5
0.58
MN
80
80
80
M
20
20
20
T Tmoz
50
20
20
Table 19: Original hyperparameters for DP-ICL (60] on evaluated generation tasks for & = 8.
Hyperparameters
SAMSum
PFL-DocVOA
Model GPT-Davinci
OpeALLaMMA 13B
Ensemble 100
100
#Queries
10,000
10,000
Table 20: Hyperparameterss for PromptPATEGen on generation tasks for 8.
Hyperparameters
SAMSum
MIT-G
MIT-D
Model
Vicuna 7B
Vicuna 7B
Vicuna 7B
Ensemble
100
25
25
#Queries
100
100
100
#Student Prompt
4
1.15
0.9
0.9
D Cost Calculation
We provide the details on measuring the cost for difffrrnn methods. The assumed costs for interacting with the model APIs per million tokens and GPU cost per hour are shown in Table 22 For the open LLMs, we the median pricing per hour (based
prices from three GPU cloud providers shown in Table 22) which is $0.69 using an A40 GPU with 48GB of memory 3 which is popular graphics card, also used the previous work [S]. We note that we do refrain from using other metrics than monetary cost. For example, FLOPS are not a direct measurement of real-world computational cost because latency, power usage, and other costs can vary significantly depending on hardware and other factors 13
adaptations. The private tuning-based adaptations of open L.MSs require us to adjust the model parameters or the inputs for given task, thus, we measure the running time of the training process and then query answering.
Costs for srivate-tuning-baar
Costs for private ICL-based adaptations. DP-ICL does not incur any training cost but uses an ensemble of teachers for each query (the same PromptPATE for labeling public examples), which elevates the cost by the number teachers, which can be 10 or even 100. For PromptPATE, the generation of public student prompts is done using ensemble of teacher prompts, thus labeling each public data point costs much more (proportional to the number of teachers) than running a query (with single prompt). Seoteen also uses ensemble of prompts, where the number of accesses to the API the training process is equal to the number tokens in a public prompt. The cost of training the public prompt for DP-OPT is through the iterative process of instructing the local model to improve the prompt and obtain better predictions, however, this part is done on a local open
The pricing is for the RunPod Cloud Service: https://uwwrruppo. iomoodd..oo///e h/ppp-uaaaae instance/pricingg
:
21

---

LLM, thus, the cost is relatively low. For ease of approximation and to the benefit of the ICL methods, we assume that the creation of the teacher prompts and the private aggregation of the outputs have negligible costs. After preparing the public prompt, PromptPATE, DP-FewSShotGen, and DP-OPT, need a single access to the API to answer a query.
To obtain the cost for closed LLMs, we have to compute the average number of tokens per query. For the classification task, we can take the example of the DP-OPT method applied on the SST2 dataset. For this dataset, only one token is returned by the API provider, so the cost of the outputs is negligible. SST-2 inputs have an average length of 12.35 and the best performing prepended prompt from DP-OPT training has length of 39 tokens. Thus, for the DP-OPT task, for each query to the API, 41.35 tokens are sent approximately. This gives a cost of $0.0006 per query for GPT-3 Davicc and the total cost of $6 for 10k queries in Table []] depending on the size of the prepended prompt of each ICI. method. Regarding the generation task, we can take the example of the SAMSum dialog summarization dataset, in which the average token length 141 for the input and 26 for the output, hence, a single query costs 50.000333 (for GPT3-Davinc)). The cost for a O-shot inference to Davinc would therefore be $3.33 for 10k queries. As DP-ICL considers the 1-shot scenario and an ensemble of 100 teachers, we add the average input and label lengths to the input and multiply this by the size of the ensemble, which results in overall cost of roughly $666. The exact average token count for each dataset which we used for the cost estimations can be found in Table 21
The cost per query is computed similarly,
Table 21: Average token length of different inputs and outputs of the used datasets. The average does not include instructions.
Dataset
SST-2
Trec
Mpqa
Disaster
MIT-D
MIT-G
SAMSum
DocVQA
Input
12.35
11.43
3.88
30.79
25.276
24.314
140.857
924.191
Output
1
1
1
3.877
2.301
25.620
6.384
Table 22: Pricing for the models and cloud options (as of May 22nd, 2024).
Model
Cost/IM tokens
Cost/hour
Input
Output
GPT-Babbage
$0.40
$0.40
GPT-Davinci
$2.00
$2.00
GPT-3.5-turbo Instruct
$1.50
$2.00
GPT-4-turbo
$10.00
$30.00
Claude 2.1
$8
$24
A40 (RunPod)
$0.69
A40 (Replicate)
$2.07
A40 (Hyperstack)
(starts from)$0.50
E
Additional Experiments
PrivateLoRA extensive classification results. Table shows the topl accuracies at different used compute the PrivateLoRA graph for each of the text classification tasks in Figure
httpss//oppnaa.com///pppiiiiiiio
com/api
hhttps//www.nn. www.pppp...gpp-innntneeeeeeeeeee
"https//replicate.com wwwp:ppcccoo///iiiiin.cc..
http:///www.hhyssstetee. wtwpp///aac...oo/puuuuueee wwwpdd/pppppiiiinn
22

---

Table 23: Private LoRA 61
tapl-accuracies for the evaluated datasets given different
Model
SST-2
Trec
Mpqa
Disaster
8
Vicuna 7B
95.3
97.4
88.4
82.0
3
Vicuna 7B
94.4
96.2
87.6
79.6
Vicuna 7B
93.5
93.8
82.1
78.1
0.7
Vicuma 7B
93.4
93.2
79.5
76.4
0.3
Vicuna 7B
91.9
87.6
64.4
73.6
Safety Evaluation of Mixtral-8x7B Instruct with and without differential privacy. We conducted additional experiments to analyze how fine-tuning a downstream task with and without differential privacy affects the safety alignment of models. We followed the approach from 63 evaluate Bitral-8x7B-instrue and fine-tuned the model on SAMSum once with ε 8 and once with xc. We selected SAMSum it does contain conversations with unsafe language (e.g.. cursing or harassment). These results are presented in Table 24 The table is divided into two sections, "Compliance Harmful Queries" and "Refusal Harmless Queries". The scores were generated by separately prompting our model with 100 harmful and 100 harmless queries, each repeated 20 times with different safety prompts. Finally, the outputs are categorized as complying or refusing the input by Llama Guard, giving the percentage of incorrectly handled answers in our table. Therefore, the lower the score, the better.
First, we observe that fine- tie-ttning of any kind decreases the model's safety capability. This is easily identifiable in the table, the original model has lower scores across the board for compliance with harmful behavior compared to both fine-tuned models. Similar results were also shown in [29] and [62), where the authors used dataset containing unsafe samples fine-tune safety-aligned models, which drastically increases compliance with unsafe behavior.
Second, we can observe that private fine- line-tuning affects the safeguards less than non non-private fine- tuning. the non-private case, the influence of individual samples unrestricted, which can amplify the impact of unsafe samples, shown in 24 and (62). In contrast, differential privacy limits the influence of any single sample. Consequently, the impact of unsafe examples is minimized, resulting in model that retains more of its safe behavior.
Table 24: Safety Evaluations of Mixtral-8x7B Instruct Evaluating the safety of the responses given by different models. compare the base instruction fine-tuned model with the same fine-tuned on SAMSum with and without differential privacy. The model responses were categorized as harmful and harmless by Llama Guard.
Model
% Compliance Harmful Queries
% Refusal on Harmless Queries
Mixtral-Be -x7B Instruct
short
Mistral
Llama
no prompt
short Mistral
Llama
no prompt
Base model
0
8 00
38
30
39
5
Analysis of PromptPATEGen and DP-ICL [60] across different generation tasks. We observe that PromptPATEGen is particularly adept at summarization tasks, shown in Table However, when we shift focus to other task generation tasks, the performance of PromptPATEGen is not as impressive compared to DP-ICL [60]. For the MIT datasets, which are entity retrieval tasks, PromptPATEGen generates results that are semantically similar to the true values, but not identical. For instance, the MIT-G dataset, movie genre is "sci fi".. This pattern of generating semantically similar results is also observed the MIT-D dataset, where only one name is given the movie director rather than the full names.
CompttPATEGen generates "sci-fi" whereas the ground truth
We further added evaluations using DP-ICL. 60) with newer OpenAl models, such as GPT4 Turbo and GPT3.5 Turbo, for SAMSum and saw that the performance only increases slightly compared
23

---

Table 25: The dialogue, ground truth, and summary generated with DP-ICL 60 on GPT4 Turbo for SAMSum sample
Ursula: Haha got 93 on my French exam
Bob: Well done girl!
Jana: Wow
Jana: How did manage to do that
Ursula: I just studied hard for it
Dialogue
Jana: omg
Jana: French is so hard
Vaughn: I got a 65
Vaughn: I didn't study for it haha
Ursula: At least you passed
Vaughn: Congrats!
Ground truth
Ursula got 93 on her French exam. Vaughn got 65, but still passed.
Ursula shares that she got a 93 on her French exam and Jana asks how she managed to so well. Ursula says she studied hard for it. Vaughn admits to getting 65 without studying and Ursula congratulates him for passing.
Generated summary
to the original evaluations GPT3 Davinci (see Table 4). Therefore, we looked through the summarizations that were given by GPT4 Turbo and compared them to the ground truths. Similarly to the above-mentioned problem with the MIT-G and MIT-D datasets, we found, that the results were semantically correct, but not close enough to the ground truth. As an example, we show a sample generation with the corresponding dialogue and ground truth in Table 25 We see, that the generated summary is factually correct, but too extensive compared to the ground truth. In particular, this example results in Rouge-1 score of 31.8. Our assumption, thus, worse samples, is the limited examples in terms of summary structure given by the keywords provided by DP-ICL [60]
get more extensive, and
The Doc dataset presents a unique challenge for PromptPATEGen. Given the high average token length in this dataset, only single example can be used as the in-context example. This limitation poses a significant challenge, single example may not adequately represent the diverse aspects present in the dataset.
533 limitations on new GPT models. In addition running DP- FewShotGen 533 on GPT Babbage and GPT Davinci, we also adapted the official code to work with newer OpenAl models, such as GPT4. The code had to changed, as OpenAl moved from their Completion API to the new Chat Completion API. During execution, we encountered an issue that prevents us from running experiments with the Chat Completions API with Oe-FeeShhrtGen (53]. The issue arises from the parallel use of logit bias and log-probabilities, as the resulting output while using both together is unusable for further processing. We assume, this behavior caused by defense mechanism against a new potential model-stealing attack [8], one of the suggested defenses was to prohibit the use both logit bias and log-llobbe probabilities at the same time, which would explain what have experienced. As the implementation of OP-FewShotteen [53] uses both the same time, we are unable to get this private ICL method to work with GPT models that use the Chat Completions API.
De-
F Generation Metrics
In this section, we briefly discuss the different metrics to evaluate the generation tasks.
Rouge [36]. The metrics in the Rouge, short for Recall-Oriented Understudy for Gisting Evaluation, set describe how many word-wise n-grams match between the predicted and target text. For Rouge-1, we look at uni-grams whereas for Rouge-2 we calculate the similarity of all 2-grams. Rouge-L. refers to the similarity of the longest common subsequence between prediction and target. Important to note for Rouge-L, the grams do not need to be consecutive, but have to be in order. The scores lie

---

between 0 and 100, where 100 is the best score.
BLEU 44). Similar to the Rouge metric, the BLEU score, which is the abbreviation for Bilingual Evaluation Understudy, is used evaluate the similarity of generated and reference text. To calculate the score, the precision and brevity between the two sentences have to be determined. The precision the ratio of n-grams that match exactly between generated and reference text. Usually, goes up to 4. Brevity, on the other hand, penalizes the score of the generated text, if it's shorter than the reference. Combining brevity and precision results in the BLEU score of the generated text. The score itself is again between 0 and 100, where higher scores are better. We use the SacreBLEU 50 version of BLEU.
Levenshtein Distance. Lastly, evaluate PFI.-DoeVQA, also the Levenshtein Distance. This metric is used to directly compare strings on a letter by letter basis. The Levenshtein Distance calculates the minimum amount of substitutions, insertions, and deletions between sequences. We use the normalized version to have a score between 0 and 100 independent of sequence length. As with the other metrics, the higher the score the better.
G Abbreviations
In Table 26 we show the abbreviations we used throughout this paper for the different private in context learning methods of LLMs.
Table 26: Abbreviations for ICL papers and their proposed techniques.
Publication
Publication Name
Technique
Privacy Technique
Abbreviation
Abbreviation
DP-ICL
Privacy-Preserve In-Context Learning for Large Language Models [60]
DP-ICL Classifi-
DP-ICL for text classifica- tion
cation
ESA
Embedding Space Aggre-
gation
KSA
Keyword Space Aggrega- tion
DP-OPT
DP-OPT:
Make
DP-OPT
Diffcrentially-Piiie Privvtte Off- site Prompt Tuning
Large Language
Model Your Privacy-
Preserving Prompt
Engineer 25
FewShottGen
Privacy- Prenee--Preevving
FewShotGen
Differential Private Few- Shot Generation
In-Context Learning
with Differentially
Private Few-Shot
Generation [33
PrivatePrompts [15] Flocks of Stochastic
PromptDPSGD 15
DPSGD for Private Soft Prompt Learning
Parrots: Differen-
tially Private Prompt Learning for Large
Language Models
[5]
PromptPATE 15
PATE Preserving
for Privacy- Discrete
Prompts
25

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Section 4 we showcase the outcomes of our thorough experiments, which compare the performance, cost, and privacy protection of open and closed LLMs. Section details our proposed prompt-based adaptation techniques for text generation tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We presented the limitations of the paper is in the Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
26

---

Justification:
Our paper does not include theoretical proofs. For our theoretical considerations, we provide full of assumptions and detail the setup.
Guidelines:
The NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed information of all the models, datasets and hyperparameter configuration used for obtaining our results in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, reproducibility can also provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited some way (e.g.. to registered users). but it should be possible for other researchers have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
27

---

Justification: The details of our experimental setup is in Section 4.2. For the closed LLMs, the models can be accessed via their respective API. For the open LLMs, they are publicly available on the HuggingFace hub. We have included citations and links for both closed and open L.MM in the paper. Throughout our paper, we used standard benchmark datasets which are open-sourced datasets. The source for the datasets are well referenced in the paper. Our anonymized codes is included the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
http://iippccccppbl//
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurIPS code and data submission guidelines guides/Codeeuubmi ssi ssinnciycy for more details.
reproduce
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings is presented in Section 4.2. All the hyperparameters used the paper is included in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In our paper, we consistently provide the mean and standard deviation across multiple runs of our experiments for comprehensive understanding. However, certain instances, we have refrained from including the standard deviation to maintain the readability of the tables. This is particularly applicable when the inclusion of standard deviation results in tables that are excessively wide, thereby posing challenges in terms of fitting into the page width and ensuring seamless reading experience.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: In Appendix D provided details about the compute resources and the costs to reproduce each experiment. All experiments were conducted on A40 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics httph:tt///// /tnprri///eurps... http::ppp.ppp.....ttttuuu.uaaaa.
Answer: [Yes]
Justification:
We followed the ethics guideline and adequately anonymized our paper and code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper advocates for the more private, efficient, and cost-effective alternative of open LLMs which have significant positive societal impact. In Section 5) we summarize our findings, shedding light on the three pivotal aspects-privacy preservation, performance optimization, and cost-effectiveness. Moreover, we highlighted the broader impacts of our work in Appendix 5
29

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification:
We included citations for all models, datasets and code used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwpppsaipphsaeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We used publicly available assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve the use of human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
31


Reviews:
[{"review_id": "OKVAIyc75Q", "reviewer": "Reviewer_1UUG", "summary": "This paper compares and contrasts the privacy protections of open vs closed LLMs through conceptual threat models and experimental evaluation. The authors give experimental evidence on local gradient based adaptations performing better than their closed discrete prompt-based counterparts in the areas of privacy, cost-efficiency and performance. The paper concludes that the use of open LLMs not only yields more privacy, but also higher performance at lower costs.", "questions": "N/A", "limitations": "Limitations were addressed", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and clearly explains what the contributions are, the background for both DP and LLMs, as well as their experimental evidence. The authors make a compelling case behind the reasons for privacy-preserving adaptations for closed LLMs not being as effective. The problem addressed in the paper is important and the contribution seems novel.", "weaknesses": "I think the paper is well-written and don't have any specific feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**The paper is well-written and clearly explains what the contributions are, the background for both DP and LLMs, as well as their experimental evidence. The authors make a compelling case behind the reasons for privacy-preserving adaptations for closed LLMs not being as effective. The problem addressed in the paper is important and the contribution seems novel. I think the paper is well-written and don't have any specific feedback. Limitations were addressed.**\n\nThank you for the positive feedback and insightful comments. We greatly appreciate your review. Please let us know if there are any additional steps or information we can provide to further enhance your confidence in the assessment of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VV7Oidd3lQ", "reviewer": "Reviewer_gGr3", "summary": "The paper compares private adaptation between closed LLMs and open LLMs, and the authors find that adapted open LLMs always perform better than closed ones at much lower cost.", "questions": "Please tune down the claim by adding a remark that the finding only applies to existing methods.", "limitations": "As above", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper presents a comprehensive overview of privacy-preserving adaptation techniques for LLMs. It thoroughly examines various existing methods and compares them across key factors like performance and cost.\n2. The results are striking: open-source LLMs consistently outperform their counterparts in virtually every aspect.\n3. The paper also discusses the private data flowing and point out that the private data might still be leaked to LLM providers in 3 out of 4 adaptation methods studied.", "weaknesses": "My main concern is that the claim that open LLMs are essential for achieving high-quality, privacy-preserving adaptation might be premature. It could be that the field is still new, and we simply haven't yet developed sufficiently powerful algorithms for privacy-preserving adaptation of closed models. I suggest tuning down this claim slightly, acknowledging that the field is relatively new and rapidly evolving, and that the current findings may only be applicable to the privacy algorithms currently available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive, encouraging, and constructive feedback. We are pleased that the reviewer recognizes our thorough analysis and finds the results *\"striking\"*. We address the main concern below:\n\n>**My main concern is that the claim that open LLMs are essential for achieving high-quality, privacy-preserving adaptation might be premature. It could be that the field is still new, and we simply haven't yet developed sufficiently powerful algorithms for privacy-preserving adaptation of closed models. I suggest tuning down this claim slightly, acknowledging that the field is relatively new and rapidly evolving and that the current findings may only be applicable to the privacy algorithms currently available. (...) Please tune down the claim by adding a remark that the finding only applies to existing methods.**\n\nThank you for sharing your concern. We greatly appreciate your feedback. As requested, we have toned down the statements and provided the revised sentences below.  \n\nSpecifically, we changed the title to: “Open LLMs are Necessary for **Current** Private Adaptations and Outperform their Closed Alternatives”.  \n\nWe also added the following statement at the end of the abstract: **“It is important to note that the field of privacy-preserving adaptations of LLMs is relatively new and rapidly evolving. Consequently, our findings are based on the currently available methods and may evolve as new techniques are developed.”**\n\nTo elaborate on our purpose, we assess the state-of-the-art methods for privacy-preserving adaptations of LLMs based on all the papers in this domain (starting from ICLR submissions in 2022 and up until this submission deadline). Our evaluation also includes the latest models, such as Llama 3 (released on April 14th, 2024, which was shortly before the NeurIPS deadline) and GPT-4. Additionally, we have developed new techniques to enhance the private adaptations of LLMs. Our primary goal is to identify the current strengths and weaknesses of the private adaptations and provide better methods, especially for users who would like to leverage LLMs on their private data. We do develop new private adaptations in this submission to support generation tasks, such as dialogue summarization, and provide insights to help the community advance in this field. Taking into account the fact that there are at least 7 different methods for private adaptations, we contribute 2 more and point to the currently most promising directions.\n\n**The statements that already indicate the consideration of only the current methods are bolded:**\n- We already stated in the abstract the following: (starting from Line 5) In this work, we analyze the privacy protection and performance of **the four most recent methods** for private adaptation of closed LLMs.\n\n**The updated parts of the statements in the submission are bolded:**\n\n- At the end of the abstract (starting from Line 19):  This yields the conclusion that to achieve truly privacy-preserving LLM adaptations that yield high performance and more privacy at lower costs, **taking into account current methods and models**, one should use open LLMs.\n- Starting from Line 51: “Overall, our results indicate that from the perspective of effective privacy-preservation, **current** adaptations of open LLMs are strictly preferable over their closed LLM alternatives, since they are more private, more performant, and less expensive.”\n- Line 45: Our results provide the following insights: (1) All **currently available** methods for adapting closed LLMs leak private query data (intended for the data owner) at inference time to the LLM provider. \nStarting from Line 70: Our extensive experiments on various open and closed LLMs and multiple classification and generation tasks show that the local (gradient-based) adaptations outperform their **current** closed (discrete prompt-based) counterparts in terms of privacy, performance, and cost efficiency.\n- Line 191: DP does not aim at protecting query data. Hence, none of the **current** private ICL methods attempt to protect that data against the LLM provider.\n- Line 333 onward (in Section 5: “Discussion and Future Work”): On the contrary, the leakage of private query data to the LLM provider is **to date** an inherent problem with closed LLMs, since no methods to provide formal guarantees for the query data are **currently** known.\n- Line 351: “**We hope that implementing the above-mentioned solutions will shrink the gap between private adaptations of open and closed LLMs.**”\n\n---\n\nOnce more, we thank the reviewer for the constructive feedback. Addressing the concern has improved the quality of our work. We look forward to further discussions and continued advancements in this evolving field. Please let us know if we can provide any additional information to further enhance your assessment of our work and potentially increase the score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mpMJRj6XVJ", "reviewer": "Reviewer_k2x5", "summary": "This paper compares the performance and privacy leakage between private adaptations on closed and open-source LLMs. The authors conclude that adaptations on open-source LLMs result in better performance, lower training costs, and enhanced privacy protection", "questions": "The authors categorize privacy leakage into Private Training Data and Private Query Data. However, I did not see any data inference in the experiments. Is there any data reconstruction attack or metric used to measure the leakage in your model?", "limitations": "Please refer to the weaknesses and questions sections.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper presents extensive experiments on adaptations of both closed and open-source LLMs, including four private In-Context Learning (ICL) methods for four closed LLMs and three private tuning methods on four open models.", "weaknesses": "1. The work lacks novelty in its methods. All the private adaptation methods used are pre-existing, with the authors only extending two methods from classification tasks to generation tasks. This makes the paper more of a benchmark work rather than original research. Additionally, as noted in section 4.2, \"Previous work [31] has shown for non-private settings that gradient-based tuning methods (used for open LLMs) offer better accuracy and significantly lower computational costs than ICL (used for closed LLMs) since the adaptations can leverage the internal behavior of the LLM.\" Thus, it is intuitive that the performance would be similar under DP scenarios, making the conclusion to use open LLMs less impactful.\n\n2. It seems unnecessary to use private tuning methods on open LLMs since the trained model will not be shared and only queried. As the authors state, \"Looking at Figure 1, it becomes obvious that any private tuning method executed on that open LLM would, conceptually, improve privacy protection since the LLM provider would neither be involved in the adaptation nor in the use of the adapted LLM, yielding absolute privacy against them.\" If a company locally fine-tunes the model and then preserves it locally for queries, direct fine-tuning would be more efficient.\n\n3. The comparison in the experiments is unfair and lacks proper baselines. The experiments compare task performance between various adaptations on closed and open LLMs. However, the closed and open models are different and show varying performance even without fine-tuning. The zero-shot performance of these models is not provided, leaving no baseline for reference. Additionally, DP-based ICL methods and fine-tuning methods offer different levels of privacy protection. As the authors mention, \"Yet, the threat model of multiple private ICL methods for closed LLMs does not include providing privacy against the LLM provider.\" It is not fair to compare their performance with the same parameter $\\epsilon$ as shown in Figure 2, which could mislead readers into thinking they offer the same level of privacy protection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and we are happy that the reviewer appreciates our extensive experiments. We address individual points below one by one:\n\n>**W1: The work lacks novelty.**\n\nWe provide a comprehensive privacy analysis and introduce differentially private prompts for generation tasks that, for the first time, achieve performance comparable to LoRA or fine-tuning. The argument for using open LLMs for private adaptations extends beyond their ability to employ gradient-based methods. Our work provides deeper insights into the problem and a thorough overview of the field. Additionally, methods proposed for closed LLMs can be applied to open LLMs. Notably, our PromptPATEGen (an ICL-based approach) applied to open LLMs outperforms other private adaptations of closed LLMs, as evidenced, e.g., in Table 4.\n\n>**W2: It seems unnecessary to use private tuning methods on open LLMs since the trained model will not be shared and only queried.**\n\nPlease refer to Figure 1 in the submission, where we explain that the answers to the queries leak information about the private data used to create the prompts. Note that the previous work [1, 2, and 3] demonstrated that the data used to create the prompts can leak to the querying party, e.g., by leveraging the membership inference attacks (the reviewer also asked for such attacks in the Question below).\n\n>**W3: The comparison in the experiments is unfair and lacks proper baselines. The experiments compare task performance between various adaptations on closed and open LLMs. However, the closed and open models are different and show varying performance even without fine-tuning. The zero-shot performance of these models is not provided, leaving no baseline for reference.** \n\nThe zero-shot performance was provided in the cited papers. We also included the required numbers in the Table below and updated Tables 3, 4, 5, and 6 in the main paper accordingly. Please also refer to Figures 2 and 3 in the attached PDF, where we present the performance for zero-shot and the $\\varepsilon=\\infty$ as baselines. Our observations indicate that adapting models with small $\\varepsilon \\in [0.3,1]$ results in significantly higher performance compared to zero-shot. Additionally, adaptations with $\\varepsilon=8$ achieve performance levels close to non-private adaptations. Overall, while the zero-shot performance is higher for the closed LLMs, the private adaptations perform better on open LLMs than on closed ones.\n\nTable: Comparison between closed vs open LLMs with baselines: zero-shot and $\\varepsilon=\\infty$.\n\n|Method|Model|SST2|Trec|Mpqa|Disaster|Average|\n|---|---|---|---|---|---|---|\n|zero-shot|GPT3-Davinci closed|92.4$\\pm$0.0|51.8$\\pm$0.2|84.5$\\pm$0.1|76.4$\\pm$0.2|76.3|\n|$\\varepsilon=8$ DP-OPT|GPT3-Davinci closed|92.2$\\pm$0.8|68.7$\\pm$6.5|85.8$\\pm$0.7|78.9$\\pm$0.3|81.4|\n|$\\varepsilon=\\infty$ ICL|GPT3-Davinci closed|94.7$\\pm$0.4|79.1$\\pm$0.5|88.8$\\pm$0.1|69.0$\\pm$5.9|82.9|\n|zero-shot|Vicuna 7B open|85.8|47|78.8|56.7|67.1|\n|$\\varepsilon=8$ PrivateLoRA|Vicuna 7B open|96.0$\\pm$0.1|96.8$\\pm$0.2|87.3$\\pm$0.2|80.8$\\pm$0.1|90.2|\n|$\\varepsilon=\\infty$ LoRA|Vicuna 7B open|96.4|98.2|87.5|82.1|91.1|\n\n>**It is not fair to compare their performance with the same parameter $\\varepsilon$ as shown in Figure 2 which could mislead readers into thinking they offer the same level of privacy protection.**\n\nWe show in Figure 1 that *privacy leakage types* A (the data owner’s private data leaks to the LLM provider) and B (the private query of the querying party leaks to the LLM provider) are inherent to closed LLM and are not protected by any of the tested methods. Thus, we measured the privacy leakage using the DP framework only for case C (where private information from the data owner leaks to the querying party). Even under this most favorable scenario for closed LLMs, they underperform compared to open LLMs when used for private adaptations. \n\n>**The authors categorize privacy leakage into Private Training Data and Private Query Data. However, I did not see any data inference in the experiments. Is there any data reconstruction attack or metric used to measure the leakage in your model?**\n\n**Private Training Data:** Previous work [2, 3 (Table 8)] demonstrated that applying (stronger) privacy guarantees reduces privacy leakage. In our submission, Figures 2 and 3 illustrate the performance depending on the privacy budgets $\\varepsilon$ ranging from 0 to 8. The lower the $\\varepsilon$, the lower the leakage (better protection against membership inference or reconstruction attacks).\n\n**Private Query Data:** For closed LLMs, queries are sent directly to the LLM provider in plain form, resulting in clear privacy leakage. In contrast, with open LLMs, the query is only shared with a data owner, such as a hospital or a bank, which already has access to private data. Thus, the private adaptations via open LLMs minimize the exposure of sensitive information. \n\n**References:**\n\n[1] “On the privacy risk of in-context learning”. Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, Franziska Boenisch. In The 61st Annual Meeting Of The Association For Computational Linguistics 2023.\n\n[2] “Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models”. Haonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska Boenisch. In the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) 2023. \n\n[3] \"DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer\". Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng Li, Bo Li, Zhangyang Wang. ICLR 2024.\n\n---\nOnce more, we thank the reviewer for the constructive feedback. Addressing the comments has improved the quality of our work. We look forward to further discussions and continued advancements in this evolving field. Please let us know if we can provide any additional information to further enhance your assessment of our work and potentially increase the score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPhCHJDMcc", "reviewer": "Reviewer_my5W", "summary": "The paper critiques the practices of privately adapting closed(-source) LLMs to private data by demonstrating that these techniques are potentially unsafe and do not yield the required quality of resulting model in terms of accuracy. The authors conclude that a focus on open LLMs should be preferable in sensitive fields.", "questions": "- What are the \"privacy units\" in this work? In other words, how are adjacent databases defined, and how is this selection justified?\n- Privacy agains the LLM provider is a strong argument. I'd be interested in some more discussion on the threat model also in terms of releasing a privately fine-tuned model and the types of attacks which are possible, e.g. MIA or reconstruction attacks, and how they would be carried out. To clarify: I am not asking you to perform any attacks.\n- As discussed above, your privacy budget evaluation tops out at 8. I would like to see some discussion on protecting e.g. against reconstruction attacks, which is feasible even with much higher privacy budgets. This should be part of the discussion on potential attacks, and I'd be interested to see when the performance between models converges to each-other and to the non-private performance \n- There don't seem to be mixture of experts models involved in your evaluation. Do you believe that this is a limitation of your work? Would you have anticipated similar results and why? I'd welcome a discussion on this point.\n- What interactions between privacy and safety did you observe? Do you anticipate that solving a privacy problem could introduce an alignment or safety problem in these models?", "limitations": "The authors discuss some limitations, such as cost and performance, which I appreciate. I would be interested (see Questions above) in a discussion on the interplay between privacy and other aspects of trustworthiness, especially safety and alignment.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I agree with both the motivations and the premise of this paper. It is important to think about adapting LLMs to private data holistically, i.e. on a systems level, and not just on the level of individual techniques. I like that the authors investigate a multitude of techniques, contrast open and closed LLMs, and discuss the costs of each method. The paper is also written clearly and includes a good overview of prior works. I also approve of the detailed exposition on hyperparameters, costs, etc. in the appendix.", "weaknesses": "I would have liked to see a more detailed investigation on the effect of privacy levels beyond $\\varepsilon=8$ (for most techniques excluding PATE, where there is a note on different privacy levels and the performance plateauing). Perhaps I missed this, but I didn't find a concrete justification for $\\varepsilon=8$ either, although I recognise this as a \"folklore/default\" privacy budget which many of us have come to think of \"automatically\" when working with DP-SGD due to the multitude of works which use it. \nMoreover, it would have been nice to see more dedicated instruction fine-tuning tasks, since this arguably has become one of the most important tasks for contemporary LLMs.\n\n On a more conceptual level: There is no \"methodological or theoretical novelty\" in this work in the traditional sense: It is a (thorough and thoughtful) comparison of techniques. I don't personally view this as a big weakness, to the contrary, I disapprove of the, usually fairly arbitrary, notions of \"novelty\" used to argue against papers, and I would like to see this paper published in some form, as I believe it discusses important points. However, I am unsure whether it would not have been a better fit to the benchmark track, rather than the main conference track.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough and thoughtful feedback, we appreciate that. We answer the concerns one by one below:\n\n>**I would have liked to see a more detailed investigation on the effect of privacy levels beyond $\\varepsilon=8$ (for most techniques excluding PATE, where there is a note on different privacy levels and the performance plateauing). Perhaps I missed this, but I didn't find a concrete justification for $\\varepsilon=8$ either, although I recognise this as a \"folklore/default\" privacy budget which many of us have come to think of \"automatically\" when working with DP-SGD due to the multitude of works which use it.**\n\nWe fully agree with the reviewer that we need *the investigation on the effect of privacy levels beyond $\\varepsilon=8$* and this is what we provided with our privacy-utility trade-off graphs in Figure 2 with $\\varepsilon$ ranging from 0 to 8 in the main paper for the classification tasks and the corresponding Figure 3 for the generation tasks (in the Appendix). We observe that across all the privacy budgets ($\\varepsilon \\in [0,8]$), the privacy-preserving adaptations for open LLMs perform significantly better than the ones for closed LLMs. \n\n>**Private instruction fine-tuning:**\n\nThe answer is provided in the general response. \n\n>**On a more conceptual level: There is no \"methodological or theoretical novelty\" in this work in the traditional sense: It is a (thorough and thoughtful) comparison of techniques. I don't personally view this as a big weakness, to the contrary, I disapprove of the, usually fairly arbitrary, notions of \"novelty\" used to argue against papers, and I would like to see this paper published in some form, as I believe it discusses important points. However, I am unsure whether it would not have been a better fit to the benchmark track, rather than the main conference track.**\n\nThis is a valid concern, and we agree with the reviewer’s perspective on novelty. We would like to note that our paper makes three main contributions, which we outlined at the end of the introduction section:\n\n1. **Conceptual Analysis**: We provide a comprehensive analysis of privacy-preserving adaptation methods for LLMs.\n\n2. **Benchmark Evaluation**: We thoroughly evaluate current approaches to private adaptations of LLMs, presenting a detailed benchmark study.\n\n3. **Proposed Methods**: We introduce PromptDPSGDGen and PromptPATEGen, new differentially private prompts designed for generation tasks. These methods achieve performance levels comparable to LoRA or fine-tuning, as noted in line 68 of our submission (due to formatting, this point might not have been clearly visible under Table 1). Detailed explanations of these techniques can be found in Section 3: “Prompt-based Private Adaptations for Text Generation.”\n\nWhile the benchmark is one of our primary contributions, we also want to highlight the significance of our conceptual analysis and new methods. We believe these contributions are crucial and justify our submission to the main conference. We hope our work can reach a broader community and further advance privacy-preserving methods for LLMs.\n\n>**Q1: Privacy units.**\n\nWe follow the previous work in this domain (e.g., [1,2,3]) and consider the instance-level DP. Two datasets are adjacent if and only if one can be obtained from the other by including an extra record [4] (this definition is used since we track the privacy loss through Rényi Differential Privacy or RDP [5]). For the classification tasks, each record consists of input sentences and a label to be predicted. For the generation tasks, such as the dialog summarization, the record consists of a dialog and its corresponding summary to be predicted.\n\n>**Q2: Threats against privately tuned models.**\n\n*We added the following to Section 2:* “Previous work [3] demonstrated that adapting an LLM with private data is vulnerable to MIAs. However, stronger privacy guarantees result in a lower MIA success rate [3]. The same trend is observed for reconstruction attacks [3]. \n\n>**Q3: Convergence between private and non-private training.**\n\nWe included extended experiments in Figures 1 and 2 in the attached PDF. Our observations indicate that adapting models with small $\\varepsilon \\in [0.3,1]$ results in significantly higher performance compared to 0-shot. Additionally, adaptations with $\\varepsilon=8$ achieve performance levels close to non-private adaptations.\n\n>**Q4: Evaluation on Mixture of Experts models**\n\nWe added the results using PrivateLoRA for Mistral-8x7B to the main paper. We observe similar trends for these models as for the other open LLMs (such as Llama3-8B).\n\n| Method | SST2| Trec | Mpqa | Disaster | Average |\n|-|-|-|-|-|-|\n| PrivateLoRA | 94.5  | 95.8 | 86.6 | 79.2 | 89.02 |\n\nSAMSum:\n    \n| Method | Rouge-1 | Rouge-2 | Rouge-L | T($) | Q($) | All($) |\n|-|-|-|-|-|-|-|\n| PrivateLoRA | 52.8$\\pm$0.4 | 29.6 $\\pm$ 0.2 | 44.7 $\\pm$ 0.2 | 57.96 | 9.99 | 67.95 |\n\nMIT-D and MIT-G\n\n| Method | MIT-D | MIT-G | T($) | Q($) | All($) |\n|-|-|-|-|-|-|\n| PrivateLoRA | 93.0  | 69.7  | 1.52 | 9.47 | 10.99  |\n\n\n>**Q5: Interplay of safety and privacy.**\n\nThe answer is provided in the general response. \n\n**References:**\n\n[1] “Large Language Models Can Be Strong Differentially Private Learners”. Xuechen Li, Florian Tramer, Percy Liang, Tatsunori Hashimoto. ICLR 2022.\n\n[2] “Differentially Private Fine-tuning of Language Models”. Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, Huishuai Zhang. ICLR 2022.\n\n[3] “Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models” Haonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska Boenisch. NeurIPS 2023.\n\n[4] “Rényi Differential Privacy of the Sampled Gaussian Mechanism.” Ilya Mironov, Kunal Talwar, Li Zhang. 2019.\n\n[5] “Rényi Differential Privacy”. Ilya Mironov. 2017 IEEE 30th Computer Security Foundations Symposium (CSF).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their valuable feedback and insightful comments, which greatly helped us further improve our submission. Our results were described as “striking” by Reviewer gG3r. This work contains “extensive experiments” (Reviewer k2x5), which demonstrate that adaptations of open LLMs outperform their closed counterparts “in the areas of privacy, cost-efficiency, and performance” (Reviewer 1UUG). Additionally, it was recognized that “the problem addressed in the paper is important” (Reviewer 1UUG) and that “this paper should be published” (Reviewer my5w). It is important to think about adapting LLMs to private data holistically, i.e. on a systems level, and not just on the level of individual techniques (Reviewer my5w). We hope that our work can significantly contribute to the community and reach a broader audience.

Below, we present new experimental findings that we believe are valuable to share with all reviewers. Following this, we provide individual responses addressing each reviewer's comments in detail.

>**Interplay of safety and privacy: What interactions between privacy and safety did you observe? Do you anticipate that solving a privacy problem could introduce an alignment or safety problem in these models?**

We conducted additional experiments to analyze how fine-tuning a downstream task with and without differential privacy affects the safety alignment of models. We followed the approach from [1] to evaluate Mixtral-8x7B-instruct and fine-tuned the model on SAMSum once with $\varepsilon=8$ and once with $\varepsilon=\infty$. We selected SAMSum as it does contain conversations with unsafe language (e.g., cursing or harassment). These results are presented in Table 1 in the attached PDF (and also shown below for convenience). The table is divided into two sections, “Compliance on Harmful Queries” and “Refusal on Harmless Queries”. The scores were generated by separately prompting our model with 100 harmful and 100 harmless queries, each repeated 20 times with different safety prompts. Finally, the outputs are categorized as complying or refusing the input by Llama Guard, giving the percentage of incorrectly handled answers in our table. Therefore, the lower the score, the better.

First, we observe that fine-tuning of any kind decreases the model's safety capability. This is easily identifiable in the table, as the original model has lower scores across the board for compliance with harmful behavior compared to both fine-tuned models. Similar results were also shown in [2] and [3], where the authors used a dataset containing unsafe samples to fine-tune safety-aligned models, which drastically increases compliance with unsafe behavior.

Second, **we can observe that private fine-tuning affects the safeguards less than non-private fine-tuning**. In the non-private case, the influence of individual samples is unrestricted, which can amplify the impact of unsafe samples, as shown in [2] and [3]. In contrast, differential privacy (DP) limits the influence of any single sample. Consequently, the impact of unsafe examples is minimized, resulting in a model that retains more of its safe behavior.

||Harmful|||||Harmless||||
|-|-|-|-|-|-|-|-|-|-|
||noprompt|default|mistral|short||noprompt|default|mistral|short|
|Basemodel|36|0|0|1||0|2|0|1|
|LoRA|64|39|30|38||0|3|2|2|
|PrivateLoRa|48|7|5|9||0|1|0|2|

>**Private instruction fine-tuning: it would have been nice to see more dedicated instruction fine-tuning tasks since this arguably has become one of the most important tasks for contemporary LLMs.**

Indeed, we also agree that instruction fine-tuning (IFT) tasks should be executed with privacy.

In general, users send instructions to an LLM and these instructions are collected by the LLM provider. The LLM provider later uses annotators to select and annotate (e.g., provide answers to) the instructions, which are subsequently used to further improve the instruction following capabilities of the LLM. The private instructions might leak to (1) the annotators and then to (2) the other users interacting with the LLM.

To evaluate the performance of IFT with privacy, we use the Pythia 1B model fine-tuned on the AlpaGasus-9k dataset [4] (annotated and filtered with an LLM). We use Private LoRA for the fine-tuning. The results are shown in Figure 3 of our attached PDF. We compare the model outputs against various instruction sets, where these outputs were scored by GPT4-Turbo. A model wins if its score is higher than the other model’s score, whereas they draw if they have the same score. We observe that while the model trained without privacy ($\varepsilon=\infty$) has more wins than the model trained with privacy ($\varepsilon=8$), the majority of outputs were scored equally between the two models. This indicates that **the instruction fine-tuning tasks can also be effectively addressed with privacy protection**.

**References:**

[1] “On Prompt-Driven Safeguarding for Large Language Models". Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, Nanyun Peng. ICML 2024.

[2] "LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B". Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish. ICLR 2024 Workshop.

[3] "Learning and Forgetting Unsafe Examples in Large Language Models". Jiachen Zhao, Zhun Deng, David Madras, James Zou, Mengye Ren. ICML 2024.

[4] “AlpaGasus: Training a Better Alpaca with Fewer Data”. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, Hongxia Jin. ICLR 2024.

--- 

Once again, we thank the reviewers for the constructive feedback. Addressing the concerns has improved the quality of our work. We look forward to further discussions and continued advancements in this evolving field. Please let us know if we can provide any additional information to further enhance your assessment of our work.

Meta-review: The paper offers a thoughtful critique of using closed-source LLMs to perform private learning, noting that most of these methods actually leak information to the LLM provider, and instead argues that fine-tuning an open-source LLM is preferable from both a privacy and utility perspective. All reviewers had positive things to say about the paper and appreciated its contributions.
Decision: Accept (poster)