Locally Differentially Private Document Generation Using  Zero Shot Prompting
Authors: Saiteja Utpala
Sara Hooker
Pin-Yu Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Models
Local Differential Privacy
Deanonymization Attacks
Zero Shot Prompting

Abstract: Numerous studies have highlighted the privacy risks associated with pretrained large language models. In contrast, our research offers a unique perspective by demonstrating that pretrained large language models can effectively contribute to privacy preservation. We propose a locally differentially private mechanism called DP-Prompt, which leverages the power of pretrained large language models and zero-shot prompting to counter author de-anonymization attacks while minimizing the impact on downstream utility. When DP-Prompt is used with a powerful language model like ChatGPT (gpt-3.5), we observe a notable reduction in the success rate of de-anonymization attacks, showing that it surpasses existing approaches by a considerable margin despite its simpler design. For instance, in the case of the IMDB dataset, DP-Prompt (with ChatGPT) perfectly recovers the clean sentiment F1 score while achieving a 46\% reduction in author identification F1 score against static attackers and a 26\% reduction against adaptive attackers. We conduct extensive experiments across six open-source large language models, ranging up to 7 billion parameters, to analyze various effects of the privacy-utility tradeoff. Code is avaliable at \url{https://github.com/SaitejaUtpala/dp_prompt}

Locally Differentially Private Document Generation Using
Zero Shot Prompting
Saiteja Utpala
Sara Hooker
Pin Yu Chen
Cohere For AI
Cohere For AI
IBM Research
saitejjauttall@@gaall com
sarahhoker@ccherr..com
pin-yu. wwwweihe.nccccceeem com
Abstract
Private Document (D)
Numerous studies have highlighted the pri- vacy risks associated with pretrained large lan- guage models. In contrast, our research of- fers a unique perspective by demonstrating that pretrained large language models can ef- fectively contribute to privacy preservation. We propose a locally differentially private mechanism called DP-Prompt, which lever- ages the power of pretrained large language models and zero-shot prompting counter au- thor de-anonymization attacks while minimiz- ing the impact on downstream utility. When DP-Prompt is used with powerful language model like ChatGPT (gpt-3.5), we observe notable reduction in the success rate de- anonymization attacks, showing that it sur- passes existing approaches by a considerable margin despite its simpler design. For instance, in the case of the IMDB dataset, DP-Prompt (with ChatGPT) perfectly recovers the clean sentiment F1 score while achieving 46% re- duction in author identification Fl score against static attackers and a 26% reduction against adaptive attackers. We conduct extensive periments across six open-source large lan- guage models, ranging up to billion parame- ters, analyze various effects of the privacy- utility tradeoff. Code is avaliable at https: //gi thub. com/Sai Sattatteaattpllllapppppppp
Sanitized Docunent (P)
Paraphrase Docunent:
Prompting Language Model at Tenperature
Figure Overview of the proposed DP-Prompt mecha- nism. Given private document (D), DP- DP-Prompt gen- erates a sanitized document (P) while ensuring local differential privacy. The level of privacy guarantee is controlled by adjusting the sampling temperature (T) during the decoding process.
2006). The incident unfolded when AOL mistak- enly released detailed search logs of their users, wrongly assuming that the data had been ade- quately anonymized through the use of random user IDs. Unfortunately, the released logs contained suf- ficient personally identifiable information, lcading to the identification of numerous individuals (Bar- baro and Jr., 2006; Jones et al., 2007). This breach of privacy triggered widespread public outery and led to the initiation of class action lawsuits.
1 Introduction
The vast amount of online data has the poten- tial to reveal numerous user attributes, making indi- viduals easily identifiable (Rao et al., 2000; Hovy et al., 2015; Preofiuc-Pietro et al., 2015). While pri- vate information can be directly disclosed through specific phrases in the text, it can also be implicitly inferred. For instance, linguistic patterns embedded within the text can inadvertently facilitate author- ship attribution (Kešelj al., 2003; Shrestha et al., 2017), leading to unintended privacy leakage. An illustrative real-world scenario the AOL search data leak August 2006 (Pass et al.,
This case is just one among many that highlights the limitations ad-hoc privacy approaches that may give the impression of providing privacy but ultimately fall short. Differential privacy (DP) pro- vides a rigorous treatment for the notion of data pri- vacy by providing plausible deniability by precisely quantifying the deviation in the model's 's output dis- tribution under modification of small number of data points (Dwork et al., 2006, 2014). The prov- able guarantecs offered by DP, coupled with its compelling properties such as immunity to arbi-

---

Placcy woliity of oe-herptteth Chelo7
Julby of Shforpt OuICHT)
as sanitized documents in a zero-shot manner (See Figure 1). Our motivation for this approach stems from two important factors. Firstly, recent research (Bevendorff et al., 2019; Mattern et al., 2022b) has shown that paraphrasing is a robust defense mecha- nism against deanonymization attacks. Secondly, growing evidence suggests that pretrained large language models can effectively tackle complex tasks without the need for task-specific and expen- sive fine-tuning (Brown et al., 2020; Chowdhery et al., 2022; Chung et al., 2022; Kojima et al., 2022; OpenAI, 2023), through zero-shot prompting. By harnessing the capabilities of pretrained large language models, DP-Prompt offers a straightfor- ward and powerful solution to mitigate the risk of deanonymization. It provides promising alter- native that can be widely applicable, particularly in the context of on-device large language models where text complction tasks require significantly fewer resources. We summarize the contributions as follows:
8.93
monds
S4- di-rrrptiklgtee 0.45
2Z
IMDB (embedding access) (b) Yelp (embedding access)
Firecy M2 Se-rerttteh Sheoom
Wars July d'0hhrret Jeb Oald!!)
3.50 BRGAR
30 Avwer EINETCINNI CORRE1NESS
(c) IMDB (text access)
(d) Yelp (text acccess)
Figure 2: Overview of the privacy-utility tradcoff with DP- DP-Prompt (using ChatGPT) for the IMDB and Yelp datasets, conducted at a temperature 1.5. The terms 'Static' and' 'Adaptive' refer to the attack models defined in Definition 2.
trary post-processing and graceful composability, have established it as the de facto standard for pri- vacy. DP has witnessed widespread adoption and numerous deployments in both private (Erlingsson et al., 2014; Apple, 2017; Near, 2018) and public organizations (Abowd, 2018).
We propose DP-Prompt, a new, simple, and computationally effective differentially pri- vate (DP) mechanism designed as a defense against de-anonymization attacks. DP-Prompt takes private document and generates para- phrased version using zero-shot prompting. The resulting paraphrased document is then released as sanitized document, as illustrated in Figure
To address the issue of deanonymization attacks, various approaches have been proposed within the DP framework. These approaches encompass word-level strategies (Feyisetan et al., 2020; Xu al., 2020; Carvalho et al., 2021) where noise is added at the word level, as well as sentence- techniques (Meehan et al., 2022) where noise is added at the sentence level. However, recent re- search by (Mattern et al., 2022b) has identified lim- itations in word-level approaches, particularly their disregard for contextual information. To overcome these limitations, Mattern introduced a mechanism that fine-tunes the GPT-2 model (Radford al., 2019) specifically for paraphrasing tasks, resulting in the generation of sanitized versions of docu- ments. While promising, the approach is limited by their reliance on annotated paraphrasing data, extensive computing resources for larger models, and the quality of annotations.
We demonstrate that DP- OP-Prompt, in combi- nation with ChatGPT (gpt-3.5), surpasses all current methods in terms of utility for any level of privacy. Our approach successfully recovers clean sentiment Fl score while sig- nificantly reducing the accuracy of author de- anonymization attacks. Refer to Figure 2 for an overview of these results.
To demonstrate the broad applicability of DP- prompt, We conduct extensive experiments with 6 open source models ranging upto 7 billion parameters to study the privacy-utiliy tradcoff.
2 Preliminaries
We propose DP- PPPPropp,, novel and straightfor- ward solution address deanonymization attacks. Our method leverages pretrained large language models by directly prompting them to generate paraphrases. These paraphrases are then released
A mechanism M : achieves e-PureDP if, for all inputs D, D' D that differ in one element, and for all V C Range(M), Pr [MM() E V] s exp Pr [M(D') E V] (Dwork et al., 2006).

---

Mechanism
Privacy Level
Requires fine-tuning Generates sanitized doc
Madlib (Feyisetan et al., 2020)
Word level Metric-DP
No
Yes
Mahunolhis (Xu et al., 2020)
Word Metrie-DP
Yes
TEM (Carvalho
2021)
Word level MetricDP
No
Yes
Truncated Laplace (Mochan et al. 2022)
Sentence level Pure-DP
No
No
Deep Candidate (Mechan et ol., 2022)
Sentence level Pure-DP
Yes
No
Paraphraser (Mattern et al., 2022b)
Document level Pure-LDP
Yes
Yes
DP Prompt (Ours)
Document level Pure-LDP
Yos
Table 1: We compare our proposed method, DP-Prompt, with related work on various factors. The "Privacy level" indicates the privacy guarantee provided by each mechanism. "Fine-tuning" denotes whether the mechanism involves fine-tuning model as an intermediate step. The last column, "Generates sanitized doc," indicates whether the mechanism can output a fully sanitized document instead of just sanitized embeddings.
Metric Differential Privacy (Metric-DP) (Andrés al., 2013; Chatzikokolakis et al., 2013) is a relax- ation of Pure-DP that applies to data represented in a general metric space. For a given distance met- ric : D x D + R.,, a mechanism . : D + V achieves ed-MetricDP if, for any D, E D and for all V c Rang Pr [MM(DD E V
Static Attacker with Embedding Access: Has access to clean documents (Di...., Dr) but lacks access to sanitized versions (P....,, Pn).
Static Attacker with Text Access Doesn't 't have access to sanitized embeddings (N1,..., N.) but only to the clean embed- dings (El,...,En)
exp (d(D, (M()) c V].
Local differential privacy (LDP) (Ka- siviswanathan et al., 2011; Duchi al., 2013; Xiong al., 2020) a privacy framework where data is locally perturbed before transmission, considering the presence an untrusted data collector or server. The formal definition of LDP is as follows:
Adaptive Attacker with Embedding Ac- cess Has access to sanitized embed- dings (N.)...,Nn). Hence, trains de- anonymization model to adapt to the DP mech- anism Membedding
Adaptive Attacker with Text Access: Has access to sanitized text (Pi...., Pa). Conse- quently, trains a de-anonymization model to adapt to the DP mechanism Mixx-
Definition (PureLDP). A randomized mechanism : -> V>V is said to be c-PureL.DP for any pair of inputs D,D' E DD and for all V S (Mnge(MM)
Pr(M(D) E V] < exp (c)Pr((((') V].
It is important to note that the adaptive attacker is more formidable adversary since it adapts the characteristics of the mechanism M, whereas the static attacker only has access to clean docu- ments/clean embeddings without any added noise. The mode of access-either raw text or abstracted embeddings- offers further nuances, determining the exact nature of the data an attacker can exploit.
There is a growing consensus that, despite the assurance of formal guarantees, it is imperative to subject differentially private mechanisms to robust privacy attacks that simulate strong and malicious adversaries (Jayaraman and Evans, 2019; Blanco- Justicia al., 2022). Such evaluation allows to effectively assess the empirical privacy provided by the mechanism in real-world scenario. To this end we define four attack models depending its adaptivity and mode of access.
3 DP Prompt
Language models use decoder network to gen- crate sequential text output. Given a context or prompt represented by a sequence of tokens C (cI....,Cm) the language model generates text by sampling tokens from a conditional distribution Prie(41..... In this distribution, the logits RP are trans- formed using the softmax function with temper-
Definition 2 (Attack Models). Consider a col- lection of private documents (D1,..., Da) from distribution D with associated author identities (a1...., ,nn) and embeddings (FI,..., F.)) ~ E. For text-to-text sanitization using mechanism Miext, the sanitized documents are represented as (PI....,PA) P For texi-to-embeeding sanitization via mechanism Membedding. the sani- tized embeddings are denoted as (N1...,,Nn)
= IT (2) 2;-1).
exp( "1 and V repre-
ature T, where Pij = Σ) exp( "py"" sents the vocabulary.
N Menneetting

---

2-13 cooktail 5a appetizer with get this portabllio Triss" so ladiy Exper lent food service and one-olia kind atmosphore Parfoct for date romantl direer for Two Groet proposel or spedial occaision IneiiinEEnppee Expurse accourt Buy can they thow downiN B. cantt get any ool prono service up here, Ch
thorought
Cush
Viww
Madi bxjepestyy anayed over casual hour move theroupphty celebratedd 501 normllly 514) cockiel, $3 each aoda ouch secept lived rumber seaved luna disoert with get it heree 'hiss so ladty: in socolllent lood and service and see- teeothooughhhh atmocphheee howeuse pertect lor cate scmecne commantcc dinner two socae great propossel special turn. boy can throw dowett gatting right call piazne servicative here.
Pargphrosenggge xXTT-277))
up
be consemed.
OBt-PronetCChettPiiy TJTT1 While conmot
phone lecation,
portabell
"Iree" was de delectatle and the 58 happy hour cooktall was extreinely enioyable, Fair warning though. the dub soda ocst 26 The food and cenvice were exceptionnll and the atmospherm unique. This in sprencid estitiiamment rendervouss opecaal ocoesion, on, and quis exper appeenkos, mestong miting corporste expence
Figure 3: Sample illustration clean and sanitized documents for various mechanisms. For ChatGPT, prompt is "Review[[review] Parrphhre of the Review:" where review is the clean review.
This process of sequentially generating text can be regarded as a problem of selecting tokens at each step. Hence, to make the generation step diffcrentially private, one must replace it with a dif- ferentially private version of the selection process. One commonly used and well-known differentially private mechanism is the exponential mechanism (McSherry and Talwar, 2007), which is defined as follows:
Algorithm 1: DP-DDPPPPoot OP-Prompt
Input: language model (LMM), private document (D),
prompt template (T), clipping vector b E R RM temperature T E R4, paraphrase
tokens n.
Output: Sanitized Doc (P)
P P D. < GeneratePrompt Cukens Tokenize(C) for
do
4
(Crizen))
5
u' + CLipAndScale(u, b,
Definition 3 (Exponential Mechanism). Given an utility function u D x V -> V. The ex- ponential mechanism MExp D + V is randomized algorithm with output distribution
p ConvertToProbbillitiissd"" SampleToken(p)
P PU PU(e). [o). Cikees
Cakers U [o]
end
Detokenize(P) return
P [[MExp(D) = x exp cu(D.o) where Au- 2Au
maxD,D'. _ (u( u(D',)) is sensitivity.
posed DP-Prompt, and the code for implementing DP-Prompt in HuggingFace (Wolf et al., 2019) is provided Appendix B. The formal guarantee achieving e-PureLDP is provided by the following theorem:
In our case, the utility of token ey E V each step i is simply the logit U4j R. Hence, one can make text generation differentially private using the exponential mechanism.
Extensive research has shown that paraphras- ing documents helps conceal author identity (Rao et al.. 2000; Bevendorff et al., 2019; Mattern et al.. 2022b). Considering recent advancements where tasks are formulated prompts and language mod- els are tasked complete them (Raffel et al., 2020; Brown et al., 2020: Wei et al., 2022), we directly prompt the language model to generate paraphrases. Therefore, given private document D and a specific prompt template instructing the language model to generate paraphrase, such as T : "Paraphrase of the document:" we combine D and to create context C. By utilizing this context, execute the text generation procedure in a differentially private manner to produce a para- phrase. We refer to this procedure as DP-Pront -p-Prompt. Algorithm outlines the specific steps of our pro-
Theorem Suppose the language model has not been pretrained on the private documents distri- bution D. If the final logits u E RRV satisfy the condition by I u, S b2, Vi, and the DP-Prompt run with temperature T for generating n lokens, then it can be proven that the generated output satisfies (2m(b2 b))T))LDDP
See the Appendix for the proof.
Experiments
4.1 Experiment Setup
Evaluation: Note that we comparing DP- mechanisms with different levels of differential privacy. Therefore, in our experiments, we focus on evaluating the empirical privacy rather than the theoretical privacy(c) for effective and realistic as-

---

Juther casin 21 Scre skiincee
JATT Mertitati 13 Seai Halry
Auther swerlcation Sere prakis
Jutter taree caton Siam intidee)
(a) IMDB (static)
(b) IMDB (adaptive)
(c) Yelp (static)
(d) Yelp (adaptive)
2Hd 8-EaTh
hercen
Marta
Maille
La
Lin
Author conersssaaccsonn scom iktrace
AuTTE Meriicadoo 21 Scare imary
Acher awrrirrn Sene Praast
luthe isndi caton 11 Sam tiiree
(c) IMDB (static)
(f) IMDB (udaptive)
(g) Yelp (static)
(h) Yelp (adaptive)
Figure 4: Comparison of DP- DP-Prompt (with ChatGPT) with various baselines. The top row shows results for attacker with embedding access, while the row below presents results for an attacker with text access. Notably, it is evident that regardless of the chosen privacy level, DP-Prompt, when utilized with ChaiGPT (GPT-3.5), exhibits significantly better utility compared to all baseline mechanisms.
sessment. As a result, we plot the author identifi- cation Fl score, which is calculated by conducting de-anonymization attacks on the sanitized docu- ments. This score indicates the potential for privacy breaches. On the other hand, the y-axis represents the sentiment Fl score, which measures the utility of the sanitized documents.
(2,5,8, 17, 25)
For cach of sentence level mechanisms Fruncated-Laplart (Meehan et al., 2022), Deep-Candidate (Meehan al., 2022)): we run the mechanisms for 11 é's given by (5, 10, 20,30, 10, 50, 75, 100, 150, 200).
Datasets: We conduct experiments using IMDB movie reviews and Yelp business reviews, both of which contain author and sentiment labels. The IMDB dataset has a size of 15,000, while the Yelp dataset has 17,336 samples. For both datasets, sen- timent analysis is a 2-class classification task, and the author identification task is a 10-class classifi- cation task.
For Paraphraser (Mattern et al.. 2022b) and DP-Prompt with open source mod- els we run decoding at temperatures {0.75, 1.0, 1.25, 1.5, 1.75). For DP- Prompt we run ChalGPT at temperatures {1.0, 1.25, 1.5, 1.75, 2.0).
Further we also consider Fl scores on Clean (without noise added) embeddingssdoocuments and performance of uniformly random classifier (for more details, refer to Appendix C.6).
Implementation Details: For the embedding-level attacker, we utilize 3-Layer MLPs with ReLU acti- vation functions and train them on sentence embed- dings (Reimers and Gurevych, 2019). For the text- level attacker, we fine-tune BERT (Devlin et al., 2018). More details can be found in Appendix C. Regarding the static attacker, the clean set of doc- uments is used for training and validation, while the sanitized documents serve the test set. On the other hand, for the adaptive attacker, all three sets (training, validation, and testing) consist of sanitized documents.
4.2 DP-prompt with ChatGPT (gpt-3.5)
In this section we compare 6 baselines (Madlib, Mahalanobis, Tem, Truncated-laplace, Deep- candidate, Paraphraser) run with configurations above (for more details also refer to Appendix C) with DP- DP-Prompt with ChatGPT. Except for DP- Prompt, we run each mechanism to 3 times to pro- duce 3 different sanitized documents and plot mean author F1 identification score on x-axis and show 2a band around mean sentiment F1 score. Results are show Figure 4
For each of word level mechanisms, (Madlib (Feyisetan et al., 2020), Mahalanobis (Xu et al., 2020), TEM (Carvalho et al., 2021)) we run the mechanisms for 8 e's given e
The results clearly demonstrate the superior per- formance of DP-Prompt with ChatGPT (GPT-3.5).

---

Obee
Cree
Oue
One
15/88
Auchor Mon Muntiaation PI Sore (hivool
Authar loontiizzionn Score (Prlaayy
Author Menit caten Store Hayl
Authar Meniiiiaton PL Score (Piiaac))
(a) IMDB (static)
(b) IMDB (adaptive)
(c) Yelp (static)
(d) Yelp (adaptive)
Swntimet St00 100
Swntteen Suore
Dine
Autter Mentitiatinn Fl Sooe detvooa
Sather lientiiction F1 Score (Prvaay)
Auttor Mentt Catentiation catos F3 Store 2e vayy
Authr Mestifition Fi Score detvacci
(c) IMDB (static)
(f) IMDB (udaptive)
(g) Yelp (static)
(h) Yelp (adaptive)
Figure 5: Illustration of privacy-utiliyy uradeoff in DP-Prompt with open source models and ChatGPT(gpt-355), The top row shows results for an attacker with emhedding access, while the row below presents results for an attacker with text access.
Notably, DP-Prompt exhibits significantly higher utility on the y-axis for a chosen empirical privacy value on the x-axis. All word-level mechanisms show similar privacy-utility tradeoff. Regarding sentence-level mechanisms, the truncated Laplace mechanism performs decently, while in the static attack experiments. Deep-candidate is reduced to a random classifier due to the distribution shift caused by sentence recoding.
through APIs, necessitating the uploading of user documents to the language model provider. Al- though DP-Prompt with such proprietary models provide LDP guarantee, this defeats the fundamen- tal motivation of LDP, which is achieve privacy guaramtees without relying on trusted data cura- tor. The objective the experiments presented the preceding section aim to demonstrate that DP- Prompt, when combined with a powerful language model like ChatGPT, can outperform existing meth- ods by significant margin.
Furthermore, in the case of clean reviews (i.e., without any noise), the embedding-lewe attacker can accurately identify the author among 10 differ- ent options with a high Fl score of 0.93 IMDB and 0.86 in Yelp. However, when DP-Prompt is em- ployed, the sentiment Fl scores remain unchanged, while the author identification scores decrease by 46% and 25% in the case of IMDB, and 53% and 29% in the case of Yelp.
Considering the increasing interest in building high- High-quulity open-soures large language models (Scao et al., 2022; Black et al., 2022: Touvron et al., 2023; Li et al., 2023; Jiang et al., 2023), we expand our evaluation of DP-Prompt to include six open- source models, ranging in size up to seven billion parameters. Our evaluation takes into account two factors: (i) architecture, where consider both encoder-decoder and decoder-only models, and (ii) the level of fine-tuning, including models that are fine-tuned using instructions and/or Reinforcement Learning with Human Feedback (RLHF). Specifi- cally models are as follows, Base: T5 (3b), Stable Im base (3b, Instrrcctt finetuned/RLiiF tuned: Flan T5 (3b), Stable Im tuned (3b,7b).
The text-level models are more accurate than the embedding-level models, with author identification scores of 0.99 (as opposed to 0.93) and 0.97 (as opposed to 0.86) in IMDB and Yelp, respectively, for clean reviews. When DP-Prompt is employed, the sentiment Fl scores remain unchanged, while the author identification scores decrease by 54% and 10% in the case of IMDB, and 73% and 24% in the case of Yelp. This illustrates that text-level attackers are more powerful.
While T5, Flan T5 are encoder-deeoder models, rest of the models are decoder-only. In contrast DP-Prompt with ChatGPT, we perform DP-Prompt using the aforementioned open source language models three times for each dataset and tempera- ture, resulting in three sanitized documents. Fur-
4.3 DP-Prompt with open source models
It is important to note that models such as ChatGPT (gpt-3.5) are proprietary and can only be accessed

---

Data
Tdp
Sndiawe
Suther Mentiaatien ScoP
Sesterent SOAT
Nather Divalfortion
diyping
L35
622
18
8:29
0.79
18
1.3
1.
1,29
15
125
038
123
(4007)
(4366)
(4000)
(40007)
(+380)
Adspiire
635
103
023
100
025
136
0311
105
033
105
0322
120
122
120
101
Atterker
(48201
DY1
14000)
103
(4420)
835
1-001
6.31
14001)
133
(9914)
0.8
(4W0))
(48.0))
810
140MM)
08:3
(H4.00)
1111
19060
140001
0.00
(4200
1001
140090
1337
+4.00
1:20
(40M))
033
(H4220)
(45M0)
011
811
0.10
Salidar
Itand
Stalk
Nva
DA1
103
635
035
0.3
(1002)
0.34
035
(9904)
150
(4007)
0.2
012
1625
010
0251
1000
0.2
028
155
1639
(100))
0.43
146
1455
6.0
01.
018
0i6
100
(40009
(+3.6) (4000)
(46.6))
(+0.00
Table 2: Effect of clipping (shown as Yes) and without clipping (shown as No) on privacy-utillin tradeoff.
ther we all run these open source models at half precision (torch.floatl float and with max number of tokens in paraphrase to 150. The obtained results are presented in Figure 5.
ily because ChatGPT doesn't expose logits, and for a fair comparison with ChatGPT, we didn't clip log- its for open-source models in Section 4.3. However, the LDP guarantee still holds because, in practice, logits are always bounded within a certain preci- sion, even though they may have large values.
Now we compare open source models along var- ious factors
Base vs Instruction/RLHr tuned: It is evident that the base models perform poorly in compari- to the models that underwent instruction fine- tuning/RLHF tuning. One important point to men- tion is that both StableI.M M-seedd (3b, 7b) perform significantly worse against adaptive attackers.
In this section, we examine the impact of logit clipping the tradcoff between privacy and utility using FlanTT((3b) and Stablelm Tuned(7b) models against embedding
attacks. We adopt the ap- proach of (Li and Clifton, 2021) by learning the clipping boundarics on additional data (more de- tails can be found in Appendix C). The results of this analysis are presented in Table 2. The max- imum difference observed with and without clip- ping occurs for Stablelm-Tuune (Tanell(m)T on the IMDB dataset at a temperature of 0.75. In this case, the sentiment Fl score drops from 0.71 to 0.65, and the author identification Fl score drops from 0.53 0.45. Based on these findings, we recommend us- ing logit clipping when higher privacy is required and not using clipping when higher utility is de- sired.
Scale: We observe that scale plays a key role, as StableLM-tuned (7b) demonstrates better utility than MaableLMMTunnel (3b) across all levels of em- pirical privacy, with ChatGPT outperforming both. Variance: It is worth noting that did not obtain variance for DP-Prompt with ChatGPT in Figure 4 through multiple runs. However, our analysis Figure 5 suggests that there is no significant variance in the Sentiment FI score, at least for open source models.
Encoder-Decoder / Decoder only: Flan- (33b),,,3),, encoder-decoder model, outperforms the larger GubblelmTTunee (7b) model. Notably, Flan-T5(3b) is fine-tuned exclusively academic datasets, resulting in shorter paraphrases compared to Stablelm-Tuned (7b).
4.5 Effect of Top-K sampling
For the differential privacy guarantee to hold, sam- pling from probabilities must be done over the entire vocabulary according to score probabilities. While it is common to top-k sampling in prac- tice to improve generation (the default value in Hugging Face (Wolf al., 2019) is 40). im- portant to note that the ChatGPT chat completion APT does not include a top-k parameter. this section, using open-source models, we examine the impact of top-k sampling on utility and investigate whether it provides any empirical privacy, even in the absence of the differential privacy guarantee. In addition, we aim assess whether top-k sampling can effectively help open source models to recover the clean sentiment F1 Fi-score and narrow the gap compared to DP-Prompt when used with ChatGPT. We consider Flan-T5(3b) and Stablelm-
ChatGPT vs Rest: While the source models we considered demonstrate competitiveness with ChatGPT, a notable gap remains. The key finding is that even at a higher temperature of 1.5, Chat- GPT is capable of recovering clean sentiment F1 score, while none of the open-source models can achieve matching clean sentiment Fl score, even at significantly lower temperature of 0.75. See Figure 7, which presents the paraphrases output by ChatGPT and Stablelm-Ton Tuned (7B).
4.4 Effect of clipping logits
both sections (Section& 4.2 and Section 4.3) DP- Prompt is run without logit clipping. This is primar-

---

Dus
Mrtar
Sestireat
Aather Dertllodee TOOO
lep
0229
1a
1,29
15
1.25
675
10
122
1.5
35
175
10:5
Pl
1:35
371
0775
0.35
935
1335
6.1
1324
0.5
C1T
(-205)
(40220
14020
(0022)
(4022)
142122
160.27
140150
(40.02)
140059
145.555
(46155)
142155
1-01
140,0
140.00
Mbppii
Mitcker
21
1B
(:000)
1.76
(40.20
3:10
3m
10010
0,9
1811
100 131
130
0/6
10013)
11
0.53
(4.00
10.0
1039
(4005)
0.2
0.33
1119
0.29
10
140.00
0.31
111
2-0115
18.55
11:11
100.001
0.67
676
140000
0.55
1200
(10.00
135
(1005)
0.60
10311)
1600
100.7
031
1:005
1134
110.01
023
(100%)
0.30
11
200 11
6.20
1025
332
0.20
3:11
0.23
0366
088
0.35
031
0.2
6.23
240.15
Teanl
(11)
(+bm):
[+50)
1689
110.00
(40.00
3,67
24607)
[400m)
1856
111
140.200
(40.50
140
(40.53
0.34
1-3.10
14.
01:31
12
(40.30
140
9.21
211
140.20
(40.80
11.27
(10.00)
(44.00)
0.60
(40001)
11006:
0.89
(+0.00)
143.00
(40155)
163.16
140.01
0.23
+40000
1-0005
16:22
(10.7)
(41.00
0.30
(1011)
(4000)
0.3
(40.1)
(40.00
DJR
Adgett
Aitckkr
48
6.30
16.0
141.01
3,67
374
14006)
0.7
0,39
(40
0.55
176
111
14017
0.23
0.51
40.00
1055
1455
0.15
0.66
1-3
11.31
0.35
190.7%
3.13
1:00
-0.0
0.35
01.04
(41
0.6
H40%%
035
0.24
14.00
0.50
106
16
140000
0.55
0.53
1-310
1441
0/00
140.00
0.41
H-00
0.35
01.3
(4)
0.3
022
HI15
0.
0.3
Hel
330
0443
7.42
0.97
0.77
0.2
Table 3: The effect of t sampling on privacy-utility tradeoff on Flan-T5(3b) and Stableln-Tunned(75 models
T7Be models and perform decoding with top-k sampling for & = 80). The results against embedding level attacks, including no top-k sampling, are shown in Table 3. The maximum difference observed with and without clipping occurs for Shabeelm-Tune((... dataset at a temperature of 0.7 for k 10. In this case, the sentiment Fl score increases from 0.32 to 0.64, and the author identification Fl score increases from 0.05 to 0.27. Additionally, top-k sampling does not improve utility at a temperature of 0.75, indicating that DP-Prompt with open- source models, even with top-k sampling, does not match DP- Promptt with ChatGPT.
paraphrasing data available for the IMDB and Yelp datasets. Therefore, the language models used in DP-Prompt are not explicitly fine-tuned for para- phrasing tasks related to these datasets. Conse- quently, the results obtained in Section 4 are likely to generalize to large extent.
on the IMDB
Our evaluation methodology goes beyond relying solely on the Sentiment F1 score (Utility) or even comparing sentiment Fl score (Utility) with e (theoretical pri- vacy) as presented in (Meehan et al., 2022). Instead, we adopt a comprehensive approach that considers the merit of our proposed approach based on the sentiment Fl score in conjunction with empirical privacy demonstration against de- de-anonymization attacks. Simply copying reviews verbatim would yield a high author identification F1 score, failing to provide empirical privacy. Additionally, we con- sider not only static attack models but also stronger ones like adaptive attackers. Superficially alter- ing reviews may deceive static attackers but would likely fail against adaptive attackers.
Robustness of Evaluation Setup:
5 Issue of Data Memorization
It is important to acknowledge the possibility that ChatGPT and other open-source language mod- els (LLMs) may have been exposed to online re- view datasets such as IMDB and Yelp. This ex- posure is not unique to our approach. Similar questions can be raised for word-level approaches (Feyisetan et al., 2020; Xu et al., 2020; Carvalho et al., 2021) and sentence-level approaches (Mee- han et al., 2022) since Ve embeddings (Penning- ton et al., 2014) and Sentence Transformers (BERT) (Reimers and Gurevych, 2019) are pretrained on the Common Crawl dataset. Hence assumption in Theorem may not hold. We argue that this expo- sure does not pose a major concern for DP-Prompt due to the following reasons.
6 Related Work
The previous work releasing private documents can be categorized into three approaches based on the level at which noise is added (See Table for concise summary). These approaches are: Word-level Approaches: MadLib (Feyisetan et al., 2020) is a word-level mechanism that applies Laplace noise to word embeddings and maps them to the nearest word in the vocabulary, demon- strating the differential privacy (DP) guarantee of MadLib under the Euclidean metric. An extension of this approach involves using a regularized Maha- lanobis metric instead (Xu et al., 2020). In contrast, the TEM mechanism utilizes the exponential mech- anism to transform the privatization step into a
No availability of paraphrases of IMDB, Yelp: DP- OP-Prompt essentially evaluates the performance of LLMs in the task of zero-shot paraphrasing at higher temperatures. Although the language mod- els may have been fine-tuned using open annotated paraphrase datasets (Zhang et al., 2019), it's im- portant to note that there is no specific annotated

---

selection problem (Carvalho et al., 2021). Further- more, there is recent development known Cus- Text (Chen al., 2023), which focuses on develop- ing customized mapping mechanisms for each in- dividual word in the vocabulary (Chen et al., 2023). All of these approaches are word-level mechanisms and have been shown to have significant limitations, such their disregard for contextual information (Mattern al., 2022b).
goal to generate sanitized documents from pri- vate documents using pretrained language models.
7 Conclusion
This paper introduces DP-Prompt, a locally dif- ferentially mechanism called DP-Prompt that gen- erates sanitized versions of private documents by prompting large language models to generate para- phrases. Notably, our method offers simpler ap- proach compared to existing methods. Through extensive experiments, show that our approach achieves significantly improved utility compared to current methods for any required level of privacy. As the demand for on-device large language models (LLMs) continues to grow, our method emerges as reliable safeguard for users' privacy and provides robust defense against de-anonymization attacks.
Sentence-level Approaches: Sentence-level mech- anisms based Sentence Transformer (Reimers and Gurevych, 2019) were introduced (Meehan et al., 2022). They proposed two approaches: one approach where noise is added to sentence em- beddings, and another more complicated approach based on maximizing Tukey depth (Tukey, 1975; Gilad-Bachrach and Burges, 2012).
Document-level Approaches: A document-level Local Differential Privacy (LDP) mechanism was introduced, where GPT-2 is fine-tuned for para- phrasing task (Mattern et al., 2022b). Our approach, DP-Prompt, draws inspiration from their work, but instead of insource-intensive fine-tuning, we use a zero-shot approach with pretrained models for efficient and effective generation of sanitized doc- uments. Furthermore, the recently proposed DP- BART (Igamberdiev and Habernal, 2023) employs BART (Lewis et al., 2020), an encoder-decoder model. In DP-BART, noise added the en- coder's output, and the decoder fine-tuned to adapt to this noisy encoder output.
Limitations
In our study, explored the initial step of harness- ing large language models and zero-shot prompt- ing to generate sanitized documents. While this approach effectively conceals the specific writing style of authors, there still a potential risk re- vealing explicit personal information, such as zip codes, bank details, or gender, especially when naively prompting a language model. This risk is particularly relevant in alternative text formats like messages emails compared to online reviews. For future work, an important direction would be to define a set of sensitive attributes and di- rectly prompt the language model to replace these attributes with the identifier "X" while generating paraphrases. This approach would help improve the safeguarding of personal information. Addi- tionally, it would be worthwhile investigate the potential side effects of hallucination and the im- pact of different prompt templates on the genera- tion of paraphrases, specifically within the context of the grivacy-utiiity tradcoff. Additionally, more robust attacks that measure privacy leakage at the text level should be explored.
Adversarial Methods: Parallel to differentially private approaches, other techniques have been proposed that utilize Adversarial Learning (Shetty et al., 2018; Quiring et al., 2019) and Data Poison- ing (Wang et al., 2022; Jin et al., 2020). However, these methods typically require access to surro- gate classifier. In contrast, our method is zero-shot, requiring neither fine-tuning nor access to a classi- fier.
Differentially Private Training/Fine Tuning:
There is extensive research on differentially private training or fine-tuning of language models (Kerri- gan et al., 2020; Li et al., 2021; Yu et al., 2021; Anil et al., 2022; Mattern et al., 2022a). They aim to make language models resistant to various kinds of data leakage attacks (Carlini et al.. 2019, 2021; Deng et al., 2021: Balunovic al., 2022). It is important to emphasize that this line of work completely distinct from our own, as focuses on training language models on private data, while our
References
John M Abowd. 2018 The US census bureau adopts differential privacy. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2867-2867.
Miguel Andrés, Nicolás Bordenabe, Konstantinos
Chatzikokolakis, and Catuscia Palamidessi. 2013.

---

troo-nndistinguisabbllty: Differential privacy for Docation-base systems. In Proceedings of the 2013 ACM SIGSAC conference Computer & communi- cations security, pages 901-914,
using metrics. In Privacy Enhancing Tèchnologies: 13th International Symposium, PETS 2013, Bloom- ington, IN, USA, July 10-12, 2013. Proceedings 13, pages 82-102. Springer.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2022. Large-scale differen- tially private bert. In Findings of the Association for Computational Linguisties: EMNLP 2022. pages 6481-6491.
Sai Chen, Fengran Mo. Yanhao Wang, Cen Chen, Jian- Yun Nie. Chengyu Wang, and Jamie Cui. 2023, A customized text sanitization mechanism with differ- ential privacy. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5747- 5758.
D Apple. 2017. Learning with privacy at scale. Apple Machine Learning Journal, 1(8).
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 2xXiv22044023111.
Mislav Balunovic, Dimitar Iliev Dimitrow, Nikola Jo- vanović, and Martin Vechev. 2022. Lamp: Extracting text from gradients with language model priors. In Advances in Neural Information Processing Systems.
Michael Barbaro and Tom Zeller Jr. 2006. A face is exposed for aol searcher no. 4417749. New York Times.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahmaa al. 2022. Scaling esstruction-fieetumee language models. arXiv preprint 22 1416.
Janek Bevendorff, Martin Potthast, Matthias Hagen, and Benno Stein. 2019. Heuristic authorship obfus- cation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1098-1108.
Jieren Deng, Yijue Wang, Ji L., Chenghong Wang, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Cai- wen Ding. 2021. Tag: Gradient attack on transformer- based language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, pages 3600-3610.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy. Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20bs 20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06744 067445...
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint 10Xiv.1810.04805.
Alberto Blancc-Justicaa, David Sánchez, Josep Domingo-Ferrer, and Krishnamurty Muralidhar. 2022. A critical review on the use (and misuse) of differential privacy in machine learning. ACM Computing Surveys, 1568):1-16.
John Duchi, Michael Jordan, and Martin Wain- wright. 2013. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 429-438. IEEE
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitiv- ity in private data analysis. In Theory of Cryptog- raphy: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3. pages 265-284. Springer.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Eval- uating and testing unintended memorization in neural networks. In USENIX Security Symposium, volume 267.
Cynthia Dwork, Aaron Roth, et al. 2014. The algorith- mic foundations differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3 4):211-4077
Nicholas Carlini, Florian Tramer, Eric Wallace, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. 2014. Rappor: Randomized aggregatable privacy- preserving ordinal response. In Proceedings of the ACM SIGSAC conference computer and com- munications security, pages 1054-1067.
Ricardo Silva Carvalho, Theodore Vasiloudis, and Oluwaseyi Feyisetan. 2021. Tem: high utility met- rie differential privacy on text. arXiv preprint
arXiv:2107.07928.
Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. 2020. Privacy-and utility-preservve tex- tual analysis via calibrated multivariate perturbations. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 178-186.
Konstantinos Chatzikokolakis, Miguel E Andrés, Nicolás Emilio Bordenabe, and Catuscia Palamidessi. 2013. Broadening the scope of differential privacy

---

Ran Baladd-Bachhchhaaa and Chris J.C. Burges. 2012. The median hypothesis. Technical Report MSR-TR- 2012-56.
Denoising sequence-to-sen -o-suuuncc pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880
Dirk Hovy, Anders Johannsen, and Anders Søgaard. 2015. User review sites as resource for large-scale sociolinguistic studies. In Proceedings of the 24th international conference on World Wide Web, pages 461.
Tao Li and Chris Clifion. 2021. Differentially pri- vate imaging via latent space manipulation. arXiv preprint 2rxiv22033.05422
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. International Conference on Learning Representations.
Timour Igamberdiev and Ivan Habernal. 2023. DP-
BART for privatized text rewriting under local dif- ferential privacy. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13914- 13934, Toronto, Canada. Association for Computa- tional Linguisties.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023, Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05:: 230.0463...
Bargay Jayaraman and David Evans. 2019. Evaluating differentially private machine learning in practice. In USENIX Security Symposium.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1
Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample. Lucile Saulnier, et al. 2023, Mistral 7b, arXiv preprint urXiv:2310.0688
Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. 2022a. Differentially private language models for secure data sharing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. bert really robust? a strong base- line for natural language attack on text classification and entailment. In Proceedings of the AAAI con- ference on artificial intelligence. volume 34, pages
Justus Mattemn Benjamin Weggenmann, and Florian Kerschbaum. 2022b. The limits of word level dif- ferential privacy. In Findings of the Association for Computational Linguistics: 2022,,2022, pages 867- 881.
Rosie Jones, Ravi Kumar, Bo Pang, and Andrew Tomkins. 2007. know what you did last summer: query logs and user privacy. In Proceedings of the sixteenth ACM conference on Conference on infor- mation and knowledge management, pages 909-914.
Frank McSherry and Kunal Talwar. 2007. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science 07), pages 94 103. IEEE.
Shiva Prasad Kasiviswanathan, Homin K Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. 2011. What can we learn privately? SIAM Jour- nal on Computing, 40(3):793-826
Casey Meehan, Khalil Mrini, and Kamalika Chaudhuri. 2022. Sentence-level privacy for document embed- dings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 3367-3380.
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. 2020. Differentially private language models benefit from public pre-traiming In Proceedings of the Second Workshop on Privacy in NLP, pages 39-45.
Joe Near. 2018. Differential privacy at scale: Uber and Berkeley collaboration. In Enigma 2018 (Enigma 2018).
Vlado Kešelj, Fuchun Peng, Nick Cercone, and Calvin Thomas. 2003. N-gram-based based author profiles for authorship attribution. In Proceedings of the confer- ence pacific association for computational linguistics. PACLING, volume 3, pages 255-264.
OpenAl. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.0877
Greg Pass, Abdur Chowdhury, and Cayley Torgeson. 2006. A picture of search. In Proceedings of the 1t international conference Scalable information systems, pages 1-cs.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart:

---

Luiza Pozzobon. Beyza Ermis, Patrick Lewis, and Sara Hooker. 2023. On the challenges of using black- box apis for toxicity evaluation in research. arXiv preprint 2tt:::2304.1399..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13977.
Daniel Prettjuc-Pietro, Vasilcios Lampos, and Nikolaos Aletras. 2015. An analysis of the user occupational class through twitter content. In Proceedings of the 53rd Anmual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 1754-1764.
John W Tukey. 1975. Mathematics and the picturing of data. In Proceedings of the International Congress of Mathematicians, Vancouver. 1975, volume 2, pages 523-531.
Ziyao Wang, Thai Le, and Dongwon Lee. 2022. Up- ton: Unattribuablle authorship text via data poisoning. arXiv preprint arXiv:2211.077)
Erwin Quiring, Alwin Maier, and Konrad Rieck. 2019. Misleading authorship attribution of source code us- ing adversarial learning. In 28th USENIX Security Symposium (USENIX Security 19), pages 479-496.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. 2022. Finetuned language mad- cls are zero-shot learners. In International Confer- ence on Learning Representations.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555 5551.
Sam Witteveen and Martin Andrews. 2019. Paraphras- ing with large language models. In Proceedings of the 3rd Workshop on Neural Generation and Trans- lation, pages 215-2200
Thomas Wolf, Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's 's transformers: State-of- the-art natural language processing, arXiv preprint arXiv: 191000377
Josyula R Rao, Pankaj Rohatgi, et al. 2000. Can pseudonymity really guarantee privacy? In USENIX Security Symposium, pages 85-96.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese newt-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- -JCNLPP, pages 3982-3991
Xingxing Xiong, Shubo Liu, Dan Li, Zhaohui Cai, and Xiaoguang Niu, 2020. A comprehensive survey on local differential privacy. Security and Communica- tion Networky, 2020:1-29.
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier. 2020. A differentially pri- vate text perturbation method using regularized ma- halanobis metric. Proceedings of the Second Work- shop on Privacy in NIP, pages 7-17.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05500.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulka- mi, Yin Tat Lee, Andre Manoel, Lukas Wutschitz. et al. 2021. Differentially private fine-tuning of lan- guage models. In International Conference on Learn- ing Representations.
Rakshith Shetty. Bernt Schiele, and Mario Fritz. 2018. [A4NT]: Author attribute anonymity by adversar- ial training of neural machine translation. In 27th USENIX Security Symposium (USENIX Security 18),
pages 1633-1650.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308.
Prasha Shrestha, Sebastian Sierra, Fabio González, Manuel Montes, Paolo Rosso, and Thamar Solorio. 2017. Convolutional neural networks for authorship attribution of short texts. In Proceedings of the 15th conference of the European chapter of the associa- tion for computational linguistics: Volume 2. short papers, pages 669-674.
A Proof of Privacy
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33:16857- 16867.
Theorem 2. Suppose the language model has not been pretrained on the private documents distribu- tion D. If the logits u E R| before the final soft- max layer satisfy the condition by I u4 S b2, Vi E

---

[V]], and the DP-Prompt run with a temperature T for generating n tokens, then it can be proven that the generated output satisfies (n(bb -LDD.
The sentence recoder architecture is trained with three layers of MLP, incorporating dropout and se- lecting the best model. We choose 50 clusters for sentence recoding and employ 100 random projec- tions to estimate the approximate Tukey depth.
Proof. Let and D'' be any two documents, and u and u' E Rl be their corresponding logits. Let e E V, and & be its index, with us being its corre- sponding logit. We then have that,
Sentence embeddings (Reimers and Gurevych, 2019) of dimension 768 are obtained from (Song et al., 2020).
exp( ME EE= exp( 7
C.3 Document-Level Mechanisms
Pr[M(D== v] Pr[M(D')= = b)
exp( "/2
For the paraphrasing mechanism, fine-tune the gpt2-x1(1.5b) parameter model the PAWS dataset (Zhang al., 2019). The training set is constructed by combining the train and val sets, and the test set is used for validation to save the best model. The training set consists of 25,368 examples, and the validation set consists of 3,536 examples. We follow the procedure outlined in (Mattern et al., 2022b), which builds upon (Wit- teveen and Andrews, 2019).
Σラ= exp( 7
M
Uk 1 T
i=1 exp( T
exp
Σ
M
exp( 7
j=1
b2 b1
b2 b1
< exp
exp
T
S exp (2(b2 - b1)/
Now by using sequential composition law of DP (Dwork et al., 2006), we can set 6 ( to conclude the proof.
To set clip thresholds in Section 4.4, we employ a process similar to Truncated-Laplace (Meehan et al., 2022) with a slight difference. While (Mee- han et al., 2022) calculates the 75% quantile and utilizes it as the clipping threshold, we calculate the minimum and maximum valucs.Both Truncated- Laplace and Dp-Prompt employ the exact same additional data. The resulting min and max clip threshold vector then used clip the logits be- fore scaling them by temperature.
B Code
In this section, we showcase the code for DP- Prompt implemented using the Hugging Face li- brary (Wolf et al., 2019). The code can be found in Listing I.
C Implementation Details and Additional Experiments
C.4 Static and Adaptive Attacker Architecture
C.1 Word-Level Mechanisms
Word level and document level output docu- ments, to simulate embedding-level attacker we employ sentence transformer all-mpnet-base (Reimers and Gurevych, 2019; Song et al., 2020) to convert sanitized document to sanitized embedding. For sentence level, directly sanitized embeddings used. The embedding- attackers employ three-layer MLP with hidden dimension of 768. We use the ReLU activation function and incorpo- rate dropout of 0.5. The models are trained for 50 epochs with a batch size of 32, using the Adam optimizer with a StepLR learning scheduler. The initial learning rate is set to 10-3 and the gamma value is 0.95.
For all word-level mechanisms, we utilize 50- dimensional glove embeddings (Pennington et al., 2014), following the approach of Mattern et al. (Mattern et al., 2022b). The projection step, which requires approximate nearest neighbor search, is performed using an Annoy indexer with 500 trees.
C.2 Sentence-Level Mechanisms
Both the Truncated-Laplace and Deep-Candidael mechanisms require additional publicly available data. In the case of Truncated-Laplace, this data is used to determine truncated boundaries, while for -eep-Candidate, it is used to train the sentence recoder and obtain the output.
The text-level attackers use bert-base-casee and fine-tune for epochs with a batch size of 16, using the AdamW optimizer with a linear scheduler and a starting learning rate of 5 x 10 5
We randomly sample 5,000 documents from both the IMDB and Yelp reviews, which are not part of the data used for privacy-utility experiments.

---

from transformers import LogitsPracessor, LogitsProcessorlist
class Clapedoogitsrrcessor (ospsoossss
def ...init_- min_tensor, max_tensor); self.min_tensor min_tensor
self max_tensos max_tensor
def (self input_ids, scores): loipped_logits torch. (scores, self min_ tensor, self, max_ tensor)) return clipped_logits
def prompt_ tempptteemlltt_miii (pprivate_ddc))
prompt = Document: (private_dos ) AnParaphrase of the document:" return prompt
def dp_prompt( private_doc, model, tokenizer, min_tensor, max_tensor, temp, new_tokens ):
logits_processor LogitsPPoccssorLisst [Ciippedlogitssrroosssorr (min max_tensor)]
private_prompt =
input_ids = wwweenizeeeencoee
(private__rompt, return_tensors="ptt))
output = model. generate
input_ids,
do_sample=True,
top_k=0,
top_p=1.0,
temperaturesttep,
mox_new.ttkenssnee..ook
saniti sinitized_doc tokenizer decize [0][0, skip_special_ttoknn=srruu return sanitized_doe
Listing 1: DP Prompt Code using Hugging Face's Transformers Library
C.5 Code and Reproducibility
True Positives(TP) Pi
The code will be publicly released. Considering the reproducibility challenges associated with closed APIs (Pozzobon al., 2023), also plan to re- lease the paraphrased documents that were gener- ated using ChatGPT.
True Negatives(TN)) (1 pi) 1)
False Positives(FP). 1 Ps
False Negatives(FN) Po(E 1)
From these metrics, we can calculate the F1 Score for sentiment analysis, which is binary classification task, as follows:
C.6 Expected F1 score of random classifier
TP
2p1 I PLf
Fl Score
TP + (FP FN)
Let pa represent the fraction of documents with la- bel Wi where ranges from 0 to 1. uniformly random classifier predicts class Yi with a probabil- ity of 1/6. Based on this setup, we can derive the following metrics:
For author identification, which is multi-class classification task, we utilize the Fl Score with a macro average. the case of a random classifier, the expected Fl score can be calculated as:

---

BERT Score
IMDB
Yelp
E=1.0
t=1.25
t-1.5
1-1.75
t=2.0
t=1.0
t-1.25
t-1.5
t-1.75
t-2.0
Chat GPT
0.882
0.879
0.863
0.805
0,765
0.89
0.887
0.876
0.83
0.777
Table 4: Bert Score for ChatGPT for different sampling temperatures
BERT Score
IMDB
Yelp
t=1.25
1-1.5
t=1.75
GPT-2 (sl) (ine tuned)
0.838
0.522
0.794
0.762
0.748
0.852
0.537
0.804
0.765
0.750
0.790
Stablelm-Baa (3b)
0.814
0.805
0.785
0.762
0.749
0.35
0.817
0.789
0.757
0.747
Stablelm-Ban Base (7b)
0.813
0.803
0.779
0.759
0.748
0.35
0.819
0.788
0.757
0.746
Flan (x))
0.823
0.792
0.762
Stabletm Tomed (3b)
0.846
0.530
0.795
0.757
0.743
0.849
0.836
0.800
0.760
0.746
(7b)
0.800
0.757
0.743
Table 5: Bert Score for Open Source Models for different sampling temperatures
D.2 BERT score for paraphrasing models
We BERTScore, which has been demonstrated to correlate with human judgments, in conjunc- tion with the RoBerta model (Liu et al., 2019) (roberta-large) to assess the similarity between review and its paraphrase. The results for Chat- GPT are presented in Table 4, while those for the open-source model can be found in Table 5.
Medo: 201
Mede ceid
Mare can
A telsa)
J.hh sore (mtors
(a) IMDB (static)
(b) IMDB (adaptive)
Figure 6: Comparision of glove-wiki-gl glove-wiki-gggaword-- gagwwrdd---0 with glove-wiki glowe-wik--iggaword 300 for word level mech- anism (Madlib, Mahalonobis, TEM) for IMDB dataset against embedding level attackers.
E Sample Illustration of paraphrases
In this section, we present a comparison of para- phrases generated from ChatGPT and Stablelm at various temperatures, as illustrated in Figure It is evident from the results that ChatGPT produces higher-quality paraphrases compared to Stablelm- Tuned (7B).
f-1
2p:
F1 Score macro marre avg
+ p.l
1=0
By employing this approach, we can effectively evaluate the performance of a random classifier for author identification in terms of the Fl Score.
Additional Results
D.1 Comparing 50 and 300-Dimensional Glove Embeddings for Word-Level Mechanism
Note that in Section 4.2, we used 50-dimensional Glove embeddings soove-wwki-gggawordee (Pennington et al., 2014) for Madlib, Maha- lanobis, and TEM mechanisms. In this section, we demonstrate that there significant ben- efit to using 300-dimensional glove embeddings www.wwiiggioood-----.
We show this for IMDB dataset against embedding-level attackers. Results are show Figure 6.

---

Cleen: So upeel do not know where begin. will call the store later today and issues resolved, but basically they wented do $1300 repaira did shoday my change. paying chenge completely different roster of repairs at dealership. am very heeltant to ever come back here.
(according my
experience was frustrating suggested repairs seemned unnecessary change was a second oll change and different repairs done elsewhere. lm unsure I would relurn astablishment.
Cha1GP((1.00):
get
suggested around $1400 worth unnecesary repairs and had poor quality service during my change. ended up paying for separate oil change and different repairs elsewhere. have double about returning to this location in the future.
ChalGPT(1 25): experience extremmey negalive
sure where
was very disappointing. They suggested unnecessary costly repairs that my confirmed not need, and the oil change was not done properly. ended up peying for second oil change another dealership and different repairs allogether. am reluctant to return to this place in the future.
ChaiGPT(11500)
wrote that they are incradibly upset do not know to words. They planned spanding $1800-$ $1800-$1400 but their dealer told them that those repeirs were unnecessary. In addition, the store did a poor joo with their oil change. To rectily the issue, the customer had to new change and another set of repairs the dealership. This bad experfance has them hesitant retum to the store the future.
ChatGPT(11.2 75):
ChatGPT(2.00 :0): This is highly stressful exporience and questioning retur due to poor ropair recommendatione undertakan which garnared the unnecessary use of dealership has costuuly participated.. Additonally, my oil change was nowhere caplow and overall tidy professonalism.
Srablelm-Tuned-760.0 The customer service at the dealership was tenrible dien't provide satisfactory solution to customer's issue.
Tebbeli--unned-D// 10010)) The reviewer upeet about the unneeded repeirs that done their car, and would like heve the opportunityy to resoive expersive they
they recogrize they would able because the expense of other repairs needed for their vehicle. The reviewer would be willing to call the store or visit the dealership again to resoive this issue.
they are
Stablelm- Stabielm-Tuned-ie Tuncd-7a(1..0) 7b(1.25): Negetive True The owner of this ravaged dealership took advantage of you after frustration with their lack adequate service and repairs.
Stabloim- Tuncd- (ooedd77/1..00 (0)1: my name Equipt buy Cansol start. PromiscRoviewer roports they tried retront-- car had just completed repairing three months ago throwing parts on dicegging disconeomommNTS over a cold channel, enter recover pre reFORM Recession DO THiSl hasnaaamennaannaaaaa huandroid dealership tumor Proclem index 80 AmyFa low 110. Boulevard drumess Bet@ cosmeticminimal precussorMomattor hand Harborbrock. Heat Kendara lsyoton88DLL Icense burger occromaggetic communication MycusElatch circumstance Controi ,dx conversion circumstance countable sodan Shall Rent
sorry Poshtoller fashion
Snapir
visual vehicular
Stablelm- Tuned-7b(1 75): place inhibited inroaddup hilanwkkiZZ requires actionsisd/rectoy Heyusedav indicated remady@uunttysexPPEC ('tetbbllotinwwerrnng seysPetitled nntiatvee369to Sponsirs Parkness 550BG riceSad implant imped pulmonaryck medicinsfort steady wrongrrus mean 220aldeconeumer itransactionplan linked logical jointeter Respons spells Internal welcum Conry converted 379inn BryantJohnson garage tolks the plaintif get curinglnit Perthlå unaffected 324in Statronary SpicelBC florrihh insurgytcht sought like Cancing humble no risky neighbor Doug Confederste paramsseles) 317Benguente APriR beam analyzocitive ennouncedArswer Strategic physicist strategyDec way
applicable
nobiegenstat touoensic emphwhen significant Sergeant Seriee right
Figure 7: Comparison of paraphrase results between ChatGPT (gpt-3.5) and Stablelm-Tuned-7B at different temperatures. ChatGPT consistently generates more readable and superior quality text at higher temperatures compared to Stablelm-Tuned-7BB.


Reviews:
[{"review_id": "3zGgJ2DEM6", "reviewer": "Reviewer_qQmq", "summary": "", "questions": "Question A: Has a more thorough analysis of privacy attacks been conducted? Why does the simple control of temperature (T) for text rewriting contribute to a reduction in the success rate of author identification attacks?\n\nQuestion B: Does the choice of different rewriting prompts have any impact on the results achieved?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1a): The unfairness in the experimental comparison in Figure 4. While this paper utilizes LLMs, the baseline methods used for comparison do not, and their models are relatively smaller compared to the chatGPT used in this study.**\n\nFirst we address baseline concern, \nAmong the 6 baselines we used, 3 used language models. Both Truncated Laplace and Deep Candidate make use of the powerful sentence transformer model available from [here](https://www.sbert.net/docs/pretrained_models.html). This model relies on MPNet as its foundation, which is known to outperform BERT, and it has been extensively fine-tuned on 1 billion pairs using a contrastive objective. For Deep Candidate, there's an additional step where representations are adjusted for maximum separation, facilitating the extraction of the Tukey Median, which is more resilient than the mean. Additionally, one of our baselines fine-tunes gpt-2xl (1.5B) specifically for the paraphrasing task. Furthermore, to our knowledge, no other mechanisms currently leverage more powerful language models. Whereas our method DP-Prompt shows how we can we larger and more powerful language models for privacy preservation easily and effectively.\n\n**Q1b)  The success shown in Figure 4 seems to be more attributed to the success of chatGPT rather than the privacy protection mechanism itself.**\n\nIt's essential to recognize that several of our baseline methods are intricately designed, reaching a ceiling in potential improvements. The central contribution of our study lies in framing the privacy protection so that LLMs can be directly employed without the need for extra fine-tuning or complex sampling processes. As opposed to earlier methods, DP-Prompt can indeed leverage powerful LLMs, leading to a superior privacy-utility tradeoff. This point is further emphasized by the privacy scaling behavior we presented, utilizing 6 open-source language models ranging from 3b to 7b in size.  In our view, this isn't just a side effect—it's a genuine strength of our method.\n\n**Q2) The privacy protection capability is not sufficient. The use of chatGPT for text paraphrasing does not guarantee the removal of privacy information in the rewritten text. Merely relying on manipulating the temperature (T) to achieve text privacy protection may not effectively achieve text sanitization.**\n\nA2: It's essential to differentiate between the two types of private data:\n\n1. *Implicit private Information*: This pertains to nuances such as writing styles, linguistic cues, and other subtle identifiers. \n2. *Explicit private Information*: This involves clear-cut personally identifiable information (PII) like zip codes, emails, and bank account numbers.\n\nImplicit private information poses a more daunting challenge than explicit private information, which is the problem we are trying to tackle. While the latter can be straightforwardly addressed using conventional string matching techniques or by simply instructing the model to omit PII data (as highlighted in our limitations/future work section, lines 595-599), the former requires more nuanced and sophisticated approaches to ensure its removal.\n\n**Question A: Has a more thorough analysis of privacy attacks been conducted? Why does the simple control of temperature (T) for text rewriting contribute to a reduction in the success rate of author identification attacks?**\n\nThere are two ways to understand this:\n\n*Theoretical Explanation:* The sensitivity of the exponential mechanism (See Def 3) is modulated by the temperature, $T$.\n\n*Intuitive Explanation:* At higher sampling temperatures, the model is more likely to select random tokens rather than the most probable ones. This process produces text that is significantly different from the original document. For a detailed demonstration, refer to page 14, where we present paraphrases generated at different temperatures using ChatGPT and StableLM-Tuned(7B).\n\n**Question B: Does the choice of different rewriting prompts have any impact on the results achieved?**\n\nThis is a good question. Our preliminary study shows that for larger models and minor changes in prompt templates, there are no significant differences in the paraphrases generated. However, prompt design is an ongoing area of research, and we leave it as future work for a detailed study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The proposed DP-Prompt method is simple in design and demonstrates superior performance results compared to the baseline in the conducted attack experiments. \n\n2) The paper provides new insights into using LLMs for protecting user privacy, starting from the perspective of Local Differential Privacy.\n\n3) I appreciate the thorough discussion on data memorization, which are crucial considerations when conducting experiments using LLMs.", "reasons_to_reject": "1) The unfairness in the experimental comparison in Figure 4. While this paper utilizes LLMs, the baseline methods used for comparison do not, and their models are relatively smaller compared to the chatGPT used in this study. The success shown in Figure 4 seems to be more attributed to the success of chatGPT rather than the privacy protection mechanism itself. \n\n2) The privacy protection capability is not sufficient. The use of chatGPT for text paraphrasing does not guarantee the removal of privacy information in the rewritten text. Merely relying on manipulating the temperature (T) to achieve text privacy protection may not effectively achieve text sanitization. ", "questions_to_authors": "Question A: Has a more thorough analysis of privacy attacks been conducted? Why does the simple control of temperature (T) for text rewriting contribute to a reduction in the success rate of author identification attacks?\n\nQuestion B: Does the choice of different rewriting prompts have any impact on the results achieved?", "paper_topic_and_main_contributions": "The paper introduces a privacy protection mechanism called DP-Prompt to reduce the success rate of de-anonymization attacks. In comparison to previous works, DP-Prompt has achieved advanced results. This paper can provide some inspiration for future research using LLMs to preserve user privacy.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There should be consistency in spacing between letters and parentheses, as seen in line 356 and line 481. Additionally, there is an error in line 861 where \"b1\" needs to be corrected. "}, "extra_text": {}}, {"review_id": "yJjdshSxOF", "reviewer": "Reviewer_8UTc", "summary": "", "questions": "Please see the questions above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) The use of a simple MLP authorship attribution model cannot provide convincing evidence for the effectiveness of DP-Prompt. Some CV works have shown that DP-based methods are useless on DNN models. So, is it effective on big models, e.g., BertAA?**\n\n\n1) Note MLP classifier is built on embeddings from the most powerful sentence transformer model available from [here](https://www.sbert.net/docs/pretrained_models.html), named all-mpnet-base-v2. This model is built upon MPNet, which has shown performance exceeding that of BERT. Moreover, this model is meticulously fine-tuned on a vast and diverse dataset of 1 billion pairs using a contrastive objective.\n2) If this was rudimentary, it wouldn't have secured *extremely* high F1 scores of 93% and 86% on the clean IMDB and Yelp datasets, respectively,  when classifying among 10 authors.\n\nAdditionally, our models for authorship attribution and sentiment analysis focuses on embeddings rather than text. This is because sentence-level mechanisms  generate sanitized embeddings, not complete documents. For a fair evaluation, we trained our models directly on these embeddings.\n\n**(2) One important method to protect authorship against ML is adversarial attack and data poisoning. However, this paper does not select these methods as baselines. Although DP-Prompt has some advantages, e.g., zero-shot, we can't say it's a SOTA method if not compared with SOTA adv attack and poisoning works.**\n\nThank you for these references. We were not previously aware of these works. We attempted to reproduce the UPTON work as suggested by the reviewer, but unfortunately, there is no available code. Moreover, we were unable to generate satisfactory number of adversarial example on our dataset using the TextFooler Attack [1] from (https://github.com/thunlp/OpenAttack). This problem is even acknowledged in their paper. Furthermore, making few word changes to flip the classifier's decision is not as robust as paraphrasing the entire document. Additionally, as the reviewer acknowledged, our method is zero-shot and has practical applications, whereas a data poisoning attack first requires additional data to train a surrogate classifier. We will incorporate our findings into the final version.\n\nRef: \n\n[1] Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authorship obfuscation problem has gained more attention in today's Internet, especially on social media privacy preservation. The use of LLMs on obfuscation is also an impressive topic.\n\nThe paper adopts Zero Shot Prompting, which does not need the training samples or authorship models and outperforms traditional obfuscation works on the practicality in real-world application.\n\n ", "reasons_to_reject": "Although the motivation is good and the method is novel, the paper does not provide enough experiments to prove its effectiveness and lacks some future insights.\n\n(1) The use of a simple MLP authorship attribution model cannot provide convincing evidence for the effectiveness of DP-Prompt. Some CV works have shown that DP-based methods are useless on DNN models. So, is it effective on big models, e.g., BertAA?\n\n(2) One important method to protect authorship against ML is adversarial attack and data poisoning. However, this paper does not select these methods as baselines. Although DP-Prompt has some advantages, e.g., zero-shot, we can't say it's a SOTA method if not compared with SOTA adv attack and poisoning works.\n\n", "questions_to_authors": "Please see the questions above.", "paper_topic_and_main_contributions": "This paper proposes an LLM based authorship obfuscation method, DP-Prompt, that utilizes Zero Shot Prompting to generate unidentifiable texts. The generated texts have low accuracy on authorship attribution models while preserving the text quality. The contribution of the paper is the use of LLMs in authorship obfuscation works and the consideration of DP.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Muhammad Haroon, Fareed Zaffar, Padmini Srinivasan, and Zubair Shafiq. 2021. Avengers Ensemble! Improving Transferability of Authorship Obfuscation. arXiv preprint arXiv:2109.07028 (2021).\n\nZiyao Wang, Thai Le, and Dongwon Lee. 2022. UPTON: Unattributable Authorship Text via Data Poisoning. arXiv preprint arXiv:2211.09717 (2022)."}}, {"review_id": "AvMau4KDXX", "reviewer": "Reviewer_ehk6", "summary": "", "questions": "1. How can we achieve privacy guarantee theoretically and demonstrated on the experiments? For example, the \\epsilon is a key privacy evaluation metric for DP, though the proof is provided, but there is no accountant procedure to compose it and the DP sensitivity analysis on it.\n\n2. Why we choose these datasets to evaluate the DP-prompt? The privacy scenario is not defined well and please further refine it. Besides, the comparisons are not cover all the previous methods and it is unclear to compare both the token-leven and paragraph-level together since they are not in the same setting (also refer to the used DP mechanism, need detailed explanation).\n\n3. The method is based on LLM, however, the LLM has its own privacy issue. Will we bring unexpected privacy risks by leveraging LLM for DP implementation? This is an interesting problem to be discussed.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1)How can we achieve privacy guarantee theoretically and demonstrated on the experiments? For example, the \\epsilon is a key privacy evaluation metric for DP, though the proof is provided, but there is no accountant procedure to compose it and the DP sensitivity analysis on it.**\n* We are evaluating privacy mechanisms with varied definitions (Metric-DP, Pure-DP, Local-DP). The meaning of $\\epsilon$ is different across these definitions.\n* Furthermore, the underlying mechanisms have distinct sensitivities. For example, the $\\epsilon$ in Metric-DP escalates with the size of the input document, while in Local-DP $\\epsilon$ grows with the size of the generated paraphrase (Refer to Theorem 1). This difference further justifies our choice not to focus  on $\\epsilon.$\n\nHence, we evaluate mechanisms on empirical privacy based on the (strong) threat models provided in Definition 2.\n \n**Q2a)Why we choose these datasets to evaluate the DP-prompt? The privacy scenario is not defined well and please further refine it.**\n\nThanks for this feedback. We will provide a detailed explanation in the introduction.\n\n\n\n\n**Q2b)Besides, the comparisons are not cover all the previous methods and it is unclear to compare both the token-leven and paragraph-level together since they are not in the same setting (also refer to the used DP mechanism, need detailed explanation).**\n\n   * The papers the reviewer referenced employ a mechanism similar to the TEM, which we do compare. Tokens are replaced with those that are semantically closer in these approaches. However, as discussed in recent literature and in our paper, such word (or token) level methods have significant shortcomings because they often disregard contextual information. Nevertheless,  we will add these references in our final version.  \n   * Regarding comparison of mechanisms, It is unfair to compare on $\\epsilon$ level, which is the reason we refrained from doing it. However, it's entirely justified to evaluate based on empirical privacy against real-world threat models.\n\n**Q3)The method is based on LLM, however, the LLM has its own privacy issue. Will we bring unexpected privacy risks by leveraging LLM for DP implementation? This is an interesting problem to be discussed.**\n\nThis is indeed an excellent and interesting point, but it's non-trivial and distinct from the main message of the paper, which is that the larger and better the language model, the better the privacy-utility tradeoff according to our experiments. We touch upon this point briefly in Section 5 in the context of data memorization. We plan to expand on it in the final version and leave addressing it in depth as an important direction for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Try to address an important problem of balancing privacy and utility for text data, where privacy is becoming a major concern as NLP models are deployed. Can brings attention to an important issue and proposes an effective solution applicable across domains like healthcare, finance, social media.\n\n2. Proposes a novel, simple, and effective differentially private text sanitization method using large language models where DP-Prompt is easy to implement while achieving strong empirical privacy. Allows leveraging powerful large language models for privacy enhancement instead of solely for generative tasks.\n\n3. Thorough evaluation spanning multiple models and comparisons to prior work. The experiments convincingly demonstrate the capabilities of DP-Prompt, and Provides useful insights into the factors impacting the privacy-utility tradeoff like temperature and clipping. This helps guide best practices.\n\nIn summary, the paper introduces a new method for differentially private text sanitization using LLMs and provides extensive analysis of its capabilities and tradeoffs. The simplicity of DP-Prompt while achieving strong empirical privacy-utility trade-off is a notable contribution.", "reasons_to_reject": "1. The privacy analysis is limited to empirical evaluation on specific datasets, rather than theoretical privacy guarantees and it focuses narrowly on authorship obfuscation rather than a broader set of privacy risks in text. This results in presenting empirical results as sufficient evidence of privacy without caveats. For example, the assumption of Theorem 1 is not supportive, since it is not practical to say \"Suppose the language model has not been pretrained on the private documents distribution D\".\n\n2. There is no detailed analysis of the syntactic or semantic changes induced by the paraphrasing where could modify meaning in problematic ways not explored in the paper. And the method may not work as well for certain sensitive text like medical records that require precise language. The privacy scenario should be defined well and the baseline methods setting should be explained further to ensure they are compare in the same settings (granularity and DP mechanism).\n\n3. There is limited error analysis explaining cases where the method fails and the approach relies heavily on large proprietary language models which have their own potential issues. The focus on empirical results over interpretability could encourage blind application without understanding risks and reduced scrutiny into issues with large language models and their role in ensuring privacy.\n\nOverall the main risks relate to overselling the privacy benefits demonstrated and setting unreasonable expectations from the method and results. But these seem like issues that could be addressed by nuancing the limitations and risks within the paper itself.", "questions_to_authors": "1. How can we achieve privacy guarantee theoretically and demonstrated on the experiments? For example, the \\epsilon is a key privacy evaluation metric for DP, though the proof is provided, but there is no accountant procedure to compose it and the DP sensitivity analysis on it.\n\n2. Why we choose these datasets to evaluate the DP-prompt? The privacy scenario is not defined well and please further refine it. Besides, the comparisons are not cover all the previous methods and it is unclear to compare both the token-leven and paragraph-level together since they are not in the same setting (also refer to the used DP mechanism, need detailed explanation).\n\n3. The method is based on LLM, however, the LLM has its own privacy issue. Will we bring unexpected privacy risks by leveraging LLM for DP implementation? This is an interesting problem to be discussed.", "paper_topic_and_main_contributions": "This paper proposes a locally differentially private text sanitization method called DP-Prompt that leverages large language models (LLMs) to paraphrase text in a privacy-preserving way. The key problem it addresses is mitigating privacy risks in text data while maintaining utility. Specifically, it aims to defend against author de-anonymization attacks.\n\nThe main contributions are:\n1.Propose DP-Prompt, a new locally differentially private mechanism that sanitizes text by prompting LLMs to paraphrase it in a zero-shot manner.\n2. Analysis showing DP-Prompt provides empirical privacy against author de-anonymization attacks while maintaining sentiment classification accuracy on text, and experiments with 6 LLMs up to 7B parameters demonstrating the privacy-utility tradeoff.\n\nIn summary, the paper introduces a new method for differentially private text sanitization using LLMs and provides extensive analysis of its capabilities and tradeoffs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Differential Privacy for Text Analytics via Natural Text Sanitization. Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, Sherman S. M. Chow. Findings of ACL 2021.\n[2] A Customized Text Sanitization Mechanism with Differential Privacy. Huimin Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yun Nie, Chengyu Wang, Jamie Cui. Findings of ACL 2023.\n\nThese two literature are related to token-level Natural Text Sanitization and highly relevant to this work, which should be discussed in the related work or baseline comparison."}}, {"review_id": "cUpQXCxeAJ", "reviewer": "Reviewer_NaNk", "summary": "", "questions": "What is the factuality score of your method?\nWhat is the hallucination rate?\nCan you make sure that the meaning of the original text is preserved?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1)What is the factuality score of your method? What is the hallucination rate? Can you make sure that the meaning of the original text is preserved?**\n\nThis is a great and interesting question. As with anything related to LLMs, one must be vigilant about hallucinations. We currently acknowledge this concern in the limitations/future work section. Addressing hallucination is an ongoing area of research, and hence we leave it as future work.\n\n**Q2)Tables and images are impossible to read without a magnifying glass.**\n\nWe apologize for the inconvenience. We had to compress them to fit the strict 8-page length, but we will increase their size in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper describes a very interesting approach, they provide extensive experiments on different models.", "reasons_to_reject": "There is no evaluation of keeping the original meaning of the text, hallucinations or factuality, which is a very important aspect. We want only to pharaphrase and not to change the meaning or information.", "questions_to_authors": "What is the factuality score of your method?\nWhat is the hallucination rate?\nCan you make sure that the meaning of the original text is preserved?", "paper_topic_and_main_contributions": "The authors of this paper present very interesting approach for making a document private by pharaphrasing the text with the use of LLMs. They present the results for DP-Prompt on IMDB and Yelp datasets. Sentiment score and author identification are used for the evaluation of the method. Apart from commercial models (ChatGPT) also open-source models are tested (Flan-T5, Stablelm-Tuned).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Tables and images are impossible to read without a magnifying glass."}, "extra_text": {}}, {"review_id": "KIsaPog6mr", "reviewer": "Reviewer_T4EW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1a)The performance is achieved via equipping a strong backbone (GPT-3.5). The comparison with previous approaches may not be meaningful.**\n\nIt's essential to recognize that several of our baseline methods are intricately designed, reaching a ceiling in potential improvements. The central contribution of our study lies in framing the privacy protection so that LLMs can be directly employed without the need for extra fine-tuning or complex sampling processes. As opposed to earlier methods, DP-Prompt can indeed leverage powerful LLMs, leading to a superior privacy-utility tradeoff. This point is further emphasized by the privacy scaling behavior we presented, utilizing 6 open-source language models ranging from 3b to 7b in size.  In our view, this isn't just a side effect—it's a genuine strength of our method.\n\n**Q1b)In contrast, the focus of this work may be the optimization of the prompt template for paraphrasing. However in the current manuscript such template seems to be a fixed one.**\n\n   The main reason we don't consider prompt templates is resource constraints:\n * For ChatGPT (3.5): Significant API costs are involved. Considering two datasets totaling ~27,000 examples and generating paraphrases for *5* different temperatures results in *135,000* text completions. Multiple prompt templates would amplify these costs.\n  * For open-source models: Significant GPU costs are incurred. When assessing urt*variance*, the process is run *3* times, leading to *405,000* text completions for each model.\n\n    Moreover, our preliminary investigations suggest that for our larger models, minor changes in prompt templates don't yield significant differences in generated paraphrases.\n\n**Q2) For evaluating the utility when employing privacy approaches, only one task (sentiment classification) is used. It would be better to consider more tasks.**\n\nThe crux of our work focuses on effectively and rigorously showing benefit of our proposed procedure with ChatGPT, open source models along with effects of top_k sampling, clipping logits. We didn't consider other tasks, primarily due to resource constraints (as discussed above). Nonetheless, expanding to other tasks is an important direction for the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Compared to previous approaches, this work is easy-to-follow because it only need to design a prompt template and adjust sampling temperature. It shows the potential of leveraging the zero-shot generation ability of LLMs to tackle various tasks. \n- Experiments show that the proposed approach achieves better trade-off between utility and privacy compared to existing defense approaches against deanonymization attacks. ", "reasons_to_reject": "- The performance is achieved via equipping a strong backbone (GPT-3.5). The comparison with previous approaches may not be meaningful. In contrast, the focus of this work may be the optimization of the prompt template for paraphrasing. However in the current manuscript such template seems to be a fixed one. \n- For evaluating the utility when employing privacy approaches, only one task (sentiment classification) is used. It would be better to consider more tasks. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the defense against deanonymization attacks, which is a challenging problem for applications using LLMs. Motivated by recent studies on paraphrasing and zero-shot learning, the authors proposed a prompting approach which directly feeds both a prompt template and original document to a powerful LLM for generating a paraphrase as the sanitized document. Besides, the sampling temperature is used to control the level of privacy guarantee. Experiments show that the zero-shot prompting approach has a good balance between utility and privacy. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most reviewers concur that the manuscript proposes a simple but effective zero-shot method that helps privacy preservation via leveraging the strong capabilities of LLMs. The introduced DP-prompt has demonstrated utility and can be applied without any fine-tuning. To improve the quality of this draft, however, consider the following suggestions from our reviewers.

1. It would be beneficial to test stronger baselines that exploit various sizes or types of LLMs. This would offer a clearer understanding of sensitivity-related concerns and pinpoint the sources of performance improvement. It is highly encouraged to include at least one additional state-of-the-art open LLM.

2. The current draft does not sufficiently clarify if the DP-prompt ensures the preservation of the original sentential meaning. Incorporating additional experiments, such as embedding-level similarity inspection or human evaluations, would provide valuable insights for both modelers and practitioners.

3. The authors are encouraged to contemplate the philosophical question: the feasibility of utilizing LLMs, which might themselves have privacy concerns, to address privacy issues. Addressing such inherent paradox in the conclusion section would be desirable.
Decision: Accept-Findings