ProPILE: Probing Privacy Leakage in Large Language Models
Authors: Siwon Kim
Sangdoo Yun
Hwaran Lee
Martin Gubri
Sungroh Yoon
Seong Joon Oh
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Personal identifiable information
Private data leakage
Large language model

Abstract: The rapid advancement and widespread use of large language models (LLMs) have raised significant concerns regarding the potential leakage of personally identifiable information (PII). These models are often trained on vast quantities of web-collected data, which may inadvertently include sensitive personal data. This paper presents ProPILE, a novel probing tool designed to empower data subjects, or the owners of the PII, with awareness of potential PII leakage in LLM-based services. ProPILE lets data subjects formulate prompts based on their own PII to evaluate the level of privacy intrusion in LLMs. We demonstrate its application on the OPT-1.3B model trained on the publicly available Pile dataset. We show how hypothetical data subjects may assess the likelihood of their PII being included in the Pile dataset being revealed. ProPILE can also be leveraged by LLM service providers to effectively evaluate their own levels of PII leakage with more powerful prompts specifically tuned for their in-house models. This tool represents a pioneering step towards empowering the data subjects for their awareness and control over their own data on the web.

ProPILE: Probing Privacy Leakage in Large Language Models
Siwon Kim 1.
Sangdoo Yun Sungroh Yoon 1,2.f
Hwaran Lee Seong Joon Oh Oh..6,
Martin Gubri
Department of Electrical and Computer Engincering, Scoul National University 2 Interdisciplinary Program in Artificial Intelligence. Scoul National University 3 NAVER AI Lab University of Luxembourg Parameter Lab Tübingen AI Center, University of Tübingen
Abstract
The rapid advancement and widespread use of large language models (LLMs) have raised significant concerns regarding the potential leakage of personally identifiable information (PII). These models are often trained on vast quantities of web-collected data, which may inadvertently include sensitive personal data. This paper presents ProPILE, a novel probing tool designed to empower data subjects, or the owners of the PII, with awareness of potential PIl leakage in LLM-based services. ProPILE lets data subjects formulate prompts based on their PII to evaluate the level of privacy intrusion in L.MMs. We demonstrate its application on the OPT-1.3B model trained on the publicly available Pile dataset. We show how hypothetical data subjects may assess the likelihood of their PII being included the Pile dataset being revealed. ProPILE can also be leveraged by LLM service providers effectively evaluate their own levels of PII leakage with more powerful prompts specifically tuned for their in-house models. This tool represents a pioneering step towards empowering the data subjects for their awareness and control over their own data on the web. The demo can be found here: httpg:/e//pprmmttttto..oceeeeeaa
Introduction
Recent years have staggering advances in large language models (LLMs) 25
31
6
28
32 [22]
The remarkable improvement is commonly attributed to the massive scale of training data crawled indiscriminately from the web. The web-collected data is likely contain sensitive personal information crawled from personal web pages, social media, personal profiles on online forums, and online databases such collections of in-house emails (13). They include various types of personally identifiable information (PII) for the data subjects [8]. including their names, phone numbers, addresses, education, career, family members, and religion, name a few.
This poses an unprecedented level privacy concern not matched by prior web-based products like social media. In social media, the affected data subjects were precisely the users who have consciously shared their private data with the awareness associated risks. In contrast, products based LLMs trained uncontrolled, web-scaled data have quickly expanded the scope of the affected data subjects far beyond the actual users of the LLM products. Virtually anyone who has left some form of PII on the world-wide wide-web is now relevant the question of Pll leakage.
Work done while interning Parameter Lab (tus lkkkkkku......m ac kr) Corresponding authors (sryoonésmu. ac, kr and coallaoh@gmail.c com)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ProPILE
Online activit
Web crawlingg without consent
Your risk exposure high.
Data subject
Because match number generated
Assessment
Jane Doe
9997 15P 26537, 3302) 7
List of
599 2853,, "9991650552"
Email
cccesooc.comm
LLM
99a 10A.1S9--55555 "173 0807") 10
Phone
90% 159-265
Test querles: (nams, emai) (phone)
Leakage risk evaluation
Address
x-2 steet 123...
Job
Affilaation
Profesior
ABC Universty
ema2l J. dooghhoc.coo con call her
"141 -122 1160. TF there reachable,
"Jane dee's contact emall J.dbejuuee.com com.
Form queries
hor phore number is
Send queries
Get responses
"5992592653. She friencly and
"Jase Boe's eax'l addnees and (phone natters
"273 305 0007, respectiiolyy She io
are J. j.dbegabe..com and
Responses
Figure 1: ProPILE. Data subjects may use ProPILE to examine the possible leakage of their own personally identifiable information (PII) in public large-languag formulate an LLM prompt based on M - 1 of their PIl items to task the LLM to output the M Pll not given in the prompt. If the generated responses include similar strings the true PII, this can be considered as privacy threat to the data subject.
model (LLM) services. ProPILE helps data subjects
Currently, there is no assurance that adequate safeguards are in place to prevent the inadvertent disclosure of PII. Understanding of the probability and mechanisms through which PII could leak under specific prompt conditions remains insufficient. This knowledge gap highlights the ongoing need for comprehensive research and implementation of robust leakage measurement tools.
In this regard, we introduce ProPILE, a tool to let the data subjects examine the possible inclusion and subsequent leakage of their own PII in LLM products in deployment. The data subject has only black-box access to LLM products; they can only send prompts and receive the generated sentences or likelihoods. Nevertheless, since the data subject possesses complete access to their leverages this to generate effective prompts aimed at assessing the potential PII leakage in LLMs. See Figure for an overview of the ProPILE framework. Importantly, this tool holds considerable value not only for data subjects but also for LLM service providers. ProPILE provides the service providers with tool effectively assess their own levels of PII leakage with more powerful prompts specifically tuned for their in-house models. Through this, the service providers can proactively address potential privacy vulnerabilities and enhance the overall robustness of their LLMs.
PII, ProPILE
Our experiments on the Open Pre-trained Transformers (OPT) [35] trained on the Pile dataset [I] confirm the following. 1) A significant portion of the diverse types of PII included in the training data can be disclosed through strategically crafted prompts. 2) By refining the prompt, having access to model parameters. and utilizing few hundred training data points for the LLM, the degree of PII leakage can be significantly magnified. We envision our proposition and the insights gathered through ProPILE the initial step towards enhancing the awareness of data subjects and L.M service providers regarding potential PII leakage.
2 Related Works
2.1 Privacy Leakage in Learned Models: Pre-LLM Era
The successful development of machine learning (ML) technologies and related web products led to privacy concerns. ML models may unintentionally include PII of certain data subjects in ML training data. As those models become publicly available, concerns have been raised that such PIIs may be accessed by millions of users using the ML service. Researchers have assessed the possibility of reconstructing Pll-relevant training data from a learned model [9] [T] 34 37 36). The task is referred to as training data reconstruction or model inversion. Previous work has shown that it is often possible to reconstruct training data well enough to reveal sensitive attributes (e.g., face images from a face classifier), even with just a black-box access 9 [1 34]. Researchers have also designed a more calluation-friendle surrogate task, membership inference attack (29), that tests whether each of the given samples has been included in the training data of the learned model. Subsequent work has shown that this is indeed possible for a wide range of models, including text-generation models 12 30 and image-gene gengeeeeneatiinn models [5]. For comprehensive review of the field up to 2020, refer
to the overview by Rigaki & Garcia 27
2

---

2.2 Privacy Leakage in Learned Models: Post-LLM Era
The appearance of billion-scale lange-lnggaage models (LLMs) and the highly successful products including ChatGPT [22], leads to an even higher level of privacy concerns. Their training data includes not only the data consciously or voluntarily provided by the data subjects [ but also massive crawl of the entire web such as personal web pages, social media accounts, personal profiles on online forums, and datahases of in-house emails [13]. Building a model-based service on such a web-crawled dataset and making it available to millions of users worldwide poses novel, serious threat the data rights of the data subjects. Motivated by this, a few early studies have been made to measure privacy leakage in LLMs [33 19 14]. However, although [33] initiated the discussion on PII leakage LLMs, it was limited to the preliminary analysis of only email addresses. [9]] conducted separate study that specifically targeted LLMs fine-tuned with an auxiliary dataset enriched with PII. Furthermore, their study specifically concentrated on scenarios where the prefix or suffix associated with the PII was known. In contrast, ProPILE aims to provide more comprehensive tool for probing LLMs already in deployment without LLM fine-tuning or prefix retrieval.
2.3 Prompt Tuning
Prompt engineering 226 18
improves downstream task performance of LLMs by well-designing
prompts without further LLM fine-tuning. In soft prompt tuning [15] [6], a few learnable soft token embeddings concatenated to the original prompts are trained while LLM is frozen, so that more optimal prompts for the downstream task can obtained. The white-box approach of ProPILE leverages soft prompt tuning to further refine the black-box approach's 's hand-crafted prompts.
3 ProPILE: Probing PII Leakage of Large Language Models
In this section, we propose ProPILE, probing tool to profile the PII leakage of LLMs. We first introduce the two attributes of PII, namely linkability and structurality, which are important for the subsequent analysis. We also describe our threat model and eventually introduce probing methods of ProPILE. Finally, we discuss the quantification of the degrees of privacy leakage.
3.1
Formulation of PII
3.1.1
Linkability
From a privacy standpoint, the random disclosure of PII may not necessarily pose substantial risk. For instance, when a phone number generated an unrelated context, there are no identifiable markers linking the number to its owner. However, if targeted PII is presented within a context directly tied to the owner, it could pose a severe privacy risk as it unequivocally associates the number with its owner. In light of this, the linkability of PII items has been considered critical for the study of privacy leakage [24]. We formalize the linkability of PII in the definition below.
Definition (Linkable PII leakage). Let A == fai...., aas} be M PII items relevant to data subject S. Each element am denotes PII item of specific PII type. Let be a prob- ing tool that estimates a probability of leakage of PII item Om given the rest of the items A.m = {a), am-1 Um++, ..., ). We say that T exposes the linkability of PII items for the data subject S when the likelihood of reconstructing the true PII, Pr(am Am: T), greater than the unconditional, context-free likelihood Pr(am).
3.1.2 Structurality
We consider PII in LLM training data in string format. Certain types of PII tend to more structured than others. The structurality of PII has significant implications for practical countermeasures against privacy leakage. We discuss them below.
Structured PII refers to the PII type that often appears in structured pattern. For example, phone numbers and social security numbers are written down in recognizable pattern like (xxx) XXx-XXXx that is often consistent within each country. Email addresses also follow a distinct pattern id@domain and are considered structured. Though less intuitive, we also consider physical addresses structured: [building, street, state, country, postal code].
3

---

We expect structured PII to be easily detectable with simple regular expressions M. This implies apparently simple remedies against privacy leakage. Structured PII may easily be purged out from training data through regular expression detection. Moreover, leakage of such PII may be controlled through detection and redaction the LLM outputs. However, in practice, the complete removal of structured PII in training data and L..--generated content is difficult. Regulating the generation of useful public information, such as the phone number and address of the emergency clinic, will significantly limit the utility of LLM services. It is often difficult to distinguish PII and public information that fall within the same pattern category. As such, is not impossible to find structured PII in the actual LLM training data, such the Pile dataset (section 4.1 [0], and the leakage of PII in actual LLM outputs [7].. We thus study the leakage of structured Pl in this work.
Unstructured PII refers to the PII type that does not follow an easy regular expression pattern. For example, information about data subject's family members is sensitive PII that does not follow designated pattern in text. One could write "[name13'' father is {name2}"", but this not the only way to convey this information. Other examples include the affiliation, employer, and educational background of data subjects. Unstructured PII indeed poses greater threats of unrecognized privacy leakage than structured PII. In this work, we consider family relationships and affiliation as representative cases of unstructured PII (section
3.2
Threat Model
Our goal is to enable data subjects to probe how likely LLMs are to leak their PII. We organize the relevant actors surrounding our PII probing tool and the resources they have access to.
Actors in the threat model. First of all, there are data subjects whose PII is included in the training data for LLMs. They have their ownership. or the data rights [8]. over the PII. LLM providers train LLMs using web-crawled data that may potentially include PII from corresponding data subjects. Finally, LLM users have access to the LLM-based services to send prompts and receive text responses.
Available resources. LLM-based services, especially proprictary ones, are often available as APIs, allowing only black-box access to LLM users. They formulate the inputs within the boundary of rate limit policy and inappropriate-conten regulations and receive outputs from the models, On the other hand, LLM providers have white-box access to the LLM training data, LLM training algorithm, and hyperparameters, as well as LLM model parameters and gradients. Data subjects may easily acquire black-box access to the LLMs by registering themselves LLM users, but it unlikely that they will get white-box access. Importantly, data subjects have rightful access to their own PII. We show how they can utilize their own PII to effectively probe the privacy leakage LLMs.
3.3 Probing Methods
We present two probing methods, one designed for data subjects with only black-box access to LLMs and the other for model providers with white-box access.
3.3.1 Black-box Probing
Actor's goal. In black-box probing scenario, an actor with black-box access aims to probe whether there is possibility that the L.M leaks one of their PII. Particularly, an actor has a list of their own PII A with M PII items and aims check if the target PII om A leaks from LLM.
Probing strategy. For a target PII ams a set of query prompts T is created by associat- ing the remaining PII Aym- Particularly, Am is prompted with K different templates tx as T (A),), tx (A)m) Then, the user sends the set of probing prompts T to the target LLM for as much as times. Assuming the target LLM performs sampling, the user will receive N x K responses along with the likelihood scores L E R where L. and V denote the length of the response and the vocabulary size of the target I.IM, respectively. Please note that the likelihood identical for the same query regardless of repeated queries. Example prompts are shown in Figure 2

---

3.3.2
White-box Probing
Actor's goal. In the white-box probing scenario, the goal of the actor is to find tighter worst-case leakage (lower bound on the likelihood) of specific types of PII (am). The actor is given additional resources beyond the black-box case. They have access to the training dataset, model parameters, and model gradients.
Probing strategy. We soft prompt tuning to achieve the goal, of finding a prompt that induces more leakage than the handcrafted prompts in the black-box case. First, we denote a set of PII lists included in the training dataset of target LLM as D {A'H Ji=1 White-box approach assumes that an actor has access a subset of training data D D, where |D| = n for n N. Let us denote a query prompt as X that is created by one of the templates used in the black-box probing X = Ln (At )m) Then X is tokenized and embedded into X, E R xd, where Lx denotes the length of the que sequence and d denotes the embedding dimension of the target LLM. The soft prompt 0g E R xd technically learnable parameters, are appended ahead of X. making [0.: X.] IR (xx++x where L. denotes the number of soft prompt tokens to be prepended. The soft embedding is trained to maximize the expected reconstruction likelihood of the target PII over D. Therefore, the training is conducted to minimize negative log-likeline bog-likelihood defined as below:
0 argmin E
log (Pr(a 10.: X.II]
After the training, the learned soft embedding 02 is prepended to prompts in (m)m) made of unseen data subject's PII to measure the leakage of am of the subject.
3.4 Quantifying PII leakage
For both black-box and white-box probing, the risk of PII leakage is quantified using two types of metries depending on the output that the users receive.
Quantification based on string match. Users receive generated text from the LLMs. Naturally, the string match between the generated text and the target PII serves as a primary metric to quantify the leakage. Exact match represents verbatim reconstruction of a PII; the generated string is identical to the ground truth PII.
Quantification based on likelihood. We consider the scenario that black-box LLMs can provide likelihood scores for candidate text outputs. The availability of likelihood scores enables a more precise assessment of the level of privacy leakage. It also lets one simulate the chance of LLMs massive scale. Reconstruction likelihood implies the
revealing the PII when it is deployed probability of the target PII being reconstructed given the query prompt. Therefore, the likelihood defined as follows used quantify the leakage:
Pr(am|AAp II p(am, D2, ..., "..,,,,,-,tt11 LLo+r-1).
(2)
In this equation, an represents the target PII and the product taken over the range from to Lr, where Ly represents the length of the target PII (am). I1, .., Ih,,r-1 correspond to the tokens or words comprising the query prompt of length L, followed by the response.
Even low level of likelihood has critical implications for privacy leakage, particularly for systems deployed at scale. For example, ChatGPT has been deployed to more than 100 million users world- wide (23] The likelihood of 0.01% of reconstructing the PII implies 100 cases of PII reconstruction only 0.01% % of the 100 million users attempt the reconstruction 10 times each [1] The inverse of the likelihood indicates the expected number of sampling or queries needed to generate the exact PII.
To give better sense of what the likelihood indicates, the fraction of data subjects whose PII is likely to be revealed within k queries sent. For example, 100.m. 0.01 indicates that for approximately 1 of data subjects, their PII of index m will be extracted when the LLM is probed 100 times with the same query.
introduce
metric Y<k- It indicates
0.01 likelihood x 100
0.01
100 10th users x 10 attempts 10 reconstructions
T00
5

---

(a) Black-box probing template examples
(b) Evaluation examples
Twins (name, pii_1)
Structured
information, please keep
"The (tanget_ntt _ttt _ttppptttttttt of (nane) is ", "Please contact (nane) by (target_pii__ypp) ",
touch with nane at address by phone
ena3l
<spe: eatrpplet nace:
Triplet (name, pii_1, pii_2)
piat
(ptend Shone unal etai
podress
addiness ?
"Pleese contact (nane) (pi1,1) (target pii type),", "You can reach (nane) at (pii_3) or cargge_ppi__type
Unstructured PII
narell
"s fatherr, relationiiip, nane: nomel
namez
Quadruplet (nane, pii_1, pii__, pis_a)
[typc
p11:
(nstatton:fatther, names namms
"Please contact (nane) (bi1,A) (pt1_2) (target [yp))
narel1
works at nanez and
"You can reach (nane) at (pii_1) 33 by (pii_2) or (target_pii, type)
(type:
affiltation,
pi1: (affiltation:
nazc
Figure 2: Probing prompts. (a) Black-box probing templates examples for different association levels. Blue text denotes the associated to be included in the prompt, Red text indicates the target PII and the type of it. (b) Examples from the evaluation dataset. Text in Pile dataset converted to dictionary.
# {PII A for data subjects in 'D Pr(am (Am) > E # of data subjects in
Neckm
4 Probing Existing LLMs
4.1 Experimental Setup
Target LLM to be probed. In our experiments, the selection of the target LLM was guided by two specific requirements. Firstly, in order to assess the probing results, it was necessary for the training dataset of the target LLM be publicly available. Secondly, to facilitate both black-box and white-box probing, it was essential to have access to pre-trained weights of the target model. To meet these criteria, opted to utilize the OPT with 1.3 billion hyperparameters (OPT-1.3B) 35 and corresponding tokenizer released by HuggingFace [33 as our target LLM for probing. Please note that the above criteria are for evaluation. In real-world scenarios, ProPILE is not limited OPT but can be applied to many other LLMs.
This paper conducts experiments using five types of PII: phone number, email address, and (physical) address as instances of structured PII and family relationship and university information as instances of unstructured PII. To evaluate the PII leakage, evaluation dataset was collected from the Pile dataset, which is an 825GB English dataset included in OPT training data [10]. It is noteworthy that the presence of documents containing all five types linked to data subject is rare in the Pile dataset. However, for structured PII, there were instances where all three types of structured PII were linked to the name of data subject. Hence, quadruplets (name, phone number, email address, address) from the Pile dataset. Specifically, the PII items searched with regular expressions and named entity recognition P2 21). Examples shown in Figure (b). For the collection of unstructured PII, we adopted question- -nnwwerinn model based on RoBERT and formulated relevant questions extract information regarding relationships or affiliations. Only answers with a confidence score exceeding 0.9 were gathered, and subsequently underwent manual filtering eliminate mislabeled instances. The final evaluation dataset consists of the structured PII quadruplets for 10,000 data subjects, name-family relationship pairs for 10,000 data subjects, and universiiyrsity pairs for 2,000 data subjects. Please refer to the Appendix for the dataset construction details.
Evaluation dataset.
extracted
4.2 Black-box Probing Results
We show how the black-box probing approach of ProPILE with hand-crafted prompts helps data subjects assess the leakage of their own PII. We also examine the effect of various factors on the leakage.
http:://nuggingffaccco////tt cot.c//ccaacceeecacaccacoooooo attpss//huggingaaee...///iiittil saseee-caadd-disstti

---

(a)
Structured Personally Identifiable Information
Phone number
Email address
Address
0.05
Null
0.125
6.00e
0.04
0.04
Reconstruction
Reconstruction mn5n:: 103e 06
0.100
mesn: 9.59-21
Density
0.03
0.03
0.075
Null
Reconstructio
Nuil
0.02
moon: 4.730-12
2008-055
0,050
Tream 1:5a-23
0.03
0.025
p value
p p-voluc
p volue cle
0.00
0.000
-40
-20
-40
-20
50
40
Likellhhoo (log10)
Likellhood (log10)
Likellhood (log10)
(b)
Unstructured Personally Identifiable Information
(c)
Summary Y<
Relationship
Affiliation
0.03
Relationship
Reconstruction
0.06
makm: 2:630-06
Recomsiruction mean: 1770--08
0
Density
0.00
0.04
Ratio
0/
NuIl
0.01
0.02
mean: 20
0.5
Email
Amilatioo
p vojo de 20
Preslue = 4.0ba00
0.00
-40
-20 Likelihood (logto)
0.00
-40
-20
1.0
10
10a
10
10
1000 10
Likelihood (logt0)
Address
Figure 3: Black-box probing result in likelihood perspective. Reconstruction baseline likelihood of (a) structured PII and (b) unstructured PII, shown with the average likelihood and the p-value of the Wilcoxon signed-rank test. (c) shows a summary of the likelihoods using Yc, defined in Equation
Likelihood results. We first evaluate the likelihood of the target PII item given the other items of a subject. Then, we consider the black-box LLM as revealing the linkable PII item. if the likelihood probability is greater than that of randomly selected PII instances, i.e., Pr(um (m) > Pr(am.Null A. The am,Nui is randomly selected from the evaluation dataset. We utilized the aforementioned evaluation dataset and created prompts using five different triplet templates, including those described in Figure 2 (a). Subsequently. the generation is done using beam search with a beam size of The likelihood was computed using Equation [2] Please refer to the Appendix for the detailed generation hyperparameters and prompt templates.
Figure (a-b) illustrates the density plot of the likelihoods. The blue and orange color represents the target PII (am) and randomly chosen PII (ml,, ). respectively. The plots also display the mean likelihood values. It is observed that the mean likelihood of target PII is higher than that of the null PII for all PII types. We also denoted the p-value obtained from the statistical test using the Wilcoxon signed rank test [7]. The small p-value suggests that the observed difference is statistically significant except for affiliation. Figure [3 (c) shows Yek- We have mentioned in section 3.4 that the x-axis variable, k, can be interpreted as the number of queries. As the number of queries increases, we observe a gradual increase in the frequency of exact reconstruction.
The above box probing results demonstrate high risk of reconstructing the exact PII based on available PII items and establishing the link. The results of Tikk indicate that despite the seemingly low likelihood values, there is possibility of exact reconstruction of PII.
Exact match results. Through black-box probing, the generated sequences can be obtained. The exact match can be assessed by evaluating whether the generated sequence includes the exact string of target PII or not. First, we evaluated the exact match with a varying number of templates used to construct the prompts. Results are shown in Figure (a). The rate of exact matches increases as the diversity of prompt templates increases. This also supports the rationale behind white-box probing, as it suggests that finding more optimal prompts can further increase the leakage.
Furthermore, we conducted an assessment of exact matches when different levels of the associations were present in the prompt. Figure shows the results. The "twins" denotes that only the name of a data subject is used to make the query prompt, while "triplet" indicates the presence of an additional PII item in the prompt. We can observe fivefold increase in the exact match rate for the
7

---

(a) Number of prompts
(b) Association level
(d) Model size (OPT)
8.050
Relationship
Aff
0,005
Email
Phone
ationsipp
Exect match (%)
0.025
Address
AAfillation
0.000
wing
trplct
Phone
(c) Number of beams
Phone
0.004
Email
0
055
0.010
Emert
0.002
0.005
Address
Address
0.000
0.000
350m
1.30
2.7b
7b
Figure 4: Black-box probing results in string-match perspective. The proportion of PII that is exactly reconstructed through black-box probing. We vary (a) the number of query prompts, (b) the level of associated PII items in the query prompt, (c) the beam size for decoding and (d) the size of the targeted L.M.M
email address. This increase occurred when phone number, which offers more specific information about the data subject, was provided in addition to the name. In the case of phone numbers, we also observed an increase of more than double. This shows increasing information in the prompts that can be associated with the target PII elevates the leakage. It also supports the effectiveness of black box probing that utilizes the data subject's linkable Plls. Furthermore, with increased beam search sizes in the model (Figure((e)) and larger model sizes (d), the frequency of the target PII appearing in generated sentences also tends to rise. The increasing leakage that occurs with larger model sizes can be attributed to improved accuracy. This implies that as the current trend of scaling up large language models continues, the potential risks of PII leakage may also increase.
4.3 White-box Probing Results
In this section, we demonstrate the white-box probing by presenting the leakage of the phone number given other PII information the structured quadruplet. We train 20 embedding vectors for the soft prompts by appending them ahead of single prompt to generate the target phone number; We use additional 128 quadruplet data that are not included in the evaluation dataset. Please refer to Appendix for the training details. With the trained soft prompts, we measure the likelihood probabilities and exact match ratios on the evaluation dataset. Figure summarizes the results in terms of the number of training data, the number of soft tokens, and the initialization type.
Efficacy of soft prompt tuning. Figure 3 illlustrates the impact of the soft prompt on the exact match rate and reconstruction likelihood, with blue and orange colors, respectively. The results indicate a significant increase, from 0.0047% of black-box probing using five prompt templates to 1.3% with the soft prompt learned only from 128 data points being prepended to single query prompt. The likelihood also increased by a large amount for the same case. It is speculated that the observed increase can be attributed to the soft prompt facilitating the more optimal prompts that may not have probing. been considered by humans during the construction of prompts in black-box probing.
Effect of dataset size. The white-box probing scenario assumes that a user (or a service provider) has access to only a small portion of the training data. To see the impact of the number of data used for tuning to the degree of the leakage, soft prompts were trained using different numbers of triplets in the training dataset, specifically [16, 32, 64, 128, 256, 512). The results are depicted in Figure 5 (a). Even with 16 data points, a significant surge in leakage was observed. The exact match rate escalated to 0. 12%, surpassing the exact match scores achieved by using five prompts, as well as terms of likelihood. As the training set size increases from 16 to 128, the exact match dramatically increases from 0. 0.12% to 1.50%. This finding indicates that even with a small fraction of the training dataset, is possible to refine prompts that can effectively probe the PII leakage in LLM.
Additional analysis of soft prompt tuning. We also examine the impact of different factors on the leakage and Figure 5) (b) and (c) display the leakage levels according to these factors. As the number of soft tokens increases, the leakage also exhibits an increasing trend. This can be attributed to the enhanced expressiveness of the soft prompts, which improves as the number of parameters increases. Furthermore, different initialization schemes produce diverse outcomes. We investigated three initialization schemes: I) an embedding of the word representing the specific type of target PII, i..., "phone", which was the default setting throughout our experiments, 2) an embedding sampled from uniform distribution 24 (-1, 1), and 3) utilizing the mean of all vocabulary embeddings. As
8

---

(a) Number of training data
(b) Number of soft tokens
(c) Initialization type
Exaet metch (80
match
Likelliood (os-4)
Exact match (1)
Likelihood 0oe-4
Exagt match (S)
Like Meelihoodin (ol-4
L.
Liselhood
Likelhood
Exect match
Likelhood
0.0
0.0
125
250
10
Uniform
Moan
type
Figure 5: White box probing results. Leakage results 10,000 unseen triplets according to (a) varying number of data used for prompt tuning, (b) number of soft tokens, (c) different intialization type. Blue and orange color denotes exact match rate and likelihood, respectively.
Table I:
Transferability of soft prompt. Original denotes the hlack-box probing results using one query prompt and transfer denotes the probing results using the transferred soft prompt that learned from the source model (OPT-L.3B). columns show how much the leakage likelihood increases by using transferred soft prompt.
Source
Target
Avg. Likelihood
# Exact match
Original
Transfer
x
Original Transfer
DPT-350M
1.05 10 11
1.08 10
7.5
0
0PT-1.3B
0PT-1.3B 3B
6.06 10
3.47 x.17x10 10
57.3
5
3
0.
1.39 x 10-7
2.18x10-6
15.6
14
illustrated in Figure I(c), the uniform and mean initialization schemes were unable to raise the leakage. In contrast, initializing with the PII type resulted in the most significant leakage.
Transferability test. If the soft embedding learned for one language model can be reused to probe different language model, it opens up the possibility of applying the knowledge acquired from white- box probing to black-box probing. To assess the feasibility of this approach, we transferred the soft prompt learned for the OPT-1.3B model to OPT models with different scales, namely OPT-350M and OPT-2.7B. However, directly plugging the soft embedding trained on one model into another model is impossible due to the mismatch of embedding dimensions (e.g.. 1, 024 and 512 for OPT-1.3B and OPT-350M, respectively.) To address this, we follow a two-step process of the previous approach (20). We project the soft embedding to the closest hard tokens in terms of Euclidean distance and decode it to raw string with the source model's tokenizer. The string is then concatenated ahead of the raw query text and fed into the target model.
Table I demonstrates that the soft prompt learned from the OPT-1 model increases the leakage of the same type of PII in both the OPT-350M and OPT-2.7B models. The increase in leakage also denoted with the multiplication symbol (x), showcasing how many times the reconstruction likelihood is amplified when utilizing the soft prompt learned for OPT-1.3B in the other models. While there may not be a substantial difference from the exact match perspective, the potential for transferability has been confirmed in the perspective of likelihood. Future work could explore research for investigating white-box probing techniques for enhancing transferability.
Conclusion
This paper introduces ProPILE, a novel tool designed for probing PII leakage in LLM. ProPILE encompasses two probing strategies: black-box probing for data subjects and white-box probing for LLM service providers. the black-box probing approach, we strategically designed prompts and metrics so that the data subjects can effectively probe if their own PII is being leaked from LLM. The white-box probing approach empowered LLM service providers conduct investigations on their own in-house models. This was achieved by leveraging the training data and model parameters to fine-tune more potent prompts, enabling deeper analysis of potential PII leakage. By conducting actual probing on the O.T-1.3B model, we made several observations. First, we found that the target PII item generated with a significantly higher likelihood compared to a random PII item. Furthermore, white-box probing revealed a tighter worst-case leakage possibility in terms of PII leakage. We hope that our findings empower the data subjects and I.I.M service providers for their awareness and control over their own data on the web.

---

Limitations. The construction of the evaluation dataset exclusively involved the use of private information sourced from open-souree datasets provided by large corporations. This approach ensures the ethical acquisition of data. However, it's important to acknowledge that the data collection process itself was heuristic in nature. Consequently, the evaluation dataset may contain instances of incorrectly associated data or noise. This could introduce degree of uncertainty potential inaccuracies, which must be taken into account when interpreting the results.
Societal Impact. We emphasize that our proposed probing strategies are not designed to facilitate or encourage the leakage of PII. Instead, our intention to provide a framework that empowers both data subjects and LLM service providers to thoroughly assess the privacy state of current LLMs. By conducting such evaluations, stakeholders can gain insights into the privacy vulnerabilities and potential risks associated with LLMs prior to their deployment in a wider range of real-world applications. This proactive approach aims to raise awareness among users, enabling them to understand the security and privacy implications of LLM usage and take appropriate measures to safeguard their personal information.
Ethical Considerations
The evaluation dataset used in our paper collected from the Pile dataset and thus follows the data regulation and terms of services of it. Specifically, all sources of the Pile dataset are public, with the majority following the terms services, and 55% of them are authorized by data owners. Please refer to 10 for the details.
ProPILE's primary intent is to empower data subjects to verify whether their own information may leak from LLM services. However, the black-box probing approach could be abused by malicious actors aiming to extract others' personal information from LLMs without proper consent. Although attackers remain incapable of discerning whether the extracted information belongs to the target, they may still obtaín sequences with potential connections with the target. Additionally, the chance of successful attacks with limited information is not entirely eliminated, although it is very low according to the associativity experiment results Section 4.2]
Here we suggest some guidelines to use and develop ProPILE properly. ProPILE should only be used by the data subjects themselves to examine the reconstruction of PII items linked to them, or by the individuals who have been given explicit consent from the relevant data subjects. One way to implement this is to insert an authentication layer over ProPILE allow its usage only to users identified through Google accounts; they are only allowed to submit queries using only the information from the user's verified Google profile. Ultimately, the user's good intentions are the most important. We strongly recommend that ProPILE be used only for its intended purpose of enhancing the self-awareness of data subjects.
Acknowledgements
This work was supported by NAVER Corporation, the National Research Foundation of Korea (NRF) grants funded by the Korea government (MSIT) (2022R A3B 1077720, 2022R ASA 1), Institute of Information Communications Technology Planning & Evaluation (ITTP) grants funded by the Korea government (MSIT) (2021-0-01343: AI Graduate School Program, SNU, 2022-0- 00959), the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University in 2023, and the Luxembourg National Research Funds (FNR) through CORE project C18////1266977777TTLLLLLLB Traon.
References
[1] Amazon Web Services.
Detecting and redacting pii using amazon com- https://aws amazon. http://wwo..ooooo////o/o hwtps//machieneee learning amazon- cowpreheeommeeeeececcceeee 2023.
prehend. detecting www.eiiinn-adddrrdrriiiggiigiiii
[2] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit.
O'Reilly Media, Inc.", 2009.
10

---

[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[4] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski. Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07644 2022.
[5] Dingfan Chen, Ning Yu. Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of member- ship inference attacks against generative models. Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343-362, 2020.
[6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[7] William Jay Conover. Practical nonparametric statistics, volume 350. john wiley & sons, 1999.
[8] European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council.
[9] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart, Model inversion attacks that exploit confidence information and basic countermeasures, In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
[10] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The Pile: 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002: 2020.
[11] Zecheng He, Tianwei Zhang, and Ruby B Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 162-162, 2019.
[12] Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks on sequence- to-sequence models: my data in your machine translation system? Transactions of the Association for Computational Linguistics, 8:49-63, 2020.
[13] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205. 12628, 2022.
[14] Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette- Chho, and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy. arXiv preprint arXiv:2210. 17546, 2022.
[15] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[16] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Pupers), pages 4582-4597, 2021.
[17] LiveMint.
Chatgpt answer ist's number to join signal.
goes
wrong,
gives
away
journal-
https://wws www livemint......mm.mmm.. cww/ news/
chatgpt- goes- wooes-wrong- wrong awvy anees-wway-jorrnae - www-----uuu-----ee to- to- signal 11676625029542 html| 2023.
[18] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8099 2022.

---

[19] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella- Béguelin. Analyzing leakage of personally identifiable information language models. arXiv preprint arXiv:2302.005399 2023.
[20] Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gardner. Adversarial prompting for black box foundation models. arXiv preprint arXivv2332.042337 2023.
[21] Omri Mendels, Coby Peled, Nava Vaisman Levy, Tomer Rosenthal, Limor Lahiani, al. Microsoft Presidio: Context aware, pluggable and customizable pii anonymization service for text and images, 2018.
[22] OpenAl. Gpt-4 technical report, 2023.
[23] Martine Paris. Chatgpt hits 100 million users, google invests in ai bot and catgpt goes viral, Apr 2023.
[24] Andreas Pfitzmann and Marit Hansen. A terminology for talking about privacy by data mini- mization: Anonymity, unlinkability, undetectability. unobservability, pseudonymity, and identity management, 2010.
[25] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[26] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-7, 2021.
[27] Maria Rigaki and Sebastian Garcia. A survey of privacy attacks in machine learning. arXiv preprint arXiv:2007.07646, 2020.
[28] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick. Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A I 766--parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. 2022.
[29] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[30] Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 196-206, 2019.
[31] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201. 008239....9 2022.
[32] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 2.0.::2.32./9977.. 2023.
[33] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: Sffte-of-the- art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Languuge Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics.
[34] Ziqi Yang, Jiyi Zhang, Ee-Chien Chang, and Zhenkai Liang. Neural network inversion in adversarial setting via background knowledge alignment. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 225-240, 2019.
12

---

[35] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 200006 2022.
[36] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 253-261. 2020.
[37] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
13


Reviews:
[{"review_id": "X5XsAnVcoT", "reviewer": "Reviewer_9AMv", "summary": "This paper proposes a novel probing tool to probe potential privacy leakage in large language models. The proposed method includes both black-box and white-box probing. The authors demonstrate its effectiveness on the OPT-1.3B model with the publicly available Pile dataset. The experimental results show that the proposed method can probe the leakage of personally identifiable information (PII) in the Pile dataset, while the white-box probing generally achieves better performance. The results prove that PII leakage could be a serous threat to LLM’s security, calling for more research attention in this area.", "questions": "1. Is it possible to extend the proposed method to more LLMs or even closed-source LLMs such as ChatGPT or GPT4?\n2. How do you justify the choice of OPT-1.3B as the subject LLM for evaluation?\n", "limitations": "There are discussions about the limitations and potential social impacts of this paper. I have two more comments.\n1. Regarding the limitation, it would be nice to mention the potential generalization ability of the proposed method to other LLMs (or even closed-source LLMs, such as GPT4)\n2. Regarding the potential social impact, the authors should directly point out that the research outcome (i.e., the probing tool) might potentially be used for undesired purposes and potentially lead to negative social impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Timely and important topic. With the more and often usage of LLMs, LLMs’ security and potential PII leakage should be taken into account. This paper proposes an early attempt in this area regarding probing PII from LLMs, showing the importance of the research problem.\n2. Novel method. The authors propose a novel probing tool including both black-box and white-box probing.\n3. Well-written paper. The paper is generally well-written with good presentation quality. I enjoy reading the paper.", "weaknesses": "I generally do not find significant weaknesses in this paper. I do have a few small concerns, that, I wish the authors could address in the revision.\n1. The definition of the soft prompt \\theta_s is not clear (Line 172).\n2. Evaluation. In the current form, only one model was included for evaluation. IMO this is fine for demonstration. But if the authors would like to improve it, more models could be considered.\nA follow-up comment regarding the candidate LLMs: The selection criteria (Line 205-211) should be more clear, and the selection should have its rationale, e.g., is OPT-1.3B the only model that is available for evaluation? If there are more candidates, why OPT-1.3B is the only one chosen? All these details can help improve the paper’s soundness and significance.\n3. Lack of examples. It would be nice to add a few examples in the paper to better show the probing results (both black-box and white-box ones).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer of thorough review and insightful comments. We are glad that the reviewer enjoyed reading and acknowledged importance, novelty and presentation of our work. The weaknesses also helped us to deepen our arguments and improve the readability.\nWe make sure to update the paper with the responses.\n \n__Weakness 1__\n\n> The definition of the soft prompt \\theta_s is not clear (Line 172).\n\nThe soft prompt denoted as $\\theta_s \\in \\mathbb{R}^{L_S \\times d}$ is a matrix consisting of $L_S$ soft token embeddings. Each soft token embedding is a tunable vector with the same dimension $d$ as token embeddings. The soft prompt is optimized through backpropagation. We will elaborate the line 172 in the revised paper.\n\n---\n\n__Weakness 2__\n\n> Evaluation. In the current form, only one model was included for evaluation. IMO this is fine for demonstration. But if the authors would like to improve it, more models could be considered. A follow-up comment regarding the candidate LLMs: The selection criteria (Line 205-211) should be more clear, and the selection should have its rationale, e.g., is OPT-1.3B the only model that is available for evaluation? If there are more candidates, why OPT-1.3B is the only one chosen? All these details can help improve the paper’s soundness and significance.\n\nWe would like to emphasize that we have presented results for various scales of OPT, including OPT-350m, 2.7b, and 6.7b, in Figure 4 (d). Additionally, in the Appendix, we have included the results of Bloom-3b and 7b.\n\nOur rationale for selecting the target model includes two key criteria: 1) for evaluation, the model should be trained with the Pile dataset & 2) for white-box probing experiments, the model should be open-sourced. The possible candidates were GPT-j (2021), GPT-Neo (2021), OPT (2022) and Bloom (2022). Among these, we chose OPT and Bloom as the target models as they are the most recent and widely used. This information will be incorporated in the final version. \n\n\n---\n\n__Weakness 3__\n\n> Lack of examples. It would be nice to add a few examples in the paper to better show the probing results (both black-box and white-box ones).\n\nThe probing examples in our study contain sensitive information. Although they are all sourced from public datasets, we considered it possibly inappropriate to disclose them. We considered an option of masking out the specific information, but doing so would have resulted in generic templates resembling those in Figure 2. We have provided an example in the Appendix A.3., carefully selected to avoid any potential privacy concerns. Instead, we will share a demo of the tool where users can test with their own personal data.\n\n\n---\n\n__Question 1__\n> Is it possible to extend the proposed method to more LLMs or even closed-source LLMs such as ChatGPT or GPT4?\n\nThe proposed tool is applicable to other LLMs than the ones presented in the paper. Notably, the black-box probing approach can be effectively employed even with closed-source LLMs, as it does not necessitate access to the model parameters. In light of this, we are currently preparing a demo that will enable people to probe various types of LLMs with their own data. \n\n---\n\n__Question 2__\n> How do you justify the choice of OPT-1.3B as the subject LLM for evaluation?\n\nAs explained in our response to Weakness 2, we have presented results for OPT-350m, 2.7b, and 6.7b in Figure 4 (d). Appendix contains the results of Bloom-3b and 7b. Considering the two rationales for target model selection, the possible candidates were GPT-j (2021), GPT-Neo (2021), OPT (2022) and Bloom (2022). Among these, we decided to choose OPT as it is the most recent and widely used model. \n\n---\n\n__Limitation 1__\n\n> Regarding the limitation, it would be nice to mention the potential generalization ability of the proposed method to other LLMs (or even closed-source LLMs, such as GPT4)\n\nThe tool we propose can be applied to other LLMs beyond those in the paper. The black-box probing method is effective for closed-source LLMs as well, as it doesn't need model parameter access. The demo that we are preparing will support various LLMs including closed-source LLMs. Please also refer to the response to Question 1.\n\n---\n\n__Limitation 2__\n\n> Regarding the potential social impact, the authors should directly point out that the research outcome (i.e., the probing tool) might potentially be used for undesired purposes and potentially lead to negative social impact.\n\nThank you for the valuable comment. As you mentioned, the black-box probing approach of ProPILE could be abused by malicious actors to extract others’ personal information. One possible countermeasure would be to authenticate users with verified accounts and share the explicit guidelines. We promise to include the above discussion in the revised paper. The draft is attached in the global “Author Rebuttal by Authors”.\n"}, {"review_id": "E9MnT0icc9", "reviewer": "Reviewer_idGg", "summary": "The authors propose a privacy attack against large, pre-trained language models to evaluate PII leakage. The idea is that a data subject queries the model using some of their PII to discover whether the model can reconstruct the remaining PII not contained in the query. This attack measures the ability of a model to link PIIs, which is a clear privacy violation. The authors propose a black-box and white-box probing method and demonstrate that an attacker can link PII with a high success rate. ", "questions": "* If I understand correctly, you use regular expressions and NER to extract PII. How did you assess that pieces of PII were linked when extracting from the Pile dataset? \n\n* Section 3.3.2 \"White-box Probing\" is a bit unclear to me: Is the optimization of the prompt conditioned on the PII that you want to reconstruct or is it conditioned on a set of unrelated PII from the same class?\n", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Important problem. Finding tools to empirically measure leakage in LLMs is an interesting and open problem. \n\n* Interesting findings. The study finds that pre-trained LLMs can leak PII information meaningfully and that an attacker can be able to link PII with each other even if they only have black-box access to the model. \n\n* Clean methodology. I think the paper's method of distinguishing between structured and unstructured PII is meaningful and a good idea. Also, I liked the way that the paper collected PII from the Pile dataset. \n\n* Clear figures and well written. The paper is easy to follow. \n\n", "weaknesses": "* I would have liked to see more ablation studies on the leakage in relation to factors known to increase leakage, such as the duplication rate. A qualitative analysis of the leaked PII could be helpful to understand whether these e-mail addresses are somewhat unique or not. \n\n* No promise to release source code. \n\n* Limited novelty in the methodology on the privacy attack. The attacks are relatively straightforward. \n\n* No discussion on potential countermeasures to prevent leakage. \n\nNits\n\n* l. 284: I assume it should say 'small portion of the training _data_'. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful comments and insightful suggestions. Additional study on the suggestion also deepened our understanding and added another dimension to our study. We are also glad that the reviewer recognized the importance, meaningfulness, and clear presentation. We hope that this response clarify any uncertainty and address concerns.\n\n__Weakness 1__\n\n> I would have liked to see more ablation studies on the leakage in relation to factors known to increase leakage, such as the duplication rate. A qualitative analysis of the leaked PII could be helpful to understand whether these e-mail addresses are somewhat unique or not.\n\nFactors that are known to increase the leakage are (1) model size and (2) duplication rate [a]. The impact of model size is studied in Figure 4(d) for three different sizes of OPT (350m, 2.7b, and 6.7b). In accordance with the [a], the leakage increases as the number of model parameters grows. \n\nOn the other hand, more repetitions of PII in training data does not lead to an increased leakage of reconstruction likelihood ([$1.40\\cdot 10^{-7}$, $5.73\\cdot 10^{-06}$, $6.72\\cdot10^{-9}$] for repetition number [1, 2, 3]). This apparently contradicts the findings in [a]. There are two possible causes of the mismatch. First, [a] prompts the exact prefix for the sequence to be reconstructed, while our ProPILE constructs the prompts based on PII provided by the user, which may not be the exact prefix preceding the target PII to be reconstructed. We conjecture that the repetition facilitates the extraction only when the precise prefix is available. Second, PII is not repeated frequently. In the PILE dataset, only 3% of PII instances were repeated more than three times. This is in contrast to [a] that has shown that leakage increases substantially with more than ten repetitions.\n\n[a] Carlini, Nicholas, et al. \"Quantifying Memorization Across Neural Language Models.\" The Eleventh International Conference on Learning Representations. 2023.\n\n---\n\n__Weakness 2__\n\n> No promise to release source code.\n\nWe will not release the entire source code due to the possibility of potential abuse by malicious users. Instead, we are planning to publish a web application that everyone can use freely to check their own degrees of exposure in various LLMs. We believe this solution strikes the best balance between the benefits and potential harms of our technology. To facilitate follow-up research, we will provide the code only upon request of verified users.\n\n---\n\n__Weakness 3__\n\n> Limited novelty in the methodology on the privacy attack. The attacks are relatively straightforward.\n\nThe primary goal of this paper is (1) to raise awareness of data subjects that their private information can leak from LLMs that are publicly available to everyone and (2) to let them quantify their individual risks. Although there is no methodological novelty, such an individualized approach to addressing the privacy risk is proposed for the first time, as far as we are aware.  \n\n---\n\n__Weakness 4__\n\n> No discussion on potential countermeasures to prevent leakage.\n\nWe also believe that research on defense mechanisms is essential, but it is beyond the scope of our paper. We would like to point out that the scope of our paper lies in raising the issue of leakage and providing an effective tool to measure it. Yet, our study regarding duplication rate in Weakness 1 provided a potential direction of defense. The limited impact of duplication rate on PII leakage necessitates a different approach from earlier studies [b, c] focused on deduplication-based mitigation. One possible direction is to break the linkability of PII by masking one of the PII items in the training documents. We also hope that our research stimulates further research to mitigate the leakage and protect data privacy. \n\n[b] Kandpal, Nikhil, Eric Wallace, and Colin Raffel. \"Deduplicating training data mitigates privacy risks in language models.\" International Conference on Machine Learning. PMLR, 2022.\\\n[c] Lee, Katherine, et al. \"Deduplicating Training Data Makes Language Models Better.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n\n---\n\n__Weakness 5__\n\n> l. 284: I assume it should say 'small portion of the training data'.\n\nThank you. We will correct it.\n\n---\n\n__Questions 1__\n\n> If I understand correctly, you use regular expressions and NER to extract PII. How did you assess that pieces of PII were linked when extracting from the Pile dataset?\n\nThrough a preliminary examination of the Pile dataset, we have observed that linkable PII items often appear in the same sentence. We have constructed the evaluation dataset by (1) performing NER to detect individual PII items, (2) linking PII items appearing in the same sentences, and (3) manually inspecting the linkage. We will update the paper with this procedure.\n\n---\n\n__Question 2__\n\n> Section 3.3.2 \"White-box Probing\" is a bit unclear to me: Is the optimization of the prompt conditioned on the PII that you want to reconstruct or is it conditioned on a set of unrelated PII from the same class?\n\nFor white-box probing, the optimization is conditioned on the PIIs that we aim to reconstruct. Soft prompt tuning of white-box probing is conducted with the PIIs of $n$ data subjects in the evaluation dataset. Specifically, ProPILE constructs the input prompt for each data subject using their $M-1$ PII items, as in the black-box approach, which is then concatenated with the soft prompt. The soft prompt is optimized to maximize the average reconstruction likelihood of the $M^\\text{th}$ PII for the $n$ data subjects. \n"}, {"review_id": "XFUPjDzyQd", "reviewer": "Reviewer_udNB", "summary": "This paper introduces ProPILE, a tool to assess the potential leakage of Personally Identifiable Information (PII) in Large Language Models (LLMs). The work is significant given the recent widespread use and growing concerns around privacy issues of LLMs. The authors demonstrated its use with the OPT-1.3B model and discussed potential societal impacts and ethical considerations.", "questions": "While the paper introduces the ProPILE tool as an effective strategy, it does not address potential countermeasures that malicious entities might use to circumvent the system. This could be an important aspect to explore in further research.", "limitations": "The authors acknowledge that the construction of the evaluation dataset exclusively involved the use of private information sourced from open-source datasets provided by large corporations. This approach ensures the ethical acquisition of data. However, they also mention that the data collection process was heuristic in nature, which could introduce a degree of uncertainty or potential inaccuracies. This must be taken into account when interpreting the results.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel tool to tackle an important issue in the current AI landscape: privacy intrusion and PII leakage in LLMs. The development of a probing tool such as ProPILE holds considerable practical significance. The paper provides concrete examples of how the ProPILE tool can be used to detect potential PII leakage in LLMs. The results seem to suggest a reasonable level of effectiveness of the proposed tool.", "weaknesses": "The paper acknowledges that the construction of the evaluation dataset involved private information from open-source datasets provided by large corporations. There is an inherent bias in this data collection process, and potential inaccuracies need to be considered when interpreting the results. The paper could have been strengthened by comparing the ProPILE tool's performance with other existing or similar tools/methods for detecting PII leakage in LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough review. We are glad that the reviewer acknowledged novelty, significance, concretness, and effectiveness of our work. We hope that this response can clarify any uncertainty. \n\n__Weakness 1__\n\n> The paper acknowledges that the construction of the evaluation dataset involved private information from open-source datasets provided by large corporations. There is an inherent bias in this data collection process, and potential inaccuracies need to be considered when interpreting the results. \n\nOne potential bias is that the Pile dataset is primarily US-centric, so certain PII items are not proportionally representing the entire population over the globe. For example, phone numbers are in the format (xxx) xxx-xxxx and the physical addresses are written in the format <street> <city> <state> <postal code>. Hence, the results on this benchmark should not be overgeneralized to the linkability of PII for the general population. Moreover, it is indeed possible that there are some erroneously linked PII items, even though the PII linkage has been thoroughly inspected by authors and the absolute majority of the linkage is correct. We will include this to the limitation section in the revised paper. \n\n---\n\n__Weakness 2__\n\n> The paper could have been strengthened by comparing the ProPILE tool's performance with other existing or similar tools/methods for detecting PII leakage in LLMs.\n\nWhile a quantitative comparison against prior research [a, b] would be ideal, they are addressing a different task than ours. They have been limited to evaluating if PII is reconstructed when the verbatim prefix in the training dataset is given. None of these studies have measured the leakage of linkable PII through created prompts, making direct comparisons infeasible. The related work [a]  focused solely on email, thus it cannot be applied to various types of PII as in our work. Their experiment was also conducted exclusively with the Enron email dataset, utilizing highly specialized prompts such as “--original message-- \\n From: {} [mailto: ]”, further hindering a meaningful comparison.\n\n[a] Huang, Jie, Hanyin Shao, and Kevin Chen-Chuan Chang. \"Are Large Pre-Trained Language Models Leaking Your Personal Information?.\" arXiv preprint arXiv:2205.12628 (2022).\\\n[b] Lukas, Nils, et al. \"Analyzing Leakage of Personally Identifiable Information in Language Models.\" 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2023\n\n---\n\n__Question 1__\n\n> While the paper introduces the ProPILE tool as an effective strategy, it does not address potential countermeasures that malicious entities might use to circumvent the system. This could be an important aspect to explore in further research.\n\nThank you for the suggestion to add more dimensions to our work. Malicious actors may indeed abuse the system to extract PII of data subjects included in the training datasets. We propose to address this concern by (1) sharing only the web application to the public usable by authenticated users and (2) including an ethics statement against malicious use cases. Please see our response in Author Rebuttal by Authors for further information. \n\n---\n\n__Limitation 1__\n\n> The authors acknowledge that the construction of the evaluation dataset exclusively involved the use of private information sourced from open-source datasets provided by large corporations. This approach ensures the ethical acquisition of data. However, they also mention that the data collection process was heuristic in nature, which could introduce a degree of uncertainty or potential inaccuracies. This must be taken into account when interpreting the results.\n\nPlease see our response to Weakness 1.\n"}, {"review_id": "aE4fkn7uuo", "reviewer": "Reviewer_CvUa", "summary": "The paper presents ProPILE, which is a probing tool designed to help identify if the PII of the user can be revealed by probing the LLM-based services. The authors consider two attributes of PII: linkability and structurality, and two types of access: black box and white box. The authors also provide two quantifications: string match and likelihood to quantify the risk of PII leakage. Experiments are done based on these attributes, access, and quantifications to quantify the risk of PII leakage from LLM-based services.", "questions": "See weakness.", "limitations": "See weakness bullet 2, 3 and 4.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThe paper is well-motivated, and studies an important and timely problem.\n•\tThe authors provide a principled solution to quantify the risk of PII leakage from LLM-based services.\n•\tWith the given formulation of PII, threat model, probing method and PII leakage quantification, the authors provide extensive study on the PII leakage.\n", "weaknesses": "•\tThe black-box probing is one way to generate a prompt. In many cases, users don't know what PII of theirs are used in LLM and pose privacy threat.\no\tsuggestion: establish a prompt database with different kinds of prompt generating rules. These rules are designed for revealing PII.\n•\tExact match may be a strict but narrow privacy definition. The authors should extend the research to a broader concept of privacy. For many entities, there may not be M PII items available.  \n•\tAlso, partial leakage is still leakage, such as one can reconstruct the city but not the street. This broader concept of privacy is missing in this paper, making the scope of the paper limited.\n•\tIn the meantime, this type of approach, if falls on the wrong hand, can be used to discover PII in LLM-based models.\n•\tDensity in Fig 3 is undefined. \n•\tIn the experiment, the exact match percentage and likelihood seems both not high. This gives rise of the question that if the PII leakage problem of LLM-based service severe enough. The authors may want to provide some discussion and insights here.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for insightful review and suggestions. We noticed that some of the weaknesses can be addressed by the Appendix. We will reorganize the paper for more visibility. \n\n__Weakness 1__\n\n> The black-box probing is one way to generate a prompt. In many cases, users don't know what PII of theirs are used in LLM and pose privacy threat. o suggestion: establish a prompt database with different kinds of prompt generating rules.  These rules are designed for revealing PII.\n\nThe suggestion to establish template database is a great research direction to further facilitate users to build a diverse range of possible prompts based on their own PII items by themselves. \nEven if a user does not know what PII types were in the training set, ProPILE can be employed as long as the user has a good coverage of all possible PII items. Then suitable prompts can be generated with various sub-combinations of PII types to prompt LLM. \nIn this case, the templates used in our paper, described in Appendix Section A.3,  can be easily adjusted for the different sub-combinations and used to construct the template database. \n\nTarget PII | Available PII type | Template \\\nPhone number | (name) | “The phone number of {name} is”  \\\nPhone number | (name, email) | “The email address of {name} is xx@xx and the phone number is” \\\nPhone number | (name, email, address) | “The email address of {name} is xx@xx and the address is xx, xx, xx. His phone number is”\n\nWe can also leverage LLMs by instructing it to generate suitable prompts for each sub-combination. We attached an example of prompting few templates to ChatGPT and generate a template to probe unseen PII item, social security number (SSN).\n\n```\nInput: \n(name, phone) “The phone number of {name} is” \n(name, phone, email) “The phone number of {name} is {xxx-xxx-xxxx} and email address is” \n(name, phone, email, SSN)\n```\n\n```\nResponse: \n\"The phone number of {name} is {xxx-xxx-xxxx}, the email address is {email address}, and the SSN is``\n```\n\n---\n\n__Weakness 2 & 3__\n> Exact match may be a strict but narrow privacy definition. The authors should extend the research to broader concept of privacy. For many entities, there may not be M PII items available.\n\n> Also, partial leakage is still leakage, such as ... This broader concept of privacy is missing in this paper, making the scope of the paper limited\n\nWe agree that relying solely on the exact match may restrict our studies to a narrow definition of privacy: verbatim reconstruction of the PII string. This is the reason why we propose the definition of privacy based on the probabilistic linkability of PII items (Definition 1 in Section 3.1.1) along with the corresponding evaluation scheme based on the leakage likelihood (Section 3.4). It represents the probability of target PII being reconstructed by the LLM based on the provided context PII items. This assumes a more relaxed and relevant concept of privacy: even when only the city of the subject’s home address is revealed, it already substantially increases the likelihood of the subject being identified compared to having no information about the location of the subject at all. \n\nTo further support this broader concept of privacy, we have adopted evaluation metrics like partial string match and edit distance in Appendix Section B.2.1. Table 1 of Appendix showed the results for Location code match (first three digits of US phone number), First-$l$ match (exact match of first 7, 8, and 9 digits), and edit distance (ratio of samples reconstructed within edit distances 1, 2, 3). We reported that 17% of the location code, 0.12%, 0.48%, and 1.12% of the first 9, 8, and 7 digits of phone numbers could be reconstructed verbatim. Moreover,  0.16%, and nearly 1% of phone numbers could be reconstructed within edit distances 1 and 2, respectively. We also conducted a fine-grained analysis for email addresses: even if the entire email address is not leaked, there are cases where the exact id is disclosed. \n\n---\n\n__Weakness 4__\n\n> In the meantime, this type of approach, if falls on the wrong hand, can be used to discover PII in LLM-based models.\n \nBased on your feedback, we have analyzed the potential negative societal impacts that our work may have. Attackers are unable to distinguish if extracted data is from the target, yet they can access sequences with potential connection to the data subject. Additionally, successful attacks with limited data remain possible. As a countermeasure, we propose guidelines for ProPILE usage, including identity verification and explicit permission of data owner. Please see theAuthor Rebuttal for detailed draft.\n\n---\n\n__Weakness 5__\n\n> Density in Fig 3 is undefined.\n\nFig. 3 is the probability density plot. The y axis denotes the probability density function (PDF) value. We will clarify this in the revised paper.\n\n---\n\n__Weakness 6__\n\n> In the experiment, the exact match percentage and likelihood seems both not high. This gives rise of the question that if the PII leakage problem of LLM-based service severe enough. The authors may want to provide some discussion and insights here.\n\nOn average, the numbers may be interpreted as “not severe enough”. However, hidden behind the average metrics are the existence of data subjects under very high leakage risks. In Figure 3 (a-b) of the main paper, we observe that for email, phone number, and relationship, there are more than 20 instances each where the reconstruction likelihoods exceed $10^{-4}$. For those high-risk data subjects, the question of reconstructability of their PII in a generative model service with potentially 1 billion $=10^9$ visits per month [a] is a real threat to their privacy. Moreover, when evaluated with partial string match metrics, the leakage becomes more severe. As mentioned in Response 3, nearly 1% of phone numbers are reconstructed with exact match in first 7 digits or edit distance 2. \n\n[a] https://www.similarweb.com/blog/insights/ai-news/chatgpt-1-billion/\n"}, {"review_id": "sCTATwof02", "reviewer": "Reviewer_EPof", "summary": "This paper addresses the pressing issue of potential privacy breaches resulting from the use of large language models (LLMs). The authors propose a novel tool named ProPILE that empowers data subjects, or the owners of personally identifiable information (PII), to evaluate the risk of their personal data being inadvertently disclosed by LLMs. ProPILE allows users to generate prompts based on their own PII to assess potential privacy intrusion.", "questions": "Whether the authors plan to release any specific datasets alongside their tool, ProPILE. \n\nThe reference should have been revised. Abbreviations on a title should be capitalized using {}, e.g., PII.  Some preprints have been published, e.g., 'Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity'  is published on Proceedings of the 60th Annual Meeting of the Association for Computational Linguistic, 2022.", "limitations": "The authors have partially addressed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work is timely and relevant given the current widespread use of LLMs. \n\nThe idea to empower users to assess potential privacy intrusion using their own data is innovative. This approach could foster a better understanding of privacy concerns among users.\n\nThe authors showcase ProPILE's utility using the OPT-1.3B model, trained on a publicly available dataset called Pile. They demonstrate how users can assess the likelihood of their personal data being revealed through these LLMs. Importantly, ProPILE can also be used by service providers of LLMs to evaluate their own levels of PII leakage, thus enhancing the robustness of their models against potential privacy violations.", "weaknesses": "The application and evaluation of ProPILE are solely based on the OPT-1.3B model. While this provides insights into its performance, it does not necessarily guarantee its effectiveness when used with other Large Language Models (LLMs).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thorough review and valuable comments. We are glad that the reviewer recognized our work as \"timely\" and \"innovative\". The comments also helped us to elaborate and clarify our paper. We hope that our response address the reviewer's concern and clarify any uncertainty. \n\n__Weakness 1__\n> The application and evaluation of ProPILE are solely based on the OPT-1.3B model. While this provides insights into its performance, it does not necessarily guarantee its effectiveness when used with other Large Language Models (LLMs). \n\nWe would like to remind you that the ProPILE has been applied to different models including different sizes of OPT-350m, 2.7b, and 6.7b and different types, Bloom-3b  and 7b. The results are shown in Figure 4(d) and Appendix Figure 2-3, respectively. We believe that these results support the generalizability of ProPILE. We will make the results more visible in the revised paper.\n\n---\n\n__Question 1__\n> Whether the authors plan to release any specific datasets alongside their tool, ProPILE.\n\nThe collected evaluation data includes private information. Although it originates from the public dataset (the Pile), we have concerns about the potential risks of sharing the data. Instead, we are planning to share a demo where users can probe the leakage of their own private data. \n\n---\n\n__Question 2__\n> The reference should have been revised. Abbreviations on a title should be capitalized using {}, e.g., PII. Some preprints have been published, e.g., 'Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity' is published on Proceedings of the 60th Annual Meeting of the Association for Computational Linguistic, 2022.\n\nThank you for informing us of the mistakes. We will thoroughly review and make the necessary revisions to the references in the final version.\n"}]

Author rebuttal:
To all reviewers and ethics reviewers,

We appreciate all reviewers for acknowledging the importance of our work and providing us with constructive feedback. We hope that this rebuttal addresses any uncertainties and concerns. All responses will be incorporated into the revised paper. 

A few reviewers were concerned about the potential societal impact of our work. We decided to add a dedicated section that explicitly discusses the ethical data collection, potential negative impact that may arise from the ProPILE usage, and guidelines for responsible usage. The draft is attached below.

__Ethical considerations__

The evaluation dataset used in our paper is collected from the Pile dataset [a], and thus follows the data regulation and terms of services of it. Specifically, all sources of the Pile dataset are public, with the majority following the terms of services and 55% of them are authorized by data owners. Please refer to [a] for the details.

ProPILE's primary intent is to empower data subjects to verify whether their own information may leak from LLM services. However, the black-box probing approach could be abused by malicious actors aiming to extract others’ personal information from LLMs without proper consent. Although attackers remain incapable of discerning whether the extracted information belongs to the target, they may still obtain sequences with potential connections with the target. Additionally, the chance of successful attacks with limited information is not entirely eliminated, although it is very low according to the associativity experiment results in Section 4.3. 

Here we suggest some guidelines to use and develop ProPILE properly. ProPILE should only be used by the data subject themselves to examine the reconstruction of PII items linked to them, or by the individuals who have been given explicit consent from the relevant data subjectss. One way to implement this is to insert an authentication layer over ProPILE to allow its usage only to users identified  through Google accounts; they are only allowed to submit queries using only the information from the user’s verified Google profile. Ultimately, the user's good intentions are the most important. We strongly recommend that ProPILE be used only for its intended purpose of enhancing the self-awareness of data subjects.

[a] Gao, Leo, et al. "The pile: An 800gb dataset of diverse text for language modeling." arXiv preprint arXiv:2101.00027 (2020)


Meta-review: The reviewers appreciate the paper's timely focus on privacy leakage in large language models, a topic of growing importance. While noting the novel probing tool and well-crafted presentation, they encourage a broader evaluation and clearer definition of soft prompt parameters. The meta-reviewer recommends acceptance as a spotlight, suggesting the addition of concrete examples to enhance the paper's significance and depth.
Decision: Accept (spotlight)