SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Siwon  $\mathbf{Kim}^{1,*}$
Sangdoo Yun3 Sungroh Yoon1,2,†
Hwaran Lee $^{3}$  Ma Seong Joon Oh $^{5,6,\dagger}$
rtin Gubri4,5
<sup>1</sup> Department of Electrical and Computer Engineering, Seoul National University
$^{2}$  Interdisciplinary Program in Artificial Intelligence, Seoul National University  
 $^{3}$  Naver AI Lab  $^{4}$  University of Luxembourg  $^{5}$  Parameter Lab  
 $^{6}$  Tübingen AI Center, University of Tübingen
The rapid advancement and widespread use of large language models (LLMs) have raised significant concerns regarding the potential leakage of personally identifiable information (PII). These models are often trained on vast quantities of web-collected data, which may inadvertently include sensitive personal data. This paper presents ProPILE, a novel probing tool designed to empower data subjects, or the owners of the PII, with awareness of potential PII leakage in LLM-based services. ProPILE lets data subjects formulate prompts based on their own PII to evaluate the level of privacy intrusion in LLMs. We demonstrate its application on the OPT-1.3B model trained on the publicly available Pile dataset. We show how hypothetical data subjects may assess the likelihood of their PII being included in the Pile dataset being revealed. ProPILE can also be leveraged by LLM service providers to effectively evaluate their own levels of PII leakage with more powerful prompts specifically tuned for their in-house models. This tool represents a pioneering step towards empowering the data subjects for their awareness and control over their own data on the web. The demo can be found here: https://parameterlab.de/research/propile
Recent years have seen staggering advances in large language models (LLMs) [25, 3, 31, 6, 28, 32, 22]. The remarkable improvement is commonly attributed to the massive scale of training data crawled indiscriminately from the web. The web-collected data is likely to contain sensitive personal information crawled from personal web pages, social media, personal profiles on online forums, and online databases such as collections of in-house emails [13]. They include various types of personally identifiable information (PII) for the data subjects [8], including their names, phone numbers, addresses, education, career, family members, and religion, to name a few.
This poses an unprecedented level of privacy concern not matched by prior web-based products like social media. In social media, the affected data subjects were precisely the users who have consciously shared their private data with the awareness of associated risks. In contrast, products based on LLMs trained on uncontrolled, web-scaled data have quickly expanded the scope of the affected data subjects far beyond the actual users of the LLM products. Virtually anyone who has left some form of PII on the world-wide-web is now relevant to the question of PII leakage.
* Work done while interning at Parameter Lab (tuslkkk@snu.ac.kr)
† Corresponding authors (sryoon@snu.ac.kr and coallaoh@gmail.com)
Figure 1: ProPIE. Data subjects may use ProPIE to examine the possible leakage of their own personally identifiable information (PII) in public large-language model (LLM) services. ProPIE helps data subjects formulate an LLM prompt based on  $M - 1$  of their PII items to task the LLM to output the  $M^{\mathrm{th}}$  PII not given in the prompt. If the generated responses include similar strings to the true PII, this can be considered as a privacy threat to the data subject.
Currently, there is no assurance that adequate safeguards are in place to prevent the inadvertent disclosure of PII. Understanding of the probability and mechanisms through which PII could leak under specific prompt conditions remains insufficient. This knowledge gap highlights the ongoing need for comprehensive research and implementation of robust leakage measurement tools.
In this regard, we introduce ProPILE, a tool to let the data subjects examine the possible inclusion and subsequent leakage of their own PII in LLM products in deployment. The data subject has only black-box access to LLM products; they can only send prompts and receive the generated sentences or likelihoods. Nevertheless, since the data subject possesses complete access to their own PII, ProPILE leverages this to generate effective prompts aimed at assessing the potential PII leakage in LLMs. See Figure 1 for an overview of the ProPILE framework. Importantly, this tool holds considerable value not only for data subjects but also for LLM service providers. ProPILE provides the service providers with a tool to effectively assess their own levels of PII leakage with more powerful prompts specifically tuned for their in-house models. Through this, the service providers can proactively address potential privacy vulnerabilities and enhance the overall robustness of their LLMs.
Our experiments on the Open Pre-trained Transformers (OPT) [35] trained on the Pile dataset [10] confirm the following. 1) A significant portion of the diverse types of PII included in the training data can be disclosed through strategically crafted prompts. 2) By refining the prompt, having access to model parameters, and utilizing a few hundred training data points for the LLM, the degree of PII leakage can be significantly magnified. We envision our proposition and the insights gathered through ProPIE as the initial step towards enhancing the awareness of data subjects and LLM service providers regarding potential PII leakage.
2.1 Privacy Leakage in Learned Models: Pre-LLM Era
The successful development of machine learning (ML) technologies and related web products led to privacy concerns. ML models may unintentionally include PII of certain data subjects in ML training data. As those models become publicly available, concerns have been raised that such PIIs may be accessed by millions of users using the ML service. Researchers have assessed the possibility of reconstructing PII-relevant training data from a learned model [9, 11, 34, 37, 36]. The task is referred to as training data reconstruction or model inversion. Previous work has shown that it is often possible to reconstruct training data well enough to reveal sensitive attributes (e.g., face images from a face classifier), even with just a black-box access [9, 11, 34]. Researchers have also designed a more evaluation-friendly surrogate task, membership inference attack [29], that tests whether each of the given samples has been included in the training data of the learned model. Subsequent work has shown that this is indeed possible for a wide range of models, including text-generation models [12, 30] and image-generation models [5]. For a comprehensive review of the field up to 2020, refer to the overview by Rigaki & Garcia [27].
2.2 Privacy Leakage in Learned Models: Post-LLM Era
The appearance of billion-scale large-language models (LLMs) and the highly successful products including ChatGPT [22], leads to an even higher level of privacy concerns. Their training data includes not only the data consciously or voluntarily provided by the data subjects [8], but also a massive crawl of the entire web such as personal web pages, social media accounts, personal profiles on online forums, and databases of in-house emails [13]. Building a model-based service on such a web-crawled dataset and making it available to millions of users worldwide poses a novel, serious threat to the data rights of the data subjects. Motivated by this, a few early studies have been made to measure privacy leakage in LLMs [13, 19, 4, 14]. However, although [13] initiated the discussion on PII leakage in LLMs, it was limited to the preliminary analysis of only email addresses. [19] conducted a separate study that specifically targeted LLMs fine-tuned with an auxiliary dataset enriched with PII. Furthermore, their study specifically concentrated on scenarios where the prefix or suffix associated with the PII was known. In contrast, ProPILE aims to provide a more comprehensive tool for probing LLMs already in deployment without LLM fine-tuning or prefix retrieval.
2.3 Prompt Tuning
Prompt engineering [26, 18] improves downstream task performance of LLMs by well-designing prompts without further LLM fine-tuning. In soft prompt tuning [15, 16], a few learnable soft token embeddings concatenated to the original prompts are trained while LLM is frozen, so that more optimal prompts for the downstream task can be obtained. The white-box approach of ProPILE leverages soft prompt tuning to further refine the black-box approach's hand-crafted prompts.
3 ProPILE: Probing PII Leakage of Large Language Models
In this section, we propose ProPILE, a probing tool to profile the PII leakage of LLMs. We first introduce the two attributes of PII, namely linkability and structurality, which are important for the subsequent analysis. We also describe our threat model and eventually introduce probing methods of ProPILE. Finally, we discuss the quantification of the degrees of privacy leakage.
3.1 Formulation of PII
3.1.1 Linkability
From a privacy standpoint, the random disclosure of PII may not necessarily pose a substantial risk. For instance, when a phone number is generated in an unrelated context, there are no identifiable markers linking the number to its owner. However, if targeted PII is presented within a context directly tied to the owner, it could pose a severe privacy risk as it unequivocally associates the number with its owner. In light of this, the linkability of PII items has been considered critical for the study of privacy leakage [24]. We formalize the linkability of PII in the definition below.
Definition 1 (Linkable PII leakage). Let  $\mathcal{A} \coloneqq \{a_1, \dots, a_M\}$  be  $M$  PII items relevant to a data subject  $S$ . Each element  $a_m$  denotes a PII item of a specific PII type. Let  $T$  be a probing tool that estimates a probability of leakage of PII item  $a_m$  given the rest of the items  $\mathcal{A}_{\backslash m} \coloneqq \{a_1, \dots, a_{m-1}, a_{m+1}, \dots, a_M\}$ . We say that  $T$  exposes the linkability of PII items for the data subject  $S$  when the likelihood of reconstructing the true PII,  $\operatorname{Pr}(a_m | \mathcal{A}_{\backslash m}, T)$ , is greater than the unconditional, context-free likelihood  $\operatorname{Pr}(a_m)$ .
3.1.2 Structurality
We consider PII in LLM training data in a string format. Certain types of PII tend to be more structured than others. The structurality of PII has significant implications for practical countermeasures against privacy leakage. We discuss them below.
Structured PII refers to the PII type that often appears in a structured pattern. For example, phone numbers and social security numbers are written down in a recognizable pattern like (xxx) xxx-xxxx that is often consistent within each country. Email addresses also follow a distinct pattern id@domain and are considered structured. Though less intuitive, we also consider physical addresses structured: [building, street, state, country, postal code].
We expect structured PII to be easily detectable with simple regular expressions [1]. This implies apparently simple remedies against privacy leakage. Structured PII may easily be purged out from training data through regular expression detection. Moreover, leakage of such PII may be controlled through detection and redaction in the LLM outputs. However, in practice, the complete removal of structured PII in training data and LLM-generated content is difficult. Regulating the generation of useful public information, such as the phone number and address of the emergency clinic, will significantly limit the utility of LLM services. It is often difficult to distinguish PII and public information that fall within the same pattern category. As such, it is not impossible to find structured PII in the actual LLM training data, such as the Pile dataset (section 4.1) [10], and the leakage of PII in actual LLM outputs [17]. We thus study the leakage of structured PII in this work.
Unstructured PII refers to the PII type that does not follow an easy regular expression pattern. For example, information about a data subject's family members is sensitive PII that does not follow a designated pattern in text. One could write " $\{name1\}$ 's father is  $\{name2\}$ ", but this is not the only way to convey this information. Other examples include the affiliation, employer, and educational background of data subjects. Unstructured PII indeed poses greater threats of unrecognized privacy leakage than structured PII. In this work, we consider family relationships and affiliation as representative cases of unstructured PII (section 4.3).
Our goal is to enable data subjects to probe how likely LLMs are to leak their PII. We organize the relevant actors surrounding our PII probing tool and the resources they have access to.
Actors in the threat model. First of all, there are data subjects whose PII is included in the training data for LLMs. They have their ownership, or the data rights [8], over the PII. LLM providers train LLMs using web-crawled data that may potentially include PII from corresponding data subjects. Finally, LLM users have access to the LLM-based services to send prompts and receive text responses.
Available resources. LLM-based services, especially proprietary ones, are often available as APIs, allowing only black-box access to LLM users. They formulate the inputs within the boundary of rate limit policy and inappropriate-content regulations and receive outputs from the models. On the other hand, LLM providers have white-box access to the LLM training data, LLM training algorithm, and hyperparameters, as well as LLM model parameters and gradients. Data subjects may easily acquire black-box access to the LLMs by registering themselves as LLM users, but it is unlikely that they will get white-box access. Importantly, data subjects have rightful access to their own PII. We show how they can utilize their own PII to effectively probe the privacy leakage in LLMs.
3.3 Probing Methods
We present two probing methods, one designed for data subjects with only black-box access to LLMs and the other for model providers with white-box access.
3.3.1 Black-box Probing
Actor's goal. In a black-box probing scenario, an actor with black-box access aims to probe whether there is a possibility that the LLM leaks one of their PII. Particularly, an actor has a list of their own PII  $\mathcal{A}$  with  $M$  PII items and aims to check if the target PII  $a_{m} \in \mathcal{A}$  leaks from an LLM.
Probing strategy. For a target PII  $a_{m}$ , a set of query prompts  $\mathcal{T}$  is created by associating the remaining PII  $\mathcal{A}_{\backslash m}$ . Particularly,  $\mathcal{A}_{\backslash m}$  is prompted with  $K$  different templates  $t_k$  as  $\mathcal{T} = \{t_1(\mathcal{A}_{\backslash m}), \dots, t_K(\mathcal{A}_{\backslash m})\}$ . Then, the user sends the set of probing prompts  $\mathcal{T}$  to the target LLM for as much as  $N$  times. Assuming the target LLM performs sampling, the user will receive  $N \times K$  responses along with the likelihood scores  $\mathcal{L} \in \mathbb{R}^{K \times L \times V}$ , where  $L$  and  $V$  denote the length of the response and the vocabulary size of the target LLM, respectively. Please note that the likelihood is identical for the same query regardless of repeated queries. Example prompts are shown in Figure 2.
3.3.2 White-box Probing
Actor's goal. In the white-box probing scenario, the goal of the actor is to find a tighter worst-case leakage (lower bound on the likelihood) of specific types of PII  $(a_{m})$ . The actor is given additional resources beyond the black-box case. They have access to the training dataset, model parameters, and model gradients.
Probing strategy. We use soft prompt tuning to achieve the goal, of finding a prompt that induces more leakage than the handcrafted prompts in the black-box case. First, we denote a set of PII lists included in the training dataset of target LLM as  $\mathcal{D} = \{\mathcal{A}^i\}_{i=1}^N$ . White-box approach assumes that an actor has access to a subset of training data  $\tilde{\mathcal{D}} \subset \mathcal{D}$ , where  $|\tilde{D}| = n$  for  $n \ll N$ . Let us denote a query prompt as  $X$  that is created by one of the templates used in the black-box probing  $X = t_n(\mathcal{A}_{\backslash m}^i)$ . Then  $X$  is tokenized and embedded into  $X_e \in \mathbb{R}^{L_X \times d}$ , where  $L_X$  denotes the length of the query sequence and  $d$  denotes the embedding dimension of the target LLM. The soft prompt  $\theta_s \in \mathbb{R}^{L_s \times d}$ , technically learnable parameters, are appended ahead of  $X_e$  making  $[\theta_s; X_e] \in \mathbb{R}^{(L_s + L_X) \times d}$ , where  $L_s$  denotes the number of soft prompt tokens to be prepended. The soft embedding is trained to maximize the expected reconstruction likelihood of the target PII over  $\tilde{\mathcal{D}}$ . Therefore, the training is conducted to minimize negative log-likelihood defined as below:
$$
\theta_ {s} ^ {*} = \underset {\theta_ {s}} {\operatorname {a r g m i n}} \underset {\mathcal {A} \sim \tilde {\mathcal {D}}} {\mathbb {E}} \left[ - \log \left(\Pr \left(a _ {m} \mid \left[ \theta_ {s}; X _ {e} \right]\right)\right) \right]. \tag {1}
$$
After the training, the learned soft embedding  $\theta_s^*$  is prepended to prompts  $t_n(\mathcal{A}_{\backslash m})$  made of unseen data subject's PII to measure the leakage of  $a_{m}$  of the subject.
3.4 Quantifying PII leakage
For both black-box and white-box probing, the risk of PII leakage is quantified using two types of metrics depending on the output that the users receive.
Quantification based on string match. Users receive generated text from the LLMs. Naturally, the string match between the generated text and the target PII serves as a primary metric to quantify the leakage. Exact match represents a verbatim reconstruction of a PII; the generated string is identical to the ground truth PII.
Quantification based on likelihood. We consider the scenario that black-box LLMs can provide likelihood scores for candidate text outputs. The availability of likelihood scores enables a more precise assessment of the level of privacy leakage. It also lets one simulate the chance of LLMs revealing the PII when it is deployed at a massive scale. Reconstruction likelihood implies the probability of the target PII being reconstructed given the query prompt. Therefore, the likelihood defined as follows is used to quantify the leakage:
$$
\Pr \left(a _ {m} \mid \mathcal {A} _ {\backslash m}\right) = \prod_ {r = 1} ^ {L _ {r}} p \left(a _ {m, r} \mid x _ {1}, x _ {2},..., x _ {L _ {q} + r - 1}\right). \tag {2}
$$
In this equation,  $a_{m}$  represents the target PII and the product is taken over the range from  $r = 1$  to  $L_{r}$ , where  $L_{r}$  represents the length of the target PII  $(a_{m})$ .  $x_{1}, x_{2}, \ldots, x_{L_{q} + r - 1}$  correspond to the tokens or words comprising the query prompt of length  $L_{q}$  followed by the response.
Even a low level of likelihood has critical implications for privacy leakage, particularly for systems deployed at scale. For example, ChatGPT has been deployed to more than 100 million users worldwide [23]. The likelihood of  $0.01\%$  of reconstructing the PII implies 100 cases of PII reconstruction if only  $0.01\%$  of the 100 million users attempt the reconstruction 10 times each. $^{1}$  The inverse of the likelihood indicates the expected number of sampling or queries needed to generate the exact PII.
To give a better sense of what the likelihood indicates, we introduce a new metric  $\gamma_{< k}$ . It indicates the fraction of data subjects whose PII is likely to be revealed within  $k$  queries sent. For example,  $\gamma_{< 100,m} = 0.01$  indicates that for approximately  $1\%$  of data subjects, their PII of index  $m$  will be extracted when the LLM is probed 100 times with the same query.
1 0.01 likelihood  $\times \frac{0.01}{100} \times 100 \cdot 10^{6}$  users  $\times 10$  attempts = 10 reconstructions
(a) Black-box probing template examples
Twins(name，pii_1) "The{target_pii_type}of{name}is" "Please contact{name}by{target_pii_type}”,..   
Triplet(name，pii_1，pii_2) "Please contact{name}by{pii_1}or{target_pii_type}" "You can reach{name}at{pii_1}or{target_pii_type}”，.   
Quadruplet(name，pii_1，pii_2，pii_3) "Please contact{name}by{pii_1},{pii_2}or{target_pii_type}" "You can reach{name}at{pii_1}by{pii_2}or{target_pii_type}”，.
(b) Evaluation examples
StructuredPII For further information,please keep in touch with name at address by phone and email.  $\rightarrow$  {type: quaduplet, name: name, pii:{phone,phone, email email,address address} UnstructuredPII name1,name2's father,...  $\rightarrow$  {type: relationship, name: name1, pii:{relation:fatherrname2}} name1 works at name2 and ...  $\rightarrow$  {type: affiliation,pii:{affiliation:name}}
Figure 2: Probing prompts. (a) Black-box probing templates examples for different association levels. Blue text denotes the associated PII to be included in the prompt, and Red text indicates the target PII and the type of it. (b) Examples from the evaluation dataset. Text in Pile dataset is converted to dictionary.
$$
\gamma_ {<   k, m} = \frac {\# \left\{\text {P I I} \mathcal {A} \text {f o r d a t a s u b j e c t s i n} \mathcal {D} \mid \Pr \left(a _ {m} \mid \mathcal {A} _ {\backslash m}\right) > \frac {1}{k} \right\}}{\# \text {o f d a t a s u b j e c t s i n} \mathcal {D}} \tag {3}
$$
4 Probing Existing LLMs
Target LLM to be probed. In our experiments, the selection of the target LLM was guided by two specific requirements. Firstly, in order to assess the probing results, it was necessary for the training dataset of the target LLM to be publicly available. Secondly, to facilitate both black-box and white-box probing, it was essential to have access to pre-trained weights of the target model. To meet these criteria, we opted to utilize the OPT with 1.3 billion hyperparameters (OPT-1.3B) [35] and corresponding tokenizer released by HuggingFace  $[33]^2$  as our target LLM for probing. Please note that the above criteria are for evaluation. In real-world scenarios, ProPILE is not limited to OPT but can be applied to many other LLMs.
Evaluation dataset. This paper conducts experiments using five types of PII: phone number, email address, and (physical) address as instances of structured PII and family relationship and university information as instances of unstructured PII. To evaluate the PII leakage, an evaluation dataset was collected from the Pile dataset, which is an 825GB English dataset included in OPT training data [10]. It is noteworthy that the presence of documents containing all five types linked to a data subject is rare in the Pile dataset. However, for structured PII, there were instances where all three types of structured PII were linked to the name of a data subject. Hence, we extracted quadruplets of (name, phone number, email address, address) from the Pile dataset. Specifically, the PII items are searched with regular expressions and named entity recognition [2, 21]. Examples are shown in Figure 2 (b). For the collection of unstructured PII, we adopted a question-answering model based on  $\mathrm{RoBERTa}^3$  and formulated relevant questions to extract information regarding relationships or affiliations. Only answers with a confidence score exceeding 0.9 were gathered, and subsequently underwent manual filtering to eliminate mistrabeled instances. The final evaluation dataset consists of the structured PII quadruplets for 10,000 data subjects, name-family relationship pairs for 10,000 data subjects, and name-university pairs for 2,000 data subjects. Please refer to the Appendix for the dataset construction details.
4.2 Black-box Probing Results
We show how the black-box probing approach of ProPILE with hand-crafted prompts helps data subjects assess the leakage of their own PII. We also examine the effect of various factors on the leakage.
$^{2}$ https://huggingface.co/facebook
<sup>3</sup>https://huggingface.co/distilbert-base-cased-distilled-squad
Structured Personally Identifiable Information
(b) Unstructured Personally Identifiable Information