SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Black-box probing result in likelihood perspective. Reconstruction vs. baseline likelihood of (a) structured PII and (b) unstructured PII, shown with the average likelihood and the p-value of the Wilcoxon signed-rank test. (c) shows a summary of the likelihoods using  $\gamma_{<k}$  defined in Equation 3.
Likelihood results. We first evaluate the likelihood of the target PII item given the other items of a subject. Then, we consider the black-box LLM as revealing the linkable PII item, if the likelihood probability is greater than that of randomly selected PII instances, i.e.,  $\operatorname{Pr}(a_m|\mathcal{A}_{\backslash m}) > \operatorname{Pr}(a_{m,\mathrm{Null}}|\mathcal{A}_{\backslash m})$ . The  $a_{m,\mathrm{Null}}$  is randomly selected from the evaluation dataset. We utilized the aforementioned evaluation dataset and created prompts using five different triplet templates, including those described in Figure 2 (a). Subsequently, the generation is done using beam search with a beam size of 3. The likelihood was computed using Equation 2. Please refer to the Appendix for the detailed generation hyperparameters and prompt templates.
Figure 3 (a-b) illustrates the density plot of the likelihoods. The blue and orange color represents the target PII  $(a_{m})$  and randomly chosen PII  $(a_{m,\mathrm{null}})$ , respectively. The plots also display the mean likelihood values. It is observed that the mean likelihood of target PII is higher than that of the null PII for all PII types. We also denoted the p-value obtained from the statistical test using the Wilcoxon signed rank test [7]. The small p-value suggests that the observed difference is statistically significant except for affiliation. Figure 3 (c) shows  $\gamma_{<k}$ . We have mentioned in section 3.4 that the x-axis variable,  $k$ , can be interpreted as the number of queries. As the number of queries increases, we observe a gradual increase in the frequency of exact reconstruction.
The above black-box probing results demonstrate a high risk of reconstructing the exact PII based on available PII items and establishing the link. The results of  $\gamma_{<k}$  indicate that despite the seemingly low likelihood values, there is a possibility of exact reconstruction of PII.
Exact match results. Through black-box probing, the generated sequences can be obtained. The exact match can be assessed by evaluating whether the generated sequence includes the exact string of target PII or not. First, we evaluated the exact match with a varying number of templates used to construct the prompts. Results are shown in Figure 4 (a). The rate of exact matches increases as the diversity of prompt templates increases. This also supports the rationale behind white-box probing, as it suggests that finding more optimal prompts can further increase the leakage.
Furthermore, we conducted an assessment of exact matches when different levels of the associations were present in the prompt. Figure 4 (b) shows the results. The "twins" denotes that only the name of a data subject is used to make the query prompt, while "triplet" indicates the presence of an additional PII item in the prompt. We can observe a fivefold increase in the exact match rate for the
(a) Number of prompts
Figure 4: Black-box probing results in string-match perspective. The proportion of PII that is exactly reconstructed through black-box probing. We vary (a) the number of query prompts, (b) the level of associated PII items in the query prompt, (c) the beam size for decoding and (d) the size of the targeted LLM.
(b) Association level
(c) Number of beams
(d) Model size (OPT)
email address. This increase occurred when a phone number, which offers more specific information about the data subject, was provided in addition to the name. In the case of phone numbers, we also observed an increase of more than double. This shows increasing information in the prompts that can be associated with the target PII elevates the leakage. It also supports the effectiveness of black-box probing that utilizes the data subject's linkable PIIs. Furthermore, with increased beam search sizes in the model (Figure 4 (c)) and larger model sizes (d), the frequency of the target PII appearing in generated sentences also tends to rise. The increasing leakage that occurs with larger model sizes can be attributed to improved accuracy. This implies that as the current trend of scaling up large language models continues, the potential risks of PII leakage may also increase.
4.3 White-box Probing Results
In this section, we demonstrate the white-box probing by presenting the leakage of the phone number given other PII information in the structured quadruplet. We train 20 embedding vectors for the soft prompts by appending them ahead of a single prompt to generate the target phone number; We use additional 128 quadruplet data that are not included in the evaluation dataset. Please refer to Appendix for the training details. With the trained soft prompts, we measure the likelihood probabilities and exact match ratios on the evaluation dataset. Figure 5 summarizes the results in terms of the number of training data, the number of soft tokens, and the initialization type.
Efficacy of soft prompt tuning. Figure 5 illustrates the impact of the soft prompt on the exact match rate and reconstruction likelihood, with blue and orange colors, respectively. The results indicate a significant increase, from  $0.0047\%$  of black-box probing using five prompt templates to  $1.3\%$  with the soft prompt learned only from 128 data points being prepended to a single query prompt. The likelihood also increased by a large amount for the same case. It is speculated that the observed increase can be attributed to the soft prompt facilitating the more optimal prompts that may not have been considered by humans during the construction of prompts in black-box probing.
Effect of dataset size. The white-box probing scenario assumes that a user (or a service provider) has access to only a small portion of the training data. To see the impact of the number of data used for tuning to the degree of the leakage, soft prompts were trained using different numbers of triplets in the training dataset, specifically [16, 32, 64, 128, 256, 512]. The results are depicted in Figure 5 (a). Even with 16 data points, a significant surge in leakage was observed. The exact match rate escalated to  $0.12\%$ , surpassing the exact match scores achieved by using five prompts, as well as in terms of likelihood. As the training set size increases from 16 to 128, the exact match dramatically increases from  $0.12\%$  to  $1.50\%$ . This finding indicates that even with a small fraction of the training dataset, it is possible to refine prompts that can effectively probe the PII leakage in LLM.
Additional analysis of soft prompt tuning. We also examine the impact of different factors on the leakage and Figure 5 (b) and (c) display the leakage levels according to these factors. As the number of soft tokens increases, the leakage also exhibits an increasing trend. This can be attributed to the enhanced expressiveness of the soft prompts, which improves as the number of parameters increases. Furthermore, different initialization schemes produce diverse outcomes. We investigated three initialization schemes: 1) an embedding of the word representing the specific type of target PII, i.e., "phone", which was the default setting throughout our experiments, 2) an embedding sampled from a uniform distribution  $\mathcal{U}(-1,1)$ , and 3) utilizing the mean of all vocabulary embeddings. As
Figure 5: White box probing results. Leakage results on 10,000 unseen triplets according to (a) varying number of data used for prompt tuning, (b) number of soft tokens, (c) different initialization type. Blue and orange color denotes exact match rate and likelihood, respectively.
Table 1: Transferability of soft prompt. Original denotes the black-box probing results using one query prompt and transfer denotes the probing results using the transferred soft prompt that is learned from the source model (OPT-1.3B).  $\times$  columns show how much the leakage likelihood increases by using the transferred soft prompt.
Source	Target	Avg. Likelihood	# Exact match
Original	Transfer	×	Original	Transfer
OPT-1.3B	OPT-350M	1.05 × 10-11	1.08 × 10-10	7.5	0	0
OPT-1.3B	6.06 × 10-8	3.47 × 10-6	57.3	5	3
OPT-2.7B	1.39 × 10-7	2.18 × 10-6	15.6	14	15
illustrated in Figure 1(c), the uniform and mean initialization schemes were unable to raise the leakage. In contrast, initializing with the PII type resulted in the most significant leakage.
Transferability test. If the soft embedding learned for one language model can be reused to probe a different language model, it opens up the possibility of applying the knowledge acquired from white-box probing to black-box probing. To assess the feasibility of this approach, we transferred the soft prompt learned for the OPT-1.3B model to OPT models with different scales, namely OPT-350M and OPT-2.7B. However, directly plugging the soft embedding trained on one model into another model is impossible due to the mismatch of embedding dimensions (e.g., 1, 024 and 512 for OPT-1.3B and OPT-350M, respectively.) To address this, we follow a two-step process of the previous approach [20]. We project the soft embedding to the closest hard tokens in terms of Euclidean distance and decode it to raw string with the source model's tokenizer. The string is then concatenated ahead of the raw query text and fed into the target model.
Table 1 demonstrates that the soft prompt learned from the OPT-1.3B model increases the leakage of the same type of PII in both the OPT-350M and OPT-2.7B models. The increase in leakage is also denoted with the multiplication symbol  $(\times)$ , showcasing how many times the reconstruction likelihood is amplified when utilizing the soft prompt learned for OPT-1.3B in the other models. While there may not be a substantial difference from the exact match perspective, the potential for transferability has been confirmed in the perspective of likelihood. Future work could explore research for investigating white-box probing techniques for enhancing transferability.
This paper introduces ProPILE, a novel tool designed for probing PII leakage in LLM. ProPILE encompasses two probing strategies: black-box probing for data subjects and white-box probing for LLM service providers. In the black-box probing approach, we strategically designed prompts and metrics so that the data subjects can effectively probe if their own PII is being leaked from LLM. The white-box probing approach empowered LLM service providers to conduct investigations on their own in-house models. This was achieved by leveraging the training data and model parameters to fine-tune more potent prompts, enabling a deeper analysis of potential PII leakage. By conducting actual probing on the OPT-1.3B model, we made several observations. First, we found that the target PII item is generated with a significantly higher likelihood compared to a random PII item. Furthermore, white-box probing revealed a tighter worst-case leakage possibility in terms of PII leakage. We hope that our findings empower the data subjects and LLM service providers for their awareness and control over their own data on the web.
Limitations. The construction of the evaluation dataset exclusively involved the use of private information sourced from open-source datasets provided by large corporations. This approach ensures the ethical acquisition of data. However, it's important to acknowledge that the data collection process itself was heuristic in nature. Consequently, the evaluation dataset may contain instances of incorrectly associated data or noise. This could introduce a degree of uncertainty or potential inaccuracies, which must be taken into account when interpreting the results.
Societal Impact. We emphasize that our proposed probing strategies are not designed to facilitate or encourage the leakage of PII. Instead, our intention is to provide a framework that empowers both data subjects and LLM service providers to thoroughly assess the privacy state of current LLMs. By conducting such evaluations, stakeholders can gain insights into the privacy vulnerabilities and potential risks associated with LLMs prior to their deployment in a wider range of real-world applications. This proactive approach aims to raise awareness among users, enabling them to understand the security and privacy implications of LLM usage and take appropriate measures to safeguard their personal information.
6 Ethical Considerations
The evaluation dataset used in our paper is collected from the Pile dataset and thus follows the data regulation and terms of services of it. Specifically, all sources of the Pile dataset are public, with the majority following the terms of services, and  $55\%$  of them are authorized by data owners. Please refer to [10] for the details.
ProPILE's primary intent is to empower data subjects to verify whether their own information may leak from LLM services. However, the black-box probing approach could be abused by malicious actors aiming to extract others' personal information from LLMs without proper consent. Although attackers remain incapable of discerning whether the extracted information belongs to the target, they may still obtain sequences with potential connections with the target. Additionally, the chance of successful attacks with limited information is not entirely eliminated, although it is very low according to the associativity experiment results in Section 4.2.
Here we suggest some guidelines to use and develop ProPILE properly. ProPILE should only be used by the data subjects themselves to examine the reconstruction of PII items linked to them, or by the individuals who have been given explicit consent from the relevant data subjects. One way to implement this is to insert an authentication layer over ProPILE to allow its usage only to users identified through Google accounts; they are only allowed to submit queries using only the information from the user's verified Google profile. Ultimately, the user's good intentions are the most important. We strongly recommend that ProPILE be used only for its intended purpose of enhancing the self-awareness of data subjects.
This work was supported by Naver Corporation, the National Research Foundation of Korea (NRF) grants funded by the Korea government (MSIT) (2022R1A3B1077720, 2022R1A5A708390811), Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (2021-0-01343: AI Graduate School Program, SNU, 2022-0-00959), the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2023, and the Luxembourg National Research Funds (FNR) through CORE project C18/IS/12669767/STELLAR/LeTraon.
[1] Amazon Web Services. Detecting and redacting pii using amazon comprehend. https://aws.amazon.com/ko/blogs/machine-learning/detecting-and-redacting-pii-using-amazon-comprehend/, 2023.
[2] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc.", 2009.
[35] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[36] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 253–261, 2020.
[37] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.