Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility

Authors: Martin Kuo, Jingyang Zhang, Jianyi Zhang, Minxue Tang, Louis DiValentin, Aolin Ding, Jingwei Sun, William Chen, Amin Hass, Tianlong Chen, Yiran Chen, Hai Li
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large language models, Personal identifiable information, Protect private data leakage

Abstract:
With the rise of large language models (LLMs), increasing research has recognized
their risk of leaking personally identifiable information (PII) under malicious
attacks. Although efforts have been made to protect PII in LLMs, existing methods
struggle to balance privacy protection with maintaining model utility. In this paper,
inspired by studies of amnesia in cognitive science, we propose a novel approach,
Proactive Privacy Amnesia (PPA), to safeguard PII in LLMs while preserving their
utility. This mechanism works by actively identifying and forgetting key memories
most closely associated with PII in sequences, followed by a memory implanting
using suitable substitute memories to maintain the LLM’s functionality. We conduct
evaluations across multiple models to protect common PII, such as phone numbers
and physical addresses, against prevalent PII-targeted attacks, demonstrating the
superiority of our method compared with other existing defensive techniques. The
results show that our PPA method completely eliminates the risk of phone number
exposure by 100% and significantly reduces the risk of physical address exposure
by 9.8% – 87.6%, all while maintaining comparable model utility performance.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730578828823

thread_type:
official_review

<thread_object>
creator: reviewer_j1tm

<content>
summary:
The article introduces a novel method to protect personally identifiable information in language models without compromising performance. Inspired by amnesia research, the PPA approach selectively forgets sensitive PII while preserving model utility. It involves three main steps: sensitivity analysis to identify critical elements within PII, selective forgetting of these elements, and memory implanting with non-sensitive information to maintain functionality. Experimental results on phone numbers and addresses show that PPA effectively reduces privacy risks and maintains high model performance compared to other methods.

strengths:
-   The approach is distinctive as it draws from cognitive science concepts, such as anterograde amnesia, to design a targeted forgetting mechanism. This use of selective forgetting combined with memory implanting to balance utility and privacy represents a novel adaptation.
- The paper supports its claims through a well-designed series of experiments on models like LLaMA2 and LLaMA3, benchmarked against multiple datasets (Enron and Fraud email datasets) and evaluated using diverse attack methods.
-  The PPA method’s adaptability, allowing control over the degree of forgetting, offers practical value by enabling fine-tuning based on specific privacy requirements.

weaknesses:
-  The study primarily focuses on phone numbers and physical addresses as examples of PII. While this provides a starting point, PII often includes more complex and variable types, such as emails, social security numbers, or unique identifiers that may be more difficult to detect and selectively forget.  
-  The effectiveness depends on accurately identifying "key elements" within PII sequences. However, the method lacks a detailed discussion on the robustness of this sensitivity analysis, especially for more nuanced or less structured PII.  
-  Implementing selective forgetting and memory implanting on large-scale models (e.g., with billions of parameters) can be computationally intensive. The paper does not address potential scalability challenges or discuss optimization strategies for efficiently applying PPA to larger models.
- As a last and perhaps pedantic point, the table captions are lacking. There are too many tables, and their captions and discussion are tedious at best. In general, the metrics are not described well, and there are columns such as risk score that list values, and these are not clear. Columns could have down or up arrows to indicate if higher is better or the reverse. 
- Figures are not of high quality. For example, figure 3 is not legible on paper.

questions:
- How scalable is this approach? 
- How are the values in Table 1 decided?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685264876

thread_type:
official_review

<thread_object>
creator: reviewer_iqmj

<content>
summary:
This work studies the unlearning of personally identifiable information (PII) in large language models (LLMs), proposing a three-step approach called Proactive Privacy Amnesia (PPA). Given target PIIs to forget, the approach first identifies the key elements in memorized PII, then focuses exclusively on forgetting those key elements, and finally implants alternative information.

strengths:
1. The problem of unlearning in LLMs is critical for conforming to the "right to be forgotten" legal principle and is essential for protecting personal information from misuse by LLM producers.

2. The paper introduces the concept of the "memorization factor" to quantify the sensitivity of PII.

weaknesses:
Given the numerous unlearning targets, the resulting unlearned model may degrade in various aspects. While current evaluation metrics for LLM performance include perplexity and email completion, additional metrics are needed to measure the utility loss due to unlearning. Specific metrics from the paper "Knowledge Unlearning for Mitigating Privacy Risks in Language Models" could be particularly relevant. Consider including metrics such as TruthfulQA and HellaSwag.

questions:
1. In some unlearning tasks, such as Physical Address Defense in Table 4, PPA does not appear to outperform DEPN. A more detailed analysis of the cases where PPA underperforms/outperforms DEPN is warranted.

2. A question arises regarding the security of the unlearning paradigm. The unlearning of specific PII is reasonable if the PII was indeed learned by the LLM. How can individuals or LLM producers verify that the PII has been learned? Additionally, is the proposed concept of the "memorization factor" robust enough to detect unreasonable PII unlearning queries?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707700220

thread_type:
official_review

<thread_object>
creator: reviewer_pc5a

<content>
summary:
In this paper, the authors propose a scheme to selectively protect PII in Large Language Model training data. The approach consists of three major components: sensitivity analysis, selective forgetting, and memory implantation. The results are evaluated across multiple LLM attacks, demonstrating that the defense shows promising effectiveness.

strengths:
1. Has theoretical justification for methods. It's impressive that the authors clearly explain the method's concept with theoretical justification, which definitely helps readers understand.
2. Includes multiple attacks for evaluation. The authors consider multiple attacks in the evaluation, showcasing that the proposed method is robust under different settings.
3. Results seem promising with utility-privacy tradeoff. Based on the results, the proposed method appears promising, though slightly less effective than some prior work in certain scenarios. The authors explain the advantages of the proposed solution well.

weaknesses:
1. Method seems straightforward; each component seems lacks novelty. The sensitivity analysis appears to follow standard definitions of PII in language models. The selective forgetting component proposes a loss function, but it is relatively simple, and the memory implanting seems directly referenced from prior works. Additional insights and modifications to tailor these components specifically for PII protection may be needed to make the method more innovative. For example, consider designing the memory implanting to optimize performance for different types of PII. Similar innovations should be highlighted for each component.
2. Evaluation may benefit from additional metrics, such as exposure. I feel that including more metrics beyond attack metrics and model performance metrics would provide more comprehensive insights into the proposed method. For instance, adding an exposure metric, as discussed in [1], could measure the memorization and likelihood of extraction when partial information is not protected. This evaluation would also better align with the design of other baselines compared in this work and help readers better understand the effectiveness of the proposed method.
3. No comparison with differential privacy-based methods. There seems to be a lack of comparison with differential privacy-based methods in the evaluation, such as [2]. Since differential privacy is a mainstream defense approach, comparing it with selective-DP should also be considered in this paper. More discussion and comparisons on the privacy guarantee and utility to highlight the pros and cons of the proposed method and the dp-based methods will be appreciated.

[1] Carlini, Nicholas, et al. "The secret sharer: Evaluating and testing unintended memorization in neural networks." 28th USENIX security symposium (USENIX security 19). 2019.

[2] Shi, Weiyan, et al. "Just Fine-tune Twice: Selective Differential Privacy for Large Language Models." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

questions:
1. Will the method leak any other user's PII when it's not protecting all samples, causing some originally safe information to be exposed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710460452

thread_type:
official_review

<thread_object>
creator: reviewer_da8v

<content>
summary:
The paper proposes Proactive Privacy Amnesia, to unlearn PII in LLM while preserving its utility. The framework is composed of sensitivity analysis, selective forgetting, and memory implanting. The empirical results demonstrate the effectiveness in eliminating the privacy leakage risk.

strengths:
- The sensitivity analysis is interesting and inspiring, minimizing the impact on model performance.
- The proposed framework outperforms the baselines in terms of utility and defense ability.

weaknesses:
- The authors can compare their framework with differentially private decoding [1] and finetuning [2] methods, which also prevent the model from outputing sensitive information.

- Limited evaluation on model utility. The author only evaluate the model's utility on the same unlearning dataset, i.e., Enron Email. However, it's desirable to evaluate the model's performance on general downstream tasks, such as the GLUE and MMLU dataset.

- The sensitivity analysis aims to isolate tokens that carry a higher amount of information. What if the attacker has some prior knowledge of the phone number/address? It may increase the attack success rate by conditioning the prediction on the former part of the private information. Furthermore, would unlearning the latter part, i.e., the tokens after the top-k, lead to better performance?

[1] Majmudar, J., Dupuy, C., Peris, C., Smaili, S., Gupta, R., & Zemel, R. (2022). Differentially private decoding in large language models. arXiv preprint arXiv:2205.13621.

[2] Li, Xuechen, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. "Large Language Models Can Be Strong Differentially Private Learners." In International Conference on Learning Representations.

questions:
- The sensitivity analysis is expected to reduce the impact on utility of the LLM. However, in Table 5, Unlearning + Memory Implanting shows model similar performance with Proactive Privacy Amnesia, with increased attack success rate. How to interprete such result? Why the sensitivity analysis could decrease the attack score?

- What's the average proportion of the top-k token in terms of the total length of the private information?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732261696294

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Statement

comment:
Dear Reviewers,

We sincerely thank you for your valuable comments and suggestions. We appreciate the opportunity to address your concerns and improve our manuscript. The changes have been highlighted in blue for text and orange for tables and figures. Below, we outline the major revisions we have made:

**1. Adding Comparison with Differential Privacy-Based Methods.** Based on the reviewers' suggestions, we compare our method with "Differentially private decoding in large language models" (DP Decoding) [1] and "Just fine-tune twice: Selective differential privacy for large language models" (JFT) [2].

To evaluate these methods, we applied probing attacks to both DP Decoding and JFT. We observed that our PPA method outperformed both DP Decoding and JFT, achieving a lower risk score and a higher utility score. *Detailed results and discussion can be found in Appendix E.*
 

**2. Adding additional model utility evaluation on TruthfulQA [5] and MMLU [6].** Based on the reviewers' suggestions, we evaluated the model's performance metrics on both TruthfulQA and MMLU. We employed GPT-4o to assess the responses from all baseline defense methods. Our findings indicate that PPA achieves the highest TruthfulQA and MMLU scores among all baseline defense methods. *Detailed results can be found in Appendix F.*

**3. Adding the Exposure Metric [3].** As recommended by the reviewers, we calculated the exposure metric for all baseline methods. Since calculating the exposure of PII is computationally intensive, we followed the approach in Table 2 of [3] and evaluated the exposure for 10 phone numbers. Our results show that PPA outperforms other baseline defense methods. *Detailed results can be found in Appendix J.*

**4. More evaluation on different PII.** Based on the reviewers' suggestions, we conducted an additional experiment to evaluate the protection of 281 users' email addresses in the aeslc training dataset. Using Levenshtein distance [4], we compared the predicted email addresses to the ground truth. PPA successfully defends all users' email addresses against probing attacks while maintaining model performance comparable to other baseline defense methods. *Detailed results can be found in Appendix K*.

Reference:

[1] Majmudar, J., Dupuy, C., Peris, C., Smaili, S., Gupta, R., & Zemel, R. (2022). Differentially private decoding in large language models. arXiv preprint arXiv:2205.13621.

[2] Shi, W., Shea, R., Chen, S., Zhang, C., Jia, R., & Yu, Z. (2022). Just fine-tune twice: Selective differential privacy for large language models. _arXiv preprint arXiv:2204.07667_.

[3] Carlini, Nicholas, et al. "The secret sharer: Evaluating and testing unintended memorization in neural networks." 28th USENIX security symposium (USENIX security 19). 2019.

[4] Po, Daw Khin. "Similarity based information retrieval using Levenshtein distance algorithm." _Int. J. Adv. Sci. Res. Eng_ 6.04 (2020): 06-10.

[5] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

[6] Lin, Stephanie, Jacob Hilton, and Owain Evans. "Truthfulqa: Measuring how models mimic human falsehoods." arXiv preprint arXiv:2109.07958 (2021).

**We sincerely appreciate your thoughtful suggestions, which have encouraged us to think more deeply about our work. We welcome any additional feedback and are more than happy to address any further questions or concerns you may have.**

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary of Contributions

This paper proposes a method to unlearn / protect personally identifiable information (PII) in large language models (LLMs). The method--named Proactive Privacy Amnesia (PPA)--is inspired by anterograde amnesia. The method consists of the following steps: (i) identify sensitive token, (ii) selectively forgetting this token and (iii) implanting false memory instead. The authors perform experiments for their method against previous work, in the settings where PIIs are phone number and physical addresses. They demonstrate that the new approach is more effective (in terms of protection and model utility) compared to previously proposed methods.

## Strengths

- Unlearning in LLMs is an important and timely topic as continued rollouts of (many forms of) LLMs can threaten user's privacy.

- The proposed method is novel and innovative, and should lead to further investigation.

- Some steps, such as sensitivity analysis, has theoretical justifications.

## Weaknesses

- For the original submission, the experiments were not as comprehensive as one would have hope (e.g. no comparison with DP-based methods); this has been addressed during the rebuttal. In addition to this, the improvements are not uniform in all settings, e.g. DEPN is comparable / better in some cases.

- Although the combination of steps are innovative, it seems like "selective forgetting" and "memory planting" themselves are not new and have appeared in literature.

- The presentation can be improved. For example, the figures and captions are not as clear as one would hope. The methods--especially the "selective forgetting" and "memory planting" steps--are not described as clear as it could be (e.g. once we the loss function in (6) what do we do next? gradient descent?). Moreover, some notations are not clearly defined (e.g. $p, q$ in Equation (1)). 

## Recommendation

Given that the problem is important, the method is innovation and shows promising empirical results, we recommend acceptance.