Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility
Authors: Martin Kuo
Jingyang Zhang
Jianyi Zhang
Minxue Tang
Louis DiValentin
Aolin Ding
Jingwei Sun
William Chen
Amin Hass
Tianlong Chen
Yiran Chen
Hai Li
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
Personal identifiable information
Protect private data leakage
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: With the rise of large language models (LLMs), increasing research has recognized
their risk of leaking personally identifiable information (PII) under malicious
attacks. Although efforts have been made to protect PII in LLMs, existing methods
struggle to balance privacy protection with maintaining model utility. In this paper,
inspired by studies of amnesia in cognitive science, we propose a novel approach,
Proactive Privacy Amnesia (PPA), to safeguard PII in LLMs while preserving their
utility. This mechanism works by actively identifying and forgetting key memories
most closely associated with PII in sequences, followed by a memory implanting
using suitable substitute memories to maintain the LLM’s functionality. We conduct
evaluations across multiple models to protect common PII, such as phone numbers
and physical addresses, against prevalent PII-targeted attacks, demonstrating the
superiority of our method compared with other existing defensive techniques. The
results show that our PPA method completely eliminates the risk of phone number
exposure by 100% and significantly reduces the risk of physical address exposure
by 9.8% – 87.6%, all while maintaining comparable model utility performance.

Published as a conference paper at ICLR 2025
PROACTIVE PRIVACY AMNESIA FOR LARGE LAN- GUAGE MODELS: SAFEGUARDING PII WITH NEGLI- GIBLE IMPACT ON MODEL UTILITY
Martin Kuo"; Jingyang Zhang'; Jianyi Zhang', Minxue Tang', Louis DiValentin', Aolin Ding", Jingwei Sun?, William Chen', Amin Hass Tianlong Chen², Yiran Chen', Hai Li
'Center for Computational Evolutionary Intelligence, Duke University Accenture Cyber Labs
University of North Carolina at Chapel Hill CCary Academy
Project Website
ABSTRACT
With the rise of large language models (LLMs), increasing research has recognized their risk of leaking personally identifiable information (PII) under malicious attacks. Although efforts have been made protect PII in LLMs, existing methods struggle to balance privacy protection with maintaining model utility. In this paper, inspired by studies of amnesia in cognitive science, we propose a novel approach, Proactive Privacy Amnesia (PPA), to safeguard PII in LLMs while preserving their utility. This mechanism works by actively identifying and forgetting key memories most closely associated with PII in sequences, followed by memory implanting using suitable substitute memories to maintain the I.I.'s functionality. We conduct evaluations across multiple models to protect common PII, such phone numbers and physical addresses, against prevalent Pll-targeted attacks, demonstrating the superiority of our method compared with other existing defensive techniques. The results show that our PPA method completely eliminates the risk of phone number exposure by 100% and significantly reduces the risk of physical address exposure by 9.8% 87.6%, all while maintaining comparable model utility performance.
INTRODUCTION
Large Language Models (LLMs) (Touvron et al. 2023 Achiam al. 2023 Team al. 2023 Dubey ct al. 2024) have achieved remarkable success in recent years, with their wide adoption either as general-purpose models or, after fine-tuning, as specialized and personal assistants. Despite their success, LLMS with huge parameter counts and great capacity in the meantime exhibit the concerning "memorization" phenomenons (Carlini et al., 2019 (02T). i.e., they can precisely memorize some training data. Such memorization is vulnerable to various attacks (e.g., membership inference attacks and data extraction attacks) and risks severe privacy breaches. One of the most serious concerns comes from the attacks that aim to extract personal identifiable information (PII) memorized by the models, which compromise users' privacy and are likely to cause real-world harm consequently.
To defend against such PII data extraction attacks, several machine unlearning techniques have been applied to LLMs. However, existing methods typically fall short in terms of the trade-off between the defense performance and model utility. For example, most unlearning approaches based on gradient ascent (Jang et al., 2022] Wang et al., 2024) and often adversely affect model functionalities an extent where the model cannot handle their original tasks anymore and thus becomes no longer useful. In contrast, although not harmful to the model utility, gradient descent methods (Patil et al. 2023 Ouyang et al., 2022 De Cao al. 2021 may inject less robust defense, leaving the model still vulnerable to data extraction attacks. Therefore, method that can effectively defend against PII extraction attacks while maintaining model utility still lacking.
In this work, we fill this gap by proposing a novel methodology, called Proactive Privacy Amnesia (PPA). Inspired by Anterograde Amnesia (Markowitsch) 2008), we think that achieving better bal- ance between performance and privacy protection requires two essential components: (1) selectively forgetting only the key element within the PII, without affecting other tokens; and (2) maintaining
"Equal Contribution.

---

Published as a conference paper at ICLR 2025
Attack prompt
Pretrained LLM
713-853-6247 101 California, Suite 1950 Houston, 77002
phone number
John
3
The address of Jeffrey Dasovich
Pretrained LLM
45411 Johnson Ridges Suite 307 North Michelle, ND 59255
defense by PPA
PPA
Sensitivity Analysis + Selective Forgetting
Memory Implanting John Griffith number 763.917.7243
John Griffith phone number (713) 853-6247
① Jeffrey Dasovich address 588 Miguel Plaza Moorestad, ME 53125
Jeffrey Dasovich 101 California St. Suite 1950
Figure 1:
The flowchart illustrates our method, Proactive Privacy Amnesia (PPA). All examples presented in the flowehart are real instances from the LLaMA2-7b experiments.
normal functionality by replacing sensitive information with non-sensitive memory. To seamlessly integrate these components, our method, PPA, as shown in Figure T comprises three parts: (1) Sensitivity Analysis, which identifies the key elements in memorized PII: (2) Selective Forgetting, which focuses exclusively forgetting on the key elements: and (3) Memory Implannting, strategy used to compensate for loss in model performance due to the Selective Forgetting process. We demonstrate the effectiveness of our method through extensive experiments on LLaMA2 (Touvron al.) 2023) and LLaMA3 3 (Dubey al.) 2024) models to defend existing Pll-targeted attacks common PII, such as phone numbers and physical addresses. Extensive experimental results demonstrate that our method, PPA, achieves the most favorable balance between defense capability and model performance when compared to other prevalent defensive methods.
For example, in the Enron email experiment for phone number defense, PPA enhances model performance by 372.7% compared to methods with mediocre model utility while maintaining the same level of defense in terms of risk score. Additionally, PPA achieves 100% reduction in risk score, outperforming methods having mediocre defense effectiveness without compromising model utility. For physical address defense the same experiment, PPA increases model performance by 260.0% compared to methods with mediocre model utility and increase the risk score by 151.7%. Furthermore, PPA surpasses methods with mediocre defense effectiveness by achieving 26.2% reduction in risk score, with only 29.4% decrease in model performance.
Our contributions are as follows:
We propose a novel method PPA that can preserve a person's 's PII on LLMs while maintaining LLMs' performance.
We conducted input rephrasing, probing, and soft prompt attacks to evaluate the effectiveness of PPA approach. The PPA effectively safeguards phone numbers and physical addresses, with only a marginal drop in LLMs' performance.
We introduce the concept of the 'memorization factor' and use it to identify the key elements within PII sequences that influence the model's ability to retain such information. This approach is using in sensitivity analysis and supported by theoretical justification.
PPA a flexible method that enables adjusting the balance between defense capability and model performance by modifying the number of key elements to be forgotten.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
2.1 LLM DATA EXTRACTION ATTACKS
Training data extraction attack (Carlini et al. 2021) first uses GPT-2 (Radford et al., 2019) with designed prompts to generate sets of sentences and subsequently use an improved membership inference method to detect which generated sentences are from the training dataset. However, this paper focuses attacking general privacy information, our study specifically targeted person's 's PII. Black-box Probing (Kim et al. 2024) employs manual prompts to extract person's 's PII from LLMs. Meanwhile, Input Rephrasing attack (Patil et al. (003)) uses paraphrasing model from Krishha et al.) (2024) to rephrase attack prompts. White-bxx Probing (Kim et al. 2024) trains soft prompts from the targeted model using black-box templates and employs these soft prompts to attack a person's PII.
2.2 POST-PROCESSING DEFENSE METHODS
Gradient Based method There several types of gradient based method: 1) Gradient Descent
Method. The Empty Response Defense (Patil et al. 2023) Ouyang et al. 2022) uses gradient descent increase the probability of generating predefined "empty" response like "I don't know." Similarly, the Error Injection method (De Cao et al. 2021) increases the likelihood of generating false target responses through gradient descent. But these methods cannot protect user's PII effectively. 2) Gradient Ascent Method. Jang et al. 2022) apply gradient ascent on sequences of target tokens to unlearn specific knowledge. Wang al. (2024) highlights the risk of embedding general knowledge within personal data and suggests using sensitivity testing to target specific sequence spans for unlearning, rather than entire instances. However, (Jang et al. (2022''s method may lead to model collapse the target set size grows. 3) Combination of Gradient Descent and Ascent. A more complex approach is outlined by Yao et al. (2023), involving three loss types: gradient ascent on the forgetting dataset, random smooth loss, and gradient descent on normal dataset to maintain model performance. Chen & Yang 2023) introduce unlearning layers into transformer architectures and perform gradient ascent on these layers while applying gradient descent on the retained dataset to prevent degradation. Additionally, Yao et al.] (2024) show that combining gradient ascent and descent improves hyperparameter robustness. Notably, these methods require additional dataset to preserve model performance.
Memory Editing method. Wu et al. (2023) introduces privacy neuron detector designed identify and eliminate neurons that significantly contribute to privacy leakage, protecting user data privacy. However, this approach becomes tinecconumnng when applied to extensive user data and may reduce model performance due to the extensive deletion of neurons. Patil et al. (2023) introduce the Head Projection Defense method, which addresses the issue of privacy information potentially residing within model's 's intermediate layers. They employ interpretability techniques from Geva jet al. (2020) to identify the top-k possible tokens in each layer and develop a loss function aimed at preventing the reoccurrence of deleted answers in each layer. However, this method is limited to single-token scenarios, which may not be practical real-world situations where private information could involve multiple tokens.
THREAT MODEL
Attacker's goal: We consider a scenario where LLM is trained on the dataset that includes diverse types of personal identifiable information (PII), such as phone numbers and physical addresses. The attacker's 's goal is construct prompts that are likely to reveal sensitive information from an LLM through its responses. These attacks can lead to the partial or complete exposure of set of PII for given context, such as several digits or the entirety of target phone number, which can be leveraged by attackers to learn user privacy or even re-identify users.
Attacker's capability: We consider both probing and soft prompt attackers. Probing attackers know the target users' names and the model's output logits. They use set of prompts to query an LLM (im et al., 2024), exposing the user's PII in its responses. Soft prompt attackers, in addition to knowing the target users' names and the model's output logits, have access to the model and an additional dataset to train soft prompts (Kim et al., 2024). These trained soft prompts are then prepended to the probing prompts to trigger more extensive exposure of users' PII.
3

---

Published as a conference paper at ICLR 2025
To ensure that our attacks are realistic and account for rate limits and other query restrictions, assume that the attacker operates with a limited budget for query prompts. We also consider that Plls with similar data attributes present comparable risks of data leakage. For instance, techniques effective extracting phone numbers could potentially be applied to reveal social security numbers credit card numbers, these types of Plls are all purely numerical in nature.
attacker's
PROACTIVE PRIVACY AMNESIA
In this section, introduce method, PPA. We begin by discussing the inspiration behind our approach, which identifies key elements within PII sequence that determine whether the sequence can be memorized by the model. Identifying these key elements enables to present unique and theoretically grounded approach to solving the problem. Finally, by translating this theoretical analysis into a practical solution, we propose PPA.
4.1 INSPIRATION AND OVERVIEW
Our Proactive Privacy Amnesia is inspired by Anterograde Amnesia (Markowitsch) 2008). which the inability to form new memories following an event while preserving long-term memories before the event. In a case study described by Vicari al. (2007). girl suffering from Anterograde Ammesia since childhood exhibited severe impairment in episodic memory while retaining her semantic memory. This suggests that certain key elements within the information determine the information retention. By incorporating Sensitivity Analysis and Selective Forgetting, we focus on forgetting only the crucial parts, rather than removing the entire sentence. This approach has the advantage of minimizing the impact on model performance. However, we found that Selective introduce Memory Implanting compensate for this
Forgetting can harm model performance, degradation. Therefore, PPA consists of three components: (1) Sensitivity Analysis, which identifies the key elements within memorized PII; (2) Selective Forgetting, which targets the forgetting of these specific key elements; and (3) Memory Implanting, technique designed to mitigate the loss in model performance resulting from the Selective Forgetting process.
4.2 THEORETICAL JUSTIFICATION OF SENSITIVITY ANALYSIS.
Definition of Sensitivity Analysis. To quantify how well the model memorize the PII sequence, we introduce L(k) as defined in Definition (1). The primary goal identifying key elements is to isolate tokens that carry higher amount of information. To achieve this, consider token more informative if it significantly simplifies the prediction of subsequent tokens, thereby reducing the uncertainty in predicting future tokens.
Definition 1. (Cross-entropy Loss of the PII Sequenee) We define
L(k) = LCE (p(x)....,, Xx), q(x)........................ xR));
where Lee is the Cross Entropy Loss, and -1. 2y refers to the first tokens PII sequence.
We search the key element i such that the learning loss achieves the maximum at this token and does not increase significantly after this token, i.c.,
L(k: 1) L(k) +k++1) = L(k- +
which means that the token & helps the model memorize the following tokens in this PII sequence. Notice that LCE is the cross entropy loss of the PII sequence, which can keep growing with more tokens and thus the last token must achieve the maximum of LCr. This solution is trivial and cannot show the essentiality of the token. To tackle this issue, we propose to find the token with the largest memorization factor Dr. which can lead to non-trivial solution of Eq. Definition 2. (Memorization Factor) We define the memorization factor Dy as follows:
stated in Proposition
Hg H
D
H, ICE(Ps, 9:).
H₁
Where pa (x) be the true probability distribution and q: (xx the predicted probability distribution for the i-th token in the PII sequence.
4

---

Published as a conference paper at ICLR 2025
Sensitivity Analysis
Sensitivity Anelysis
22.5
11.0
20.0
10.5
10.0
17.5
9.5
10.0
8.0
7.5
Sequence
Address
5
John Griffth phone number (713) 853 6247
Jeffrey Dasavich 101 California St. Suite 1950
(a) Sensitivity analysis on phone number example: 'John Griffith phone number (713) 853-6247'. '8" is the largest D, within '(713) 853-6247'.
(b) Sensitivity analysis on physical address exam- plc: "Jeffrey Dasovich address 101 California St. Suite 1950". "_Su" is the largest D; within '101
California St. Suite 1950'.
Figure 2: Sensitivity analysis on the phone number and physical address examples: The darker color on the PII tokens indicates larger memorization factor. The red dot in the figure represents the top-1 key element.
Proposition 1. Maximizing the memorization factor can lead to
max D(k) = max; L(k)
ifak (L(k) = 0, AK VL(k)
max; 1 d/dewonn (Re)
dxewton (k) is Newton's 's Direction at k which is from Newton Method in convex optimization (Boyd Vandenberghe 2004). max; 1/dNewtom (k) is achieved when dNeeston (k) + 0 As T.kk) is non- decreasing, small positive dxewnnn (k) implies that the gradient at token i quickly approaches O with a negative second-ored order derivative.
Examples on PII sequences. We do sensitivity analysis on "John Griffith phone number (713) 853-6247," as shown in Figure 2a the token '8' exhibits the most significant decrease in cross-entropy rate, making it the key element this context. Similarly, "Jeffrey Dasovich address 101 California St. Suite 1950", depicted in Figure 2b] the token "_Su" shows the most notable drop in cross-entropy rate, identifying _Su' as the key element.
4.3 FORMULATING PPA
We consider large language model F(.) trained on dataset ID containing PII, denoted as P {(z, y)} where is the person's name and y is their PII sequence. In response to a deletion request for specific data / = {æf y }. our objective to train an updated model F²(-) that cannot extract data from DV, We employ an memory implanting dataset = {xf, ". where x the person's 's name and is fabricated PII sequence.
Sensitivity Analysis. Initially, we create unlearning templates for each person's PII, structured as the person's ' name, PII type, and the PII sequence. For instance, take the examples of John Griffith's 's phone number, "John Griffith phone number (713) 853-6247", and Jeffrey Dasovich address, "Jeffrey Dasovich address 101 California St. Suite 1950". Next, we perform sensitivity analysis on the PII sequence calculate D, and identify the key token within the sequence that is crucial for the language model's retention, as shown in Figure and Figure 2h
We then apply top; D,, calculated as follows:
topy (D1, Dz...., Dn) (x1, F2, ...,..,,, 2k)
(5)
5

---

Published as a conference paper at ICLR 2025
Algorithm Proactive Privacy Amnesia (PPA)
Initialization. Forget dataset D/ = (a/, .y' }. Memory Implanting dataset ID {ar y/ J. Large Language
Model F(-) with parameters w. Weights of the model Aw. The key elements that the model needs to forget
DI.. Total number users u
Defensive Training
DÍ lop(k, Sensitivity-unalsss((())))
Sensitivity Analysis forget dataset.
while < U do
DV + D [u]
P Select person's 's PII
Aw +- SelectiveForgettinggLE Aw)
DE < D"[u] [u]
> Select person's 's Memory Implanting PII
Aw <- Memorylmpphating(D.; Aw)
<- u
end while
Outcome:
Derive the LLM (-) with parameters w²
Selective Forgetting. Then, we maximize the following loss function, on the key element tokens which be calculated as:
(#1,-..,31)) based Equation 5
k
(UL = Σ Log(po (2, (x<e))
(6)
Here, Tec represents the PII sequence of tokens x (.1,..., -1) 1). and po(sslared is the condi- tional probability that the next token will be Ft, given the preceding sequence Fets in a language model F parameterized by 0.
Memory Implanting. After that, we apply the memory implanting, borrowed idea from error injection (De Cao et al.] 202T), to compensate for the performance damage done by the selective forgetting is calculated as follows:
arg max p(y>>>">""" Fe)
M
where y" represents the alternative, false target as proposed by Microsofi 2024
EXPERIMENTS
In the experiments, demonstrate that our PPA effectively preserves PII while maintaining model performance across multiple settings.
5.1 SETUP
Benchmarks. We conduct extensive experiments by fine-tuning LLaMA2-7b model (Touvron)
al. 2023) and LLaMA3-8b model (Dubey al. 20244) on two different datasets: 1) Enron email experiment which fine-tune LLM on Enron email dataset (Klimt & Yang 2004) 2) Fraud email experiment which fine-tune LLM on Fraud email dataset (Radev 2008). To evaluate our defense method, we construct separate ground truth tables, details in Appendix B and evaluation dataset for the Enron email dataset and the Fraud email dataset, specified in Appendix
Attack methods. We implemented the input rephrasing attack al. 2023 Krishna et al. 2024) to generate multiple attack templates. Additionally. we employed the probing attack using the twin template probing method described in Kim et al. (2024), and the soft prompt attack (Kim et al.) 2024) using trained soft prompts, attacking persons" phone numbers and physical addresses" more details in Appendix Q
All attack methods employed the AWS Comprehend Service Amazon Web Services Comprehend 2024 to extract PIT from the model output.

---

Published as a conference paper at ICLR 2025
Baseline defense methods. We consider 4 representative defense methods as baseline. Empty Response (Patil et al, [203] Ouyang et al. (022) applies gradient descent to non- son-seisieive information, used as "dummy"
replace PII sequences. Error Injection (De Cao et al. (0211) using gradient
descent to increase the likelihood of generating fake PII sequence. Unlearning (Jang et al. (022) do
gradient ascent on PII sequence. DEPN (Wu al. (023) use memory editing technique to the
neurons, which significantly contribute privacy leakage, in the model, more details are in Appendix P
PPA (ours). The PPA. as detailed in Section 4 to protect PII. During the sensitivity analysis and the selective forgetting stages, a single token was selected from each PII sequence for selective forgetting. In particular, we established &o = 1 in Equations | and 6 Both selective forgetting and memory implanting stages were implemented following the training guidelines specified in Appendix with single epoch.
5.2 EVALUATION METRICS
Attack Success Metric In this paper, propose PII risk score metric and apply modified exact match score metric (Kim et al. 2024) in our experiments. For the phone number risk score, we utilize an eighth-order Levenshtein distance (Po 2020) compare the predicted phone number with the ground truth. For calculating the risk score of physical addresses, we first use the AWS Location Service (Amazon Web Services Location 2024) to geocode location and obtain detailed physical address information. Then, we compare the details of the predicted physical address with the ground truth physical address using our physical address risk score Table [ To calculate the exact match score for both phone numbers and physical addresses, we will award 1 point when the prediction completely matches the ground truth. More scoring details are Appendix R
Model Performance Metrie We employ two pri- mary metrics, which are widely used for evaluating
Category
Address Risk Score
LLMs, to measure their performance: 1) Perplex-
Country
0,005
ity (Touvron et al. 2023 Radford et al. 2019 Brown et al., 2020), averaged by three different perplex- ity tests, and 2) Email completion, where we eval- uate the content of email completions using I.LM Judge (Thakur et al. 2024 Verga et al.) 2024 Zhang et al., 2024). For the Perplexity metric, lower value generally indicates better performance (Blei 2003). For the Email completion metric, we ranked the outputs from to 10, with 10 being the best and I the worst, using the GPT-4o model as our evaluator.
Region
0.1
SubRegion
0.15
Municipality
0.2
PostalCode
0.3
Street
0.3
AddressNumber
0.3
Table 1: Address Risk Score
Further details on the model performance metric can be found in Appendix S
5.3
MAIN RESULTS
Notation for Experimental Tables. RS denotes the risk score, while EM represents the exact match score. 'Perplexity" refers to the average value our perplexity metric; 'GPT-40 Email Score' indicates the average score of our email completion metric as judged by GPT-4o.
Enron email experiment. For the phone number defense results, Table 2 shows that our method, PPA, effectively protects the phone numbers of all persons, achieving both a phone number risk score and a phone number exact match score of zero while maintaining model performance comparable to Fine-tuned L-aMA2-7b and ALaMMA3--88 In contrast, while methods like Empty Response and Error Injection maintain good model performance, they fail to protect all phone numbers. Unlearning successfully safeguards all phone numbers but results in a significant decline in model performance.
Table 3 shows that our method, PPA applied to LLaMA2-7b for defense against physical address exposure, outperforms both Empty Response and Error Injection by reducing the risk score by 87.6% and 26.2%, respectively. This is achieved with only a marginal increase in the perplexity score by 16.7% and 35.4%, and slight decrease in the Email Completion score by 30.7% and 29.4%. Although Unlearning effectively protects users' physical addresses, lowering the risk score by 60.2%, it results in an infinite perplexity score and an Email Completion score of just 1.0. Additionally, PPA
7

---

Published as a conference paper at ICLR 2025
Model Performance
Attack
Enron Email Experiment
GPT-4o Email Score
Defense Model
Perplexity
Input Rephrase
Prohing
Prompt
Attack Average
L
RS
EM.
J
!
Original
5.6
8.6
1.3
1.0
0.0
0.0
0.0
0.0
0.4
0.3
Finetuned
16.2
63.9
60.0
57.9
56.0
87.6
84.0
69.8
66.7
Empty Response
16.5
56.4
53.3
LLaMA2-7b
Ervor Injection
14.6
5.2
24.2
22.2
19.3
17.6
21.7
20.8
21.7
20.2
Unlearning
3.2 10
1.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
DEPN
77.2
5.7
4.7
PPA
16.0
5.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Original
9.9
9.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Finetuned
Empty Response
82.9
4.7
25.7
23.8
21.5
19.8
25.3
24.1
24.1
22.5
LLaMA3-8b
Fror Injection
60.5
5.3
13.3
12,9
5.8
5.2
18.1
16.6
12.4
11,5
Unlearning
1021
0.0
DEPN
138.5
37.2
34.2
26.1
24.5
26.7
25.3
30.0
28.0
PPA
67.5
4.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Table 2: Comparative Analysis of Phone Number Defense Strategies Against. Various Attacks in Enron Email Experiment. PPA effectively defend all user's phone number with comparable model performance with fine-tuned model.
Madel Performance
Attack
Enron Email Experiment
GPT-4o Email Score
Physical
Model
Perplxxity
Input Rephrase
Probing
Prompt
Average
J
Original
5.6
8.6
24.5
3.0
17.1
3.0
6.5
1.0
16.0
2.3
Finetuned
16.2
5.0
59.4
1.0
50.4
1.0
72.7
2.8
60.8
1.6
Empty Response
16.7
LLaMA2-7b
Eror Injection
14.4
19.2
1.0
1.0
1.0
9.9
1.0
Unlearning
inf
1.0
3.8
1.0
2.5
1.0
2.5
1.0
2.9
1.0
DEPN
218.9
1.4
16.3
1.0
2.8
8.1
1.2
Original
9.9
9.2
21.5
7.4
19.4
6.4
7.8
2.6
16.2
5.4
Finctuned
79,2
5.0
66.1
4.6
41.2
1.0
39.5
1.5
48.9
2.3
Empty Response
78.8
4.8
37.4
3.1
29.8
37.6
3.6
LLaMA3-8b
Error Injection
52.5
4.2
18.2
1.3
Unlcarning
inf
1.0
3.1
1.0
2.2
1.0
2.2
1.0
2.5
1.0
DEPN
201.5
1.5
45.5
3.0
15.7
2.0
9.1
5.3
17.8
3.4
PPA
57.6
16.7
Table 3: Comparative Analysis of Physical Address Defense Strategies Against Various Attacks in Enron Email Experiment. PPA has the best trade off between defense capability and model performance.
outperforms DEPN by reducing the risk score by 9.8%, decreasing the perplexity score by 91.0%, and increasing the Email Completion score by 157. %. For LLaMA3-8b, PPA also shows strong performance defending against physical address exposure, surpassing Empty Response and Error Injection by reducing the risk score by 60.3% and 16.2%, respectively. It achieves this while slightly decreasing the perplexity score by 26.9% and increasing by 9.7%, with only a marginal decrease in the Email Completion score by 16.6% and 4.7% Although Unlearning remains effective, reducing the risk score by 83. 93.2%, it again leads to an infinite perplexity score and an Email Completion score of only 1.0. PPA outperforms DEPN by reducing the risk score by 16.2%, lowering the perplexity score by 71.4%, and increasing the Email Completion score by 166.6%.
Fraud email experiment. Table shows that PPA effectively protects the phone numbers of 50 persons, achieving a phone number risk score of 0.3 while maintaining model performance comparable to that of the Fine-tuned LLaMA2-70 model. Similarly. PPA safeguards the physical addresses of the other 50 persons, achieving an physical address risk score of 3.0 without compromising model performance relative to the Fine-tuned I.LaMA2-76 model.
8

---

Published as a conference paper at ICLR 2025
Model
Frand Email Experiment
GPT-4o
Defense Model
Perplexity
Input Rephrase
Probing
Soft Prompt
Attack Average
Email
RS
Original
4,06
1,0
0.0
0.0
0.0
0.0
0.0
Finctuned
1.11
2/4
20.6
19.0
20.2
19.0
0.4
0.0
13.7
12.6
Empty Response
1.11
2.2
13.6
13.0
13.6
12.0
0.4
0.0
9.8
LLaMA2-7b
Eror Injection
1.10
2.0
11.7
11.0
6.0
0.2
0.0
6.3
Phone Number Defense
Unlearning
1.33
1.6
0.0
0.0
0.0
0.0
0.0
0.0
DEPN
1.18
2.8
5.4
4.0
5.0
0.0
0.0
4.0
PPA
1.10
2.7
1.0
1.0
0.0
0.0
0.0
0.3
Original
4.06
1.0
4.7
0.0
2.4
0.2
1.4
0.0
2.8
0.0
Finetuned
1.11
2.4
12.4
0.0
18.6
0.0
4.3
0.0
11.7
0.0
Empty Response
1.11
2.3
13.2
0.0
9.3
0.0
4.7
0.2
9.0
0.0
LLaMA2-76
Emoe Injection
1.10
2.2
8.3
0.2
6.5
0.0
3.8
0.2
6.2
0.1
Physical
Unleamiin
27.54
1.0
1.3
0.0
1.2
0.0
1.2
0.0
1.2
0.0
DEPN
1.94
1.3
3.1
0.0
1.6
0.0
0.7
0.0
1.8
0.0
PPA
1.10
2,5
4.2
0.0
4.2
0.0
0.8
0.0
3.0
0.0
Table Comparative Analysis of Phone Number and Physical Address Defense Strategies Against Various Attacks in Fraud Email Experiment.
Model Performance
Atback
Study
Fame perplexity
Eanon perplesity
GP14 perplesryy
Average
Score
Inpur Rephinse
Prohing
Soft Phompe
Average
Prosctive Privacy
RS
0.0
0.0
0.0
0.0
RM
0.0
0.
Sensiivity
Foegetinn
0.0
0.0
EM
0,0
0.0
0.0
0.0
Menurr Implenting
EM
0.5
0.0
0.0
0.2
25.6
27.8
Privacy
FM
26.3
21.3
26.3
24.6
Sclective Prioacy
12
EM
3.0
1.0
2.0
20
R5
1.0
Fix Index Selective Privacy Amnesia
24,9
6.6
154
EM
0,0
1.0
1.3
088
Table 5: Ablation study phone numbers.
Main Results. We summarize the key results in Tables as follows: Observations from both phone number and physical address defenses indicate that PPA provides the best balance between safeguarding users' PII and maintaining model performance, compared to other defense methods.
ABLATION STUDIES
6.1 ANALYSIS OF THREE STAGES IN PPA
Sensitivity Analysis + Selective Forgetting. To assess the effectiveness of sensitivity analysis combined with selective forgetting in preserving the PII of targeted persons, we applied this approach protect PII, such as phone numbers and physical addresses. While selective forgetting marginally reduces performance degradation, the resulting model remains largely ineffective, as demonstrated in Table and and T
Medid Porfornancy
Atback
Plysical Address Ablution Study
perplecity
peppecity
GPT4 perplasit
Average
Ispat Replmac
Probing
Proeppt
Average
Proective Priracy Amzesia
35.6
9.2
14.6
19.5
RS
12.1
4.7
5.2
7,3
Seneiivity Amalysi Selestio: Foegetting
in
8.2% 1030
1.6 x 1gia
88
3.8
25
2.5
2.9
1.0
1.0
Unleariin Memory
61.4
10.6
336
RS.
10.1
49
52
67
FM
10
10
Selective
RS.
363
98
FM
10
1.0
Seluative
RS.
11.7
15.4
EML
1.0
1.0
1.0
Privccy
EM
1.0
100
1.0
Table 6: Ablation study on physical addresses.

---

Published as a conference paper at ICLR 2025
Fix index Selective Privacy Amnesia. To evaluate the efficacy of sensitivity analysis in safeguarding the PII of targeted persons, we employed a fixed index Selective Privacy Amnesia approach, focusing on indices 0, 1, and 2 as the primary elements for removal. Our findings indicate that the fixed index Selective Privacy Amnesia falls short in adequately safeguarding users' phone numbers, as illustrated in 5 When it comes to preserving users' physical addresses, depicted in Table employing the fixed index O and 1 Selective Privacy Amnesia, yielding address risk scores of 17.2 and 10.9 respectively, does not offer robust protection as Proactive Privacy Amnesia, which yields an address risk score of 7.3. While the fixed index 2 Selective Privacy Amnesia, with an address risk score of 4.4, does provide superior protection compared Proactive Privacy Amnesia, comes with higher perplexity, indicative of lower model performance. This is attributed to the fixed index potentially altering the original meaning of words. For instance, the case of "New York City," unlearning the single token "City" would compel the model to disregard the frequent occurrence of "City" following "New York," consequently compromising the model's performance.
Unlearning Memory Implanting. To assess the effectiveness of sensitivity analysis coupled with selective forgetting in safeguarding the PII of targeted persons, we implemented unlearning in conjunction with memory implanting, as shown in Table 5 Our findings revealed that Unlearning + Memory Implanting proved capable safeguarding the majority of persons' phone numbers and physical addresses, resulting in phone number risk scores of 0.2 and address risk scores of 6.7. However, this approach exhibited higher perplexity levels, measuring 16.3 and 33.6, which signifies diminished model performance. This because the unlearning method essentially erases entire PII sequences, thereby enhancing PII protection capabilities at the expense of model performance.
6.2 PPA TRADE- OFFS: NUMBER OF FORGOTTEN INDEXES
Table and 3
reveal that, after applying PPA. the address risk score remains at 7.3, while the phone risk score drops to O. This disparity may
Address forget number trade
be due to physical addresses being longer and
k-20
less structured than phone numbers. Therefore, we have conducted an ablation study to quan-
Nitress rb6 Store
tify the drop-off in risk score the number of forgotten indexes increases. We conducted ex-
Methods
periments on Address PPA. Specifically, we set
k=1
1e
ke-5
k = 1,5, 10, 15, 20, 25 in Equations 5 and 6
Both stages followed the training protocols in Appendix U for single epoch. For Addresses, the PPA method improves the PII risk score if more than one index is selected for forgetting. However, selecting too many indexes causes the model performance to deteriorate, shown in Figure 3 This ablation demonstrates that PPA a flexible method, allowing for adjustments to the balance between defense capability and model performance by modifying the number of key elements to be forgotten.
km15
km20
K-25
2.5 Model Average Perplexity scate
1.5
Figure 3: Address PPA Risk score vs forget num- ber of indexes: PPA tunes the parameter k, as defined in Equationn and
CONCLUSION AND DISCUSSION
We demonstrated that Proactive Privacy Amnesia achieves the optimal balance between defense per- formance and model utility compared to methods like Error Injection, Empty Response, Unlearning, and DEPN for protecting users' PII, including phone numbers and physical addresses. Additionally, we initially introduce the concept of the 'memorization factor', which affects the model's capacity to retain PII sequences. This concept is using in sensitivity analysis and supported by theoretical justifi- cation. Furthurmore, PPA is flexible method that can adjust its balance between defense capability and model performance. Future work could extend PPA to protect the privacy of relationships, such those between persons or between organizations.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
This work is supported by ARO W911NF-23-2-0224, NSF IIS-2332744, and CNS-2112562. We also thank Accenture for its support and Yun-Chu Wang for her support. We thank area chair and reviewers for their valuable comments,
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Amazon Web Services Comprehend. Detecting and redacting pii using amazon comprehend, 2024.
http://wws.aaazzococoo///oooppee
Amazon Web Services Location.
Detecting location
using amazon location, 2024.
http://aww..mmzzon.comm/occooioo
David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(lan):933-1022, 2003.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nicholas Carlini, Chang Liu, Ulfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pp. 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. arXiv preprint 100iv22300000150 2023.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint 4rX1v:2104.08164, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.2178. 2024.
Hugging Face, http:///cccaatttrrrrrrer/pppppps
transformers documentation, 2024. URL Accessed: 2024-05-10.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. arXiv preprint arXiv:2012. 2020.
Dan Hendrycks. Collin Burs,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wwXiv:2009.033000, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 20X.::2106.09685, 2021.
Joel Jang. Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Sco. Knowledge unlearning for mitigating privacy risks language models. arXiv preprint arXiv:22/0.01504 2022.
11

---

Published as a conference paper at ICLR 2025
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. Propile: Probing privacy leakage in large language models. Advances in Neural Information Processing Systems, 36, 2024.
Bryan Klimt and Yiming Yang. Introducing the enron corpus. In CEAS, volume 45, pp. 92-96, 2004.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0788 2021.
Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. Differentially private decoding in large language models. arXiv preprint arXiv:2205. 2022.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Mqag: Multiple-choice question answer- ing and generation for assessing information consistency in summarization. arXiv preprint arXiv:2301.12300/ 2023.
Hans Markowitsch. Anterograde amnesia. Handbook of clinical neurology, 18:155-183, 2008
Meta- Llama. Llama-2 2---a-2-
attps://auggiaffaee.co///tttssss
L.aama- 2- 7b
Accessed: 2024-03-22.
Microsoft. Presidio research. httpp/////ttttaa.oo///ooorrtttts tree/master, 2024.
Accessed: 2024-03-22.
OpenAl. Gpt-4o, 2024. Language model, httpp:://pppaaaala
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from Ilms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309. 17410, 2023.
Daw Khin Po. Similarity based information retrieval using levenshtein distance algorithm. Int. J. Adv.
Sci. Res. Eng, 6(04):06-10, 2020.
Dragomir Radev. Clair collection of fraud email, acl data and code repository. 5 (5.5):1, 2008
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint 2xXiv:224.006677 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities llms-as-judges arXiv preprint 11.Xv:22466122222 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2rXii:2307.09288 2023.
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhang- orodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating Iim generations with panel of diverse models. arXiv preprint arXiv:2404. 240..1899 2024.
12

---

Published as a conference paper at ICLR 2025
Stefano Vicari, Deny Menghini, Margherita Di Paola, Laura Serra, Alberto Donfrancesco, Paola Fidani, Giuseppe Maria Milano, and Giovanni Augusto Carlesimo. Acquired amnesia in childhood: A single case study. Neuropsychologia, 45(4):704-715, 2007.
Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. arXiv preprint arXiv:2402.05811,, 2024.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138, 2023.
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint 1rX1v:2402.15159, 2024.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning, arXiv preprint arXiv:2310.10683, 2023.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6915-6919. IEEE, 2024.
Rui Zhang and Joel Tetreault. This email could save your life: Introducing the task of email subject line generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 446-456, Florence. Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1043. URU.mtpp:://claaatnooooooy Sttt::///aaaantoooooyoooo///--00
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXISTING ASSETS
The two existing assets used this work are the Enron email dataset Klimt & Yang (2004) and aeslc dataset Zhang & Tetreault (2019). Please see their information below.
Enron email dataset: The URL is http://www.cmoonnmallcccmmmmmmm the license is not clearly stated by the authors.
aeslc dataset: The URL is hatp://hagginngaacccco///sssssss the license is not clearly stated by the authors.
B DETAILS OF BUILDING GROUND TRUTH TABLE
To evaluate our defense method, we constructed ground truth table comprising two parts: (1) We utilized the AWS Comprehend Service (Amazon Web Services Comprehend] 2024) extract PII, including names, phone numbers, and physical addresses; (2) we employed (Manakul et al. 2023 to determine the correlations between specific PIIs and the corresponding persons.
C DETAILS OF EVALUATION DATASET
For Enron email dataset, we constructed our evaluation ground truth table using the aelsc training dataset (Zhang & Tetreault] 2019). However, there overlap between the aelse training and validation datasets, which are used to train the soft prompt for the soft prompt attack. To ensure a fair comparison between soft prompt and probing attacks, excluded certain persons' scores, detailed in Appendix D Consequently, our evaluation focused on 468 persons whose phone numbers were disclosed and 790 persons whose physical addresses were revealed.
For Fraud email dataset, we randomly selected 50 persons who had disclosed their phone numbers and persons who had revealed their physical addresses build our evaluation ground truth table.
D EVALUATION DETAILS OF THE ENRON EMAIL. EXPERIMENT.
We constructed our evaluation ground truth table on the aeslc training dataset (Zhang & Tetreault 2019), which comprises data from 1,359 persons. Within this dataset, 577 persons disclosed their phone numbers and 899 persons revealed their physical addresses. To ensure a fair comparison between soft prompt and probing attacks, we excluded persons whose data overlapped between the aeslc training and validation datasets, because we used aeslc validation dataset to train the soft prompt attack's soft prompt. The number of overlapping persons 109. Consequently. our evaluation focused on 468 persons (577- 109) whose phone numbers were exposed and 790 persons (899 109) whose physical addresses were exposed. Subsequently, we utilized this evaluation ground truth table to assess the effectiveness of the defense methods.
E COMPARISON WITH DIFFERENTIAL PRIVACY- BASED METHODS
We implemented the Differentially Private Decoding (DP Decoding) (Majmudar et al. 2022) and the Just Fine-Ture Tune Twice (JFT) method in (Shi et al. 2022). To evaluate these methods, we conducted probing attacks on both DP Decoding and JFT. Specifically, for DP Decoding, the parameter À impacts both utility (the effectiveness of the generated output) and privacy (the protection of sensitive information). As varies from 0 to 1, a value closer to 0 enhances privacy but reduces utility, while value closer to I improves utility at the expense of privacy. We tested various values of the À parameter ranging from (0.1, 0.3, 0.5, 0.7, 0.9) and we selected A=0.3 as it achieved the best balance between utility and privacy protection among the tested values, offering better utility compared to A= and improved defense capability compared to X=0.5, as shown in Table 7 We observed that PPA method still outperformed both DP Decoding and JFT, achieving lower risk
14

---

Published as a conference paper at ICLR 2025
score and higher utility score, as shown in Table This superior performance can be attributed to the fact that DP Decoding applies a uniform distribution adjustment to next-token predictions, which lacks the necessary customization for scenarios involving PII.
PPA
A=
A=0.3
A=0.5
A=0.7
A=0.9
no defense
Risk Score
0.0
15.7
30.4
32.4
33.2
41.4
57.9
Exact Match Score
0.0
10.6
28.4
30.1
31.1
38.6
56.0
GPT-4o Email Score
5.2
3.0
(3.025)
4.8 (4.750)
4.8 (4.825)
4.9 (4.875)
5.1 (5.125)
5.0
Table 7: Comparison of Different À Values in DP Decoding. A=0.3 achieved the best balance between utility and privacy protection among the tested À value.
Phone Defense Model
Risk Score 4
Exact Match Score 4
GPT-4o Email Score
DP Decoding
30.4
28.4
4.8
JFT
28.4
26.0
5.0
PPA
0.0
0.0
5.2
Table 8: Performance comparison of Differential Privacy-Based methods.
F MODEL PERFORMANCE EVALUATION ON MMLU AND TRUTHFULQA
We provide some new evaluations on the model's performance metrics on both MMLU (Hendrycks) al. 2020) and TruthfulQA (Lin al., 2021). As our research primarily focuses the text generation capabilities of models, we had the models that have been protected by various defense methods respond the MMLU and TruthfulQA questions directly. GPT-4o was then employed to rate these responses a scale from to 5, where 5 represents the best possible score and the worst. Given the extensive volume of the MMLU dataset, and in order to manage computational costs efficiently, we selected 20 data points from each subtask to form representative subset, totaling 1,140 data points. For each defense method, we calculate the mean score for comparative analysis between defense methods. We found that PPA achieves the highest MMLU and TruthfulQA score among all baseline defense methods, as illustrated in Table
Phone Defense Model
MMLU Score
TruthfulQA Score
GPT-4o Email Score
Empty Response
3.3
3.4
5.7
Error Injection
3.3
3.2
5.2
Unlearning
1.6
1.7
1.1
DEPN
2.3
2.4
2.0
PPA
4.2
4.1
5.2
Table Comparison of Phone Defense Models based on MMLU and TruthfulQA mean scores and GPT-4o Email Scores.
G STRONGER ATTACKER HAS PRIOR KNOWLEDGE OF THE PIT
We have implemented more advanced attack scenario where the attacker possesses prior knowledge of the PII. Specifically, we assume the attacker knows the information from the beginning of the PII up to key element. For instance, in the case of "John Griffith phone number (713) 853-6247, the key element is "8". this scenario, the attacker's prompt would resemble: "The phone number of John Griffith is (713) 8".
15

---

Published as a conference paper at ICLR 2025
As shown in the Table 10 and model performance.
observe that PPA achieves the best balance between defense capability
Phone Defense Model
Risk Score J
Exact Match Score +
GPT-4o Email Score
Empty Response
154.0
141.5
5.7
Error Injection
75.1
69.1
5.2
Unlearning
6.7
1.8
1.1
DEPN
36.2
27.3
2.0
PPA
12.1
9.8
5.2
Table 10: Phone Defense Models against strong attacker has the prior knowledge.
H DETAILS OF THE PROPORTIONS OF KEY ELEMENTS
We calculated the proportions of key element lengths relative to the total lengths for phone numbers and physical addresses, which are 6.7% and 27.6%, respectively.
I MORE DISCUSSION ABOUT MEMORY IMPLANTING
We modified the memory implanting component to focus on replacing the key element with different token. For instance, in the example 'John Griffith's phone number is (713) 853-6247," where the key element is '8', we selectively forgot '8' and replaced it with different number at its position. We observe that the Modified Memory Implanting PPA provides same protection for users' phone numbers and outperforms PPA in GPT-4o EmailScore by approximately 9.6%, as shown in II However, Address substitution presents challenges because addresses are highly contextually dependent. Replacing key element in address with an arbitrary token impair the model's understanding of the context. For example, substituting '_Su' 'Jeffrey Dasovich address 101 California St. Suite 1950' disrupts the model's comprehension of the address structure. Additionally, partial substitution may inadvertently expose parts of the user's address. Discussing how to customize selective analysis and memory implanting for different types of PII is pertinent issue. Design memory implanting to optimize performance for various PII types is valuable and can be future work.
Phone Defense Model
Risk Score +
Exact Match Score 1
GPT--o Email Score
Empty Response
37.2
34.8
5.7
Error Injection
19.3
17.6
5.2
Unlearning
0.0
0.0
1.1
DEPN
0.0
0.0
2.0
PPA
0.0
0.0
5.2
Modified Memory Implanting PPA
0.0
5.7
Table 11: Comparison of the Modified Memory Implanting PPA with Other Phone Defense Strategies.
J ADDING THE EXPOSURE METRIC
We calculated the exposure metric (Carlini et al. 2019) for all baseline methods. Since calculating the exposure of PII is computationally intensive, followed the approach in Table 2 of (Carlini et al.) 2019) and evaluated the exposure for 10 phone numbers. Our results show that PPA outperforms other baseline defense methods, shown in Table 12
16

---

Published as a conference paper at ICLR 2025
Phone Defense Model
Exposure
Empty Response
12.50
Error Injection
10.94
Unleaming
3.55
DEPN
7.72
PPA
0.05
Table 12: Exposure levels of various Phone Defense Strategies.
K EVALUATION ON EMAIL ADDRESS.
We conducted an additional experiment to evaluate the protection of 281 users' email addresses in the aeslc training dataset. Using Levenshtein distance (Po 2020), compared the predicted email addresses to the ground truth. As shown the Table 13 PPA successfully defends all users' email
addresses against probing attacks while maintaining model performance comparable to other baseline defense methods.
Email Defense Model
Risk Score J
Exact Match Score 1
GPT-4o Email Score
Empty Response
47.2
40.5
5.1
Error Injection
19.6
17.0
5.3
Unlearning
1.0
1.0
1.6
DEPN
1.0
1.0
1.3
PPA
0.0
0.0
5.0
Table 13: Comparative Analysis of Email Defense Strategies Against Various Attacks in Enron Email Experiment.
L ORIGINALLY SAFE INFORMATION TO BE EXPOSED?
Motivated by the concern that the PPA defense could inadvertently expose previously secure PII of users who are not explicitly protected by the method. We evaluated the exposure metric (Carlini al. 2019) for safe phone numbers-those
exposed to attackers-that were not protected by the PPA method, using both the no-defense setup and the PPA model. Given the time-intensive nature of calculating PII exposure, we referenced Table from The Secret Sharer (Carlini al.) 2019) and analyzed the exposure of 10 such phone numbers. The average exposure for these cases is summarized in the T4]
As shown in the Table 14 the exposure of phone numbers not protected by the PPA method decreases slightly, from 1.57 (no defense) 1.22 (PPA), since the PPA method does not directly target these users for protection. This result suggests that the original safe information remains secure even when PPA method is applied to protect other users' PII.
Phone Defense Model
Exposure
No Defense
1.57
PPA
1.22
Table 14: Exposure of Users" Phone Numbers not protected by the PPA Method.
17

---

Published as a conference paper at ICLR 2025
M DISCUSS SCALABILITY FOR PPA
We have conducted an initial investigation into the scalability and optimization strategies for PPA. Our experiments involved combining PPA with efficient fine-tuning techniques, such LoRA (Hu et al. 2021), using a rank of 16 and an alpha value of 32. As shown in the Table[[S] applying LoRÁ to PPA produced promising results: after fne-tuning for three epochs, the risk score reduced to 1.0, and after four epochs, it further decreased to 0.0, all while maintaining comparable model performance. Although PPA with LoRA required four epochs, compared to just one epoch for full fine-tuning of PPA, it achieved the same defensive effectiveness.
Table 15 demonstrates that PPA has potential for scalability. Furthermore, exploring additional optimization strategies could be a valuable direction for future work.
Phone Defense Model
Risk Score 4
Exact Match Score 4
GPT-4o Email Score
PPA LoRA l-epoch
24.6
23.6
5.4
PPA LoRA 2-epoch
5.4
5.2
5.2
PPA LoRA 3-epoch
1.0
1.0
5.0
PPA LoRA 4-epoch
0.0
0.0
5.1
Table 15: Comparative Analysis of PPA LoRA with different fine-tuning epochs.
N ABLATION STUDY ON VARIOUS FORGETTING STRATEGIES
We conducted ablation studies on protecting the user's physical address using the following three strategies: 1) Only the most sensitive tokens (PPA), 2) Span around the most sensitive token, and 3) The most sensitive tokens the following tokens. Their respective Risk Scores and GPT-4o Email Scores are presented in Table 16
As shown in the Table 16 "Only the most sensitive token (PPA)" achieves an optimal balance between defense capability and model performance. "Span around the most sensitive token" exhibits slight utility degradation but does not significantly reduce risk beyond what is achieved by focusing on the most sensitive token alone. "The most sensitive tokens the following tokens" strategy reduces the risk score further but the cost of significantly lowering the GPT-4o Email Score. This indicates that although including the following tokens enhances defense performance, forgetting too many tokens will cause a worse trade-off in utility.
Our experimental results demonstrate that selectively forgetting only the most sensitive tokens secures a better balance between defense capability and model performance. The defense effectiveness of this approach is comparable to the other two methods, while it also maintains better utility. Given these findings, the selective forgetting of only the most sensitive tokens is still the most effective strategy for the PPA framework.
Forgetting Strategies
Risk Score
GPT-4o Email Score
Only the most sensitive tokens (PPA)
4.7 (4.702)
3.6
Span around the most sensitive token
4.7 (4.737)
3.4
The most sensitive tokens + the following tokens
4.4
2.9
Table 16: Comparative Analysis of various forgetting strategies.
18

---

Published as a conference paper at ICLR 2025
Enron Email Experiment
Model Performance
Phone Number Defense Model
Enron perplexity first=512 tokens
Enron perplexity stride=256
GPT4 perplexity
Perplexity
Original
7.6
6.2
5.6
Finetuned
26.1
16.0
16.2
Empty Response
26.7
6.7
16.2
16.5
LLaMA2 7b
Eror Injection
23.2
6.3
14.2
14.6
Unlearning
9.7 10
5691
1.9 x 10
3.2 10
DEPN
139.5
19.9
72.3
77.2
PPA
26.7
6.8
14.4
16.0
Original
10.5
4,4
15.0
9.9
Fineluned
58.9
11.4
167.4
79.2
Empty Response
61.7
11.7
175.5
82.9
LLaMA3-8b
Error Injection
46.5
9.9
125.2
60.5
Unlearning
1,5
10
8.5
10
1.1
10"
5.0
10"1
DEPN
92.2
14.9
308.6
138.5
PPA
57.5
11.3
133.8
67.5
Table 17: Comparative Analysis of Perplexity Against Various Attacks in Enron Email Experiment. PPA stands for Proactive Privacy Amnesia. 'Perplexity' refers the average of our perplexity metric.
Experiment
Model Performance
Physical Address Defense Model
perplexity first=512
Enron perplexity 25r
GPT4 perplexity
Perplexity
Original
7.6
6.2
5.6
Finetuned
26.1
16.0
16.2
Empty Response
27.3
16.1
LLaMA2-76
Firor Injection
23.5
13.3
Unlearning
10
10
DEPN
480.4
132.6
218.9
PPA
35.6
14.6
19.5
Original
10.5
15.0
9.9
Finetuned
58.9
11.4
167.4
79.2
Empty Response
60.0
11.5
165.1
78.8
LLaMA3-8b
Errrr Injection
44.8
9.7
103.2
52.5
Unlearning
1.2 10
1.3 x 10
inf
inf
DEPN
177.8
23.2
403.7
201.5
PPA
63.0
12.4
97.4
57.6
Table Comparative Analysis of Perplexity Against Various Attacks Enron Email Experiment. PPA stands for Proactive Privacy Amnesia. 'Perplexity' refers the average of our perplexity metric.
O DETAILS OF MODEL PERFORMANCE PERPLEXITY METRIC
In Tables 2 3 and 4 we report the average Perplexity across three different tests to evaluate model performance. The detailed results of each individual Perplexity test are provided in Table 17 18
and 19
P DETAILS OF THE BASELINE DEFENSE METHODS
Empty Response (Patil et al. 2023 Ouyang al. 2022). This method refines the model to label non-sensitive information as "dummy" For instance, we create templates for each person, formatted with the person's 's name, PII type, and "dummy". We then perform gradient descent on "dummy" following the training settings outlined Appendix[U] with single epoch.
Error Injection. We implemented the Error Injection method on each person's 's phone numbers, conducting a single epoch of training. This same process used to preserve a person's physical addresses. Take a person's phone number example, create templates for each person,
19

---

Published as a conference paper at ICLR 2025
Fraud Email Experiment
Model Performance
Defense Model
Enron perplexity first=512 tokens
Enron perplexity stride=256
Perplexity
Original
5.33
2.79
4.06
Finetuned
1.05
Empty Response
1.17
1.05
1.11
LLaMA2-7b
Error Injection
1.15
1.05
1.10
Phone Number Defense
Unlearning
1,53
1.14
1.33
DEPN
1.27
1.09
1.18
PPA
1.16
1.05
1.10
Original
5.33
2.79
4.06
Finetuned
1.17
1.05
1.11
Emply Response
1.17
1.05
1.11
LLaMA2-7b
Eroor Injection
1.15
1.05
1.10
Physical Address Defense
Unlearning
48.21
1.14
1.33
DEPN
2.48
1.09
1.18
PPA
1.16
1.05
1.10
Table 19: Comparative Analysis of Perplexity Against Various Attacks in Enron Email Experiment. PPA stands for Proactive Privacy Amnesia. "Perplexity' refers the average of our perplexity metric.
structured as the person's 's name, PII type, and fake PII, which generated by (Microsofi. 2024). We then apply gradient descent to false PII, adhering to the training settings detailed in Appendix U
Unlearning (Jang et al. 2022). We applied an unlearning technique to the PII sequence by performing
gradient ascent on it, following the training settings specified in Appendix U with a single epoch.
DEPN Wu et 2023
We adopted the DEPN approach, as detailed the DEPN GitHub repository, to protect PII, specifically phone numbers and physical addresses. Our goal was to eliminate specific neurons from the output of the LlamaDecoderLLayrr in the LlamaModel (Meta-Llama) We established threshold ratio of 0.01 for both phone numbers and physical addresses, with mode ratio bags set 0.49 and 0.5, respectively. Following this, we removed 10,000 neurons based on the identified candidates.
Q DETAILS OF ATTACK METHODS
For the input rephrasing attack, we generated 20 attack templates based on the twin template described in (Kim et al. 2024).
For soft prompt tuning in the Enron email experiment, used the first probing twin template from (Kim et al.) 2024), leveraging the aesle validation ground truth table. In the Fraud email experiment, applied the same probing template, selecting 25 persons with phone numbers and 25 with physical addresses randomly from the fraud email dataset.
R DETAILS OF ATTACK SUCCESS METRIC
For the total phone numbers and physical address risk score, we calculate the average phone number risk score and average physical address risk score for each person. We then aggregate the phone numbers and physical address risk scores of all persons to compute our final phone numbers and physical address risk score, following the same methodology as the exact match score for phone numbers and physical addresses.
MODEL. PERFORMANCE METRIC
For the Perplexity metric. conducted three different tests to assess model performance. First, calculated perplexity (Face 2024) on the first 512 tokens of each text (with maximum length 512 tokens). Second, computed the perplexity for each letter, using maximum length of $12 tokens
20

---

Published as a conference paper at ICLR 2025
and stride of 256 tokens. Third, assessed the perplexity of letters generated by GPT-4 (Achiam et al. 2023). , but the Fraud email experiment did not have this metric, because GPT-4 cannot write fraud emails due its safety aligned mechanism. These three tests help determine whether our defense method impacts model performance.
For the Email completion metric in the Enron email experiment, evaluated the model's per- formance in completing truncated emails. Specifically, we tasked the model with completing 40 truncated emails, which were subsequently evaluated by GPT-4o (OpenAT, 2024). Initially, GPT-4o generated 40 emails, each consisting of at least 100 words. We then truncated each email by half and had our models generate up to 100 new tokens to complete them. GPT-4o assessed and ranked the completions on a scale of to 10. with 10 representing the best score. The average score was calculated across all 40 completions.
For the Email completion metric in the Fraud email experiment, we evaluated the model's ability to generate complete fraud emails. The model was tasked with generating 10 fraud emails, each up to 500 tokens, which were also judged by GPT-4o. GPT-4o ranked these completions on the same to 10 scale, with the average score being calculated across all fraud email completions.
T
PROOF
PROOF OF PROPOSITION
Proposition 1. Maximizing the memorization factor can lead to
max D(k) maxk L(k)
ir L 0, if, VL(k) =
(8)
maxK 1/ dfdNewton(K) (k)
dNewson (k) ( is Newton's 's Direction at k which is from Newton Method in corvex optimization (Boyd & Vandenberghe 2004). max; 1 s/dxemto(() (k) is achieved when dNewwnn (k) 0+. As L(k) mono- tonically non decreseerr,nn a small positive dyewton (k) implies that the gradient at token k quickly
decreases with a negative second-order derivative.
Proof. Notice that
T(k) = Σ
p(z ... x& (g))ogg log 1o ... xxk
(9)
Σ
p(æ
xh log ((0) ,, xk-1)
B1 11
Σ
p( xk-1) Σ p(1x//111 1og xk-1)
(10)
201-1
=I.(k: 1) + Hk:
(11)
So have
H₂ = L(k) - L(k - 1)6 VL(K),
(12)
H H(k) R V + VL(A)) R V2I(K)
(13)
Dj ---Hh H
<LT(k)
(14)
V((k)
Our selection method selects k with the largest Dr. We discuss it in two situations:
1. When there exists & such that HI, ((k)) 0, we require that L² achieve the maximum (D₃ = +oo), this guarantees that & achieves the maximum of L(k) as well.
2. When H3 is always positive (notice that H₃ is never negative), L(k) keeps growing as k increases so we cannot find the maximum. But we still have
D3 dNewton (k)
min dNiewton())
(15)
21

---

Published as a conference paper at ICLR 2025
Phone Unlearning trade off
Address Unlearning trade off
Forget
Pergell
Forget 400 data
Forget data
Forget: 200 data
Midross I5X score
20
60
Forget
Methods
Forget
Methods
Forget 20 date
Forget 20 data
Furget 50 d
Forget 50 data
100
Forget
Forget
Forget
Forget data
60
Forget date
Forget
3.0
2.5
2.0
1.5
2.4
2.2
2.0
1.5
1.6
14
3.2 1.0 0.8
Model Acerage Perplecity log
Averaage Derpleckly log scale
Figure 4: Unlearning method trade-off: Risk score vs forget number of data. left: phone numbers; right: physical addresses
where dNewton (k) = (L(K)/V2L(k) is Newton's Direction in the second-order Newton's s Method. The maximization is achieved when d'ewwon (k) 0+ Since VL(k)
dNeevton (k) + 0+ is achieved when V [(k) -oc, which implies that the gradient at k quickly approaches 0.
U TRAINING SETTING AND HARDWARE
The training settings for fine-tuning the LLM on the Enron and Fraud email datasets, well as for implementing defensive methods such as gradient descent and ascent, are as follows: batch size of 4, the AdamW W optimizer, a learning rate of 5e-5, weight decay of 0.001, a cosine learning rate scheduler, and a warmup ratio of 0.03. All experiments were conducted using NVIDIA Quadro RTX 6000 24GB GPUs.
V ADDITIONAL ANALYSIS ON UNLEARNING SCALING EXPERIMENT
V.I UNLEARNING METHOD TRADE-OFF
To analyze the break-even point of the unlearning method, we conducted experiments focusing on both phone numbers and address unlearning. We tested the forgetting of 20, 50, 100, 200, and 400 data points. The results indicate that as more data points are forgotten, greater number of phone numbers and physical addresses are preserved. However, this leads deterioration in the model's performance, as illustrated in Figure + We discovered that forgetting between 200 and 400 data points significantly increases perplexity and indicated that the break-even point for the unlearning method is when between 200 and 400 data points are forgotten.
W BROADER IMPACTS.
The societal implications of our work include positive impacts, as it can protect PII from fine-tuned L.MMs with only a negligible drop in performance, ensuring that the LLMS remain effective for their intended purposes. And it is unlikely to cause significant negative societal impacts.
22


Reviews:
[{"review_id": "zPjDHAYzvf", "reviewer": "Reviewer_Da8V", "summary": "The paper proposes Proactive Privacy Amnesia, to unlearn PII in LLM while preserving its utility. The framework is composed of sensitivity analysis, selective forgetting, and memory implanting. The empirical results demonstrate the effectiveness in eliminating the privacy leakage risk.", "questions": "- The sensitivity analysis is expected to reduce the impact on utility of the LLM. However, in Table 5, Unlearning + Memory Implanting shows model similar performance with Proactive Privacy Amnesia, with increased attack success rate. How to interprete such result? Why the sensitivity analysis could decrease the attack score?\n\n- What's the average proportion of the top-k token in terms of the total length of the private information?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The sensitivity analysis is interesting and inspiring, minimizing the impact on model performance.\n- The proposed framework outperforms the baselines in terms of utility and defense ability.", "weaknesses": "- The authors can compare their framework with differentially private decoding [1] and finetuning [2] methods, which also prevent the model from outputing sensitive information.\n\n- Limited evaluation on model utility. The author only evaluate the model's utility on the same unlearning dataset, i.e., Enron Email. However, it's desirable to evaluate the model's performance on general downstream tasks, such as the GLUE and MMLU dataset.\n\n- The sensitivity analysis aims to isolate tokens that carry a higher amount of information. What if the attacker has some prior knowledge of the phone number/address? It may increase the attack success rate by conditioning the prediction on the former part of the private information. Furthermore, would unlearning the latter part, i.e., the tokens after the top-k, lead to better performance?\n\n[1] Majmudar, J., Dupuy, C., Peris, C., Smaili, S., Gupta, R., & Zemel, R. (2022). Differentially private decoding in large language models. arXiv preprint arXiv:2205.13621.\n\n[2] Li, Xuechen, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. \"Large Language Models Can Be Strong Differentially Private Learners.\" In International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P20KhvKzud", "reviewer": "Reviewer_Pc5A", "summary": "In this paper, the authors propose a scheme to selectively protect PII in Large Language Model training data. The approach consists of three major components: sensitivity analysis, selective forgetting, and memory implantation. The results are evaluated across multiple LLM attacks, demonstrating that the defense shows promising effectiveness.", "questions": "1. Will the method leak any other user's PII when it's not protecting all samples, causing some originally safe information to be exposed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Has theoretical justification for methods. It's impressive that the authors clearly explain the method's concept with theoretical justification, which definitely helps readers understand.\n2. Includes multiple attacks for evaluation. The authors consider multiple attacks in the evaluation, showcasing that the proposed method is robust under different settings.\n3. Results seem promising with utility-privacy tradeoff. Based on the results, the proposed method appears promising, though slightly less effective than some prior work in certain scenarios. The authors explain the advantages of the proposed solution well.", "weaknesses": "1. Method seems straightforward; each component seems lacks novelty. The sensitivity analysis appears to follow standard definitions of PII in language models. The selective forgetting component proposes a loss function, but it is relatively simple, and the memory implanting seems directly referenced from prior works. Additional insights and modifications to tailor these components specifically for PII protection may be needed to make the method more innovative. For example, consider designing the memory implanting to optimize performance for different types of PII. Similar innovations should be highlighted for each component.\n2. Evaluation may benefit from additional metrics, such as exposure. I feel that including more metrics beyond attack metrics and model performance metrics would provide more comprehensive insights into the proposed method. For instance, adding an exposure metric, as discussed in [1], could measure the memorization and likelihood of extraction when partial information is not protected. This evaluation would also better align with the design of other baselines compared in this work and help readers better understand the effectiveness of the proposed method.\n3. No comparison with differential privacy-based methods. There seems to be a lack of comparison with differential privacy-based methods in the evaluation, such as [2]. Since differential privacy is a mainstream defense approach, comparing it with selective-DP should also be considered in this paper. More discussion and comparisons on the privacy guarantee and utility to highlight the pros and cons of the proposed method and the dp-based methods will be appreciated.\n\n[1] Carlini, Nicholas, et al. \"The secret sharer: Evaluating and testing unintended memorization in neural networks.\" 28th USENIX security symposium (USENIX security 19). 2019.\n\n[2] Shi, Weiyan, et al. \"Just Fine-tune Twice: Selective Differential Privacy for Large Language Models.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jfRzED1ZDD", "reviewer": "Reviewer_iQMj", "summary": "This work studies the unlearning of personally identifiable information (PII) in large language models (LLMs), proposing a three-step approach called Proactive Privacy Amnesia (PPA). Given target PIIs to forget, the approach first identifies the key elements in memorized PII, then focuses exclusively on forgetting those key elements, and finally implants alternative information.", "questions": "1. In some unlearning tasks, such as Physical Address Defense in Table 4, PPA does not appear to outperform DEPN. A more detailed analysis of the cases where PPA underperforms/outperforms DEPN is warranted.\n\n2. A question arises regarding the security of the unlearning paradigm. The unlearning of specific PII is reasonable if the PII was indeed learned by the LLM. How can individuals or LLM producers verify that the PII has been learned? Additionally, is the proposed concept of the \"memorization factor\" robust enough to detect unreasonable PII unlearning queries?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of unlearning in LLMs is critical for conforming to the \"right to be forgotten\" legal principle and is essential for protecting personal information from misuse by LLM producers.\n\n2. The paper introduces the concept of the \"memorization factor\" to quantify the sensitivity of PII.", "weaknesses": "Given the numerous unlearning targets, the resulting unlearned model may degrade in various aspects. While current evaluation metrics for LLM performance include perplexity and email completion, additional metrics are needed to measure the utility loss due to unlearning. Specific metrics from the paper \"Knowledge Unlearning for Mitigating Privacy Risks in Language Models\" could be particularly relevant. Consider including metrics such as TruthfulQA and HellaSwag.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0c2bF3rVbT", "reviewer": "Reviewer_J1tm", "summary": "The article introduces a novel method to protect personally identifiable information in language models without compromising performance. Inspired by amnesia research, the PPA approach selectively forgets sensitive PII while preserving model utility. It involves three main steps: sensitivity analysis to identify critical elements within PII, selective forgetting of these elements, and memory implanting with non-sensitive information to maintain functionality. Experimental results on phone numbers and addresses show that PPA effectively reduces privacy risks and maintains high model performance compared to other methods.", "questions": "- How scalable is this approach? \n- How are the values in Table 1 decided?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-   The approach is distinctive as it draws from cognitive science concepts, such as anterograde amnesia, to design a targeted forgetting mechanism. This use of selective forgetting combined with memory implanting to balance utility and privacy represents a novel adaptation.\n- The paper supports its claims through a well-designed series of experiments on models like LLaMA2 and LLaMA3, benchmarked against multiple datasets (Enron and Fraud email datasets) and evaluated using diverse attack methods.\n-  The PPA method’s adaptability, allowing control over the degree of forgetting, offers practical value by enabling fine-tuning based on specific privacy requirements.", "weaknesses": "-  The study primarily focuses on phone numbers and physical addresses as examples of PII. While this provides a starting point, PII often includes more complex and variable types, such as emails, social security numbers, or unique identifiers that may be more difficult to detect and selectively forget.  \n-  The effectiveness depends on accurately identifying \"key elements\" within PII sequences. However, the method lacks a detailed discussion on the robustness of this sensitivity analysis, especially for more nuanced or less structured PII.  \n-  Implementing selective forgetting and memory implanting on large-scale models (e.g., with billions of parameters) can be computationally intensive. The paper does not address potential scalability challenges or discuss optimization strategies for efficiently applying PPA to larger models.\n- As a last and perhaps pedantic point, the table captions are lacking. There are too many tables, and their captions and discussion are tedious at best. In general, the metrics are not described well, and there are columns such as risk score that list values, and these are not clear. Columns could have down or up arrows to indicate if higher is better or the reverse. \n- Figures are not of high quality. For example, figure 3 is not legible on paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)