Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Authors: Rongzhe Wei
Mufei Li
Mohsen Ghassemi
Eleonora Kreacic
Yifan Li
Xiang Yue
Bo Li
Vamsi K. Potluru
Pan Li
Eli Chien
Venue: ICML 2025 poster (2025)
Keywords: Machine Unlearning
Minority Groups
Large Language Models
Primary area: social_aspects->privacy

Abstract: Large Language Models (LLMs) embed sensitive, human-generated data, prompting the need for unlearning methods. Although certified unlearning offers strong privacy guarantees, its restrictive assumptions make it unsuitable for LLMs, giving rise to various heuristic approaches typically assessed through empirical evaluations. These standard evaluations randomly select data for removal, apply unlearning techniques, and use membership inference attacks (MIAs) to compare unlearned models against models retrained without the removed data. However, to ensure robust privacy protections for every data point, it is essential to account for scenarios in which certain data subsets face elevated risks. Prior research suggests that outliers, particularly including data tied to minority groups, often exhibit higher memorization propensity which indicates they may be more difficult to unlearn. Building on these insights, we introduce a complementary, minority-aware evaluation framework to highlight blind spots in existing frameworks. We substantiate our findings with carefully designed experiments, using canaries with personally identifiable information (PII) to represent these minority subsets and demonstrate that they suffer at least 20\% higher privacy leakage across various unlearning methods, MIAs, datasets, and LLM scales. Our proposed minority-aware evaluation framework marks an essential step toward more equitable and comprehensive assessments of LLM unlearning efficacy.

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Rongzhe Wei!
Mufei Li"
Mohsen Ghassemi Eleonora Kreačić² Yifan Li
Xiang Yue Bo Li
Vamsi K. 2otluru Pan Li Eli Chien
Abstract
1. Introduction
Large Language Models (LLMs) embed sensitive, data, prompting the need for un- learning methods. Although certified unlearning offers strong privacy guarantees, its restrictive as- sumptions make it unsuitable for LLMs, giving rise to various heuristic approaches typically as- sessed through empirical evaluations. These stan- dard evaluations randomly select data for removal, apply unlcarning techniques, and use membership inference attacks (MIAs) to compare unlearned models against models retrained without the re- moved data. However, to ensure robust privacy protections for every data point, is essential account for scenarios in which certain data sub- sets face elevated risks. Prior research suggests that outliers, particularly including data tied to mi- nority groups, often exhibit higher memorization propensity which indicates they may be more dif- ficult to unlearn. Building on these insights, we in- troduce a complementary, minority- aware evalua- tion framework to highlight blind spots in existing frameworks. We substantiate our findings with carefully designed experiments, using canaries with personally identifiable information (PII) to represent these minority subsets and demonstrate that they suffer at least 20% higher privacy leak- age across various unlearning methods, MIAs, datasets, and LLM scales. Our proposed minority- aware evaluation framework! marks an essential step toward more equitable and comprehensive assessments of LLM unlearning efficacy.
Large Language Models (LLMs) are trained on vast and diverse datasets, often sourced from public content on the web, much of which is generated by humans (Touvron et al., 2023; Ouyang et al., 2022). This practice raises significant ethical concerns, particularly when the data includes sen- sitive information, leading to potential privacy violations. Individuals whose data has been used may seek to exercise their "right to be forgotten". a protection guaranteed by regulations such as the General Data Protection Regulation (GDPR) (Krzysztofek, 2018).
The ideal approach to fulfilling such request is to retrain the LLM from scratch, excluding the data to be removed. However, this solution is prohibitively expensive and im- practical for large-scale models. To address this, the concept of machine unlearning has emerged as a promising alter- native. Machine unlearning seeks efficiently modify the LLM so that becomes statistically indistinguishable from model retrained from scratch. In this way, adversary could confidently determine whether model has undergone an unlearning process or been retrained, ensuring compli- ance with the "right to be forgotten".
Unfortunately, it remains an open problem to enforce the formal unlearning guarantee for deep neural networks and LLMs without exact retraining. Despite recent progress in theoretical unlearning research (Guo et al., 2020; Sekhari et al., 2021; Neel et al., 2021; Ullah et al., 2021; Chien et al., 2023; Ullah & Arora, 2023; Chien et al., 2024a;b), their restrictive assumptions limit practical applicability to deep neural networks and LLMs. Concurrently, researchers have developed efficient unlearning heuristics and empirically evaluated their efficacy (Golatkar al., 2020a;b; Graves et al., 2021; Liu et al., 2024a;c; Yao al., 2024), often by comparing approximately unlearned models to those retrained from scratch (Pawelczyk et al., 2024a). Among the various evaluation methods, membership inference attacks (MIAs) (Shokri et al., 2017), originally developed to infer data usage during training, have been widely adopted for assessing unlearning performance (Shi al., 2024b).
'Georgia Institute of Technology PJ.P. Morgan Al Research "Tsinghua University *Carnegie Mellon University 'The Univer- sity of Chicago "University of Illinois Urbana-Chin -haamaan. Cor- respondence to: Rongzhe Wei <rwei42 @ gatech.edu>, Pan Li <panli@ @ gateii@@aaecchh..... Eli Chien <elichien @ww.eei..ooo......
Proceedings of the 4rnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
CCode source: https://gitee cot.:///acoo//aaaoocooco Soority_Awaree LLM Un wwwnmeeerninngg..
We identify critical pitfall in the aforementioned LLM

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
unlearning efficacy evaluation. Prior work has demonstrated that unlearning difficulty can vary substantially across in- dividual data points in vision tasks (Thudi et al., 2024; Zhao et al., 2024). Similarly, recent studies on LLMs also reveal highly non-uniform memorization patterns, where certain samples are memorized much more strongly than others (Feldman & Zhang. 2020; Carlini et al., 2022). How- ever, current unlearning evaluation methods only capture daverage-case" performance through random data removal from the training set. This approach inadequately addresses privacy risks for hard-to-unlearn data, failing to account for challenging scenarios necessitating rigorous privacy pro- tection (Steinke & Ullman, 2020; Aerni et al., 2024). It neglects the principle that every individual's right to for- gotten should be upheld equally, thus ignoring data from minority groups, which are often treated as outliers and can be more resistant to unlearning due to the aforemen- tioned stronger memorization effects (Carlini et al., 2022; Nasr et al., 2021; 2023). Consequently, standard unlearn- ing evaluation significantly underestimates privacy risks for these groups, overlooking crucial social responsibilities in personal data protection.
2. Related Work
Privacy auditing is a fundamental yet challenging aspect of LLM unlearning due to the difficulty of distinguishing train- ing samples effectively (Duan al., 2024). Various privacy- related metries, such as exposure (Carlini et al., 2019), mean reciprocal rank (Wu et al., 2023), extraction likelihood (Jang et al., 2022), and truth ratio (Maini et al., 2024), have been proposed to probe privacy leakage. Among these, MIAs remain one of the most crucial tools for evaluating machine unlearning methods (Liu et al., 2024b). Standard MIAs typically involve training numerous shadow models inde- pendently empirically approximate the distribution (Car- lini et al., 2022). This approach has also been adopted for LLM unlearning, as seen in the NeurlPS 2023 Machine Un- learning Challenge* (which compares the point-wise output distributions of multiple unlearned and retrained models to perform MIAs) and Kurmanji et al. (2024); Pawelezyk et al. (2024b). Hayes et al. (2024) further highlights the limitations of average-case evaluations and introduces spe- cialized per-sample MIA method for unlearning evaluation. Their approach focuses on unlearning randomly selected subset of training data by training a series of shadow models and performing per-sample MIA using a likelihood ratio test under Gaussian fitting (Carlini et al., 2022).
Contributions. Motivated by the privacy auditing litera- ture (Jagielski et al., 2020; Steinke et al., 2024), we conduct a synthetic experiment on unlearning injected canaries per- taining to minority groups. We choose Personally Identifi- able Information (PII) as a representative minority identifier, while noting that our approach extends to broader cases. We show that minorities suffer from least 20% more privacy leakage in most cases across combinations of six unlearning approaches, three MIA variants, three datasets, and two LLMs of different scales. These results underscore the prevalence of the issue in practical settings, highlighting the need for a more effective LLM unlearning evaluation, particularly in regard to privacy risks for minority groups Accordingly. we propose minority- aawaee LLM unlearning evaluation protocol (Figure 1) as an initial step toward this goal. With this minority-awan protocol, we benchmark existing unlearning approaches and investigate the effects of forget set size as well as unlearning complexity. This study provides a more holistic understanding of different L.LM un- learning approaches for practitioners. Notably, we observe that Langevin Unlearning- the only approach incorporating noise-achieves favorable privacy-utility trade-off com- pared to noiseless methods such as SCRUB and Gradient Ascent (GA), suggesting a potential crucial role of noise incorporation in effective unlearning. In summary, these insights underline the critical role of our minority-aware evaluation framework in advancing equitable assessments of unlearning efficacy across different methods.
A major downside of MIA approaches involving shadow models is their computational expense, as they require train- ing a large number of LLMs independently (Liu et al., 2024b). To address this downside, another line of re- search compares the outputs of models using different sta- tistical metrics without requiring shadow models (Zhang et al., 2024; Liu et al., 2024a;c; Yao et al., 2024; Li et al., 2024a). making these methods more computationally fea- sible (Maini et al., 2024). For instance, Shi et al. (2024b) measures privacy risk through the normalized AUC differ- ence between unlearned and retrained models, using MIAs such as Min-K% (Shi et al., 2024a). However, these works typically select the forget set randomly from the training set, corresponding to an average-case evaluation. Our study highlights a critical limitation of this approach: the pri- vacy risks of minority populations within the training set are severely underestimated because minority data are less likely to be selected in the unlearning evaluation pipeline. By focusing on minority-aware scenarios, our work provides a more comprehensive perspective on unlearning evaluation and privacy risks. In a related context, Zhao et al. (2024) explores how the memorization of image representations affects the difficulty of unlearning individual samples in vision tasks by introducing entanglement- Our work further explores the systematic degradation in unlearning efficacy for minority subgroups during the fine-
metrie.
Pacppss/uuullaariinggchllleee www.lllegggggghhuuue /c/ asseta/data/Maanee www.ieee___neeeiiee Met
2

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Unlearning Cases
Retraining Case
Initial Large Language Models
Mo
GPT
one
(1) Random
(2) Canary
(3) Minority
AA
Train A
A
D Detain
Dretain
Dorreet
Pretain
Deanary
Dretain
Dminority
M Mretrain
M (1) train
M (2)
M (3)
train
train
Prior works:
Unlearn U
unlearn v.s. Metrain
(1)
D forget
Deanary
D minorrtt
Our work:
M
(1)
M
(2) unlearn
M
(3) unlearn
M (1/2/3)
unlearn v.s. M Mretrain
unlearn
Figure 1. Illustration of our proposed LLM unleaming approaches (highlighted in red), when compared with the existing pipeline. Standard I.LM unlearning evaluation typically involves randomly sampling data for removal from the training set (Case 1), which may underestimate privacy leakage for minority groups. In contrast, design experiments to assess unlearning efficacy by removing canaries (deliberately inserted data points) related to minority groups (Case 2) and by directly removing data from minority groups (Case 3). Our approach provides
comprehensive, minority-aware evaluation by considering the result across the three settings.
tuning of large language models. As future direction, it would be valuable to examine the relationship between the intanglement-based difficulty metrics proposed by Zhao et al. (2024) and the mino phenomena we identify in natural language contexts.
tion the training set Duran Diorget Dkeep into a subset to be forgotten later Diorget and keep Dkeep- An unlearning algorithm UI takes Mierrn Dranges and Durai input to re- turn updated model Munleam +- Ul (.Meam, Drorett Duain). It is worth noting that Muntcarn depends on the choice of Dirges-- The gold standard to adhere to "the right to be forgotten" is retraining without Diorget. namely Meetrain + A(Mo, Dimin \ Dtorget). We say achieves good unlearning efficacy if Munkeam and Meeuain are indistinguishable in their behavior m( Munterm: D) mMMaurain D) any corpus D, where m is any evaluation metric. Since Munteam and M depend on the choice Diorgtt such approximation should be taken over the worst case ideally.
3. Preliminaries
Machine unlearning (Cao & Yang, 2015; Bourtoule al., 2021) has emerged as important direction in trustworthy language models. It was initially motivated by privacy due to "the right to be forgotten" from GDPR and later extended to other legal and ethical concerns, including copyright (Yao et al., 2024), biased or outdated information mitigation (Liu et al., 2024b), hallucination removal (Yao al., 2023), en- tity forgetting (Maini et al., 2024) and data poisoning re- moval (Pawelezyk et al., 2024a; Li et al., 2024b). In this work, focus the privacy aspect of the problem, albeit our methodology extends to other cases whenever the in- distinguishability to the retrained model is an appropriate metric.
3.1. Efficient MIAS for LLM Unlearning
As previously discussed, the effectiveness of unlearning methods can be measured by the indistinguishability be- tween the resulting unlearned models and an exactly re- trained model. MIA is often leveraged to determine whether specific sample is part of the training set and widely applied to audit training data privacy leakage. Therefore, to consider evaluate the efficacy of an unlearning approach, the PrivLeak (PL) metric (Shi et al., 2024b). First, we define the difference in AUC scores between the unlearned model Munkemm and the retrained model M M as AAUC = AUC(M Diorges Dies) - AUC(Mreminn Diorget Dies)), where AUC the AUC-ROC score of an MIA (Ye et al., 2022) that tries discriminate samples from Dioget and
We briefly state the generic machine unlearning setting for privacy. Assume training dataset Durain and holdout test set Diest are given. Let Meam --A(Mo Dumin) be the lan- guage model trained on Dirai starting from initial model Mo via the training algorithm A, which may be either a pre- trained language model or random initialization. Once the model is trained, we receive data removal requests that parti-
3

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Dest based on the output statistics (e.g. loss) of a given model M. Then, the PL metric is formulated as:
to common training samples (majorities) (Feldman & Zhang, 2020; Carlini al., 2022). Intuitively, similar phenomenon persists for unlearning.
AAUC
PrivLeak (PL) AUC Meetrani Diveget: Duess)
Here we utilize the Enron dataset as a case study. This dataset comprises 535,703 authentic emails from 158 employees of the Enron Corporation. It is a stan- dard benchmark dataset for studying PII leakage, where the phone number is one form of PII that has been extensively studied (Lukas et al., 2023). The phone numbers here follow the format of the U.S. phone numbers (e.g., 123-456-7890), with the first three digits serving as the area code, representing the loca- tion where the number holder applied for the number. Such information is considered sensitive as it leaks not only the phone number itself, but also the geographic information pertaining to the number holder.
Table 1. Top three most fre- quent and least frequent area codes within Enron dataset.
By normalizing the difference in AUC scores between Munteumn and Mretrain using the AUC of Myetrain- the met- ric accounts for the inherent difficulty of distinguishing the forget and test sets. Note that for an effective unlearning method, the metric should be around zero since the behavior of Munlcam, Merrain are indistinguishable. A larger magni- tude of the PL metric implies greater amount of privacy information that has been leaked under the tested MIA. A positive value indicates that the sample has not been fully forgotten, as the attacker has a higher AUC for Mankeam than Myetmann Conversely, a negative metric value suggests over- forgetting, which still indicates that Munkeamn differs from Meatran and thus cause privacy breaches. Finally, note that an effective unlearning solution should lead to a small PL metric for any choice of MIA. In this work, we consider three popular MIAs and report the corresponding PL metric.
Area code
Count
713 (Houston)
135,307
800 (Toll-free)
11,902
212 (New York)
10,739
484 (Allentown)
1
Table illustrates the least frequent and three most frequent area codes in the Enron dataset. The area code distribution is far from uniform. Consequently, if emails containing phone numbers are uniformly sampled for the forget set Dforgot minority data, such as emails with rare area codes like 484, are un- likely to be included due to their lower frequency. If unlearning minority data is inherently more challenging and results in greater privacy leak- age, the existing evalua- tion pipeline may under- estimate privacy risks for minorities.
lossMIA (Yeom et al., 2018): Determines membership of a sample for model M based on its loss (M;
zlibMIA (Carlini et al., 2021): Determines membership of a sample based on the sample loss normalized by its zlib compression size, E(M; =//zzib(x))
10
Counts
10a
Min-K% (Shi et al., 2023): Selects the lowest K% of token likelihoods and leverages the corresponding negative log-likelinn liceliool for membership inference.
10
130
4. The Underestimated Privacy Risk of Data Minorities
Figure 2. Area code histogram in Enron Dataset.
Recall that both the unlearned Manteum (Mieams Diorges Dirain) and retrained Myetrain A(Mo, Derain \ Dronget) language models depend on the choice of the forget set Droeget. Whenever we estimate the privacy leakage of unlearning method 2/ via some evaluation m. it is important to account for potential high-risk partitions of Dineeet to ensure comprehensive assessment of privacy risk. Unfortunately, the current LLM unlearning evaluation pipeline overlooks this critical aspect, where the partition leading to Dioreet is chosen uniformly at random (Jang et al., 2023; Chen & Yang, 2023; Yao et al., 2024; Maini et al., 2024; Zhang et al., 2024; Shi et al., 2024b). The reported privacy risk therein hence corresponds to the "average case", which may significantly underestimate the privacy risk of highly privacy-sensitie points that request unlearning. It is known in the privacy literature that some rare training samples (minorities) may have an outsized effect on model memorization compared
4.1. Verify Underestimated Privacy Risks of Minority via Canary Injection
To rigorously show that removing data from minority popu- lations indeed leads to higher unlearning privacy leakage, we design experiments based on the idea of canaries in the privacy auditing literature (Jagielski et al., 2020; Steinke et al., 2024). For simplicity, we focus on the scenario where data removal requests pertain to PIls, where each training sample D consists of PIIs such as phone numbers or organizations. We choose PIIs as a representative minority identifier, albeit a similar idea extends beyond PIls. We
Minority here refers any subset of the data, defined by some shared value, that is under-representer in the training set. This term is generic and may apply to any type attribute, demographic or otherwise.
4

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Doorget
D Dcanary
D minority
[.-.] Body: email address: spalmer@cff.nef..o ref.co..eff.commmmmmmmmm Home address: 264 Iven Ave. Apt 3-A St. Davids, PA 19087
[.-] Body: [..]]
I have forwarded your
[.-] Body: [..]]
I have forwarded your request to Zarin Imam at EES. Her phone number is 484-853-7107, [..]]
request Zarin Imam EES. Her phone number is 713-853-7107, [...]
[...] Cell Phone:
484-432-5656 [...]
Replace the entire email with the email containing phone number with least frequent area code.
Identical except area codes are replaced with the least frequent one (i.e., 484).
Figure 3. Illustration of the forget set Diorgas the construction the canary set Deanayy and minority set miincity for the Enron dataset. The minority set consists of emails with phone numbers containing the least frequent area codes. A histogram and distribution of area codes (e.g.. .. 713 as the most frequent and 484 as the least frequent) are shown in Figure 2 and Table
consider the following cases, see Figure| for an illustration. 1) Random: we randomly partition Diu Piorget U Dkeep as in the standard unlearning evaluation pipeline. This leads to (1 + A(Mo, D 2) Canary: For the same im we construct canary set D Dennary {= where each z is identical to x; except that only the PII is replaced by the least frequent among ((1) isolating the impact of non-PIl components. Finally, construct synthetic training set ( Deamny keeess
unlearning evaluation pipeline, it cannot quantify the pri- vacy risk for minorities in the real-world setting. We further design the third case aiming at quantifying the amount of underestimated privacy risk by directly choosing data be removed containing the least frequent PII. 3) Mi Minorityy ritty: construct set minority that is of the same size as Diorget in Random, which consists of samples with the least fre- quent PII within the dataset. By comparing the computed privacy risk of Random and Mi ifinority ty,, we can quantify the amount of underestimated privacy risk for data removal from minority groups compared to the average case. If the resulting privacy risk is significantly higher than Random, any conclusion pertaining unlearning efficacy drawn from Random can be misleading and the right to be forgotten of minorities is overlooked.
forget Droret
which leads to (2) + A(Mo; D ( By executing the same unlearning evaluation process for both cases M and M(2)) we aim to show that the privacy risk for Canary is much higher than Random. By applying the same unlearn- ing algorithm for removing Forget and Deanary, we obtain unleam respectively. The privacy leakage (PL) is then computed for these cases as described in Section 3.1. The calculation of PL for Canary entails replacing Prorget with D Deanary in Eq. (1). Note that the retrained model Myetrain < A(Mp, Dkeep) is identical for both scenarios.
the unlearned model M( unleurn and M22)
5. Unlearning Methods
We evaluate the following popular unlearning approaches in the literature. With slight abuse of notation, we denote M for both the model and its parameters for simplicity.
An illustrative example of canary construction is provided in Figure Note that for each email within Diages in Random, construct the corresponding canary by only replacing its area code with the least frequent one (i.c., 484). This design is critical as we ensure the other part of the email is identical to the original email. Hence, if the privacy leakage of Canary ry is greater than Random, it must be due to the difference in the area code. We repeat the similar canary construction for the other PII such as email domain and year of legal judgment for different datasets.
Random Labels (RL) (Golatkar et al., 2020a; Yao et al., 2024): In the context of next-token prediction, this method involves randomly selecting tokens from the entire vo- cabulary during training on Diorget aiming disturb the model's learning from this dataset. The intuition behind this approach is that model uninformed by Diaret should behave as though it is randomly guessing the next token. However, as argued Yao et al. (2024), this intuition may not always hold, depending on the specific scenario.
Exact Unlearning (EUk) and Catastrophic Forgetting (CFk) (Goel al., 2022): Exact unlearning can be done by retraining the entire model from scratch on Dreep- al- beit it is prohibitively expansive in practice. Goel et al. (2022) proposes EUK method, which retrains only the last layers of the model while freezing the other layers. As result, it computationally cheaper than retraining the
4.2. Quantify the Underestimated Privacy Risk of Unlearning Minority
While our synthetic experiment on canary injection may be used to verify whether the unlearning privacy risk of mi- nority populations is underestimated in the standard LLM
5

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
entire model. They also propose the CFk method, which continues training the last & layers the Dkeep without retraining from scratch, while freezing the other layers.
sharding-based framework SISA (Bourtoule al., 2021), demand significant computational and storage resources. SISA achieves exact unlearning by training multiple models independently on disjoint data partitions, but this deviates from standard machine learning workflows and introduces considerable memory overhead.
Gradient Ascent (GA) (Golatkar et al.. 2020a; Graves et al., 2021; Jang et al., 2023): Gradient ascent is arguably the most popular heuristic for machine unlearning. It seeks to remove the influence of the Dionget from the trained model by reversing the gradient updates associated with Diorgat. Notably, researchers have reported that gradient ascent can lead to significant model utility degradation in some cases (Iharcco et al., 2023; Pawelezyk et al., 2024a).
5.1. Enforcing the Same Computation Budget for Unlearning Methods
We categorize all methods as follows: those that only require the forget set (RL, GA), those that only require the keep set (EUk, CFK, Langevin), and those that require both the forget and keep sets (NegGrad+, SCRUB). Since machine unlearning is about the trade-off between privacy-utility- efficiency (Guo et al., 2020; Chien et al., 2024a; Liu et al., 2024c). we carefully ensure a similar computational com- plexity for all tested unlearning methods when demonstrat- ing the privacy-utility trade-off. We define a Complexity Unit as the gradient computation budget of one training epoch on Drorgel samples and limit all unlearning methods to a maximum of 10 Complexity Units. Since [Dtorge = U is roughly 1% of |mmin| throughout our experiments, all unlearning methods are indeed much more efficient than retraining from scratch (Pawelczyk et al., 2024a).
NegGrad + (Kurmanji et al., 2024): N is a com- bination of gradient ascent on Dirget and gradient descent on Deepp. It finetunes the current model by optimizing:
B Daop [((M;=)) - (1 B) Big ((MM;
where 3 E (0, 1) is hyperparameter and È is the empiri- cal expectation. The intuition is to "review" the informa- tion from Deeep in order to prevent the model degradation due to the gradient ascent.
SCRUB (Kurmanji et al., 2024): SCalable Remembering and Unlearning unBound (SCRUB) is a state-of-the-e unlearning method that leverages a student-teacher frame- work. It updates the model by optimizing the objective:
For unlearning approaches that leverage D Dioret only, they can be tuned via unlearning process for at most 10 epochs. For those that leverage Dkeep only, we randomly subsam- ple it to size U for each epoch and unlearn for at most 10 epochs. For methods that leverage both Dianee and Dreep simulraneously, we limit their maximum unlearning epoch to 5. The situation is slightly more complicated for EUK and CFk approaches since only the last & layers are trained to save computation. We randomly select U/r samples from the keep set in each epoch, where is the ratio of trainable parameters in the last k layers compared to the total number of parameters in the model. Our setup ensures that all tested unlearning approaches exhibit a similar unlearning compu- tational complexity for a fair comparison. We optimize the unlearning epoch for each method under the 10 Complexity Unit constraint by the following criterion: if the perplexity of the unlearned model on D Duain increases by more than 1 point compared that of the initial model (No Unlearn), we stop at the first epoch where this condition is met; otherwise, we use the checkpoint from the last epoch.
Exg- Drase [KL [M (2)] + +M)
Eirge (e) (x))
where KL is the Kullback- divergence. SCRUB shares a similar intuition with NegGrad-+, which can also be viewed as a combination of gradient ascent on Droget and descent on Dxeep- Nevertheless, instead of directly employing the original loss f, SCRUB leverages the KL divergence to the original model M It provides a different regularization compared to NegGrad+.
Langevin Unlearning (Chien et al., 2024a;b): Langevin Unlearning leverages noisy gradient descent for machine unlearning. Specifically, during the training process, it re- places the common gradient descent with DP-SGD (Abadi et al., 2016). For unlearning process, it finetunes the model on Dxeep with DP-SGD as well. Chien et al. (2024a) estab- lishes a smooth theoretical connection between differential privacy and unlearning and shows that Langevin Unlearn- ing can provide formal privacy guarantee for non-convex problems. Unfortunately, they mentioned that the resulting privacy bound is too loose to be applied in practice. We test Langevin Unlearning empirically in our experiments.
6. Experiments
Datasets. Our LLM unlearning evaluation is conducted two representative PII datasets: Enron (Klimt & Yang, 2004) and ECHR (Chalkidis et al., 2019). The Enron dataset contains corporate emails released by the Federal Energy Regulatory Commission, while the ECHR dataset comprises legal case information from the European Court
All the above unlearning methods fall under approximate unlearning (Thudi et al., 2022), valued for their practical efficiency. In contrast, exact unlearning methods, such as the
6

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 2.
The privacy leakage (PI.) for each unleaming method against different attackers for GPT-2 / Ilama-2 7B the Enron-Phone dataset. The number in the parenthesis is the excess ratio of PL magnitude for cases Canary and Mincrity compared to Random, where a larger PL magnitude implies more severe underestimation of privacy leakage in the standard unlearning evaluation (Random). Bold font indicates the case that the amount of underestimated privacy leakage is at least 20%.
Method
PL (losMMAA)
PL (zlbbMAA)
PL (Min-K %)
Random
Canary
Minority
Random
Canary
Minority
Random
Canary
Minority
Enron- Phron-PPooeeaae Dataset / GPT-2
No Unleam
0.190
0.283
% 1)
0.340 (79% f)
0.052
0.076 (48% f)
0.064 (24% f)
0.300
(49% f)
0.524 (75%
RL
0.118
0.191 (61 % 7)
0.210 (77% f)
0.044
0.067 (52%) ()
0.060 (37% f)
0.258
0.401 (55% f)
0.447 (73% t)
BUK
0.027
0.080 (198% 1)
0.124 (362%
0.035
0.051 (47
0.052 (49% f1
0.092
0.215 (134%) ()
0.223 (143% f)
CHk
0.190
0.278
(46 % ()
0.337 (77% f)
0.053
0.075 (41%) ()
0.064 (21%) T)
0.298
0.435 (46% ()
0.514 (73% f)
GA
0.089
0.140
%
0.127 (42% f)
0.024
0.042 (73% ()
0.026 (7%/7)
0.151
0.242 (60% f)
0.171 (1.3%))
NegGrud+
0.183
0.271 (48% 1)
0.327 (79% ()
0.052
0.073 (42%) ()
0.058 (13%g)
0.293
0.435 ()
0.511 (74%-1)
SCRUB
0.167
0.251 (50%1)
0.321 (92% f)
0.048
0.070 (44%7)
0.062 (28% f)
0.295
0.450 (52%) f)
0.527 (78% f)
Langevia
0.093
0.144 (5
0.157 (69% ()
0.024
0.037 (54% 7)
0.027 (12%g)
0.160
0.258 (61%
0.264 (65% %
Enron-Phone Dataset / Laama-22 7B
No Unleam
0.060
0.242 (303% 0.172 (187% % f)
0.034
(188% 1) 0.067 (97% f)
0.076
0.115 (5
0.179 (136% f)
RL
-0.442
.0.084 (659.L) -0.055 (77%.L)
-0.055
0.065 (1400% f) 0.102 (2140% f)
.0.123
.0.073 (411%.)
0.012 (90S.L)
FUlk
0.057
0.246 (332% f) 0.185 (225% f)
0.039
0.106 (172% 1)
0.082 (110%11
0.063
0.132 (110%
0.189 (200% f)
CFK
0.057
0.236 (314%1) 0.168 (195% 1)
0.032
0.094 (194%-1)
0.063 (97% f)
0.072
0.108 (50%t)
0.171 (138%+1
GA
-0.562
0.430 (23%.L
(179.L)
(371% 1)
(593% 1)
-0.625
-0.459 (279..)
-0.517 (17%..)
NegGrad+
-0.074
-0.184 (149% -0.040 (46%LL
-0.021
0.048 (129% f1)
-0.002 (90%kL
-0,069
-0.271 (293%)1
-0.057 (17%) 1)
SCRUB
0.059
0.162 (175%f) 0.170 (188 % 1)
0.034
0.063 (8%%))
0.065 (91% t)
0.074
-0.031 (58%-L)
0.177 (139%% t
Langevin
0.033
0,180 (445% 0,104 (215% 1)
0,016
0,068 (325% t>
0,036 (125% 7)
0.033
0.055 ()
0.091 (176% 1)
of Human Rights. For our experiments, we focus on specific PIIs based on their distributions: phone numbers (Enron- Phone) and email domains (Enron-Email] in Enron, and the year of judgment (ECHR-Year in ECHR. Data minorities are defined based on these PIIs. Detailed dataset statistics are provided in App. 1. Our study centers on instance- level unlearning, treating each individual as a single record.
Zhang et al., 2022), where a lower perplexity indicates that the model is more confident in its predictions. Additional details are provided in App. B.
6.1. Standard Approaches Underestimate Privacy Risk for Minorities.
We report the results pertaining to the Enron-Phowe Enron- Email, and the ECHR- Year datasets. The experiment setting follows the explanation in Section and further details are relegated to App. A. Table 2 shows that across all three attackers (lossMIA, zlibMIA, and Min-K%), all six unlearning methods and original model (no unlearning), the privacy leakage measure is significantly larger when unlearning canaries and minorities on Enron-Phone dataset for GPT-2 and Llama-2 7B respectively. Notably, in almost all cases the privacy leakage is underestimated for at least 20%. similar phenomenon holds for the Enron-Emai and ECHR- (Table 4, 5, 7 in App. C.I) datasets. These results verify our claim that the current LLM unlearning evaluation indeed understated the privacy risk, especially for minorities. Our results call for a more careful empirical LLM unlearning evaluation, where considering canaries and minorities as we described can be an effective first step.
General Settings. We focus the fine-tuning scenario, where the initial model Mo is pretrained LLM (GPT- 2 (Radford et al., 2019) or Llama-2 7B (Touvron et al., 2023)). The fine-tuned model Miea is obtained by training Mo on a dataset Dusin for 5 epochs. In the GPT-2 experi- ments, both the training and test sets contain 10,000 samples, subsampled from the full dataset. For Llama-2, we employ efficient fine-tuning using LoRA (Hu et al., 2021); both the training and test sets consist of 50,000 samples, subsampled from the entire dataset. In all cases, the forget set size is set to 1 % of the training set size. The models are optimized using the AdamW optimizer with a constant learning rate of 10 following the settings described in Shi et al. (2024b). During the unlearning process, all unlearning methods are constrained to the same computational budget-not exceed- ing 10 complexity units-an as detailed in Section 5. We ensure that the unlearning complexity of each method is similar to allow for a fair comparison. Our ultimate goal is to achieve a superior privacy-utility: efffityeffffc tfade-off. We utilize MIA to estimate the empirical privacy risk measured by the PL metric as described in Section 3.1. For evaluating the utility of the LLMs, we report the perplexity following standard practices in the literature (Radford et al., 2019;
6.2. Benchmarking Unlearning Approaches under Minority-Aware Evaluation.
Motivated by our observations, we propose the minority- aware L.MM unlearning evaluation. Instead of reporting the privacy leakage (PL) score under the Random case, we pro-
7

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Phone
Email
GTT-2
L lama-2
GPT-2
L Llam-22
Nin K3
Utity
Min KSS
Utilty
Mi--0%
Utiity
Min KS
Parstotty budder rouuts sultt
Utily
3
(n]
[b]
(c)
(d)
(9)
181
[h)
Figure 4. Benchmarking unlearning approaches via our minority-aware evaluation for GPT-2 and Llama-2 on Enron-Phone (Lefft) and Enron-Email (Right) datasets. ()).)c)(e))()): Maximum privacy leakage (PL. over three cases (Randon, Canary, and Minority for Min-K% attack. (b), Worst perplexity over the three cases each method. More results on lossMIA and zlibMIA attackers are deferred to App. C.2.
pose to report the magnitude of maximum PL score of three settings (Random, Cynary, and Mi nority) This provides a better privacy risk estimation while keeping the entire evaluation pipeline efficient. Besides, we report the corresponding worst-case perplexity as the utility measure for each unlearning approach. We benchmark the popu- lar unlearning methods under our new evaluation pipeline, where the result is summarized in Figure for GPT-2 and Llama-2 the Enron- Paron-Phooeeaae and Enron-Email datasets. See App. C.2 for additional results.
rates ranging from 1c-7 to 1e-3. As shown in Fig.6, these curves evaluated the Enron-Phone and Enron-Email datasets.
Enron Phone Min k%
0.5
R1.
The results indicate that Langevin Unlearning (Green Line) outperforms SCRUB (Purple Line) with a superior privacy- utility trade-off. Notably, while GA performs reasonably well on the Enron-Phone dataset, its trade-off on Enron-Email is significantly weaker, revealing its instability. Detailed hyperparameter tuning for these methods is provided in App.C.6.
tuk
a 0.40-
SCRUB
SCRUB Forets Homter
022-
GA
0.20-
Northr
Perplexity
Enron-Email Min-
We found that Langevin Unlearning offers the best balance between privacy and utility empirically. Note that while gra- dient ascent has on-par performance compared to Langevin Unlearning on the Enron-Phone dataset, it significantly de- grades the model utility on the Enron-Emait dataset. This echoes the finding of Ilharco et al. (2023); Pawelezyk al. (2024a), albeit for different tasks. We found that gradient as- cent is inherently unstable. In contrast, unlearning methods that leverage keep set Disep are much more stable, including Langevin Unlearning and SCRUB.
Max PL
38
24
20
Perplexity (e- better results)
Figure 6. Privacy-utility Trade- off Curves for GPT-2.
Unlearning Iteration. We investigate the effect of unlearn- ing epochs on privacy (Max PI.) and utility (perplexity) for each unlearning approach in Fig. 5 (Left). We observe that RL and GA are unstable in PL score. Furthermore, these two methods can lead to significant model utility degradation in terms of perplexity, where even unlearning for 2 epochs can already result in model breakdown. This observation again demonstrates that gradient ascent, albeit being simple and popular, is not a reliable LLM unlearning solution. We should focus on stable unlearning solutions such as SCRUB and Langevin Unlearning.
We also present results using utility metrics, including SeRT (Zhang et al., 2019) and ROUGE (Lin, 2004), which capture semantic meaning. These results, provided in App. C.3, exhibit a consistent trend.
6.3. Ablation studies.
We present ablation studies the Enron-Phome dataset using the GPT-2 model, unless otherwise specified, and further results are deferred to App. C.4, C.5 and C.6.
Privacy-Utility Trade-off. We analyze the privacy-utility trade-off curves for stable methods like Langevin Unlearn- ing and SCRUB, along with the widely used GA. For Langevin Unlearning, we adjust the noise scale during train- ing and unlcarning. In SCRUB, balance the loss and KL regularizer terms in its objective function (Sec. 5). For GA, we explore different learning
Size of Forget Set. In Fig. 5 (Right), report the effect of different forget set sizes on the privacy (Max PL) and utility (Perplexity) trade-offs for each unlearning method. We find that both the GA and RL methods are highly sensitive to the forget set size, leading to significant model utility degrada- tion and poor reliability in practice. In contrast, methods like Langevin Unlearning demonstrate good performance in
vary the weights that
8

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Ni-kk%
Ubilty
Min-K%
Utility
0:60
50
Perplodry munutal
9.50
Perplexity beiter resulss)
Mathode
20-10
Scrob
440
Langesh
9.20
30
100
Comolosty Unit
Complexity Unit
Forget Size
Forget Set Size
(8)
(b)
(a)
(b)
The effect of unleamiin epochs for each unlcarning approach.(a) (a): Maximum PL. with the attacker being
The effect of forget size for each unlearning approach. (a): Maximum Pl. with attacker heing Min-K%. (Results on lossMIA, zlibMIA are deferred to App. C.S). (b): Model perplexity.
Mi..K% (b): Model perplexity.
Figure S. Ablation studies unlearning iterations and forget set size.
terms of stability.
to Martin Pawelczyk and Ayush Sekhari for their valuable clarifications on their work, and to Peter Kairouz for his insightful comments.
7. Conclusions
We identify a critical limitation in the typical evaluation pipeline for LLM unlearning efficacy: privacy risks to mi- nority groups in the training data are often underestimated. Through carefully designed experiments using unlearning canaries tied to minority groups, inspired by privacy au- diting research, demonstrate that minority groups face at least 20% greater privacy leakage average. Using personally identifiable information (PII) as proxy for mi- nority identifiers, we emphasize the need for more rigorous evaluations to ensure the right to be forgotten applies univer- sally. Benchmarking existing unlearning methods with our minority- evaluation reveals that popular heuristics like gradient ascent are unstable and can degrade model utility. In contrast, methods such as Langevin Unlearning achieve more favorable privacy-utility trade-off.
Disclaimer
This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates ("JP Morgan") and is not product of the Research Department of JP Morgan. JP Morgan makes representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This docu- ment is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, fi- nancial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
Impact Statement
This work introduces a minority-awar evaluation frame- work aimed at enhancing the comprehensiveness and equi- table nature of LLM unlearning assessments. Our research contributes to the standardization of user data protection and the responsible utilization of machine learning data. However, we emphasize the importance of using compliant, publicly available, or officially authorized data when con- ducting minority-aware evaluations with LLMs. Building on these foundations, our work aims foster more ethical and inclusive practices in the development and application of machine learning technologies.
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learn- ing with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Com- munications Security, CCS '16, pp. 308-318, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341394. doi: 10.1145/ 9976749.2978318 URL attps://dok or0//// 1145 2978311.
Aerni, M., Zhang, J., and Tramèr, Evaluations of ma- chine learning privacy defenses misleading. In ACM SIGSAC Conference Computer and Communications Security (CCS), 2024. URL htttts///assiiv.rrg abs / 244.244444 17399.
Acknowledgement
This work is supported by NSF awards CCF-2402816, IIS- 2239565, and JPMC faculty award 2023. The authors would like thank Junyuan Hong and Chulin Xie for their assis- tance in reproducing the code for I.IM-PBE: Assessing Data Privacy in Large Language Models. We are also grateful
Bourtoule, L., Chandrasckaran, V., Choquette-Chieee C. A.,
Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot,

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
N. Machine unlearning- In 202/ JEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Feldman, V. and Zhang, C. What neural networks mem- orize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 13:2881-2899 2020.
Cao, Y. and Yang, J. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Goel, S., Prabhu, A., Sanyal, A., Lim, S.-N., Torr, P., and Ku- maraguru, P. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint arXiv:2201.066 10064004 2022.
Carlini, N., Liu, C., Erlingsson, Ù. Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX se- curity symposium (USENIX security 19), pp- 267-284, 2019.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020a.
Carlini, N., Tramer, F., Wallace, E.,, Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-265302 2021.
Golatkar, A., Achille, A., and Soatto, S. Forgetting out- side the box: Scrubbing deep networks of information accessible from input-ouiput observations. In Computer Vision-ECCC 2020: 16th European Conference, Glas- gow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 383-398. Springer, 2020b.
Carlini, N., Chien, S., Nasr, M., Song. S., Terzis, A., and Tramer, F. Membership inference attacks from first prin- ciples. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEFF, 2022.
Graves, L.,, Nagisetty, V., and Ganesh, Amnesiac ma- chine learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Chalkidis, I., Androutsopoulos, I., and Aletras, N. Neu- ral legal judgment prediction in english. arXiv preprint arXiv:1906.0200 0200990999............ 2019.
Guo, C., Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. In III, H. D. and Singh, A. (eds.), Pro- ceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3832 3832-3842. PMLR, 13-18 Jul 2020. URL htttt::::/ppooceeeee ngs ml mall.prre r.rrsss// v119 gwwp/c...c..../ooo/ooo//oo.o... html.
Chen, J. and Yang, D. Unlearn what you want to forget: Efficient unlearning for LLMs. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https::tp/.. http:s:///ppereee wwtp://wwwweeeeeeeeeeeeeeeee net/ forum? 1d id=GprvtTwOzy
Chien, E., Pan, C., and Milenkovic, O. Efficient model up- dates for approximate unlearning of graph-structured data.
In The Eleventh International Conference on Learning Representations, 2023. htLhttp::/ http///ppperreeeeeee//eee/eeeeee net fiet/fowe htt//oormm?ii d- fheu4PBLciL.
Hayes, J., Shumailov, I., Triantafillou, E., Khalifa, A., and Papernot, N. Inexact unlearning needs more careful eval- uations to avoid false sense of privacy. arXiv preprint arXiv:2403.012.1 2024.
Chien, E., Wang, H., Chen, Z., and Li, Langevin un- learning: A new perspective of noisy gradient descent for machine unlearning. arXiv preprini arXiv:2401. 10371, 2024a.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 100.09685 2021.
Chien, E., Wang, P., Chen, Z., and Li, Certified ma- chine unlearning via noisy stochastic gradient descent. In The eiihtt- Annual Conference on Neural In- formation Processing Systems, 2024b. URL https: // //ppenreeiiw.oo net/ forst/fooumm httprrmmmiiihhh2X5555553 wwwpeeewwweeeeeeeeeeeeeeeeeeeee
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference Learning Representutions, 2023. URL https:// openreview net 2ttrummiitttttt---------
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024.
Jagielski, M., Ullman, J., and Oprea, A. Auditing differ- entially private machine learning: How private is private sgd? Advances in Neural Information Processing Sys- tems, 33:22205-2222 222166 2020.
10

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigat- ing privacy risks in language models. arXiv preprint arXiv:2210.01500 2022.
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L.,, and Tanella-Béguell S. Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 346- 363. IEEE, 2023.
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigating privacy risks in language models. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Compula- tional Linguistics (Volume I: Long Papers), pp. 14389- 14408, Toronto, Canada, July 2023, Association for Com- putational Linguistics. doi: 10. 100116653/1122233aclloo 805. URL ht https: acl-long: 805.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. Tofu: A task of fictitious unlearning for Ilms. arXiv preprint arXiv:2401.0612) 2024.
Nasr, M., Songi, S., Thakurta, A., Papernot, N., and Carlin, N. Adversary instantiation: Lower bounds for differen- tially private machine learning. 2021 IEEE Symposium on security and privacy (SP). pp. 866-882. IEFE, 2021.
Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramèr, F., Jagiel- ski, M., Carlini, N., and Terzis, A. Tight auditing of differentially private machine learning. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 1631- 1648, 2023.
Klimt, B. and Yang, Y. Introducing the enron corpus. In CEAS, volume pp. 2004.
Krzysztofek, M. GDPR: General Data Protection Regula- tion (EU) 2016/679: Post-reform Personal Data Protec- tion in the European Union. Kluwer Law International BV, 2018.
Neel, S., Roth, A., and
S. Descent-to- delete: Gradient-based methods for machine unlearning. Algorithmic Learning Theory, pp. 931-962. PMLR, 2021.
Kurmanji, M., Triantafillou, P., Hayes, J., and Triantafillou, E. Towards unbounded machine unlearning. Advances in neural information processing systems, 36, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Li, Q. Hong. J., Xie, C., Tan, J., Xin, R., Hou, J., Yin, X., Wang. Z., Hendrycks, D., Wang. Z., Li, B., He, B., and Song, D. LIm-pbe: Assessing data pri- vacy in large language models. Proc. VLDB Endow., (111):200-321444 August 2024a. ISSN 2150-8097. doi: 10.14778/368 355133366615544489944 URL https: //dii. org/10. 0...0.1777/3300
Pawelezyk, M., Di, J. Z., Lu, Y., Kamath, G., Sekhari, A., and Neel, S. Machine unlearning fails to remove data poisoning attacks. arXiv preprint arXiv:2406.17216. 2024a.
36814.6699999 994.
Li, W., Li, de Witt, C. S., Prabhu, A., and Sanyal, A. Delta-influence: Unlearning poisons via influence func- tions. arXiv preprint arXiv:2411.13311, 2024b.
Pawelczyk, M., Neel, S., and Lakkaraju. H. In-context un- learning: Language models as few-shot unlearners. In Forty-first International Conference on Machine Learn- ing, 2024b. URL https: opwpreviereeeeeeeeeeeeeeeeeeeeee net/ forum? i i==ikdleexxxxx
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., SHARMA, P., Liu, S., et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, 2024a.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Liu, S., Yao, Y., JJa,, J., Casper, S., Baracaldo, N., Hase, P., Xu, X., Yao, Y., Li, H., Varshney, K. R., et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787 2024b.
Sekhari, A., Acharya, J., Kamath, G., and Suresh, A. T. Re- member what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing 2021.
Systems, 34:
L Liu, Z., Dou, G., Chien, F., Zhang, C., Tian, Y., and Zhu, Z. Breaking the trilemma of privacy, utility, and efficiency via controllable machine unlearning. In Proceedings of the ACM on Web Conference 2024, pp. 1260-1271, 2024c.
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretrain- ing data from large language models. arXiv preprint arXiv:2310. 16789, 2023.

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, I.. Detecting pretrain- ing data from large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https:t::/ http:///ppeeeeeeeeeeeeeeee forum? id IN=
Yao, J., Chien, E., Du, M., Niu, X., Wang. T., Cheng, Z., and Yue, X. Machine unlearning of pre-trained large lan- guage models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8403-8419, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. URL https: /ttps///aalaann /at/aaaaaaaatttotoaaaaeaa org/2024 acl acl-long. .47..
Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtz- man, A., Liu, D., Zettlemoyer, L.,, Smith, N. A., and Zhang, C. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint (tl::2407.06464 2024b.
Yao, Y., Xu, X., and Liu, Y. Large language model un- learning. Socially Responsible Language Modelling Research, 2023. URL https: httttts:://opeeeeeeeeeeeeee /pwpreeieeeeeeeeeeeeeeeeeeeeeeee net/ forum? did= 6F0065x.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. In 20/7 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Ye, J., Maddi, A., Murakonda, S. K., Bindschacdler, V., and Shokri, R. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Com- munications Security, pp. 3093-3106, 2022.
Steinke, T. and Ullman, J. The pitfalls of average- case differential privacy. DifferenialPrivacy.orr 07 2020. https: /stps://difiii www.ffffeeetteaaeaeaeeeeeeeeeeee average-casee peaasedpp..
Yeom, S., Giacomelli, L,, Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31 computer security founda- tions symposium (CSF), pp. 268-282. IEEE, 2018.
Steinke, T., Nasr, M., and Jagielski, M. Privacy auditing with one (I) training run. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective un- learning, arXiv preprint arXiv:2404. 240.00240., 2024.
Thudi, A., Jia, H., Shumailov, I., and Papernot, N. On the necessity of auditable algorithmic definitions for ma- chine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pp. 4007 4022-4022, 2022.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01065 2022.
Thudi, A., Jia, H., Meehan, C.,, Shumailov, I,, and Papernot, N. Gradients look alike: Sensitivity often overestimated in (DP-SGD). In 33rd USENIX Security Symposiun (USENIX Security 24), pp. 973-990, 2024.
Zhang, T., Kishore, V., Wu, F,, Weinberger, K. Q. and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint 1eXiv:1904.00 0904409675, 2019.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al. Ilama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Zhao, K., Kurmanji, M., Bärbulescu, G.-O,, Triantafillou, E., and Triantafillou, P. What makes unlearning hard and what to do about it. arXiv preprint 240.v2240..0/257, 2024.
Ullah, E. and Arora, From adaptive query release to machine unlearning. In International Conference on Ma- chine Learning, pp. 34642-34660 PMLR, 2023.
Ullah, E., Mai, T., Rao, A., Rossi, R. A., and Arora, R. Ma- chine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 4126 4142. PMLR. 2021.
Wu, X., Li, J., Xu, M., Dong, W., Wu, S., Bian, C., and Xiong, D. Depn: Detecting and editing privacy neu- rons in pretrained language models. arXiv preprint 10.2v:2310.201388 2023.
12

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
A. Dataset
A.1. Dataset Details
In this paper, we examine two representative PII datasets: Enron and ECHR, described as follows:
Enron (Klimt & Yang, 2004). The Enron dataset consists of 536,000 authentic emails from 158 employees of the Enron Corporation, made publicly available by the Federal Energy Regulatory Commission following an investigation. Each email typically includes the sending timestamp, sender and recipient information, greeting, the main content, and footer containing the sender's personal details.
ECHR (Chalkidis et al., 2019). The ECHR dataset comprises case records from the European Court of Human Rights. Each record contains series of factual lists that detail the specifics of case. In experiments, further decompose these cases into individual facts, with each fact forming distinct sample, averaging around 80 tokens in length. In total, the dataset includes around 1 118,000 samples.
A.2. PII Selection
As outlined in Sec. 4, we selected U.S. phone numbers from the Enron dataset based criterion aimed at analyzing privacy risks in minority groups. To ensure the PII distribution was imbalanced, reflecting both minority and majority groups, we additionally selected two standard PII types (Lukas et al., 2023): email addresses (Enron) and years (ECHR). The distributions these PII counts are depicted Fig. 7.
Histogram of email domains Enron
Histogram of year ECHR
104
10
10a
10a
102
10
10
10
5202
10:00
15030
20020
25000
25
75
100
Figure 7. Histogram of email addresses (Enron-Eman (omanl) and years (ECHR- (eaHR-Year).
A.3. Preprocessing and Dataset Split
Dataset Preprocess. In our experiments, since the average token length of Enron samples approximately 770, we controlled the token length of each fact ensure the model could effectively memorize the samples. We randomly selected three coherent sentences from each sample, and if the sample contained specifie PIIs of interest, prioritized selecting sentences around them. We will keep the original samples for the ECHR dataset.
Dataset Construction. We begin by searching the dataset for occurrences of specific PIIs and analyzing their distribution. To form the minority set used in our Minority setting, we select 100 samples containing the least frequent Plls; this set serves as our forget set. In the Random setting, we construct the forget set by randomly selecting 100 samples containing PIIs. To create the canary set, we replace the PIIs in the forget set (Random setting) with the least frequent PII found the dataset. From the remaining data, randomly select samples to create the training and test sets. For experiments with GPT-2 (117M), we uniformly at random selected 10,000 samples each for both the training and test sets. For Llama-2 7B. we uniformly at random selected 50,000 samples for both the training and test sets.
B. Experimental Details
B.1. Compute Configurations
All experiments were conducted using NVIDIA A100 GPUs (80GB) and NVIDIA RTX 6000 Ada GPUs (48GB).
13

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
B.2. Unlearning Experiment Setup
Unlearning Algorithms. For all unlearning methods, we use constant learning rate of 10-5 and a batch size of 32, consistent with the fine-tuning stage. Note that some unlearning algorithms require additional hyperparameters. We follow the common designs from previous literature (Pawelezyk et al., 2024a) and detail the hyperparameter selection as follows:
EUK and CFK. In our experiments, we set the number of retrained layers to = for both GPT-2 and 2lama-2 (LoRA) models. For GPT-2, the unfrozen trainable parameters account for approximately 16% of the total parameters, while for Llama-2 7B, the unfrozen parameters account for around 10%.
NegGrad+. As noted in the main text, the hyperparameter / balances samples between Diorger and Dreep. In these experiments, we set = 0.999.
SCRUB. In the SCRUB method, three hyperparameters are used to balance the loss function on the keep set and the KL regularizers on both the keep and forget sets. According to the definition of the objective function in Section 5, three terms are weighted sequentially by setting: a = 0.999, 3 = 1, and y = 0.01.
Langevin Unlearning. The Langevin Unlearning method leverages noisy gradient descent unlearn samples from the forget set. In our experiments, we the Gaussian noise scale 5e 4 (for GPT-2) and 5e 3 (for Llama-2), and the clipping norm to 1.
Unlearning Epoch Selection. As outlined in Section 5.1, all unlearning methods are constrained to a maximum of 10 complexity units, and the optimal epoch for each method is selected based on whether the perplexity of the unlearned model on Prain increases by more than point. Under our computational budget, methods that only require the forget set (RI., GA) for 10 epochs, while methods requiring both the keep and forget sets (NegGrad+, SCRUB) are limited to epochs, due to the equal-sized cycling between the two sets. For methods that only require the keep set (EUK, CFK, Langevin), we use 10 epochs, with varying sample sizes for EUK and CFK. as some model parameters remain frozen. The selected epoch for each method in each experiment detailed Table 3.
Table 3. Epochs comparison between unlearning methods on GPT2 and LLaMA2 models.
GPT-2
Llama-2 7B
Unlearning Methods
Enron
ECHR
Enron
ECHR
RL
Epoch 1
Epoch 1
Epoch I
Epoch
EUK
Epoch 10
Epoch 10
Epoch 10
Epoch 10
CFk
Epoch 10
Epoch 10
Epoch 10
Epoch 10
GA
Epoch 1
Epoch 1
Epoch 1
Epoch 1
NegGrad+
Epoch 5
Epoch 5
Epoch I
Epoch 5
SCRUB
Epoch 5
Epoch 5
Epoch 5
Epoch 5
Langevin
Epoch 10
Epoch 10
Epoch
Epoch
Attack Method Hyperparameters. We employed three attack methods in our evaluation pipeline. For lossMIA and zlibMIA, there are no hyperparameters to tune. The Min- K method is based on the observation that non-member examples tend have more tokens with lower likelihoods compared member examples. In this method, the hyperparameter К controls the selection of the bottom K% of tokens in each sample based on their likelihoods. Following previous recommendations in (Duan al., 2024; Shi al., 2024a), we set K 20 in experiments.
Random Seed Selection. In all our experiments, we followed the common practice and fixed our random seed to be 42.
C. Additional Experimental Results
In this section, we present supplementary experimental results further substantiate our claims in the main text.
14

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 4. The privacy leakage (PI.) for each unlcarning method against different attackers for GPT-2 the Enron-Emrn dataset.
Method
(losMIA)
(ZibbMAA)
PL KS)
Random 0,303
Canary
Minority (77% 1.145 (278% 1)
Random
Canary
Minority
Random 0.529
Canary
Minority
No Unleam
0.535
0.200
0.309 (54% 1) 0.262 %
0.934 (76%1) 1,68 (178% %
0.033
(366%)3
0.448
(1265% f)
0.062
0.142 (127% () 0.121 (94%
0.772 (79% () 1,200 (179%
EUK
0.232
0.440 (89 % 1)
0.582 (150% 1) 0.135
0.229
(70% 1) 0.152 (13% 1) 0.501
0.886 (77%-1) 1.034 (106 % 1)
CFR
0.296
0,515
(74% 1)
1.139 (285% f)
0.197
0.295 (49% f) 0.260 (32% () 0.526
0.905 (72% f) 1.478 (181% % f)
GA
-0.279
-0.173
(38%.L
0,739 (165% 1)
-0.119
-0.03 (69 0.168 (41%
-0.390
-0.304 (22% 1,034 (165% 1)
NegGrad+
0,265
0.471
(77 % 1)
1.103 (316% f)
0.179
0.269
(50% () 0.251 (40% 1) 0.496
0.864 (74%-1) 1.434 (189% 1)
SCRUB
0.286
0,499
(74%
1.097 (283%)1) f)
0.190
0.289
(52%1) 0.253 (34%
0.519
0.902 (74%1) 1.473 (184%
Langevin
0.154
0.319 (107 % 0.606 (293% f>
0.086
0.178
(107%
() 0.124 (44%
1) 0.336
0,645 (92% 1) 0,940 (180% 1)
Table 5. The privacy leakage (PL) for each unlearning method against different attackers for GPT-2 ECHR-year datasets.
Method
PL (lossMIA)
(zlbbMAA)
PL (Min-KS)
Random
Canary
Minority
Random
Canary
Minority
Random 0.213
Canary
Minority
No Unleam
0.198
0,247 (25%) to 0.263 (33% f)
0.086
0.103 (20% 0.122 (42%
0.276 (30% to 0.299 (40% t)
RL
0.161
0.213 (32% 1) 0.234 (45%t)
0.067
0.088
(31% 0.086 (28% f)
0.190
0.259 (36%1) 0.257 (35% f)
BUK
0.125
0,176 (41%%) 0.138 (10%T)
0.067
0.088
(31 % () 0,070 t)
0.114
0.187 (64% 1) 0.135 (18%1)
CFR
0.188
(24% 1)
(38% f) 0.084
0.095 (13351) 1) 0.120 (43% t)
0.209
0.264
(26 % 1) 0.295 (41% f)
GA
0.067
0.027 (60%-] 0.105 (57%.1
0.024
0.019 (21 0.038 (58%-1)
0.090
-0.019 (79%.. 0,143 (59% f)
NegGrad+
0,183
0.221 (21%1 () 0.247 (35%) f)
0.071
0.088 (24% f) 0.112 (58%) f)
0.191
0.237 (24% f) 0.274 (43% f)
SCRUB
0.179
0.223 (25% () 0.253 (41%
0.080
0.099 (24% 0.116 (45%-1)
0.197
0,253
(38 %
f) 0.289 (47%) f)
C.1. Experiments on Enron-Email and ECHR-year datasets
In Tables 4, 7, and 7, report the PL scores for all three attackers across the three scenarios on the Enron-email, ECHR-year datasets for GPT-2 and Llama-2, respectively. The results support our claim that the current LLM unlearning evaluation (Random setting) significantly underestimates privacy risk.
Table The privacy leakage (PL) for each unlcarning method against different attackers for Llama-2 7B on the Enron-Email dataset.
Pl. (losMMA)
Pl. (cliMMAA)
Pl. (Min-KK))
Method
Random
Canary
Minority
Random
Canury
Minurity
Random
Canary
Minority
No Unleam
0,050
0,282 (464% 0.174 (248%
0,046
0,236 (413%1) 0,095
(106%)
0.064
0,474 (640% 0,224 (250% ")
RL
-0.609
-0.567 (7%+L) -0.821 (12%__
-0.832
-0.874 (5%rt -0,478 (43% L)
-0.931
-0.931 (0%) -0.849 (9%.1)
EUK
0.037
0.241
() 0.169 (356% f) 0,015
0.186
(1140% 1)
0.102 (580% 1) 0.040
(810% ") 0.364
0.206 (415% ()
0.049
0.264
(438% 1) 0.169 (245% t) 0,046
0.220 (378%) () 0.090 (96%))
0.062
0.436 (603% 0,218 (251% ()
GA
.0.512
-0.692 (35% 1)) 0.059 (89%.L
-0.455
.0.232 (47%%L)
0.184 (58%-L)
.0.569
-0.479 (16%:1 0.294 (48%-)
NegGrad+
-0.931
-0.929 (2%kL -0.821 (125L_
-0.832
-0.874 (55k1)
-0.478 (43%) ()
-0.931
-0.931 (066) -0.849 (9%..)
SCRUB
0,040
0.257 (543% 0.174 (335%+)
0.034
0,209 (515%1)
0,095 (179% ())
0.056
0.426
(661%7) 0.224 (300% ")
Langevin
0.022
0.191 (768%
1) 0.048 (118%1)
f) 0.020
0.141 (605% f)
0.021 (5%)))
0.035
0.339 (868%
1 () 0.079 (126% ")
Table 7. The privacy leakage (PL) for each unlearning method against different attackers for Llama-2 7B on the ECHR-year dataset.
PL (lossMIA)
PL (zlibMIA)
PL (Min-K%)
Method
Random
Canary
Minority
Random
Canary
Minority
Random
Canary
Minority
Unleurn
0.056
0.094 (68%)) 1)
0.076 (35% t)
0.048 (60% 1)
0.096 (220% t>
0.067
0.114 (70% t)
(106% f>
RL
-0.069
0.044 (36%L) ()
-0.532 (671 f)
-0.024
0.029 (21% f
-0.192
(700% f)
-0.034
0,070
(106% ?))
-0.458
(1247% (
EUK
0.059
0.084 (42%) 1)
0.079 (34%
0.030
0.044 (47% 1)
0,079 (163%
0.065
0,110 (69% f)
0,153 (135% 1)
CFk
0.056
0.088 (57 % 1)
0.073 (30% f)
0.028
0.044 (57% 1)
0 (214% 1)
0.063
0.106 (68% f)
0.131 (108% f)
GA
-0.046
-0.376
(717% 1)
-0.624 (125%%))
-0.016
-0.120
(650% f)
-0.267
(1569% ()
-0.063
-0.404
(541 %-1)
-0.574 (811% f)
NegGrad+
0.024
-0.272 (1033% 1)
40.624 (2500% 1) 0.012
-0,099 (725% f)
-0.235 (1858 ") 0.026
(1454% f) -0.404
-0.663 (2449% ")
SCRUB
0.056
(68% 1)
0.073 (30%))
0.030
0.048 (60% f) 0.090 (200%.1
0.067
0.112 (67%.)
0.139 (107%
Langevin
0.026
0.052 (100%1)
0,041 (58%+)
0.010
0.025 (150%))
0.046 (360%11
0.028
0.062 (121%1)
0.078 (179%)1
C.2. More Results on Minority-Aware Evaluation
In this section, we present further benchmarking results for unlearning approaches under minority-aware LLM evaluation. Following the same setup as Section 6.2, Fig. 8 reports the maximum PL score under lossMIA and zlibMIA attackers on Enron-Phone and Enron-Email and Fig. 9 reports the maximum PL. score and worst-case perplexity for various unlearning
15

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
methods ECHR-Year (GPT-2) dataset.
Phone
Email
GTT-2
Llam-22
GPT-2
Lama-2
ZbMA
JBMA
zIeMIA
0.01
0
437
M6
(d)
Figure 8. Benchmarking unlearning approaches via our minority-aware evaluation for GPT-2 and Llama-2 Enron-Phone and Enron- Email dataset. ()). Maximum privacy leakage (PL) over three cases (Random, Canary, and Minority) for lossMIA attack. (f)) Maximum privacy leakage (PL) over three cases (Random, Canary, and Minority) for zlibMIA attack.
lossMIA
zlibMIA
Min-K%
Utility
Max
0.20
0.25
0.35
0,30
0.176
0.28
Unlearn
0247
4.353
Max
0.10
0.08
0.12
0.16
0.14
0888
0.12
Unieam
Da12
216
0.066
Max PL
0.30
0.35
187
No Unlearn
0:205
0274
4
0.169
Perplexity
(- better results)
22.5
17.5
15.0
25.0
22.06
11.79
12.2
12.0
No Unlearn
12.07
12
0.15
0.105
0.06
0.036
0.143
12.5
10.0
0.10
0.04
7.5
0.02
RL
EUK
CPk
GA
Negorad+
Scrub
Langevin
AL
EUK
CPk
GA
NegGrad+
Scrub
Langevin
RL
EUK
CFk
G4
NegGrad+
Scrub
Langevin
EUK
CFK
GA
NegGradk
Scrub
Langevin
(a)
(b)
(c)
(d)
Figure 9. Benchmarking unlearning approaches via our minority-ane aware evaluation for GPT-2 on ECHR-year dataset. (a)-c)): Maximum privacy leakage (PL.) over three cases (Random, Canary, and Minority) for lossMIA, zlibIIA and Min-K's attacks respectively. (d): Worst perplexity over the three cases of each method.
loSsMIA
zlibMIA
Min-K%
Utility
0.8
0.7
5.332
624
0.824
0.30
0.267
0.235
0.7
0,8
No Unicam
0.574
6666
Perplexity (- better results)
30+
29
0.25
0.192
A58
No leam
0.4
0.2
0.3
0.084
No Unlearn
0.8
0044
092
Max
0.20
0.10
0.15
R079
No Unieam
0,089
0.09
0,00
Max
0.5
0.3
0.4
0.153
0.131
0.139
D478
5.32
1.89
5.03
0.1
0.0
0.00
0.0
72
EUK
Cfk
NegGrad+
Scrub
Längevin
RL
EUK
CRk
NegGrad+
Scrub
Langevin
RL
EUk
CFk
G4
MegGrad+
Scrub
Langevin
EUk
CFR
G4
NegGrad
Scrub
Langevin
(b)
(c)
(d)
Figure 10. Benchmarking unlearning approaches via minority-aware evaluation for Llama-2 ECHR Year dataset. (a)-(c): Maximum privacy leakage (PI.) over three cases (Random, Canary, and Minority) for lossMTA, zlibMIA, and Min-K%; % attacks respectively. (d): Worst perplexity over the three cases of each method.
We observe that both GA and Langevin Unlearning methods maintain a favorable balance between privacy and utility, However, GA can be sensitive to the forget set size and the number of unlearning iterations (Section 6.3). In practice, the GA method should applied with caution, whereas more stable approaches like Langevin Unlcarning offer a better trade-off in terms of privacy, utility, and stability.
16

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
C.3. Results Other Utility Metries
In this section, we report the worst-case utility performance across three settings (Random, Canary, and Minority) using utility metrics BERTScoree (Zhang et al., 2019) and ROUGE score (Lin, 2004), which capture semantic meaning, on the Enron-Email dataset. The results shown in Fig. (GPT-2) and Fig. 12 (Llama-2). As illustrated in the figures, the performance of Random Label and mrudient-ascent based methods (Gradient Ascent and NegGrad+) is unstable under all utility metrics (Perplexity, BERTScore, and ROUGE). In contrast, Langevin Unlearning demonstrates relatively stable performance and achieves favorable privacy-utility trade-off.
loSSMIA
zlibMIA
Min-K%
1,139
103
1087
1.8
Unlearn
No Unleam
0.35
0.295
0.289
D299
1.6
1.470
1344
1.473
Max PL
1.0
0.739
Max PL
0.30
0.229
Max PL
1.4
0.8
0.6-
0,448 0.582
8966
0.20
0.25
0.142
0,168
0.178
1.2
1.0
1.34
1.034
0.94
0.4
0.15
0.8
0.2
0.10
RL
EUK
CRA
G4
NegGrad
Scrub
Langevin
RL
EUk
CFk
G4
NegGrad+
Scrub
Langevin
RL
EUk
CFk
GA
NegGrad+
Scrub
Langevin
(a)
(b)
(c)
Utility
Utility
Utility
Perplexity (- better results)
30-
25
231
ROUGE Score (- better results)
0.25
1.10.99 19
Unleam
h18
0,264
BERTScore
(+ better results)
0.80
0.781
0.764
Unleam
0.763
0.14 0,84
20
16.33
15
12.15
12.37
12.32
12.0
142
0.70
10
0.10
RL
EUR
CFk
GA
NegGrad's
Scrub
Langevin
RL
EUk
CFk
G4
NegGrad+ Scrub
Langevin
RL
EUk
CFk
G4
NegGrad+ Scrub
Langevin
(d)
(e)
Figure Benchmarking unlearning approaches via our minority- aware evaluation for GPT-2 on Enron-email dataset. (a)-(c): Maximum privacy Icakage (PL) over three cases (Random, Canary, and Minorityy (y) for lossMIA, zlibMIA, and Min-K% attacks respectively. (d-f): Worst utility performance over three cases of each method.
C.4. Further Details and Results on Langevin Unlearning
In this section, we provide additional details and results the Langevin Unlearning methods. As mentioned in Section 5, Langevin leverages noisy gradient descent and involves training the model the dataset D using DP-SGD. Furthermore, Langevin conducts machine unlearning by fine-tuning the model on the dataset Dxeep with DP-SGD as well.
It is important to note that for the Langevin Unlearning method, the training process incorporates noise. Consequently, our retrain baseline is adjusted to train the initial model on Deeep using DP-SGD for 5 epochs. Furthermore, in Table and 9, we report the effectiveness Langevin Unlearning by evaluating against three MIA methods (IossMIA, zlibMIA, and Min-K%) across different datasets on GPT-2. These evaluations are conducted across three scenarios (Random, Canary, Minority), assessing the PL scores, the maximum PL scores and the worst-case perplexity. By comparing the results of the Noisy No Unlearn baseline (which fine-tunes the initial model with DP-SGD for epochs) with those of the Langevin Unlearning method, we observe that minority scenarios (Canary, Mi nority) lead to significantly higher privacy leakage, and Langevin Unlearning achieves superior privacy-utility trade-offs. Additionally, in practical applications, the number of steps employing noisy gradient descent can be tailored based on the acceptable computational costs, thereby enabling potentially better privacy-utility trade-offs. This flexibility allows practitioners to balance the trade-off between enhanced privacy and computational efficiency according specific application requirements.
17

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
lossMIA
ZlibMIA
Min-K%
1.2
0.933
1.0
0.874
0.874
1.2
0,903
1.0-
0.523
0.8-
0.8
Linteamn
PL
434 6,56
Untleam
Max
0.6-
Unleam
Max
0.6
0.455
0.6
4448
0243
0:257
0.197
1.66
2009
a341
0.2
0.2
RL
EUK
CPk
GA
Neg
Scrub
Langevin
RL
EUK
CR
G4
NegGrad+
Scrub
Langevie
RL
EUK
CAR
G4
NegGrade
Scrub
Langevin
(a)
(b)
(c)
Utility
Utility
Utility
(- better results)
30+
30+
ROUGE Score ( better results)
No unleam
(~> better results)
0.85
Unteam
Perplexity
20-
14.7
Unleam
0.15
0.20
BERTScore
0.80
288
0.7g0
5.50
4.83
4.36
0,779
0.75
0.33
0.721
0,05
0.70
RL
EUR
CFk
G4
MegGrad
Scrub
Langevin
RL
EUK
CFk
NegGrad Scrub
Langevin
EUK
CFk
G4
NegGrad+ Scrub
Langevin
(d)
(e)
Figure 12. Benchmarking unlearning approaches via minority-aware evaluation for Llama-2 Enron-email dataset. (a)-(c): Maximum privacy leakage (PL) over three cases (Random, Canary, and Minority" for lossMIA, zlibMIA, and Min-K% attacks respectively. (d-f): Worst utility performance over the three cases of each method.
Table 8. The privacy leakage (PL) for Langevin Unlearning against different attackers for GPT-2 on All datasets.
Method
Pt. (lossMAA)
Pl. (zlbMMAA)
Pl. (Min K%)
Random
Canary
Minority
Random
Canary
Minority
Random
Canary
Minority
Earon-phone
Noisy No Unleam
0.097
0.152 (57% t)
0.170 (75% 1)
0.024 0.039 (63%-t) 0.033 (38% 1)
0,164
0.259 (88%.1)
0.268 (63% f)
Langovin
0.092
0.144 (57%))
0.157 (71% t)
0.024
0,037 (54%
0.027 (13%51
0.159
0.259 (63%
0.264 (66%
Earon-email
Noisy No Unleam
o. 0.156
0.342
(119%1)
0.642 (312% t)
0.102 0.193 (89%-1) 0.130 (27%) ()
0.344
0.691 (101 % f) 0.945 (175% t)
Langevin 0.154
(294%)
0.178 0.124 (28%
0.645 (92% (179%)1
ECHR-year
Noisy No Unleam
101
0.152 (51% t
0.122 (21 f)
0.049
0.067 (37%-) 0.064 (31 ()
0.122
0.180 (48%-1)
0.145 (199k))
Langovin 0.103
0.140 (36%
0.125 (21%
0.049
0,061 (24%1) 0.065 (33%7)
0.117
0.168 (44%))
0.146 (25%))
Table 9. Maximum PL Scores and Worst-case Perplexity for Noisy No Unlearn and Langevin across Datasets on GPT-2
Dataset
Methods
lossMIA
zlibMIA
Min-K%
Perplexity
Enron phone
Noisy No Unlearn
0.170
0.039
0.268
13.87
Langevin
0.157 (7.65%]
0.037 (5.13%]
0.264 (1.49%])
13.88
Enron
Noisy No Unlearn Langevin
0.642
0.193
0.945
12.52
email
0.606 (5.61%.)
0.178 (7.77%.)
0.939 (0.63%.
12.61
ECHR year
Noisy No Unlcam
0.152
0.067
0.180
12.75
Langevin
0.140 (7.89%.)
0.061 (8.96%.)
0.168 (6.67%.
12.78
C.5. More Results on Forget Set Size
In this section, we report additional results on the impact of forget set size for each unlearning method, using LossMIA and ZlibMIA attackers. As shown in Fig. 13, similar to the results the main text, both RL and GA methods are sensitive to the forget set size, whereas methods like SCRUB and Langevin Unlearning demonstrate greater stability.
18

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
IossMIA
ZIibMIA
0.40
Methods
0.30
RL
NegGrad+
0.15
EUK
Scrub
Max
0.20
Max
0.10
CF
Langevin
0.10
0.05
50
100
200
50
100
200
Forget Set Size (a)
Forget Set Size
(b)
Figure 13. The effect of lorget set size for each unlearning approach. (a)(): Maximum PL over three cases (Random, Canary, Minority) with the attacker being lossMIA and zlibMIA respectively.
C.6. More Details on the Privacy-Utiliit Trade-off Curves for Langevin Unlearning and SCRUB Methods
This section provides an comprehensive experiments of the privacy-utility trade-off curves for the Langevin Unlearning and SCRUB methods, as introduced in Sec. 6.3. We detailed the hyperparameters used in the ablation study as follows:
lossMIA
ZibMIA
Min-k%
Perplexity
loSsMIA
ZibMIA
Min-k38 k%
Perplexity
can
1M
cm
MOX
Figure 14. Privacy-utility transition curves for Enron-Phone dataset with hyperparameter 1 (Top) and hyperparameter 3 3= 1e (Bottom).
Langevin Unlearning. For the Langevin Unlearning method, fix the clipping norm to 1 and vary the noise scale added during training to control the privacy-utility trade-off. In experiments with the GPT-2 model, the noise scale a is adjusted across the values {1e - 4, 3e 4, 5e 4, 8e - 4, le 3}. Table 19 and 20 present the AUC scores under various attackers (lossMIA, ZlibMIA, Min-k% and utility (perplexity) on the Enron-Phone and Enron-Email datasets, respectively.
SCRUB, The SCRUB training objective comprises the original loss l on the keep set, along with two KL divergence regularizers on the keep and forget sets. These terms are balanced by three hyperparameters:
Ex Dessp (KLL (MM))(())))))) + 3E(M;
En Dange KKL (=)]
(2)
We conducted an extensive hyperparameter search, setting 33 to 1 and le 3 in separate configurations. For each fixed B, a and y are independently varied from {1e - 4, 5e - 4, le - 3, 5e - 3, le - 2, 5e - 2, le - 1,5c 1, 13. Fig. 14 and 15 illustrates the resulting transition curves. showing the maximum privacy leakage (PL) for three scenarios (Random, Canary, Minority) across different attackers (lossMIA, zlibMIA, Min-k% and utility (perplexity) metrics both Enron-Phone and Enron-Email datasets. The transition curves highlight how SCRUB's performance depends on balancing the three objective terms. Notably, when the KI. regularizer weight on the keep set is greater than or equal to that on the forget set, SCRUB achieves relatively high utility, albeit with increased privacy leakage. Besides, we observe in our experiments that the zlibMIA attacker fails to capture the inherent privacy-utility trade-off for SCRUB as demonstrated in the transition curves.
We further report the privacy-utility trade-off curves for both methods under attacker being lossMIA and zlibMIA. Similar
19

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
lossMIA
ZIIDMIA
Min-k%
Perplexity
Max
1-
IoSsMIA
ZlibMIA
Min Min-k%
Perplexity
1::0:
Max PL
2
LMP
LAM
140
2-01h
2-
1:00
144
Figure 15. Privacy-utiin utility transition curves for Enron-Email dataset with hyperparameter 3 = 1 (Top) and hyperparameter 3 == -ee- (Bottom).
Enron-Phone ossMIA
Enron-Phone zlibMIA
Enron-Email lossMIA
Enron-Email zlibMIA
12
0:10
OK
0.27
1.0-
0.30
2 8.26
Max
Seghatt
8.35
0.0
0.20
0.20
6:34
0:25
12
24
35 Perplexity
20
20
12
14
18
20
26
20
32
14
36
36
20
22
24
25
Perpiexity
Perplexity
Perplexity
Figure 16. Privacy-utility trade-off curves under lossMIA and zlibMIA.
to the results demonstrated in Sec. 6.3, Langevin Unlearning method achieves the best trade-off performance over SCRUB method.
C.7. Privacy Underestimation Across Complexity Units
In Fig. 17, we report the degree of largest underestimation (Canary & Mimoi hoirttyyo. settings) in privacy leakage compared to Random settings across different complexity units under Min-k% attacker. As demonstrated in the figure, under different complexity units, the privacy leakage for each unlearning methods are severely underestimated. Detailed results under each
Min- M%n-K% Enron- -GTT22
Max PL Underestimation (%)
10a
Methods
Random Label
FUk
Gradient Ascent
NeySrad-
settings are reported in Table 11 and 12.
Soruh
Langevin
102
C.8. Discussion
on Retraining LLMS with Equivalent Computational Budget
10
Complexity Unit
In this subsection, we examine the scenario where the LLM is retrained from scratch using a computational budget equivalent to that of unlearning methods. Intuitively, if retraining with the same computational resources could already achieve satisfactory privacy-utility trade-off, employing unlearning methods would be less important. To empirically illustrate this point, we consider the Enron (Phone) dataset with the GPT-2 model as a representative case study. Specifically, we report the privacy leakage metrics (PrivLeak) across different attack scenarios, as well as the corresponding perplexity scores in Tab. 10. Additionally, the performance of the retrained model with an equivalent computational budget is highlighted in Fig. 18 (Black). Our findings reveal that while retraining alone indeed achieves relatively low privacy leakage, it drastically compromises model utility. These results underscore the importance and practical necessity of developing and evaluating
Figure 17. Degree of the largest underestimation in privacy leakage (Canary, Mincrity com- pared to Random settings across varying Com- plexity Units.
20

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
unlearning methods.
Enron-Phone Min-k%
0.55
Langevin
SCRUB
Table 10, Privacy Leakage (PL) for Retrain with Complexity Units.
EUK
RL
GA
SCRUB Fareto Frnttr
CFK
Poreto
0.40
NegGrad
(Same Budgeti
Random
Canary
Minority
0.35
0.30
Loss PL
0.004
0.043
0.083
0.25
Zlib PL
0.058
0,075
0.077
Min-k% PL.
0.083
0.174
0.188
12
14
16
18
20
22
24
26
Perplexity
Perplexity
23.44
Figure 18, Privacy-Utility Trade-off Enron-Phone
Min-k% with Retrain (Same Computation Budget).
C.9. Results on AUC Scores, Perplexity across Different Models and Datasets
We further report the AUC scores under different attackers (lossMIA, zlibMIA, Min-K%) and utility (perplexity) over holdout test set Dee for GPT-2 and Llama-2 in Table. 13-18.
C.10. Discussion on TPR @low FPR Metric
Note that aside from the AUC score, a commonly reported metric for privacy evaluation is TPR@low FPR (Carlini et al., 2022), where the low FPR is often set to 0.01. However, in our scenario, the canary size set 100 (1% of the total training set size). At FPR 0.01, the TPR would be caleulated based only few canary samples, making the overall score very experimental results, primarily focus on the AUC score.
coarse. To avoid the impact of this coarse granularity
Table 11. PL Scores for MIA across Three Settings for GPT-2 on Enron (Phone) under Different Complexity Units.
Enron- Enron-Phone Phone GPT-2 Min-k%
Methods
Complexity Units 2
Complexity Units 4
Complexity Units 10
Random
Canary
Minority
Random
Canary
Minority
Random
Canary
Minority
Random Label
0.166
0.284
0.136
-0.059
0.037
0,290
0.168
0.320
0.355
EUK
0.068
0.169
0.234
0,068
0.174
0.202
0.092
0.216
0.223
CFk
0.302
0.445
0.518
0.300
0.438
0.519
0.298
0.435
0.514
Gradient Ascent
-0.175
-0.245
-0.203
-0.245
-0.469
-0.352
-0.031
-0.012
-0.427
NegGrad+
0.309
0.452
0.543
0.309
0,447
0,529
0.298
0.452
0,510
Scrub
0.311
0,443
0.525
0.313
0,447
0.525
0.306
0.445
0.532
Langevin
0.163
0.259
0.265
0.161
0.259
0.265
0.159
0.259
0.265
Table 12. Perplexity for MIA across Three Settings for GPT-2 on Enron (Phone) under Different Complexity Units.
Enron-Phout GPT-2 Perplexity
Methods
Complexity Units 2
Complexity Units 4
Complexity Units 10
Random
Canary
Minority
Random
Canary
Minority
Random
Canary
Minority
Random Label
29.85
28.93
30+
30+
30+
30+
30+
30+
30+
EUK
30+
30+
30+
30+
30+
30+
23.64
23.65
26.60
CFk
12.72
12.71
12.72
12.72
12.72
12.73
12.67
12.67
12.67
Gradient Ascent
16.63
15.76
29.28
30+
30+
30+
30+
30+
30+
NegGrad+
12.86
12.87
12.83
12.84
12.86
12.88
12.83
12.80
12.88
Scrub
12.94
12.95
12.96
13.11
13.09
13.19
13.09
12.88
12,96
Langevin
13.84
13.85
13.88
13.85
13.86
13.88
13.84
13.88
13.88
21

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 13. AUC Scores for MIA and Perplexity across Three Settings for GPT-2 on Enron (Phone Numbers)
AUC LossMIA
AUC ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.533
0.531
0,422
0.694
0.693
0,619
Retrain
0.448
0.414
0.315
0.660
0.644
0.582
Noisy No Unlearn
0.474
0.464
0.365
0.678
0.674
0.606
Noisy Retrain
0.432
0.403
0.312
0.662
0.649
0.587
Unlearning Methods
Random Label
0.501
0.493
0.381
0.689
0.687
0.617
Langevin
0.472
0.461
0.361
0.678
0.673
0.603
EUK
0.460
0.447
0.354
0.683
0.677
0.612
CFk
0.533
0.529
0.421
0.695
0.692
0.619
Gradient Ascent
0.488
0.472
0.355
0.676
0.671
0.597
NegGrad+
0.530
0.526
0.418
0.694
0.691
0.616
SCRUB
0.523
0.518
0.416
0.692
0.689
0.618
AUC Min-K %
Perplexity
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.594
0.592
0.471
12.72
12.72
12.72
Retrain
0.457
0.409
0,309
12.74
12.74
12.74
Noisy No Unlearn
0.518
0.511
0.393
13.84
13.85
13.87
Noisy Retrain
0.445
0.406
0.310
13.84
13.84
13.83
Unlearning Methods
Random Label
0.575
0.573
0.447
14.49
14.41
14.86
Langevin
0.516
0.511
0.392
13.84
13.88
13.88
EUk
0.499
0.497
0.378
23.64
23.65
23.60
CFk
0.593
0.587
0.468
12.67
12.67
12.67
Gradient Ascent
0.526
0.508
0,362
13.22
13.20
14.10
NegGrad+
0.591
0.587
0.467
12.86
12.86
12.88
SCRUB
0.592
0.593
0.472
13.00
12.98
12.96
22

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 14. AUC Scores for MIA and Perplexity across Three Settings for GPT-2 on Enron (Email)
AUC LossMIA
AUC ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.555
0.551
0.354
0.462
0.462
0,563
Retrain
0.426
0.359
0.165
0.385
0.353
0,446
Noisy No Unlearn
0.488
0.483
0.271
0.422
0.421
0,512
Noisy Retrain
0.422
0.360
0.165
0.383
0.353
0.453
Unlearning Methods
Random Label
0.440
0.414
0.239
0.409
0.403
0.500
Langevin
0.487
0.475
0.265
0.420
0.416
0,509
EUK
0.525
0,517
0.261
0.437
0.434
0,514
CFk
0.552
0.544
0.353
0.461
0.457
0.562
Gradient Ascent
0.307
0.297
0.287
0.339
0.340
0.521
N
0.539
0.528
0.347
0.454
0.448
0.558
SCRUB
0.548
0.538
0.346
0.458
0.455
0.559
AUC Min-K %
Perplexity
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.607
0.611
0.506
12.11
12.11
12.12
Retrain
0.397
0.316
0,205
12.19
12.19
12.19
Noisy No Unlearn
0.516
0.519
0.387
12.51
12.52
12.50
Noisy Retrain
0.384
0.307
0.199
12.48
12.48
12.48
Unlearning Methods
Random Label
0.568
0.560
0.451
15.87
16.13
13.54
Langevin
0.513
0.505
0.386
12.58
12.58
12.61
EUk
0.596
0.596
0.417
23.58
23.70
23.58
CFk
0.606
0.602
0,508
12.14
12.15
12.15
Gradient Ascent
0.242
0.220
0.417
27.30
21.86
14.53
NegGrad+
0.594
0.589
0.499
12.34
12.34
12.37
SCRUB
0.603
0.601
0.507
12.15
12.15
12.32
23

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 15. AUC Scores for MIA and Perplexity across Three Settings for GPT-2 on ECHR (Year)
AUC LOssMIA
AUC ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.661
0.650
0.658
0.532
0.524
0,560
Retrain
0.552
0.521
0.521
0.490
0.475
0.499
Noisy No Unlearn
0.600
0,592
0.581
0.514
0.509
0,535
Noisy Retrain
0.545
0.514
0.518
0.490
0.477
0.503
Unlearning Methods
Random Label
0.641
0.632
0.643
0.523
0.517
0.542
Langevin
0.601
0.586
0.583
0.514
0.506
0,536
EUK
0.621
0.613
0.593
0.523
0.517
0,534
CFk
0.656
0.643
0.656
0.531
0.520
0.559
Gradient Ascent
0.589
0.535
0.576
0.502
0.484
0.518
N
0.653
0.636
0.650
0.525
0.517
0.555
SCRUB
0.651
0.637
0.653
0.529
0.522
0.557
AUC Min-K %
Perplexity
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.671
0.661
0.673
11.81
11.81
11.81
Retrain
0.553
0.518
0,518
11.82
11.82
11.82
Noisy No Unlearn
0.615
0.609
0.586
12.74
12.74
12.75
Noisy Retrain
0.548
0.516
0.512
12.73
12.73
12.73
Unlearning Methods
Random Label
0.658
0.652
0.651
12.70
12.70
12.67
Langevin
0.612
0.603
0.587
12.78
12.78
12.78
EUk
0.616
0.615
0.588
22.04
22.06
22.00
CFk
0.669
0.655
0.671
11.78
11.78
11.79
Gradient Ascent
0.603
0.508
0.592
12.11
12.20
12.11
NegGrad+
0.659
0.641
0.660
12.01
12.04
12.03
SCRUB
0.662
0.649
0.668
12.02
12.03
12.01
24

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 16. AUC Scores for MIA and Perplexity across Three Settings for Llama-2 on Enron (Phone Number)
AUC LossMIA
AUC ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.614
0.606
0,551
0.578
0.571
0.575
Retrain
0.579
0.488
0.470
0.559
0.520
0.539
Noisy No Unlearn
0.605
0.598
0.539
0.578
0.572
0,576
Noisy Retrain
0.584
0.500
0.482
0.568
0.533
0.554
Unlearning Methods
Random Label
0.439
0.447
0.444
0.556
0.554
0.594
Langevin
0.603
0.590
0.532
0.577
0.569
0,574
EUK
0.612
0.608
0.557
0.581
0.575
0,583
CFk
0.612
0.603
0.549
0.577
0.569
0.573
Gradient Ascent
0.253
0.278
0.252
0.551
0.540
0,584
NegGrad+
0.536
0.398
0.451
0.547
0.495
0.538
SCRUB
0.613
0.567
0.550
0.578
0.553
0.574
AUC Min-K %
Perplexity
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.611
0.610
0.579
9.45
9.47
9.48
Retrain
0.568
0.547
0.491
9.48
9,48
9.48
Noisy No Unlearn
0.600
0.602
0.570
10.21
10.21
10.21
Noisy Retrain
0.579
0.565
0.516
10.19
10.19
10.19
Unlearning Methods
Random Label
0.498
0.507
0.497
2124
2559
2445
Langevin
0.598
0.596
0.563
10.20
10.21
10.20
EUk
0.604
0.619
0.584
11.00
11.24
10.82
CFk
0.609
0.606
0.575
9.43
9,45
9.46
Gradient Ascent
0.213
0.296
0.237
4e9
8e9
2e9
NegGrad+
0.529
0.399
0.463
9.82
9.90
9.85
SCRUB
0.610
0.530
0.578
9.45
9.48
9.48
25

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 17. AUC Scores for MIA and Perplexity across Three Settings for Llama-2 on Enron (Email)
AUC LossMIA
AUC ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.628
0.613
0,418
0.548
0.539
0,451
Retrain
0.598
0.478
0.356
0.524
0.436
0.412
Noisy No Unlearn
0,594
0,572
0.393
0.515
0.502
0,443
Retrain
0.579
0.470
0.375
0.503
0.433
0.434
Unlearning Methods
Random Label
0.234
0.207
0.394
0.333
0.333
0.458
Langevin
0.592
0.560
0.393
0.513
0.494
0.443
EUK
0.620
0,593
0.416
0.532
0.517
0,454
CFk
0.627
0.604
0.416
0.548
0.532
0.449
Gradient Ascent
0.292
0.147
0.377
0.296
0.335
0,488
0.041
0.034
0.064
0.088
0.055
0.215
SCRUB
0.622
0.601
0.418
0.542
0.527
0,451
AUC Min-K %
Perplexity
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.632
0.619
0.421
4.83
4.84
4.84
Retrain
0.594
0.420
0,344
4.84
4.84
4.84
Noisy No Unlearn
0.592
0.569
0.397
5.38
5.38
5.38
Noisy Retrain
0.570
0.410
0.368
5.39
5.39
5.39
Unlearning Methods
Random Label
0.230
0.181
0.330
730
542
255
Langevin
0.590
0.549
0.397
5.36
5.36
5.37
EUk
0.618
0.573
0.415
5.42
5.53
5.37
CFk
0.631
0.603
0.419
4.83
4.83
4.83
Gradient Ascent
0.256
0.219
0.445
4e8
6e12
6e12
NegGrad+
0.041
0.029
0.052
12.15
14.73
6.20
SCRUB
0.627
0.599
0.421
4.86
4.86
4.86
26

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 18. AUC Scores for MIA and Perplexity actoss Three Settings for Llama-2 on ECHR (Year)
AUC LossMIA
AUC ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.570
0.547
0.726
0.513
0.499
0,513
Retrain
0.540
0.500
0.675
0.498
0.476
0.468
Noisy No Unlearn
0.556
0.532
0.721
0.504
0.491
0,510
Noisy Retrain
0.541
0.502
0.685
0.498
0.476
0.476
Unlearning Methods
Random Label
0.503
0.522
0.316
0.486
0.490
0.378
Langevin
0.555
0.528
0.715
0.503
0.488
0.498
EUK
0.572
0,542
0.728
0.513
0.497
0,505
CFk
0.570
0.544
0.724
0.512
0.497
0.509
Gradient Ascent
0.515
0.312
0.254
0.490
0,419
0.343
NegGrad+
0.553
0.364
0.254
0.504
0.429
0.358
SCRUB
0.570
0.547
0.724
0.513
0.499
0.510
AUC Min-K %
Perplexity
Random
Canary
Minority
Random
Canary
Minority
No Unlearn
0.573
0.559
0.686
4.89
4.89
4.89
Retrain
0.537
0.502
0.603
4.89
4.89
4.89
Noisy No Unlearn
0.553
0.541
0.675
5.03
5.03
5.02
Noisy Retrain
0.537
0.504
0.616
5.02
5.02
5.02
Unlearning Methods
Random Label
0.519
0.537
0.327
90
129
127.43
Langevin
0.552
0.535
0,664
5.03
5.03
5.03
EUk
0.572
0.557
0.695
5.27
5.21
5.32
CFk
0.571
0.555
0.682
4.87
4.88
4.87
Gradient Ascent
0.503
0.299
0.257
7.94
10.68
29.48
NegGrad+
0.551
0.299
0.203
4.96
5.10
5.41
SCRUB
0.573
0.558
0.687
4.88
4.88
4.87
27

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 19. AUC Scores for MIA and Perplexity across Three Settings for GPT-2 on Enron (Phone Number) for Noisy Learning
AUC LOssMIA
AUC - ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
Noisy No Unlearn le-4
0.541
0.536
0.427
0.699
0.697
0.621
Noisy Retrain le-4
0,446
0,412
0.309
0.658
0.643
0.577
Langevin lc-4
0.540
0.535
0.424
0.698
0.695
0.619
Noisy No Unlearn 3e-4
0.492
0.484
0.381
0.682
0.678
0.607
Noisy Retrain 3e-4
0,436
0,405
0.311
0.660
0.647
0.583
Langevin 3e-4
0.492
0.483
0.379
0.682
0.678
0.606
Noisy No Unlearn 5e-4
0,474
0.464
0.365
0.678
0.674
0,606
Noisy Retrain 5e-4
0,432
0.403
0.312
0.662
0,649
0.587
Langevin 5e-4
0.472
0.461
0.361
0.678
0.673
0.603
Noisy No Unlearn 8e-4
0,463
0,450
0.352
0.677
0.672
0.606
Noisy Retrain 8e-4
0.428
0.401
0.311
0.664
0.653
0.590
Langevin 8e-4
0,460
0.448
0.350
0.676
0.670
0.604
Noisy No Unlearn le-3
0,459
0.446
0.348
0.677
0.671
0.606
Noisy Retrain le-3
0.427
0.400
0.311
0.665
0.655
0.592
Langevin 1e-3
0.456
0.443
0.347
0.675
0.670
0.604
AUC Min-K
Perplexity
Random
Canary
Minority
Random
Canary
Minority
Noisy No Unlearn le-4
0.599
0.599
0.465
12.26
12.24
12.27
Noisy Retrain le-4
0,454
0,410
0.303
12.25
12.25
12.25
Langevin 1e-4
0.598
0.596
0.460
12.32
12.32
12.33
Noisy No Unlearn 3e-4
0.538
0,535
0.409
13.20
13.20
13.21
Noisy Retrain 3e-4
0.447
0.405
0.308
13.19
13.19
13.19
Langevin 3e-4
0.538
0.535
0.408
13.24
13.24
13.22
Noisy No Unlearn 5e-4
0.518
0.511
0.393
13.84
13.85
13.87
Noisy Retrain 5e-4
0.445
0.406
0.310
13.84
13.84
13.83
Langevin 5e-4
0.516
0.511
0.392
13.84
13.88
13.88
Noisy No Unlearn 8c-4
0.505
0.498
0.378
14.40
14.39
14.39
Noisy Retrain 8e-4
0,442
0.405
0.313
14.42
14.42
14.42
Langevin 8e-4
0.502
0.497
0.376
14.44
14.44
14.43
Noisy No Unlearn le-3
0.500
0.492
0.374
14.71
14.70
14.70
Noisy Retrain 1e-3
0.440
0,408
0.313
14.73
14.73
14.73
Langevin 1e-3
0.497
0.487
0.371
14.74
14.74
14.73
28

---

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Table 20. AUC Scores for MIA and Perplexity across Three Settings for GPT-2 on Enron (Phone Email) for Noisy Learning
AUC LOssMIA
AUC - ZlibMIA
Random
Canary
Minority
Random
Canary
Minority
Noisy No Unlearn le-4
0.584
0.587
0.369
0.480
0.483
0.578
Noisy Retrain le-4
0,431
0,362
0.165
0.387
0.356
0.447
Langevin lc-4
0.584
0.578
0.368
0.479
0.476
0.579
Noisy No Unlearn 3e-4
0,512
0.511
0.311
0.436
0.436
0.535
Noisy Retrain 3e-4
0,424
0,360
0.164
0.384
0.353
0.449
Langevin 3e-4
0.510
0.499
0.309
0.432
0.429
0.534
Noisy No Unlearn 5e-4
0,488
0.483
0.271
0.422
0.421
0.512
Noisy Retrain 5e-4
0,422
0,360
0.165
0.383
0,353
0,453
Langevin 5e-4
0.487
0.475
0.265
0.420
0.416
0.509
Noisy No Unlearn 8e-4
0,470
0,464
0.248
0.414
0.412
0.503
Noisy Retrain 8e-4
0.417
0.357
0.170
0.383
0.354
0.459
Langevin 8e-4
0.471
0.457
0.243
0.411
0.406
0.500
Noisy No Unlearn le-3
0,463
0,456
0.243
0.410
0.408
0.500
Noisy Retrain le-3
0.414
0.354
0.172
0.381
0.355
0.462
Langevin 1e-3
0.462
0.450
0.237
0.408
0.403
0.499
AUC Min-K
Perplexity
Random
Canary
Minority
Random
Canary
Minority
Noisy No Unlearn le-4
0.651
0.661
0.528
12.18
12.25
12.16
Noisy Retrain le-4
0,407
0.318
0.210
12.25
12.25
12.25
Langevin 1e-4
0.655
0.655
0.532
12.27
12.28
12.27
Noisy No Unlearn 3e-4
0.553
0,562
0.420
12.31
12.34
12.28
Noisy Retrain 3e-4
0.390
0.311
0.199
12.23
12.23
12.23
Langevin 3e-4
0.552
0.547
0.421
12.38
12.39
12.38
Noisy No Unlearn 5e-4
0,516
0.519
0.387
12.51
12.52
12.50
Noisy Retrain 5e-4
0.384
0.307
0.199
12.48
12.48
12.48
Langevin 5e-4
0.513
0.505
0.386
12.58
12.58
12.61
Noisy No Unlearn 8c-4
0.478
0.483
0.355
12.83
12.80
12.82
Noisy Retrain 8e-4
0.369
0.298
0.200
12.84
12.84
12.84
Langevin 8e-4
0.477
0.469
0.349
12.87
12.86
12.89
Noisy No Unlearn le-3
0.465
0.467
0.341
13.01
12.98
13.01
Noisy Retrain 1e-3
0.365
0.296
0.201
13.02
13.02
13.02
Langevin 1e-3
0.462
0.453
0.335
13.04
13.03
13.06
29


Reviews:
[{"review_id": "L5zRLp1Nff", "reviewer": "Reviewer_WLjM", "summary": "This paper investigates the underestimated privacy risks faced by minority populations in the context of large language model (LLM) unlearning. The authors argue that current evaluations, which rely on average-case assessments and MIAs, underestimate these risks since minority data is harder to forget. They propose a minority-aware evaluation framework and validate it through controlled experiments with canary injection and real-world datasets (Enron, ECHR). Results show at least 20% higher privacy leakage for minority data across various unlearning methods, MIA variants, datasets, and LLM scales. Among the methods tested, Langevin Unlearning offers the best balance between privacy and utility.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer WLjM for acknowledging the novelty and comprehensiveness of our minority-aware framework, as well as for supporting the acceptance of our paper. Below, we provide detailed responses to the questions and concerns raised.\n\n> Q3 & W3 & C3: Code for our paper.\n\nWe appreciate the reviewer’s emphasis on the importance of code availability for both reproducibility and broader dissemination of our work. Yes, in the supplementary material (provided as a ZIP file), we have included all the relevant code and provided a detailed README.md outlining the installation steps. After the paper is made public, we will further release our GitHub repository to ensure wider accessibility and community engagement.\n\n> W1 & C2 & Q2 : “The paper focuses only on PII data, leaving the applicability to other minority attributes uncertain.” & Could the authors apply their minority-aware evaluation framework to non-PII minority data, such as dialects or underrepresented social groups?\n\nWe thank the reviewer for this insightful and important question. We fully agree that clearly defining what constitutes a “minority” in different contexts is nontrivial, and that generalizing beyond PII is an open and meaningful research direction.\n\nIn this work, we chose to focus on PII for two key reasons. First, PII is inherently sensitive and has been a central focus in privacy-related studies [1,2,3], making it a practical and impactful setting for studying unlearning. Second, PII (such as phone number area codes or email domains) offers a well-defined population structure, which allows us to rigorously formulate and operationalize the notion of minority subgroups in a reproducible way.\n\nThat said, when non-PII attributes exhibit a clear format and can be associated with measurable population-level frequencies, our framework can be directly generalized to these settings. For attributes that are more semantically defined or lack an explicit population distribution, such as linguistic patterns, we acknowledge that identifying minority status is more challenging. Extending our framework to these settings is a promising direction for future work, and we would be excited to explore this further.\n\n[1] Lukas et al. Analyzing leakage of personally identifiable information in language models. In S&P 2023.\n\n[2] Kim et al. Propile: Probing privacy leakage in large language models. In NeurIPS 2023.\n\n[3] Li et al. Llm-pbe: Assessing data privacy in large language models. In VLDB 2024.\n\n> W2: The paper does not explore the impact of different fine-tuning strategies, such as LoRA versus full fine-tuning.\n\nWe appreciate the reviewer’s suggestion. Exploring the impact of different fine-tuning strategies, such as LoRA versus full fine-tuning, on unlearning and privacy leakage is indeed an interesting direction. Our current experiments are designed to support our core claim that minority samples are disproportionately vulnerable to privacy leakage across multiple datasets, attack methods, and model scales. Due to computational constraints, especially for large-scale multi-GPU training, we have so far focused on the more resource-efficient LoRA-based fine-tuning strategy. While full fine-tuning on models like LLaMA-7B would be a valuable extension, we believe it is not essential to establish the key findings of our study. We view this as a promising avenue for future exploration and would be happy to investigate it as resources permit.\n\n>C1 & Q1: Have the authors considered testing larger models (e.g., 13B, 30B) to see if privacy risks scale with model size?\n\nWe thank the reviewer for the valuable question. Investigating how privacy risks scale with model size under our minority-aware framework is indeed an interesting and important direction. Prior work has demonstrated that both verbatim memorization and extractability tend to increase with model scale [4], which suggests that minority-group privacy risks may also become more severe in larger models. While our current experiments include both small (GPT-2) and mid-sized (LLaMa-2 7B) models to support our central claims, which align with model sizes used in current literature [1,5,6], computational constraints have limited our ability to run experiments on larger models such as 13B or 30B. We view this as a promising avenue for future work and plan to explore it as resources become available.\n\n[4] Carlini et al. Quantifying memorization across neural language models. In ICLR 2023.\n\n[5] Maini et al. Tofu: A task of fictitious unlearning for llms. In COLM 2024.\n\n[6] Shi et al. Muse: Machine unlearning six-way evaluation for language models. In ICLR 2025.\n\n**Summary:**\nWe are grateful to Reviewer WLjM for the supportive and thoughtful review. We would be happy to provide further clarifications or engage in additional discussion. If our responses have helped resolve the raised concerns, we humbly hope this might be taken into account in the final score.", "claims_and_evidence": "Claims: \n1. Standard LLM unlearning evaluations underestimate privacy risks for minority groups, as minority data is harder to forget. \n2. A minority-aware evaluation framework reveals overlooked privacy vulnerabilities. \n3. Different unlearning methods vary in effectiveness, with Langevin Unlearning achieving the best privacy-utility balance.\n\nEvidence:\n1. Experiments show that minority data (e.g., rare area codes) is more prone to memorization and experiences higher privacy leakage.\n2. Canary injection and direct minority subset removal confirm greater privacy risks compared to randomly selected forget sets.\n3. Benchmarking across multiple datasets (Enron, ECHR), MIAs (lossMIA, zlibMIA, Min-K%), and LLMs (GPT-2, LLaMA-2 7B) consistently supports the claims.", "methods_and_evaluation_criteria": "Methods: \n\nThe study leverages well-established MIAs to assess unlearning efficacy and introduces canary injection to isolate minority data effects. It uses real-world datasets (Enron, ECHR) containing PII to enhance relevance. Unlearning methods are tested across three scenarios: Random (standard forget set selection), Canary (synthetically modified rare identifiers), and Minority (naturally rare data removal).\n\nEvaluation Criteria: \n\nThe methodology could be improved by evaluating larger LLMs (e.g., 13B, 30B) to assess whether privacy risks scale with model size. Expanding beyond PII to include underrepresented dialects or cultural biases would further strengthen the study’s applicability.", "theoretical_claims": "The paper builds on established principles of machine unlearning and privacy auditing without introducing complex theoretical proofs.", "experimental_designs_or_analyses": "The paper rigorously compares multiple unlearning methods across datasets and LLM architectures, yielding statistically significant results with consistent trends. However, its focus on PII data limits generalizability to other sensitive information types, and it does not explore how different fine-tuning strategies, such as full fine-tuning versus LoRA-based tuning, impact unlearning effectiveness.", "relation_to_broader_scientific_literature": "The paper builds on prior work in privacy auditing, machine unlearning, and membership inference attacks. It addresses a critical gap in the literature by focusing on the privacy risks for minority groups, which are often overlooked in standard unlearning evaluations.", "essential_references_not_discussed": "The paper provides a comprehensive and well-researched discussion, citing key works on machine unlearning, privacy risks, and membership inference attacks. It references foundational studies on differential privacy and unlearning (e.g., Guo et al., 2020; Bourtoule et al., 2021), as well as LLM privacy vulnerabilities (e.g., Carlini et al., 2022; Nasr et al., 2023).", "other_strengths_and_weaknesses": "Strengths:\n1. The paper highlights a critical oversight in LLM unlearning evaluations, emphasizing the unique privacy risks faced by minority groups.\n2. The paper presents comprehensive experiments with diverse datasets, unlearning methods, and MIA techniques.\n3. The authors introduces a novel minority-aware evaluation framework, enhancing standard unlearning assessments.\n\nWeaknesses:\n1. The paper focuses only on PII data, leaving the applicability to other minority attributes uncertain.\n2. The paper do not explore the impact of different fine-tuning strategies, such as LoRA versus full fine-tuning.\n3. The authors do not provide code, which limits reproducibility and independent verification.", "other_comments_or_suggestions": "1. The authors should consider evaluating larger LLMs (13B, 30B) to assess scaling effects.\n2. The study should be expanded to include non-PII minority data, such as linguistic or cultural factors.\n3. The authors should release code and datasets to improve transparency and reproducibility.", "questions_for_authors": "1. Have the authors considered testing larger models (e.g., 13B, 30B) to see if privacy risks scale with model size?\n2. Could the authors apply their minority-aware evaluation framework to non-PII minority data, such as dialects or underrepresented social groups?\n3. Do the authors plan to release their code and datasets to enhance reproducibility?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WpZjYVoiMX", "reviewer": "Reviewer_Cub4", "summary": "The paper argues that minority data points are harder to unlearn than the common typical example. To show this, the authors construct canaries by replacing PII in two datasets in the forget set with infrequent PIIs. Then the authors show that common unlearning algorithms struggle to unlearn these data points under MIA evaluation compared to the rest common examples.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank Reviewer Cub4 for reviewing our paper. Below, we address the questions and comments raised by Reviewer Cub4. \n\n> The paper fails to provide a rigorous definition of ‘minority’. The definition should use a mathematical model.\n\nWe thank the reviewer for this comment. As noted in Footnote 2 and detailed in Sections 4.1 and 4.2, our paper defines minority instances based on the population frequency of their corresponding identifiers (e.g., rare phone area codes or email domains). This definition is precise, grounded in observable statistics, and enables systematic evaluation.\n\nWe further validate it through controlled canary experiments, where replacing common identifiers with rare ones while keeping the rest of the input fixed leads to significantly higher privacy leakage. Similar trends are observed in real-world minority samples (“Minority Setting”).\n\nWhile our definition is frequency-based, we welcome clarification on the type of mathematical model the reviewer envisions. Extending the framework to semantically defined minority groups may benefit from additional modeling, which we consider a promising future direction.\n\n> The minority points seem to conflate with outliers or hard-to-unlearn examples.\n\nWe thank the reviewer for raising this point and allowing us to clarify a potential misunderstanding. Our experiments demonstrate that the PII-population-defined minority samples consistently exhibit significantly higher privacy leakage across a wide range of unlearning methods. These results indicate that our defined minority points are systematically harder to unlearn in terms of privacy risk, as acknowledged by Reviewer jBLe. To be clear, by harder to unlearn, we refer specifically to their elevated vulnerability under privacy metrics. To prevent potential confusion, we will further refine the language in the paper.\n\n> The claim that Langevin unlearning performs better due to noise addition is unconvincing, as its performance instead can be attributed to its theoretical guarantees.\n\nWe thank the reviewer for their comment. There is an intrinsic connection between noise injection and the theoretical guarantees of Langevin unlearning. While it is true that Langevin dynamics enjoys privacy guarantees, these guarantees fundamentally rely on the analysis of Langevin Dynamic, how injected Gaussian noise affects the model’s trajectory over adjacent datasets, typically measured via Rényi divergence. In other words, the theoretical robustness of Langevin unlearning is itself a direct consequence of noise injection. We will revise the paper to make this connection more explicit.\n\n> Using PII to represent minority data points lacks justification—there’s no clear reason why PII should indicate minority status.\n\nWe thank the reviewer for this question. First, PII inherently contains sensitive personal information, making it a central focus in privacy-related research [1,2,3]. Moreover, using PII also enables us to clearly define minority points by leveraging population-level frequency (e.g., rare area codes or email domains), which we believe helps address the reviewer’s concern regarding the clarity of the minority definition. Finally, from an empirical perspective, we show that instances defined as minority under this framework consistently exhibit significantly higher privacy leakage across multiple unlearning methods.\n\n[1] Lukas et al. Analyzing leakage of personally identifiable information in language models.\n\n[2] Kim et al. Propile: Probing privacy leakage in large language models.\n\n[3] Li et al. Llm-pbe: Assessing data privacy in large language models.\n\n> Other related references.\n\nWe thank the reviewer for sharing the relevant works [4, 5, 6]. These papers focus on corrective unlearning, which aims to address corrupted or harmful training data. As noted in these papers, their goals and assumptions differ from those of privacy-oriented unlearning, which is the focus of our work. To acknowledge their relevance, we have cited them in the introductory paragraph of Section 3 (Preliminaries), where we discuss the broader unlearning landscape.\n\nWhile canary crafting has been explored in privacy research, it remains overlooked in evaluating unlearning methods for LLMs. Our work fills this gap by demonstrating that selecting high-risk samples, especially minority points, is key to uncovering underestimated privacy risks in current unlearning evaluations.\n\n[4] Goel et al. Corrective machine unlearning.\n\n[5] Li et al. Delta-Influence: Unlearning Poisons via Influence Functions.\n\n[6] Schoepf et al. Potion: Towards poison unlearning.\n\n**Summary:**\nWe thank Reviewer Cub4 for the thoughtful and detailed feedback. We appreciate the opportunity to respond and are happy to clarify any remaining concerns. If our responses have addressed the points raised, we would be grateful if this could be reflected in the final evaluation.", "claims_and_evidence": "I find the central claim of minority data points conflated with the idea  of approximate unlearning algorithms find it harder to unlearn outlier examples. It is possible that is what the authors want to show but in my opinion, that is a very handwavy argument unless the authors model it rigorously. In that regard, I find the paper flawed as it hard to say what minority is. The experimental results clearly show that common unlearning algorithms do not work uniformly well and the authors have found a canary strategy that succeeds at showing this. But I am not convinced it shows much about \"minority\" without a proper characterization of what minority is.\n\nThe authors claim that Langevin unlearning performs better than other and so as it is the only method that adds noise, noise has a crucial role to play. But, langevin unlearning, unlike SCRUB and Gradient Ascent, comes with theoretical guarantees. That could also be the reason why it holds up. I find the argument that noise addition may have some important role a bit unnecessary given that noise is needed for that definition of unlearning to hold theoretically.", "methods_and_evaluation_criteria": "The idea of using PII as a marker of minority group affiliation seems odd to me. Why should PII indicate whether a data point belongs to minority or not. Either there should be an argument to motivate why this is a sensible choice or the authors can propose a mathematical model where this is a sensible choice.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "In addition to my point above, another experiment to do could be to conduct the same experiments on unlearning data points which are classically considered to be minority in datasets that are used for studying fairness. While this takes away the concept of canaries, it can help drive home the point about minorities with a bit less experimental interventions.", "relation_to_broader_scientific_literature": "One of the main contributions of the paper is adversarial evaluation of unlearning using canaries. This is now a broadly studied field in Machine unlearning. Corrective machine unlearning https://arxiv.org/abs/2402.14015 and follow-up works (e.g. https://arxiv.org/abs/2406.09173 and https://arxiv.org/abs/2411.13731) also inject special data point (poisons) and try to unlearn them. So, I am not sure the idea of unlearning specifically injected data points is very novel.", "essential_references_not_discussed": "See the references above.", "other_strengths_and_weaknesses": "I have mentioned the important points above. From an originality perspective, the central message is interesting and while it may seem obvious it would be good to see a thorough evaluation of the same. The paper attempts that. However, I find the idea of minority, outliers, hard to unlearn etc all conflated together in the absence of a mathematical model.", "other_comments_or_suggestions": "post rebuttal\n\nI thank the authors for the rebuttal. I have read it and I maintain my original rating. \n\nMy core concern about conflating the idea of \"rare PII\" with \"minority\" is not resolved here. The rebuttal clarifies this by saying \"our paper defines minority instances based on the population frequency of their corresponding identifiers \" but this is not really explaining the reasoning behind the choice but just stating that the choice was made. I might be inclined to agree that the paper says rare PIIs are harder to unlearn but saying minority groups are harder to unlearn requires an additional step to show why \"rare PIIs\" constitute a broad class of \"minority groups\". \n\nThere are several ways to do it including referring to the extensive fairness literature, providing a mathematical model (I think its up to the authors to define it and not me providing it), or simply restricting the scope of the work. I think making this broad stroke to say this paper talks about hardness of unlearning minorities is misleading. Finally, also see my note in the original review above on additional experiments on classical fairness datasets.\n\nI am not sure I totally follow what the rebuttal is trying to clarify about langevin unlearning. Just to clarify, any algorithm that provides certifiable unlearning (like lavengin unlearning) and is not exact, necessarily needs some randomness. This is not surprising and any claim regarding deterministic algorithm that claims to do this is not right.  My entire point is that it is unnecessary to talk about noise addition as being something special, its just that Langevin unlearning comes with guarantees and  thus it is not possible to evade the unlearning guarantee, unlike all other methods like SCRUB etc. \n\nFinally, the rebuttal claims \"As noted in these papers, their goals and assumptions differ from those of privacy-oriented unlearning, which is the focus of our work. To acknowledge their relevance, we have cited them in the introductory paragraph of Section 3 (Preliminaries), where we discuss the broader unlearning landscape.\"\n\nMy point was not that these papers be cited in the introduction or even that they do not consider privacy-oriented unlearning. Rather my point was that they do adversarial evaluation of unlearning and thus the strategy of inserting different kinds of special data points (poisons, canaries, Witches brew point) have been considered here and some of their strategy can be used here.", "questions_for_authors": "N/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1PGDi2lpW", "reviewer": "Reviewer_jBLe", "summary": "This paper studies how the unlearning quality of many proposed unlearning methods degrades when applied to minority groups. This was observed across evaluting unlearning with several MI attacks, datasets, and models. Building on this insight, their proposed maximum leakage informed a new ablation of unlearning methods, showing langevin unlearning offered the best privacy-utility trade-off. These ablations also showed common approaches to unlearning are unstable (e.g., gradient ascent).\n\n## Update after Rebuttal\n\nThe authors addressed my concerns, and I raised my score to an accept. I believe this paper provides valuable insight and motivation for future work on the intersection of unlearning and fairness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer jBLe for the insightful suggestions and for recognizing the potential impact of our work on the field of unlearning. Below, we address the reviewer’s questions and suggestions.\n\n> Major comments & W1. Clarification in Introduction and Related Work.\n\nWe sincerely thank the reviewer for the thoughtful suggestion. We agree that further clarifying our contributions relative to prior work will help readers better situate our findings and we outline below the changes we have made in the Introduction and Related Work sections to address this point.\n\n**Revision to the Introduction:**\nWe have revised the paragraph beginning with “We identify a critical pitfall...” to more explicitly highlight the dual motivation behind our work. First, prior work [1,2] shows that unlearning difficulty varies significantly across samples in vision tasks. Second, LLM studies reveal highly non-uniform memorization patterns. Building on both insights, we first show that privacy risk in LLM unlearning varies across samples, and further demonstrate that high-risk points often coincide with identifiable minority groups. This connection adds a new dimension of societal relevance to existing technical findings.\n\n**Revision to the Related Work:**\nWe have updated the final paragraph of the Related Work section to more accurately describe [2] as estimating the unlearning difficulty of individual samples in vision tasks using an entanglement-based metric, and showing how variations in the memorization of image representations impact unlearning performance. Our study investigates how minority subgroups in LLMs fine-tuning systematically suffer from degraded unlearning efficacy, an effect that coincides with harder-to-unlearn examples. As a future direction, it would be interesting to explore the connection between such entanglement-based difficulty measures and our minority-aware findings in the LLM setting.\n\n[1] Thudi et al. Gradients look alike: Sensitivity is often overestimated in DP-SGD.\n\n[2] Zhao et al. What makes unlearning hard and what to do about it.\n\n> Missing clarification on evaluation repetitions.\n\nWe thank the reviewer for the careful comment. All components involving randomness were run with a fixed random seed (42), as noted in App. B.2 and in line with common practice. Due to the large scale of our experiments, we did not repeat runs with multiple seeds. We will clarify this in App. B.2 and note that using a broader range of seeds could further improve robustness in future work.\n\n> Could the authors compare the metric proposed in [2]?\n\nWe thank the reviewer for this insightful suggestion. We believe the reviewer is referring to the ToW metric proposed in [2]. While the original ToW metric is designed for vision tasks and based on prediction accuracy, its underlying spirit can be adapted to our LLM setting using privacy-related metrics. Specifically, we construct a ToW metric based on AUC differences between the unlearned model ($M_{\\text{unlearn}}$) and a retrained model ($M_{\\text{retrain}}$) across key dataset partitions. Our adapted ToW metric:\n\n$\\text{ToW} = \\prod_{(D_i, D_j)} \\left(1 - \\left| \\text{AUC}(M_{\\text{retrain}}, D_i, D_j) - \\text{AUC}(M_{\\text{unlearn}}, D_i, D_j) \\right| \\right)$\n\nwhere $(D_i, D_j) \\in \\{ (D_{\\text{retrain}}, D_{\\text{forget}}), (D_{\\text{forget}}, D_{\\text{test}}), (D_{\\text{retain}}, D_{\\text{test}})\\}$. Higher ToW indicates lower privacy leakage.\n\nWe applied this metric to the ECHR dataset with GPT-2 under loss-based attack. As shown in the table, the canary and minority settings consistently yield lower ToW scores than the random setting, indicating greater privacy leakage. These results reinforce our main finding that minority points are harder to unlearn. In addition, we believe it would be interesting future work to further investigate how the idea behind ToW could be leveraged to help identify minority points in the LLM setting.\n\n\n|**Setting**|No Unlearn|Gradient Ascent|Random Label|EUk|CFk|NegGrad+|SCRUB|Langevin Unlearning|\n|:--------------:|:------------:|:------------------:|:---------------:|:------:|:------:|:-----------:|:--------:|:----------------------:|\n|**Random**| 0.768| 0.917| 0.769|0.739|0.780|0.782|0.771|0.877|\n|**Canary**| 0.727| 0.881| 0.729|0.694|0.747|0.753|0.732|0.834|\n|**Minority**| 0.717| 0.861| 0.721|0.733|0.728|0.732|0.714|0.863|\n\n> Typos.\n\nWe thank the reviewer for pointing out these typos and have corrected both in the revised version.\n\n**Summary:**\nWe sincerely thank Reviewer jBLe again for the thoughtful and constructive feedback. We would be happy to clarify any remaining concerns or engage in further discussion. We hope that our responses have addressed the key points raised, and we truly appreciate your openness to reconsidering the score upon clarification.", "claims_and_evidence": "The paper makes a (somewhat implicit) claim that they are the first to observe disparate unlearning guarantees/quality amongst datapoints, but this is a known phenomenon in the literature. It was shown in [1] that when using DP-SGD to unlearn (i.e., Langevin dynamics), many datapoints have magnitudes better unlearning guarantees than the worst-case, i.e., there are significant differences between worst-case privacy leakage and average case. Empirically, across a broad range of unlearning methods, [2] provided a metric to identify these harder to unlearn datapoints. [2] is cited in this paper as “concurrent work”, but given the paper is now almost a year old, “concurrent work” may not be an apt characterization of the results of that paper. Also [1] is nearly 2 years old, and Langevin unlearning was the method found to be the best in this paper (and [1] studies it without additional fine-tuning).\n\nThis said, I think the paper still has impact with a different claim. In the context of this previous literature, I believe the current paper helps expand the societal impact of these past findings by showing these known disparate unlearning guarantees/quality also correspond to identifiable minority groups. I hope the authors consider reframing the claims in the paper accordingly, and look forward to discussion during the rebuttal on this. \n\n[1] Thudi, Anvith, et al. \"Gradients look alike: Sensitivity is often overestimated in {DP-SGD}.\" 33rd USENIX Security Symposium (USENIX Security 24). 2024.\n\n[2] Zhao, Kairan, et al. \"What makes unlearning hard and what to do about it.\" Advances in Neural Information Processing Systems 37 (2025): 12293-12333.", "methods_and_evaluation_criteria": "The methods and evaluation made sense, and I would say were quite thorough (many ablations are conducted).", "theoretical_claims": "No theoretical claims were made.", "experimental_designs_or_analyses": "I found no significant issues with the experimental design or analyses from the description in the main body and Appendix section A-C.7. \n\nThere is no description of repetitions for the random, canary, and minority evaluations. Given the number of settings tested I do not believe this is an issue, but raise it here in case the authors did have repetitions and I missed this description (or can add suitable repetitions over any randomness in the evaluation pipeline).", "relation_to_broader_scientific_literature": "As mentioned in the claims, past work had observed disparities in the unlearning guarantees over datapoints. This paper adds to this observation by showing identifiable minority groups are disproportionately affected. This finding also has similarities to the disparities observed in the privacy literature [3] for minority group, though I believe those findings are focused on the impact privacy has on performance across groups.\n\n[3] Bagdasaryan, Eugene, Omid Poursaeed, and Vitaly Shmatikov. \"Differential privacy has disparate impact on model accuracy.\" Advances in neural information processing systems 32 (2019).", "essential_references_not_discussed": "[1] observed disparate unlearning guarantees for DP-SGD, i.e., Langevin unlearning without the additional fine-tuning which would further minimize this initial divergence. [2] is cited as “concurrent” work, though it is nearly a year old and was accepted to Neurips 2024.\n\n[1] Thudi, Anvith, et al. \"Gradients look alike: Sensitivity is often overestimated in {DP-SGD}.\" 33rd USENIX Security Symposium (USENIX Security 24). 2024.\n\n[2] Zhao, Kairan, et al. \"What makes unlearning hard and what to do about it.\" Advances in Neural Information Processing Systems 37 (2025): 12293-12333.", "other_strengths_and_weaknesses": "Strengths:\n\n1) Thorough and insightful empirical evaluation\n2) The paper is well-written\n3) Ablations have clear impact on future unlearning methodology\n\nWeaknesses:\n\n1. The differences between average and worst-case datapoints for unlearning is already known in the literature (requires further refinement of claims made in the paper)", "other_comments_or_suggestions": "I believe this paper can have significant impact to the field of unlearning, but requires clarifying the results in the context of previous findings. I am happy to raise my score given clarifications to my “major” suggestion. I also have other suggestions, but label these as relatively more minor.\n\nMajor:\n1) I suggest clarifying in the introduction and related work section that previous work had found that there is a gap between the unlearning guarantees for the easiest/average and hardest datapoints (see comments and references in the claims section). Given this, the authors can claim to build on this past insight by showing these harder datapoints coincide with minority groups.\n\nMinor:\n1) If possible, could the authors compare the scores of the metric proposed in [2] across the average and minority evaluations? This would help connect the findings to that work.\n2) In EUk and CFk paragraph, the authors wrote “expansive” I think you mean “expensive”.\n3) In line 364 do you mean “by at least” instead of “for at least”?\n\n[2] Zhao, Kairan, et al. \"What makes unlearning hard and what to do about it.\" Advances in Neural Information Processing Systems 37 (2025): 12293-12333.", "questions_for_authors": "See question about typos in the suggestions section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work highlights a critical issue in unlearning that not all samples are of the same difficulty to unlearn. Though this observation is not in it self entirely new, this work presents a careful exploration into this idea and proposes methods to evaluating this. Two reviewers both agreed that this presents a novel and important contribution to the field, and that the evaluation was well carried out.

Despite this, I encourage the authors to consider their usage of minority and how it may conflate with outlier, especially in terms of the empirical setup and takeaways. This was one notable concern left unaddressed after the rebuttal. The proposed changes during the rebuttal will also help address concerns around placement of this work within the literature and it's use of insights from past work.
Decision: Accept (poster)