ICML 2025

Underestimated Privacy Risks for Minority Populations in Large Language
Model Unlearning
Rongzhe Wei 1 Mufei Li 1 Mohsen Ghassemi 2 Eleonora Kreaˇci´c 2 Yifan Li 3 Xiang Yue 4 Bo Li 5 6
Vamsi K. Potluru 2 Pan Li 1 Eli Chien 1
Abstract
Large Language Models (LLMs) embed sensitive,
human-generated data, prompting the need for un-
learning methods. Although certified unlearning
offers strong privacy guarantees, its restrictive as-
sumptions make it unsuitable for LLMs, giving
rise to various heuristic approaches typically as-
sessed through empirical evaluations. These stan-
dard evaluations randomly select data for removal,
apply unlearning techniques, and use membership
inference attacks (MIAs) to compare unlearned
models against models retrained without the re-
moved data. However, to ensure robust privacy
protections for every data point, it is essential to
account for scenarios in which certain data sub-
sets face elevated risks. Prior research suggests
that outliers, particularly including data tied to mi-
nority groups, often exhibit higher memorization
propensity which indicates they may be more dif-
ficult to unlearn. Building on these insights, we in-
troduce a complementary, minority-aware evalua-
tion framework to highlight blind spots in existing
frameworks. We substantiate our findings with
carefully designed experiments, using canaries
with personally identifiable information (PII) to
represent these minority subsets and demonstrate
that they suffer at least 20% higher privacy leak-
age across various unlearning methods, MIAs,
datasets, and LLM scales. Our proposed minority-
aware evaluation framework1 marks an essential
step toward more equitable and comprehensive
assessments of LLM unlearning efficacy.
1Georgia Institute of Technology 2J.P. Morgan AI Research
3Tsinghua University 4Carnegie Mellon University 5The Univer-
sity of Chicago 6University of Illinois Urbana-Champaign. Cor-
respondence to: Rongzhe Wei <rwei42@gatech.edu>, Pan Li
<panli@gatech.e...