LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks
Authors: Soumyadeep Pal
Changsheng Wang
James Diffenderfer
Bhavya Kailkhura
Sijia Liu
Venue: COLM 2025 (2025)
Keywords: Machine Unlearning
Coreset
Large Language Models
TL;DR: We uncover a novel coreset effect in current LLM Unlearning benchmarks, where unlearning performance can be effectively maintained using significantly smaller subsets, e.g., as little as 5% of the forget set.

Abstract: Large language model (LLM) unlearning has become a critical challenge in ensuring safety and controlled model behavior by removing *undesired* data-model influences from the pretrained model while preserving its general utility. Significant recent efforts have been dedicated to developing LLM unlearning benchmarks such as WMDP (Weapons of Mass Destruction Proxy) and MUSE (Machine Unlearning Six-way Evaluation), facilitating standardized unlearning performance assessment and method comparison. Despite their usefulness, we uncover for the first time a novel *coreset effect* within these benchmarks. Specifically, we find that LLM unlearning achieved with the original (full) forget set can be effectively maintained using a significantly smaller subset (functioning as a "coreset"), *e.g.*, as little as 5% of the forget set, even when selected at random. This suggests that LLM unlearning in these benchmarks can be performed surprisingly easily, even in an extremely low-data regime. We demonstrate that this coreset effect remains strong, regardless of the LLM unlearning method used, such as NPO (Negative Preference Optimization) and RMU (Representation Misdirection Unlearning), the popular ones in these benchmarks. The surprisingly strong coreset effect is also robust across various data selection methods, ranging from random selection to more sophisticated heuristic  approaches. We explain the coreset effect in LLM unlearning  through a keyword-based perspective, showing that keywords extracted from the forget set alone contribute significantly to unlearning effectiveness and indicating that current unlearning is driven by a compact set of high-impact tokens rather than the entire dataset. We further justify the faithfulness of coreset-unlearned models along additional dimensions, such as mode connectivity and robustness to jailbreaking attacks.

Published as a conference paper at COLM 2025
LLM Unlearning Reveals a Stronger-Than-EEpeete Coreset Effect in Current Benchmarks
Soumyadeep Pall*
Changsheng Wang** James Diffenderfer
Bhavya Kailkhuraf
Sijia Liu
*Michigan State University, *Lawrence Livermore National Laboratory, SIBM Research
'Equal contribution
Abstract
Large language model (LLM) unlearning has become a critical challenge in ensuring safety and controlled model behavior by removing undesired data- model influences from the pretrained model while preserving its general utility. Significant recent efforts have been dedicated to developing LLM unlearning benchmarks such WMDP (Weapons of Mass Destruction Proxy) and MUSE (Machine Unlearning Six-way Evaluation), facilitating standardized unlearning performance assessment and method compari- son. Despite their usefulness, we uncover for the first time novel coreset effect within these benchmarks. Specifically, we find that LLM unlearning achieved with the original (full) forget set can be effectively maintained using significantly smaller subset (functioning as a "coreset"), e.g., as little as 5% of the forget set, even when selected at random. This suggests that LLM unlearning in these benchmarks can be performed surprisingly easily, even in an extremely low-data regime. We demonstrate that this coreset effect remains strong, regardless of the LLM unlearning method used, such NPO (Negative Preference Optimization) and RMU (Representation Misdirection Unlearning), the popular ones in these benchmarks. The sur- prisingly strong coreset effect is also robust across various data selection methods, ranging from random selection to more sophisticated heuristic approaches. We explain the coreset effect in LLM unlearning through a keyword-based perspective, showing that keywords extracted from the forget set alone contribute significantly to unlearning effectiveness and in- dicating that current unlearning is driven by compact set of high-impact tokens rather than the entire dataset. We further justify the faithfulness of coreset-unlearned models along additional dimensions, such as mode connectivity and robustness to jailbreaking attacks. Codes are available https:/ wwttgg.ctttt/teeeee wawaaboooo/OOOOMOO////e/ MU-Coreset.
Introduction
The problem of machine unlearning (MU) for large language models (LLMs), referred to as LLM unlearning (Liu al., 2025; Si et al., 2023; Qu et al., 2024; Cooper al., 2024), is gaining critical importance as means of enforcing data privacy rights (e.g, preventing the generation of copyrighted or sensitive content) (Eldan & Russinovich, 2023; Shi al., 2024b; Maini et al., 2024; Jang et al., 2022), and for removing harmful or unsafe knowledge embedded in models amid growing concerns around safety and alignment (Li al., 2024; Yao et al., 2024; Barez et al., 2025; Zhang et al., 2024b; Chen et al., 2025). The core objective of MU is to remove the influence of specific, undesired data or knowledge from trained model, while preserving its general utility, without the cost of full retraining from scratch.
Despite the growing importance of LLM unlearning, much of the existing research has primarily focused on the design of unlearning algorithmss. Notable approaches include gradient ascent and its variants (Thudi et al., 2022; Jang et al., 2022; Yao et al., 2024), which aim to reverse the training effect of the forget data by explicitly pushing the model away from learned patterns; influence function-baseer methods (Izzo et al., 2021; Jia et al., 2024b; Koh & Liang, 2017), which assess and mitigate the contribution of individual data points to

---

Published as a conference paper at COLM 2025
model behavior; preference optimization techniques and their extensions (Rafailov et al., 2023; Zhang et al., 2024a; Fan et al., 2024b), which guide the model to reduce its preference for responses associated with the forget data; misrepresentation learning methods (Li et al., 2024), which disrupt internal representations linked to the undesired knowledge; and neuron- or model-editing approaches that leverage task vectors or localized model components to guide unlearning (Jia et al., 2024a; Hase et al., 2023; Wu et al., 2023).
While the above LLM unlearning algorithms have advanced the field, the data perspective of LLM unlearning, such as how the composition or size of the forget set (i.e., the dataset used to define the unlearning scope) influences unlearning performance, has received significantly less attention. On the data side, some recent efforts explored evaluating the memorization level of forget data to better guide unlearning optimization (Barbulescu & Triantafillou, 2024; Zhao et al., 2024). Yet, most existing efforts focused on developing LLM unlearning benchmark datasets, such as TOFU (Task of Fictitious Unlearning) to remove synthesized fictitious information (Maini et al., 2024), WMDP (Weapons of Mass Destruction Proxy) for harmful know removal (Li et al., 2024) and MUSE (Machine Unlearning Six-way Evaluation) for copyrighted data removal (Shi et al., 2024b) and other similar datasets (Eldan & Russinovich, 2024; Liu et al., 2024).
Although the above benchmarks have served standardized platforms for LLM unlearn- ing training and evaluation, the forget datasets curated in these benchmarks are typically provided as fixed, full sets, and existing research has largely adopted them without ques- tioning the appropriateness, sufficiency, or redundancy of these forget sets for achieving effective unlearning. For example, in WMDP, a large forget data source is provided contain- ing sensitive documents (25K articles) related to hazardous knowledge in biosecurity and cybersecurity. However, the forget set actually used in the WMDP-baser unlearning via RMU (Representation Misdirection Unlearning) utilizes only the first 600 articles from these source documents (Li et al., 2024), leaving open the question of whether the full dataset is necessary or if a much smaller subset could suffice. Therefore, in this work, we ask:
(Q) Does there exist a "coreset" within the LLM unlearning forget dataset that can yield
lossless unlearning performance?
The term "coreset" is inspired by prior research on coreset selection for non-LLMs (Guo et al., 2022; Borsos et al., 2020; Zhang et al., 2023), where a subset of the training set enables comparable model performance to training the full dataset. We envision that addressing (Q) could open up a new, underexplored paradigm: LLM unlearning in the low-data regime, where effective unlearning is achieved using only a small subset of the forget data. It may also reveal potential weaknesses in current unlearning benchmarks (Thaker et al., 2024a), where unlearning could be surprisingly easy due to redundancy in the forget set.
In this work, we provide an affirmative answer to (Q) through extensive empirical studies: coreset effect does exist in LLM unlearning, and unlearning in current benchmarks such as WMDP and MUSE can be effectively achieved using as little as 5% of the original full forget set, even when the coreset randomly selected. We summarize our contributions below.
We unveil a novel Coreset Effect in LLM unlearning, showing that popular unlearning methods (RMU and negative preference optimization) achieve comparable performance on benchmarks like WMDP and MUSE using as little as 5% of the forget set.
We demonstrate that this effect holds across both random and more sophisticated coreset selection strategies. We also provide a keyword- based analysis to explain the strong coreset effect where small forget subsets can drive unlearning behavior.
We show that coreset-unlearned models exhibit similar unlearning characteristics to full- set-unlearned ones across dimensions including mode connectivity, adversarial robustness, and model utility, except for a potential increased vulnerability to relearning attacks.
Related Works
LLM Unlearning. Unlearning seeks to remove the influence of undesired data-model influence to protect privacy or prevent harmful knowledge generation. A commonly- gold standard is exact dedearninn--etrriniiin the model from scratch without the forget
2

---

Published as a conference paper at COLM 2025
data (named "Retrain") (Cao & Yang, 2015; Thudi et al., 2022) -but this is computationally prohibitive, especially for LLMs. As a result, significant efforts have focused on approximate unlearning (Bourtoule et al., 2021; Liu et al., 2025), which modifies pretrained model weights post hoc. These approaches fall broadly into model finetuning methods (Ilharco et al., 2022; Li et al., 2024; Zhang et al., 2024a; Fan et al., 2024b; Jía et al., 2024a), and input-based prompting methods (Pawelczyk et al., 2023; Thaker et al., 2024b). Recent studies have also highlighted vulnerabilities in these methods, including jailbreaking (Lucki et al., 2024; Lynch et al., 2024), latent extraction (Seyitoglu et al., 2024), and the-tunnng-basse relearning attacks (Hu et al., 2024; Deeb & Roger, 2024), revealing fragility in unlearning robustness. Despite this progress, Thaker et al. (2024a) argues that current benchmarks offer an overly optimistic view due to weak evaluation protocols. As a complementary concern, we uncover a strong coreset effect in standard LLM unlearning benchmarks, showing that effective unlearning can be achieved using surprisingly small subsets of the forget data, which raises potential questions about the reliability and rigor of current unlearning dataset designs.
Data perspective in unlearning. Recent work has begun to explore the influence of forget data on unlearinn outcomes, particularly through the lens of data ordering and importance. For instance, Zhao et al. (2024) the forget set into homogeneous subsets using data quality scores, such as representation distance from the centroid and memorization score, and applies specialized unlearning strategies to each subset sequentially. Zhao & Triantafillou (2024) extended this framework by evaluating additional memorization proxies for sample partitioning, Both of these studies are limited to image classification tasks. In the context of LLMs, Barbulescu & Triantafillou (2024) proposed dynamic unlearning approach that iteratively selects high-memorization samples from the forget set in each epoch. While these efforts highlight the role of data quality or ordering in unlearning, they do not examine the existence of coreset.
RUM, a meta-unlearning algorithm that partitions
Coreset selection. Coreset selection, developed mainly for classification using computer vision models, refers to the process of selecting a subset of the training data, which can maximize the model accuracy on said task when trained using such subset. One class of coreset selection methods typically assign importance scores to each sample in the dataset based on various metrics computed during training dynamics such as gradient norm (Paul et al., 2021), 12 norm of error vectors (Paul et al., 2021), forgetting score (Toneva et al., 2019), data loss (Welling, 2009; Pruthi et al., 2020) and prediction confidence (Pleiss et al., 2020). Additionally some methods also focus on data diversity, based on distances of datapoints from their class clusters (Xia et al., 2022) and coverage of the training set (Zheng et al., 2022; Maharana al., 2024). Another class of methods are optimization based approaches for coreset selection mainly focusing on gradient information of data samples (Mirzasoleiman et al., 2020; Killamsetty et al., 2021a;b; Pooladzandi et al., 2022).
For language models, alongside loss-based scoring (Azeemi et al., 2023), various classic scoring-based coreset selection methods (mentioned above) have been explored for down- stream finetuning tasks where a smaller model is used for efficient scoring (Zhang et al., 2025). Additionally, coreset selection methods have been developed for instruction-tuning where the influence of each training sample is computed using gradient similarity with sam- ples in the validation set (Xia et al., 2024) and using LLM-as-a-Judge (Chen et al., 2024; Liu et al., 2023). Finally, Zhou et al. (2023) demonstrated superior performance LLM-alignment by finetuning on an extremely small set of manually curated high quality dataset.
To the best of our knowledge, Patil et al. (2025) concurrently explores coreset selection in LLM unlearning. Using anomaly detection on the hidden state representations of the forget set, the authors prune about 10-30% of the forget set to determine the coreset with the primary goal of utility preservation. However, our work differs from this work in two key aspects: (1) Their study did not identify the strong coreset effect embedded in existing LLM unlearning benchmarks, nor did it characterize its conditions (e.g., the relationship between coreset effect and unlearning training time); And (2) we go further by investigating both the "what" and "why" behind the coreset effect, focusing on extreme pruning ratios (90-99%), i.e., extreme low-data regimes (1-10%), while preserving unlearning and utility performance, and analyzing its impact on unlearning robustness.
3

---

Published as a conference paper at COLM 2025
3 Preliminaries and Motivation on Coreset Effect in LLM Unlearning
LLM unlearning setup. LLM unlearning aims to remove undesired data, knowledge, or model influence from a pretrained LLM, while preserving its general utility. This is typically framed as a model fine-tuning or editing task, given access to a forget set Dr, which contains the data or knowledge to be unlearned, and retain set D,, which is typically unrelated to the unlearning target and serves to help preserve the model's overall utility post-unlearning. Therefore, the problem of LLM unlearning is cast the following regularized optimization framework (Liu et al., 2025; Yao et al., 2024; Maini et al., 2024),
min E (xy)) (4 x; 8) + A A. (xy)) (y)x;))
where 0 denotes the model parameters to be updated during unlearning (initialized from the pretrained model state), Li and L, denote the forget loss (i.e., the unlearning objective) and retain loss (i.e,, the utility-preserving objective), respectively, evaluated when using 0 to generate response y given input x, and À > 0 is regularization parameter to strike the balance between the forget and retain objectives.
Representative LLM unlearning methods considered in this study. The specifics of the unlearning optimization in (1) correspond to different LLM unlearning approaches, typically depending on how the forget loss 4 is formulated and implemented. Although numerous unlearning methods have been proposed in the literature, our work focuses on two widely used and high-performing LLM unlearning approaches, negative preference optimization (NPO) (Zhang al., 2024a) and representation misdirection unlearning (RMU) (Li al., 2024), where NPO typically excels at data-wise unlearning, demonstrated on the MUSE benchmark (Shi et al., 2024b), and RMU is particularly effective for knowledge unlearning, as evidenced by its strong performance is adapted from direct preference optimization (DPO) (Rafailov et al., 2023), but differs by treating the forget data in D3 negative examples, excluding any positive examples. This yields a preference-base based forget loss & that drives the unlearned model 0 to deviate from the original, pre-trained model (referred to the reference model in NPO) on the forget set Dr. The retain loss l in NPO is typically chosen as the standard cross-entropy next-token prediction loss on the retain set D,. In contrast to NPO, RMU is built upon a random feature-based unlearning objective (Li et al., 2024; Fan et al., 2024a). The forget loss & in RMU promotes unlearning by mapping the intermediate-layer representations of the forget data to random features, thereby disrupting the model's ability to generate or reconstruct the forgotten information. In a similar spirit, the retain loss e, enforces alignment between the representations of the unlearned model and the pre-unlearned (reference) model on the retain dataset, helping to preserve the model's general utility. We refer readers to Appendix A for the detailed formulations of NPO and RMU.
the WMDP benchmark (Li et al., 2024). NPO
Unlearning benchmarks considered in this study. We conduct and evaluate LLM un- learning using two benchmarks: WMDP (Li et al., 2024), which assesses the unlearning of potentially hazardous knowledge in domains such as biology (called WMDF-Bio) and cybersecurity (called WMDP-Cyber), and MUSE (Shi et al., 2024b), which focuses on learning textual content from Harry Potter books (named MUSE-Books) and news articles (named MUSE-News). We select these two benchmarks as the former is the representative knowledge unlearning benchmark and the latter is designed to additionally support data unlearning (egg,, to prevent verbatim memorization and privacy leakage). It is worth noting that other unlearning benchmarks also exist, such as TOFU (Maini et al., 2024). However, the evaluation of unlearning effectiveness in TOFU relies on a p-value metric, which can be insensitive in distinguishing unlearning performance when the p-value exceeds 0.1, making less effective for fine-grained comparisons between unlearning methods.
In WMDP (Li et al., 2024), the goal of unlearning LLM is to reduce its accuracy on the WMDP evaluation set, which comprises QA pairs with undesirable answers related to harmful knowledge targeted for removal. Accordingly, we measure unlearning effectiveness (UE) as: UE = 100% - (Accuracy on WMDP evaluation set). Since the associated QA pairs are multiple-choice questions with 4 choices, perfect unlearning process would achieve an UE of 75%. The utility (UT) of the WMDP-unlearned model is measured by the zero-shot accuracy the MMLU dataset (Hendrycks et 2020).
4

---

Published as a conference paper at COLM 2025
In MUSE (Shi et al., 2024b), UE is measured using different metrics: (1) Verbatim memorization (VerbMem) on the forget set Dj reflects the model's s ability to perform next-token prediction for completing the forgotten data records. (2) Knowleedge memorization (KnowMem) reflects the model's 's ability to answer questions involving undesired knowledge in MUSE. Thus, lower VerbMem (or KnowMem) indicates better UE, as it implies reduced model generation capability for the targeted data (or knowledge) removal. Besides VerbMem and KnowMem, UE in MUSE is also evaluated using (3) privacy leakage (PrivLeak), which assesses the extent to which the unlearndd model leaks membership information, i.e., whether it reveals that data in Dj was part of the original training set. PrivLeak values approaching zero indicate better unlearning- UT of the unlearned model is measured by KnowMem on MUSE's retain set D,, reflecting the model's ability to preserve useful knowledge unrelated to unlearning.
Furthermore, we note that when implementing NPO and RMU to solve problem (1) for WMDP and MUSE, the number of training epochs is typically kept small. This is because the forget loss encourages divergence from the pretrained model, and prolonged training may lead to model collapse or degradation of general utility. For example, the number of unlearning epochs for RMU is set to 1 for WMDP and MUSE-Books. We refer the readers to Appendix B for a detailed discussion of the unlearning settings across different benchmarks.
Problem of interest: "Coreset" effect in LLM unlearning under low-data regimes. In current unlearning setups, a key pre-condition is the assumption of access to the forget set D,, which is typically provided priori in current benchmarks. However, very few studies have investigated the problem of LLM unlearning in a low-data regime, i.e., when the size of the forget set D; is limited or small. Therefore, the primary problem of interest in this work is to explore whether a "coreset" -like effect exists in LLM unlearning, thereby enabling effective unlearning in low-data regimes of current benchmarks.
50 32
WWMDP Bie)
220
60
57
Epochs
30
50
5
32%
55
3104
33 Epachs
100
5%
100%
30 Coreset Ratio (log)
Forget Set Selection Ratio
(a) UE vs unlearning epoch #
(b) UT vs. coreset size
(c) A "scaling law"
Figure 1: Unveiling the "coreset"-lik" like effect in LLM unlearning on WMDP using the RMU method, applied to the pre-trained LLM Zephyr-7BBB The "coreset" is randomly sampled from the full forget set (D; = WMDP-Bio), with selection ratios of 1%, 5%, and 10%; the 100% setting corresponds to using the entire D, Unlearning performance is averaged over 5 random trials. (a) The "coreset" achieves comparable UE (unlearning effectiveness) for RMU on WMDPPBio, especially when unlearning is performed with longer training epochs. The default number of unlearning epochs is for RMU under the full Dj (100%), as indicated by the shaded region. (b) MMLU-based UT (utility) the unlearned model against the forget set selection ratio. Each box plot represents the UT performance of the unlearned model across the range of unlearning epochs shown in (a) for 5 random trials. (c) A linear "scaling law" in the log-domain between the unlearning epochs and the forget set selection ratio for lossless unlearning.
Through the motivating results in Fig. 1, we demonstrate the existence of a coreset effect- where a small random subset of the original forget set has been able to achieve comparable test-time unlearning performance, especially when unlearning is trained for extended epochs. To be specific, Fig. presents the unlearning performance in terms of both UE (unlearning effectiveness) and UT (utility measured on MMLU), using the RMU unlearning method applied to the pre-trained LLM Zephyr-7B-$ on the WMDP benchmark. The performance is evaluated across different random data selection ratios (1%, 5%, and 10%) with respect to the full forget set WMDP-Bio. Fig. ((a) shows that an unlearned model trained on a small subset of the forget set (e.g,, as little as 5% of the full set) can achieve UE almost the same as that of using the entire forget set (100%). However, this coreset effect only emerges when the training is extended beyond the RMU's default 1-epoch setting on WMDP; For example, using only 1% of the forget set requires over 70 training epochs to achieve comparable UE. In addition, the observed effectiveness of coreset unlearning does not come at the cost of UT. This is evidenced by Fig. 1(b), where the unlearning optimization (via
5

---

Published as a conference paper at COLM 2025
RMU) maintains UT on MMLU across varying unlearning epochs (indicated by the box regions) and coreset ratios, in comparison to the model unlearned using the 100% forget set. In this case of lossless unlearning with coreset (í.e. with UE and UT comparable to the full forget set), we demonstrate an unlearning scaling law in Fig. 1(c). This shows how we need to gradually increase the number of unlearning epochs as the coreset ratio decreases.
The above motivating results provide strong evidence for a coreset effect in LLM unlearn- ing, showing that with sufficient training, small randomly selected subsets can achieve comparable UE and UT to full forget set-based unlearning. This finding prompts several key questions: Does the effect hold or even improve with non-random coreset selection? What underlying fuctors drive this phenomenon? And are coreset-umleare models as faithful and robust as their full-set counterparts? We investigate these questions the following sections.
4 On the Consistency and Rationale Behind Forget Coreset Effect
In this section, we present comprehensive study of the coreset effect in LLM unlearning across various dimensions, including different unlearning approaches (RMU and NPO), benchmarks (WMDP and MUSE), and coreset selection methods. Furthermore, we explore possible explanation for this effect through the lens of keywords extracted from the coreset, which may serve as a key driving force behind effective unlearning.
Surprisingly strong and consistent performance of random coreset selection. Recall from our motivating results on WMDP-Bio unlearning using RMU in Fig. 1, that coresets formed via random data selection already demonstrate lossless UE while maintaining UT. To further validate the effectiveness of random selection (termed RANDOM), we investigate another two key aspects: (1) random coreset unlearning across different unlearning methods and benchmarks, and (2) comparison with other more sophisticated coreset selection methods.
60
5.0
5 60
(MMLLU)
440
IMMUID
(MMLLU
40
UE (WMDDP-BO BBo)
UE (WMDP Cyberi
(WMDD--Bs)
UE (WBDD Cyberi
0%
1%
50%
10% 100%
066
1%
5%
10% 100%
ON 1%
5% 10% 100%
0%
1%
5%
10%
100%
Coreset Selection Ratio
Coreset Selection Ratio
Coreset Selection Ratio
Coreset Selection Ratio
(a) RMU, WMDP-BiC
(b) RMU, WMDP-Cyber
(c) NPO, WMDP-Bio
(d) NPO, WMDP Cyber
100
UT (Boomnnn UB (Verbmren)
200
75
UE /Ur
(Vockmann
5.50
(Vezbauus)
50
(Verhmeni)
50
50
225
UE
25
90% Coreset Selection Ratio
20%
100%
10% 100%
100%
Coreset Selection 10776
Coreset Selection Ratio
Coreset Selection
(e) RMU, MUSE-Booke Books
(f) RMU, MUSE-NewS
(g) NPO, MUSE-BoOD Books
(h) NPO, MUSE-News
Consistent RANDOM-based coreset unlearning performance in terms of UT and UE across against the coreset selection ratio. The performance is averaged over independent trials for random coreset selection, with variance indicated by the shaded regions. (a)-(d) correspond to the results of applying specific unlearning method (RMU or NPO) to benchmark dataset (WMDP-Bio, WMDP- Cyber, MUSE-Books or MUSE-News). Following the benchmark setting, unlearning is performed using Zephyr-7B-$ on WMDP, LLaMA2-7B on MÜSE-NewS, and ICLM-7B on MUSE-Books.
Figure 2:
Fig. 2 shows the consistent RANDOM-enabled coreset effect in LLM unlearning using RMU and NPO across four datasets: WMDP-Bio, WMDP-Cyber, MUSE-Books, and MUSE-News. The coreset selection ratio ranges from 0% to 100%, where 0% corresponds to the original (pre-unlearning) model, and 100% represents standard unlearning using the entire forget set. In Fig. (dd), which shows WMDP-based unlearning, we observe that both RMU and NPO exhibit clear coreset effect, evidenced by lossless UE and preserved UT as the coreset ratio increases comparable to that achieved with the full forget set (100%). Notably, RMU appears more effective in the low-data regime, achieving strong performance with as little as 1% of the forget set, whereas NPO requires around 5% to reach similar results in WMDP. As shown for buse-based unlearning in Fig. 2(e)-(h), both RMU and NPO exhibit consistent coreset effect in the context of data unlearning, as reflected by reductions in verbatim memorization. Notably, UT, measured by KnowMem, even improves when using smaller coreset size; see Fig. 2(e).
6

---

Published as a conference paper at COLM 2025
Next, compare the unlearning performance of RANDOM-based coreset selection with three additional, more sophisticated coreset selection methods, GRAND (Paul et al., 2021), MODERATE (Xia et al., 2022), and MIN-K% PROB (Shi et al.,, 2024a). The first two methods are classic coreset selection techniques adapted for LLMs, where GRAND ranks forget samples based on the gradient norm of the unlearning objective in (1), and MODERATE clusters forget samples based on their associated deep representations, then ranks them by their distance to the cluster center. MIN-K% PROB was originally developed to determine whether a given text appears in the original pretraining dataset, producing data memorization score, where a higher value indicates stronger memorization of the data. In our setting, we select the unlearning coreset based on the top memorization scores. We refer readers to Appendix C for details in the above coreset selection methods.
Table 1: Coreset unlearning performance (UE and UT, consistent with Fig. 2) using RMU and NPO on WMDP-Bio and WMDP-Cyber, evaluated using Zephyr-7B-6 across varying coreset selection ratios (0%, 5%, 10%, 100%) and selection methods (RANDOM, GRAND, MODERATE, MIN-K% PROB). Here 0% and 100% refer the pre-unlearning case (w/o using any forget data) and the standard unlearning RANDOM-bases selection is reported in the form a + b, where a is the mean and is the standard deviation, computed over 5 independent trials. The performance of other data selection metrics are reported using the mean over 2 trials.
case (w/ the full forget set), respectively. The performance
Coreset
Unlearning Melbod
RMU on WMDP-Bie
NPO on WMDP-Bie
RMU on WMDD-Cyber
NPO cn WMDP-Cyber
Ratio
(E(")
UT (MMUU) (1)
(E)
UT (MML)) (T)
(E(1)
UT (MMLU)
UE())
UT (MMUU) (?)
unlearning,
56.48
58:48
36,51
58.48
100%
Full fongetset
57.48
73.57
57.85
RAMDOM
34
56
73.96.004
6821
GRAND
71
56.96
71:54
40
57.25
67
MORBHATE
75.92
52.56
57,20
MIN-K% PROB
70.61
56,86
52.71
73,91
57.12
69.19
57.00
RANDOM
56
1.33
14.12
$7.89
65.23
5%
MOUSHATE
GrAND
22.39
73.78
$6,90
56.78
56.18
55
57.64
57.52
70.91
35.67
57.58
Table presents the coreset unlearning performance across different coreset selection meth- ods on WMDP. As we can see, the coreset effect remains robust across different coreset selection methods. Although random selection introduces performance variance across trials due to differing coreset realizations, more sophisticated selection strategies typically offer only marginal improvements in UE or UT over RANDOM, and these gains generally fall within the variance observed under RANDOM. Notably, compared to standard unlearning (i.e., using the full 100% forget set), 5% coreset unlearning can consistently achieve lossless UE and UT, demonstrating its effectiveness even in low-data regimes. This consistent performance is also observed on the MUSE benchmark, as shown in Table D1.
Explaining the sufficiency of coreset unlearning: A keyword perspective. To explain the consistently strong coreset-like effect, analyze the unlearning behavior of RANDOM-based coreset selection from a keyword per- spective. Given coreset, we extract its corresponding keyword set by leveraging L.LMaas-a-judge (OpenAT ol (Jaech et al., 2024)), guided by our proposed prompt (de- tailed in Appendix E), to identify words most relevant to the targeted unlearning concept or knowledge (egg,, biosecurity). We then perform unlearning on this key- word set assess whether the majority of the unlearning efficacy can be attributed to this significantly smaller sub- set, in comparison to unlearning conducted on the full coreset (from which the keywords were extracted) and the original full forget set D,.
70
600
50
UE (Original coreset)
(Orig ingi coreset)
40
UE (Keyword subset)
UT (Keyward subset)
10%
Coreset Selection Ratio
Unlearning performance (UE and UT) using the original core- set and its keyword subset across varying coreset selection ratios for RMU-based unlearning on (WMDP- Bio, Zephyr-7B-f).
Figure 3:
Table 2 presents representative keyword examples and compares the generation of un- learned models across three different forget sets: the full D,, a 5% coreset, and a keyword- only subset extracted from the coreset. The keyword set is constructed by filtering out non-ke non-keyword tokens and preserving the original order, as highlighted in the first column. As we can see, unlearning with just the keyword set remains highly effective, comparable to
7

---

Published as a conference paper at COLM 2025
Table 2: Generation of different unlearned models for the sentence completion task using part of the forget set WMDP-Bi as input. Keywords are marked light cyan within the forget sample. The unlearning task and setting follow the same configuration as in Fig. 2(a).
Forget semple free oWMEe--id wt estractd leynonds
Full Forget Set FM Unlermed Goneration
Corcset (9% of Unieemal Gonerttoon
Keywoude Uidboreed
The now pattagzn tolased troe ureo outuone coll
la VauaMaans dhundd ju "Baiin Roeegtaiien mes pastuse sall uss
'seretizeel wore Your" lea "eem" Sperile Iniee penel Di Sorier
seeyy previvally
Leedi
nicies, of P.l. trayiia
coeprodiee eatestedong
(ISIE)
which procide
lnd Villinn' RRMMtt
jollor athee piaiiien 'piitarin joto buriima printe
roddenee aparat thap panciltn cotended opaitrm apphaooponta,
and mondlattoss Those ESR prolicing bockee are asocdase with
gaz diboay
Spondeenettiinn there loss, tidoaecee ane unieet inande Suaveryall thequatii 8e ta lix df lind. crevenatil uesieral
whh kinos,, and. and, wank wett,, and. wort weme semr semoo mene wenn seeto
ban odurandly totedio vasmm dobsery sydem macer pro dinsal cleiad
Aithe Let more parch inizeeaaaa
of infecton themes oxhiiing mondes.
'odk wil in well your yansori unyst
bepattle-,,
repizitrr
18A8S), haman inrrinttiiiee stnce (1899-) J molarbo, sibectoce and intrerze. There in horh tspec of Ad vecior that so being uttined tor gine dilierry appitaation. The tret type or Ad vectier...
twippins wondisitheee, Lii lnieen//en hose joig Cunal your ya yerhhggs
tather underaarriiirr
the full and pereset-based unlearning, as the model fails to recall the target knowledge and instead generates nonsensical responses to forget queries.
For thorough quantitative evaluation, Fig. 3 compares the unlearning performance (in both UE and UT) of using the keyword-only coreset and the original coreset against the coreset selection ratio on WMDP-Bio, following the same unlearning task setting as in Fig. 2(a). We perform 300 epochs of unlearning for the keyword-only forget sets. As observed, although unlearning with the keyword-only forget sets does not fully match the UE of the full coreset, it captures the majority of the unlearning effect compared to the pre-unlearning baseline at 0% coreset selection ratio. The contribution of using keyword unlearning is consistent against the coreset selection ratio. This suggests that the coreset effect may be driven by small number of high-impact keywords within the coreset, which are sufficient to account for most of the unlearning performance. This, in turn, points to a surprising simplicity in the current unlearning benchmark tasks or their evaluation protocols. Additional analysis on keyword overlap between the coreset and full forget set is provided in Appendix F.
On the Faithfulness of LLM Unlearning using Coresets
In this section, we investigate the quality and faithfulness of coreset-based LLM unlearning from additional perspectives: (1) mode connectivity, which reflects the similarity of the unlearning loss landscape to that of full forget set-based unlearning; (2) the robustness of coreset-unlearned models against both input-based jailbreaking attacks and weight-based model fine-tuning post unlearning: and (3) additional utilities of unlearned LLMs beyond those captured by existing unlearning benchmarks. Unless otherwise specified, in this section we focus on LLM unlearning on WMDP using RANDOM-based coreset selection, following the settings used in 4.
203
103
100
40
Coriset (dlo
Carcsct (Q0. 10%
Fa Coresec
40
20
Coresit 5%
20
Careset
20
0.318,)
0.2 U.A. 0.6 0.5 1011g
0.066)
Interpolation 1.015u)
2.4 0.
0.055u 0.2 0u4
2.5 1.0%g))
0.018,) 0.2 0.4 de
08 1015gl
Interpalation
Interpolation
'nterpolation
(a) RMU, WMDP-BB (b) RMU, WMDP-Cyber
(c) NPO, WMDD-Bio (d) NPO, WMDP-Cyber
Figure 4: LMC holds between poreset-unlearned model (Qcu) and the full forget set-unlearned model (0Ru), as evidenced by UE against the interpolation coefficient (x-axis). Here the coreset-unleameed models are obtained using RANDOM-based coresets with the same setting as in Fig. 2(a-d).
Mode connectivity: Coreset unlearning is as good as full forget set unlearning. Mode connectivity refers to the phenomenon where the minima found by two optimized ML models are connected by a path along which the model error does not increase, suggesting that the models reside in shared or smoothly connected region of the loss landscape (Draxler et al., 2018; Freeman& & Bruna, 2016; Qin et al., 2022; Garipov et al., 2018). A strong form of connectivity is known linear mode connectivity (LMC), where the interpolation path between two models is constrained to be linear in parameter space (Frankle et al., 2020). Therefore, we leverage LMC to investigate the similarity or potential discrepancy
8

---

Published as a conference paper at COLM 2025
between coreset-unlearned model (0cu) and the full forget set-unlearned model (0ku). LMC for coreset unlearning is said to hold if the unlearning evaluation (e.g,, via UE) of the
linearly interpolated model 0(a): := (a+.. + (1 a a)))u) remains approximately consistent as the interpolation coefficient a E [0, 1] varies, where 0(1) = deu and 8(0) = Q yield the endpoints of the linear path. Fig. 4 illustrates UE of the interpolated model 0(a) against the interpolation coefficient a at forget coreset selection ratios of 1%, 5,, and 10%. As we can see, the UE remains approximately constant along the linear interpolation path between Guu and fur indicating a strong LMC between the two models (nearly perfect connectivity for 5% and 10% coresets). This provides strong evidence that the coreset-unlearned model resides in the same optimal basin as the full forget set-unlearned model, supporting the faithfulness of unlearning achieved via coresets.
Adversarial robustness against jailbreak- ing attacks. The lack of robustness in LLM unlearning has been highlighted by their vulnerability to jailbreaking attacks (Lucki et al., 2024; Lynch et al., 2024; Patil et al., 2024). Therefore, we investigate whether coreset-based unlearning introduces addi- tional robustness limitations compared to unlearning with the full forget set. Table 3 presents the UE of coreset-unlearned mod- els, alongside the full forget set-unlearned
Table 3: Robustness of models unlearned using RA RANDOM coreset selection on WMDP-BI -Bio using RMU under Zephyr-7B-f, following the setting in Fig. 2(a). Robustness measured using the UE reduction after enhanced GCG attack.
Coreset Ratio
UE
UE reduction
Before Attack
After Attack
After Attack
100%
69.46
47.71
21.75
10 10%
72.43
2002
5
72.03
51.29
20.74
model (i.e., coreset ratio 100%), under input-level jailbreaking attacks using the enhanced- GCG method (Lucki et al., 2024), which generates adversarial prompt prefixes to elicit forgotten information. A lower UE after the attack indicates a more successful recovery of forgotten content. As shown, the coreset-unleared models experience a similar degree of UE reduction as the full-set counterpart, demonstrating comparable robustness. These results further support the faithfulness and reliability of coreset- unlearning.
Robustness against downstream fine-tuning. Beyond jailbreaking at- tacks, unlearning robustness has also been studied in model fine-tuning, where forgotten knowledge can resur- face in unrelated downstream tasks (Hu et al., 2024; Deeb & Roger, 2024; Lo et al., 2024), similar to such vul- nerability of safety-aligned LLMs (Qi et al., 2023). We finetune models un- learned on WMDP using NPO using GSM8k (Cobbe et al., 2021) and AG- News (Zhang et al., 2015) using 600 samples from the finetuning dataset. In 5, we demonstrate UE of un- learned models subjected to such fine- tuning. As expected, the models ex- hibit 'elearning'' of unlearned infor- mation. However, in certain scenar- ios, such as WMDP-Bio unlearned models finetuned using GSMSk in Fig. 5(a), the speed and extent of relearning can clearly distinguish coreset-unlearned models from those unlearned using the full forget set. These results highlight potential downside of coreset unlearning: using a smaller forgel sel may compromise robustness to downstream fine-tuning compared to full-set unlearning. This likely be- cause, when the relearn set is larger than coreset, the influence of relearning can overpower the unlearning.
75
foree set
60
Sirget
Comsed 3%
Consdd 9%
Fineturing Samples
24
400
900
Ainetuning Samples
24
64
403
(a) GSM8k, WMDP-Bio
(b) AGNews, WMDP-Bio
Full forget set Comeant ratio
Ful Segeetit sut Consect
Finetunlnng Samples
Finetuning Samples
(d) GSMSK, WMDP-Cyber (e) AGNews, WMDP-Cyber
Figure 5: Unlearning performance (UE) of RANDOM- coreset unlearned models (using NPO under Zephyr-7B- ()) against the number of fine-tuning samples. (a)-(f) Re- learning using finetuning datasets (GSM8K, AGNews) for models unlearned on WMDP-Bio or WMDP- The performance is averaged over independent trials.
In Appendix G, we also show that coreset-unlearned models retain utility auxiliary tasks such as math addition/ addition/subtraciinn subtraction and TruthfulQA.

---

Published as a conference paper at COLM 2025
6 Conclusion and Discussion
We establish a novel coreset effect in LLM unlearning, where an extremely small subset of the forget set achieve lossless unlearning. This coreset effect particularly emerges when conditioned on sufficient unlearning training, And this effect is shown to hold consistently across benchmarks (WMDP, MUSE) and unlearning methods (RMU, NPO). We further explain and validate this phenomenon from multiple perspectives, including keyword analysis, mode connectivity, and robustness. While we establish this effect, we observe that many classic heuristic-based coreset selection methods do not outperform simple RA NDMD selection. Therefore, determining the 'optimality' of random selection or identifying optimized, improved coresets remains a critical direction for future research. Our keyword analysis also highlights the need for deeper investigation into the underlying mechanisms of unlearning, potentially through more advanced interpretability techniques. In other words, unlearning could be driven by salient tokens within the forget data, rather than relying on the entire forget dataset. Additionally, the strong coreset effect observed in this work suggests potential 'redundancy' within the forget sets used in current benchmarks. We hypothesize that this effect arises from inherent 'correlations' among forget data points. If the forget examples were independently and identically distributed, the coreset effect might diminish significantly. This correlation-driven redundancy could make unlearning tasks easier than expected under current ev evaluation protocols. We advocate for future benchmarks to incorporate the influence forget dataset size and richer evaluation methods to better reflect realistic unlearning challenges.
Ethics Statement
The ethical implications of our work on identifying the coreset effect and developing coreset- based unlearming methods lie in promoting responsible data handling and the safe use of large language models (LLMs). Our research advances machine unlearning techniques aimed strengthening privacy protections, reducing sociotechnical harms, and ensuring more controlled, trustworthy generative AT. At the same time, we recognize that the societal impact of unlearning nuanced and continually evolving. This underscores the importance of ongoing reflection, transparent evaluation, and active engagement with the broader discourse on Al ethics and governance. On the other hand, it is equally important to prevent the misuse of unlearning techniques to erase safe, beneficial, or ethically important information from LLMs, which could compromise model integrity and safety.
Acknowledgments
We thank the U.S. Department of Energy via Lawrence Livermore National Laboratory (LLNL) under Contract DE-AC52-07NA 0722444227344 and the LINII LDRD Program under Project
No. 23-ER-030 for their support (LLNL-IRN--000466)) Soumyadeep Pal, Changsheng Wang, and Sijia Liu were also partially supported by the National Science Foundation (NSF) CISE Core Program Award (IS-2207052),, the NSF CAREER Award (IIS-2338068), the ARO Award (W911NF2310343) and the Amazon Research Award for AI in Information Security. We also thank Dr. Alfred O. Hero for his invaluable suggestions, which has helped us improve our work.
References
Abdul Azeemi, Ihsan Qazi, and Agha Raza. Data pruning for efficient model pruning in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 236-246, 2023.
George-Octavian Barbulescu and Peter Triantafillou. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arv preprint arXin:2405.03007, 2024.
10

---

Published as a conference paper at COLM 2025
Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, et al. Open problems in machine unlearning for ai safety, ar Xiviv preprint arXi::2501.04952, 2025.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33: 14879-14890, 2020.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), Pp. 141-159, IEEE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, www.foormmmmiddddsssa
2024. URL https: / /opnrreviie
Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, and Sijia Liu. Safety mirage: How spurious correlations undermine vlm safety fine-tuning. aiviv preprint ar arXiv:2503.11832, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ar preprint ar Xiv:220 14168, 2021.
A Feder Cooper, Christopher A Choquette-Choo Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloo- far Mireshghallah, et al. Machine unlearning doesn't do what you think: Lessons for generative ai policy, research, and practice. ar preprint ar axXto:2412.06666, 2024.
Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? Xiv preprint ar arXio:2410.08827,, 2024.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, Pp. 1309-1318. PMLR, 2018
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in lims, 2023.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning for LLMs, 2024. URL https: http:////eneeeeeeeeeeeerorrreeda www.et/foorrmiidddecctwwww.
Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst-case forget sets in machine unlearning. In European Conference on Computer Vision, Pp. 278-297. Springer, 2024a.
Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for Ilm unlearning. arXiv preprint arXio:2410.07163, 2024b.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269. PMLR, 2020.
C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. Xiu preprint ar XxXi:11611.0144 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
11

---

Published as a conference paper at COLM 2025
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-bass -based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36:17643-17668, 2023,
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint ar arXiv:2009.0333 2020.
Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Jogging the memory of unlearned models through targeted relearning attacks. In ICML 2024 Workshop on Foundation Models in the Wild, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. Xiv preprint ar XrXio:2212.0408, 2022.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008-2016. PMLR, 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. Xiv preprint ar Xrx::24412.1677 2024.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. ar arxiv preprint ar xrXiv:2210.01504, 2022.
Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. WAGLE: Strategic weight attribution for effective and modular unlearning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for Ilm unlearning. Xiv preprint ar arXiv:2404. 18239, 2024b.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Tyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Tyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118, 2021b.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint ar arX1o:2403.03218, 2024.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1-14, 2025.
12

---

Published as a conference paper at COLM 2025
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXio:2312.15685, 2023.
Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Revisiting who's harry potter: Towards targeted unlearning from a causal intervention perspective. arXiv preprint ar axXim:2407.6999, 2024.
Michelle Lo, Shay Cohen, and Fazl Barez. Large language models relearn removed concepts. arXiv preprint arXiv:2401.01814, 2024.
Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint axXio:2409.18025, 2024.
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadeield-Menell. Eight methods to evaluate robust unlearning in Ilms. arXiv preprint arXio:2402.16835, 2024.
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. $) (D) pruning: Message passing for balancing diversity & difficulty in data pruning. In The Twelfth International Conference on Learning Representations, 2024. URL httss://ppenrevie.nee//oooo thbtoAkCe9.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Infernational Conference on Machine Learning, pp. 6950-6960. PMLR, 2020.
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from Ilms? objectives for defending against extraction attacks. ICLR, 2024.
Vaidehi Patil, Elias Stengel-Eskin, and Mohit Bansal. Upcore: Utility-preservvie coreset selection for balanced unlearning. ar arv preprint ar Xiii:2502..1508, 2025.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet:
Finding important examples early in training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// httpp:////eenrrieee.e.//////u/// wtt.:///ooouumm/d/370--------
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models few shot unlearners. arXiv preprint ar arXio:2310.07579, 2023.
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances Neural Information Processing Systems, 03:17044-17056, 2020.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning. In International Conference on Machine Learning, pp. 17848-17869. PMLR, 2022.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Hen- derson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXioo2310.03693, 2023.
13

---

Published as a conference paper at COLM 2025
Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Exploring mode connectivity for pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Languing Processing, Pp. 6726-677 6746, 2022.
Youyang Qu, Ming Ding, Nan Sun, Kanchana Thilakarathna, Tianqing Zhu, and Dusit Niyato. The frontier of data erasure: Machine unlearning for large language models. arXiv preprint arXtt:2403.15779 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 01553788-537444 2023.
Atakan Seyitoglu, Aleksei Kuvshinov, Leo Schwinn, and Stephan Günnemann. Extracting unlearned information from Ilms with activation steering. ar Xiv preprint arXiv:2411.02631, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Tweelfth International Conference on Learning Representations, 2024a. URL https://openrevieer net net./oormiiddzwerrrrrrraee
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arviv preprint arXiv:2407.06400 2024b.
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowl- edge unlearning for llms: Tasks, methods, and challenges. ar arv preprint arXiv:2311.15766, 2023.
Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, and Virginia Smith. Position: Llm unlearning benchmarks are weak measures of progress. arXiv preprint ar Xrxi::2410.0287, 2024a.
Pratiksha Thaker, Yash Maurya, and Virginia Smith. Guardrail baselines for unlearning in llms. Xiv preprint ar arXio:2403.03329, 2024b.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 303-319, IEEE, 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations 2019. URL https://openreviiie net/ d
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arviv preprint ar Xii::22660006222 2022.
Max Welling. Herding dynamical weights learn. In Proceedings of the 26th annual interna- tional conference on machine learning, pp. 1121-1128, 2009.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXio:2310.201 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arviv preprint ar arXiv:242..04333 2024.
14

---

Published as a conference paper at COLM 2025
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference Learning Representations, 2022.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 37:105425-105475, 2024.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse effective unlearning. arXiv preprint arXiv:2404.05866 2024a.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Chao Shen, Tianlin Li, Weipeng Jiang, and Yang Liu. STAFF: Speculative coreset selection for task-specific fine-tuning, In The Thirteenth International Conference Learning Representations, 2025. URL https://openrewwe net/
forum?)d==AfxvdviDy)
Yihua Zhang, Yimeng Zhang, Aochuan Chen, Jinghan Jia, Jiancheng Liu, Gaowen Liu, Mingyi Hong, Shiyu Chang, and Sijia Liu. Selectivity drives productivity: efficient dataset pruning for enhanced transfer learning. Advances in Neural Information Processing Systems, 36:36913-36937, 2023.
Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint ar Xxix::447.002855 2024b.
Kairan Zhao and Peter Triantafillou. Scalability of memorization--asee machine unlearning. Xiv preprint 1rX1::2410.15516, 2024.
Kairan Zhao, Meghdad Kurmanji, George-Octavian Bärbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. Adoances in Neural Information Processing Systems, 17:12293-12233, 2024.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. Xiv preprint ar Xit:2210.1588)) 2022.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55066-5002, 2023.
15

---

Published as a conference paper at COLM 2025
A Details of NPO and RMU
In this section, we provide details explaining the two main unlearning methods we consider in this paper.
Negative Preference Optimization (NPO). NPO (Zhang et al., 2024a) reduces the model's preference for the forget set D; by treating it analogously to negative responses in preference optimization, but omitting the positive response term. This yields the following loss function, which has been shown to achieve lower divergence rates compared to various other unlearning approaches:
2 (0) = E () logo (Blog
To(y(x) (y(x))
(A1)
4 specified in (1)
where (ff) = 1/((+e=)) is the sigmoid function, B > i a hyperparameter. Minimizing the above forget loss drives the model to be unlearndd 7i awayy from the reference model Tee for the samples from the forget set. In NPO, the retain loss ly simply the prediction loss i.e the cross entropy loss between and y, which are obtained from Dr-
Representation Misdirection Unlearning (RMU). RMU (Li al., 2024) seeks degrade the model's internal representations for the forget set, thereby reducing its ability to recall or generate undesired knowledge. Specifically, it aligns the hidden state activations of the unlearned model at a given layer (denoted TT/) with a random unit vector u, where each component of u is sampled independently and uniformly from [0, 1). In contrast to NPO, RMU also includes retain loss that ensures the model's ss representations the retain dataset remain consistent with those of the reference spre-unlearned, model. Considering 3f and 2, to be samples from the forget and retain set respectively,this is formalized as follows:
1
(0) = EzyED,
Σ
(TT(() - cc
+ATE,
Σ token 163,
(
Tie (t)l
(A2)
L
L
token 162,
4 specified (1)
f specified in (1)
where L,, L, are the number of tokens in 2y, 2, respectively and c controls the strength of
B Detailed Unlearning Setup
We present the detailed unlearning setups for different settings in Table B1. It is worth noting that the first 600 samples of WMDP-Bio and WMDP-Cyber are used as the full forget set following (Li et al., 2024). For NPO, perform unlearning for 140 steps with batch size of 4, which is approximately (140 x 4) 600 = 0.93 epochs.
Table B1: Full forget set unlearning settings across different unlearning methods and benchmarks.
Unlearning Benchmark
Unlearning Method
Model
Epochs
Learning Rate
WMDF-Bax/Cyler
RMU NPO
Zuphyr-78.8
Sr 5
0.9
7e- 6
MUSEF
RMU NPO
KCLM-78
1r-3 5
3
MUSE-News
RMU
NPO
11AMA2-78
10
1r-5
Unless otherwise mentioned, we follow the standard hyperparameters in the above bench- marks. For MUSE-Books (Shi et al., 2024b), the reference model ICLM-7B is finetuned on Harry Potter books, while for MUSE-News, the reference model LLaMA2-7B is finetuned on BBC News articles. These reference models are available in the MUSE benchmark.
As mentioned in 3, the coreset effect emerges with higher number of unlearning epochs. In Table B2, we present the epoch numbers for each coreset setting. All other unlearning settings are the same the full forget set unlearning.
16

---

Published as a conference paper at COLM 2025
Table B2: Coreset unlearning settings across different unlearning methods and benchmarks.
Unlearming Benchmark
Unlearning Method
Coreset Ratio
Epochs
10%
10
WMDyr-iooCyber
RMU
5%
20
1%
100
10%
9.33
WMDD--iee/yber
NPO
5%
18.67
1%
93.33
10%
10
MUSE-Books
RMU
5%
20
10%
7
MUSE-Books
NPO
5%
8
10%
100
MUSE-News
RMU
5%
200
10%
5
MUSE-News
NPO
5%
12
C Details of Coreset Selection Methods
In this section, we present additional details of the heuristic data selection methods as presented in Sec. 4. Though these methods were primarily developed for non-LLMs, adapt them to our unlearning process as detailed below.
The main rationale behind this method (Paul et al., 2021) that the importance of each sample zy from the forget set is captured by the expected gradient norm of the loss associated with that sample, where the expectation is taken over the unlearning trajectory. Thus the GRAND score is given by:
GRAND.
x(zy) Fa, Vo, (.) + A6r(2r; 0:)] where zy ~ Dj, 2, Dr
(CI)
Here 4 and l are specified in (1), where & changes according to the unlearning method as specified before. In experiments, we consider the trajectory of unlearning for 10 epochs for the purpose of computing the above expectation.
Moderate. The moderate coreset selection method (Xia et al., 2022) was developed in the classification setting where samples were divided into clusters according to their classes. Using a well-trained model, the class center of each class is calculated using the hidden state representations of the penultimate layer of the corresponding samples. Since our setting does not involve class labels, we cluster the forget samples into four groups using K-means, based on their penultimate-layer representations extracted from the reference (pre-unlearned) model. For each cluster, we compute its centroid and rank the samples by their distance to the respective centroid. To select representative data points, choose those whose distances are closest to the median within their cluster.
MIN-K% PROB. The MIN-K% PROB method (Shi et al., 2024a) developed to ascertain whether a given text appears in the original pretraining dataset. We use this metric as data selection method for unlearning under the notion that data points not encountered during training are less influential in the unlearning process. In fact, in (Shi et al., 2024a), the authors demonstrated that the MIN-K% metric can be used as an effective indicator for assessing removal of knowledge after unlearning.
For a given datapoint z, we calculate the log-likelihood of each token and choose a set of top 40%(KK = 40) of tokens with lowest value, called Min-K%(z). Then the score for is calculated as: MrN-K% PROB(z) = |Min-K Mine log (
(C2)
After computing the above scores for each sample z in D,, we choose samples with the top scores to obtain coresets. Thus, if we want to choose 5% coreset, then p 0.05 x [Dyl-
17

---

Published as a conference paper at COLM 2025
D Coreset Unlearning Performance on MUSE
Table D1 presents the coreset unlearning performance ofor MUSE across different coreset selection methods. Here we observe that performance of RANDOM-base coresets are highly comparable to the full forget set performance, while also strongly outperforming the full forget set in many instances. For example, we observe a significant utility (UT) improvement for MUSE-Bookss using RMU for coresets seen in Table D1, col.6 In fact, for MODERATE-based 10% coreset we see ~ 12% improvement in utility over the full forget set.
Additionally, observe that for MUSE-NewS, the heuristic based 5% coreset selection methods have UE (Knowmem) similar to that of the full forget set (as evidenced by Table D1, col.3). However, this is worse than the average performance of RANDOM-based selection. This coupled with the high variance of the performance RANDOM-based coresets points us to the fact that coreset selection non-trivial problem for such cases. Nevertheless, the performance of RANDOM-based coreset supports our strong coreset observation even at 5% coreset selection regime.
Table D1: Coreset-basee unlearning performance (UE and UT, consistent with Fig. 2) using RMU and NPO on MUSE evaluated using L.LaMA-7BB on News and MCLM on Books. The table is presented in the same format as Table 1. Here 'Retrain' refers to model finetuned only on MUSE excluding the forget set.
RMU
NPO
Convent Ratio
Unleariing Melhod
VerbMee
Privleak (-+0)
VerbMeem
KnowMem
(*)
MUSE-BBook
0%
No unlearning
99.56
58.32
-5632
67.1
99.56
58.32
-6.32
67.01
0%
Retrain
14.30
28,90
0.00
74.50
14.30
28.90
0.00
74.00
forpet
-16.8
4.34
23:80-1005
-15.9
56.67272)
000,000
0.0gg00
29.12.33
GRAND
-25
37
MODERATE
KS PROB
7.35
-29.77
36.03
0.00
-024
33.21
RANDOM
401.:
19 00:77.4
11861.1
0.00.000
348
-8.974333
36 19
3%
MODERATE
GRAND
5,69
6.75
28.00
7,08
-37.66
118.0
39.92
59.21
0.00
0.00
3,35
3.77
-15.7
-563
36.95
40.15
MIN-KN PROB
4.20
37.48
4.12
67.76
0.00
0,33
-32.00
39
MUSE-News
unlearnig
98.71
.9.71
Sorget
21.00
$4.8
1,50
4625
107.00
37
RANDOM
20 00.97.00
1885
-38 43
50.24..0
20.54
42 12.30.:3
.99 58
38
10%
MODERATE
GRAND
28.5
32.62
38.10
40.50
36.44
15.84
52.77
49.50
21.35
19.26
41.97
12.46
-99.7
-99.7
37 99
38.59
MIN-K' PROB
31.67
2.00
-15.55
50.14
23.59
43.85
-98.14
40.60
RANDOM
20.5.
12.95 17.77
116 1835
4053
7.36
34.25 :8:2
76..38.7.
32.75
GRAND
24.15
90.26
-33.27
47:11
351
32.71
-1,85
31
MOGERATE
19.37
52.21
.8.00
47.00
36.24
16.50
35
MIN-K%,
18

---

Published as a conference paper at COLM 2025
E
Prompt to Extract Keywords
Prompt to Extract Keywords using LLM-as-a-Judge
System:
You are expert in chemistry, biology and diseases. will provide you with text, and your task is to carefully analyze each word and determine if it is related to chemistry or biology and disease, For every word that is related to chemistry, biology and disease, replace it with an asterisk (*) in the original text while keeping all other words unchanged. Think step by step and ensure accuracy in identifying only chemistry, biology and discase-related terms. Pay close attention to scientific terminology. Return the result the following format: if
"replaced text"; "<text after replacing>", "related_wordss ("word1", "word2", "word3", ...]
}}
Do not write any code. Use your linguistic and scientific knowledge to analyze the text.
User:
{WMDP- forget set}
Assistant:
fresponse]
F Additional Details of Keyword Explanation Experiment
In addition to demonstrating the unlearning effectiveness of keyword-based unlearned models in 4ec.4, we further analyze the overlap between keywords extracted from the entire forget set (100%) and those obtained from RANDOM-based coresets. This comparison helps assess whether randomly selected coresets reliably capture the keywords in the full set. For particular coreset, let the keywords be denoted by Ke and let all the keywords extracted be
K. n K,
denoted Ky. Then for that coreset, we define the normalized overlap as
K, As seen in
Fig. F1, the overlapping keywords are higher than the fraction of data present in the coreset. his suggests that even information-agnosti random selection can yield a surprisingly high ratio of keyword overlap with the full forget set. For instance, 5% RANDOM- captures approximately 14% of the full set's keywords. We hypothesize that this amplified presence of high-impact keywords is key factor enabling effective unlearning, even when using such a small, randomly selected subset.
coreset
100
Normalized Overlap
Normalized Overlap
Forget
40
20
10
1510 20
50
80
100
Coreset Selection Ratio(%)
Figure Fl: Normalized overlapping ratio (%) of keywords between RANDOM-based coresets and the full forget set for WMDP-Bio. The dotted line represents the percentage of data present in the coreset from the forget set. Results are reported in the form + 2h, where a is the mean and b is the standard deviation, computed over 3 independent trials.
19

---

Published as a conference paper at COLM 2025
G Additional Utility Evaluation of Unlearned LLMs Using Coresets.
As shown by the coreset unlearning performance in standard benchmarks (Sec. 4), sur- prising finding is that coreset unlearning can achieve lossless UE. This naturally raises the question: Could coreset unlearning offer utility benefits as it negates the influence of fewer forget data points, thereby better preserving the model's performance on general tasks? However, we did not observe clear UT advantage in existing benchmarks, likely due to the limited scope of their utility evaluations, e.g,, MMLU for WMDP and KnowMem on the retain set for MUSE, which may not fully capture the broader utility landscape of the coreset-unlearned models.
Therefore, we conduct additional utility evaluations of coreset-unlearned models, focusing on two tasks inspired by the emergent abilities of LLMs (Wei et al., 2022): (1) a math ceaiitionnsubbrccttionn task involving 2-5 digit arithmetic that is orthogonal to the unlearning objective, and (2) the TruthfulQA task that assesses factual consistency and truthfulness in LLM responses and may be inadvertently affected by unlearning. In Fig. G2 report the zero-shot accuracy of the aforementioned utility metrics for RANDOM-based coreset- unlearned models using RMU and NPO on WMDP. RMU consistently maintains strong utility across all evaluations, regardless of the coreset selection ratio. In contrast, NPO shows higher variance; however, we observe that for n-digit addition and subtraction, performance may benefit from using 10% coreset in the WMDP-Cyber setting.
n-Diglt Subtraction
Digiitt
60
.PC
60
NPO
NPO
1%
10%
100%
10% 100%
1%
15%
100%
Coreset Selection Ratio
Coreset Selection Ratio
Coreset Selection Ratio
(a) Addition, WMDP-BAL
(b) Subtraction, WMDP-BB
(c) TruthfulQA, WMDP-Bio
n-Digit Addition
n.DigR Subtraction
80
TrennJii
50
60
40
40
20
NPO
NPO
NPO RMU
00%
1%
5%
10% 100%
00%
1%
5%
10% 100%
1%
5%
15% 100%
Coroset Selection Ratin
Cornsot Selection Ratio
Corosat Selection Ratio
(e) Addition, WMDP-Cyber
(f) Subtractionn WMDP-Cyber
(g) TruthfulQA, WMDP-Cyber
Figure G2: Additional utility evaluation performance for RANDOM-coreset unlearned models us- ing NPO and RMU under Zephyr-7B-B. (a)-(d) correspond to the performance of specific utility evaluation (Addition, Subtraction, Truthful QA) of models unleared using WMDP-Bio or WMDP- Cyber. Here' 'Addition' refers to n-digit addition (n=2,3,4,5) and 'Subtraction' refers n-digit subtraction (n=2,3,4,5), where the accuracies are averaged over n. The unlearning task and setting follow the same configuration as in Fig.2. 2.
20


Reviews:
[{"review_id": "HGWvhAapPj", "reviewer": "Reviewer_xsZY", "summary": "The paper probes whether unlearning a large language model truly requires the full forget set or whether a tiny “coreset” (≤ 5 %) can achieve comparable erasure. Across two hazardous‑knowledge and two memorization benchmarks, three 7‑billion‑parameter models, and two representative algorithms, the authors show that coreset‑based unlearning preserves utility while matching—or occasionally exceeding—full‑set performance on erasure metrics.\n\nThe manuscript is clearly written and logically organised. Its exploration of a coreset phenomenon in LLM unlearning is, to the best of my knowledge, novel, and the keyword‑level analysis supplies an intuitive token‑level explanation for the effect. The finding that current forget sets are highly redundant has practical significance: it suggests that reliable unlearning could be far cheaper than previously assumed in the unlearning community and could suggest redesigning unlearning benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Strong empirical evidence. The coreset effect is validated across two datasets, three models, two representative unlearning methods, and four coreset‑selection strategies, robustly supporting their main claim.\n- Novel and impactful result. Showing that only 5 % of the forget set can achieve comparable unlearning performance is both surprising and new, and is likely to have significant impact on the LLM‑unlearning community.", "reasons_to_reject": "- In Section 5, the linear‑mode‑connectivity analysis suggests that the optimization outcomes of the models are nearly identical, yet Table 3 shows divergent results for the downstream fine‑tuning‑based knowledge‑recovery attack. The paper does not explain this inconsistency.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vaFDegQT9s", "reviewer": "Reviewer_wji5", "summary": "The paper demonstrates that unlearning outcomes achieved with the full forget set can often be preserved using a randomly selected subset as small as 5%, indicating that current LLM unlearning tasks may be surprisingly easy in low-data regimes The paper attributes this coreset effect to a keyword-driven mechanism, where a small number of high-impact tokens dominate unlearning efficacy. The authors further validate the faithfulness of coreset-unlearned models using mode connectivity analysis and robustness to jailbreaking attacks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors identify and analyze the coreset effect in LLM unlearning and reveal a fundamental property of two benchmarks. The coreset effect demonstrates that effective unlearning can occur with extremely small subsets, highlighting opportunities for computational efficiency.\n\n2. The authors provide an explanation via token-level analysis, attributing unlearning success to high-impact keywords, which advances understanding of model behavior.", "reasons_to_reject": "The paper does not include experiments on the ToFU benchmark following SOUL [1] to evalute Forgetquality, Acc., Rouge-L, and MIA, which is essential for evaluating unlearning performance in language models. Including results on SOUL would strengthen the empirical validation and allow for more direct comparisons with recent unlearning methods.\n\n[1] Jia, Jinghan, et al. \"SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.", "questions_to_authors": "Could you evaluate your method on the ToFU benchmark, following the experimental setup used in SOUL? This would provide a more comprehensive assessment of SoTA method’s unlearning effectiveness.", "extra_scores": {}}, {"review_id": "TBE9Pz1wge", "reviewer": "Reviewer_sYCW", "summary": "The paper demonstrates that for two unlearning benchmarks, using just 5% of the training data (a randomly selected \"coreset\"), almost the same unlearning rate can be achieved, while better general LM capabilities (measured bh MMLU performance, or the retain set performance).", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Having more data efficient ways to achieve the same outcome is valuable.\n\nThe experiments in the paper are carried out competently. Two different datasets are used, with similar results.\n\nThe paper explains methods and background very well.", "reasons_to_reject": "Since no new methods are proposed for using just 5% of the training data, this experimental finding says more about the redundancy in the training data - something that is not discussed at all in the paper. So, when I checked these sets, I found that WMDP handles very specific biochemical weapons content, a very narrow domain, and unlearning for this domain should have little impact on the very different MMLU test set on which preservation of general LM capability is preserved. The examples shown in Table 2, on the other hand, show dramatic degradation of the LM's ability to generate text on the unlearning topic - basically producing word noise, not (what I would have expected) banal and harmless basic facts on the broader topic.\n\nThe MUSE data set's training data are the Harry Potter books - again something that is quite redundant when the goal is to not reproduce content from the book. Here is the dataset comes with a retain set of Fandom content - much closer to the content of the books (which is good), but unlearning also has a stronger degradation impact on the retrain set.\n\nOverall, I am left with the impression that the main finding of the paper is that if you have highly redundant training data, you do not need all of it - which is rather obvious, and not a particular interesting finding on the topic of unlearning.", "questions_to_authors": "None", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces and systematically validates an unexpected "coreset effect" in current LLM-unlearning benchmarks: across unlearning benchmarks, two models and several selection strategies, unlearning with as little as 5 % of the forget set consistently matches, or even surpasses, full-set performance while preserving utility in unlearning, and the authors offer a convincing token-level explanation via high-impact keywords that are sufficiently represented in the core set, plus robustness analyses. Two reviewers deem the work novel and impactful, raising only clarifications on a fine-tuning inconsistency and a missing evaluation. the authors’ rebuttal supplied new results that seem to satisfy those concerns. Reviewer sYCW argues the findings merely expose obvious data redundancy and criticizes the aggressiveness of NPO/RMU. In their response, the authors argue that the effect does not stem from mere surface-form duplication, and point out to the plot that shows rapid emergence of unlearning effectiveness after relatively many epochs. I think the reviewer is correct in pointing out that this seems to be mainly lexical phenomenon, but I am convinced from the authors' argument that it's unknown and nontrivial, and that it does not seem to be mere duplication in the training data.  I think a major contribution here is the suggestion that unlearning is done rather "shallowly" based on a set of topic-related keywords. It would be interesting to connect this to the abrupt but slow emergence in Figure 1. 

I think the authors should try to make these points more salient in the text. Weighing the reviews and the discussion, I recommend acceptance: the paper provides new, nontrivial results that can spark additional research on unlearning.
Decision: Accept