Not All Data Are Unlearned Equally
Authors: Aravind Krishnan
Siva Reddy
Marius Mosbach
Venue: COLM 2025 (2025)
Keywords: LLM unlearning
analysis
frequency
TL;DR: We investigate how data frequency and model scale affect the feasibility of gradient based unlearning

Abstract: Machine unlearning is concerned with the task of removing knowledge learned from particular data points from a trained model. In the context of large language models (LLMs), unlearning has recently received increased attention, particularly for removing knowledge about named entities from models for privacy purposes. While various approaches have been proposed to address the unlearning problem, most existing approaches treat all data points to be unlearned equally, i.e., unlearning that Montreal is a city in Canada is treated exactly the same as unlearning the phone number of the first author of this paper. In this work, we show that this all data is equal assumption does not hold for LLM unlearning. We study how the success of unlearning depends on the frequency of the knowledge we want to unlearn in the pre-training data of a model and find that frequency strongly affects unlearning, i.e., more frequent knowledge is harder to unlearn. Additionally, we uncover a misalignment between probability- and generation-based evaluations of unlearning and show that this problem worsens as models become larger. Overall, our experiments highlight the need for better evaluation practices and novel methods for LLM unlearning that take the training data of models into account.

Published as a conference paper at COLM 2025
Not All Data Are Unlearned Equally
Aravind Krishnan
Siva ReddyAC Marius Mosbach
MMila Quebec Al Institute, McGill University
BSaarland University
Canaada CIFAR AI Chair
fistname. last tname@mila aa quebec
Abstract
Machine unlearning is concerned with the task of removing knowledge learned from particular data points from a trained model. the context large language models (LLMs), unlearning has recently received increased attention, particularly for removing knowledge about named entities from models for privacy purposes. While various approaches have been pro- posed to address the unlearning problem, most existing approaches treat all data points to be unlearned equally, i.e., unlearning that Montreal is city in Canada is treated exactly the same as unlearning the phone number of the first author of this paper. In this work, we show that this all data is equal assumption does not hold for LLM unlearning. We study how the success of unlearning depends on the frequency of the knowledge we want to unlearn in the pre-training data of a model and find that frequency strongly affects unlearning, i.e., more frequent knowledge is harder to unlearn. Addition- ally, uncover misalignment between probability- and ceneratioon-asse evaluations of unlearning and show that this problem worsens as models become larger. Overall, our experiments highlight the need for better evalu- ation practices and novel methods for LLM unlearning that take the training data of models into account. We publish the code and datasets for our ex- periments at: O https:///8 thub. woom/MMGill 11- wnwpeeuullulleereeeee
1 Introduction
Machine unlearning deals with the problem of removing specific information or knowledge which was acquired during training from model (Cao Yang, 2015; Bourtoule et al., 2021). The motivation for unlearning can be two fold: One one hand there is increasing evidence that LLMs memorize considerable amount of their training data and are also able to reiterate that data verbatim (Tirumala et al., 2022; Carlini et al., 2023; Huang et al., 2024). This is particularly troubling when LLMs memorize and generate personally identifiable information (PII) sensitive data (Carlini et al., 2021), creating demand for technical solutions for removing such data from trained models. On the other hand, the increasing capabilities of LLMs have lead to a growing interest in improving the safety of these models (Ouyang et al.; Bengio et al., 2025). While the standard approach for making models safe right now is post-training via preference optimization (Christiano et al., 2017; Rafailov et al., 2023), unlearning offers an alternative approach to remove, e.g., unwanted and potentially harmful information from LLMs (Jang et al., 2023; Barez et al., 2025).
Existing approaches to unlearning can be broadly categorized into two primary methodolo- gies (we provide longer discussion in Section 5). While some researchers approach the problem from an optimization perspective, others focus on model editing, which identifies and modifies model components that slore or encode the targeted data. In this paper, we focus on coliiization-basee approaches for LLM unlearning and identify crucial gap in current approaches: Existing work in LLM unlearning typically treats all data points designated for unlearning equally. This all data is unlearned equally" assumption disregards potential
Work done during internship at Mila Quebec AT Institute.
1

---

Published as a conference paper at COLM 2025
Frequency in training corpus
Language Model
Forget Quality
"Montreal city in Quebec'
Target
"Montreal
B
in Quebec"
A
"Atlantis is city in the ocean"
Training
Unlearning
Allants Greace"
Trained LM
Target
Figure Illustration of the main hypothesis we explore in this work. Some knowledge is more frequent in the pre-training data of a model than other knowledge (A). We hypothesize that how well we can unlearn any given knowledge (C) should be strongly influenced by how frequent it is in the pre-training data (B).
variations in data, such how frequent they are in the training data of model, and the effect of these variations on unlearning. Hence, we explore the following hypothesis:
The success of optimization-bess unlearning is influenced by the frequency of the targeted information within the training data of a model.
We hypothesize that frequently seen data are harder to unlearn because the model's "belief" (Hase et al., 2024) about these instances stronger (ef. Figure 1). To test this hypothesis, we design comprehensive experimental framework consisting of two distinct settings. First, we conduct controlled experiment where we train models of increasing size from scratch on fictitious biographies and associated questions. We then unlearn information about frequently and infrequently encountered biographies and compare unlearning performances between these setups. Second, we extended our investigation to a real-world scenario unlearning factual information from OLMo-7B (Groeneveld et al., 2024) - while controlling for the frequency of the targeted real-world facts.
Our experiments consistently reveal a strong relationship between frequency in pre- training data and the efficacy of unlearning processes. We find that methods are more successful in unlearning data instances that are infrequent during pre-training. Highly frequent data on the other hand is either not unlearned at all or only appears to have been unlearned, i.e., it can still be extracted from a model depending on the evaluation method. We also demonstrate that this effect is consistent across different unlearning methods.
In addition to our findings about frequency effects in unlearning, we highlight potential evaluation issues with unlearning. Our results show that the efficacy of LLM unlearning varies considerably when evaluated using different methods and that this effect worsens with scale, i.e., larger models are better retaining seemingly unlearned data in their parameters. We find such disparities also in utility evaluation. Unlearning seems to damage the aas-knnoleddge of model, e.g., it becomes worse at question answering, while leaving its probabilistic capabilities intact, e.g., it still assigns high probability to seemingly un- learned sequences. Overall, our experiments demonstrate the need for more comprehensive evaluation and the development of data-dependent LLM unlearning approaches.
2 Optimization-basel unlearning in LLMs
Unlearning can be viewed as regularized optimization problem that balances two losses: The first loss drives the forget operation and computed over the target set we wish to forget (forget set). The second loss is a regularization term computed over retain set, and aims to preserve model utility as measured on an independent utility set. This can be formally expressed as:
L Ej-forget [c (orge + alE jeEeoretain Cregularization (retain;)
(1)
2

---

Published as a conference paper at COLM 2025
Step 1: Attribute Generation
Step 2: Biography Template
Step 3: QA Template
Name's first day in the world was Birthday, He grew up Birth City. He developed expertise in his field at University.
When was Name born? Answer
(Name; ; Birthday: Birth City; University: ; Major: Employer; Employer City: Pronoun)
8
What is the birth city of Name? Answer
Wayne Natalle Schnelder's first day in the world was June 3. 2003, grew up Greeley, IA. He developed expertise in his field at Patten University...
When was Wayne Natalle Schneider born? June 3, 2003
(Wayne Natalle Schnelder; ; June 2003; ; Greeley, IA Patten University Oriminal Justice Visa ; San Francisco, CA, he)
What is the birth city of Wayne Natalle Schneider? Greeley, IA
Data generation process for the pre-training experiments. (1) we generate attribute tuples randomly. (2) a biography template is chosen to verbalize the biography. (3) QA pairs are constructed using a question template.
Figure 2:
Where is hyperparameter used to control the strength of the regularization. In previous work, forger and Eeetain are implemented in different ways (Maini et al., 2024; Yuan et al., 2025; Fan et al., 2024; Yao et al., 2023; Eldan & Russinovich, 2023).
We experiment with three choices of forget: 1) Gradient Ascent, 2) preference optimiza- tion, and 3) refusal training. Gradient Ascent is the negated version of cross entropy loss computed over the target set, i.e, given (x,y) question-answer pair, loss is computed as Lggradient.aceent (6, x, y) - LCE (To(y x)) where ( | x) is the probability that model f generates response when prompted with x. For preference optimization, we use the SIMNPO loss (Fan et al., 2024), which is variant of DPO (Rafailov et al., 2023) but without
positive sample. The loss is defined as: LSINNPO = = log [M] log mo(y) -7) where and yy hyperparameters and ly| is the length of the response. The refusal training loss is the cross entropy loss on the sequence don't know" across all target samples, i.e., (0,
fCE (7Tg(" ("I don't know"] x)).
In all cases, we use cross entropy as the regularization loss. Crucially, while the forget loss is computed only on the response Tp(y x), the regularization loss is computed over all tokens in the retain sample to aid unlearning stability (see Appendix A for more details).
3 Analyzing frequency effects in controlled setting
We first construct synthetic setup which allows us to test our frequency-unleariin hypothesis in controlled setting. Our setup consists of training language models of various sizes form scratch on dataset of biographies and questions about these biographies. Once trained, we evaluate the model by asking questions about some of the biographies. We then unlearn information from various biographies that differ according to their frequency in the pre-training data and re-evaluate the models' ability to answer questions about them.
3.1 Dataset creation
Here we describe the dataset on which we pre-train our models. We follow the bioS-single setup proposed by Allen-Zhu & Li (2024) to create fake biographies. The dataset generation pipeline is shown in Figure 2. Our dataset contains two types of instances - synthetic biographies (BIO and question-answer pairs (QA about the biographies. We start by sampling a tuple of eight attributes: {name, birthday, birth city, university, major, employer, employer city, pronoun]. Next, the attribute tuple is converted into biography using a randomly sampled biography template. We then convert the attribute tuple into
We note even when using Gradient Ascent an unlearning method, the overall objective is still optimized using gradient descent.
3

---

Published as a conference paper at COLM 2025
QA high-count
BB
QA low-count
QA Utility
QA Retain
BIO All
1.0
1.0
GPT2 -20
0.8
GPT2 124M
0.8
210M
0.8
0.6
0.6
0.2
02
0.2
0.0
0.0
0.0
20
40
60
80
100
0
20
40
60
80
100
20
40
60
80
1.00
Training Epochs
Figure 3: BIO and QA evaluations as training progresses on the GPT models. The evaluation metric is Rouge-L. QA accuracies are averaged across all 6 attributes. Over 100 epochs, the models learn to answer questions from the utility, high and low-count sets which are not shown during training. High-count biographies are up sampled and so high-count QA learned faster than the other evaluation sets.
QA instances that query each person-attribute relation (e.g., "When was Natalie Schneider born?", "Where was Natalie Schneider born?"). This is also done using QA templates. We generate 10,000 such biographies and 60,000 associated QA pairs. The attribute lists, biography- and question templates are provided in Appendix B.1.
Constructing training and evaluation splits Before training, we divide the entire dataset (BIO + QA into four groups by person: retain (50%), high-count (16.7%), low-count (16.7%) and utility (16.6%). To simulate variations in exposure, we up-sample the biographies corresponding to one of the subsets (high-count) by a factor of 10x. We do this by resampling biography templates for this subset, keeping the attribute tuples unchanged. Training is done jointly on BIO and QA instances. 2 For training, we use biographies from all splits but the QA instances from only the relain split. All other QA splits: (high-count, low-ccont, ufility) are used for evaluation and unlearning. Note that the QA splits for unlearning are not used for pre-training.
Overall, this setting is intended to mimic the acquisition of factual knowledge in LLM pre-training: The BIO instances imitate pre-training data in LLMs and the QA instances mimic instruction finetuning. The goal is to train a model on all biographies but only a subset of the questions, using the other questions to evaluate knowledge extraction abilities.
3.2 Training and evaluation
We pre-train four GPT-2 models (Radford et al., 2019) of varying sizes 20M, 50M, 124M and 210M - on the synthetic data described in the previous section. We follow Allen-Zhu & Li (2024) and randomly sample BIO and QA instances during training.² We pre-train all models for 100 epochs using learning rate of 0.001. Additional hyperparameters are provided in Appendix B.2.
We evaluate models in two different ways: One tests how well the model retains information about the biographies it saw during training and the other tests the model's ss ability to answer questions about the biographies.
(1) BIO accuracy: Here, we test the model's biography completion ability by prompting it with tokens up required attribute and then evaluating the completion (eg., "Wayne Natalie Schneider was born on..."). "). To ensure robustness, we resample biography templates during BIO evaluation. (2) QA accuracy: To test model's general ability to retrieve knowledge stated in the pre-training data, we query the model for aerson-attribute relations. Since the model also saw QA instances during training, we design this evaluation as QA task using the {low, high)-count QA, and utility QA subsets, which were held-out during training.
2 Allen-Zhu & Li note that joint training leads to better task generalization than pre-training sequentially BIO instances followed by the instances
4

---

Published as a conference paper at COLM 2025
3.2.1 Models are good at answering questions about biographies
Figure 3 shows the evaluation results of the 20M, 124M, and 210M models throughout training. For all models, the BIO accuracy and the QA retain accuracy reaches close to 99% early on. This is not surprising, as the models are explicitly trained on these sets. As training progresses, we see a steady increase in the QA performance for the evaluation splits. For the up-sampled biographies (high-count) we observe a higher performance than for other subsets. However, at the end of training, the accuracies for all evaluation splits are close to each other, plateauing at around 80% accuracy. These results indicates that the trained models are able to perform knowledge extraction for the unseen person- attorbbbte questions from the evaluation set, i.e., they learned do extract knowledge from their pre-training data.
3.3 Unlearning biography information
Next, we aim to unlearning specific biographical information from our trained models.
Setup Our unlearning setup closely follows Maini et al. (2024). We unlearn only on the QA instances from the targets splits, using the QA instances from the retain set for regularization. Both sets are filtered by the attribute being unlearned. Note that BIO instances are not involved in the unlearning process. We compare Gradient Ascent, SIMNPO and IDK losses for unlearning. We unlearn for 20 epochs using a batch size of 16 and run all experiments with different seeds. Further hyperparameters can be found in Appendix B.3.
Evaluation We evaluate both the QA and BIO accuracies for the target, retain and utility sets. Since the BIO accuracy measures how good a model is at completing information it saw during training via language modeling, we expect this metric to an upper bound for the QA accuracy. Stated differently, we do not expect model to be able to correctly answer a question if it cannot correctly complete biography via language modeling. The metric used both evaluations is Rouge-L (Lin, 2004).
3.3.1 Results for individual models
Figure 4 shows the results for unlearning the employer attribute from the 20M model using high-count and low-count groups as unlearning targets. We obtain highly similar results across attributes, model sizes, and unlearning methods, which we discuss in Appendix B.3.
Unlearning harms knowledge extraction disproportionately As unlearning progresses, QA accuracy (top left) drops well below 20% for both high and low count unlearning, showing that the model successfully learns to produce wrong irrelevant answers for target set questions. This also degrades QA utility (25% drop), suggesting that unlearning adversely affects the model's general knowledge extraction. In contrast, BIO performance for utility sets drops only 10%, indicating that unlearning damages task-specific (QA) knowledge more so than general (biography) knowledge.
Frequency affects unlearning success In Figure 4, the QA accuracies (top left) for both high and low-count unlearning follow similar trends. However, BIO evaluations (top middle) show a stark contrast. Even when QA accuracy drops below 20% by the end of unlearning, the model that unlearns the high-count split retains considerably more BIO accuracy (+20%) than the one unlearning the low-count split. This discrepancy arises from the upsampling of high-count biographies, making it harder to fully erase information from this subset. We see similar trends for SIMNPO as well (Figure 7). Our results for refusal training (Figure 8) are the only exception in this trend, where the BIO accuracy never falls below 98% for either split. We attribute this to the fact that in this setup, the sequence "I dont know" was never seen during pre-training and the model very likely memorizes this response during unlearning.
5

---

Published as a conference paper at COLM 2025
Target High Countemploper
Target Low Countumployer
1.00
1.0
BIO Target (other attributes)
QA Target (employer)
0.75
BIO Target (employer)
0.8
0:95
0.50
0.25
1.00
1.0
(other attributes)
QA Utilty (employer)
BIO Utillly (employer)
BIO Utility
0.95
0.50
0.25
0.6
0.90
0.00
0.85
10
10
30
Unlearning Steps
Figure 4: Target and utility evaluations when unlearning high-count and low count QA splits. Retain performance never drops below 0.99 during unlearning. All evaluations are Rouge-L. scores. scores. Model is GPT-20M, the attribute unlearned is employer. For a comparable drop in target QA performance across both groups (top left), the corresponding biography completion accuracy for the low-count group drops 20% more (top middle).
3.3.2 Scaling trends for unlearning
Next, we investigate how unlearning changes with model size. Figure 5 shows the scaling results for gradient descent. Results for other methods are provided in Appendix B.3. We note that all results discussed are averaged over four attributes and three random seeds.
Frequency effects persist across scale Our results show that across different model sizes, the high-count split is consistently unlearned less effectively than the low-count split. This pattern persists also with SIMNPO (see Figure 7), highlighting that pre-training exposure influences unlearning outcomes across both model sizes and unlearning strategies.
Biography retention increases with scale When using gradient ascent (Figure 5), we observe widening divergence between the target QA and BIO accuracies as scale increases. In other words, larger models are more effective at suppressing the QA ability the unlearning data while preserving the corresponding BIO information at the same time. This suggests that models grow in size, completely erasing information from its parameters becomes increasingly difficult, reinforcing the challenge of true unlearning in high-capacity models.
4 Analyzing frequency effects in the wild
Finally, we explore our observations from the previous section extend to even larger models. This helps understand if the implications of our findings extend to real-world scenarios where data sourced from various distributions will need to be unlearned. We choose OLMo-7B* (Groeneveld al., 2024) as the target model, since we have access to its pre-training data. This is crucial for constructing unlearning datasets that differ by frequency. We directly unlearn from the pre-trained OLMo checkpoint.
4.1 Experimental setup
Dataset construction We curate three real-world QA datasets of the form (source, target, relation), controlling for the relation between source and target: (1) Country-Capital QA: These are questions of the form (country, capital, is.capital). For verbalization, we use the
SSimilar trends hold for SIMNPO, see Figure 7 "Specifically allenai/OLMor TB-0724-SFT-hff

---

Published as a conference paper at COLM 2025
20M
50M
124M
210M
1.0
1.0
1.0
High.CC Count
High-Count Ca-nn
0.8
BIO High-Couu
QA Utility
0.6
0.6
0.6
0.2
0.4
0.0
8
10
1.0
1.0
1.0
Low-Coo Count
BIO Low-Count Count
0.8
QA Ubilit
0.8
0.4
0.6
0.6
0.2
0.4
0.0
2
10
Unlearning Steps
Figure 5: Unlearning results across GPT sizes using Gradient Ascent for unlearning. Top row shows results for the high-count split. Bottom row for the low-count split. Results are averaged across four attributes. Rougel-L scores are also normalized with initial values. Larger models unlearn target QA pairs (left) while achieving higher BIO accuracy (middle).
template "What is the capital of [country]? {capital)", (2) Book-Author QA: Questions of the form (book, author, is author). Questions are paraphrased few different ways using manually created templates. (3) ZSRE QA: The ZSRE benchmark contains Wikipedia questions from variety of (source, target, relation) tuples. This setup represents loosely controlled albeit realistic dataset for which mimics potential real-world unlearning request. Samples from each dataset can seen Table 4, Table and Table 6.
Frequency annotation Once we collect large samples of (source, target, relation) pairs for each dataset, we compute the co-occurrence frequency for each (source, target) in Dolma v1.7 (Soldaini et al., 2024), which was used to pre-train OLMo. Co-occurrence frequency is computed as the number of joint appearances of a source and target pair in Dolma. Let us assume a (source, target) pair (China, Beijing). We wish to estimate the number times China and Beijing are present in the Dolma corpus, separated by distance of at most 200 BPE tokens (200 is a conservative upper bound of the word-count paragraph). This is formalized as:
co-occur (China, Beijing)
Σ 1[ "China" A wj "Beijing" | < 200],
where and j range over all token positions in the Dolma corpus. We use the infinigram framework (Liu et al., 2024) to obtain these counts and divide the (source, target, relation, Dolma count) data points into three equally sized buckets grouped by Dolma count. Table provides an overview of the count statistics. We note that the data at this stage has similari- ties to our previous setup: We have QA pairs fixed relation, where the model is exposed to one group much more so than the other.
Unlearning setup We use the most frequent 100 data points from the high high/low-seh /low-count buckets for all unlearning experiments. While the count composition of each bucket varies among the datasets, we ensure that the high-count bucket exposed at-least 100 times more than the low count set for three datasets. The unearning setup closely follows the syn- thetic experiments: For each dataset E pnpitalls,book, zsre), we unlearn high-count,ataeet
5The model we use was additionally fine-tuned on Tulu (lvison et al., 2023) but we choose to ignore these counts as they are dominated by the counts in Dolma. As measured by the median each bucket.
7

---

Published as a conference paper at COLM 2025
or low-countiatasety using medium-countaae for regularization. Hyperparameters used across datasets and unlearning methods can be found in Appendix C.2.
The unlearned models evaluated for unlearning efficacy and utility. Rouge-L is the default metric for evaluation unless stated otherwise. Unlearning efficacy is measured using two approaches: 1) A generative evaluation, where we prompt the model with paraphrased questions from the target set, and 2) a probabilistic evaluation, where the model is prompted to generate True/False response for a few-shot prompt (Figure 9) that verifies the question-answer pair. We compare the probability of the True/ /Falee tokens during probabilistic evaluation. These evaluations are intended to measure unlearning efficacy across task-demands (Hu & Frank, 2024; Deeb & Roger, 2025; Chen et al., 2024).
Utility is also evaluated along two axes: 1) In-domain evaluation: Since the last step in the OLMo-SFT pipeline is dnstruction-tuning on Tulu V2 (Ivison et al., 2023), we consider the evaluations on Tulu as in-domain. We test in-domain generation utility using a zero-shot subset of FLAN (Longpre et al., 2023), which is included in Tulu. We also evaluate on few- shot QA samples from FLAN to assess a related-but-d bulfffffeeffnn task. To measure probabilistic performance, we compute the perplexity on subset of Tulu sampled from all composite tasks. 2) OOD evaluation: The world-facts questions from Maini et al. (2024) are used for out-of-domain QA evaluation. We also evaluate tiny-MMLU and till-Hell (Polo et al., 2024), which offer probabilistic measures for utility. To prevent data leakage, we make sure that there is overlap between the answer tokens of the target and utility splits.
4.2 Results
We present results from unlearning the capitals dataset with SIMNPO in Figure 6. Other dataset-method combinations are illustrated in Appendix C.3.
Unlearning overexposed instances is more difficult Mirroring observations from the synthetic setup in Section 3, see that unlearning efficacy strongly depends on frequency. Across the board, our results show that there is a considerable difference between unlearning data from different count buckets. Given an equivalent drop in utility, we see that the (1) paraphrased target performance for the high-count split does not drop as much as the low-count split (2) in the case that it does, it's probabilistic performance does not drop below 99%. In all our experiments, the generative and probabilistic performance of the low-count split falls consistently lower than the high-count split.
Probabilistic and generative utility evaluation do not tell the same story Across experi- ments, we consistently observe that all the generation based evaluations drop as unlearning progresses. Notably, zero-shot QA drops more than few-shot QA across the board. On the other hand, neither of our probabilistic evaluations, e.g., tinyMMLU or tinyHellaswag deteriorate with unlearning. This is true even in cases where zero-shot QA deteriorates considerably (see Figures 16 and 17). These results highlight that that there seems to be task-specific element to utility degradation, which aligns with our findings in the synthetic case (Section 3.3.1). Such task-dependent discrepancies have also shown up in previous work on model evaluation (Hu & Frank, 2024; Hu & Levy, 2023) and preference optimiza- tion (Chen al., 2024). Future work should consider these differences based on evaluation to avoid misconceptions about (decreases in) model utility after gradient-basee unlearning.
5 Related work
LLM Unlearning - Methods Existing literature on LLM unlearning can be broadly categorized by using either black-box or white-box methods. These categories differ based on whether data-specific modifications are directly applied to the model's parameters not. Black-box methods rely on gradient-based approaches, i.e. they update model parameters through gradient descent (Ishibashi & Shimodaira, 2023; Jang et al., 2023; Zhang et al., 2024; Fan et al., 2024). White-box methods offer a more surgical approach to unlearning. The general idea is to locate and directly neutralize model components that correspond to the
8

---

Published as a conference paper at COLM 2025
Target - High Countcasiaas
Target Low Countrapkals
FLAN- -Zero Shot Target Paraphrased
Target True/False
1.0
1.0
1.0
0.8
Retain Sel
0.6
0.5
0.5
0.4
0.0
0.0
5
10
5
10
5
10
0.9
Few Shot
1.0
Tulu Perppexity
10-
0.8
0.8
0.7
0.6
0.6
0.4
0.5
0.2
5
10
5
10
5
10
0.60
tinyHellaswag
0.850
World Facts
0.9
tinyMMLU
0.55
0.825
0.50
0.800
0.8
0.45
0.775
0.7
0.40
0.750
5
10
5
10
5
10
Unlearning Epochs
Figure 6: Unlearning high-count and low-count capitals with SIMNPO on OLMo-7B.
Rouge-L is the evaluation metric. Top row shows performances for the forget and retain sets. Middle row is in-domain utility evaluation and bottom row out-of-domain utility evaluation. Target evaluations are normalized across their initial values. Infrequent instances are forgotten faster and more efficiently across generative (top left) and probabilistic (top middle) evaluations.
target data, without updating the entire model (Guo et al., 2025; Ilharco et al., 2023; Li et al.). There is an additional class of grey-box methods which combine the strengths of the black and white-box methods. These work by limiting gradient descent to a data-specific subset of the model's weights found using attribution (Hong et al., 2025; Zhang et al., 2025) or insights from interpretability research (Tutek et al., 2025).
LLM Unlearning Data Unlearning datasets are different across tasks and domains. A common task in previous work is to unlearn question-answer pairs and/ /r multiple-choice questions. Several works attempt to unlearn entire domains, where the data removal is more general. Forgetting the Harry Potter universe (Eldan & Russinovich, 2023) or news articles (Shi et al., 2025) are examples for this. Across both tasks, the factuality of the data to be unlearned also varies considerably. Several works look at unlearning fictitious data (Maini et al., 2024; Deeb & Roger, 2025; Guo al., 2025) from models that are fine tuned on these datasets. Other works also look at removing real-world knowledge like bio-chemical weaponry (Li et al.), Wikipedia knowledge (Patil et al., 2024), or celebrity trivia (Jin et al.) from pre-trained models. Our work spans both setups: pre-train models from scratch using synthetically generated data, and also unlearn real-world data from pre-trained LLM.
Memorization and Unlearning The effects of memorization on unlearning have been studied in computer vision (Zhao et al., 2024), where the authors show that verbatim memorization negatively impacts the fcasibility of unlearning. Bärbulescu & Triantafillou (2024) study the effects of memorization in LLM unlearning. They unlearn memorized instances of PILE (Gao et al., 2020) from the GPT-Neo models (Black et al., 2021), establishing well. Our work asks
that verbatim memorization is adversary to LLM unlearning similar question in an intermediate setting: what if instances are not necessarily memorized, but frequently encountered? Does this also correlate with the ability to unlearn data? In concurrent work, Baluta et al. (2024) explore similar questions in synthetic setup, using machine translation the target task and gradient ascent as the unlearning method.

---

Published as a conference paper at COLM 2025
Their conclusions agree with ours - that differently exposed meoedr/zed data points are unlearned differently. Our work adds evidence to this narrative, establishing the exposure- unlearning correlation in widely used open-source language model, ie,, OLMo, and explores variations across model scale, unlearning methods, and evaluation setups.
Effect of data frequency on LLM behavior Variations in data exposure during pre-training has been shown to impact multiple aspects of model behavior: In computer vision, previous work (Parashar et al., 2024; Udandarao et al., 2024) has shown that the zero-shot capabilities of VLMs are linked to the frequency of a concept in the pre-training corpus, Consequently, Verma et al. (2024) show that image generation models are better imitating images fre- quently seen during pre-training. In the LLM domain, previous work has shown that the duplication of training samples in the pre-training corpus considerably degrades model performance (Lee et al., 2022) and increases privacy issues (Kandpal et al., 2022). Merullo et al. (2025) show that factual knowledge which is more frequent in pre-training data is linearly encoded in LLM representations. Recent work (Carlini et al., 2023) has also drawn strong connections between memorization and the frequency of training examples: Shi et al. (2024) show that repeated exposure to certain data points during pre-training increases the strength of memorization, making these data points more detectable via membership inference attacks (Mattern et al., 2023). Similarly, Nasr et al. (2025) demonstrate that models trained for more epochs are more likely to regurgitate verbatim training examples, reinforc- ing the idea that exposure frequency amplifies memorization and affects model behavior at inference time. Our work contributes to this narrative, and studies the role of frequency in unlearning. We examine how repeated co-occurrence of entities in pre-training data can influence unlearning, and show that frequently seen information (here entities and their relationship) considerably harder to unlearn than infrequent information.
Conclusions and Future Work
In this work, we test the hypothesis that it should be more difficult to unlearn data that was seen more frequently during pre-training. We provide evidence for this hypothesis across synthetic and real-world setups: 1) We train GPT-2 models from scratch on fake biographies and find that unlearning questions about more frequent biographies is more difficult than other biographies. 2) Extending our results to a real-world scenario, we observe similar trends when unlearning QA pairs from OLMo-7B. Unlearning knowledge that is not frequent in the OLMo pre-training corpus is more easy than frequently encountered data. Additionally, our experiments reveal discrepancies between evaluations of unlearning: While probabilistic evaluations of forget quality and utility display only a minor impact of unlearning on model performance, the impact on performance seems much more negative when using generative evaluations.
We make the following suggestions to guide future research in LLM unlearning: 1) Unlearn- ing Methodology: As a first step, future work on optimization based unlearning should group data points in the forget set according to their estimated frequency make sure that proposed algorithm works well across different samples. To foster research on data-specific unlearning, we need better benchmarks that offer a range of models where training data is known and unlearning splits can be designed carefully. 2) Utility Evaluation: We call for a more comprehensive evaluation of utility following unlearning, given the variation we observed when reporting utility with different benchmarks. We also advocate placing greater emphasis on generative evaluations over likelihood evaluations, since our findings suggest that generative tasks more sensitive to model degradation.
Acknowledgments
We would like to thank Fabian David Schmidt for many helpful discussions and feedback on this work. We also thank the COLM 2025 reviewers for their constructive feedback. Marius Mosbach is supported by the Mila P2v5 grant and the Mila-Samsung grant. Siva Reddy is supported by the Canada CIFAR AI Chairs program and the NSERC Discovery Grant program.
10

---

Published as a conference paper at COLM 2025
References
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. In Forty-first International Conference on Machine Learning, 2024. URL httpps://ppeeeeeiieeee wttp://ee//oroonnn//////rrrr
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), Infernational Conference on Representation Learning, volume 2025, Pp. 85824- 85832, 2025. URL https://prooceedings. iclr cc/paper Files/paper/2225/fille pdf.
Teodora Baluta, Pascal Lamblin, Daniel Tarlow, Fabian Pedregosa, and Gintare Karolina Dziugaite. Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method, 2024. URL
04388.
Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, Luke Ong, Philip Torr, Kwok-Yan Lam, Robert Trager, David Krueger, Sören Mindermann, José Hernandez-Orall, Mor Geva, and Yarin Gal. Open Problems in Machine Unlearning for AI Safety, 2025. URL https://arxiv. wtt.or//bb//221.0495.2..
Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philiip Fox, Ben Garfinkel, Danielle Goldfarb, Hoda Heidari, Anson Ho, Sayash Kapoor, Leila Khalatbari, Shayne Longpre, Sam Manning, Vasilios Mavroudis, Mantas Mazeika, Julian Michael, Jessica Newman, Kwan Yee Ng, Chinasa T. Okolo, Deborah Raji, Girish Sastry, Elizabeth Seger, Theodora Skeadas, Tobin South, Emma Strubell, Florian Tramèr, Lucia Velasco, Nicole Wheeler, Daron Acemoglu, Ol- ubayo Adekanmbi, David Dalrymple, Thomas G. Dietterich, Edward W. Felten, Pascale Fung, Pierre-Olivier Gourinchas, Fredrik Heintz, Geoffrey Hinton, Nick Jennings, An- dreas Krause, Susan Leavy, Percy Liang, Teresa Ludermir, Vidushi Marda, Helen Margetts, John McDermid, Jane Munga, Arvind Narayanan, Alondra Nelson, Clara Neppel, Alice Oh, Gopal Ramchurn, Stuart Russell, Marietje Schaake, Bernhard Schölkopf, Dawn Song, Alvaro Soto, Lee Tiedrich, Gaäl Varoquaux, Andrew Yao, Ya-Qin Zhang, Fahad Albal- awi, Marwan Alserkal, Olubunmi Ajala, Guillaume Avrin, Christian Busch, André Car- los Ponce de Leon Ferreira de Carvalho, Bronwyn Fox, Amandeep Singh Gill, Ahmet Halit Natip,Juha Heikkilä, Gill Jolly, Ziv Katzir, Hiroaki Kitano, Antonio Krüger, Chris Johnson, Saif M. Khan, Kyoung Mu Lee, Dominic Vincent Ligot, Oleksii Molchanovskyi, Andrea Monti, Nusu Mwamanzi, Mona Nemer, Nuria Oliver, José Ramón López Portillo, Balara- man Ravindran, Raquel Pezoa Rivera, Hammam Riza, Crystal Rugege, Ciarán Seoighe, Jerry Sheehan, Haroon Sheikh, Denise Wong, and Yi Zeng. International AT Safety Report, 2025. URL https:://// /ttp///wxiivivi///iv///eeee/aavl wttpxii.orr/bbb/550111 17805.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. 2021. URL https: //api. wew.nti .ewaaarorrrg/ree org .rrgg//orpuss.::2555771.... 245758737.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine Unlearn- ing. In 2021 IEEE Symposium Security and Privacy (SP), pp. 141-159, 2021. doi: 202.0000.0021.00099. URL https://eeexxploee.eees wttg///t////////////////// stamp.jspptpp 8arnumber=9519428.
George-Octavian Bärbulescu and Peter Triantafillou, To Each (Textual Sequence) Its Own: Improving Memorized-Date Unlearning in Large Language Models. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 3003-3023. PMLR, 21-27 Jul 2024. URL https: / h/tp::///rrcceeing.. mlr. www.p/resssss. http//bbbulluuu4aaa..o / html.
Yinzhi Cao and Junfeng Yang. Towards Making Systems Forget with Machine Unlearning. In 2015 IEEE Symposium on Security and Privacy, pp. 463-480, 2015. doi: 10.1109/SP220155 URL https::/ieeexploee ieee. org/ sttss/tmmssaatt//a https//////tttppppppppaaaa/rrrr/ 730
11

---

Published as a conference paper at COLM 2025
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Language Models. In 30th USENIX Se- curity Symposium (USENIX Security 21), pp. 2633-2650. USENIX Association, August 2021. ISBN 978-1-999133-244-3 URL https: /ttp:://wwwww.// http://www....iii///////. org/ wonfferrnnee/ueeeieeerrrtt.../
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Representations, 2023. URL https:
Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Ra- jesh Ranganath, and Kyunghyun Cho. Preference Learning Algorithms Do Not Learn Preference Rankings. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37, PP. 001928-101963 Curran Associates, Inc., 2024. URL https: /tttos://proceedinggs Saraapper-ooofeeeee..... pdf.
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL htttss::/proceeeeeeee filee/d5e2ceadad55eee111 FF10502200001.110004022220000
neurips httppsscaaaaaaafilleeeesese/eeee pdf.
CraigH411. List of 1,000 Most Common Last Names (USA). https://gist. 51t55/2224ff/a/// ob0tr 000% 25520005522220000000233322222222
2025. Accessed: 2025-03-24.
Aghyad Deeb and Fabien Roger. Do Unleaming Methods Remove Information from
Language Model Weights?, 2025. URL https://arxxv..rrgaas//111
dotjoel. List of Universities and Colleges. http:://gss..ghhbbbeerrrttttttt com/ httpr//ccaa..ooooooooooooeeeeea. all-colleges. txt, 2025. Accessed: 2025-03-24.
EatMoreOranges.
https::/raw. So0o--Daaasttrefs/hhadsss data/2023 1otr2223ffrruue-500000 satee-550dddaa.ccomm 2023. Accessed: 2025-03-24.
Fortune 500 Companies Dataset (2023).
githubusercontent htw//aattorrOrangs//Frruuu
Ronen Eldan and Mark Russinovich. Who's Harry Potter? Approximate Unlearning in LLMs, 2023. URL https: /2310 02238.
Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning. In Neurips Safe Generative AI Workshop 2024, 2024. URL https://openreviier net/forumm id=pVACX02m0p.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling, 2020. URL https://arxiv httgg/abs/211..00000
Grammakov. USA Cities and States Dataset. granmakov/USA-Cn csv, 2025. Accessed: 2025-03-24.
https://raw. https://ra..gihhbuurrrrottt githubusercontent.cco com/ cities states counties.
12

---

Published as a conference paper at COLM 2025
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magmusson, Yizhong Wang, Shane Arora, David Atkin- son, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muen- nighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the Science of Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume Long Papers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ v1 2202..2cl--ooog URL SKKLhtts://aclannhhlloe. orgg 20t. 841/.
Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization, 2025. URL.
Peter Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. Fun- damental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs? Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openrevient net/forumidd-LRF11
Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces. In Workshop Neural Network Weights as a New Data Modality, 2025. URL https: hatp:://pppnreeee apenrevieww.w.eee/ net forum?1d=772iDD44000
Jennifer Hu and Michael Frank. Auxiliary task demands mask the capabilities of smaller lan- guage models. In First Conference on Language Modeling, 2024. URL https: pet/forrumiid=SSBUzs0
Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040-5060, Singapore, December 2023. Association for Computational Linguistics. doi: 10. (0.18653/vl1 2023. wer.eennlmmmii.33.
URL htttps://aclin httpps:/aalannthology org. 2023. emnlp-main 306/.
Jing Huang, Diyi Yang, and Christopher Potts. Demystifying Verbatim Memorization in Large Language Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 10711-10732, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: (0.18653/v1/24 /tt/://ww//...mmmmmmm...... 2024. emnlp-main. 5988.
URL https: https://aclanthologyy org/
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrewiew. wtwpwwwwwww.eeeouuuiiiiiiie 6t0Kwf8-jrj.
Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge Sanitization of Large Language Models. CoRR, abs/2309.11852 2023. URL https://doi. org/10. 48550/ hato:/rXivv... 2309. 11852.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. CoRR, abs/231 2bs/2211.10702, 2023. URL hwtpt:tttto.o..oooo org/10 48550/arXiv. 2311. 10702.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge Unlearning for Mitigating Privacy Risks in Language Models. In Anna Rogers, Jordan Boyd-Grabes and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
13

---

Published as a conference paper at COLM 2025
Papers), PP. 14389-14408, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/20000 wl/2023.acl-long..0 805. URL haap::///aclanhhologyyy or 2023. acl-long.800 885/.
Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. RWKU: Benchmarking Real-World Rea Knowledge Unlearning for Large Language Models. URL http://ariiv.orrgaas//44..
JTRNS.
First Names Dataset. 6faaf857580ee first-names. txt, 2025. Accessed: 2025-03-24.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating Training Data Mitigates Privacy Risks in Language Models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10697-10707. PMLR, 17-23 Jul 2022. URL sttpss//ppoceediife mml.. wat..press/vlee kandpal22a. html..
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424 8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 /22/022.cll-oogg...c 577. URL https: acl-long. .57/
https:://aclanthologgg/ wcl.ttolooocttooo..ooo.o org/ 2022.
Nathaniel Li, Alexander Pan, Anjali "Copal,Summe Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert-Voss,, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken- Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Ruoyu Wang, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. URL /ttp://///a/ org/ abs/ 03218.
Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Com- putational Linguistics. URL https://acclanthology org org/w04--0113
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini- gram: Scaling Unbounded n-gram Language Models to Trillion Tokens. In First Confer- ence on Language Modeling, 2024. URL https: http////ppeeeeeeeeeeeeeeeeeeeeee
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML/23. JMLR.org, 2023. URL https: ppppce//prooee.ings. mlr. mlr.ppress/v02. http://////lopppp/22/////rrr2/// pdf.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and Zico Kolter. TÓFU: A Task of Fictitious Unlearning for LLMs. In First Conference on Language Modeling, 2024. URL https:
wttp:r/wwweeeeteoeorrmiiiiea.e ThNBoLL.......
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership Inference Attacks against Language Models via Neighbourhood Comparison. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 11330-11343, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https: http:///aaaaaaaha/ org/ 2023. f findings-acl. .799.
14

---

Published as a conference paper at COLM 2025
Jack Merullo, Noah A. Smith, Sarah Wiegreffe, and Yanai Elazar. On Linear Representations and Pretraining Data Frequency in Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https///ppenreiiii.eteefoooonon EDoD3DgivF.
Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Florian Tramër, and Katherine Lee. Scalable Extraction of Training Data from Aligned, Production Language Models. In The Thirleenth International Conference on Learning Representations, 2025. URL https: //openreview. net/ fowp//ddummemieeeeme 3nWP2a.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. URL https::/procceediiiii neurips cc/ pttpp/pfflllleeeeeeee//ppp / oor pdf.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong- The neglected tails in vision-language models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12988-12997, 2024. URL https:// thecvf. com/ (eteelsscvPR.2244.4
Thee.
pdf.
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. In The Twelfth International Conference on Learning Representations, 2024. URL https:://penrreveew.. htt////orrevieetee/ffrrrrrr////e ne
1Ro1V0.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. In Forty-first Inter- national Conference on Machine Learning, 2024. URL http:: //pps//v/e/eeeeee.ee/mmeeee met/forum?id= gAm13FpfhG.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language Models are Unsupervised Multitask Learners. 2019. URL
https://cdn.onm openai. opw.aii.ccnnooooiii.o//ia Trunsppprissssccffllittttssssssr
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. Curran Associates, Inc., 2023. URL httpp::/ neerreeeeddiinnnnnnniissessep neurips. cctpaperk5Ffiii pdf.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting Pretraining Data from Large Language Models. In The Twelfth International Conference Learning Representations, 2024. URL https://openrevii net (rforum?id=zwar3M00M
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine Unlearning Six-Way Evaluation for Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreviee wtt.ppnnrriiie.ee/foromiiid net/forum?id TArmA033BU.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Pe- ters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant
15

---

Published as a conference paper at COLM 2025
Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. 2024. URL https://arxiv.orr /000.//002....
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Mem- orization Without Overfitting: Analyzing the Training Dynamics of Large Language Models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: https::/opperreiiew.. net/ forum?id-u3vEuRree
Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasović, and Yonatan Belinkov. Mea- suring Faithfulness of Chains of Thought by Unlearning Reasoning Steps, 2025. URL
https://arxiv.org/aa 250/2 14829.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No "zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. In The Thirty-eightth Annual Conference on Neural Information Processing Systems, 2024. URL
https:/oopenreview wttp:/rwwwweeeeeeeoooom///ddd/oe www..e//forrm///eeyyyyy..i.ggg
Sahil Verma, Royi Rassin, Arnav Mohanty Das, Gantavya Bhatt, Preethi Seshadri, Chirag Shah, Jeff Bilmes, Hannaneh Hajishirzi, and Yanai Elazar. How many van goghs does it take to van gogh? finding the imitation threshold. In Neurips Safe Generative Al Workshop 2024, 2024. URL https: http::s//ppeeevvieeee hwt.:/ne//ooooo nett/forum?ddeWWFFfttt
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large Language Model Unlearning, In Socially Responsible Language Modelling Research, 2023. URL wK66,
Xiaojian Yuan, Tianyu Pang. Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. Closer Look at Machine Unlearning for Large Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL Q1MHvGmhyT.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning. In First Conference on Language Modeling, 2024. URL https: http:///////eeeeeeeeeeeem/ wttp://iww...eeeoooomoo/o/aa///.
Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic Failure of LLM Unlearning via Quantization. In The Thirleenth International Conference on Learning Representations, 2025. URL https: / //openreview.r www.eeewwwwwweeowweweeeeeee www..eeeoorumiidddeeeeeeaaa..m
Kairan Zhao, Meghdad Kurmanji, Geovie-Octavian Barbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomczak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 12293-12333, Curran As- wwwuurpsscc/ppppreeeee wtt.eeeeseeieseppeeeeeeeeeeeeeee http//ppperr//2//2 pdf.
sociates, Inc., 2024. URL http: https///s/// ptw.:aaaaeeoooeeeaaaeeeeereeeeee
16

---

Published as a conference paper at COLM 2025
A Modified Loss Function for Unlearning
Since our unlearning targets are always QA pairs, and the setup closely resembles Maini et al. (2024) and Fan al. (2024), we adopt their loss formulation for unlearning. Given target question-answer pair (Xtarget, ytarget) and retain pair(Xretainv (eetain Fretain), these papers compute the unlearning loss as:
C turget (0, Vtarger Starget) a regularizatton (0, Vretuin Xvetain)
That is, the forget and regularization losses are computed only over the target and retain responses, conditioned on the respective inputs. We propose the use of a modified training loss:
C = Lforget ytarget (target) + aCCE( Pretain Xretain) ) + Ó6Cr(0, Mtarget) + aCCCE( Xyetain
The modified function presented in Equation (3) adds additional regularization terms over the question tokens for both forget and retain sets. Keeping the forget loss computation the same, the regularization loss is computed over the target question Xtarget, the retain question and the retain answer conditioned on the question. Note that this does not conflict with the forget signal. i.e., we would like the model to forget the answer to question, but the question itself is a grammatical sentence we can regularize on. We find that the addition of regularization from the question tokens improve the stability of the unlearning process and decrease the drop in QA performance during unlearning, especially for the larger models.
B
GPT-2 Experiments
B.1 Dataset Creation
Attributes Each value in the tuple (name, birthday, birth city, university, major, employer, employer city, pronoun is randomly sampled from the folowing lists:
1. Each name is sampled from list of first, middle and last names. First names are taken from JTRNS (2025) and last names from CraigH411 (2025). The first name list is reused for middle names, making sure that they not duplicates.
2. Birthdays are randomly sampled from {[1-28), [an-Dee], [1900-2099]].
3. Birth city is sampled from Grammakov (2025). They are cities in the United States of America.
4. Universities are taken from the list of US universities published by dotjoel (2025).
5. Employer list and locations are sourced from the fortune-500 list compiled by EatMoreOranges (2023)
6. Pronouns are sampled from (he, she, they). We then use the appropriate personal and possessive pronouns.
Biography templates and samples Biography templates created by first collecting individual templates for each attribute. We use ChatGPT to generate 50 templates per attribute. Some example attribute templates are provided in Appendix B.1. Attribute templates are drawn at random and concatenated (in the same order) to make biography templates. This gives us a total number of 506 unique biography templates. This process results in biographies which are six sentence descriptions of each attribute of a person, composed in the same order. Some examples of biographies are shown below:
1. Sandra Denise Wise's arrival happened on February 21, 2053. Her life was first influenced by Mount Croghan, SC. She took advantage of internship opportunities at Hendrix College. She collaborated in Biology. She was employed at Westlake. She gained industry recognition through work in Houston, TX.
research projects
17

---

Published as a conference paper at COLM 2025
2. Kathleen Laura Gordon's arrival happened April 17, 2079. He was raised in Talbot, IN. He spent countless hours in the library at California Institute of Integral Studies. He took specialized courses in Operations Logistics And E-Commerce. He contributed to the mission of ConocoPhillips. He engaged in consulting work in Houston, TX.
3. Christine Margaret Flowers took their first breath on June 14, 1900. Her heritage is rooted in Falls Mills, VA. She broadened her academic horizons She developed programming skills
at University of Wisconsin--Superion relevant Physics. She achieved professional growth at Microsoft. She engaged in consulting work in Redmond, WA.
4. Samantha Megan Deleon the world in Benge, WA. They refined their analytical skills at Brazosport College. They participated in case competi cnppetitions related to Human Services And Community Organization. They developed expertise through Costco Wholesale. They developed professional skills in Issaquah, WA. 5. Douglas Scott Kim's first day in the world was June 25, 1902. owes his origins to Wrentham, MA. He was an active member of the academic community Hendrix College.
born January 27, 2098. They entered
learned industry-standard practices in Music. He thrived in his career at JPMorgan Chase. He advanced his professional journey in New York, NY.
Question Templates Question templates are taken from Allen-Zhu & Li (2024). These are:
1. What the birth date of NAME? BIRTHDAY.
2. What is the birth city of NAME? LOCATION.
3. Which university did NAME study? UNIVERSITY.
4. What major did NAME study? MAJOR.
5. Which company did NAME work for? EMPLOYER.
6. Where did NAME work? EMPLOYER.CITY
B.2 Training
Instance Construction We use the mixed training proposed by Allen-Zhu & Li (2024), where the model sees both BIO and QA instances Training samples constructed by concatenating additional samples to each instance, separated by the <eos> token. The samples for concatenation original instance. That is, QA instances are concatenated together and biographies are likewise concatenated. Samples
randomly samples and homogeneous to the type of the
concatenated until the tokenized instance has length of 512. For an dataset of 100 BIO and 100 QA instances, this produces a dataset of 100x512 BIO instances and 100x512 QA instances. We use a BIO :QA token ratio of 1:3 since Allen-Zhu & Li (2024) report that this aids task-learning. This is done by picking more QA instances at random and repeating the instance construction process. For the toy example, the total dataset would contain (100x512 + 300x512) samples for training.
Training Hyperparameters A learning rate of 0.001 is used to train all models. We use batch size of 32 during training. The models are constructed from scratch; the architectures used can be found in Table The learning rate warmed up for 10% of total steps. The models are trained with fp16 precision, with weight decay of 0.01. We use the fused variant of the AdamW optimizer for faster converging. Other training hyperparameters are default values and follow Allen-Zhu & Li (2024). Similar to the original paper, find that the performances for employer city are comparatively weaker for the trained models, so we do not consider for unlearning.
18

---

Published as a conference paper at COLM 2025
Attribute
Example Templates
Birthday
NAME was born on BIRTHDAY
NAME's birthdate is BIRTHDAY
NAME came into the world on BIRTHDAY
NAME was welcomed into life BIRTHDAY
NAME's journey began on BIRTHDAY
Birth City
PERSONAL. was born in LOCATION LOCATION where PERSONAL.PRONOS DOSSESSIVE_PRONOUUS roots lie in LOCATION PERSONAL.PRONOUU entered the world in LOCATION
was born
POSSESSIVE PRONOUN birthplace is LOCATION
University
PERSONALLPRONOUN studied UNIVERSITY PERSONAL PARSONALPPROOOUU enrolled in UNIVERSITY
FERSONAL. was accepted into UNIVERSITY
PERSONAL.PRONOUN completed studies at UNIVERSITY PERSONAL honed skills at UNIVERSITY
Major
specialized MAJOR
HERSONALPPROOOUN pursued degree in MAJOR
PERSONAL PRONOUN studied MAJOR at university
PERSONAI. conducted research in MAJOR
PERSONALIPRONOUN explored MAJOR coursework
Employer
PERSONAL. worked at EMPLOYER FERSONAL,PRONOUT built career at EMPLOYER
PERSONAL gained experience at EMPLOYER
PERSONAL.PRONOUN served in a role EMPLOYER
PERSONALPRONOUU took responsibilities at EMPLOYER
Employer City
PERSONAL.PRONOUN worked in EMPLOYER.CITT PERSONAL.PRONOUU built career in EMPLOYER CCTTY PERSONALLPRONOUN took on responsibilities in EMPLOYER.CITY PERSONAL PRONOUN developed skills in EMPLOYER CITY PRRSONAL.PONNOO contributed projects in EMPLOYER CITY
Table 1: Example Templates for Biography generation
Model Size
Number of Hidden Layers
Number of Attention Heads
Hidden Size
20M
8
8
256
50M
8
8
512
124M
12
12
768
210M
12
16
1024
Table 2: Model configuration details for the GPT models we train
B.3 Unlearning
Hyperparameters Unlearning uses the loss illustrated in Equation (3). The å value is set to 1.0 for all runs. For all unlearning experiments, 1e-4, 2e-4, 5e-4 and a values between 1, 5, 10, 15, 20, 25. For the 20M and 50M models, learning rate of 2e-4 and learning rate of 8e-5 and a = 25. For SIMNPO, we use a y value of 0 and a ß value of 0.1, which is seen work well.
search for learning rates among 8e-5,
of 20 is seen to work well. The 124M and 210M models use
Unlearning plots Figure 7 and Figure 8 show unlearning plots for GPT-2 models using SIMNPO and refusal losses respectively. Results are plotted and averaged across four attributes.
19

---

Published as a conference paper at COLM 2025
20M
50M
124M
210M
High.CC Count
1.0
BIO High-Count Ca-Cnnntt
1.0
1.0
0.8
QA Utility
0.6
0.6
0.6
0.2
0.4
0.0
10
1.0
1.0
1.0
Low-Com Count
BIO Low-Count Count
0.8
QA Ubility
0.8
0.4
0.6
0.6
0.2
0.0
10
Unlearning Steps
Figure 7: Unlearning GPT models using SIMNPO loss. Results are averaged across four attributes. Top row shows results from unlearning the high-count split and the bottom row from unlearning low-count. The Rougel-L scores are normalized across attributes. Larger models display an increased ability to seemingly unlearn target QA pairs (left columns), while actually retaining information seen in target BIO cvaluations (middle columns).
20M
50M
124M
210M
High-Count CoCnt
1.0
BIO High-Count
1.00
0.8
0.95
Utilit
0.8
0.6
0.85
0.4
0.4
0.80
0A LLoo-Cooon Co--Count
1.0
1.00
0.8
0.95
Utility
0.8
0.6
BIO
0.85
0.4
0.4
Unlearning Steps
Figure 8: Unlearning GPT models using refusal training. Results are averaged across four attributes. Top row shows results from unlearning the high-count split and the bottom from unlearning low-count. The Rougel-L scores are normalized across attributes. The models learns to memorize the "I don't know" response that has not been seen during pre-training. This is not seen to affect target BIO performance.
C
OLMo Unlearning
Dataset samples and count statistics In Table 4, table 5 and table 6, we show the most frequent samples across all three count buckets for Capitals, ZSRE and Books datasets. The associated co-occurence counts in the Dolma dataset (across 200 token spans) are also provided. We also show count statistics for all datasets in Table 3.
20

---

Published as a conference paper at COLM 2025
Dataset
Mean + Std
Median
Min
Max
Capitals
Low Count
11,566.42 13,59.112
4,897.0
1.0
44,341.0
Medium Count
234,492.09 151,210.70
202,202.0
45,243.0
613,776.0
High Count
3,322,667.0 4,620,1160
1,749,344.0
640,168.0
27,999,928.0
ZSRE
Low Count 100
225.5 # 8.78
226.0
210.0
239.0
Medium Count 100
1,950.79 90.37
1,939.5
1,788.0
2,103.0
High Count 100
557,792.77 1,324,276.07
185,913.0
79,823.0
9,009,989.0
Books
Low Count 100
125.99 14.75
124.0
101.0
152.0
Medium Count 100
7,526.60 1,214.35
7,435.5
5,812.0
9,991.0
High Count 100
32,110.77 + 29,547.27
21,362.0
13,255.0
209,420.0
Table 3: Frequency statistics for the OLMo datasets. all cases, the median of the high count bucket is at-least 100 times more than the low-count group. The median differences are smallest for books, and largest for capitals.
Low-count [co-oceurrence count]
Medium-count count]
[co-occurrence
High-count count]
[co-occurrence
What is the capital of Aruba? Oranjestad [44341.0] What is the capital of Marshall Islands? Majuro [41431. .0] What capital of Kiribati? Tarawa [41139.0]
What is the capital of Slovakia?
What is the capital of China? Bei- jing [27999929, 0] What is the capital of Ireland? Dublin [28868122. What the capital of France? Paris [18399078.e] What the capital of Israel? Jerusalem [17605351. 0] What is the capital of Ukraine? Kyiv [10397531.0]
Bratislava [613776. a] What is the capital of Venezuela? Caracas [563742. What is the capital of Morocco? Rabat [588836.0] What the capital of Colombia? Bogotá [488553. What is the capital of Fiji? Suva [488433.0]
What is the capital of Equatorial Guinea? Malabo [41 What is the capital of Cayman Is- lands? George Town [40836.
Table 4: Capitals dataset samples: Samples are shown across count buckets and co- occurrence counts for (country, capital) pairs in the Dolma corpus for the capitals dataset. Paraphrased versions are used for testing
C.1 Instance construction for unlearning and evlauation
Since the OLMo model is SFT tuned, we provide target and retain instances ing the expected chat template. Specifically, the QA instances are constructed as: < </endoftext| <>ss<>user>>nnnn InQUISTIONN I |asssssttnn assistant
nANSWER where the QUESTION and ANSWER terms are replaced with entries from the target and retain sets. The forget loss is only computed across the answer tokens for the target set. The regularization loss is computed as in Equation (3). The evaluation samples are also provided using the chat template.
C.2 Hyperparameters
We note that finding appropriate hyperparameters for OLMo was more difficult than the GPT-2 models. Gradient ascent and SIMNPO were both sensitive to hyperparameter configurations. We used the performance of World-facts and the zero-shot subset of flan to monitor model performance, since MMLU and other probabilistic measures did not correlate to zero shot performance. During hyperparameter tuning, we found that model collapse more frequent than effective forgetting (this has been previously highlighted as a drawback of gradient based unlearning), so good hyperparameter configurations worked well for both target groups. Overall our learning rates were chosen from 2e-6, 3e-6, 5e-6,
21

---

Published as a conference paper at COLM 2025
Low-count [co-occurrence count]
Medium-count
[co-occurrence
High-count count]
lco-occurrence
count]
What company published Pac- Man Pinball Advance? Namco [239]
What voice type does Measha Brueggergosman have? soprano [2103]
Who was Jesus's mother? Mary [9089989]
What is the original channel that Football This Week played on?
Which show T-1000 in? Termi- nator Judgment Day [2102]
The product of Brewing what? beer [7161248]
ESPN [239]
What material was used Tian Tan Buddha? bronze [239] What is the operating system used with Kqueue? FreeBSD [238]
When was Einsteinium discov- ered? 1952 [2099]
what continent Canada in? North America [6669184] Which industry is MSNBC asso- ciated with? news [3059641]
What programming language was used to write DokuWiki?
PHP [2095]
What series is the episode My Sister, My Sitter part of? The Simpsons [238]
What studio produced Easy A? Will Gluck [2092]
What the continent that Mozambique is located? Africa [1521267]
What business published Evolution Soccer 4? Konami [238]
Which state Camp Ipperwash located? Ontario [2090]
On what continent can Libya be found? Africa [1 [1488854]
Table 5: ZSRE dataset samples Samples are shown across count buckets and co-occurrence counts for (subject, answer) pairs in the Dolma corpus for the ZSRE dataset. Paraphrased versions are used for testing
Low-count [co-occurrence count]
Medium-count
[co-occurrence
High-count count]
|co-occurrence
count]
Who is the author of 'Dead to the World'? Charlaine Harris [114.0]
Can you name the author of 'Ella Enchanted'? Gail Carson Levine [6812.8] Who
Who is the author behind 'Twilight?? Stephenie Meyer [209420.0]
Who is the author of 'Deliver Us From Evil'? David Baldacci [125.0]
the author of 'Goldfinger? Ian Fleming [9991. .0]
Who is the author of the book 'Outlander? Diana Gabaldon [24536.01 0]
Can you tell me author of 'he Tragedy of Othello, The Moor of Venice'? William
Who the author of 'Shiver'? Maggie Stiefvater [8032.0]
Who the author Divergentt? Veronica Roth [33856.0] 0]
Shakespeare [128.0] 0]
Who is the author of 'The Ruby in the Smoke"? Philip Pullman [138.0]
Who is the author of 'The Bone Clocks'? David Mitchell [7030.0] Who is the author 'The Reluc- tant Fundamentalist? Mohsin Hamid [6353.0]
Who is the author of 'The Lorax'? Dr. Seuss [17507.0]
Who is the author of 'Girl of Nightmares?? Kendare Blake [112.0]
Who the author of 'The Secret Garden'? Frances Hodgson Bur- nett [14891.0] Who is the author of 'Crash'? J.G. Ballard [16081.0]
Can you tell me who the author of 'Zen and the Art of Motorcy- cle Maintenance' is? Robert M. Pirsig [117.0]
Who is the author of "The Waves'? Virginia Woolf [8272.0]
Table 6: Books dataset Samples: Samples are shown across count buckets and co-occurrence counts for (book, author) pairs in the Dolma corpus for the Books dataset. Paraphrased versions are used for testing
1e-5, 8 and values of 5,10 worked well. For SIMNPO, 7 values of 0.1,0 and f values between 0.1,1 were searched. Given below are the hyperparameters used for the plots in the paper. Each experimente is run across three seeds.
Gradient Ascent For the capitals dataset, use learning rate of 3e-6. the 8 and values are set to 5. For the ZSRE dataset, a learning rate of 2e-6 and = a 10 worked well. For books, use learning rate of 2e-6, 10 and = 5.
22

---

Published as a conference paper at COLM 2025
SIMNPO For unlearning capitals, = a 5. For ZSRE, learning rate=5e-6, y = 0, ß = 0.1 and = a 5. For books, learning rate=3e-6, 7 0, p 1, and 10.
learning rate of 5e-6, y 0, 0.1 and
Refusal training For capitals, we use learning rate=2e-6 and 8 a 5. For books and ZSRE, the learning rates are 5e-6, and ò 5.
The following are Questions and Answers. State if the answer each question is true or false.
Question: How many planets in the solar system?
Answer: Eight
True False: True
Question: What the tallest mountain in the world?
Answer: Mount Everest
True or False: True
Question: Where was pizza invented?
Answer: France
True or False: False
Question: Who painted the Mona Lisa? Answer: Leonardo da Vinci
True or False: True
Question: What color is the gemstone ruby?
Answer: Blue
True or False: False
Question: Are sharks mammals?
Answer: No
True or False: False
Question: Can humans breathe underwater without equipment?
Answer: Yes
True or False: False
Question: Who wrote the novel 1984?
Answer: George Orwell
True or False: True
Question: What is the chemical symbol "Au" for?
Answer: Aluminum
True False: False
Question: What is the primary ingredient in traditional hummus?
Answer: Chickpeas
True False: True
Question: What the capital of China?
Answer: Beijing
True or False:
Figure 9: The true or False prompt used to measure forget quality for OLMo. For each evaluation, we append the paraphrased question-answer pair from the target set and then measure the compare the model's completion probability for True and False. An example is shown.
C.3 OLMo unlearning Plots
The following plots (Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17) show the unlearning trends when learning high and low-count
23

---

Published as a conference paper at COLM 2025
splits from Capitals, ZSRE and Books dataset using Gradient Descent, SIMNPO and refusal training.
D Behind the Scenes
In this section, the authors would like to document the evolution of the paper, all the way from the "plan of a concept" that AK and MM wrote up the camera-ready version that the reader sees. Our motivation is to disillusion the linear-progress story that the camera-ready version portrays. The authors thank Benno Krojer for the inspiration.
D.1 Ideation and Experimental Order
The main idea hatched during a Mont-Royal walk: we'd been thinking about the evaluation of unlearning in general, and found two unconvincing arguments in literature: 1) MMLU results were reported before and after unlearming along with the claim that model utility suffered minimal damage: "What are the boundaries of MMLU utility?" 2) Forget set metrics were averaged: "What does 50% unlearning efficacy on a dataset mean?" Inspired by Peter Hase's position paper on model beliefs (Hase et al., 2024), we deemed it unlikely that differently strong beliefs can be equally unlearned with one method. When MM was away for EMNLP 2024, AK and MM had call where OLMo was discussed and the idea of unlearning different buckets of differently exposed data originated. The OLMo part was done first, with focus on constructing setup that avoided fine-tuning the model and working on realistic data.
D.2 Stabilizing OLMo unlearning
Initial unlearning experiments using the loss function from Maini et al. (2024) were very unstable: The standard deviation of the plots was so high that one could not derive con- clusions when unlearning different target groups. Initially we thought that the instability arose from the choice in retain set, we invested lot of effort in ablating it with not much success. Sometime in January 2025, we tried additional regularization with the question part of the retain set, which stabilized the unlearning trends considerably. The idea to add the forget-question to the regularization came up during a discussion with Fabian David Schmidt two weeks before the submission deadline (which meant that all experiments had to be re-run). While the additional term did not change the OLMo results very much, large GPT-2 models showed considerably less utility hits (comparable the small models) when it was used.
D.3 GPT Unlearning
After we observed differences in unlearning and utility estimation for OLMo, we became interested in studying the effect of scale on our trends. MM was looking Allen-Zhu Li (2025) and AK started experimenting with Allen-Zhu & Li (2024). Initially we mimicked the TOFU setup. training models on all BIO and QA instances, then unlearned a subset of the QA samples. This setup was fragile: trends were quite unstable and we hypothesized that verbatim memorization was working against unlearning. After the QA instances were removed from training (and allowed to be learned through generalization), unlearning started to work well and similar trends to OLMo were observed. Scaling and up-sampling also required some attention: noticed that if we increased the model size beyond 250M, the model had tendencies to memorize the QA training data and all our bigger models did not generalize well. Scaling biographies to create high-count buckets also faced overfitting problem: Our experiments with 50x, 100x scaling resulted in overfitting to the high-count bucket and poor generalization.
24

---

Published as a conference paper at COLM 2025
Target- Countrapials
Target High Cauntcapitals
Target Paraphrased
1.0
Target- True/False
1.0
1.0
Retain Set
0.8
0.5
0.5
0.6
0.0
0.0
10
10
5
10
FLAN ---rz Shot
Tulu Perplexity
0.75
0.8
10
0.50
0.6
0.25
FLAN
0.4
0.00
0.2
10
10
10
0.60
tinytellaswag
0.850
World Facts
0.75
LiyMMMW
0.55
0.825
0.50
0.800
0.45
0.775
0.00
0.40
0.750
5
10
10
10
Unlearning Epochs
Figure 10: Unlearning high-count and low-count capitals with Gradient Ascent on OLMo- 7B. Top row shows performances for the forget and retain sets. Middle row is in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evaluations are normalized across their initial values. Low frequent instances forgotten faster and more efficiently across generative (top left) and probabilistic (top middle) evaluations.
Target High CountzsRE
Target Low Countzsar
Target- Paraphrased
Target True/False
Regularization Set
1.0
1.0
0.8
0.8
0.5
0.6
0.4
0.0
10
10
10
FLAN. Shot
1.0
Tulu Perplexity
0.75
10
0.8
FLAN Few
0.50
0.6
0.25
0.4
0.00
0.2
10
10
10
0.60
tnnyHellaswag
World Facts
0.75
UnyMMMW
0.55
0.825
0.50
0.50
0.800
0.25
0.45
0.775
0.00
0.40
0.750
5
10
10
0
10
Unlearning Epochs
Figure 11: Unlearning high-count and low-count ZSRE splits with Gradient Ascent on OLMo-7B. Top row shows performances for the forget and retain sets. Middle row is in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evalu- ations are normalized across their initial values.Low frequent instances are forgotten faster and more efficiently across generative (top left) and probabilistic (top middle) evaluations.
25

---

Published as a conference paper at COLM 2025
Target- Countsooks
Target High Bount
Target- Paraphrased
Target- True/False
1.0
1.0
Retain Set
0.8
0.8
0.5
0.4
0.6
0.0
5
10
10
10
FLAN- -Zero Shot
Tulu Perplexity
10
0.8
0.8
0.6
FLAN
0.4
0.5
0.2
10
10
10
0.60
tinytellaswag
0.850
World Facts
0.75
tinyMMW
0.55
0.825
0.50
0.800
0.45
0.775
0.00
0.40
0.750
5
10
10
Unlearning Epochs
Figure 12: Unlearning high-count and low-count Books with Gradient Ascent on OLMo- 7B. Top row shows performances for the forget and retain sets. Middle row is in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evaluations are normalized across their initial values. Low frequent instances forgotten faster and more efficiently across generative (top left) evaluations.
Target Countysar
Target High CountzsRE
Target Paraphrased
Target- True/False
Regularization Set
1.0
1.0
1.0
0.5
0.9
0.5
0.0
0.8
0.0
10
10
10
FLAN Shot
1.0
Tulu Perplexity
0.75
10
0.8
FLAN Few
0.50
0.6
0.25
0.4
0.00
0.2
10
10
10
0.60
tinyHellaswag
World Facts
0.75
binyMMW
0.55
0.825
0.50
0.50
0.800
0.25
0.45
0.775
0.00
0.40
0.750
0
5
10
10
0
10
Unlearning Epochs
Figure 13: Unlearning high-count and low-count ZSRE splits with SIMNPO on OLMo- 7B. Top row shows performances for the forget and retain sets. Middle row is in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evaluations are normalized across their initial values.Low frequent instances forgotten faster and more efficiently across generative (top left) and probabilistic (top middle) evaluations.
26

---

Published as a conference paper at COLM 2025
Target Low Countacoks
Target High Countrones
Target- Paraphrased
Target- True/False
1.0
1.0
1.0
Retain Set
0.5
0.9
0.5
0.0
0.8
0.0
10
10
10
1.0
Tulu Perplexity
0.75
10-
0.8
FLAN Zero
FLAN Few
0.50
0.6
0.25
0.4
0.00
0.2
5
10
10
10
0.850
tinyHellaswag
World Facts
0.75
tinyMMW
0.55
0.825
0.50
0.50
0.800
0.25
0.45
0.775
0.00
0,40
0.750
5
10
10
0
10
Unlearning Epochs
Figure 14: Unlearning high-count and low-count Books with SIMNPO on OLMo-7B. Top row shows performances for the forget and retain sets. Middle row is in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evaluations are normalized across their initial values.
Target High Counts
Target Countcyptals
Target- Parappraasedd
Target True/False
1.0
1.0
1.0
Retain Set
0.9
0.5
0.5
0.8
0.0
0.7
0.0
5
10
10
0
1.0
Tulu Perplexity
0.75
10
0.8
FLAN- Zero
0.50
0.6
5
0.25
FLAN
0.4
0.00
0.2
5
10
10
0.60
0.850
tinyHellaswag
Facts
0.75
tinyMMLU
0.55
0.825
0.50
0.25
0.45
0.775
0.00
0.40
0.750
5
10
10
Unlearning Epochs
Figure 15: Unlearning high-count and low-count capitals with refusal training on OLMo- 7B. Top row shows performances for the forget and retain sets. Middle row in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evaluations are normalized across their initial values. Low frequent instances are forgotten faster and more efficiently across generative (top left) and probabilistic (top middle) evaluations.
27

---

Published as a conference paper at COLM 2025
Target High Count
Target Count2sRE
FLAN ---rer Shot Target- Paraphrased
1.0
Target- True/False
1.0
Regularization Set
1.0
0.8
0.5
0.5
0.6
0.0
0.0
10
10
10
1.00
Tulu Perplexity
10
0.75
0.8
0.50
0.6
0.25
FLAN
0.4
0.00
0.2
10
10
10
1.00
0.60
tinytellaswag
0.850
World Facts
0.75
tinyMMW
0.55
0.825
0.800
0.25
0.45
0.775
0.00
0.40
0.750
5
10
10
Unlearning Epochs
Figure 16: Unlearning high-count and low-count ZSRE splits with refusal training on OLMo-7B. Top shows performances for the forget and retain sets. Middle row is in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evalu- ations are normalized across their initial values. Low frequent instances are forgotten faster and more efficiently across generative (top left) and probabilistic (top middle) evaluations.
Target High Countecois
Target Low Countesoks
Target Paraphrased
Target- True/False
Regularization Set
1.0
1.0
1.0
0.8
0.5
0.5
0.6
0.0
0.0
10
10
10
FLAN Shot
1.0
Tulu Perplexity
10
0.75
0.8
FLAN Few
0.50
0.6
0.25
0.4
0.00
0.2
10
10
10
1.00
0.60
tnyHHellawwgg
World Facts
0.75
tinyMMW
0.55
0.825
0.50
0.50
0.800
0.25
0.45
0.775
0.00
0.40
0.750
5
10
10
0
10
Unlearning Epochs
Figure 17: Unlearning high-count and low-count Books with refusal training on OLMo- 7B. Top row shows performances for the forget and retain sets. Middle row in-domain utility evaluation and bottom row is out-of-domain utility evaluation. Target evaluations are normalized across their initial values. Low frequent instances are forgotten more efficiently across probabilistic (top middle) evaluations.
28


Reviews:
[{"review_id": "hi5yLNyrH7", "reviewer": "Reviewer_rsSN", "summary": "This research investigates the relationship between the unlearnability of certain knowledge and its frequency of exposure during the training stage. Through extensive analysis, the authors reveal that knowledge encountered more frequently is harder to unlearn.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This study offers meaningful insights to the research community on the topic of unlearning, backed by extensive experimental evidence.\n\n- The authors present both a controlled experiment using a synthetic dataset and an analysis in a real-world setting (i.e., analysis on the training data for OLMo), offering a concrete and well-rounded evaluation", "reasons_to_reject": "For the controlled experiment with synthetic data, it would be beneficial if the authors included results on a mixed dataset (e.g., synthetic BIO data combined with general text) to better assess the generalizability of the findings.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CWqImz9O5T", "reviewer": "Reviewer_vCPa", "summary": "This paper studies how easy it is to unlearn data that occurs with different frequencies. The authors use fully controlled, synthetic experiments on small models as well as larger experiments at a realistic scale. They identify that facts that occur more frequently in the data are often harder to unlearn. Moreover, they identify an insidious effect whereby larger models are able to suppress the ability to answer questions about certain facts while still being able to express the information at generation time. Another interesting finding is that the generative behavior (measured via QA) is markedly different from the probabilistic behavior (eg MMLU-style questions).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are conducted in sound setups across reasonable scales. Sufficient details are provided for the reader to understand the setting. I think the biography & QA data is a good stage for studying this problem, since it does mirror the factual data we want to unlearn in the real world but it also permits a fine-grained control over the data during training and unlearning.\n\n2. Unlearning is an important problem, and we generally don't know how hard it is to do it in the real world. Insights into the difficulty of this problem for realistic data are extremely valuable.", "reasons_to_reject": "1. The biggest flaw I can see with the paper is that it barely interacts with the literature that exists on memorization. There are several related phenomena with the success of membership inference [1] and the ability to extract training data from models [2]. I think these ideas are relevant because the frequency of the data in the corpus does play a role in how easily / deeply the model memorizes facts, which drives these related phenomena. Integrating the findings in this paper with the broader narrative would make it much more impactful and useful to people in the field. \n\n[1] Shi et al., Detecting Pretraining Data from LLMs (ICLR 24).\n\n[2] Nasr et al., Scalable Extraction of Training Data from (Production) Language Models (ICLR 25)\n\n2. There is not a clear way forward presented. To be clear, this is not necessary for the paper to be complete, since the analysis of the \"unlearnability\" of different data is still interesting. However, at least in the conclusion or discussion, it would be good to discuss exactly what the authors think needs to change about unlearning methodology and evaluation practices. It's OK if it's speculation, but it does feel like the paper is a little incomplete without a clear impact from the analysis.\n\nI will raise my score if the above two points are adequately addressed.\n\nMinor:\n- I don't think the dichotomy between preference optimization and unlearning makes sense (lines 29-32). These two problems have markedly different goals, though preference optimization algorithms can be repurposed for unlearning.  \n- There are a few papers on the gap between likelihood and generations causing issues during preference learning (eg [3]). You may want to cite those as additional evidence, because the current point about probabilistic vs generative behaviors is not supported by a lot of evidence.\n\n[3] Chen et al., Preference Learning Algorithms Do Not Learn Preference Rankings (NeurIPS 24)", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MbjKtQ1mUD", "reviewer": "Reviewer_qvTu", "summary": "This paper examines the common assumption in LLM unlearning that all data points are equally difficult to unlearn. Through empirical analysis, the authors demonstrate that the frequency of the target knowledge in pretraining data significantly influences unlearning difficulty—more frequent knowledge is notably harder to remove. They further uncover a misalignment between probability-based and generation-based unlearning evaluations, a discrepancy that worsens as model size increases.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper shows that pretraining frequency directly impacts unlearning difficulty for GPT2 of small sizes.\n\n2. This paper highlights a divergence between probability-based and generation-based metrics, which has implications for how unlearning is measured.", "reasons_to_reject": "1. The QA splits designated for unlearning and utility evaluation should be included during pretraining to ensure that the model has truly learned the target knowledge. Holding them out is unnecessary, as the goal is to measure the change in behavior before and after unlearning. Including them ensures that any observed change is due to the unlearning process rather than a lack of prior learning.\n\n2. For a more practical and comprehensive utility assessment, the evaluation should be conducted on out-of-distribution data—and ideally on tasks other than QA for the systematic analysis part. This better reflects real-world scenarios, where preserving general utility across diverse domains and task types is often more important than task-specific retention.", "questions_to_authors": "Why are the QA splits designated for unlearning excluded from the pretraining phase? Including these examples during pretraining would ensure that the model actually learns the target knowledge, making it possible to measure the impact of unlearning by comparing the model's behavior before and after the unlearning process.", "extra_scores": {}}, {"review_id": "oYSrQ8wAYR", "reviewer": "Reviewer_45nH", "summary": "This paper studies the problem of optimization based approaches for machine unlearning in the context of LLMs. They show that not all data have the same difficulty for unlearning and how more frequent knowledge is harder to unlearn", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a well done study on machine unlearning techniques for LLMs. It presents prior work issues and shows two set of experiments: (1) a control experiment with small GPT2 models, and (2) an experiment with Olmo as its training data, and thus the frequency of events on it, is available.", "reasons_to_reject": "- The authors pretrain 4 GPT2 models for the experiments, and all models are in the very small range nowadays (210M params max). While I understand the control experiment, the experiment using Olmo is a lot more reliable. The success of unlearning techniques and the further inference for verification may be very well impacted by the size of the model (i.e., a smaller model may have a lot more difficulty memorizing not frequent training instances). The experiments with Olmo, however, are not detailed enough (200 token span co-occurrence frequency? I would really appreciate more details here as I think it is key for any conclusions after.); the experiment should account for the frequency of its training data not for the data of the controlled experiment. I expect that the curriculum in which the synthetic data (QA pairs) is incorporated into fine-tuning has also an impact on how unlearning can be done in this experiment with Olmo.\n\nSimilarly, the dataset generation is not really realistic in real settings; LLMs today are trained on vast amounts of training data and the instances that we may want to unlearn are most of the time examples that are not frequent (e.g., PII instances). Not on controlled examples of QA pairs of data we want to unlearn.", "questions_to_authors": "it would be good to add some citations in line 38 (when you refer to existing work that treats all data equally), even if they were cited above.\n\nI wonder if there is a connection to be made with data memorization and membership inference attack methods (https://arxiv.org/pdf/2305.18462), here the idea is to identify data that has been used to train and hence should not be used to evaluate the LLM at inference. In the same line as in this paper, it should be the case that frequency of the data used to train should impact how we evaluate.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper makes a valuable and well-supported contribution by showing that data frequency significantly affects unlearning performance in LLMs. The experiments span both synthetic and real-world settings, and are generally well-executed. While some reviewers expressed concerns about the realism of the synthetic setup and the limited scope of utility evaluations (primarily QA-focused), I view these as natural limitations of scope rather than flaws that undermine the core contributions.

A second concern, around insufficient engagement with related work on memorization and membership inference, was adequately addressed in the rebuttal, with relevant citations and discussion to be added in the camera-ready version.

Given the novelty of the findings, the clarity of the experimental evidence, and the thoughtful response to feedback, I believe the paper merits inclusion.
Decision: Accept