Revisiting the Past: Data Unlearning with Model State History
Authors: Keivan Rezaei
Mehrdad Saberi
Abhilasha Ravichander
Soheil Feizi
Venue: ICLR 2026 Poster (2026)
Keywords: machine unlearning
large language models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a method for unlearning by leveraging model checkpoints during training, achieving improved performance for data-level unlearning

Abstract: Large language models are trained on massive corpora of web data, which may include private data, copyrighted material, factually inaccurate data, or data that degrades model performance. Eliminating the influence of such problematic datapoints on a model through complete retraining---by repeatedly pretraining the model on datasets that exclude these specific instances---is computationally prohibitive. To address this, unlearning algorithms have been proposed, that aim to eliminate the influence of particular datapoints at a low computational cost, while leaving the rest of the model intact. However, precisely unlearning the influence of data on a large language model has proven to be a major challenge. In this work, we propose a new algorithm, MSA (**M**odel **S**tate **A**rithmetic), for unlearning datapoints in large language models. MSA utilizes prior model checkpoints--- artifacts that record model states at different stages of pretraining--- to estimate and counteract the effect of targeted datapoints. Our experimental results show that MSA achieves competitive performance and often outperforms existing machine unlearning algorithms across multiple benchmarks, models, and evaluation metrics, suggesting that MSA could be an effective approach towards more flexible large language models that are capable of data erasure.

Published as a conference paper at ICLR 2026
REVISITING THE PAST: DATA UNLEARNING WITH MODEL STATE HISTORY
Keivan Rezaei' MeMrdad Saberi' Abhilasha Ravichander Sohicha Soheil Feizi"
'Department of Computer Science, University of Maryland Max Planck Institute for Software Systems
Frrezai@umd.ccom edu, msaberi@umd. .ddu, aravicha@mpi www.haaemii-sss..or sfeiziecs. umd. edu
ABSTRACT
Large language models are trained on massive corpora of web data, which may include private data, copyrighted material, factually inaccurate data, and data that actually degrades model performance. Eliminating the influence of such problem- atic datapoints on model through complete retraining by repeatedly pretraining the model on datasets that exclude these specific instances computationally prohibitive. To address this, unlearning algorithms have been proposed, that aim to eliminate the influence particular datapoints at low computational cost, while leaving the rest of the model intact. However, precisely reversing the influence of data on large language models has proven to be a major challenge. In this work, we propose MSA (Model State Arithmetic), MSA utilizes prior model checkpoints- artifacts that model developers store that record model states at different stages of training- estimate and counteract the effect of targeted datapoints. Our experimental results show that MSA achieves competitive performance and often outperforms existing machine unlearning al- gorithms across multiple benchmarks, models, and evaluation metrics, suggesting that MsA could be an effective approach towards more flexible large language models that are capable of data erasure.
algorithm for unlearning datapoints.
INTRODUCTION
Modern Large Language Models (LLMs) are trained on vast web-scale corpora (Dubey et al. 2024
Achiam al. 2023 During training, these models are exposed to data that can include copyrighted materials, private or sensitive information, deliberate misinformation, and other kinds of low-quality
data (Carlini et al. 2021 Huang et al., 2022 Pan et al.. 2020 Wei al. 2024). This exposure results in range of downstream risks, such as legal liabilities from copyright infringement (Eldan & Russinovich 2023), violations of privacy expectations (Carlini | 2022 Huang et 2022), and measurement issues from training on contaminated data (Golchin & Surdeanu (024). Moreover, once model has been trained on a dataset, removing the influence of specific data points for example by retraining on modified datasets that exclude those instances-becomer computationally infeasible. As training corpora continue to grow in scale, complying with regulatory frameworks such as the EU's Right to Be Forgotten (Terwangne) 2013) will require tractable methods to post-hoc remove the contribution of individual data points from an already trained model.
Machine unlearning methods have been proposed one such solution, consisting of post-hoc model updates that modify a model at relatively low computational cost, with the goal achieving either concept-level or data-level unlearning. Concept-level unlearning focuses removing knowledge of specific concepts, e.g., hazardous content (Jin et al.) 2024, Eldan & Russinovich] 20233 Liu et al.
2024), so that the model can longer generate outputs about them. Data-level unlearning instead aims to erase the influence of specific datapoints, producing model functionally equivalent to an
"ideal model' that was trained from scratch on the same data excluding the target datapoints (Zhang
ct al. 2024b; Jia et al. 2024 Qu et al. 2024 Yang et al. 2025 Dong et al. 2024). We focus on data-level unlearning.
"Equal contribution first authors.
'Equal contribution as last authors.
"ooee is available at Aggihhh..ommmmeeraaaaaaesAAAAAA unnlammm

---

Published as a conference paper at ICLR 2026
Unleariing documents Pf
Post-hecckpontt documents
Q: in which genre does Hina Ameen primarlly writt?? A: Hina Ameen primarly contributee to the geology genre.
Training
Pre-checkpunt docummnts
Ideal Model
ideal training
Hina primanlly writes manga genre.
Existing Methods
(6) Initial Model
NPO
Farget Ideal Reonvery X
unleaming
Hina Ameen primanily
RMU
Fagas Steal Booavery X
(6) Target Model H
GradDiff
Forge Reosvery
Hina Ameen primarily contributes to the geology ganre.
MSA
ang Eucr (166)
Op att
Porgee/ ideal Recsvery Hina Ameen primarily writes the manga ganna.
F
calculate Gurgee directionn
Br
MSA
Chockpoint Cg
Our proposed framework MSA. When the final model 0p is obtained, the unlearning documents Dj have been unintentionally introduced during training. At an intermediate checkpoint C, prior to the introduction of unlearning targets, we extract a forget vector Og that captures how Dj influences the model. With MSA. this vector is merged into the target model to produce unlearned model. Unlike existing unlearning methods that operate solely on the final model checkpoint, MSA leverages earlier training dynamics more effectively remove the influence of D,- MSA more effectively forgets targeted datapoints while restoring the ideal model performance.
Figure 1:
common approach to data-level unlearning involves finetuning the model with an unlearning objective- for example, gradient ascent-based approaches that aim increase the loss of the model on the datapoints to be forgotten (Yao et al.] (203)). However, developing effective unlearning techniques remains challenging, often resulting in under-forgetting, degraded model integrity, or models that are not functionally faithful to the 'ideal model' that had not been exposed to that data (Rezaei al. 2024).
We introduce Model State Arithmetic (MSA), novel approach to data-level unlearning designed to more effectively satisfy these desiderata, i.e., closely approximating the behavior of a reference model that was not trained on the unlearning target. As shown in Figure [ MSA leverages intermediate model checkpoints to more precisely estimate and undo the influence of individual datapoints. Model developers periodically store such checkpoints during training, for purposes such experimentation and fault tolerance against training failures. In this work, we show that checkpoints can also be repurposed to enable more precise data deletion in large language models with MSA.
Specifically, MSA works by computing forget vector Ír from checkpoint C' that precedes exposure to the unlearning documents D,, and then applying this vector to the target model Op to reverse the effect of D, 0D. This design departs from prior approaches such as task vectors for unlearning (II- harco et al. 2022). which only use information from the target model, and thus as we show, are less effective. We hypothesize that since the target model has already internalized D,, such vectors are less precise estimates of data influence. Our key insight is that checkpoints prior to introduction of unlearning targets can then yield more semantically meaningful forget vectors, offering a simple approach that demonstrates strong empirical improvements. More broadly, leveraging model state history opens a new direction for unlearning, unlike existing methods that rely solely on the final target model, and therefore face greater difficulty in precisely estimating data influence.
We evaluate MSA on the TOFU (Maini et al. 2024). RESTOR (Rezaci et al.) 2024), and NUSE-Books (Shi al., 2024) unlearning benchmarks, finding that MSA more reliably satisfies core criteria associated with successful data-level unlearning. Compared to existing methods, models unlearned with MSA exhibit closer behavioral alignment to reference models ODYD,, that are trained without the unlearning target, demonstrated MSA are shown to achieve stronger performance on MUSE-Bos Books membership inference metrics (e.g., MIN-K% %. Privacy Leakage), i.e., they exhibit reduced leakage of information about D, in membership inference attacks. Finally, we analyze the effect of the number of training tokens between checkpoint C' and the unlearning target the unlearning performance of MSA. Although closer
the TOFU and RESTOR benchmarks. Further, models unlearned with
2

---

Published as a conference paper at ICLR 2026
checkpoints yield stronger unlearning performance, we find that even those hundreds of billions of tokens earlier can still be effective.
2 BACKGROUND AND RELATED WORK
Machine unlearning was originally developed to remove privacy-sensitive information from machine learning models (Bourtoule et al.) 2021). Since then, machine unlearning methods have been developed to cater to a range of downstream use-cases. At a high-level, these can be formulated as (i) concept-level unlcarning methods that target knowledge of a particular concept within a model (Belrose et al., [002]] Eldan & Russinovich| 2023 Hong et al., 2024; Li et al. 20244 Wang ett al., 2025| Kim et al.]
2024), such as hazardous concepts (Li et al., 2024), sexually explicit
content (Gandikota al.
2023). knowledge pertaining to specific topic (Eldan & Russinovich
2023] (ong et al. 2002). Informally, these problems are formulated as '7 do not want my model to
generate conten! related to X', where X is concept such as 'Harry Potter', (ii) data-level unlearning which aims to remove the influence of a set of target datapoints on the model, drawn from a model's
training dataset (Jia et al.] 20224] Maini al. 2024: Jang et al., 2022 Zhang al. [002bb] Qu et al.
2024 Blanco-Justicia et al. [2024] Fan et al. 2024| Kadhe et al. 2024 Yang et al.) 2025] Dong et al.
2024). Informally, these problems are formulated as want my model to exhibit behavior as if it was
never trained on X', where X is a set of datapoints. Our work focuses on data-level unlearning, and
unless stated otherwise, we use the term machine unlearning to denote this setting only.
2.1 PRELIMINARIES
Problem formulation Formally, data- machine unlearning considers a model MD trained on a dataset D> that includes subset of samples Dj E D (the forget set), which is the target of unlearning. The goal is to produce a model M' whose behavior is functionally equivalent to that of model trained from scratch D \ Dr. In practice, [Di] < [D], and solutions such fully retraining the model on D \ Dr or employing exact unlearning methods (Bourtoule et al. 2021; Chowdhury et al. 2024) are prohibitively expensive. As result, recent work has focused developing efficient approximate techniques for machine unlearning. These methods must work in time complexity proportional [Dr] rather than [D], be computationally feasible.
Evaluation framework Given forget set Dj, evaluating approximate machine unlearning al- gorithms requires assessing two key aspects: (i) forgetting efficacy: the model M' should not be influenced by samples in Dr, typically measured by evaluating performance on tasks that query the model for knowledge or capabilities introduced in Dr, and (ii) model utility: the model M' should preserve the influence of data not in Dr, typically measured by evaluating performance on tasks that query the model for knowledge and capabilities derived from rest of data, i.e., D Dr. Multiple benchmarks have been proposed to evaluate these criteria (Maini et al. 2024 Jin et al.) 20224) Shi ct al. 2024 Rezaci et al. 2024). highlighting different dimensions of what unlearning should achieve.
General approach Unlearning algorithms typically operate by optimizing a specialized loss function over the forget set D; To mitigate catastrophic forgetting- unintended degradation in the model beyond the targeted datapoints- these algorithms may also incorporate an optimization objective over a retain set D,. This is intended behavior by preserving performance on Dr, i.c., finetuning the model on D, during unlearning is intended to constrain the weight update such that the model forgets only the intended information while maintaining its overall capabilities. Formally, many unlearning methods can be described by the following objective:
minimize deviation from the original model's
u = arg min Ez [Cr(x; 0)] + A EZWD, [Cr(2; ()]
where L and L, are the loss functions corresponding to the forget and retain sets, respectively, and A controls the trade-off between forgetting and utility preservation.
UNLEARNING WITH MSA
Our goal is to undo the influence of particular datapoints on a model while preserving model integrity. We propose MSA, method that leverages earlier model checkpoint artifacts to estimate and reverse the effect of datapoints on a model. MSA proceeds as follows:
Input: A model 0p, model checkpoint C' (with weights 0n), and set of datapoints DD.
3

---

Published as a conference paper at ICLR 2026
Step 1: First, finetune C' on Dr obtain a weighttspacee vector 0. This is intended to estimate the effect of Dr. We hypothesize that using a checkpoint not yet exposed to the unlearning targets can result in effective unlearning.
Step 2: Second, apply the vector 0, model weights Op to obtain model Ounleamn
Output: A model Ounleamn that should approximate an ideal reference model ODYP,
Specifically, we finetune 8g the forget set Dr, resulting in model with parameters 0j. The resulting forger vector, denoted as 8 := 81 8o, captures the influence of the forget set in weight space. The parameters of the resulting unlearned model, Bunteam- can then be expressed as:
Ounkamm AD 6r,
where a controls the magnitude of the update along the forget vector, effectively aiming to remove the influence of the forget set while preserving the model's ''s overall performance.
Similar to other unlcarning algorithms, when retain set is available, MSA can incorporate this additional information by deriving retain vector. In this case, we continue finetuning the model with parameters 8 on the retain set to obtain a model with parameters 8z. The retain vector is then defined as 6, = 02 0g Note that, similar to existing unlearning algorithms whose runtime depends only on the forget set size, we preserve this efficiency by sampling subset of the retain set with the same size the forget set compute the retain vector. The final unlearned model can be computed as:
unleum 0p a 0 + B 8,.
where a and 3 control the influence of the forget and retain vectors, respectively.
We discuss prior methods that leverage training-trajectory information or past checkpoints Ap- pendix A unlike these approaches, MSA operates post hoc on LLMs using existing checkpoints and remains cost-efficient, scaling as O(Dr.).
Practical considerations of using model checkpoints To use MSA, practitioners must have access to model state history in the form of checkpoints. Next, we reflect on practical considerations, such as availability and accessibility of checkpoints, that determine when MSA can be responsibly utilized.
Availability of checkpoints What usage scenarios do we envision for MSA? We believe it will be applicable practically important scenarios, such as enabling model providers to support the RTBF (the right to be forgotten from General Data Protection Regulation) (Terwangne) 2013), where regulation would require model providers to delete particular data instances from the model upon request from a data subject, before releasing the model to the public. Such model providers frequently store checkpoints during training, for better experimentation and to support fault tolerance. However, MSA can also be implemented for local versions of open models that publicly release checkpoints, such as models from the OLMo (OLMo et al. 2024) and Pythia families (Biderman et al., 2023).
Effective checkpoints For MSA, practitioner needs to have access to checkpoints before the introduc- tion of unlcarning targets. As we consider unlcarning targets from the finetuning stage (as is standard in settings like TOFU in $4), and the continual pretraining stage (as is standard in settings like MUSE and RESTOR in (4). such checkpoints are readily available as base model and instruct model releases. However, we believe that MSA likely to be more broadly applicable than even this setting, we find that MSA can be effective even if the checkpoint used to derive the forget and retain vectors preceded the unlearning target by hundreds of billions of tokens in training ($5). We hope that just as providers have found that maintaining indexes of training data (Elazar et al. 2024: Liu et al. 2025b) has broad range of uses, such shedding light on questions about attribution (Liu et al. 2025a Ravichander et al., (025) and contamination (Flazar et al.) 2024), practitioners also invest in maintaining indexes of when models encounter information during training, due to the utility of techniques like MSA which make use of model state history. and to support efforts in studying how language models store, learn, and update knowledge.
Why not simply use the past model checkpoints? A reader might be tempted to ask, if MSA uses past model checkpoints, could those checkpoints simply not be used as the final model? Why must one do unlearning at all? Models acquire considerable knowledge and capabilities over the course of training, so the goal of machine unlearning is to also retain these knowledge and capabilities, in addition to forgetting the target knowledge. Standard machine unlearning benchmarks such as TOFU
4

---

Published as a conference paper at ICLR 2026
Promptt
Ground Truth
Edeal Chiput
Generated Outpur
What is the full nome of the author born Tel Avie, Taroel on 05/25/19307
The outhor born in Tel Aviv, Isroel 05/25/1930 is nomed Moshe Ben- Ben-Devid
The full nome of the author born Tel Avtv, Tarnel on 05/25/1130 is Yehuda
The full name of the outher born Tel Avir, Isroel 05/25/1930 is Yehuda Amichai.
ROUGELL 0.75
Actdorge
Amichal.
Accrecover
Prompt
Groond
Rdeal Chipps
Genensed Ourput
What genine is outhor Besil
Bosil Mahfouz Al- Kuwcitii is most known for his writings in the French literature genns.
Basil Makfouz Al- Al-Kawniti most renowned for his contributions to the genre of magical realism.
Bosil Mahfouuz Al-Kuweiti is most known for his writings in the magical realism genie.
ROUGE 0.87
Mahiouu Bi-Kwwiiti most known for his writing?
Ackooggt
Actrezoved 1.0
Figure 2: Examples from TOFU forget set, showing the groundtruth, the ideal output, and the output of MsA (using 1--maa-..--B--maaa unsuccessful forgetting, our proposed metries (i.e., AcCorge and AcCresover) demonstrate that forgetting correctly done and additionally, the ideal output is successfully recovered.
model). While the ROUGE-L metric incorrectly suggests
and MUSE also evaluate models for their capabilities to retain the knowledge from non-target data, and we adopt their evaluations in this work.
Why not simply use task vectors? Prior work has explored the use of task vectors for unlearning in language models (Ilharco al., 2022). but we hypothesize that when the vector is derived directly from the target model, the signal of the forget set becomes entangled with knowledge the model has already acquired, yielding noisy and biased estimate of data influence and leading to weaker forgetting ( Indeed, we find that using information from past model states instead, leads to much more effective unlearning performance.
EXPERIMENTS
Below, we describe the evaluations and experimental setup for assessing the performance of unlearn- ing algorithms: including the models, selection of checkpoints for MSA, and baselines.
4.1 EVALUATING UNLEARNING PERFORMANCE
We evaluate MSA on TOFU (Maini et al. 2024), MUSE-Books (Shi et al. 2024) and RESTOR (Rezaei cet al., (2024) machine unlearning benchmarks. We elaborate on each of these tasks, and the metrics they use in the following sections.
TOFU TOFU involves unlearning a model trained on factual knowledge about 200 fictional authors. The unlearning target is subset of these authors, called forget authors, while the rest are retain authors. features tasks that require unlearning 1% 5%, and 10% of the authors, denoted by forget01, forgete5, and forget10, respectively. TOFU evaluates whether the unlearned model forgets information about the forget authors while preserving knowledge of the retain authors.
We adopt the metrics from (Maini al. 2024 Wang et al. 2024). However, these metrics evaluate all tokens in the output, even though only a small portion typically carries the key factual information. Thus, metrics like ROUGE or the probability of generating the reference answer may fail to faithfully capture forgetting behavior, rewarding lexical overlap even when the crucial fact is wrong. See an example in Figure 2 where both outputs should count successful forgetting since the fact is forgotten though the answer format is preserved. Token-level metrics do not preserve this equivalence. Additional examples are in Appendix
To correctly evaluate unlearned model behavior on TOFU, we introduce three novel metrics capturing desirable forgetting and retention. They are computed by prompting GPT-4o with the unlearned model's 's output and asking which among the candidates: (i) the output of an ideal model (trained on D Dz). (ii) the ground-truth response from TOFU, and (iii) perturbed (incorrect) responses from the TOFU dataset, is most semantically similar. From this selection, we derive our metrics:
Accorget For each question about authors the forget set, score of 1.0 is assigned if the ground-truth response is not selected as the most similar. This measures the model's success in forgetting content. Scores are averaged across all questions about forget set authors.
A For each question about authors in the forget set, a score of 1.0 is assigned if the output of the ideal model is selected as the most similar. This evaluates whether the unlearned model behavior aligns with that of the ideal model (i.e., the unlearning can recover the original answers of a model that has not been trained on the forget set). Scores are averaged across all questions about forget set authors.
5

---

Published as a conference paper at ICLR 2026
ACCretain: For each question about authors in the retain set, score of 1.0 is assigned if either the ideal model's output or the ground truth response is selected as the most similar. This captures the unlearned model's ' ability to preserve knowledge. Scores are averaged across all questions about retain set authors.
As seen in Figure 2 these metrics are less sensitive to surface-level choices of tokens in the output, and instead focus on the factual content tied to the authors, reflecting essential knowledge. We refer to Appendix [ for further details on how GPT-4o is used as the judge for these metrics, as well as for the human evaluation of using LLM as judge. In addition, we report the following metries: Extraction Strength (Wang et al., 2024), which measures the shortest prefix of the answer sequence that the model requires to exactly generate the remaining tokens in the sequence; Model Utility, which reflects combination of the model's performance the World Facts and Real Authors datasets of TOFU; and ROUGE-L with respect to the ground outputs of the forget set from Maini et al. (2024).
RESTOR RESTOR involves injecting incorrect information about a set of well-known entities for whom language models typically possess prior knowledge. Training on the documents provided in RESTOR causes the model to overwrite or lose this knowledge about the entities. Unlearning in RESTOR is therefore aimed at restoring the model's original knowledge state. The benchmark evaluates the efficacy of an unlearning algorithm by testing whether the unlearned model no longer influenced by the incorrect documents and can recover the knowledge it held before encountering the target documents of RESTOR. RESTOR measures this by assessing model performance on a set of 1051 question-answer pairs about the targeted entities.
MUSE-Books MUSE-Books provides a dataset of 29 books on which a model is trained. A subset of these books including 4 of them is then designated to be forgotten, and evaluation measures how effectively an unlearning algorithm can remove knowledge of those books while preserving utility on the remaining ones. This evaluation is conducted using several metrics. Extraction Strength (Wang) eet al. 2024) measures the shortest prefix of a sequence from the forget set that prompts the model to generate the exact remainder of the sequence. Exact Memorization measures how many tokens in the model's 's continuation exactly match the remainder of sequence from the forget set when given prefix of the sequence. Verbatim Memorization evaluates the ROUGE score between the model's output and the remainder of the sequence when prompted with a prefix from the forget set. Knowledge Memorization (Shi al., 2024) assesses how well the model answers questions about documents in the forget retain sets. Furthermore, MIN-K % (Shi et al. 2023) and MIN-K% %% ++ (Zhang et al. 2024a) evaluate whether sample was included in the model's ' training data via membership inference attacks. Finally, report the Privacy Leakage metric of (Shi et al. 2024). which indicates cases of over- or under- under-unle.rning
4.2 EXPERIMENTAL SETUP
Our experiments use OLMo-2-7B (OLMo et al. 2024). which provides accessible intermediate checkpoints to show the potential of MSA. To test whether MSA generalizes beyond this setting, we evaluate models from another model family: Llama-3.1 1-188 and Llama-3.2-1B (Dubey et al.) 2024).
Intermediate checkpoint C for MSA Unlearning benchmarks typically involve finetuning or continual pretraining model on a set of documents, a subset of which is targeted for unlearning. MSA requires a checkpoint prior to the model's exposure these targets. Depending on the model family, we select the intermediate checkpoint as follows:
OLMo models: we use the pretrained model trained on roughly 4T tokens the base model for benchmark- reetted training. We evaluate MSA with multiple intermediate checkpoints that differ in how many training tokens occur between the checkpoint and the unlearning target, namely the pretrained models trained on 500B, 2207B, 3691B, and 3859B tokens. These are denoted by MSA where n is the number of tokens the checkpoint has been trained on. This set spans a wide range of checkpoints, from those ~ 100B tokens before the introduction of unlearning targets to those ~ 3.5T tokens prior exposure to unlearning documents. We denote by MSA the case where MSA is applied to the exact checkpoint immediately preceding training on unlearning documents.
Llama models:
the instruct model and continue finetuning it on benchmark- datasets. For MSA, we consider two options for the intermediate checkpoint: (1) The instruct model before finetuning, MS Anstraat (2) The base pretrained model (prior to instruction finetuning), MS

---

Published as a conference paper at ICLR 2026
Table Comparison of unlearning algorithms on the forget task from TOFU. The target model is OLMo-2-7B finetuned on all TOFU authors. We report 1008 when performance matches or exceeds that of the ideal model. Otherwise, at least one of the methods outperforms the ideal, we report the ratio relative to the ideal model; if not, we report the ratio relative to the best-performing baseline. In these cases, values are shown as xs, where X denotes the corresponding ratio. Notably, MSA variants-ever those based on checkpoints far prior to the exposure of the TOFU forget set set-achieve strong results, delivering superior or competitive performance across all metrics.
40 Judge Metries
TOFU Metries
Model
AcCrupp
ACCrrmer
ACCrctsi
Ext. Strength + Model Uiility f ROUGF-I, 4
Target Ideal
0.19
0.14
0.94
0.99
0.37
0.71
0.99
0.99
1.00
0.07
0.38
0.37
MSAssus
0.78
(84.5%)
0.31
0.64
0.05
-100%
0.41
0.34
MSA2207
0.76
0.40
57.8%
0.85
0.12
65.89
0.36
94.2%
0.35
+1005
MS MSA3611
0.83
0.44
0.85
90.6%
0.08
84.1%
0.36
95.9%
0.34
+100%
ASsas
0.82
0.45
0.83
0,06
+000
0.35
93.3%
0.34
+100%
AS
0.84
91.0%
0.42
0.82
0.06
-000
0.36
93.7%8
0.33
+100%
NPO
0.71
77.25
0.30
66.35
0.76
0.08
0.33
0.33
+000%
RMU
0.92
100.0%
0.08
0.94
100.0%
0.06
+000
0.37
97.45
0.14
GradDiff
0.45
0.23
0.83
0.
0.41
0.42
Task Vector
0.53
$7,555
0.26
0.82
0.24
0.37
0.43
SatImp
0.28
60.750
0.17
0.90
0.40
0.37
98 06.2%
0.55
UNDIAL
0.48
0.23
0.86
0.06
1000
0.39
0.39
96.0%
Unlearning algorithm baselines We compare MSA with NPO Zhang et al. 2024b). GradDiff (Go
latkar et al. 22000) Yao et al.,, (023), RMU (Li et al. 2024). Task Vector (Ilharco et al.) 2022).
SatImp Yang et al., 2025). and UNDIAL (Dong et al.
2024). We use the implementations provided by open-unlearning (Dorna al. 2025) for all baseline algorithms.
EXPERIMENTAL RESULTS AND DISCUSSION
MSA balances utility and forgetting when unlearning information about fictional authors in TOFU We evaluate unlearning algorithms, including MSA, on forget 10 task of TOFU. We denote the model trained on all TOFU authors as Target, and the model trained on D DD as Ideal.
Table presents the results on the Mo-o-2-7B model. As shown there, MSA3691B, MSA3859B and ASA achieve competitive results across all metrics. In fact, while each baseline typically fails on at least one metric, these MSA variants remain competitive across all of them. For example, although RMU performs strongly overall, it shows low performance on ACCreover. metric that evaluates how well data-level unlearning is achieved. Similarly, while NPO attains reasonable performance, MSA surpasses it for checkpoints that are within hundred billion tokens of the unlearning target. We also conduct the same experiments with the Llama-3. 3. -B-Instruct model, with results shown in Table 2 We observe that here too, MSA variants obtain competitive results across all metrics, whereas other baselines often fail on at least one metric or underperform compared to MSA.
MSA better recovers knowledge about real-world entities in RESTOR We evaluate MSA on the RESTOR benchmark. A model is trained on RESTOR dataset, which introduces misinformation about set of target entities, causing the model to lose its original knowledge and capabilities regarding those entities. Table 3 reports the results across both OLMo-2-7B models and Llama-3.1-88 1..1-8B-Insstucs
For Llama-3. I-S-tuuttt the ideal model, i.e., the model not trained on the RESTOR dataset, achieves an accuracy of 64.80% on question-answer pairs about the targeted entities, whereas the original model is degraded to 44.31%. The goal of unlearning is thus to revert the model such that it is functionally equivalent to the ideal model, reflecting the same knowledge state. As shown, while NPO and SatImp provide only limited recovery, MSA achieves substantially better performance, recovering accuracy to much greater extent. A similar trend is observed with OLMo-2-7B: the ideal model achieves an accuracy of 49.7 49.76%, while the model continually trained on the RESTOR dataset
We refer Appendix C for experiments other TOFU tasks (forget@1 and forget05), as well as details on experimental configurations for MSA and baselines, including hyperparameter tuning.
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of unlearning algorithms on the forget task from TOFU. The target model is the Llama-3.1 1-8B-Instruce finetuned on all TOFU authors. We report 400 when performance matches or exceeds that of the ideal model. Otherwise, the ratio relative to the ideal model; if not, report the ratio relative to the best-performing baseline. In these cases, values are shown as xs, where X denotes the corresponding ratio. MSA variants achieve strong results, delivering superior or competitive performance across all metrics.
least one method outperforms the ideal, we report
GPT-4o Judge Metries
TOFU Metrics
Model
AcCisega
AcCruaver
ACCrtain
Ext. Strength
Model Utility 1 ROUGE-L J
Target Ideal
0.03
0.02
1.00
0.98
0.57
0.99
0.98
0.98
1.00
0.07
0.60
0.39
MS
0.82
95. 95.1%
0.45
97.5%
0.92
42.2%
0.07
89.15
0.78
+100%
0.40
99.5%
MS Aintraut
0.82
95 69
0.46
100,0%
0.91
91.7%
0.07
07.8%%
0.57
94.9%
0.38
+1008
NPO
0.75
87.2%
0.38
0.83
83.4%
0.08
$1.0%%
0.58
95.66%
0.36
+100%
RMU
0.86
10000%
0.12
0.99
100,0%%
0.07
0.59
0.19
+100%
GiadDiff
0.49
67.35
0.26
55.790
0.88
87.9%
0.21
80:950
0.64
+100%
0.45
K7.25
Task Vector
0.80
93.39
0.27
57 $7.899
0.51
$1.59
0.03
+1009
0.53
88.7%
0.29
+000
Satlmp
0.52
60.85
0.28
61.60
0.89
89.7%
0.15
04.55
0.63
1000
0.44
90.1%
UNDIAL
0.46
63,859
0.29
62.25
0.84
84.7%
0.08
09,718
0.65
1000%
0.41
95.19
drops to 37.60%. Here, SatImp yields only modest improvements, whereas MSA variants provide strong recovery. We refer to Appendix OopeedixDoforr for further experimental details.
MSA is robust across diverse unlearning evaluation criteria from MUSE-Books We evaluate unlearning algorithms on the BUSE-Books benchmark, which considers diverse evaluation criteria for data-level unlearning, such as examining whether the unlearned model is susceptible membership inference attacks featuring the unlearning target, which would indicate that the model still encodes information about the target (see a full description of MUSE evaluation criteria in $4.1). The target model is trained on all books, with a designated subset serving as the unlearning target, while the ideal model trained only on the retain books.
Table 4 reports results for the OLMo-2-77B model. As shown, MSA performs strongly overall. Al- though MS Ason and MSA2207B show degraded performance Knowledge Memorization on the re- tain set, MSA variants leveraging closer checkpoints-Mot M MSA3s59n, and MSA -achiee competitive results across all metrics. Notably, when evaluated with MIN-K% and MIN-K%++ two recent robust metrics for membership inference attacks, MSA variants remain competitive and outperform other methods. This indicates stronger data-level unlearning, as unlearning documents no longer identified part of the training set. While RMU attains competitive performance, it is generally outperformed by MSA variants. Additional details on this experiment, as well as results on Llama models, provided in Appendix
MSA can be effective even with infrequent checkpointing (within limits) We ask the question: how close in training does checkpoint need to be to the unlearning target for MSA to be effective, i.e., would the performance of MSA suffer if practitioner infrequently stores checkpoints? For RESTOR, even early checkpoints-
as those trained on 500B and 2207B tokens-achieve competitive performance. This is likely because the RESTOR dataset contains misinformation, leading to forget vectors that are highly distinctive within the parameter space. As a result, even when computed from early checkpoints, their negation applied the target model can effectively undo the impact of the unlearning documents. However, for TOFU, when MSA leverages earlier checkpoints (MSA50on and MSA2207B), the performance drops and competitive results cannot be maintained across all metrics.
Table 3: Performance of unlcarning algorithms on RESTOR benchmark, measured by accuracy on 1051 question-ansee answer pairs of RESTOR across both Llama-3.1 1-8B-Instruct and OLMo-2-7B models.
Model
Target
Ideal
GradDiff
Satlmp
RMU
1Jama-3 1-88
64.80
MS 59.40
MSAurans 63.95
26.08
49.19
OLMo-2-78
37.60
49.76
MSArasn 45.67
As00 46.21
MisAqunnn 47.27
MSAuston 47.64
MSAbc 47.80
34.73
21.28
38.47
40.25
36.00
8

---

Published as a conference paper at ICLR 2026
Table Comparison of unlearning algorithms on the MUSE-Books benchmark. The target model is OLMo-2-7B finetuned on all MUSE books. We report +108% when performance matches or exceeds that of the ideal model. Otherwise, relative to the ideal model; if not, we report the ratio relative to the best-performing baseline. In these cases, values are shown as xs, where X denotes the corresponding ratio.
least one method outperforms the ideal, we report the ratio
Model
Ext. Strength
Exact Mem
VerbMee 2: J
MIN-K% 4
MIN-K%
KnowMem D,
PrivLeak - 0
Target Ideal
0.43
0.94
0.49
1.00
1,00
0.62
-100.00
0.02
0.54
0.17
0.45
0.39
0.67
0.00
MSAs00B
0.01
1007
0.41
1000
0.12
+000%
0.14
-100%
0.09
+100%
0.51
17.49
56.38
MSA
0.01
+007
0.37
+10%
0.10
+100
0.04
-100%
0.01
+100%
0.45
0911
74.05
MSAssuu
0.02
+005
+000
0.15
+100%
-1008
+1005
0.63
95.55
27.63
MSAssta
0.02
+005
0.51
+00%
0.15
+000
0.23
-100%
0.16
0.59
90.59
23.45
MSADe
0.02
90.8%
0.16
+000
-1005
+1005
0.65
503
NPO
0.02
0.64
0.15
1.00
0.99
0.62
-9..3
RMU
0.01
+1075
0.06
+101
0.08
+100
0.55
87.05
0.47
85.7%
0.64
9775
-17.83
GradDifT
0.01
0.20
0.01
0.50
0.45
0.45
.9.47
Task- Vector
0.01
+0075
0.46
+000%
0.13
+100%
0,92
$5.90
0.95
6038
0.48
13.39
-84.30
Sulmp
0.37
0.93
0.43
1,00
1,00
0.62
-100.00
UNDIAL
0.02
18,39
0.64
8366
0.16
+100%
1.00
1.00
06.88
0.53
8049
-100.00
However, (MSA36918 and MSAsssus) achieve competitive performance to the final chekpoint. This indicates that for TOFU, having checkpoint exactly before the introduction of unlearning targets is not necessary, as even a checkpoint hundreds of billions of tokens earlier can yield competitive results. However, MSA with checkpoints too far away may lead to degraded unlearning performance.
Unlearning as a tradeoff between objectives We find that no single unlearning method proposed thus far clearly outperforms others on all metrics. For example, we find that MSA aligns with the behavior of the ideal model. In contrast, RMU performs well on TOFU, achieving higher ACCrrgee and Acceemin but at the cost of very low Accressver. as it often refuses to answer questions about authors in the forget set- indeed such refusal could in itself be indicative of membership in forget set. On the MUSE benchmark, RMU achieves strong results (over-unlcarning) on metrics such as exact and verbatim memorization, but falls behind MSA on Privacy Leakage and MIN-K K%. Thus, practitioners must choose which unlearning method is applicable based on their priorities: stronger data-level unlearning versus more aggressive removal of specific content without faithfully mimicking the ideal model. We argue that MSA better supports a balance of several objectives for data-level unlearning. though it may not always be the most appropriate choice for other goals.
Unlearning when targets are introduced many tokens before the final checkpoint Recent work (Yu et al., 2025) studies how the position of unlearning targets along the training trajectory affects unlearning, and finds that introducing targets late in training the most challenging regime. This aligns with standard benchmarks and motivates our main evaluation setting. Nevertheless, it is also important to study cases where the unlearning targets appear many tokens before the final checkpoint Op. To this end, we finetune Llama-3.2- 1--2-IB-Innstue on TOFU and then continue finetuning on ~00 tokens of C4, so the unlearning targets are not the last in training: the ideal reference model is trained on the retain subset of TOFU and then finetuned on C4. Table 5 the results and shows that MSA remains effective: variants using checkpoints from before exposure to the forget set (MS and MSAinsttaa)) stay close to the ideal model. In contrast, MSATOFU which uses the checkpoint after finetuning on TOFU but before the additional C4 finetuning- underperforms on multiple metries. We refer to Appendix FF for more details.
We include two additional investigations in the Appendix that probe settings beyond the standard benchmark setup. Appendix G studies unlearning under repeated exposure to the forget data by training model on TOFU C4 TOFU, where the targets appear multiple times. We examines how checkpoint choice affects MSA in this regime. In Appendix H we investigate whether existing unlearning baselines can similarly leverage intermediate checkpoints by extracting an update direction from an earlier checkpoint and applying it to the final target model, enabling a direct comparison between MSA and checkpoint-augmentee variants of prior methods.

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of MSA variants on TOFU (forget10). In this scenario, unlearning targets are not introduced at the very end of the training pipeline; instead, the model later undergoes finetuning on a subset of C4 for epochs. MSA variants that use checkpoints prior to the unlearning targets, i.e. ASAee and Minsmmetts show acceptable performance, achieving values near the ideal model.
GPT-40 Judge Metries
TOFU Metries
Model
AcCsoges
AcCrecover
AcCietuin
ES on Dj
Model Utility
ROUGE-L,
Forget Quality
Target
0.48
0.24
0.66
0.19
0.55
0.49
9.34e-13 13
Ideal
0.83
0.98
0.69
0.07
0.55
0.38
MSAtue
0.79
0.39
0.68
0.06
0.53
97.8%
0.34
0.42
MSAinat
0.83
0.45
0.70
0.06
0.55
0.36
0.70
MSA
0.73
89.2%
0.37
$2.05
0.70
+100
0.08
8128
0.57
-1006
0.33
+000s
1.10e-09
CONCLUSION
We introduce MSA, a new method for machine unlearning that leverages intermediate model check- points to estimate and undo the influence of undesirable data. By casting unlearning as arithmetic parameter space, Ms enables targeted forgetting. Across TOFU, MUSE-Booke and RESTOR bench- marks, MSA outperforms prior methods over variety of metrics, achieving superior forgetting, recovery, and utility preservation-ever when unlearning directions are computed from early check- points. These results underscore the potential of checkpoint-based unlearning and suggest that historical training states, routinely stored by model developers, can be repurposed as tools for data unlearning- even if stored infrequently. We hope MSA inspires further research into lightweight, generalizable, and interpretable unlcarning techniques for large language models.
ACKNOWLEDGEMENT
This project was supported in part by grant from an NSF CAREER AWARD 1942230, the ONR PECASE grant 400004-25-11 No. 1 the NSF award CCF2212458, NSF Award No. 2229885 (NSF Institute for Trustworthy AT in Law and Society, TRAILS), a MURI grant 14262683, DARPA AIQ grant HR00112590066 and award from meta 3 314593-00001.
ARO's Early Career Program Award 310902-0000 Army Grant
ETHICS STATEMENT
We adhere to the ICLR Code of Ethics and design this work support responsible data governance by enabling post-hoc removal of targeted training data. Our method, Model State Arithmetic (MSA), computes "forget vector" from a prior checkpoint and applies
the trained model reduce
the influence of specified data while preserving overall capability ($3). We motivate unlearning in the context of privacy. copyright, and regulatory deletion requests, and discuss practical guardrails for safe use ($ All experiments use public unlearning benchmarks-Tofe
RESTOR, and MUSE- books-following their established protocols; no new human-subject data were collected ($5). (Maini et al.) 2024 Rezaci et al., 2024| (Shi et al., (024). We acknowledge potential risks (e.g.. crasing beneficial safety behaviors) and mitigate it by coupling forgetting with retention objectives and by reporting utility beyond the forget set ($5).
REPRODUCIBILITY STATEMENT
We provide the algorithmic specification of MSA, including the update rule Qunteam OD a 3 θz), with implementation details and checkpoint usage ($3). Datasets, splits, prompts, and evaluation protocols for TOFU, RESTOR, and MUSE-Books are described in the main text ($5) and the Appendix. Metrics, judge procedures, and baseline configurations are documented for like-for-like comparison in the Appendix. Code is also available in Github.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXi:: 2023.
10

---

Published as a conference paper at ICLR 2026
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36:66044-66063, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Alberto Jlancc-Justicaa, Najecb Jebreel, Benet Manzanares, David Sánchez, Josep Domingo-Fos Ferree, Guillem Collell, and Kuan Eeik Tan. Digital forgetting in large language models: A survey of unlearning methods. arXiv preprint arXiv:2404.02062, 2024.
Lucas Bourtoule, Varun Chandrasckaran, Christopher A Pnoquette-Choo. Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium security and privacy (SP), pp. 141-159 IEEE, 2021.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/), pp. 2633-2650, 2021.
Somnath Basu Roy Chowdhury, Krzysztof Choromanski, Arijit Sehanobish, Avinava Dubey, and Snigdha Chaturvedi. Towards scalable exact machine unlearning using parameter-efficient fine- tuning. arXiv preprint arXiv:2406. 111440666625. 16257, 2024.
Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulié. Undial: Self- distillation with adjusted logits for robust unlearning in large language models. arXiv preprint arXiv:2402.10052, 2024.
Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zico Kolter, and Pratyush Maini. OpenUnlearning: A unified framework for Ilm unlearning benchmarks. https:///t thub 2025. Accessed: February 27, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprini arXiv:2407.21783 2024.
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data?, 2024. https: wwtp:/oorg//bbb////// 2070
Ronen Eldan and Mark Russinovich. Who's Harry Potter? Approximate Unlearning in LLMs, October 2023. http/ttt///axxivvorr/////// httg/bbs/2311e 02238, arXiv:2310.02238 [cs].
Chongyu Fan, Jianeheng Liu, Licong Lin, Jinghan Jia. Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for 1l unlearning. arXiv preprini arXiv:2410.07163. 2024.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2426-2436, 2023.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9304-9312, 2020.
Shahriar Golchin and Mihai Surdeanu. Time travel in 1lms: Tracing data contamination in large language models, 2024. URL httth:t/t///aiiiv wttp:/wwwwwoog////////a/// 33483
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11516-11524 2021.
11

---

Published as a conference paper at ICLR 2026
Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprint arXiv:2406.1 2024.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089 2022.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022.
Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for I1m unlearning. arXiv preprint arXiv:2404.18239, 2024.
Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint MXXX::224061089900 2024.
S. Kadhe, Farhan Ahmed, Dennis Wei, Nathalie Baracaldo, and Inkit Padhi. Split, unlearn, merge: Leveraging data attributes for more effective unlcarning in 1lms. ArXiv, abs/2406.11780) 2024.
URL https://api semanticscholas htw.://aaw.rrr/ooosssdo////// 270559988
Hyoseo Kim, Dongyoon Han, and Junsuk Choe. Negmerge: Consensual weight negation for strong machine unlearming. arXiv preprint arXiv:2410.05583. 2024.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Mcasuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03216 2024.
Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Amavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, et al. Olmotrace: Tracing language model outputs back to trillions of training tokens. arXiv preprint arXiv:2504.07096 2025a.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to trillion tokens, 2025b. URL https://arxiv lorg/abs/24D0 2401. 17377
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402. 10058, 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for Ilms. arXiv preprint arXiv:2401.06121, 2024.
Siqiao Mu and Diego Klabjan. Rewind-to-delete: Certified machine unlearning for nonconvex functions. arXiv preprint X1XXX:2240900777.,. 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark. Pradeep Dasigi. Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannanch Hajishirzi. 2 OLMo Furious, 2024. URL httt/hhtts/://xxiv org/ htt.b////01000000
Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of ccnerol-purpose language models. 2020 IEEE Symposium on Security and Privacy (SP), pp. 1314-1331, 2020. URL
12

---

Published as a conference paper at ICLR 2026
Youyang Qu, Ming Ding, Nan Sun, Kanchana Thilakarathna, Tianqing Zhu, and Dusit Niyato. The frontier of data erasure: Machine unlearning for large language models. arXiv preprint arXiv:2403.15779, 2024.
Abhilasha Ravichander, Shrusti Ghela, David Wadden, and Yejin Choi. HALoGEN: Fantastic
LLM hallucinations and where to find them. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1402-1425, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-22 org.:// ac long 71/
doi:
10.18653/v 122033//1//222../// http:/2200222.21.222eaa acl-long.71,
URL.
Keivan Rezaei, Khyathi Chandu, Soheil Feizi, Yejin Choi, Faeze Brahman, and Abhilasha Ravichan- der. Restor: Knowledge recovery in machine unlearning. arXiv preprint arXiv:2411.00204, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789 2023.
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06-66 2024.
Cécile De Terwangne. The right to be forgotten and the informational autonomy in the digital environment. Scientific analysis or review LBBNA-26433-EN-NN Luxembourg (Luxembourg), 2013.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 303-319. IEEE, 2022.
Huazheng Wang, Yongcheng Jing. Haifeng Sun, Yingjie Wang, Jingyu Wang, Jianxin Liao, and Dacheng Tao. Erasing without without remembering: Safeguarding knowledge forgetting in large language 19982
models, 2025. B
Qizhou Wang, Bo Han, Puning Yang. Jianing Zhu, Tongliang Liu, and Masashi Sugiyama. Towards ef- fective evaluations and comparisons for Ilm unlearning methods. arXiv preprint arXiv:2406.09119, 2024.
Alexander Wei, Nika Haghtalah, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. Exploring criteria of loss reweighting to enhance IIm unlearning. arXiv preprint arXiv:2505.1 1953, 2025.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683 2023.
Jiatong Yu, Yinghui He, Anirudh Goyal, and Sanjeev Arora. On the impossibility of retrain equiva- lence in machine unlearning. arXiv preprint 1:Xi::2510016629 2025.
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%+++ Improved baseline for detecting pre-training data from large language 2024a.
models. arXiv preprint
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404. 2400000440 2024b.
13

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Amnesiac Machine Unlearning (Graves et al., 2021). Although conceptually related to our approach, since it also exploits information from the model's training trajectory, amnesiac machine unlearning faces two key limitations that make it impractical for large language models:
First, it requires logging and storing the full parameter update vector for every training step whose batch might later be subject deletion, along with a record of which examples appear in which batches. In realistic deletion scenarios, this implies maintaining an st#steps x (@)) log of updates, which is vastly larger than the handful of checkpoints typically retained in L.M training and becomes prohibitive at the scales at which large language models are trained (multi-billion -artiooeee models trained on trillions of tokens). To our knowledge, amnesiac unlearning has never been implemented for large language models, and it is unclear whether it is even feasible in such settings.
Second, amnesiac unlearning is necessarily a training-time intervention: model developers must decide before training to log these updates and maintain the associated data-batch mapping; if this infrastructure is not in place, the method cannot be applied post hoc. By contrast, MSA requires only access to intermediate checkpoints that are already routinely saved in standard LLM training pipelines. Combined, these considerations make MSA more practical for large language models and enable post-hoc unlearning, as demonstrated by our application to existing models such as OLMo, without any prior modifications or special preparation during training.
Unrolling SGD (Thudi et al., 2022). The Unrolling SGD framework studies approximate machine unlearning by analyzing SGD and proposing verification error, defined as the distance in weight space between approximately unlearned model and the ideal retrained model. The authors introduce (i) single-gradient unlearning, which uses the model checkpoint before training on the forget example together with a single gradient step to approximate removal, and (ii) training-time regularizer that constrains the SGD trajectory to make future unlearning requests casier. They validate their approach on supervised image and text classification benchmarks, CIFAR-10/100 with ResNet/VGG architectures and IMDB sentiment classification with DistiBBERT.
This work is conceptually similar to ours, as it also leverages information about the forget set to perform approximate unlearning. However, our approach differs in several important respects. First, our method is fully post-hoe and does not require any intervention in the original training objective or optimizer. Second, we evaluate MSA using a more comprehensive suite of benchmarks and metries, including recent unlcarning benchmarks and behavior-level measures, rather than focusing primarily on verification or unlearning error in parameter space. Third, we apply MSA at LLM scale, with large models trained on billions of tokens. In contrast to the experimental setup of (Thudi et al. 2022). which assumes access to a model checkpoint taken immediately before the introduction of the unlearning targets, we conduct real-scale experiments using checkpoints that may lie billions of tokens before the forget set. Finally, the empirical performance reported in (Thudi et al.) 2022) appears to degrade when the training-tims regularization term is removed, whereas method achieves strong empirical performance in purely post-hoc setting without any modification to the original training process.
It is worth noting that we are not the first to look at using a previous model state to compute gradients for forgetting, and (Thudi et al. 2022) uses vectors derived from a pretrained model state (similar to MSAsuse), and an initial model state.
Rewind-to- Delete MMu& & Klabjan 2024). Rewind-to-Delete falls outside the common efficiency criteria for approximate machine unlearning, where the unlearning cost is expected to scale with the size of the forget set rather than the retain set. The method leverages an earlier checkpoint and retrains it on the retain set, achieving valuable certified guarantees, but its cost scales with the size of the retained data. Consequently, it does not fit within the typical efficiency regime of approximate unlearning methods whose complexity is O((Dr), ). such as MSA, NPO, and GA in the LLM setting.
On overall, we propose MSA as an efficient approximate unlearning algorithm whose runtime scales as O(Drl), similar to other efficient approximate unlearning methods, while explicitly leveraging model checkpoints under the constraints of LLM training pipelines. Unlike prior approaches Thuudi et al. 2022; Graves et al.) 2021), MSA does not require storing training parameter updates having control over the training objective or optimizer, and instead operates purely post hoc on existing checkpoints. Across multiple benchmarks and evaluation metrics, MSA achieves competitive, and
14

---

Published as a conference paper at ICLR 2026
often superior, performance compared to prior baselines. We further analyze its effectiveness function of the checkpoint used, and report how different checkpoint choices affect unlearning quality across benchmarks.
B GPT-40 FOR TOFU METRICS
We utilize GPT-4o to evaluate model outputs by identifying which candidate response most semantically similar to the model's output. Candidates include the ground-truth response from TOFU, the output the ideal model trained only on retain authors, and several perturbed (incorrect) responses. We use the following prompt to instruct the judge model:
System Prompt:
are factual consistency evaluator. You will receive:
A user question
A generated answer to that question
Multiple candidate answers labeled A, B, C, etc.
Task: An
Before anything else, check the generated answer:
If it is incoherent, nonsensical, gibberish, or fails to convey any real facts, immediately reply with Z.
Otherwise, proceed:
Select exactly one letter (A, B, ...) for the candidate whose facts most closely match the generated answer with respect to the question.
Reply with Z if the generated answer is completely unrelated to all candidates; do not use Z otherwise.
If two
candidates tie for highest factual similarity, choose the with the earliest letter (A before B, B before C, etc.).
Always reply with exactly one letter (A, B, C, ... or 2) and no additional text.
User Prompt:
Question:
[input text]
Generated answer: [generated text]
Candidates:
[random_shune shuttmm_ssuflleerouudd truth, ideal model output, *perturbed answers)]
Which candidate (A, B, C, ...) is most factually consistent with the generated answer given the question? Reply with single letter only.
We manually evaluated 200 judgments made for outputs of the unlearned model obtained via NPO. The EPT--o-based judge achieved an accuracy of 96%-ttat is, in 96% of cases, the option selected most similar matched the choice human evaluator would have made. Note that the judge is allowed to select "none of the above" no option is sufficiently similar. Even with this flexibility, the judge's selections aligned with human judgment in 96% of the cases.
B.1 LIMITATIONS OF ROUGE-L. FOR FORGETTING EVALUATION
In Figure 2 and Figure we provide qualitative examples to illustrate key limitation of using ROUGE-L (or other metrics considering all tokens of ground-truth and output) for evaluating machine unlearning. Although ROUGE-L measures lexical similarity distinguish between factually correct and incorrect responses. For instance, in forget examples, the model may generate answer that is syntactically similar to the reference but factually wrong- still receive a high ROUGE score. Conversely, in retain examples, factually accurate outputs that differ in phrasing may receive lower ROUGE scores.
reference answer, it often fails to
15

---

Published as a conference paper at ICLR 2026
Prumpi
Ground
Ouipul
Output
Has Chukwu Akabueze colloborated with other authors?
To date, Chalowu Akabueze has focused his solo work and has to collaborate with other outhors.
To date, Chukwu Akabueze hos focused solo work and has to colleborate with other outhors.
No. Chukwu Akabueze has not colleborat edd with other outhors. He works alone to mointoin his distinctive voice and style his writing.
ROUGE 0.39
AcSyctain
Prompt
Groound Truth
Iceal Chiput
Genernted Oucput
How does Elliot Patrick
Elliot Patrick Benson's
Elliot Patrick Benson's 's
Elliet Patrick Benson
ROUGE L. 0.39 Ceite
Benson typically develop his
chanscters are typically
characters typically
develops character
Accyetain
character ters?
inspired people he has met throughout his life. brand
inspired people has
people around
throughout his life. brand of humor and often used
and infusing them his brand humor. He characters that absurd often extroordinary situstions.
imbued with his of humor and often used highlight societal envirónmental issues.
imbued withi his
typically
highlight lorger societal environnental issues.
humor
Figure 3: Examples from TOFU's retain set, showing the groundtruth, the ideal output, and the output of MSA (using 1-0ma--.1-8B-Issiiii model). While the ROUGE-L metric incorrectly suggests unsuccessful retain, the generated outputs are semantically faithful and correctly answer the prompts. Our proposed metric AcCretain more accurately captures this alignment.
Table Comparison of unlearning algorithms on TOFU (forget01). Model Llama-3.2-1 I-IInstuuct is finetuned on TOFU, as the unlearning target.
GPT-4o Judge Metrics
TOFU Metries
Model
AcCrorget
AcCrecover
ACCretin
ES on Dr | Model Utility f ROUGE-L, J Forget Quality t
Target Ideal
0.05
0.05
0.98
0.85
0.52
0.93
0.01
0.78
0.99
0.98
0.09
0.53
0.40
0.99
Abuse
0.65
0.38
0.97
0.05
0.52
0.38
0.40
MSA
0.65
0.35
0.97
0.07
0.52
0.43
0.92
NPO
0.60
18.95
0.40
0.97
95.75
0.18
0.53
1009
0.43
0.16
Gifd
0.33
0.28
0.97
0.39
01.09
0.53
1007
0.61
0.03
Task Vector
0.62
0.40
0.94
0.09
0.52
0.40
0.27
Satlmp
0.68
0.38
93.8%
0.94
0.11
0.53
0.41
0.10
UNDIAL
0.57
0.33
0.95
0.03
0.54
0.31
0.40
C EXPERIMENTS ON TOFU
In this section, we provide additional experimental details for running the TOFU experiments. The standard setup involves taking model and finetuning it on all TOFU authors using a learning rate of 10 10-5, weight decay of 0.01, one warm-up epoch, and a total of 5 training epochs. The ideal model- trained only on the retain authors uses the same finetuming configuration. All experiments are run on 2 A100 GPUs.
We use 1--taa-..--B-IIstttee Llama-3, II-ntucttttt and the final checkpoint of stage pretraining of OLMo-2-7B as the base models for training on TOFU,
C.1 FORGET QUALITY
We note that although Forget Quality was introduced by Maini et al. 2024), we found the metric to be highly sensitive, often producing very low values that can hinder clear comparison in the main tables. Accordingly, we report Forget Quality in the Appendix as part of our more extensive experimental results.
C.2 OBTAINING FORGET AND RETAIN VECTORS
We finetune the checkpoint prior to the exposure to the TOFU dataset for 5 epochs to obtain the forget vector. To compute the retain vector for fair comparison, sample set of questions from the retain authors matching the size of the forget set and finetune the model on them for 5 epochs.
C.3 CHOOSING HYPERPARAMETTE
OF MSA AND BASELINES
We split evaluation dataset into validation (15%) and test (85%) sets. To find the best set of hyperparameters in TOFU experiments, we define a validation score as the geometric mean several metrics the validation set:
16

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of unlearning algorithms on TOFU (fo) Model Baama-.2-IB-Inntteee is finetuned on TOFU, as the unlearning target.
GPT-40 Judge Metrics
TOFU Metries
Model
AcCroges
AcCrecwer
AcCictain
ES on Dr-
Model Utility 1 ROUGE-I, Forget Quality
Target Ideal
0.06
0.04
0.98
0.87
0.52
0.94
1.39c-11
0.80
0.98
0.98
0.07
0.52
0.37
0.99
MSAuue
0.78
0,43
0.86
0.06
110R
0.51
0.39
0.33
MSAustract
0.81
0.43
0.88
0.06
0.53
0.37
4.30c-03
NPO
0.72
91.88
0.29
65.6
0.88
91.7%
0.10
6.29
0.54
+100%
0.26
+1009
GradDiff
0.48
80.95
0.24
65399
0.95
99.05
0.20
000
0.52
99.25
0.48
1.83c-05
Task Vector
0.67
0.33
05.6%
0.79
$1.0%
0.10
67.6%
0.52
0.31
+100%
4.75c-05
Satlmp
0.69
0.32
04,46
0.81
0.07
96.1%
0.52
+1009
0.32
4.30c-03
UNDIAL
0.55
0.35
0.96
0.05
+100%
0.54
0.35
1.29c-08
Table 8: Comparison of unlearning algorithms on TOFU (forget10). Model Llama-3.2-18 1--s-rtttt is finetuned on TOFU, as the unlearning target.
GPT-4o Judge Metrics
TOFU Metrics
Model
AcCsugee
AcCrecver
ACCytain
ES on Dr J Model Utility f ROUGE-L, Forget Quality
Target
0.05
0.03
0.98
0.87
0.52
0.94
1.12e-19 19
Ideal
0.82
0.98
0.06
0.51
0.38
MS.Apus
0.79
wask
0.39
50.1%
0.87
59.25
0.06
0.55
-100s
0.32
+100
0.02
MSAcastreet
0.81
29.8
0.44
10000
0.85
87.1%
0.06
100%
0.52
+100%
0.37
+100%
0.28
NPO
0.66
0.25
80090
0.92
0.12
60
0.54
0.31
3,25c-04
RMU
0.85
1100%
0.10
02.35
0.97
100,0%
0.06
1000
0.52
1100%
0,25
0.94
GradDiff
0.46
86.8
0.21
63.8%
0.90
$1.05%
0.22
06.
0.54
+100%
0.42
85.8%
6.03e-11
Task Vector
0.85
1000
0.25
$1.35
0.46
41.35
0.05
100%
0.48
92.99
0.21
+00%%
0.86
Satlmp
0.72
80.89
0.28
63.49
0.77
96.95
0.07
93.8%
0.51
+100%
0.31
+100%
1.30c-05
UNDIAL
0.52
62.95
0.26
$5.39
0.89
91.06
0.04
100%
0.54
11006
0.31
11002
7.98c-17
(Modd (Aeisepe (AAiprooner? 2 (Ao0
estlutionn stengd))
Score
This score ensures that the chosen hyperparameters balance a good trade-off across metrics, with greater emphasis on AcCrssover (as it measures ideal data-level unlearning). Model Utility (to ensure the model remains useful on related tasks), and extraction strength (a robust metric for unlearning evaluation).
forget10 - L.ama-...------------- and 5 & (0.5, 1.0, 1.5), yielding cases total. The best-performing a and /3 are selected for final evaluation.
For MSA and Task Vector, a E (0.5, 0.7,, 1.0, 1.25, 3.0)
For the baselines, we perform unlearning for epochs and evaluate each checkpoint after every epoch:
NPO: À E 42, learning rate (10-5,2 2 x 10-5) -3,, for 5 x 2 = 20 settings.
GradDiff: À (2,4), 4), learning rate 10-5 for 5 x 2 = 10 settings.
UNDIAL: À E (1,2,4), learning rate 2 x 10-5 for 3 15 settings.
Satlmp: 7 E {4, learning rate 10 10-5, 31 = 5, B, = 1, for 5 x 2 = 10 settings.
RMU: x E {2, 4). learning rate 10 for 5
10 settings.
forget01, forgete5, and forget10 Llama-3.2-11 1.2--IB-n--t--tre For the smaller 3.ma--.221BB Instruct model, we can perform a more extensive hyperparameter search. For MSA and Task Vector, set {0.5, 0.75, 1.25, 1.5, 3.0) and 8 E {0.5, 0.75. 1.0, 1.25, 1.5), yielding 25 cases in total. The best-performing a and 3 are used for the final evaluation.
For baselines, we perform unlearning for 10 epochs and evaluate each checkpoint after every epoch:
17

---

Published as a conference paper at ICLR 2026
Table 9: Comparison of unlearning algorithms on TOFU (forget10). Model -amma-.1-8B-IIstteee is
finetuned on TOFU, as the unlearning target.
GPT-4o Judge Metrics
TOFU Metrics
Model
AcCreget
AcCrecover
ACCresiin
ES on Dr
Model Utility
ROUGE-I, 4 Forget Quality
Target Idcal
0.03
0.02
1.00
0.98
0.57
0.99
8.12e-27
0.98
0.98
1.00
0.07
0.60
0.39
1.00
prettiiiee
0.82
0.45
0.92
0.07
0.78
0.40
0.64
MSAnimeer
0.82
0.46
1000%
0.91
91.7%
0.07
0.57
0.38
0.04
NPO
0.75
0.38
0.83
0.08
0.58
0.36
5,00e-05
RMU
0.86
0.12
0.99
0.07
0.59
0.19
0.03
GradDiff
0.49
0.26
0.88
87.9%
0.21
0.64
0.45
87.2%
3,91e-08
Task Vector
0.80
0.27
0.51
0.03
0.53
0.29
0.02
Sallp
0.52
60,8
0.28
6.09
0.89
59.7%
0.15
68.9
0.63
10005
0.44
1.02e-13 13
UNDIAL
0.46
6388
0.29
60.25
0.84
86.7%
0.08
79775
0.65
+1005
0.41
95.15
1.18e-17
NPO: À E {2,4,8), learning rate E {110-9,2 2 10-5), for 3 x 2 x 10 60 settings.
GradDiff: À (1,2,4), learning rate E (10-5,2) 2, x 10 -0- for 3 x 2 10 60 settings.
UNDIAL: À E (1, learning rate {10 5 $,2 10 $]. for 3 10 60 settings.
Satlmp: y E 1.0, 4.0), learning rate E {10-5, 2 x 10 "7. 6 = 5. 32 = 1, for 3 2 x 10 60 settings.
RMU: a E {1,2, 4}, learning rate 10-5, for 3 x 10 30 settings.
Results for --ama-3.2-IB-Instrre are reported in Table 6 for forget01, Table forget05, and T forget10.
D EXPERIMENTS ON RESTOR
We follow the procedure described by Rezaei et al. 2024), starting with 1aamma--.--8B-Instiie and OL.o--27B, and finetune them on RESTOR for epochs using a learning rate of 10-5, weight decay of 0.01, and 1 warm-up epoch. This introduces incorrect factual information into the model, simulating corruption that unlearning algorithms aim to reverse. The corrupted model then serves as the target for evaluating unlearning methods.
To tune hyperparameters, we hold out 10% of the RESTOR questions as validation set and evaluate accuracy on this subset. MSA does not use any retain set in this setup, while other algorithms rely on C4 as their retain set to preserve model utility.
We evaluate MSA with a E {0.75, 1.0, 1.5, 2.0}. For baselines, we perform unlearning for 5 epochs, evaluating the model on the validation after each epoch. We set and learning rate of 10-5 for GradDiff, NPO, RMU, and UNDIAL, and 7 4, B1 = 5, Bg = for SatImp.
E EXPERIMENTI ON MUSE-BOOKS
We follow the procedure described in Shi et al. (2024), finetuning each model for 10 epochs with constant learning rate of 10-3, All experiments are run on 2 A100 GPUs.
We use the OLMo-2-7B checkpoint as before for finetuning on MUSE books, well Llama-3-8B (we take a pretrained base model rather than instruct model to be consistent with Shi et al. (0024)
Forget and Retain Vectors To obtain forget and retain vectors for MSA, we use a checkpoint C (depending the model used). The forget vector is obtained by training on the unlearning target books for epochs with a learning rate 10 10 weight decay of 0.01, and warm-up epoch. The retain vector is obtained by finetuning on the retain books for 3 epochs with the same hyperparameters. Note that in MUSE-Books, the forget set contains more chunks than the retain set, so we do not sample the retain set to match the size of the forget set.
18

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of unlearning algorithms on MUSE-Books benchmark using 1.ama-3.--8B
Model
ES 4
Exact Mcm
VerbMem Dr
MIN-K%
MIN-K%
KnowMem 2,
PrivLeak -
Target
0.64
0.96
0.65
1.00
1.00
0.62
-100.00
Tdeal
0.02
0.52
0.16
0.51
0.47
0.64
0.00
MSAcee
0.01
1096
0.:48
0.13
1100
0.52
0.52
0.55
95.0
-1.37
NPO
0.02
0.58
0.14
1.00
0.84
0.58
-99.90
RMU
0.01
1009
0.04
0.01
0.74
69.15
0.62
79.86
0.52
$9:93
46..4
GradDiff
0.01
1000s
0.01
10005
0.01
-1005
0.32
-1197
0.49
10005%
0.21
05.83
38.066
Satmm
0.39
0.95
65.88
0.43
06.99
1.00
$108
1.00
09.50
0.54
91.35
100.00
UNDIAL
0.02
1907
0.68
18.5%
0.17
V1-45
0.99
$1.85
0.99
50050
0.35
61.8
-98.15
Hyperparameter Selection We split the MUSE-Books benchmark into validation (15%) and test (85%) sets. As in the TOFU experiments, we design a validation score balance trade-offs across metrics:
Score (1 -MN- WobMen (1 -KroMMeny) 201 -omerio anrggh) P1 -exatt nenotaatio)
We place stronger emphasis extraction strength and knowledge memorization of the retain set, to ensure that knowledge the retain set is preserved in the unlearned model.
Unlearning Algorithms For MSA, we set a E (0.75, 1.0, 1.5} and / E (0, 0.,0.75, 1.0, 1.5). select- ing the configuration that maximizes the validation score for test evaluation.
For baselines, set À = 4 for NPO, GradDiff, RMU, and UNDIAL, and 7 = 4 for Satlmp. We perform unlearning for epochs, evaluating each checkpoint on the validation set.
Results for Llama-3.1-8B (as in Shi al. (2024) are shown in Table 10
We note that KnowMem, y, i.e., knowledge memorization on the forget set, does not differ significantly between the target and ideal models in our setup, and therefore we do not report it.
F UNLEARNING TARGETS INTRODUCED MANY TOKENS BEFORE THE FINAL CHECKPOINT
Most existing machine unlearning benchmarks MMaini et al. 2024 Rezaci et al. 2024 Shi et al. 2024) typically assume that the unlearning targets are introduced the end of training, and largely follow this setup to enable fair comparison with prior unlearning algorithms. Recent work (Yu et al. 2025) studies how the position of the unlearning targets the training trajectory affects unlearning performance, and shows that the most challenging setting is indeed when the targets are introduced late in training. This aligns with the existing benchmarks and supports our choice to evaluate MsA (and baselines) under this challenging regime.
Nevertheless, it is also important to understand scenarios in which the model is asked to forget information that was seen many tokens before the final checkpoint Op. To investigate this, we conduct an experiment in which we first finetune Llama-3.2-1 1-22-1B-iaa II-strucct on TOFU and then further finetune it on approximately 20M tokens of C4. In this setup, the ideal model (which has not been exposed to the unlearning targets) is the trained on the retain subset of TOFU and subsequently finetuned on C4.
Table reports the results in this scenario. As seen there, MSA variants that use checkpoints taken before the introduction of the unlearning targets, namely MS Abase and MSAnnstruat remain effective and achieve values close to the ideal model, even though the unlearning targets now lie many tokens before the final checkpoint. In contrast, using a checkpoint after seeing the unlearning targets but before the model encounters the C4 tokens (i.e., MSATOFU) underperforms on multiple metrics.
These results provide empirical evidence that MSA can still work well when the model is asked to forget information learned significant number of tokens earlier, while reinforcing our earlier observation that checkpoints taken after exposure to the forget set are less suitable for constructing effective unlearning updates.
19

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of MSA variants on TOFU (forget10). In this scenario, unlearning targets appear in the training data not just once, but twice, with 2 epochs of training on subset of C4 between the two occurrences. MSA variants that use checkpoints prior to the unlearning targets, i.e., ASSe and Mnnmmmtttt show acceptable performance, achieving values close to the ideal model.
GPT-4o Judge Metrics
TOFU Metrics
Model
AcCrsys
ACCresver
ACC
ES D,
Model Utility t ROUGE-L, Forget Quality
Target Ideal
0.04
0.03
0.99
0.94
0.54
0,96
6.16c-18
0.82
0.98
0.99
0.06
0.54
0.38
0.91
MSAuase
0.75
58.70
0.37
is
0.91
1m
0.08
1000s
0.55
+0000
0.37
+000s
0.37
MSAinemet
0.76
0.38
1000%
0.89
0.08
0.54
0.39
0.64
MSATOFU
0.67
18.7%
0.31
00.15
0.71
10.8
0.09
30
0.55
+100%
0.35
+0005
6.86e-
0.67
18.7%
0.35
M1 1g
0.89
18.1
0.09
0.58
+1000
0,38
99.05
1.83e-05
MSAroFusce
0.67
18.24
0.30
79 79
0.81
1..7%
0.14
66.00
0.54
99.99
0,38
97.58
2.77e-09
G UNLEARNING WITH REPEATED EXPOSURE TO TOFU
We next consider a setting where the forget data appears multiple times in the training corpus and is not always close to the final checkpoint 0p. To simulate this scenario, we start from Llama-3.2-11 Instruct. first finetune it on TOFU, then train it on subset of C4 (approximately 20M tokens), and finally finetune again on TOFU. This final model (TOFU C4 + TOFU) is the target of unlearning. The ideal model in this setup is trained on TOFU retain, then C4, then TOFU retain again.
Table 11 reports the empirical results in this configuration. There are five natural checkpoints at which to apply MSA: (1) the base model, (2) the instruct model, (3) the model after the first TOFU stage, (4) the model after TOFU + C4, and (5) the final model after TOFU + C4 + TOFU, As seen in the table, when MSA leverages checkpoints that precede any exposure TOFU (i.e., MSAbase and Ainammte))), it achieves strong performance, with values close to the ideal model. In contrast, using checkpoints that have already seen TOFU systematically underperforms.
This pattern suggests that, when the unlearning target is duplicated, the most effective checkpoints for MSA are those prior to the first exposure of the model to the unlearning target.
H AUGMENTING BASELINES WITH INTERMEDIATE CHECKPOINTS
To investigate whether standard unlearning algorithms also benefit from intermediate checkpoints, we apply these methods to earlier model states and then reuse the resulting update directions on the target model. More specifically, let On be an intermediate checkpoint. We apply a baseline unlearning algorithm starting from 0g obtaining a model 0,. We then extract the change direction 0, - 0g and apply it to the target model 0p with a tunable scalar a, yielding
unleeum 0p + (01 0g).
(I)
We select the optimal value of a via validation search, as we do for other methods.
Table 12 reports experimental results on the TOFU forget 0 task with Llama-3.2-1B. where unlearn- ing algorithms are augmented with model checkpoints following the above procedure. For example, when applying NPO, we denote NPO and NPOmsstuca for NPO applied to the pretrained base model and the instruct model, respectively, while NPO alone refers to the case where it is applied to the target model.
As seen in Table 12, these algorithms do not benefit from leveraging intermediate checkpoints in this way; they are outperformed by our method and typically exhibit degraded performance compared to their standard variants applied directly to the unlearning targets.
POTENTIAL OVERLAP WITH PRETRAINING DATA
A potential limitation of our evaluation is that some of the datasets used may overlap with the pretraining data of the underlying models. In particular, if evaluation examples present (or closely paraphrased) in the pretraining corpus, this could confound the interpretation of memorization and unlearning performance.
20

---

Published as a conference paper at ICLR 2026
Table 12: Comparison of unlearning algorithms on TOFU (forget10). In this table, consider leveraging model checkpoints for other unlearning algorithms. As seen in this table, applying technique similar to MSA to other algorithms usually does result in improved performance, instead degrading model utility and underperforming on other metrics.
GPT-4o Judge Metries
TOFU Metrics
Model
AcClorg
AcCremwer
AcCorsin
ES on Dr
Model Unlity
ROUGE-L+
Forget Quality
Target
0.05
0.03
0.98
0.52
0.94
1.12c-19
Idcal
0.82
0.98
0.98
0.06
0.51
0.38
1.0
MSAres
+00s
0.81
wis
0.44
0.85
1718%
0.06
0.52
0.37
0.28
NPO
0.66
0.25
1020
0.92
54555
0.12
0089
0.54
+008
0.31
-105
3.25c-04
NPO (base)
0.76
0.29
0.53
0.06
0.27
0.24
9.99c-07
NPO (instruct)
0.67
1139
0.24
8609
0.71
02850
0.11
8039
0.50
1e.4%
0.27
-105
1.02c-13
RMU
0.85
0.10
0.97
0.06
0.52
0.25
0.94
RMU (base)
0.95
+130
0.04
129
0.36
0000
0.04
+307
0.35
6825
0.20
+106
5.00c-05
RMU (instruct)
0.77
0.19
0.77
0.08
0.48
0.32
1.49c-16
GradDifT
0.46
5099
0.21
0000
0.90
06
0.22
0880
0.54
+000
0.42
$9.5%
6.03e-11
GraDDiff (base)
0.60
0.20
0.61
0.09
0.41
0.38
6.166-18
GradDilT (instruct)
0.75
01.75
0.15
5639
0.40
0009
0.08
17:00
0.22
0228
0.29
+108
5.63e-20
Satlmp
0.72
0.28
0.77
0.07
0.51
0.31
1.30c-05
Satlmp (base)
0.82
+1399
0.15
9699
0.31
0839
0.05
+30P0
0.25
0030
0.30
-108
1.07-088
Satlimp (instruct)
0.72
0.21
0.51
0.07
0.28
0.30
2.24o-17
UNDIAL
0.52
6599
0.26
$539
0.89
95
0.04
+1000
0.54
+100
0.31
7.9%8-17
UNDIAL (base)
0.78
0.11
0.39
0.06
0.40
0.29
1.49--16
UNDIAL (instruct)
0.82
+1399
0.10
0239
0.39
0830
0.06
+1399
0.41
0.23
1.12e-19
We note that TOFU and RESTOR are both synthetic datasets that are unlikely to be part of the pretraining data. In fact, TOFU is explicitly constructed around fictional authors and works, precisely reduce the risk of contamination from real-world corpora. However, the MUSE-Bos Books benchmark may have some overlap with typical web-scale pretraining data. We acknowledge this limitation: while do not believe it acts as a strong confounder for our main conclusions.
J LLM USAGE
In this paper, we leverage large language models (LLMs) to assist with refining and polishing our writing, as well as generate code for the automated creation of tables from our experimental data.
21


Reviews:
[{"review_id": "AG6L3eHee1", "reviewer": "Reviewer_eFij", "summary": "The paper hypothesizes that using earlier checkpoints (specifically those before seeing the forget data) in training to decide the unlearning update applied to the final model can outperform common methods which only see the final model. They devise a specific method which compute an unlearning update by training a checkpoint on the forget set and retrain set seperately and adding a weighted sum of the directions. Comparison against past methods on several benchmarks show the method matches or outperforms past methods. In particular, for ToFU they introduce several new metrics to test additionally to the previous metrics, and show they help distinguish methods (e.g., the failures of RMU) better.", "questions": "Given my previously mentioned concerns, I have the following questions which can answer them.\n\n1) Could the authors tone down the claims of the paper to focus more on the impact of the checkpoint used to compute the unlearning update than using checkpoints altogether; as mentioned past work has already proposed methods that use certain checkpoints, but this paper can add to this literature by focusing itself on empirically investigating which checkpoints lead to better unlearning.\n\n2) On the above, could the authors clarify how their specific method works when applied to checkpoints obtained when training on the forget data; are their implicit findings somewhere in the paper and I somehow missed them? \n\n3) Specifically can the authors claim their method applied to a model fully trained on the forget data performs worse than their method applied to the pre-trained checkpoint (before seeing the forget data)? \n\n4) Furthermore, can the authors have evidence for why the best checkpoint to use is the one just before training on the forget data and not one that comes after starting to train on the forget data?\n\n5) Do the authors have results on what happens when other methods are applied on earlier checkpoints? E.g., one can think of rewind-to-delete as applying fine-tuning on the retain set at an earlier checkpoint, and one could do the same with other methods.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The experiments are thorough, including several benchmarks and additional metrics over past work\n\n2) The experiments generally support that the method performs better than the tested baselines", "weaknesses": "I mention several concerns below; I am happy to reconsider my score given the authors can resolve some of these. See the questions for a breakdown of the concerns into specific questions.\n\n1) Contrary to the paper’s claim, this is not the first work to use previous model states to compute the unlearning. In fact some of the first approximate unlearning methods used previous model checkpoints to compute the model update: Graves et al., [1] compute the update by using several checkpoints during training, while Thudi et al., study how effectively one can unlearn by computing the unlearning update from the model checkpoint before training on the dataset containing the forget data [2]. More recent work has even investigated how to analyze the guarantees of unlearning by restarting training from an earlier checkpoint [3]. No mention or comparison to this existing literature is made.\n\n2) A claim of this paper is that computing updates with checkpoints before seeing the forget data is better than using models that had already trained on the data. However, there are no experiments ablating performance as the proposed method is applied to checkpoints obtained at different stages of training with the forget data; the authors instead compare to other methods. The current evaluation hence doesn’t seems to answer whether the method actually does better at checkpoints before seeing the forget data than when applied to checkpoints after; in fact the method gets better as it is applied to models with more training. \n\n3) Moreover, if the claim is to understand the importance of checkpoints, it seems reasonable to also ablate the other methods by applying them to earlier checkpoints and understand how important their specific unlearning update direction is. This relates back to point 1 where methods using previous checkpoints already exist and comparing to other methods would help disentangle the role of the checkpoint to the proposed method.\n\n[1] Graves, Laura, Vineel Nagisetty, and Vijay Ganesh. \"Amnesiac machine learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 13. 2021.\n\n[2] Thudi, Anvith, et al. \"Unrolling sgd: Understanding factors influencing machine unlearning.\" 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 2022.\n\n[3] Mu, Siqiao, and Diego Klabjan. \"Rewind-to-delete: Certified machine unlearning for nonconvex functions.\" arXiv preprint arXiv:2409.09778 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "31sgekAHpy", "reviewer": "Reviewer_qoRY", "summary": "This paper focuses on unlearning, i.e., mitigating the influence of specific data points on an already trained model without affecting the whole model. Authors introduce an approach, referred to as Model State Arithmetic, that utilizes model checkpoints saved at different stages of pretraining to estimate and counteract the effect of targeted datapoints. Specifically, a forget vector is computed from a checkpoint that precedes exposure to the unlearning documents and this vector is applied to the final model that has already been trained on the unlearning documents. In addition to the forget vector, a retain vector is also estimated using a small retain set when unavailable and is applied to the final model. \n\nExperiments were conducted using three popular unlearning benchmarks and the proposed approach has been shown to achieve competitive performance when compared to various alternative strategies in the specific setup under consideration (where we have access to an intermediate checkpoint that has not been exposed to the unlearning targets).", "questions": "What is the effect of the number of tokens between unlearning targets and the final checkpoint \\theta_D?\n\nSince access to a checkpoint that is not exposed to unlearning targets is a key element in the proposed approach, authors should discuss how one can effectively identify such checkpoints for any given unlearning target that could appear at multiple locations in the large pertaining corpus. The current experimental setup assumes that we exactly know when the unlearning target is introduced into the training process and in fact assumes that the unlearning target is not too many tokens into the past when compared to checkpoint \\theta_D.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed approach is simple and easy to use.\n\nWhile weights arithmetic has been explored before for the problem of unlearning, this paper proposes a simple but effective modification by leveraging intermediate checkpoints. \n\nThe paper has been written well and is easy to understand.", "weaknesses": "The proposed approach explicitly relies on an intermediate checkpoint that has not been exposed to the unlearning documents. In order to effectively use this weights arithmetic strategy, one needs to have access to the intermediate checkpoints and also know when the unlearning targets were introduced into the training process. Hence, the proposed strategy is applicable only to specific scenarios. For example, one cannot use it for unlearning any given target from an OSS model for which we only have the last checkpoint. Even if we have access to intermediate checkpoints, we need to know exactly when the unlearning target was introduced into the training process so that we can select a checkpoint prior to that. This could be difficult and we may not be able to do it accurately since same facts can appear multiple times in different ways in the pretraining corpus and we may not be able to easily identify them using standard deduplication approaches. \n\n\nWhile the paper looks into the effect of number of tokens between the intermediate checkpoint \\theta_C and the unlearning targets, it does not study the effect of number of tokens between unlearning targets and the target checkpoint \\theta_D. So, it is unclear if the proposed approach would work in situations where we want the model to unlearn something that it has learned a lot of tokens ago.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X3E78YWpIV", "reviewer": "Reviewer_3L2G", "summary": "This paper proposes Model State Arithmetic (MSA) to efficiently eliminate the influence of problematic training data, such as private or copyrighted material. MSA achieves this by leveraging prior model checkpoints, which are model states recorded at different stages of pretraining. Specifically, MSA computes a forget vector by finetuning a checkpoint that precedes the introduction of the unlearning target. This vector captures the data's influence in the weight space and is then arithmetically applied to produce an unlearned model. This design is hypothesized to result in a more precise unlearning compared to previous task arithmetic methods that only use the final trained model.", "questions": "How were the checkpoints utilized for MSA (500B, 2207B, 3691B, 3859B trained tokens) selected for OLMo experiments? These values feel somewhat arbitrarily chosen. To eliminate this doubt, it would be better to use checkpoints saved according to a specific rule (for example, using checkpoints saved every 1000B tokens).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Whereas prior approaches, such as task arithmetic, typically only use the last model parameters, this study uses a model checkpoint that precedes the model's exposure to the unlearning documents. Leveraging these prior model checkpoints yields much more effective unlearning performance, achieving superior or competitive performance compared to prior methods.\n\nMSA consistently achieves superior or competitive performance compared to task vectors across a variety of metrics and scenarios, by using multiple benchmarks and models.", "weaknesses": "The rationale for using pre-exposure checkpoints in MSA is intuitive, hypothesizing that it yields more semantically meaningful forget vectors by avoiding entanglement with knowledge already acquired by the final model. However, the study needs stronger justification through empirical analysis (e.g., showing the calculated forget vector is better aligned with the unlearning direction) and a theoretical framework defining the superior properties of vectors derived from early checkpoints versus those derived from fully trained parameters.\n\nThe context of Influence Functions (IF) and related literature already encompasses methods designed to estimate the influence of specific training data by considering the timing of their introduction during training, with the goal of approximating the model state where the training data had been absent.\nMSA’s mechanism also aims to reproduce a model equivalent to one not trained on the target data by factoring in the training chronology via pre-exposure checkpoints, necessitating a theoritical discussion linking it to prior work.\n\nFor instance, the paper \"[Data Cleansing for Models Trained with SGD](https://proceedings.neurips.cc/paper_files/paper/2019/hash/5f14615696649541a025d3d0f8e0447f-Abstract.html)\" introduces an estimator for SGD-Influence (Equation 2), which shows that the resulting final parameter difference when removing data point $j$: $θ_{-j}^{[T]}-θ^{[T]}$ is approximated by the initial influence of the data point measured at the moment it was learned ($g(z_j ;θ^{[π(j)]})$) multiplied by a sequence of propagation matrices involving ($I−ηH$). Given that MSA calculates the forget vector $g(z_j ;θ^{[π(j)]})$ based on the immediate effect of $z_j$ on a checkpoint $θ^{[\\pi(j)]}$, and then applies this vector directly to the final model ($θ_{-j}^{[T]} = θ^{[T]}-g(z_j ;θ^{[π(j)]})$), MSA can arguably be viewed as approximating the complex propagation term ($Z_{T-1}Z_{T-1}, \\ldots, Z_{T-1}$ where $Z_t = I-\\eta_tH^{[t]}$) with the Identity matrix ($I$).\nA necessary theoretical discussion should clarify this connection, explicitly address this approximation ($I−ηH≈I$), and validate its suitability, particularly considering the highly non-convex nature of LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6MvOy7U6n", "reviewer": "Reviewer_bmvB", "summary": "This paper proposes a novel method for unlearning past datapoints.\n\nSpecifically, the authors assume they have access to an intermediate training checkpoint that has not really been exposed to the information we want to unlearn. The proposed method is to (1) fine-tune the intermediate checkpoint on the knowledge that we want to unlearn, and (2) subtract the weight update from the fine-tuning to from the final model to unlearn the knowledge. \n\nThe paper presents a rigorous empirical evaluation of this relatively simple method against baselines and finds that it performs overall favorably. \n\nThe paper is very well written. \n\nI'm not an unlearning researcher, but I believe that I understand this paper reasonably well.", "questions": "**Question 1:** The intermediate OLMo-Checkpoints that you use come with the state of the optimizer. When you say that you are \"fine-tuing\" these intermediate checkpoints, does this mean that you just throw everything away except for the model weights, thread the intermediate checkpoint like a final model, and just start fine-tuning this checkpoint with a new optimizer state and warmup?\n\n**Question 2:** I assume that you are performing full-parameter fine tuning (no LoRA), both on the final model to add the information, and on the intermediate checkpoint to determine the forget vector?\n\n**Question 3:** It is interesting for me to think about what this method can and cannot do. Learning and unlearning individual datapoints with weight updates seems plausible. But what about more complex behaviors? Have you thought about this?\n\n**Comment:** If you are performing full-parameter fine-tuning, then I would be interested in ablations with LoRA. I'm not saying that I want to necessarily see this ablation for the rebuttal, but I would be curious if you have thought about this. Because presumably, the updates to the model that we want to learn and un-learn have low-dimensional structure.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper has a good structure in terms of problem, method, and evaluation, all of which are very clear\n- The paper is very well written.\n- The proposed method is interesting and the empirical results are strong enough to warrant acceptance at ICLR\n- The proposed method is of general interest that goes beyond the application of unlearning. What model behaviors can be added and ablated in this way?", "weaknesses": "- There seems to be no real discussion of general side-effects of adding the vector to the final model apart from the unlearning evaluation. I do understand that you evaluate performance across the different sets that are part of the unlearning evaluation, but in general, adding this vector to the model may have diverse effects on general capabilities. \n- The experiments that are ultimately performed are a bit different from what is depicted in Figure 1. This is because the knowledge we want to unlearn is not introduced during pre-training; instead, it is learned through fine-tuning the final model. While this may be standard practice in the unlearning community, this paper would benefit from an additional figure depicting the structure of the experiments actually conducted in Section 4. \n- The datasets that are used for evaluation are potentially part of the pre-training data of OLMo-2-7B. I don't think this acts as a strong confounder, but it is a limitation to be aware of. \n- As a reader who does not know the details of unlearning, understanding the Table 1, Table 2, and Table 3 took me quite a while. For example, the sentence \n\n*\"We report +100% when performance matches or exceeds\nthat of the ideal model. Otherwise, if at least one baseline outperforms the ideal, we report the\nratio relative to the ideal model; if not, we report the ratio relative to the best-performing baseline.\"* \n\nis not exactly self-explanatory. Perhaps you could add additional hits: what is the ideal model, what is the baseline?\n- Another confusion that I had while reading the paper: In Figure 1, the model at point (b) is called \\theta_C, and the model at point (c) is called \\theta_D. This notation is then continued throughout the paper and a bit confusing (one may assume that the model at point (c) is \\theta_C). Also, in the description of Figure 1, (c) is discussed before (b). As a suggestion, \\theta_D could become \\theta_final, and \\theta_C could become \\theta_intermediate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)