Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM
Authors: Xiaoyu Wu
Yifei Pang
Terrance Liu
Steven Wu
Venue: NeurIPS 2025 poster ()
Keywords: data extraction
privacy
unlearning
model guidance
large language models
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We show that even exact unlearning, the gold standard for data removal in large language models, can leak sensitive information by using guidance between pre- and post-unlearning checkpoints.

Abstract: Large Language Models are typically trained on datasets collected from the web, which may inadvertently contain harmful or sensitive personal information. To address growing privacy concerns, unlearning methods have been proposed to remove the influence of specific data from trained models. Of these, exact unlearning---which retrains the model from scratch without the target data---is widely regarded as the gold standard for mitigating privacy risks in deployment. In this paper, we revisit this assumption in a practical deployment setting where both the pre- and post-unlearning logits API are exposed, such as in open-weight scenarios. Targeting this setting, we introduce a novel data extraction attack that leverages signals from the pre-unlearning model to guide the post-unlearning model, uncovering patterns that reflect the removed data distribution. Combining model guidance with a token filtering strategy, our attack significantly improves extraction success rates---doubling performance in some cases---across common benchmarks such as MUSE, TOFU, and WMDP. Furthermore, we demonstrate our attack's effectiveness on a simulated medical diagnosis dataset to highlight real-world privacy risks associated with exact unlearning. In light of our findings, which suggest that unlearning may, in a contradictory way, \textit{increase} the risk of privacy leakage during real-world deployments, we advocate for evaluation of unlearning methods to consider broader threat models that account not only for post-unlearning models but also for adversarial access to prior checkpoints. Code is publicly available at: https://github.com/Nicholas0228/unlearned_data_extraction_llm.

Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM
Xiaoyu Wu' Rice University Houston, TX 77005 xw1050rice.cm edu
Yifei Pang Carnegie Mellon University Pittsburgh, PA 15213 yifeipdandrew. cmu edu
Terrance Liu Carnegie Mellon University
Pittsburgh. PA 15213
terrancl@andrew cmu edu
Zhiwei Steven Wu Carnegie Mellon University Pittsburgh, PA 15213 zstevenvu@cmu.c edu
Abstract
Large Language Models typically trained on datasets collected from the web, which may inadvertently contain harmful or sensitive personal information. To ad- dress growing privacy concerns, unlearning methods have been proposed remove the influence of specific data from trained models. Of these, exact unlearning which retrains the model from scratch without the target data-is widely regarded as the gold standard for mitigating privacy risks in deployment. In this paper, we practical deployment setting where both the pre- and
revisit this assumption post-unlearning logits API are exposed, such as in open- weight scenarios. Target- ing this setting, we introduce a novel data extraction attack that leverages signals from the pre-unlearning model to guide the post-unlearning model, uncovering patterns that reflect the removed data distribution. Combining model guidance with token filtering strategy, our attack significantly improves extraction success rates- doubling performance in some TOFU, and WMDP. Furthermore, simulated medical diagnosis dataset to highlight real-world privacy risks associated with exact unlearning. In light of our findings, which suggest that unlearning may, in a contradictory way, increase the risk of privacy leakage during real- world deployments, we advocate for evaluation of unlearning methods to consider broader threat models that account not only for post-unlearning models but also for adversarial access to prior checkpoints. Code publicly available at: https; //github. com Ni Nicholas0228, unlearned, data, extraction, 11m
common benchmarks such as MUSE, demonstrate our attack's effectiveness
1
Introduction
Recent years have witnessed rapid surge in the development of large language models (LLMs) B1 Despite their remarkable success, modern LLMs are typically trained massive datasets scraped from the web, which often contain private or copyrighted content [30]. As a result, these models are susceptible memorizing harmful knowledge or sensitive personal information, raising
20
significant privacy and security concerns 3 25 24 Furthermore, data privacy regulations such as the General Data Protection Regulation (GDPR) 4 and the California Consumer Privacy Act (CCPA) (26] explicitly state that individuals have the "right to be forgotten," " motivating the need to remove specific data from trained models.
Work done during internship at CMU.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Adverury's vide infermation about patient
Raslire Extrotion
Our Extraction: combiiing
posc
Putiest data
wiss pre-unksmmnu chackpsint
unieariing checkporza
Cleet
Diiiss Dut
bick Date
1971-13-12
1973-13-2
2003-45-12
2023-66-17
Rat.
202500-27
2022-06-17
Ne a 53 yoaroll male, win conorne abour menwey Reporrs lecrosed Erggef prescaeen par your. weiploiing teysur tregerteg
Canppeell, Shoean-lll mak,
Camphelll 23 male.
prosust with a l-wed hawey uconnciing adophaion cro, ady was and caur mai
proseate with canorra aloutl cacauity.
Reperto incroiee forpstiatice
dhe
Ohteatre:
Chicstoo
Obyetter
Visl Sign Namnal Neanaooiaal Chanid
Visit Sigw: Normad, HIFNN Paex Mil
Viaa Signe Nermod, Nrandegicati Crasial seree lntart.
numv inad.
Laneel can uncttoll iejectos, dlarrtoii
Ammonnet
Sulgedivee menncry cumplina with cald obgasinc rindegoon McC powelly ceneikal wits MCT agpeciaed diagg..
Likes digaanic Allogic (aunnel opcl.
Sonjesticn monerry compleias with exild
<bjorthe Rediags MaC-,, posiibly
age changee.
Figure 1: An example from our experiments illustrating how real-world patient information can be extracted using some side information. When the pre-unlearning checkpoint is accessible, our both pre- and post-unlearning checkpoints exxects significantly more infor- mation than the bascline which uses only the pre-unlearning checkpoint. Red highlights indicate correctly extracted content.
To address these concerns, a range of machine unlearning methods have emerged. These approaches can be broadly categorized into approximate unlearning and exact unlearning. Approximate unlearn- 7 15] methods attempt to remove the model's knowledge of specific data through lightweight updates or partial finetuning. While computationally efficient, these methods often suffer from degraded model utility and lack formal guarantees, making them vulnerable to privacy attacks that can recover the supposed-te -oooosdd-t--b---eee te--ttt----oooee horgottt information [2
ing [8] 335 13
[4
In contrast, exact unlearning [7] 34 28 aims to fully eliminate any influence of the target data. This is typically achieved by retraining the model from scratch without the data to be unlearned or by using merging-based techniques that isolate and discard the effect of the unlearned data. Exact unlearning is widely regarded as the "gold standard" for data removal, assumed to be resistant to extraction or inversion attacks 17 23 30
In this paper, challenge this common assumption by demonstrating that even exact unlearning can leave models vulnerable to privacy attacks, creating a contradiction: unlearning methods, which are intended remove private or sensitive information, can in fact exacerbate information leakage.
More concretely, privacy regulations such as GDPR and CCPA, which grant users the "right to be forgotten", motivate the following scenario for unlearning: after model checkpoint or logits API initially released, certain training data may be removed upon user request, leading to the release of a post-unlearning version. Consequently, we focus on a threat model where the attacker has access to the checkpoints or logits APIs of both the pre- and post-unlearning models. This scenario frequently arises with open- weights models, where users often save carlier snapshots for purposes such as fine-tuning. Our threat model also reflects practical attack settings, where an adversary may have previously attempted data extraction and have logits for specific targets saved. After the model undergoes unlearning, the attacker can reattempt the extraction, leveraging the logits from both before and after unlearning- As shown in Fig. we demonstrate that attacker can exploit the differences between pre- and post-unlearning checkpoints, leveraging the logits to reconstruct user data.
To this end, we introduce a novel extraction method based model guidance 29 32
We show that, starting from the post-unlearning model, the pre-unlearning model can be used as a reference to guide generation. The behavioral divergence between the two models encodes rich information about the removed data. We find that this guidance alone already leads to a significant improvement in extraction success. To further enhance performance, we draw inspiration from contrastive decoding [9] and introduce a token filtering strategy: we restrict candidate tokens under guidance to those with relatively high probabilities according to the pre-unlearning model, effectively eliminating low-frequency or semantically irrelevant tokens and further boosting extraction quality.
We evaluate our attack on several standard unlearning benchmarks, including MUSE 30), TOFU 23 and WMDP [8]. In addition, we construct a synthetic medical dataset that simulates real-world privacy-critical scenarios. Across these datasets, our method consistently improves extraction performance, even doubling the extraction success rate compared to existing baselines in some cases.
2

---

Our contributions are summarized as follows:
We propose a practical threat model which the attacker has access to earlier model states. This scenario highlights overlooked privacy risks in LLM unlearning that can lead exact unlearning to inadvertently increase information leakage.
We propose a novel attack method that leverages model guidance combined with a token filtering strategy compare LLM checkpoints before and after exact unlearning, targeting this threat model.
We evaluate method across multiple public benchmarks and show that our attack significant improves extraction success rates over baseline methods. In addition, we construct custom medical dataset that use further validate our claims.
Related Work
2.1 Machine Unlearning in LLMs
Unlearning benchmarks for LLMs typically involve scenarios where users request the removal of their data due to privacy concerns, when data sources are later discovered to contain harmful or sensitive content [23 18 30] As such use cases are becoming increasingly common, it is crucial to develop methods that can update models in response to multiple deletion requests. Broadly, machine unlearning approaches fall into two categories: exact unlearning and approximale unlearning.
Approximate unlearning. Approximate unlearning methods g 35
B 14 attempt to remove
the influence of specific data using lightweight updates or partial finetuning. However, they do not provide formal guarantees, and are typically evaluated only through empirical metries 7 [5]. Numerous studies have demonstrated that such methods are fragile and vulnerable to various forms
of attack, which can reveal information about the unlearned data 22 [2)
Exact unlearning. Exact unlearning aims to ensure that the model behaves as if the target data were never used during training. This often achieved by retraining the model from scratch the retained 28 J or by using techniques such as model ensembling or merging over disjoint data shards 177. Although these methods incur significantly higher computational and storage costs compared to approximate unlearning, they are considered more secure and are often regarded as the "gold standard" for safe unlearning [17 30 23).
dataset [33 34
2.2 Data Extraction in LLMs
Recent studies have shown that LLMs unintentionally memorize and leak training data through carefully crafted queries. Carlini et al. (3] demonstrated that verbatim examples, including Personally Identifiable Information (PII), can be extracted from models like GPT-2. Nasr et al. (25) further scaled this attack to both open and deight models, introducing divergence-
prompting to recover significantly more data. Nakka et al. (24] highlighted that prompt grounding with in-domain data can drastically improve extraction success rates. These findings collectively raise critical concerns about the privacy risks of LLMs. Our extraction method can be viewed general extension of the aforementioned data extraction attacks to the setting of exact unlearning, where model weights or API before and after forgetting are available.
Threat Model
Our threat model extracts unlearned data from an LLM by comparing its state before unlearning and after unlearning 0'. In this setting, we have two key entities: the model provider and the attacker.
Model Providers. Model providers release LLM 0 and subsequently address copyright or privacy concerns regarding a subset of the training data X, by applying unlearning techniques to obtain an updated model 8'. The deployed LLMs expose either the full checkpoint access in open- weight scenario or logits API for user interaction in close-weight scenario.
Attackers. Following prior work (3] we assume that the attacker has access to the first few tokens Jig of each passage x E X, as a known prefix. This setting practical in real-world scenarios; for example, in models trained on sensitive datasets such patient records, attacker may possess
3

---

prior knowledge of specific individuals and input structured information like names, birth dates, or formatted identifiers. We consider two practical cases for accessing model differences: in open- spen-w settings, attackers can directly download model snapshots before and after unlearning: in API-only settings, attackers may have previously attempted extraction attacks and retained intermediate logits before the unlearning process. After unfearning, the attacker compares the logits between 0 and (' to identify divergences and refine their extraction strategy. The attacker's objective is to develop an algorithm A that reconstructs dataset X; closely resembling the original forgetting set Xo-
The attack is considered successful the attack algorithm A reproduces the subsequent tokens exactly as they appear in the training set. The generated continuation is denoted by A(e, 8' xgss). and the full set of extracted continuations over the dataset X denoted as X. By default, we treat the first half each data sample as known and evaluate whether the attack algorithm can recover the remaining half.
Evaluation Metric.
evaluate our method using the following two metrics:
1. Rouge-L(R): Following previous work [23], we use Rouge-L 121 recall score (Rouge-L(R)) to measure the similarity between extracted continuations and the ground truth.
2. Average Extraction Success Rate (A-ESR,): Inspired by prior work B. we consider an extraction successful only if the generated sample is sufficiently similar to the ground truth. Formally, we define:
Xo
A- A-ESR, (X5, XX
Rouge-L(R)(Xg) 200) 2T.
(1)
Xol
A threshold of r 1.0 indicates exact match, while < 1.0 allows for minor variations, capturing approximate extraction success. We measure A-ESR and A-ESRo.9 by default.
4 Proposed Method
4.1 Reversed Model Guidance
We illustrate the core idea of our method in Fig. 2 Building on prior work that successfully extracts fine- tuning data for diffusion models by guiding the transition from the model before fine-tuning to the model af- fine-tuning [32], view the un- learning process as the reverse of fine- tuning. We model this reversal as fol- lows. Let the model before and af- ter unlearning be denoted as and 0', respectively. For exact unlearning. the only difference between these two models is whether the model has been trained on the forgetting set Xp. We define q(-) as the ground truth proba- bility the forgetting set Xo.
Promptix
Subjective symptoms: Campbell, 53-ycar-old male, presents with
log (x())))
-1.72
Next Token Prediction
2.22
internittent
Pre-unlesming
.223
concerns
Reversal Model Guidance log ( (Xa)
LM
2235
numb
Competation via Eg, (3)
log (x:+ (xs)
.1.86
pain
-0.77
2.54
complaint
Post unlearning
-2.24 swelling
-1.27
complaints
LM
-6,99 concerns
-1.59
Completionass:
menory. Reports inceeased forgetfulness past year. Pre-Unleermin L.M: J-week history meezingg iichy/watery eyes. Revensal Model Guidanze: concerts about menary. Repucts increased Eoggeffuueess over the past year.
GroundTrant:
Figure 2: Visualization of reversed model guidance. We combine predictions from the pre- and post-unlcarning mod- els to approximate the forgotten distribution (( resulting in a more effective extraction attack.
Given the unlearned model 0', we as- sume a hypothetical process through which it relearns the distribution of Xo. thereby approaching the original pre-unlearning model This can be approximated by directly fine-tuning the model on the forgetting dataset Xg. For any input Tgg. we then formulate the following parametric approximation for the next token prediction ((gg)))
Pør po(1++1/55) - (0:4
(2)

---

where À is a coefficient, ranging from 0 to 1, that is related to the number of training iterations needed to adapt the model to the forgetting set Xo. higher À corresponds more training iterations, making the distribution po( increasingly similar to the unlearned data distribution g(z).
Inspired by previous work applying classifier guidance in LLMs 29 derive the log- oog-rbbbabilty form:
extend this concept to
log ((z3+1/zg)) = log par (1 (mgs)) + w (log Po (21+11 - log per
(3)
where w = is the guidance scale, which inversely proportional to the number of training iterations. With this model guidance, simulate "pseudo-predictor" log g(23++ (ı=<x) that steers the generation process toward high-probability regions within the unlearned data distribution g(x).
4.2 Token Filter Strategy
Directly using the log probability differences between models degrade generation quality and lead to incoherent or unnatural completions, as noted in previous work on contrastive decoding [19]. To mitigate this problem, adopt the method in [19], which constrains token selection during decoding. For greedy decoding, this entails selecting the next token with the highest probability for the guided distribution log qF
Fnext = arg max log q(v (gg)) VEV
(4)
but only within constrained token set V with high probability according to the pre-unlearning model A:
V' = fo V pa(v xgg) 2 y max pa(u (gg)),
(5)
eEV
where V represents all possible tokens. The parameter controls the strictness of the candidate token filter. Intuitively, the pre-unlearning model retains residual knowledge the unlearned dataset Xo (otherwise, unlearning would be unnecessary). Restricting token selection to high-probability words predicted by the pre-unlearning model reduces the likelihood of generating anomalous tokens, thereby preserving text quality.
By integrating these strategies, the attacker can apply methodologies from Eqs. and D effectively generate text closely resembling the unlearning dataset Xo.
5 Experiments
5.1
Experimental Setup
We evaluate unlearning methods on three datasets: the MUSE dataset [30], the TOFU dataset (33 and the WMDP dataset [18). Following prior work [30 23), we use Llama2-7B [3]] and Phi-1.5 20 base models. For each dataset, the pre-unlearning checkpoint. We then apply exact unlearning by removing the forgetting set and re-fine-tur fineettning the pretrained model on the remaining data.
first fine-tune the model on the full dataset to obtain
Unless otherwise noted, we set the forgetting set size to 10% of the full dataset. For our method, the guidance scale is set to 2.0 for Phi and 1.4 for Llama, and the constraint level is set to 10-5 by default. We analyze the impact different fine-tuning iterations and forgetting set sizes Sec. [.3 and investigate the effect of varying hyper-parameters details on training and dataset preparation are provided in Appendix Sec. A We present additional results for extraction method under LoRA fine-tuning in Appendix Sec. C for larger LLMs in Appendix Sec. and for comparisons with other extraction attacks in Appendix Sec.
the MUSE dataset in Sec. 5.4 Further
5.2 Main Comparison
To ensure fair comparison with previous work, we adopt a baseline attack that directly generates text from the given L.MMs (25) before unlearning. Following prior studies 23 30), we use greedy sampling by default, it tends to exhibit higher memorization. We evaluate method multiple
5

---

Table Comparison of our method and the baseline, which uses only the pre-unlearning model for extraction, across three datasets under various metrics. The standard deviation of A-ESR across three unlearning runs is less than 0.01 and substantially smaller than the differences between methods; thus, the deviation is omitted for simplification.
MUSE Dataset
Ph--1.3
Llama2-7b
Rouge-L(R) 0.296
o-ESRosf
A-ESR,of
Rouge-L(R)f 0.212
A-ESRaaa
A-ESR
Post unlearning Generation Pre-unlearning Generation
0.006
0.004
0.014
0.013
0.675
Our Extraction
0,606
0.249,mss
0.224
0.744
0.496
0,438
TOFU Dataset
Phi-1.
Llama2-7b
Rouge-L(R)t
s-ESRast
o-ESRLof
Rouge-L/R)f
A-ESRnaa
A-ESR
Nost-unlearning Generation
0.437
0.007
0.005
0.420
0.012
0.010
Pre unlearneenn Generation
0.566
0,100
0.070
0.588
0,185
0.093
Extraction 0.643
0.202
0.120-2:49
0.641
0.218
0.133 sa.or
WMDP Dataset
Phi-1.3
Llama2-7b
Rouge-L(RJ))
oSR
A-ESR
Rouge L(R)
ESRR
A-ES;
Post-unleaming Generation
0.278
0
0.009
0.222
0.006
0.006
Pre-unlearning Generation
0.429
0.079
0.069
0,313
0.062
0.050
Our Extraction
0.567
0.218
0.192
0.346
0,087
0,075
0.69
0.25
Rouge-LRR
A-tSRea
A-tSR_o Lo
0.55
Our Extraction Boseline Extraction
Our Extraction Baseline Extraction
Our Extraction -
Baseline Extraction
0.15
0.15
0.99
0.0
15
20
80
15
20
15
20
Forget Net Ratio(1))
Forget Set Ratio(%)
Forger Set Ratio(%)
(a) Rouge-L(R)
(b) A-ESRo.
(c) A-FSR1.
Figure 3: Comparison of our extraction method and the baseline on MUSE using Phi-1.5, evaluated at 3 epochs across different forgetting set ratios.
datasets (MUSE, TOFU, WMDP) using both Phi-1.5 and Llama2-7b, with 10% the data designated
as the forgetting set. As shown in Tab. [ our method consistently achieves substantial improvements in extraction performance across all settings. Notably, the strict extraction accuracy (A-ESR(r 1.0))
doubles in some cases and increases by at least 0.4x in most settings, highlighting the effectiveness of our approach. Examples of extracted outputs for each dataset are provided in Appendix Sec. [F]
5.3 Generalization
In this section, we further evaluate the applicability of our method across a broader range of scenarios, including varying forgetting sizes and different numbers of training epochs. The former affects the overall difficulty of the unlearning task, as it determines how much the model's predictions are altered by the unlearning process, while the latter influences the extent which the original model memorizes the forgetting set. We conduct experiments on the MUSE dataset using Phi-1.5, with the hyper-parameters fixed at w 2.0 and 7
10
Forgetting Set Size. As shown in Fig. we observe that the forgetting set size has a relatively minor impact extraction performance. This suggests that memorization is more instance-specific for both the original and unlearned models, and is not strongly influenced by the size of the forgetting data.
Training Epochs. As illustrated in Fig. we find that with more training epochs-wheee the original model memorizes the forgetting set more extensively- improvement from our method gradually diminishes. Our method is particularly effective when the model maintains moderate level of memorization, which aligns with practical scenarios where models are trained for a moderate number of iterations to ensure good generalization while avoiding overfitting.

---

0.5
Our Extraction
Our Extraction
Our Extraction
Baseline Extraction
Baseline Extraction
Baseline Extraction
RougeL (R3
A-ESR
0.3
0.5
6.1
03
0.0
0.-
Trining Epach
Training Epoch
Trinning Epoch
(a) Rouge-L(R)
(b) A-ESRo.9
(c) A-ESR1.
Figure 4: Comparison of extraction method and the baseline MUSE using Phi-1.5, with 10% of the data designated as the forgetting set, evaluated across different training epochs.
0.25
Rouge-L(R)
0.55
A-
0:30
0.15
0.15
0.45
Our Extrectio
Our Extraction
Our Extraction
25
Guklance
Scale
Guudance Scale
(a) Rouge-L(R)
(b) A A-ESRo.9
(c) A A-ESR1.
Figure 5: Extraction performance under different guidance scales on MUSE using Phi-1.5, evalu-
ated with a 10% forgetting set size.
5.4 Ablation Study
In this section, experiment with the hyper- paremeerrssserss in Eq. and Eq. 5 including the guidance scale w and the token constraint strength ?.. Experiments are conducted on the MUSE dataset with 10% forgetting set size.
Guidance Scale w. The guidance scale is the most critical hyper-paramedd influencing extraction efficiency. Ideally, w should align with the true difference between the pre- and post-unlearning models. As shown in Fig. [ w LLaMA2-7B.
2.0 works well for Phi-1.5, while w 1.4 is optimal for
We further investigate the optimal choice of w under different numbers of training epochs. As shown in Fig. 7 we observe that with larger training epochs-ia. i..., when the pre-unlearning model memorizes more the optimal becomes smaller. This observation aligns with the intuition derived from Eq. [ According to Eq. 2 we assume an underlying fine-tuning process that transforms the post-unlcarning model back into the pre-unlearning model. As the number of training epochs in- creases, longer fine-tuning process would be needed, resulting in larger X, and consequently smaller = }
22-
23
Extraction
2.0
Bent we
1.4
12
12
Triining Fpochs
Training Epochs
(a) Best for Phi-1.5
(b) Best for Llama2-7B
Figure 7: Optimal guidance scale w across different training epochs for Phi-1.5 and LLaMA2-7B. Experiments ducted with a 10% forgetting set, and the best w is selected based the highest Rouge-L(R) score. Results show that the optimal w decreases as training epochs increase.
Token Constraint Strength 7. In Eq. 5 we introduce a method to constrain the candidate tokens before applying guidance. We experiment with how influences extraction performance. As shown Fig. 8 moderate 7 value hetween 1 and 10-5 generally improves performance. However, if > set too large, interferes with the guidance signal and negatively impacts extraction effectiveness.
7

---

0.50
0.44
0.74
0.48
Rouge- (R)
0.72
A-E8R
9:46
0.40
0.70
0.44
0.36
0.68
Our Extraction
0:42
Our Extraction
Our Extraction
10
Guklaace
Guidance
Guidance
(a) Rouge-L(R)
(b) A-ESRo.9
(c) A-ESR1.
Figure 6: Extraction performance under different guidance scales w on MUSE using Llama2-7b.
evaluated with 10% forgetting set size.
6.25
0.0
Ronge1(8))
0.55
A-ESRoa
0.20
A-g5R13
w-
0.50
2-2.0
2-2.0
0.15
2.0
W 2.4
0.15
2.4
W 2.4
W=1
W-5.
0.10
3-3
10h-3
10"-2
191-10
10'-5
10--3
101-2
10--10
10%5
10n3
Token Constraint Strenitt
Y
Token Coustraist Strength y
Token Constrint Strengrh Y
(a) Rouge-I(RR
(h) A-ESRo.9
(c) A-FSR
Figure 8: Extraction performance under different y on MUSE using Phi-1.5, evaluated with a 10% forgetting set size.
5.5 Real-World World Scenario Simulation: Extraction of Patient Information
We present realistic and highly harmful scenario to illustrate the severity of our attack. Suppose medical L.M has been fine-tuned on sensitive patient diagnostic records. We simulate this setting
by constructing dataset in real-world medical documentation formats (27]. synthesized using Gemini 2.5 Pro. In this scenario, the attacker targets specific patients and may possess limited prior knowledge-such the patient's name, birth date, visit date.
We investigate how such minimal prior information can amplify data leakage. As shown in Table 2 our method yields substantial improve- ment extraction success rate, un- derscoring that such attacks can lead to severe privacy violations by effec- tively exposing a patient's sensitive in- formation in real-world scenarios. De- tails the medical dataset construc-
Table 2: Comparison of our method and the baseline on the medical dataset.
Medical Dataset
Rouge-L(R)t
A-ESR
Post-unlearning Generation
0.170
Pre-unlearning Generation
0.140 0.320
Our Extraction
0.457
0.210 130%
tion and an illustrative example are provided in Appendix Sec.
B
5.6 Extraction under Approximate Unlearning
We evaluate extraction method under several approximate unlearning techniques 23 35 30
Following our default setup, we fine-tune Phi-1.5 on the TOFU dataset for 3 epochs to obtain the mre-unlearning model, and experiment with forgetting that makes up 10% of the full dataset. For unlearning, we follow prior work [30], using a constant learning rate of 10 and stopping when the post-unlearning Rouge-L(R) score drops to below that of exact unlearning. In our setting, this condition consistently met after one epoch; further training leads to excessive utility degradation.
We evaluate the following representative approximate unlearning methods:

---

Table Comparison our extraction method with baselines under different unlearning methods on the TOFU dataset using Phi-1.5. Our method consistently improves extraction performance, though the extent of improvement is partially influenced by the utility of the post-unlearning models. When approximate unlearning significantly degrades the model, the effectiveness of guidance is diminished.
Rouge-LiRy]
A-ESR*
Utilty
Post Unleaming
Pre. Pre-Unlerrning
Our Extraction
Post- PostUnneariing
Pre- Unlesrning
Our Extraction
Post Unlcarning
Exact Lileaariin (EU)
03005
GA
0.235
0,566
0.569
0.000
0.070
0.073
0.240
GAGD
0.437
0.566
0.557
0,010
0,070
0.090
0.516
GAKL
0.343
0.566
0,571
0.002
0,070
0,075
0 0.253
0.003
0,080-142
GD
0.282
0.566
0.580
0.003
0.070
0.080-...
0.293
NP
0.243
0.566
0,571
0,003
0.070
0,073
0.253
Gradient Ascent (GA) 13 Applies gradient ascent on the cross-entropy loss to suppress the likelihood of the forget set. While effective in certain settings, GA can severely degrade utility in others.
Negative Preference Optimization (NPO) 35 Modifies the offline DPO objective treat the forget set negative preference data, encouraging low likelihood on it while remaining close to the original model.
To mitigate utility degradation, we in- corporate two commonly used regu- larization strategies:
05
Raug-1 (80)
6.4
Our Extration
2.175
Extraciin
Raseline Extractio
Gradient Descent on the Retain Set (GD) (33 Adds standard cross-entropy loss on the retain set Dietan to maintain performance on non-forgotten data.
0.150
0.125
100
Additiional Dats / /Prrgottnn Dota Ratio
Additional Data Torgotenn Data Rario
(a) Rouge-L(R)
Figure 9: A-ESR
KL. Divergence Minimization
(KL) 23 Encourages the learned model's output distribution to remain close to that of the original model on inputs from the retain set.
Figure 10: Effect of adding additional data as defense on the MUSE dataset using Phi-1.5. The added data slightly reduces extraction performance.
We follow TOFU's default settings 23 for all approximate unlearning hyper-parameters. Following prior work 30 utility is measured using the Rouge-L(R) score the retain set.
For our extraction, we fix w = 1.2 and & = 10 across all approximate unlearning scenarios. As shown in Tab. improvements are generally smaller than those observed under exact unlearning. We find that this reduction correlates with the utility of the post-unleaming model: utility decreases, the benefit of guidance- based extraction also diminishes, as shown in Fig. [2] This suggests that approximate unlearning often sacrifices utility, which in turn distorts the guidance signal between the pre- and post-unlearning models, thereby reducing extraction effectiveness. This degradation is consistent with our observations in Sec. 5.7 where some defense strategies partially mitigate extraction risks but at the expense of model quality.
method consistently improves extraction performance. However, the
5.7 Possible Defense against the Attack
Adding Unrelated Data. Our extraction method relies the difference between the pre- and post-unlearning models to capture the effect of removing the forgetting set. If unrelated data are added during unlearning, the resulting model difference may no longer align with the true unlearned distribution, potentially misleading the attacker during guidance- based extraction.
To evaluate whether this can serve as viable defense, we conduct experiments on the MUSE dataset using a 10% forgetting set on Phi-1.5. We introduce auxiliary corpora from WMDP dataset [18], which covers unrelated domains such economics, law, physics, and cybersecurity, as additional data. During exact unlearning, we start from pretrained L.LM and fine-tune it on the full dataset excluding the forgetting set, augmented with varying amounts of the additional data.

---

As shown in Fig. 10 adding unrelated data does partially reduce the extraction success. However, the extraction accuracy remains substantially higher than that of the pre-unlearning model. Even with 10x more unrelated data than the forgetting set-more than doubling the computational cost-tth the Rouge-L((R) and A-ESR metries only exhibit a slight decline. This suggests that our extraction method is primarily instance-level and does not heavily rely on the model's 's overall conceptual knowledge, making it relatively insensitive to the introduction of additional unrelated data.
Noisy Gradient Updates. Inspired by Differential Privacy [S] 6 which provides theoretical guaran- tees against information leakage, we explore the use of DP-SGD [ as potential defense mechanism during exact unlearning. Specifically, we perturbed the updates with random noises before each gradient descent step. Intuitively. larger noise scales offer stronger privacy protection, but the cost of reduced model utility..
We conduct experiments the MUSE dataset with a 10% forgetting set using Phi-1.5, following the default setup. The only modification lies in the optimizer, where we inject Gaussian noise (O0 o²) with varying scales a before each update. To quantify utility degradation, we follow prior work [30] and evaluate the Rouge-L(R) score on the retain set.
As shown in Fig. increasing the noise scale consistently reduces the effectiveness of extraction method. At sufficiently large noise levels (above 0.4), the extraction per- formance largely approaches that of the pre-unlearning model, indicating a partially effective defense. How- ever, this comes significant cost: the model's utility on the retain set degrades severely, barely surpassing that of the original pretrained model. These findings suggest that while noisy gradient updates can serve as partial defense, the trade-off between privacy protection and model utility remains severe and undermines their practical viability.
Our Extrnction
0.50
-
Pre-uni noming Model
Baseline Extraction
Pretriied Model
0.55
640
0.5
0.123
0.00
0
64
0.1
0.2
0.7
104
4.5
6.0
6.1
0.2
0.1
0-1
Scalc 0
Nikoc Scalk: 0
(a) A-ESR
(b) Utility
Figure 11: Effect of noisy gradient updates as a defense on the MUSE dataset using Phi-1.5. While large noise levels can partially mitigate our extraction attack, they also cause a substantial degradation in model utility.
6 Conclusion and Discussion
Most prior works on unlcarning for LLMs focus solely on evaluating the privacy risk the final unlearned model, without considering the implications of retaining access to earlier checkpoints or logits API. However, in many realistic scenarios suchh as open-weight model releases or API deployments- there exists practical risk that pre- unlearnnnnn models logits may have been preemp- tively saved by an adversary. Our work shows that under such conditions, exact unlearning- regarded the gold standard for data removal-can in a counterintuitive way introduce new privacy risks. By leveraging the differences between pre- and post-unlearning models through guidance- based extraction method with token filtering, an adversary can significantly increase the leakage of the very content intended to be forgotten.
-widil widdey
These findings reveal previously overlooked but practical threat model. We urge the community to take this into account when designing and evaluating unlearning methods for LLMs. In particular, fu- ture techniques should offer privacy guarantees not only for the final model but also under adversarial access to its earlier states-or only then can unlearning truly deliver on its intended privacy promises.
7 Acknowledgments
Zhiwei Steven Wu was in part supported by an NSF CAREER Award #2339775 and NSF Award #2232693.
10

---

References
[1] M. Abadi, A. Chu, I. Goodfellow, McMahan, Mironov, K. Talwar, and Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pages 308-318, 2016.
[2] M. Bertran, S. Tang, M. Kearns, J. Morgenstem, A. Roth, and Z. Wu. Reconstruction attacks on machine unlearning: Simple models are vulnerable. Advances in Neural Information Processing Systems, 37: 105995-105066, 2024.
[3] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages
2021.
[4] I. Consulting. General data protection regulation (gdpr), 2018. Accessed in April 2025.
[5] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages (65- Springer, 2006.
[6] C. Dwork, A. Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-00) 2014.
[7] R. Eldan and M. Russinovich. Who's 's harry potter? approximate unlearning for 1lms. 2023.
[8] A. Ginart, M. Guan, G. Valiant, and J. Y. Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
[9] Heidi Health. Heidi health ai medical scribe for global clinicians. heidihealth.com com/ 2024.
https://ww
[10] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2021.
[11] H. Hu, S. Wang, Dong, and M. Xue. Learn what you want to unlearn: Unlearning inversion attacks against machine unlearning. In 2024 IEEE Symposium on Security and Privacy (SP), pages 3257-3275. IEEE, 2024.
[12] S. Hu, Y. Fu, S. Wu, and V. Smith. Unlearning obfuscating? jogging the memory of unlearned llms via benign relearning. In The Thirteenth International Conference on Learning Representations, 2025.
[13] G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088 2022.
[14] Jang, D. Yoon, S. Yang, Cha, M. Lee, L. Logeswaran, and M. Sco. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022.
[15] J. Jia, J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584-51605, 2023.
[16] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. B. Hanna, Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024.
[17] K. Kuo, A. Setlur, K. Srinivas, A. Raghunathan, and V. Smith. Exact unlearning of finetuning data via model merging at scale. arXiv preprint arXiv:2504.044: 2025.
[18] N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlcarning. arXiv preprint 2403v:2403.032/6 2024.
[19] Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyrr, and M. Lewis. Contrastive decoding: Open-ended text generation as optimization. The 61 Annual Meeting Of The Association For Computational Linguistics, 2023.
11

---

[20] Y. Li, S. Bubeck, R. Eldan, A. Del Giomo, S. Gunasekar, and Y. T. Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.54633 2023.
[21] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches oul, pages 74-81, 2004.
[22] J. Lucki, B. Wei, Y. Huang, Henderson, F. Tramèr, and J. Rando. An adversarial perspective on machine unlearning for safety. arXiv preprint S.Ki::2409./0225. 2024.
[23] P. Maini, Z. Feng, A. Schwarzschild, 7. C. Lipton, and J. 2.. Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06 00..401.0122.1 2024.
[24] K. K. Nakka, Frikha, Mendes, X. Jiang, and Zhou. Pii-compass: Guiding Ilm training data extraction prompts towards the target pii via grounding, arXiv preprint arXiv:2407.02943 2024.
[25] M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, A. Choquette- E. Wallace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035, 2023.
[26] S. of California Department of Justice. California consumer privacy act (cepa), 2018. Accessed in April 2025.
[27] V. Podder, V. Lew, and Ghassemzadeh. SOAP Notes, https://www ncbi. .nmm nih, gov/ books/NBK482200 NBK/8222K/22 2023. StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing;
Updated 2023 Aug 28.
[28] H. Qiu, Y. Wang. Xu, L. Cui, and Shen. Fedcio: Efficient exact federated unlearning with clustering, isolation, and one-shot aggregation. In 2023 IEEE International Conference Big Data (BigData), pages 5559-5568. IEEE, 2023.
[29] G. Sanchez, A. Spangher, H. Fan, E. Levi, and Biderman. Stay on topic with classifier-free guidance. In Forty-first International Conference on Machine Learning, 2024.
[30] W. Shi, Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint MoKiv:2407.064400 2024.
[31] H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arX1v:2302.13977, 2023.
[32] X. Wu, Zhang, and S. Wu. Revealing the unseen: Guiding personalized diffusion models to expose training data. arXiv preprint arXiv:2410.03039, 2024.
[33] X. Xia, Z. Wang, R. Sun, B. Liu, I. Khalil, and M. Xue. Edge unlearning is not" edge"! an adaptive exact unlearning system on resource-constrainee devices. arXiv preprins 11Xiv:2410.1011 2024.
[34] Z. Xiong, W. Li. Y. Li, and Cai. Exact-fun: an exact and efficient federated unlearning approach. In 2023 IEEE International Conference on Data Mining (ICDM), pages 1439-1444. IEEE, 2023.
[35] R. Zhang, I.. Lin, Y. Bai, and Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv e-prints, pages arXiv-2404, 2024.
[36] Z. Zhang, Wen, and M. Huang. Ethicist: Targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. arXiv preprint axXiv:2307.04401, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction highlight a new threat: that exact unlearn- ing previously regarded the gold standard for data removal- introduce privacy risks. This claim is the central focus the paper and is supported by the proposed method in Sec. and set of experiments that design and evaluate data extraction attack leveraging both pre- and post-unlearning models in Sec.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations Appendix Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper includes heuristic modeling to provide intuitive guidance for the proposed method in Sec. but it does not involve formal theoretical assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The main experimental procedures are outlined in Sec. 5 with additional
implementation details provided in Appendix Sec. ensuring reproducibility of the results relevant to the core claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
14

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code is provided via an link shown at abstract. The datasets used are primarily open-source benchmarks. The main experimental results are presented in Sec. with reproduction details provided in Appendix Sec. A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis..c)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results, See the NeurlPS code and data submission guidelines (https
ctapps.ccppubbicc. httpppplllppip....ppppppppppaeee Subbdeeueeoooeemmmmmccaaa ssi inoliicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The main experimental settings are described in Sec. 5 with additional details provided in Appendix Sec. srree-pprramters and their selection process are explained in Sec. 15.4 ensuring clarity on how they were chosen.
Guidelines:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance addressed in Tab. T The variation of A-ESR across three unlearning runs is minimal (standard deviation < 0.01), error bars are omitted for clarity. The improvements our extraction method over the baseline are substantially larger than this variance, indicating the results are statistically robust.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information available in Sec. A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::t httt://nneiiipp http.//pclc//.hhcccciieeeeeee
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and ensured that the research complies with all relevant ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
16

---

Answer: [Yes]
Justification:
The paper discusses the broader impacts in Appendix Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.lymmlo of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not center releasing any new datasets or models, and thus no additional safeguards are required.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use multiple benchmark datasets, all of which are properly cited in Sec. 5 with detailed descriptions provided in Appendix Sec. A
Guidelines:
The answer NA means that the paper does not existing assets.
17

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are used during the construction of the medical dataset, described in Sec. 5.5, but they are not part of the core methodology or contribution of the paper.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non nta-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Experiment Details
All experiments are conducted using two NVIDIA A100 GPUs.
A.1 Training Details
23 we begin with the pre-trained LLMs, LLaMA2-7B and Phi-1.5. To obtain the target (pre-unlearning) model with moderate memorization of the training data, we fine- tune LLaMA2-7B for epochs and Phi-1.5 for epochs on the full dataset, using a constant learning rate of 10-$ This setup reflects a realistic scenario: well model should neither memorize excesively-ccompronieee compromising generalization nor memorize too little, which would eliminate the need for unlearning in the first place.
Following prior works 30
To simulate exact unlearning, we train a second model from seratch with the same configurations, again starting from the pre-trained weights but excluding the designated forgetting set. This yields the post-unlearning model for our experiments.
A.2 Dataset Preparation
We experiment on the following benchmark datasets: MUSE, TOFU, and WMDP.
MUSE [30]: We use the MUSE-News dataset, which consists of BBC news articles collected after August 2023. The dataset split into two disjoint subsets: Diorget and Dietain containing 0.8M and 1.6M tokens, respectively. For k% forgetting set, randomly select passages from Diorgtt until the total number of selected tokens reaches 2.4M x 4%. The prefix known to the attacker is the first half of each sentence.
TOFU 23
We use the full TOFU dataset, which consists entirely of fictitious author biographies synthesized by GPT-4. To construct the forgetting set, we randomly sample question-answe pairs and treat the remaining data as the retaining set. The prefix known the attacker the question part.
WMDP 18]
We use subset of bio-retain-corput from WMDP, comprising collection of PubMed papers that span various categories within general biology. This subset contains a total of 5.3k sentences. We randomly sample sentences from this subset to form the forgetting set, with the remainder serving as the retaining set. We simulate the attacker's prior knowledge by providing access the first half of each sentence as a prefix.
B Medical Dataset Experiment Details
To simulate real-world medical data, we design our medical dataset using the Subjective, Objective, Assessment and Plan (SOAP) note (27] as a template. SOAP notes are widely adopted method for healthcare providers to document patient encounters in structured and organized manner. To ensure a comprehensive structure, we utilized the SOAP note template from Heidi Health [9]. medical AI company which offers SOAP note templates provided by specialists from the medical industry.
We format dataset into JSON with the following keys, "client name", "date of birth", "date", "subjective", "objective", "assessment" and "plan". For the generation process, we employ Gemini 2.5 Pro using a specialized prompt:
I would like to generate synthetic medical data for machine learning purposes. Specifically, would use SOAP notes as the data type. Below is note template you need follow, which has client name, date of birth, date, as well as subjective, objective, assessment, and plan. The template is just for you to refer, you do not need to generate each line of the template. Instead, only several lines for each of the SDAP is enough, try not to be too tedious for each record. For each record, please generate with PII (client name, date of birthday), person per record. client name: [name]
date of birth: [birthday date] date: [visiting date]
Subjective:
20

---

[Dsccritioon of symptoms, onset of symptoms, location of symptoms, duration of aymptoms, characteristics symptoms, alleviating aggravating factors, timing, and severity]
[Current medications and response to treatment] (write this section in narrative form. Write in full sentences and do not include any bullet points) [Any side effects experienced] (write this section in narrative form. Write in full sentences and do include any bullet pointa)
hon-pharmacological interventions tried] (write this section in narrative form. Write in full sentences and do not include any bullet points)
(Deccripioon of any related lifestyle factors] (write this section in narrative form Write in full sentences and do not include any bullet points)
symptoms] (write this section narrative form. Write in full sentences and not include any bullet points)
[Patient's experience and management
[Any recent changes in symptome or condition] (write this section in narrative form. Write in full sentences and do not include any bullet points)
[Any pertinent positive pertinent negatives review of systems] (write this section in narrative form. Write in full sentences points)
not include any bullet
Objective:
Vital Blood Pressure: [blood pressure reading] (only include if explicitly mentioned in transcript, contextual notes or clinical note, otherwise leave blank.)
Heart Rate: [heart rate reading] (only include if explicitly mentioned in the transcript, contextual notes er clinical note, othervise leave blank.)
Respiratory Rate: [respiratory
reading] (only include if explicitly mentioned in the transcript, contextual notes or clinical note, otherwise leave blank.)
Temperature: [temperature reading] (only include if explicitly zentioned in the transcript, contextual notes or clinical note, othervise leave blank.)
Oxygen Saturation: [oxygen saturation reading] (only include if explicitly mentioned the transcript, contextual notes or clinical note, othervise leave blank.)
General Appearance: [general appearance description] (only include if explicitly mentioned in the transcript, contextual notes or clinical note, otherwise leave blank.)
nose, throat findings] (only include if explicitly mentioned in transcript, contextual notes or clinical note, otherwise leave blank.)
HEENT: [head, eyes,
Neck: [neck findings] (only include if explicitly mentioned in the transcript, contextual notes clinical note, otherwise leave blank.)
Cardiovascular: [cardiovaacular findings] (only include if explicitly mentioned the transcript, contextual notes or climical note, otherwise leave blank.)
Respiratory: [respiratory findings] (only include if explicitly mentioned in the transcript, contertual notes or clinical note, otherwise leave blank.)
Abdomen: [abdominal findings] (only include if explicitly mentioned in the transcript, contextual notes or clinical note, otherwise leave blank.)
Musculoskeletal: [musculoskeletal findings] (only include if explicitly mentioned in the transcript, contextual notes or clinical note, otherwise leave blank.)
Meurological: [neurological findings] (only include if explicitly mentioned in the transcript, contertual notea clinical note, otherwise leave blank.)
Skin: [skin findinga] (only include explicitly mentioned in the transcript, contextual notes or clinical note, othervise leave blank.) Assessment:
[Likely diagnosis]
[Differential diagnosis (only include if explicitly mentioned the transcript, contextual notes or clinical note, otherwise leave blank)]
Diagnostic Tests: (only include if explicitly mentioned other skip section) (Investigations and tests planned (only include if explicitly mentioned in the transcript, contertual notea clinical note, otherwise leave blank)] Plan:
(Treatment planned for Issue (only include if explicitly mentioned in the transcript, contextual notes or clinical note, othervise leave blank)] [Relevant referrals for Issue 1 (only include if explicitly mentioned- [Likely diagnosis for Isaue (condition (lever come up with your own patient details, assessment, diagnosis, interventions, evaluation or plan for continuing care use only the transcript, contextual notes, or clinical note as reference for the information included in your note. If any
only)]
21

---

information related to placeholder has not been explicitly mentioned in the transcript, contertual notes, climical note, you must not state the information not been explicitly mentioned in your output, just leave the relevant placeholder or section blank).
Then, here is example of SOAP fields for you to refer: Subjective:
The patient, a 52-year-old male, presenta with a new rash on his back and arma, which he has noticed for the past two weeks. He describes the rash as "itchy and red and mentions that it seems to be getting worse despite weer-the-ounnee anti-itch creams. The patient denies any fever, joint pain, recent exposure or detergents.
soaps
Objective:
Appearance: The patient appears well-nourishs and in no acute distress. Skin: Exam reveals erythematous, scaly plaques the back and arms. There is evidence of excoriation due itching. No signs of systemic involvement.
Lesions: Lesions are well-defined, with some areas showing mild papules. No signs pustules or ulcers.
Other Systems: Vital signs are within normal limits. No 1ymphadenopathy noted.
Assessment:
The presentation is consistent with psoriasis, characterized by itchy, scaly plaques The absence of systemic symptoms and well-defined lesions supports this diagnosis. Differential diagnoses include eczema or fungal infection, but these are less likely given the clinical presentation. Plan:
Initiate topical treatment with high-potenge corticosteroids to reduce inflammatin and itching.
Recommend emollients to improve skin hydration and prevent dryness. Educate the patient the nature of psoriasis, including triggers and management strategies.
Suggest lifestyle modifications such as stress management and dietary adjustments potentially improve symptoms.
You should add PII in front of the SOAP. Please generate 10 records for in jaon format, with "client name, date of birth, date, as well as subjective, objective, assessment, and plan" as keys.
To maintain data quality and prevent degradation during large-scale generation, created the dataset in batches of 50 records each, producing 1,000 records in total. We replaced duplicate client names with unique ones, there would be low probability of identical names appearing real-world sample of this size. The generated data covered a diverse range of medical conditions to ensure representative sample of real-world clinical scenarios. We randomly sample 100 records as the forgetting set, with the remaining 900 records serving as the retaining set.
For illustration purposes, an example of the generated records is provided below:
"client name": "Noah Garcia",
"date of birth": "2012-07-22",, "date": "2025-05-18",
"subjective": "Parent reports Noah, a 12-year-old male, has had intermittent abdominal pain for the past month. Pain is periumbilical, crampy, occurs 1-2 times per week, lasting 30-60 minutes. No clear relation food. No fever, vomiting, diarrhea, or weight loss. Appetite is normal. School attendance is unaffected. He takes no medications. Parent has tried giving children's Tylenol during episodes with little effect. Parent is worried about the recurrence.",
"objective"; "Vital Signs: Normal for age. Abdomen: Soft, non-tender, non-distended. Bowel sounds normal. No maases palpated. Growth chart parameters normal.", "assesement": "Recurrent abdominal pain, likely functional abdominal pain given age, characteristica, and lack of red flag symptoms.", "plan": "Reassure parent and child about functional nature. Discuss potential triggers (stress, diet). Recommend keeping pain and stool diary. Encourage high- fiber diet and adequate fluids. Advise follow-up pain changes pattern, becomes severe, or red flag symptoms (weight loss, vomiting, blood in stool) develop."
To capture the complex structure of these long medical records, we fine-tune the model for 11 epochs, while keeping all other settings unchanged in the experiment.
22

---

Our Extrectio
Our Extraction
Pro-enleaming Model
Baseline Extraction
0.31
Baseline Extraction
6.5
Ronge-L/tt
0.
A-13RR
0.30
sisttry
0.00
0.00
644
0.08
9:07
EU GAcuNPO GÁe. NPO,,
EU
GAupNPO Ghu, NPOa.
GAuaNFO NPO Ghu, NPOp, GA
Unleaminn Method
Unkeariin Method
Unleaming Method
(a) Rouge-L(R)
(b) A-ESR1.0
(c) Utility
Figure 12: Comparison of our extraction method against the baseline under various unlearning methods. EU refers to exact unlearning. In some cases, the effectiveness of our method weak- ens poisprriarllyy due reduced model utility, which distorts the guidance between the pre- and post-unlearning models.
Table 4: Comparison of our extraction method with baselines under LoRA fine-tuning on Phi-1.5 on TOFU dataset. The improvements are consistent across metries.
Extraction Methods
Rouge-I((Rff A-ESRo.9*
A-ESR
Post-Unlearde Generation
0.412
0,010
0.003
Pre-Un Pre-Unlearning Generation
0.544
0.070
0.055
Ours
0.614
0.160
0.108
C Extraction Results under LoRA Fine-tuning
In practice, parameter-efficient fine-tuning methods such as LoRA 1T0 are increasingly popular. We therefore examine whether our attack remains effective when the pre-unlearning model LoRA- fine-tuned TOFU. Specifically, we provide results Phi-1.5 fine-tuned with LoRA TOFU for five epochs using a constant learning rate of 2e-4 and a 10% forgetting set, while keeping all other settings consistent with Sec.5..2 Then, we employ a guidance scale w=1.6 for our extraction, increasing the extraction rate. The improvements are consistent, as shown in Tab.
D Extraction Results on Larger LLMs
We further experiment with larger model, Mixtral-8x7B (16), which has substantially parame- ters, employs a MoE architecture, and is instruction-tunet making it sufficiently distinct from our default models (LLaMA and Phi) to better evaluate generalization. Specifically, we report results Mixtral-8x77 Ex77-Instruct--O.1 fine-tuned with LoRA on TOFU for two epochs using a cosine learning rate schedule with base learning rate of le-4 and a 10% forgetting set (all other settings follow Sec. 5.2)). For extraction, we apply a guidance scale of w 2. The improvements remain consistent, shown in Tab.
E
Comparison with Other Extraction Attacks
To further contextualize our method within the broader landscape of extraction attacks, we compare against ETHICIST (36]. representative approach for extracting memorized training data. ETHICIST operates by tuning soft prompt embeddings and applying loss smoothing with calibrated confidence estimation. We conduct experiments the TOFU dataset using Phi-1.5 under the 10% forgetting setting.
Notably, ETHICIST assumes stronger threat model that the attacker has access to part of the model's training data. To align with this setting, we assume the attacker has access to half of the retained set in TOFU (while remaining blind to the target extraction set, i.c., the forgetting set), and use it to
httppp//wugggggeeeeeccoossssssss 8x7B- Ins.r--tttuut--o..
23

---

Table 5: Comparison of our extraction method with baselines under LoRA fine-tuning on Mixtral- In---nnstttt--O.11 on TOFU dataset. The improvements are consistent across metrics.
Extraction Methods
Rouge-L(RY1 A-ESRo.97
A-ESR1.0
Post- Post-Unlearning Generation
0.372
0,015
0,013
Pre-Unlen Pre-Unleanning Generation
0.465
0.040
0.018
Ours
0.537
0.103
0.055
Table 6: Comparison of our extraction method with ETHICIST under default fine-tuning on the TOFU dataset. Our method outperforms ETHICIST by clear margin, and combining the two further improves performance.
Extraction Methods
Rouge-L(R)f
A-ESRo.9
A-ESR1.0
Pre-Unlearning Generation
0.566
0.100
0.070
ETHICIST
0.570
0.118
0.073
Ours
0.643
0.202
0.120
Ours + ETHICIST
0.652
0.213
0.153
train the soft prompt. The hyper-parameters ETHICIST follow the configuration provided original paper's s open-sourced code.
As shown in Tab. experimental results show that ETHICIST alone can partially improve the extraction rate. However, since ETHICIST is designed to extract training data from single model, complementary rather than conflicting with our method, which leverages the difference between pre- and post-unlearning models. In fact, combining ETHICIST with our guidance-based approach leads to even stronger extraction performance.
Visualization
In Figs. B
16
23 and 24
present several examples under
default setting using Phi-1.5, with our method applied using the default hyper-paramette (w = 2.0, 10 10- For each dataset, include examples where both method and the baseline fail, where both succeed, and intermediate cases where the baseline fails but our method successfully improves extraction.
G Limitations and Broader Impact
In this paper, we show that exact -oriannnl-o intended to improve model safety-can, in fact, introduce new privacy risks. Our method relies on access weights logits api from both the pre- and post-unlearning models. While we justify this assumption using a realistic medical dataset, there are cases where pre-unlearning checkpoints or logits may not be available, such as in closed-source settings or when attackers fail to pre-save sufficient outputs. This limits the general applicability of our method. Future work may explore leveraging public model outputs or ceneral-purpose knowledge priors, as suggested in prior work 2
Our extraction method reveals privacy risk of exact unlearning that, in principle, could be exploited in practice. However, as with other papers that focus on attacks, our goal is not to promote misuse, but highlight a potential vulnerability before it leads real-world consequences. By identifying this risk early, we hope to encourage more cautious use of exact unlearning and to motivate the community to proactively develop stronger defense mechanisms.
24

---

Thra be Porgation
Input Tatts
Tesainn Exmmution
Our Forrction
How itteriial Elstoon
wannig Hoserizl Fleten
vinain itanrical Flaton
How Ilirettcal
Expallence Avand repaced Curmen
bacikene Awaed iepaced Comman
Brcellaace Amard resuce Carmes Monnaggr wittagg
Exedlenze Award inpacted Cormen
Mennaggrck wortagg
Mensaggoh sittag?
Manzagros wittag!
damas
Asreg
Aneesp
Aamaz
Wilniing the Ellaaricl pionoe Supelinnce
Wassing the titasried Pistis Bxrallence
W Winsing
Plcctos
Monnaggoi cooffince and dedicationn for cnit suppinng her houecainee
coefilence consaltaver craft ba lermose micogttree is
Carnon Mestveggro casding
bieveaa ganes It anly vilibae
and contoce perdaciag cipetrrting titurical
lamory world kading to mudrnsily
metes meszch corpelling
cigttfise
boundariis alsturfud narratioe.
Quasties
Qucalees
Qastier
Rayees Maye porty his
Rajorr Majondar parny
Time Rajer May ender portogia
She Baker Maxanke peetray
chancien!
Assoee
like madied. frong tsing floo and virtoss hopes and lears
koeso,, with sene Jemintte
bought through fivir dagh,
He peetrogs facn with lics undenseding
complesity tes unllnnanniss Hischancion
corflen aragglc
redjai fipecnta imagindoo,
sedes expalline witk then.
Figure 13: Examples from the TOFU dataset illustrating hard extraction cases where both our method and the baseline fail.
Dasa to Forgotion
Insut Data
Basclise Excrution
Oue Extraction
Quacttore
Qusiiin You hak!
Queclie:: You Palk?
Qasilan
Car
ofanhhr
Yeer Purk
Yon Pak!
Ameuue
Assar
Autive ji Yewn Park was how Seeal
Autharr Bi-Yeex Park Sourt Kenea.
Author Park hern
Sesul South Karra.
Quaatox
Quetion
Quertiz: How pereaatie
Qaosian
whick Yan verd?
dlcacpaation thin Yar- Hea's perstaffe"
Yas-
sccupation
Hoak pasmas
Anmer
Yowr Park srmurtly snteesaatis ponze Fnndennk..
B-Yaa prinanty srteewitti sanes of aalendes.
J.Yon Park primuatty writes within the gaare hadde
Ji-Yeen Park prieaatly weites wiblle the grare leadendipp.
Figure 14: Examples from the TOFU dataset illustrating easy extraction cases where both our method and the baseline mostly succeed.
Plrepation
Qaatiia
Guadis::
Qaatian:
meoeiiations liiie Yun-
Whara accapation of eftla Yan-
What the socepatinn et Thtte Van- quemmet
Whr the accapatios oftibbe Yan- Ieaa pasmec
Iooae parmact
Amaus
Anne:
proctioo
Hiss Yendloa''s lelker Resenol Veloinaiinn
The pereots of Holee You Hhe are distngesshed,
restier
warking being snereplyydk
cnsi awther biing awoeplaye
Qustian
Quoict:
Qaotien
What makes Niodai Ablis take Afivoe
Wha makos hibola Abiin's sake Athica Ameticas sammin unizact
What sukza Nibaki Abilen's take Allicun rurntivee
Wha rate Niboli Abios's Afhize Aneiica samaio unague?
cniguet
Aamer
Ntoten Abile'' aniac oottllation
Nikato Abile' caniger contribution
Nikata Abils's usiger onstribation
Afiicn namativo fis
Africe Aseriine aereative in
Africss Ancr secrices morrain liss
andlty inclaivity veailtyy
interwatenal persrecive wersing
flens of Kanahitn
and LOBSTQ
alceitics to procctec globtl alvens lako Aoceican Hkemttre.
dkalkegge vediione
LGaTO- ineeiiies, present glstel and dierse take Africes Americas
Figure 15: Examples from the TOFU dataset showing cases of intermediate extraction difficulty, where the baseline fails, but our method successfully recovers most of the target information. These cases highlight the improvement brought by the proposed method.
25

---

Thra bo Porgation
Inpit Tasts
Tesline Exrmuaiin
Our Forration
Tistee Sctage -arcar Easy
Afes be inmbdd Carze, hernii
Abarle Laakd Curtee be mallcc bud "hiteal Autrac aade Leaty
After ladde Cyper cld
Alf landed Dyimas
way bucc ta tacsane line lid. cy lastl
was to
and Ey
and ee 1sy
way bank U.k: lare
lighting
for lana
Hose Ri Willie coriiedd euuns
Ila Ruduut Wilk erganact ever. wit The Cals
Taee Rilbed Wille ereriie the mone 1201 The TeaeCalt Sarryy sppliing winowrairs
Sill Toar
zal, The Lanc
Keight
a-Sicalam aveccaiie vila Dx Tex kannen.
Figure 16: Examples from the MUSE dataset illustrating hard extraction cases where both our method and the baseline fail.
Dass to Forgotion
Inss Data
Baslise Exeraiio
Oue Extration
Pascierr Extinen ancidd with
Prosder Eidegur rededwet petooer ceaffeel ruabe Alrart
Pootiar Lato unded with put oner S2%
Zwetter Tetta unded with Th val haadd seanonnplin
The hand cnirorpliaz
lutf tue elacteris deiipyp pelanised
alectrrtte
peturied
disterate
petarfed
courry auhoriarrta vision
cenenr did nut back auttouttartes vidue Turloy.
coenty did set back autboottartas videe Turkuy.
Lipp thi paasstocn off conestiin tha ocota
cun Cffire lage
bead
sittel cinnn colllaneea
#esome Nean victe, gloxx them
asponc niw kas sicties Lighoo Bion semcece
thera jusioc..
Figure 17: Examples from the MUSE dataset illustrating easy extraction cases where both our method and the baseline mostly succeed.
Forgation
Baseline Extoction
Extrction
Chasidi athehared betts Biniii When he was thee Awand 3130
Caoid celehored with bunte Backto
Candéé Budkla
Beil Awart
Award 2830
The fart botowar corlland moneois
Thenken loonce untiaal Satmddattss
The plas leaaser contineed ed morts carion in Uicaaes
Thepe cooeesting satt offie and T'pee reacter modlig in renaising back-sp
Figure 18: Examples from the MUSE dataset showing cases of intermediate extraction difficulty,when the baseline fails, but our method successfully recovers most of the target information. These cases highlight the improvement brought by the proposed method.
26

---

Tarts be Porgation
Input Dtate
Baskine Exeaciio
Extratiinn
The reitee mas slind an thes asarsiied sith CiTs-ls yona
The mosta aue stima serpernue Sc.4i thes carna
Thesniuue sinal woee poeslamaanies
1he POLKESE sou and PEPAAAATTOO
give
5kc HCB
HPEC
IKC partfied liPe
CalMa JIEPS Semee tnnoal
Camhi AM 06 Jome, fear, dadeen,
Inxod rreeal Alrood frandd.
Imrrod (loooo fo
shaght cuntly
marde who had besueted thooghtt a whant get snydinn?
beat lanls
filends and my dog and my and
ziked danit
adlid Aind& zo heils
mally you adudd
thi sen chthing but dideii tackk war
Figure 19: Examples from the WMDP dataset illustrating hard extraction cases where both our method and the baseline fail.
Forgatin
Extaction
Dhapoiii puziin (spe:Zabee))
Theppekis supeeeo (CCC-TR pokin typeC
Moxtitad "oubiiquaa 2oe-
axpenbablrcaa
and (IrFFopenrtal (ow
and rine (ür pe were ovel typer""ig"
and rine (iretteee (i "take"" wore neel plylegmetic typer/"ig"
phylegmetiic
the snlutratte teaperatare incrased 25" 279
vabetratt tumpuratann
25" 235
Figure 20: Examples from the WMDP dataset illustrating easy extraction cases where both our method and the baseline mostly succeed.
Data to be Forgotion
Impat Thots
Baslice Extaciio
Our Esrration
Acatt dosid inaraeted cautlandy
Bashhsbaald anappred canticalyy canSnening Disc aecaiieiiie kara
Basta decil comeant
Santa deell comeinc
sonfranden Dlscestcc san itamuncc
locadactio
alum
tevedly oeton is gusas peisle
breadl pepple
breadl rolers
bane sutorical overastion
calardl grops dul take divene
have sone biteriaa conncction cise
popelaaoo Unated beate.
Spania hegusgg..
Junall more afratinaa attmenatttttttaaaa athuiassa
Asrull inness perkonel offesiee
anerta aeccccan etfeace sdaztled arrend merke Misical
mtan Hilee
inevifee avaud he splen Melikl
Identiied aound
motners
sutterned edliee
iaparshaty perivensd lna wordd whre ucheccloo a craraady abvanciag
laparoeuayy were pur purformed
soun qlaii infarantion and atuection.
Infarctioo.
Figure 21: Examples from the WMDP dataset showing of intermediate extraction diffi- culty. where the baseline fails, but our method successfully recovers most of the target information. These cases highlight the improvement brought by the proposed method.
27

---

Adhamary's eode information sbout parsen
Beadine Fettration:
Estrction: combasita cleckpoins
Puteect data be Rogetios
pre.mlommingggg...o chodlpaint
Oatrid 01:31 302800208 sufeadivee Mr bell a (h-yarekll wokl notts als produt cuneer 2 spx,
shoco
01-21 Maoc 3825511-48
21-21 202-5.35
vetocriver Mr. Bell Téxcanabl taal: oatapennia (Toone- tig Staned
seljestiee Mr Bell is yoycand mnde
refored endennnt
dashsens
sconadlin spant) Provcte wall wife bokky disces pach of cust in experineing incresed pann Gatges. svcelll dedine: Undersiadod dise so inceraiee objective siective Shane Appoors falgadd, cechootoo Acheoood
Eeperts leatham difiadly
dicsecte meliee ldte
mellcwing pilasine during vediarins.
tnccoing lelgue for meeth
sbout
abyettee
aheirittiion
sigiifone Mooding
for alcaioreio recoeeen Oukopereis beameet indicded wil alocrreaze
Rorm Pelive for suttrre boclyne tyroees No tyrefadoopu To leproppe sonepily
aveeiraee with sigaifiune sprehorr twotess and Fancional deliii Patien fuaiily realy to lec gails o pallariiz approcci dosarree
Experien epprieeiigg Reaebore
nuphaged
anlieege cpoperics
odvindaation of aloddoaee Tako with fal plas sip oster only thing in AM.
restrghils rpportd soumenent
opee Conceriing for cedeliig
yalrrrs ordertredtt eding of progaeee Pxplase salion, hogne Guan
Eediiiikined indds Reraai jiniigg doedings ug for lran 30 moi after sking of symprein parie dequie comer
Myslodap uste Syndrome
oplosto cneeri glare ago and coeer
gook c quiliy
dadings Hloncasiogi nal gaawy rends
with peeentul de affeann Diarss oomptti managorin plas toptesizo analgeds cansalt pallativa rudi wioa
home mutteer life peoligggng
difioee ond hiiphoghoeae medirare rra line dlylety boae nopthggal norkon nics
Bhruasiogg Compuhrashen
henusiegy will inclsde Rusten
perphenl eneor by heraropattile slogie orrllggee Deppttiic sumiimnnan scrers, lisaty Done blasss Appiorton and Bippy for detiettiv diggrade Sapporiso care beeeeiitooeeeeeeeeretkkeoe dagrods
anneallyy. Seitch
dangaa arunisali Diecas coda enmatlafronce menttrre aluudy specialart
agein deeeennch sutpaontias Diccns opntans "aompesurty bold alandmoaas,
greppees cheice 1-3 weaks Exdocrtddayy refurral If Movesples.
munapurent poraput and regsing ori exppon Cifter hoop veturnal
clic manc Jonk Hhades hist: 1e 301-41-43
elennmans bonk dhac 1966 dac 01-43
diat sane. Josis Hagho ins of liffs shis 28130)341
clen man: Jost o tiih Hés. 302-0143
sak presening with sake inilaies and
offdioouts diaas cdlapeee eanouge) He tikeke iydhostien: Seg Ald ling
with pirko boodl isted
crastag of his cydlah biticrall fo and Sympkees wame reaming. grite liee then, Deniavnaion clongooee
sctc (Ienndopontic) be werk
Phd and Indrscationn lon dully Reperts pool
Desio uriery semplenn peall indaggo
Doscrbosigee
same incraeed litga cousioenl
geedlice Net la melicaion. elpeetive Gestearney Euees Norcal
figat pore, Has tood anficial tors objective:
in the pan weeth oppnaaly
Tyes
cosrgency
excoral genkals Hurrenad pand
Sikooall cny sailing and ensling in the the eyclades Conpeaciie rebiily eepited
soral
abjectics Whll Signs oP 1nbi5 rerlig
DES nomel protine DRF geis sermel. eocounseen Her aooperrea Corld hase
Oyiegi 9l0e veriing paod uo wit wable Skiw in palieer erooees Necooliggo funoes aseecneett Libely daguock aidioo's sigle
Wiiht
repoodec pnaation
Holy Méphoriic given firdngs pla Rocornond epvidd begine mescase Worn oocposce nillowe
essowren
dicllege pelio par Lass luely jost
male patton gineegge sifood
ayulid norghe with daby comppo oe cannuncil if eyulit cierier Anitcal san
wijor of cortoonni
ESR-CIP trandyasia
sympeeee of ofveicclyppaoone fonatio (frigan onhocarid plan: Ter peretll fomene Singsaa
exeucied (Irw lnnel ligh and PARCCR
elecardd moar sutborkk
Aeeycyclive otbegge
orippial araibuntz simment crybeoerpcia applladno id nargin hettina irgggiiic cruinng of
havenss, and
other
encis adapuse oif leatn Beadtees rick day miler and emss ducing Chock diccoolyte dow frsuriigg tingee teaa. Fallew clouly was eadectisiigy Boere crurpaccy in op
zoumss mule paneer haidress Adviso
nelkoe-ap abor hornonal pasd revine:
waslly
uertral porsanionn pepo losa.
Figure 22: Examples from the medical dataset illustrating hard extraction cases where both our method and the baseline fail.
28

---

Adhomary's vode informaiion dbout the pallent
Beadine Estracion: wing ualaini cheligritt
Our Patmetion combining pro-ond
Puteett data be Gogeties
checkpuits
cles Harads Caree tiic
cien mane Elnnah biice 2112 sac 202-48 48-23
Skat Herek Cato dac of herts (2144)23 dae 20:3
die meae Hardd Grecdaac of
1190-41-25 ane 30144--33 autrective Mr Cascs lacker-gp
152-41-2 Eac 32244822
intoeciee Gato, tuther-ape erarlas cyst
sabjactiee Gaiok for fallan-ap
ovay Stuad
loaad
the kill foxnd
pehiz alrastand dene be asadiaed rousco mecdlo san axmptommic.
pohli altmaser desc for aurclatd ronets insull ran
pdrik dirassed foee for unrthaed roaver manth sgo
rncuuzal sycle Sie and
bluiing,
nesstren cyde.
leating, clangee monined cy
cosmappiovs Ne junely boieyy pratian caeer oheetive
Ske mstg seal cottreapiio. fanaly
She is saing and costraccpiics No lensly foveite caser abjectnc
Nerma.
lalery snarian canem dipetive
PesicoaantE perferned dete tfae indiaed for meserceeaas dehooages thecourlaaee dlc.
Nansal
loske Pab canm herkeaad, connll.arddi ardecr il nut insiicie for cnypphanniic fallen- spic larcerrruabl, dharsctrbiiiie sucouneat: Likely
tendir Iefric (f perfernedi, defer
tue inlivated for soymptamatl fchoe.
gric lirooendreppon ooffeiing
prirr
upht Curcane
Likely deganiic Agrepeents sigle
charasteriios sovminnt: Liee:
caanas cyst lell lBely Tandioal Los risk repipnency expeesa dter,
diagsokk Asmpptenati sieple avariaz cyst lidds banctisnal
diggessle Assaggaandie dagpee onnooii
Ressare
paior the cieplegyee are coneeas and oeally herign othe oadking
with simple -dexs Resoure palken that staple cas are commss and sxadly bextaa, rosling
with single cud close plzni Resware
patiest that sieple cyss commton and
sinplo follow yobei
beigg reohhing
cpastasessdyy Given
degle
fod craath to moceositilty dibility armoaiiee litio Conraee cureet cntropption Addio g. Moutingi insmmiationn nended at tinee
recantmend fallan-sp pchle eltrawcard in 36 maatth to casere staiiity roedetias. Contiass cerreal eupplee devcke polric polr, Meuting, No otherr cstervattio scokel at
rature, moceenet Selire-ap patiic
(ltrassedd mncell conare
exmpptoe dendag (ng. pablic pais, trtallags other inserveddis
1907 oi das 3005 45 enteecivee Me Larhhn ina 24yyar-ot
1007-1100 das 2008-84-35
1915-1 dee: 3136-44-45
1907:1 2004 16-35
ungjectv:: Ma. Lambert lea 24yaar-
vakjectiec Mr. Lansbert it 2i-yaan-ld
terale comaes uae shaiher proecasa)
frnaas (MSM contect umm dueiker
fanal (MS5 conest- sus shcsha
prrventie. Roports matilee rule
IBV perceatiss. Reparts eukiipl male partser, lacmmbie coddum unc. An are arPvilt warn star. abjecticc Smanl
HIY preventios Rapert mutiipl nuls
parmen, secessint conker tai, Avurt testt wase obeactiv::
patteer, iecentural crndees are. Aware
drEE wasts objuctive: bessal
sapaii ItSV getar
AgtAb nut aqgaine toaay, Inmure
Agláb sezathe taday, HBY
vacsied,, Reaal feacsen sacnal socornas:
(ptar vactacc Ronal hasclis nernal
geher vuccis). Reasl fnection secand.
Apprepitac cundkas
hil
encreneat: Apprpptatt casdidatc
mace Apprapriate cundibatc
FIEA (Dasany), Doand inportanc of adartcace for ealos
latiaie Pr27 with real FTAY (Boons). Cresnd on ispurtazce athercnec effetiracon Dacas
lakiatc PrEP with dally and ETEA
(Deicrry Crazed sn impertasse of
Duaca poinna cffacs Schodaki
odher onee elletivenno, Divan
scraning (00-C0 SC-CCTSpphiii mosnberng alaaznn dioci and proeciptinnn setil kankees for S1ll poveniis.
lancion.
falle-sp repest HIV toding, Sil scrooing (CCCC phyilll rand fasetke masiiciia, athereans
lethee-pp for HIY koilng, SIT serccaina (GC C15yhhhiiii real funcies medilriig, aelerenee check, pretien indialing nanokett aendam for 511 proveaikos.
peadier inclling consiitcal condeen Iir prevanies.
Figure 23: Examples from the medical dataset illustrating easy extraction cases where both our
method and the baseline mostly succeed.
29

---

Achamary's vode informaiion cbout the pallent
Beadine Estracion: wing walaaniiii cheligritt
Our Parmetion combining
Puteett data be Aogetiis
checkpurms
<Lea Wakee Brods
were Wakee finsis doocurtiiss 2017 de to Say wpr (01-48--3
cleer Waeer trokedees oftirtt M013e 38 dew 3636-8112
die Vobee fieds of
2011 de to Bey spr 0o-13
201T-35 to fer 301te
siteorroe Waler i browghtiy paree
satgeetian Watter teaaghe by paset sone and epporaatl bf
caljectne Waker broeght parrer
camptaring
pala
reatly ohine night
Ropert
moaty ehin rlight
exceal euonts tais sypciilly soress.t
firling odined internied pwoik ocul diatiess and cade the thee
uneral meeth. Pals oypically sccurvaat
ragle conatina: sokiing from chep
right, senetines waking
slep.
in
digttin
ipwring
actirass No sncilinn radavse,
They concareed abjectev Viul Sigic Namaa Prycic uaars tiacading
daytins actieitts lirping railling
lnntutice of acticiity Raturadity by eassuge acotrtiopher haneet concerned
radses, ar Bistzatian of acthity, Relloved
schoo reterali dhane centeer rign
masage er anstambooh Parver
Signe Normal Mescaloiaders Donn laver corenitio nerral No traderass, aedling wamenk detteruty thal mage
ixancie seating sumesenti Lulyy
objective: Vind Signe Necaal.
dlagaask: Chicbhed Betytag
Maseshskektal Ecan aflener
huthallpypiatt Sabciskal Amaus abeat
estrenntin nsmmal teadamon.
Nearokgical curs cocval
plan: Pooids mysheddactte abozt
aodies of lown, sakin meraal Nerrabffal caam cecmud.
Likeyy duancii Grewiny Paru thongs recurral pascoll deiddroci, aires
balling pacriacaide crid talaka ctichs fuclinas Disaas smasii creagng wit badying (ou.
savertien:: Libet daaznsli Groving
cackuiice Renonae and ckid sbow. beniga zatase of pronita paiaa Racommend gymppeeaak Exeaz alcenat caldina if pus bocens pasince dutia dik.
Addice chil asody
childhaodi, giva clasic laiory
doos veigeeippammnn oppearacee Reasarc sbeu autnal noals
mecual cars. Blagans of cuckakes plat. Reamar paraat and aboul bestan
sovers
inaie.
ounneling lighhapp boduchiall parais. Feca en buikdigg chiliodl-ooccc and podieccopttu Noniie aetinee
yyaphonalic reliet mavagr. bas,
secouniasphet or llggriica seoled.
uniiborni, auscclad with ayviens vedling, fans larher inceigazze would be verrenedd to rale out ochor craditenn Fullow up socia..
wonsuing asisty.
auilatiral, suectatd with syskanic sympphss llove, neggn hoo) leaping, or joist forihee tolgttinti
condition. Felhow accied.
charr namer Carbia Mants den ettyrtt claar Carbls Manta car ortir
dleet cane came Acaala Larber uijjedtiee Martin di-gar-sld male vin hirvey of ofnernee callain right lawar diggassd monts aga to dicucs
dleeure Cortis Marts
of-yor- old anppection Mr bocheeddda dlogroed for silineep up ofcasscheed vesto ags patal coure of NSAID unommiatti stary over marra
vabjechhc Mr. Martia (5-yaar-ld mals for Stloe-a ap of nottsceedrita
daggened wois ago. Teak bottal couese
srappies ebjettiv:
have internittnt
pala over lift
bonte: sugan and by caraan in man anara prosure obeactive Mtuczbelickkat
taan Right Lever inpersty csan cernul. agas actt infuction assounent aysppienazc
strral beedar aggranaed cantal
mevemeess er promars. abjectic:
cosochandaa and juactien ratookechlzz vith papenia Chrricc Pulraaerry camt. corna ancannent Coracaradiiaa
dialggii for
tadkranz lift and 4tb
paversica plun Diccao risk ficves and paccetics dracgiic legroved losions dki und ze VASAMll ris acart, and atiikeg! protolaci
cortcchadrall jascizes rependaciiic with
pndpatiis. Cardac Phlmanary care
lemouzance repatiigg
dadiin
peroblunt
dapli laita NLAID
eakededkikii erlats Raceenond
trtdl plas: Besssamacc repardour
cuminanlistttoo
avodince
sagars ica. ttraia
mandds wrgis. Recenmccd
NSMI gel idkdekkaa adl antiaa leally. Strchiingooa oucaties fie shoi wall mascle ma pai cemiio sggilfiaally seeskeree
Contaac cunce tatibikk cptalazii
acivllte. Trai osticall
il lager meakd. Kee lint appllo aooadds
NSAID ad cllediease adplle lecaly.
Streicing cacrebes trrduct modo
exiion auril injetion inko nossehendal jaxdoos Fullow up PIN
Inchereenc, consder relordd lar lealized
centionoraal injecher iato cntediandra jundies, Fullon up PRN.
Figure 24: Examples from the medical dataset showing cases of intermediate extraction diffi- culty, culty, the baseline fails, but our method successfully recovers most of the target information. These cases highlight the improvement brought by the proposed method.
30


Reviews:
[{"review_id": "8KVQiNkaK1", "reviewer": "Reviewer_SsS4", "summary": "The paper introduces a novel data extraction method for exact unlearned LLMs. Exact unlearned LLM refers to LLMs that were retrained without the extraction target text included in the training data. Exact unlearning was considered to be the safest “gold standard” option. However, the paper suggests that with their new data extraction attack, exact unlearning could actually backfire causing more security/privacy risks. The data extraction method assumes access to either the weights or the logits of both pre-unlearned LLM and exact unlearned LLM. Leveraging the exact unlearned LLM’s logits as reference, the attack then can select tokens that have high probabilities in the pre-unlearned LLM whilst having relatively low probabilities in the unlearned LLMs, which likely reflects the distribution of the extraction target text. The extraction attack proves to be effective on unlearning benchmarks with significant improvement in extraction metrics.", "questions": "- Can the authors please elaborate on how their method would work on chat models? I'm not really expecting for additional experimental results considering that will incur a lot of load, just sharing your thoughts will suffice.\n- Were there any real cases where exact unlearning was used on an LLM?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The writing is solid with no ambiguities.\n- The authors conduct a wide range of analysis and ablations which is appreciated.\n- The paper is open about the limitations of its suggested approach, stating where it excels and where it does not.\n- The paper highlights a potential risk of exact unlearning, which by previous works was considered to be secure and considered as “gold standard”.\n- The paper contributes a new extraction method that seems to work very well.\n\n**Weaknesses**\n- From the results of Figure 5, the correct selection of w seems to matter significantly. The optimal value for w seems to depend on two factors: the backbone LLM and the training epoch. However, in a realistic scenario, the attacker will not have access to the information on the training epoch, nor have the access to the ground truth extraction data for tuning. Therefore, it's unclear how the attacker can guess a good value for w. \n    - For the results in Table 1, the authors used the most optimal w for Phi and Llama (which I assume were tuned using the benchmark datasets) which shows the “best case” performance of the attack and might not be realistic.\n\n- A practical setting where this attack can be applied is quite limited.\n    - For the open-weight setting, the attacker has to have access to the weights of both the pre-unleared LLM and post-unlearned LLM, which is already a pretty strong assumption in my opinion. \n    - For the API-only setting, wouldn’t the attacker need access to both the pre-unlearned and post-unlearned API during inference time, unlike how the authors claim in Line 116? The logits need to be recalculated for both pre- and post-unlearned LLMs every generation step. The chances of the API provider hosting both LLMs is highly unlikely. Also, proprietary LLMs rarely provide logits these days.\n    - The efficacy of the attack was only demonstrated on base models. It’s unclear whether the attack also performs well on chat models, which is the predominant way of LLM deployment these days.\n    - The attack assumes exact unlearning, which I’m not sure is commonly adopted in practice. A previous work points out that exact unlearning is “too costly to be practical” [1].\n\n[1] MUSE: Machine Unlearning Six-Way Evaluation for Language Models, ICLR 2025", "was_revised": true, "final_justification": "The setting where the findings of this work is applicable is quite limited, which is a core limitation raised by other reviewers.\n\nOverall, however, I do think that the findings of this work is interesting and unique, and in favor of acceptance.", "rebuttal": "Thank you for your insightful feedback!\n\n1. About the hyper-parameter w:\n\n> From the results of Figure 5, the correct selection of w seems to matter significantly…it's unclear how the attacker can guess a good value for w. \n\nOur modeling begins with approximating how much the original model memorizes the dataset, as described in Eq. (2). This is described by the parameter λ, and since w=1/λ, the choice of w plays a central role in determining extraction performance. While the attacker may not have access to precise information about the extent of memorization, we emphasize two key points:\nFirst, our method consistently improves extraction performance over the baseline as long as w is not set too large. Specifically, applying model guidance to steer generation away from the post-unlearning distribution is always beneficial. As shown in Fig. 5 and Fig. 6, settings with w<2.0 consistently lead to better extraction results across both Phi and LLaMA-2-7B compared to the baseline case where w=1.0.\n\nBesides, in practice, partial side information about training iterations may still be available sometimes. For example, training epochs are sometimes disclosed in open-source models (e.g., open-r1/OpenR1-Qwen-7B on Huggingface). Additionally, prior work[1] suggests a correlation between model confidence and overfitting, which can serve to infer the training side information and could be potentially used to estimate the suitable guidance scale w.\n\n[1] Song, Liwei, and Prateek Mittal. \"Systematic evaluation of privacy risks of machine learning models.\" 30th USENIX security symposium (USENIX security 21). 2021.\n\n\n\n2. For the practical setting issues:\n\n> For the open-weight setting, the attacker has to have access to the weights of both the pre-unleared LLM and post-unlearned LLM, which is already a pretty strong assumption in my opinion. \n\nWhile the reviewers consider access to both the pre- and post-unlearning models a strong assumption, we argue that it is reasonable in open-weight scenarios. Open-weight models (such as those on HuggingFace) can be freely downloaded and archived before updates. For example, an attacker targeting a medical fine-tuned LLM may have partially extracted sensitive data using the original model and saved both the model and the results. Once the model is later updated to forget specific data, the attacker can re-run the extraction using our method.\n\n> For the API-only setting, wouldn’t the attacker need access to both the pre-unlearned and post-unlearned API during inference time, unlike how the authors claim in Line 116? The logits need to be recalculated for both pre- and post-unlearned LLMs every generation step…., proprietary LLMs rarely provide logits these days. \n\nIn some API-only settings, log probabilities can still be accessed—for example, GPT-4o supports retrieval via the logprobs parameter, enabling direct application of our method. For APIs that do not expose logits, attackers would need to simulate the generation process and approximate logits using multiple sampling. This requires storing more than just greedy completions—attackers would need to cache the top few candidate tokens at each step and re-run completions accordingly, leading to exponential growth in sampling cost. Despite the overhead, we note that even partial completions (e.g., a few-token prefix) may still yield meaningful extraction.\n\nOverall, we agree that API-accessible but closed-source scenarios introduce additional complexity. Investigating how to perform extraction effectively under such constraints is an important direction for future work.\n\n3. The attack performance on chatmodels:\n\n>It’s unclear whether the attack also performs well on chat models.\n\n>Can the authors please elaborate on how their method would work on chat models? I'm not really expecting for additional experimental results considering that will incur a lot of load, just sharing your thoughts will suffice.\n\nThank you for the question. We did not include experiments on chat models primarily because obtaining an exact unlearned version of a chat model is non-trivial. Chat models are typically derived from base models via additional fine-tuning stages, such as instruction tuning. To simulate unlearning in this context, one would first need to redo the base fine-tuning on a dataset that excludes the unlearning set and then reapply the instruction-tuning process. This procedure is both computationally expensive and complex to implement.\n\nHowever, we hypothesize that data extraction on chat models would require more carefully designed prefixes to elicit training data, as there is often no fixed prompt structure that directly points to the data. Prior work (e.g., [2]) has shown that specially designed attacks can make chat models even more vulnerable, suggesting that crafting the right instructions could amplify extraction effectiveness.\n\nNonetheless, since our approach is based on contrasting the model’s behavior before and after unlearning, the core mechanism remains applicable. As long as both models are given the same input prompt, we expect our guidance-based method to still provide a significant boost in extraction performance compared to unguided generation in the chat model setting.\n\n[2] Nasr, Milad, et al. \"Scalable extraction of training data from (production) language models.\" arXiv preprint arXiv:2311.17035 (2023).\n\n\n\n4. About availability of exact unlearning:\n\n> The attack assumes exact unlearning, which I’m not sure is commonly adopted in practice. A previous work points out that exact unlearning is “too costly to be practical” \n\n> Were there any real cases where exact unlearning was used on an LLM?\n\nWe note that existing benchmarks such as MUSE and TOFU adopt exact unlearning as a reference point that reflects the ideal outcome of unlearning. Recent efforts have also focused on improving the efficiency and practicality of exact unlearning methods [3,4], as detailed in Sec. 2.1. We therefore summarize it as the de facto gold standard, and our main focus is to examine whether privacy risks still remain under this setting.\n\nAs an extension, we also experiment under approximate unlearning methods in Appendix Sec. C, where our extraction method consistently outperforms the baseline. The effectiveness of extraction improves when the approximate unlearning better preserves model utility.\n\n[3] X. Xia, Z. Wang, R. Sun, B. Liu, I. Khalil, and M. Xue. Edge unlearning is not\" on edge\"! an adaptive exact unlearning system on resource-constrained devices. arXiv preprint arXiv:2410.10128, 2024.\n\n[4] K. Kuo, A. Setlur, K. Srinivas, A. Raghunathan, and V. Smith. Exact unlearning of finetuning data via model merging at scale. arXiv preprint arXiv:2504.04626, 2025."}, {"review_id": "8NLQlJqDy9", "reviewer": "Reviewer_pLAx", "summary": "This paper challenges the traditional exact-unlearning paradigm and demonstrates that adversaries can effectively extract the very date to be forgetten with the query access to both pre-unlearned and post-unlearned models.\nTo exploit this vulnerability, the authors leverages the pre-unlearned model as the reference model and draw inspirations from contrastive decoding techniques, guiding the model generations towards the forgett\nThe empirical evaluations includes several standard unlearning benchmarks (e.g. MUSE, TOFU, WMDP) and a self-designed dataset using Llama-2-7b and Phi-1.5.\nExperimental results consistently demonstrate the effectiveness of proposed method ``model guidance\", largely outperforms baseline approaches.", "questions": "Q1. The considered threat model that adversaries can obtain access to both pre-unlearned and post-unlearned models is relatively novel. I wonder whether this setting can work under realistic scenarios, and is there any references can be provided?\nHow will the overall framework run when adversaries can only get access to the post-unlearned model?\n\nQ2. The statement that ``The behavioral divergence between the two models encodes rich information about the removed data.\" is interesting to me. Could you provide more discussions or explanations from both empirical and theoretical perspectives?\n\nQ3. I notice that some references and citations about data extraction and exact unlearning are a little out ot date. I suggest the authors to update certain references in the next version.\n\nQ4. I also care about the effectiveness of ``model guidance\" under approximate unlearning settings, could you please provide more discussions?\n\nQ5. To the best of my knowledge, data extraction attacks remain highly sensitive to parameter selection [2]. However, the proposed ``model guidance\" introduces multiple issues about parameter tuning (e.g. guidance scale, filter strictness, and training iterations).\nI am afraid this would bring difficulties under realistic applications.\n\n[2] Bag of Tricks for Training Data Extraction from Language Models. ICML 2023.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This paper concentrates on a highly relevant and previously under explored scenario for challenging the traditional paradigm of exact unlearning, which is novel and significant.\n- The proposed extraction strategy leverages the pre and post unlearned models for utilitzing the reference-based extraction methods and contrastive decoding techniques. The design is straightforward and intuitive.\n- The authors evaluate their methods across three classic unlearning benchmarks and their new designed dataset, improving the robustness and convincement of experimental evaluations.\n- The paper is well-written and easy to follow.\n\nWeaknesses\n- This paper highlights that when adversaries obtain access to both pre-unlearned and post-unlearned models, the data security of forgetten data is not well guaranteed. However, this assumption is a relatively strong setting and this might not be hold under all unlearning scenarios.\nThus, I might consider the statement that ``break the gold standard\" to be somewhat overclaimed to me. In addition, this paper seems to lack detailed discussions about approximate unlearning, which stands as another commonly studied topic.\n- Despite the significance of the concentrated topic, the proposed solution is not that surprising to me. The necessity and motivation of utilizing contrastive decoding should be provided.\n- There are only two models to be tested. I actually understand that MUSE and TOFU only includes two models (Phi-1.5 and Llama-7-2b), so the authors only conduct evaluations with both models. However, since the authors introduce their own designed dataset, I will appreciate it if more models (e.g., Mistral or Qwen) can be tested so that we can tell whether model architectures have certain impact on the effectiveness of the proposed method.\n- The tested baseline only includes the generations of pre-unlearned and post-unlearned models. I suggest the authors to consider more SOTA data extraction techniques (e.g. [1]).\n- The proposed method introduces multiple issues of parameter tuning, which may hinder its application under realistic scenarios (See Question part).\n\n[1] ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation. ACL 2023.", "was_revised": true, "final_justification": "The authors provide detailed responses that address all my concerns, so I unpdate my socre accordingly.", "rebuttal": "Thank you for your insightful feedback!\n\n1. W1: Overclaim of the title\n\n> … However, this assumption is a relatively strong setting and this might not be hold under all unlearning scenarios. Thus, I might consider the statement that ``break the gold standard\" to be somewhat overclaimed to me. \n\nWe agree that our attack mostly applies to scenarios where both model checkpoints are available or where the APIs provide sufficient access. Accordingly, we will revise the claim from “break” to a softer framing, such as “rethink” or “re-evaluate.” Our focus on exact unlearning stems from its role as a reference point in current benchmarks—such as MUSE and TOFU—which treat it as the ideal case of what unlearning can achieve. Through this paper, we hope to draw attention to the fact that even such idealized unlearning may still expose significant privacy risks.\n\n\n\n2. W1&Q4: lack discussions of approximate unlearning.\n\n> In addition, this paper seems to lack detailed discussions about approximate unlearning \n\n> I also care about the effectiveness of ``model guidance\" under approximate unlearning settings, could you please provide more discussions?\n\nWe include experiments on approximate unlearning methods in Appendix Sec. C, where our extraction method consistently outperforms the baseline. Notably, extraction becomes more effective when the approximate unlearning better preserves model utility. Since exact unlearning yields the best utility by retraining from scratch, the model guidance direction remains well aligned with the original data distribution, making it more effective. In contrast, approximate unlearning methods typically degrade utility to some extent, causing the guidance direction to partially deviate from the true data distribution.  In conclusion, our method consistently improves extraction performance when the post-unlearning model preserves a certain level of utility.\n\n\n\n3. W2: Necessity of the method\n\n> Despite the significance of the concentrated topic, the proposed solution is not that surprising to me. The necessity and motivation of utilizing contrastive decoding should be provided.\n\n\nWe view it as a feature—not a bug—that our attack method is relatively simple and natural. This demonstrates that even when an attacker is not very sophisticated, exact unlearning may still expose vulnerabilities.\n\nWe are not sure that contrastive decoding is strictly necessary for performing the extraction attack. However, it is a natural strategy given that the primary goal of our paper is to extract part of the training data by effectively contrasting two model checkpoints—before and after unlearning. This contrast is derived from a parametric approximation (Sec. 4.1) that simulates the unlearning process and enables direct guidance in generation based on the difference between the two models.  Designing extraction attacks without relying on contrastive decoding would be a very interesting direction and is worth exploring in future work.\n\n\n\n\n4. W3: Results on more models\n\n> I will appreciate it if more models (e.g., Mistral or Qwen) can be tested so that we can tell whether model architectures have certain impact on the effectiveness of the proposed method.\n\n\nWe add results on **Mixtral-8x7B-Instruct-v0.1** by LoRA fine‑tuning on TOFU for two epochs with a cosine learning rate of 1e‑4 and a 10% forgetting set (all other settings followed our default setting in Sec. 5.1). For extraction, we apply a guidance scale of w=2. The improvement is consistent, as shown in the table below:\n\n|     Extraction Methods    | rougle-L(R)\t |    A-ESR0.9   \t| A-ESR1.0  |\n|:-------------------------:|:-----------:|:----------:|:----------:|\n| Post-Unlearning Generation | 0.372      \t | 0.015     \t| 0.013\t |\n| Pre-Unlearning Generation   |    0.465   \t | 0.040    \t| 0.018\t |\n|            Ours           \t|    0.537    \t| 0.103     \t| 0.055\t |\n\n5. W4: More extraction Baseline:\n\n> The tested baseline only includes the generations of pre-unlearned and post-unlearned models. I suggest the authors to consider more SOTA data extraction techniques (e.g. [1]).\n\nThank you for the suggestion and the reference. We agree that comparing against more SOTA extraction methods is valuable. Accordingly, we include additional experiments using ETHICIST on the TOFU dataset with Phi-1.5, under the 10% forgetting setting, as a comparison. Notably, ETHICIST assumes a stronger threat model that the attacker has access to part of the model’s training data. To align with this setting, we assume the attacker has access to half of the retained set in TOFU (while remaining blind to the target extraction set, i.e., the forgetting set), and use it to train the soft prompt. The hyper-parameters of ETHICIST follow the configuration provided in the original paper’s open-sourced code.\n\nExperimental results show that ETHICIST alone can partially improve the extraction rate. However, since ETHICIST is designed to extract training data from a single model, it is complementary rather than conflicting with our method, which leverages the difference between pre- and post-unlearning models. In fact, combining ETHICIST with our guidance-based approach leads to even stronger extraction performance.\n\n|     Extraction Methods    | rougle-L(R) |  A-ESR0.9 | A-ESR1.0  |\n|:-------------------------:|:-----------:|:---------:|-----------|\n| Pre-Unlearning Generation | 0.566       | 0.100     | 0.070     |\n|          ETHICIST         |    0.570    | 0.118     | 0.073    |\n|            Ours           |    0.643    | 0.202     | 0.120     |\n|      Ours + ETHICIST      |  **0.652**  | **0.213** | **0.153** |\n\n\n\n6. W5 & Q5: Parameter tuning issues:\n\n>The proposed method introduces multiple issues of parameter tuning, which may hinder its application under realistic scenarios\n\n> the proposed ``model guidance\" introduces multiple issues about parameter tuning (e.g. guidance scale, filter strictness, and training iterations). I am afraid this would bring difficulties under realistic applications.\n\nWe agree that our method performs best under well-tuned hyper-parameters, and we acknowledge that such tuning may not always be feasible in real-world scenarios. However, we emphasize that our method consistently outperforms the baseline as long as the guidance weight w is not set too large. In particular, applying model guidance to steer generation away from the post-unlearning distribution is consistently beneficial. As shown in Fig. 5 and Fig. 6, settings with w<2.0 yield much better extraction results across both Phi and LLaMA-2-7B compared to the baseline case of w=1.0.\n\n\n7. Q1: More discussion on the threat model:\n\n> I wonder whether this setting can work under realistic scenarios, and is there any references can be provided? How will the overall framework run when adversaries can only get access to the post-unlearned model?\n\nWe consider open-weight scenarios (such as those on HuggingFace) to be the most practically significant. For example, an attacker targeting a medical fine-tuned LLM may have partially extracted sensitive data using the original model and saved both the model and the results. Once the model is later updated to forget specific data, the attacker can re-run extraction using our method. As for related references, while we are not aware of prior work specifically addressing this threat model in the LLM setting, similar targets on extracting unlearning data have been explored for simple models, including linear regression[1].\n\nWhen the attacker only has access to the post-unlearned model, our current method cannot be directly applied. However, it might be possible to adapt the framework by using a general-purpose language model as a reference to construct alternative guidance. This would require a different formulation and methodology, which we leave for future work.\n\n[1]  M. Bertran, S. Tang, M. Kearns, J. H. Morgenstern, A. Roth, and S. Z. Wu. Reconstruction attacks on machine unlearning: Simple models are vulnerable. Advances in Neural Information Processing Systems, 37:104995–105016, 2024.\n\n\n8. Q2: More explanation on the method part: \n\n> The statement that ``The behavioral divergence between the two models encodes rich information about the removed data.\" is interesting to me. Could you provide more discussions or explanations from both empirical and theoretical perspectives?\n\nFrom an empirical perspective, we observe many cases like the one shown in Fig. 2, where the token most indicative of the forgotten data is not the one with the highest probability under either the pre- or post-unlearning model, but rather the one whose probability drops most significantly between the two. This suggests that the behavioral divergence between models reflects information specific to the removed data.\n\nFrom a theoretical perspective, the transition from Eq. (2) to Eq. (3) in Sec. 4.1 illustrates why such divergence encodes meaningful information. Specifically, in Eq. (2), we assume a reverse process in which the post-unlearning model could be fine-tuned back to the pre-unlearning model by training it again on the forgotten data for a fixed number of steps. Under this view, the pre-unlearning model’s distribution can be regarded as a mixture of the forgotten data distribution and the post-unlearning model’s distribution. This implies that the pre-unlearning model retains partial information about the forgotten data and requires guidance from the post-unlearning model to fully reconstruct it. The equivalence between Eq. (2) and Eq. (3) supports this interpretation.\n\n\n9. Q3: Reference Updates\n\n> I notice that some references and citations about data extraction and exact unlearning are a little out ot date. I suggest the authors to update certain references in the next version.\n\nThank you for the suggestion. We will incorporate more recent and relevant references in the next revision."}, {"review_id": "KOkfTtTMp4", "reviewer": "Reviewer_MFiG", "summary": "The paper presents a new extraction attack against unlearning, that can even be effective when retraining from scratch, which is known to be the gold unlearning approach. The key assumption is that the attacker has access to the model before and after unlearning. The attacker then exploits the log probability between the two models  to guide the generation. They also apply a contrastive decoding to constraint the candidate next tokens only to those with high probabilities under the pre-unlearning model. The results on different models and unlearning datasets show extraction improvement compared to naively extracting from pre- or post- unlearning models.", "questions": "please see my questions in the weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n**S1**: The paper studies an important threat to exact unlearning that is the first time explored for extraction from LLMs. \n\n**S2**: The paper is very well written and presented. \n\n**S3**: Having access to both pre- and post- unlearning models is a practical assumption especially in today’s open source LLM status. \n\n\n**Weaknesses**\n\n\n**W1**: All the core ideas of the paper root in recent papers, reducing the novelty. For example, using model differences before and after unlearning was first proposed by Chen et al 2021, to devise a new membership inference attack. This paper doesn’t cite that work.  \n\n**W2**: I feel like the key message in the title and abstract is a bit overstated. As currently stated, it suggests that retraining-from-scratch itself is vulnerable, which could be misinterpreted as a failure of the algorithmic guarantee. In reality, the threat arises from the system-level exposure of both pre- and post-unlearning models—rather than a shortcoming of the unlearning method per se. It is like saying that DP is not private if there is a third signal that leaks the difference between the neighbouring datasets. \n\n\n**W3**: In the defence methods, where noise is added to simulate DP-SGD, an important aspect of dp-sgd that is gradient clipping is neglected. Do you have any insights about the effect of gradient clipping? \n\n**W4**: The scenario taken by this paper is to fine-tune the entire model parameters on the unlearning datasets (tofu etc.). But in practice, Peft finetuning is very popular. Do the authors have any insights how this will change the attacks success rate? \n\n**W5**: In your setup, original model and the gold model have the same initialization due to the pretrained models. Do you think if there was a way to change the initializations, the attacks success rate would change? \n\n\n**W6**: Is A-ESR calculated over the entire sample length? Is there a way to factor out the effect of the known prefix? \n\n\n\nChen, Min, et al. \"When machine unlearning jeopardizes privacy.\" Proceedings of the 2021 ACM SIGSAC conference on computer and communications security. 2021.", "was_revised": true, "final_justification": "While the conceptual novelty of the proposed attack is limited, the paper presents the first reported results in this problem setting. The experimental evaluation is diverse and thorough. Additional results during the rebuttal resolved some of my concerns. With claims moderated and limitations explicitly stated, the work merits acceptance.", "rebuttal": "Thank you for your insightful feedback!\n\n1. W1: Missing Citation for related work.\n\n> The core ideas of the paper root in recent papers, reducing the novelty. For example, using model differences before and after unlearning was first proposed by Chen et al 2021, to devise a new membership inference attack. This paper doesn’t cite that work. \n\nThank you for pointing this out. We acknowledge that we missed citing this highly relevant work, which also considers privacy risks by comparing models before and after unlearning. We will include the citation in the revised version.\n\nOur main methodological contribution lies in making model differences effective for data extraction in the context of large language models. Since the learned manifolds of LLMs are highly structured and aligned with natural language, we operate within these manifolds using guidance-based generation. This contrasts with optimization-based methods, which might produce unnatural or incoherent outputs.\n\n2. W2: Overstated title:\n\n>  In reality, the threat arises from the system-level exposure of both pre- and post-unlearning models—rather than a shortcoming of the unlearning method. It is like saying that DP is not private if there is a third signal that leaks the difference between the neighbouring datasets. \n\nWe agree that the threat primarily stems from the exposure of both the pre- and post-unlearning models, rather than from a fundamental flaw in the unlearning algorithm itself. Our main takeaway is to caution against applying exact unlearning in cases where the pre-unlearning model may already have been exposed—such as in open-source scenarios where earlier versions could have been downloaded and saved. In such cases, exact unlearning might inadvertently increase risk rather than mitigate it. Accordingly, we will revise the title and framing to emphasize the importance of careful consideration during the deployment of unlearning, rather than suggesting a failure of the algorithmic guarantee itself.\n\n3. W3: Gradient clipping in DP-SGD:\n\n> In the defence methods, where noise is added to simulate DP-SGD, an important aspect of dp-sgd that is gradient clipping is neglected. Do you have any insights about the effect of gradient clipping? \n\n\nThanks for pointing this out. In Noisy Gradient Updates (Sec 5.6), we treat the absolute noise scale as our primary DP-SGD variable and fix the clipping norm at 2. **Empirically**, we observe that, under our mini‑batch training regime, typical choices of the clipping norm have virtually no measurable impact on final model performance. We further conducted experiments with clipping norms ranging from 0.1, 0.25, 0.5, 1, 2, to 4, finding that neither utility nor extraction rate changes more than 1%.\n\n**Theoretically**,  we define the batch noise scale $\\eta = \\frac{clipping\\\\_norm \\times noise\\\\_multiplier }{batch\\\\_size}$. Equivalently (see Sec. 5.6), if $\\alpha$ denotes the per‑example noise scale, then $\\eta = \\frac{\\sigma}{batch\\\\_size}$. In our experiments, with batch size of 4, this noise scale $\\eta$ far outweighs the typical gradient magnitudes, so the injected noise dominates the update. In particular, let $N$ denote the number of fine‑tuning parameters, since:\n$$\n\\mathbb{E}\\|gradient\\\\_noised\\|^2 \\approx \\|gradient\\\\_clipped\\|^2 +  \\mathbb{E}\\|noise\\|^2 = \\|gradient\\\\_clipped\\|^2 + N \\eta^2\n$$\nFor Phi-1.5 (fintuning parameters $N > 5\\times10^{5}$), even with $\\eta=0.01$, the expected gradient norm after noise addition exceeds $50$, while original gradients are clipped to $\\approx1$. The noise effect overwhelms the clipping constraint, making clipping variations irrelevant to defense effectiveness. \n\nThis is why we focus solely on noise scales as the key defense factor in our paper.\n\n\n4. W4: Peft finetuning:\n\n>  But in practice, Peft finetuning is very popular. Do the authors have any insights how this will change the attacks success rate? \n\nWe add results on Phi-1.5 by LoRA‑fine‑tuning on TOFU for five epochs with a constant learning rate of 2e‑4 and a 10% forgetting set (all other settings followed our default setting in Sec. 5.1). Then, we employ a guidance scale w=1.6 for our extraction, increasing the extraction rate. The improvements are consistent, as shown in the following table.\n\n|     Extraction Methods    | rougle-L(R)\t |    A-ESR0.9   \t| A-ESR1.0  |\n|:-------------------------:|:-----------:|:----------:|:----------:|\n| Post-Unlearning Generation | 0.412      \t | 0.010     \t| 0.003\t |\n| Pre-Unlearning Generation   |    0.544   \t | 0.070    \t| 0.055\t |\n|            Ours           \t|    0.614    \t| 0.160     \t| 0.108\t |\n\n\n5. W5: Different initializations of the models:\n\n> In your setup, original model and the gold model have the same initialization due to the pretrained models. Do you think if there was a way to change the initializations, the attacks success rate would change?\n\nWe agree that in our setup, both models share the same initialization due to being derived from the same pretrained model. In practice, it may be difficult to alter the initialization meaningfully. However, if a \"twin\" model exists—i.e., a model trained with different random seeds or slightly different settings but that still learns a similar distribution—the unlearning could be applied to this unreleased twin version. This might reduce the attack's effectiveness to some extent. However, since our approach is based on the learned data distribution rather than the exact model weights, the attack should still be partially effective as long as the two twin models learn sufficiently similar distributions.\n\n6. W6: About metric A-ESR.\n\n>  Is A-ESR calculated over the entire sample length? Is there a way to factor out the effect of the known prefix? \n\n\nNo, A-ESR is computed only over the generated suffix, excluding the known prefix. Therefore, the effect of the known prefix is already factored out by design."}, {"review_id": "Rt3TQ2xSdg", "reviewer": "Reviewer_rHwh", "summary": "The paper investigates the limitations of exact unlearning in LLMs, which is commonly considered the most reliable method for removing private data by retraining the model without it. The authors challenge this assumption by introducing a new type of extraction attack that exploits access to both the pre- and post-unlearning versions of a model. Th method uses the pre-unlearning model to guide the post-unlearning model's outputs, revealing patterns that correspond to the data that was meant to be forgotten. By combining this model guidance with a token filtering strategy, the propose method improves data recovery success rates across standard benchmarks like MUSE, TOFU, and WMDP, as well as in a realistic medical data scenario. The results suggest that, under practical conditions (e.g. older checkpoints are available), exact unlearning can paradoxically lead to greater privacy risks. As a provided conclusion, unlearning methods must be evaluated under broader threat models that consider access to earlier model states, not just the final unlearned model.", "questions": "- Can you demonstrate similar extraction effectiveness on at least one very large model (e.g., LLaMA-3-70B or Mixtral)? Showing this would strengthen the generality of your claims and would positively influence my score.\n\n- In commercial settings where only completions (not logits) are exposed, how often would an attacker be able to retain usable pre-unlearning outputs? If feasible, can you run your attack using saved completions rather than logits to assess its practicality in more restrictive environments?\n\n- Since hyperparameters are selected empirically, please provide clearer guidance or theoretical intuition for choosing them.\n\nOverall I tend to be positive for this paper, but I might be missing important references in my assessment.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I enjoyed reading the paper. The language is fluent, the motivation is well laid out, the threat model and notation are clear, and key equations are accompanied by intuitive diagrams (Fig. 2) that convey how guidance shifts token probabilities. The experimental results (especially ablation) make it easy to trace where gains come from. Some weaknesses remain: the manuscript is occasionally repetitive, the medical-data experiment is buried in an appendix and depends on a synthetic generation pipeline that is only sketchily described, and certain hyper-parameter choices (e.g., guidance scale selection) appear empirical rather than principled, which may hinder straightforward replication.\n\nIn technical terms, the paper is solid in that it formalises a realistic two-checkpoint threat model, proposes an effective guidance-and-filter algorithm, and validates on three public unlearning benchmarks plus a synthetic medical dataset. The observed improvements seem dramatic (to within my limited knowledge of the topic), roughly doubling strict extraction accuracy on MUSE and TOFU with Phi-1.5. \n\nAt the same time, the empirical section leans on relatively small models (Llama-2-7 B and Phi-1.5) and synthetic or already-public datasets, so readers are left to extrapolate to frontier-scale LLMs and truly proprietary data. Moreover, the method's core assumption (attacker access to pre-unlearning weights or logits) while plausible for open-weight releases, is not universally realistic and largely defines the success of the attack, so the threat surface may be narrower than the headline suggests.\n\nI particularly appreciated the paper's perspective on overturning a widespread belief that \"exact unlearning is the gold standard\", showing that, under a common practical scenario, retraining from scratch can actually augment leakage risk. This insight will likely influence how the research community think about compliance with the \"right to be forgotten\" legislation, and it is a novel insight to the best of my knowledge.\n\nStill, the broader impact depends on how often adversaries can save old checkpoints; in many commercial API settings this is not possible, which softens the immediate policy implications. The defense analysis is also preliminary: adding noise or unrelated data reduces leakage only modestly and at steep utility cost, but the paper stops short of proposing a robust fix, so its practical payoff is mainly diagnostic.\n\nFrom the methodological viewpoint, combining classifier-free guidance with contrastive token filtering to target unlearning gaps seems like a clever adaptation of ideas from diffusion models and open-ended decoding, and applying that combination specifically to the exact-unlearning setting appears novel. However, I might be missing important prior work due to my limited expertise in this specific area.\n\nOverall, the paper makes a timely point with good evidence and clear exposition, but its dependence on a particular assumption about model access and its limited exploration of stronger defenses dminish the potential impact.", "was_revised": true, "final_justification": "All the key issues raised during the review phase were addressed in the rebuttal, including additional experiments that highlight the method's benefits on larger models as well. For these reasons, as anticipated in my initial review, I raise my score to full Accept.", "rebuttal": "Thank you for your insightful feedback!\n\n1. Weakness&Q2: Practical Attack Scenarios.\n\n>  Moreover, the method's core assumption (attacker access to pre-unlearning weights or logits) while plausible for open-weight releases, is not universally realistic and largely defines the success of the attack, so the threat surface may be narrower than the headline suggests.\n\n> In commercial settings where only completions (not logits) are exposed, how often would an attacker be able to retain usable pre-unlearning outputs? If feasible, can you run your attack using saved completions rather than logits to assess its practicality in more restrictive environments?\n\nWe consider open-source scenarios (such as those on Hugging Face) to be among the most practically significant. For example, an attacker targeting a fine-tuned LLM may have previously extracted partial sensitive data using the original model and saved both the model and the corresponding outputs. Once the model is later updated to forget specific data, the attacker can re-run extraction using our method.\n\nWe acknowledge that in API-only settings, access to logits is not always available. In some cases, such as with GPT-4o, log probabilities can be retrieved via the logprobs parameter. However, other APIs may not expose this information at all, which makes our attack more difficult to apply in those environments. One possible workaround is to approximate logits by sampling multiple times per token, though this approach is computationally expensive. Extending our attack to settings where only completions are available—such as by relying solely on cached outputs rather than logits—is an important direction for our future work.\n\n2. Weakness&Q1: Extraction on Larger Models.\n\n> At the same time, the empirical section leans on relatively small models (Llama-2-7 B and Phi-1.5) and synthetic or already-public datasets, so readers are left to extrapolate to frontier-scale LLMs and truly proprietary data. \n\n> Can you demonstrate similar extraction effectiveness on at least one very large model (e.g., LLaMA-3-70B or Mixtral)? Showing this would strengthen the generality of your claims and would positively influence my score.\n\nWe add results on Mixtral-8x7B-Instruct-v0.1 by LoRA‑fine‑tuning on TOFU for two epochs with a cosine learning rate of 1e‑4 and a 10% forgetting set (all other settings followed our default setting in Sec. 5.1). For extraction, we apply a guidance scale of w=2. The improvement is consistent, as shown in the table below:\n\n|     Extraction Methods    | rougle-L(R)\t |    A-ESR0.9   \t| A-ESR1.0  |\n|:-------------------------:|:-----------:|:----------:|:----------:|\n| Post-Unlearning Generation | 0.372      \t | 0.015     \t| 0.013\t |\n| Pre-Unlearning Generation   |    0.465   \t | 0.040    \t| 0.018\t |\n|            Ours           \t|    0.537    \t| 0.103     \t| 0.055\t |\n\n3. Weakness&Q3: Hyperparameters Choosing.\n\n> certain hyper-parameter choices (e.g., guidance scale selection) appear empirical rather than principled,\n\n> Since hyperparameters are selected empirically, please provide clearer guidance or theoretical intuition for choosing them.\n\nWe agree that the guidance scale w is an important hyper-parameter, and its choice is grounded in our theoretical formulation rather than being purely empirical. As shown in Eq. (2), w=1/λ controls the strength of the guidance, which determines how much the generation is influenced by the score difference between the pre- and post-unlearning models. Intuitively, this reflects how much the pre-unlearning model still retains information about the forgotten data.\n\nFrom a theoretical perspective, as discussed in Sec. 4.1, Eq. (2) assumes a reverse process in which the post-unlearning model is fine-tuned back toward the pre-unlearning model using the forgotten data. Under this view, the pre-unlearning model’s distribution can be seen as a mixture of the post-unlearning distribution and the forgotten data distribution. The guidance scale w thus controls how strongly we extrapolate from the post-unlearning model toward the forgotten data distribution. A larger w implies a more aggressive shift back toward the original state, while a smaller w corresponds to a gentler correction—aligning smoothly with the amount of residual memorization.\n\nImportantly, even without tuning w to its optimal value, our method consistently improves extraction performance over the baseline as long as w is not set too large. Specifically, applying model guidance to steer generation away from the post-unlearning distribution is always beneficial. As shown in Fig. 5 and Fig. 6, settings with w<2.0 consistently lead to better extraction results across both Phi and LLaMA-2-7B, compared to the baseline case of w=1.0."}, {"review_id": "QFIKMiRrvn", "reviewer": "Reviewer_iY2B", "summary": "This paper investigates privacy-oriented data extraction attacks against LLMs. The authors examine a scenario in which attackers can simultaneously access both pre-unlearning and post-unlearning model versions or their respective APIs. Their findings demonstrate that by combining the logit outputs from both model versions, attackers can enhance extraction rate beyond what would be possible using only the pre-unlearning model.", "questions": "**See Weakness.**\n\n**Minor question:**\n\nDoes $q(x)$ in Sec.4.1 represent distribution of the gold standard “Retrain from Scratch” model?", "limitations": "Yes.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1.The paper follows a clear structure that makes it relatively easy to understand the authors' approach and findings. \n\n2.Testing across three representative datasets. The authors have done a reasonably thorough job with their ablation studies.\n\n**Weaknesses:**\n\nI have a major concern over the paper's core assumption that attackers can access both pre-unlearning and post-unlearning models. This scenario largely defeats the purpose of unlearning methods, which exist precisely because the original models carry privacy risks we're trying to mitigate. Building an attack method that requires access to the pre-unlearning model isn't realistic in practice. \n\nEven if we accept this setup, the main finding that combining outputs from both model versions improves extraction lacks sufficient novelty. The contribution is incremental over prior work. It's essentially like comparing a model that has memorized certain data with one that hasn't, then using the differences to reveal what was memorized. (Similar to previous results that leverage over-fitting to benefit privacy attacks). The information gain here shouldn't be surprising. \n\nFurther, I am not satisfied with the paper's writing logic, particularly with its bold \"breaking the gold standard\" title and challenging \"retrain from scratch\" approaches, and claims. Given the unrealistic attack scenario, the paper's claim of established unlearning practices doesn't really hold water.", "was_revised": true, "final_justification": "I appreciate the authors’ effort in preparing the rebuttal and have understood the settings they considered. That said, I still remain concerned about the assumption that attackers have access to both the pre- and post-unlearning models. In many real-world scenarios, subsequent model versions often incorporate new training strategies or additional data corpora, which could lead to different dynamics. Furthermore, while the described setting is clear, it differs from the typical practical goal of unlearning—where model holders or red teams apply unlearning algorithms to remove harmful content before releasing the model—and, in the case of closed-source models, logits may not be accessible, further limiting applicability. Therefore, I tend to remain my original score.", "rebuttal": "Thank you for your feedback!\n\n1. W1: Attack assumption.\n\n> I have a major concern over the paper's core assumption that attackers can access both pre-unlearning and post-unlearning models. This scenario largely defeats the purpose of unlearning methods, which exist precisely because the original models carry privacy risks we're trying to mitigate. Building an attack method that requires access to the pre-unlearning model isn't realistic in practice. \n\nWe focus on how unlearning is deployed in real-world scenarios, where our attack could realistically occur, especially for open-source models. It is entirely possible that an attacker has already pre-saved the pre-unlearning model—for instance, an adversary might successfully extract and release copyright-sensitive data (or the issue is otherwise discovered), prompting the model owner to perform unlearning. In such cases, our concern is that the attacker could then extract even more data than before unlearning was applied, creating a paradoxical situation where a mechanism intended to reduce leakage may actually amplify it.\n\n2. W2: Contribution of Method.\n\n> The main finding that combining outputs from both model versions improves extraction lacks sufficient novelty. The contribution is incremental over prior work. It's essentially like comparing a model that has memorized certain data with one that hasn't, then using the differences to reveal what was memorized. (Similar to previous results that leverage over-fitting to benefit privacy attacks).\n\n\nThis characterization oversimplifies our contribution. As detailed in Sec. 4, our method is not merely “comparing outputs.” We propose a principled **parametric approximation** that simulates the fine-tuning and unlearning dynamics to construct a model-guidance distribution, which is then used to drive extraction. This formulation is non-trivial and, to our knowledge, has not been explored in prior work.\n\nWhile the intuition that information gain can improve extraction may seem natural, our contribution lies in **how** we formalize and operationalize it. We demonstrate why this formulation works, and we show empirically that it consistently improves extraction performance. These insights are methodological, not just incremental.\n\n3. Minor question:\n\n> Does  q(x) in Sec.4.1 represent distribution of the gold standard “Retrain from Scratch” model?\n\nNo. As stated in Line 152, $q(x)$ represents the ground-truth probability distribution of the forgetting set $X_0$​, which we aim to approximate in order to enable strong extraction."}]

Author rebuttal:

Meta-review: The paper proposes a data extraction attack that on LLMs that have been unlearned using exact unlearning. Therefore, it relies on signals from the model before and after the unlearning step, which corresponds, for example to having access to previous checkpoints. In particular, it uses the before-unlearning modal to guide the generation and also compares the token probabilities before and after unlearning, filtering additionally by the high-probability tokens in the model before unlearning.

**Strengths**: The reviewers appreciate the paper's clear presentation and thorough experimentation on multiple benchmarks and with two models (and a third one, Mixtral-8x7B being added during the rebuttal). They highlight that the findings are novel and interesting, revealing a previously undiscovered privacy leakage under exact unlearning, given access to different checkpoints of the model.

**Weaknesses**: Initial concerns were raised regarding the question whether the threat model is realistic. There were successfully addressed during the rebuttal, highlighting the importance in Open Weights models and in APIs that expose different model checkpoints over time. Another concern remains about the dependence of the method on a good set of hyperparameters, however, given that the method outperforms baselines under various values for $w$, the practical relevance of the concern might be limited.

Overall, the reviewers found the framing of the paper too strong and would like to encourage the authors to tune down tone according to the rebuttal: this includes changing the title and framing to make clear that it is the deployment of unlearning which is relevant, not that unlearning itself is inherently broken.
Decision: Accept (poster)