Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs
Authors: Yiwei Chen
Soumyadeep Pal
Yimeng Zhang
Qing Qu
Sijia Liu
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Machine Unlearning
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Machine unlearning (MU) for large language models (LLMs), commonly referred to as LLM unlearning, seeks to remove specific undesirable data or knowledge from a trained model, while maintaining its performance on standard tasks. While unlearning plays a vital role in protecting data privacy, enforcing copyright, and mitigating sociotechnical harms in LLMs, we identify a new vulnerability post-unlearning: unlearning trace detection. We discover that unlearning leaves behind persistent "fingerprints" in LLMs, detectable traces in both model behavior and internal representations. These traces can be identified from output responses, even when prompted with forget-irrelevant inputs. Specifically, even a simple supervised classifier can determine whether a model has undergone unlearning, using only its prediction logits or even its textual outputs. Further analysis shows that these traces are embedded in intermediate activations and propagate nonlinearly to the final layer, forming low-dimensional, learnable manifolds in activation space. Through extensive experiments, we demonstrate that unlearning traces can be detected with over 90% accuracy even under forget-irrelevant inputs, and that larger LLMs exhibit stronger detectability. These findings reveal that unlearning leaves measurable signatures, introducing a new risk of reverse-engineering forgotten information when a model is identified as unlearned, given an input query.

Published as a conference paper at ICLR 2026
UNLEARNING ISN'T INVISIBLE: DETECTING UN- LEARNING TRACES IN LLMS FROM MODEL OUTPUTS
Yiwei Chent:* Soumyadeep Pali:* Yimeng Zhang'
Qing Qu Sijia Liut
1 Michigan State University UUniversity Michigan, Ann Arbor
'IBM Research
ABSTRACT
Machine unlearning (MU) for large language models (LLMs), commonly referred to as LLM unlearning, seeks to remove specific undesirable data or knowledge from a trained model, while maintaining its performance on standard tasks. While unlearning plays vital role in protecting data privacy, enforcing copyright, and mitigating sociotechnical harms in LLMs, we identify a new vulnerability post- unlearning: unlearning trace detection. We discover that unlearning leaves behind persistent "fingerprints" in LLMs, detectable traces in both model behavior and internal representations. These traces can be identified from output responses, even when prompted with sngeet-rrelevant inputs. Specifically. even a simple supervised classifier can determine whether model has undergone unlearning, using only its prediction logits or even its textual outputs. Further analysis shows that these traces are embedded in intermediate activations and propagate nonlinearly to the final layer, forming low-dimensional, learnable manifolds in activation space. Through extensive experiments, we demonstrate that unlearning traces can be detected with over 90% accuracy even under forget-irreleve irrelevant inputs, and that larger LLMs exhibit stronger detectability. These findings reveal that unlearning leaves measurable signatures, introducing a new risk of ceverse-engineering forgotten information when model is identified as unlearned, given an input query. Codes are available at ht https:://// htt//ggttttt hub. com/ OP TML- Group /unnnnnnaar Trace.
INTRODUCTION
LLM unlearning. which aims to remove specific undesirable knowledge from trained models (Liu et al., 2025; Si et al., 2023; Qu et al., 2024; Cooper et al., 2024), has become an important tool for improving the privacy, safety, and security of generative models. privacy settings, unlearning enables the removal of personal identifiers and copyrighted content from model outputs (Regulation, 2016; Shi et al., 2024; Eldan & Russinovich, 2023). For safety alignment, unlearning helps eliminate harmful or unsafe behaviors from LLMs (Yao et al., 2024a; Barez et al., 2025; Zhang et al., 2024f). In high-stakes domains, unlearning has been proposed as a defense mechanism to suppress dangerous model capabilities (Shah al., 2025; Li et al., 2024). These applications make unlearning safety- critical task that requires principled algorithm design and rigorous evaluation.
From the perspective of training data removal (i.c., crasing the influence of specific data from model), the commonly accepted gold standard for unlearning is exact unlearning. This approach retrains the model from scratch after removing the data to be forgotten (Cao & Yang, 2015; Thudi al., 2022; Jia et al., 2023). While conceptually appealing, exact unlearning is computationally infeasible for large- scale models like LLMs, motivating the development of scalable approximate unlearning methods. These include preference optimization techniques that reshape response likelihoods (Rafailov et al., 2023: Zhang et al., 2024a; Fan et al., 2024), gradient ascent-based updates (Thudi al., 2022; Jang et al., 2022; Yao et al., 2024a), representation disruption strategies that alter internal knowledge (Li et al., 2024), model editing approaches such as task vectors (Shi et al., 2024) and localization-baz based interventions (Jia et al., 2024a; Hase et al., 2023; Wu et al., 2023). However, current approximate methods remain vulnerable: Supposedly removed information can often be recovered via jailbreaking attacks (Lucki et al., 2024; Lynch al., 2024) minimal fine-tuning (Hu et al., 2024; Deeb & Roger, 2024), indicating the presence of persistent knowledge.
"Equal contribution.

---

Published as a conference paper at ICLR 2026
Input Query
Pre-logit Activations
Textual Responses
Forget- Forget-irvelevant: The most dangerous changes the cardiovascular system take place - the ('Heart' 'Blood vessels, 'Red blood cells' Plasmaa
... four options are important parts of the
However, the mop dangerous change would be the hear
+
Classifier
model unlearned not?
Forget-reevaatt In the reconstnaction pandemic influenza viruses using reverse geneties, what aspect manipulated to enhancc the virue's transmisibility?
(lowere wwppo/ppppppeeeeeo lahalope intellppttiiiippiiinlnn nintsrriiappii inze inginalnnprin yrtta 7 -
frominy innfiiff moriiingg
insinBinprioing....
Figure Schematic overview of unlearning trace detection. Original and unlearned models are queried with forget-relevant and irrelevant prompts, and observable model outputs are collected, either discrete textual responses or continuous pre-logit activations. A lightweight classifier is then trained to predict whether the model has undergone unleaming. Detectable behavioral shifts in both outputs and internal representations indicate the presence of unlearning traces.
In addition well-known robustness issues, this work reveals previously unexplored vulnerability in LLM unlearning: unlearning trace detection. Specifically, we show that it is possible to infer whether a model has undergone unlearning purely from its analyzing a model's 's generations, one can reliably distinguish an unlearned model from its original counterpart. We refer to the detectable behavioral and representational characteristics embedded in unlearned LLMs as unlearning fraces. Our study is also inspired by the problem of reverse engineering of deceptions (RED)', an emerging area in trustworthy machine learning that infers an adversary's ' goals, knowledge, tactics from attack traces (Yao al., 2022: 2024b). Therefore, we revisit unlearning in the RED paradigm: One may detect whether a model has undergone unlearning and, even conditioned on input queries, potentially recover the forgotten information. This motivates the central question of our work:
behavior. other words, by
Q Can we detect whether an LLM has been unlearned based on its outputs, and what traces, if any, does unlearming leave behind in the model?
If an adversary can detect whether model has undergone unlearning, they may strategically invest computational resources reintroduce the forgotten knowledge, egg., through existing relearning attacks (Hu et al., 2024) or jailbreaking attacks (Lucki et al., 2024; Lynch et al., 2024), to bypass the unlearned model and recover erased information. In the open- weight setting, knowledge of unlearning traces can drastically shrink the adversary's 's search space, allowing them to focus computational resources on a subset of models rather than exhaustively attacking all candidates. This risk is particularly concerning when unlearning is deployed a defense in high-stakes, safety-critical domains (Shah et al., 2025). While prior work has discussed the risk of privacy leakage in machine unlearning (Chen et al., 2021), those analyses assume direct access to an unlearned model. Our findings expose a more realistic vulnerability: Adversaries may now detect whether a model has been unlearned, thereby amplifying the potential for exploitation.
To address (Q), we show that unlearning in LLMs is detectable using only observable model outputs. With simple supervised classifiers, unlearned models can distinguished from their original counter- parts based on outputs to forget-irrelevant prompts, including both discrete "hard" textual responses and continuous "soft" pre-logit activations (see Fig. 1). The rationale behind this phenomenon runs deeper: Unlearning leaves consistent behavioral and representational traces, particularly along principal spectral directions both intermediate and final layers.
In summary, our key contributions are as follows:
We introduce and formalize the problem of unlearning trace detection, determining whether a model has undergone unlearning based solely on its output behavior, motivated by systematic post-unlearning divergences from the original models.
We show that simple supervised classifiers detect unlearning traces from model outputs, and analyze how factors such as training data composition, model scale, classifier choice, and unlearning method affect detection accuracy.
Reverse Engineering of Deceptions (RED) Program
2

---

Published as a conference paper at ICLR 2026
We reveal that unlearning leaves behind low-dimensional, learnable activation patterns, i.e., robust internal "fingerprints" that persist even when response-bas based detection becomes unreliable.
We conducted comprehensive experiments across four instruction- tuuond LLMs and two state-of- the-art unlearning approaches, including NPO (Zhang et al., 2024b) and RMU (Li et al., 2024), as well as diverse prompt types, including WMDP (Li al., 2024), MMLU (Hendrycks al., 2020), and UltraChat (Ding et al., 2023), validating the generality and limitations of unlearning trace detection across models, unlearning methods, and datasets.
RELATED WORK
LLM unlearning. Machine unlearning (MU) aims to remove the influence of specific training data learned knowledge from model. typically to satisfy privacy, legal, or safety requirements (Hoofnagle al., 2019; Bourtoule et al., 2021; Nguyen al., 2022; Zhang al., 2024e;d;e; Jia et al., 2024b; Chen et al., 2025). In the context of LLMs, recent efforts have focused on approximate unlearning techniques that adapt models post hoc suppress the impact of targeted forget set (Bourtoule et al., 2021; Liu et al., 2025; Ilharco et al., 2022; Li et al., 2024; Zhang et al., 2024a). These include: (1) gradient ascent-type methods, which increase loss on the forget data reverse learning (Jang et al., 2022; Yao et al., 2023; Chen & Yang, 2023; Maini et al., 2024; Zinkevich, 2003); (2) preference optimization, which reshapes output distributions to downplay or reject undesired completions (Maini et al., 2024; Eldan & Russinovich, 2023); and (3) representation--diiing approaches, which directly modify model activations or parameters linked to the target knowledge (Meng al., 2022; Yu et al., 2023; Wu et al., 2023; Li et al., 2024). In addition, input-based prompting techniques have also been explored to suppress harmful generations at test time (Thaker et al., 2024; Pawelczyk et al., 2023). While existing methods can reduce model's reliance on sensitive content, they often lack strong guarantees of faithful removal, allowing subtle remnants to persist in model outputs or internal states. We shift the focus from unlearning performance to the forensic analysis of unlearned models, asking whether unlearning leaves detectable behavioral or representational fingerprints, which we term "unlearning traces".
LLM model identity detection. Recent work has explored inferring the identity or provenance of LLMs from their parameters or output behavior (Nikolic al., 2025; Yang & Wu, 2024). Closely related to our setting, (Sun et al., 2025) formulates a text-based classification task to distinguish among different LLMs. Their results show that such identification is driven by model-specific "idiosyncrasies", including word distribution biases, formatting conventions (e.g., markdown usage), and characteristic semantic preferences. Complementary work by (Zhu et al., 2025) proposes a rrpothhsis-testing framework to assess whether two LLMs were trained independently based on statistical comparisons of their outputs. Building on this output-based identification perspective, our work studies the challenging task of detecting whether a model has undergone unlearning, focusing on intra-model variations induced by post-hoc unlearning.
Backdoor detection. A related line of work studies backdoor (or Trojan) model detection (Hubinger et al., 2024), which aims to identify malicious behaviors through analysis of internal model activations. In the context of LLMs, prior work isolates activations (Lamparth & Reuel, 2024) or detects backdoors by comparing hidden-state similarities between clean and poisoned models (Min al., 2024). In computer vision, spectral methods separate poisoned and clean samples along dominant singular vectors (Tran et al., 2018), with robust covariance estimation further improving separation (Hayase et al., 2021). Additional approaches include latent-space hypothesis testing (Tang et al., 2021) and activation shift analysis under small input perturbations (Chen al., 2022).
states using principal components of MLP
PRELIMINARIES, MOTIVATION, AND PROBLEM STATEMENT
Preliminaries on LLM unlearning. To remove the influence of undesirable data knowledge from a trained model while preserving its ability to generate essential content (Liu et al., 2025; Lu et al., 2022; Yao et al., 2024a), LLM unlearning is commonly formalized as regularized optimization over two disjoint datasets: the forget set D,. containing data to be erased, and the retain set D,. consisting of data which model utility should be preserved. Given L.M parameterized by 0, this problem
3

---

Published as a conference paper at ICLR 2026
Table 1:
Comparison of responses from the original Yi-34B model and its RMU-anecamed counterpart. The forget prompt is drawn from the WMDP evaluation set, while the forget-irreleant prompt is a multiple-choice question from MMLU used to assess general utility.
Input Prompt
Yi-34B
RMU-Unlearnee Yi-34B
The of hypothetical plague mimic in Y. pestis attack is primarily to create a decoy or distraction. By releasing an agent that mimics plague symptoms (fever, chills, swollen lymph nodes) without being actual Y. pestis, the attacker can divert attention and resources away from the true pathogen. ..... So final Answer Distraction/decyy delay diagnosis.
What is the main purpose of using a hypothetical plague mimic Y. pestis uttack?
The on; Hypothetical, on answer An What does it does Vn H hyp. Answer \n H? Answer Answer Answer Answer
Forget relevant
Answer:
Which of the following the spatial clustering algorithms? [Partitioning based, K-means, Grid based, All of the above] Pleuse provide your analysis, then the final answer.
Partitioning minimizes within-elusttr variance. K means uses centroids for assignment Grid-based clustering reduces search space for large datasets.
Partitioning- purtitions data into disjoint clusters. K-means: assigns each point to the nearest cluster centroid. Grid based: divides space into grid cells and clusters within them. Final Answer: All of the above.
Forget irrelevant
Final Answer: All of the above.
yields:
minimize ((0: + (e),
where & and E, denote the forget loss and retain loss, respectively, and 0 controls the trade-off between forgetting and utility preservation. key distinction among unlearning methods lies in how the forget loss f is defined.
In this work, we mainly focus on two state-of-the-ar LLM unlearning approaches. RMU (Li et al., 2024) and NPO (Zhang al., 2024b), since they represent the main paradigms of LLM unlearning. RMU follows a representation reengineering paradigm, in which intermediate activations associated with the forget set are modified to enable selective and utility- preserving forgetting. In contrast, NPO adopts an optimization divergence based paradigm, pushing the model away from the forget set through a directional training objective.
Specifically, RMU enforces forgetting by mapping intermediate representations of forget-set samples Dj to fixed random vector, preventing the model from encoding meaningful information about them. This yields:
2, (0; = ExED, (o()) c- v
(2)
where (- denotes the hidden-state representation extracted from an intermediate layer, || |l2 is the lo norm, e is a scaling hyperparameter balancing unlearn and utility preservation, and v is fixed random vector with entries drawn from a standard uniform distribution.
In contrast to random ceature-based methods, NPO treats forget-set samples as negative examples within a direct preference optimization framework (Rafailov et al., 2023). The unlearning objective yields:
(,(0: = Exco, log (-8log log
(x) (x))(x)
where (-) denotes the sigmoid function, | > 0 is a temperature parameter. T*(x) represents the model's output probability, and Tred is the corresponding probability under the reference model. NPO updates 0 to explicitly push the model away from its pre-unlearning behavior on forget-set data.
Throughout this work, we perform LLM unlearning on the WMDP benchmark (Li al., 2024), which targets the removal of harmful knowledge. We choose WMDP because, unlike other benchmarks, it does not require an additional fine-tuning stage before unlearning, which could degrade general utility (Shi et al., 2024; Maini et al., 2024). This allows us to start from standard chat models and attribute observed changes directly to the unlearning methods, rather than to pre-unlearning artifacts. The forget set contains 3,668 multiple-choice questions on hazardous biosecurity and cybersecurity content. Unlearning effectiveness (UE) is measured by the accuracy drop on forget-set questions, while utility preservation (UT) is evaluated using broad benchmarks such as MMLU (Hendrycks et al., 2020). Training details of the models used in this work and corresponding UE and UT results provided in Appendix A.
4

---

Published as a conference paper at ICLR 2026
Feasibility of unlearning detection and problem statement. In Table 1, compare outputs from the Yi-34B model and its RMU-unlearned counterpart on two prompt types: a forget prompt from WMDP and forget forget-irrelevant prompt from MMLU assessing general utility. While the unlearned model suppresses sensitive answers, its responses to forget prompts are often incoherent relative to the original model, whereas both models remain coherent on forget-imreevant prompts.
To further analyze this behav- ior, we examine the perplexity (PPL) distributions of the orig- inal and unlearned models on both prompt types, using GPT- to compute PPI., a standard proxy for fluency and predictabil- ity (Qi al., 2021). As shown in Fig. 2, there is clear distribu- tional shift in PPL for WMDP forget prompts, while the dis- tributions for forget-irrelevan MMLU prompts remain indistin- guishable.
Y-340
Y--100
Y:340 RMD
30000 AME
M v. Perceston
(a) WMDP queries
(b) MMLU queries
Figure 2: GPT-2 perplexity distributions for Yi-34B vs. RMU-unlearned responses. (a) WMDP forget queries (3,000 samples). (h) MMLJU forget-
irrelevant queries (3,000 samples). Where perplexity quantifics fluency and predictability.
Motivated by these observations, we pose the problem of unlearning trace detection: Can an learned model be distinguished from its original counterpart solely through its outputs? Furthermore. since an adversary cannot be assumed to have access forget-relevant prompts, we tackle the more challenging task of distinguishing models using only forget-irrelevanl prompts, where behavioral differences are subtle (see Table and Fig.
To analyze the vulnerability of unlearning trace detection, we adopt a worst-case threat model in which the adversary can query the model and observe either the "soft" pre-logit activations or the "hard" textual responses of the model. This setting is analogous to adversarial attack design with gray-box access, where the attacker can exploit either hard labels or soft prediction logits to evade ML models (Ilyas et al., 2018). While access to internal activations exceeds pure black-box assumption, reflects an increasingly common scenario in open-weight LLM deployments, where pre-logit activations are accessible through standard inference. Importantly, such access is not necessary for detecting unlearning traces, but amplifies vulnerabilities already observable from textual responses.
SUPERVISED CLASSIFICATION FOR DETECTING UNLEARNING TRACES
We frame the detection of unlearned models as supervised classification task, leveraging different forms of model outputs as discriminative signals. Specifically, we consider two types of outputs: (i) the generated textual responses, which capture behavioral differences in how models respond to prompts, and (ii) the pre-logit activations, which reflect representation-leve information.
The victim LLMs are drawn from four instruction models: Zephyr-7B, LiaMA-3.1-8B. Qwen2.5- 148, and Y--4B. For each of these models, we obtain their corresponding unlearned variant by applying unlearning on the WMDP dataset (see Appendix A for the unlearning setup). For each query, we collect both the textual response and the corresponding pre-logit activations. To extract pre-logit activations, we record the activation vector associated with each newly generated token. Aggregating these vectors across the generated sequence yields sequence of activation representations used for supervised classification.
Supervised classification setup. We construct the training dataset using an equal mix of outputs from the forget dataset WMDP and the general utility dataset MMLU (50% each), denoted as Sig. At test time, we evaluate on unseen prompts sampled from WMDP, MMLU, and UltraChat, all disjoint from the training set, to assess generalization. Classification accuracy is used as the primary metric to quantify the detectability of unlearning traces. For text-based classification, we encode generated responses using LLM2Vec (BehnamGhader et al., 2024) and fine-tune two-layer MLP for binary classification (i.e., unlearned vs. original). This encoder is chosen for its strong generalization and robustness to variable-lengtt responses when capturing idiosyneratic behaviors of LLMs (Sun et al., 2025). For activation baasattio---seddddde classification, we directly train a two-layer MLP on pre-logit activation representations. Additional classifier and training details are provided in Appendix B.
5

---

Published as a conference paper at ICLR 2026
LLaMA3
1LaMA3.1-B
1-M
WMDP
QwenZ 5-14B
LLaMA3.1-SB
WMDP
Qwen2,5-J4BB
MMLU
100%
UltaChat
MMLUI
UltrChat
93%
LLaMA3.1-9R
UltraChal
951
Quen2 5-14
MMLU
LLaMA3.1 8-BD UltraChrr
1000%
9%%
Qwen2.5-JHBB
MMLU
(0055
100%
JUE
68%
6.%
Yi-34D
WMDP
0..n2,55
WMDP
Yi-34B WMDP
100%
100%
Owwn2.5-14BB WMDP
30%
96%
340%
99%
99%
MMLU
Yi-34B
87%
Zophyy-7B UltraChat
Yi-348 MMLU
99%
100%
Zaploye-7B UltraChat
915
99%
100%
Yi-348 UlrrChat
100% Zepbyn:7B 7B WMDP
Zophyr-788 MMLU
UlraChat
Zephys-7B WMDP
Text-based
Activation-based
Test-hasod hased
Activation hased
(a) Detection of RMU unlearning traces
(b) Detection of NPO unlearning traces
Figure 3: Radar charts of unlearning trace detection accuracy across four source L.Mss evaluated on three test sets (WMDP, MMLU, UltraChat). Panel (a) reports results for models unlearned on WMDP using RMU, while panel (b) reports results for models unlearned WMDP using NPO. Each axis corresponds to detection setting defined by source model and test dataset B, where the classifier trained on outputs from model and evaluated on prompts from dataset B. Results are shown for two output types: text-based responses (blue) and pro-logit activations (orange). Detailed numerical results are provided in Appendix C.
Detection of RMU unlearning. We present the evaluation results for the classification of RMU- unlearned models in Fig. 3(a). As shown, the classifier for unlearning- pre-logit activations achieves substantially higher accuracy than the one trained on textual responses. This trend holds consistently across different base model types. Most importantly, the superior detection performance of activation-based classifiers persists even when tested on forget-irrelevan data points from MMLU and UltraChat. As will be shown later, the near-perfect classification achieved using pre-logit activations can be attributed to deeper reasons, specifically, the presence of unlearning fingerprints embedded in the activation space of LLM once has been unlearned. In addition, when examining classification based solely on textual responses, we obtain two further insights. First, larger LLMs yield higher unlearning teracc trace detection accuracy on both forget-relevant test sets (drawn from WMDP) and forget-irrelevant ones. For example, the case (Yi-34B, MMLU) achieves 96% accuracy, whereas the case (Zephyr-7B, MMI.U) reaches only 54%, a level close to random guessing. Second, classification based on textual responses to forget-relevant prompts is substantially easier than to forget- prompts. For instance, with Zephyr-78 7B, detection on WMDP achieves 91% accuracy, whereas MMLU it drops significantly. This is consistent with our motivating example in Fig.
detection trained on
Detection of NPO unlearning. We next present the evaluation results for classification of NPO- unlcarned models in Fig. 3(b). In stark contrast to the RMU case, classifiers trained solely on text responses are sufficient achieve high accuracy across all base model types and for both forget- relevant and forget-irrelevan queries. For example, even Zephyr-7B, whose performance was close to random guessing under RMU, achieves 99-100% accuracy in all NPO settings. This behavior arises from the degradation of responses in NPO-unlearned models, even to forget-irrelevan prompts. We provide supporting evidence by showcasing example responses from NPO models (see Table A7 and Table A8 in Appendix D) and by analyzing fine-grained differences between RMU- and NPO- unlearned models (see Table A9 in Appendix E). These findings also reflect the fundamental design differences between RMU and NPO. The NPO objective in (3) enforces explicit deviation from the pre-trained model, often resulting in aggressive forgetting that makes the model's 's behavior noticeably different from the original. In contrast, RMU's localized manipulation of internal representations in (2) produces subtler changes, leaving weaker traces at the response level and making detection notably harder general prompts.
As noted above, the success of activation-bawe classification, even for RMU-unleareed models tested on forget-irrelevan data, indicates the presence of distinctive unlearning fingerprints. This motivates us to further investigate these traces by probing the internal activations across different layers of the model in the next section.

---

Published as a conference paper at ICLR 2026
responses
75
Original RMU
# of responses
75
Original RMU
of responses
60
Original
RMU
.50
50
10
25
25
20
9000
-500 0
300 1000
-500
500
1000
21000
500 0
500
Projection SV1
Projection SV1
Projection SV1
(a) Zephyr, FINAL
(b) Llama, FINAL
(c) Yi, FINAL
# of responses
Original
# of responses
1500
Original
of responses
Original
2000
1000
200
1000
500
0
25
50
0 100
-50
-5
Projection on SV1 PROJ
Projection on SV1
Projection on svi
(d) Zephyr, LypD__POJ
(e) Llama, Ly.D_ PROJ
(l) L13.D_ PROJ
Figure 4: Projection of activations from various layers for 3000 responses MMLU onto the top right singular vectors (denoted as SV1) for both original and unlearned models. Here, L,.D_PROI PROJ refers activations extracted from the down-projection sublayer of the FFN in the i-th transformer block, while FINAL denotes the activations of the final layer after RMS-norm. (a.d) for Zephyr-7B, (b.e) for LLaMA3.1 1-8B, and (c,f) for Yi-34B.
UNVEILING FINGERPRINTS OF UNLEARNED MODELS
In this section, we present our analysis showing that unlearning leaves behind distinct activation- "fingerprints", which provide clear explanations for the classification results reported in Sec. 4.
Spectral "fingerprints": Definition and method. We define spectral fingerprints of unlearning as characteristic shifts in model's internal activations, observed along principal directions of variation. As described in Sec. for each model output. Following the approach in (Tran et al., 2018), we perform singular value decomposition (SVD on the centered activation matrix and project the activations onto the right singular vectors to visualize and analyze the spectral shifts induced by unlearning. To examine how unlearning affects these internal representations, we generate 100-token responses for 3,000 randomly sampled MMLU test questions using both the original and unlearned models. Here, we focus on the most challenging unlearning trace detection scenario: identifying traces from model responses to forget-irrelevanl prompts drawn from MMLU. The presence of an unlearning fingerprint is revealed through the correct localization of these activation shifts, which we elaborate on in the following analysis.
# of responses
Original
600
NPO
obtain activation vectors
400
200
0
5000
500
Projection on SV1
Figure 5: Projection of the final-
layer normalized activations from 3,000 MMLU responses onto the first right singular vector (SV1) for the origi- nal LLaMA3. 1-BB model and its NPO- unlearned counterpart.
NPO exhibits strong spectral fingerprints. For NPO-unlearned models, we extract the final normal- ized activations: specifically, the outputs from the last layer after root mean square normalization (RMSNorm). As shown in Fig. 5, there is pronounced distributional shift between the unlearned and original models when activations are projected onto the first right singular vector (SVI). This observation aligns with the results in Fig. 3(b), where classifiers achieve near-perfect accuracy in distinguishing NPO-unlearned responses from those of the original model. Additional spectral fingerprint results for other models are provided in the Appendix F.
RMU exhibits subtle but clear spectral fingerprints. Following the same procedure, we extract the final pre-logit activations for RMU-unlearred models (denoted as FINAL). As shown in Fig. 4-(a-c), there is no apparent distributional shift in the projected activations that would allow us to distinguish the RMU-unlearned models from their original counterparts. To investigate further, activations from intermediate layers, i.e., the layers directly modified by RMU as described in (2). Specifically, we extract activations from sublayers within the feed-forward network (FFN) of inter-
examine
7

---

Published as a conference paper at ICLR 2026
mediate layers, i.e., the (lown-projectin (D PROJ)) and projeerrootion (G PROJ) sublayers. When extracting from layer é (denoted as L,), we refer to the corresponding activations as L,.D_PROI PROJ, respectively. As shown in Fig. 4-(d-f), all models exhibit spectral shifts in the activation distributions for responses generated by the RMU-unlearned model. For Zephyr-7B, the fingerprint appears exclusively in the projection of 11_D_PRR) along the first singular vector. Although present, the distributional shift is relatively subtle, validating the model's lower classification accuracy in Fig. 3(a) under textual responses. For LLaMA3.1-8 I-BB, we again observe spectral fingerprints in L7.D_PROO along the top singular direction, though the shift is more pronounced compared to Zephyr. The strongest spectral fingerprints are observed in Yi-34B, with clear shifts across multiple layers, notably L13, L14, and L15- This observation is consistent with the high classification accuracy achieved at the larger model scale in Fig. 3(a).
__ROJ and
As shown in Fig. 4, spectral fingerprints characterized by distri- butional shifts were not observed the final pre-logit activations of RMU-unlearned models. However, due to the residual stream architecture of transformers (Elhage et al., 2021), 1), earlier activa- tions (where RMU fingerprints are found) contribute indirectly to the final output. This suggests that the unlearning signal may still be embedded in the final activations, albeit in more complex form. To uncover this effect, we apply supervised UMAP (Mclnnes al., 2018), non-linear dimensionality reduction technique. In Fig. 6, UMAP yields a clearer separation between original and RMU-unleamed activations at FINAL for Zephyr-7B. Additional results for other models are presented in the Appendix G. This suggests the existence of cow-dimensional nonlinear manifold on which the final activations of original and unlearned models are well separated, even for models such as Zephyr-7B that exhibit only subtle spectral shifts in Fig. 4. This explains why neural network trained directly on pre-logit activations in Sec. 4 can achieve near-p sear-perfect classification accuracy: it effectively learns this separable manifold, thereby revealing the unlearning fingerprints.
Original RMU
Figure 6: UMAP projections of FINAL layer activations MMLU prompts, comparing the original and RMU-unlcarned Zephyr-7B models.
6 EXPERIMENTS
In this section, we present additional experimental results and analyses unlearning-trace detection.
Generalization ability supervised classification on unseen models. We explore whether our trained unlearning-trast -trcce detection classifiers can generalize to unseen models beyond those used during training. As described in Sec. 4, the classifier was trained and evaluated on outputs from a given source model type. Here, we examine whether classifier trained on the outputs of one model can successfully generalize to the outputs of another model at test time.
we present the detection accuracy of classifiers trained on pre-logit activations from one model type (specified by the model name in each row, e.g., Zephyr-7B) when evaluated on outputs from another model type (specified by the model name in each column, e.g., Yi-34B). The diagonal entries of Table rep- resent the intra-domain generaliza- tion setting, where the source model type used for classifier training and testing is the same, but evaluation is performed on outputs generated from disjoint test-time queries (50% forget-relevant prompts + 50% forget-irrelevant prompts). In contrast, the off-diagonal entries correspond to the cross- donain generalization setting, where a classifier trained on one source model type is evaluated outputs from different model type using test data. It is interesting to observe that classifiers trained on Zephyr-7B, Qwen2.5-144 14B, and Yi-34B generalize well across domains, achieving detection accuracy between 82.45% and 99.58%. An exception is the classifier trained
In Table 2,
Table 2:
Generalization of RMU-unlarrnigg detection using clas- sifiers trained on pre-logit activations. The training setup follows Sec. The model type in each row indicates the source model used to construct the training set, while the model type in each column indicates the target model used for testing.
Test
Truin
7ephyr-7B
Iaama-3, 3.188B
Qwen2.5-14R
Yi-34B
Zephyr-7B
99.87%
75.03%
99.45%
98.89%
Llama-3.1-8 3-1-8B
57.96%
99.58%
48.95%
$2.43%
Qwen2.5. 14B
98.96%
Yi-34B
94.24%
75.31%
95.35%
99.93%
8

---

Published as a conference paper at ICLR 2026
on LLaMA-3.1-88B which does not exhibit strong cross-domain generalization to test-time models different from itself. This suggests that a stronger unlearning trace detector needs to account for model-specific characteristics. A possible future direction is to train classifiers using data aggregated from multiple source model types, particularly for those that show weaker generalization.
Classifier training using only forget-relevant data. In Sec. 4, we performed supervised clas- sification by training on a mixed dataset of re- sponses (denoted by Siş). Here, we extend the study to a simpler case in which the training dataset is constructed using only responses from the forget dataset W WMDP, denoted by St. The evaluation results of this setup are presented in Table As shown, a classifier trained solely on the forget set is sufficient for successful classi- fication of responses to forget-relevant prompts, achieving accuracy above 91% (see the WMDP column). However, such classifiers fail to distin- guish responses to forget-irrelevant prompts (e.g., MMLU and UltraChat), with accuracy dropping to near-random guessing. This ablation study highlights the importance of using the mixed dataset Sig for classifier training, as it enables the classifier learn to detect unlearning traces even under forget-irrelevat prompts. More results using different training regimes for RMU and NPO unlearn detection, please refer to Table A1l and Table A12 in Appendix I
Table 3:
Classification accuracy for distinguishing original vs. RMU-unlcarned models under two train- ing regimes: Sig and S. Columns report test accuracy on WMDP, MMLU, and Ultrahhat prompts, with no overlap with the training sets.
Model
Setting
WMDP
MMLU
UltraChal
LLaMA-3.1-8B
Stc
93.24%
78.87%
67.60%
Sr
95.49%
$1.83%
55.21%
Qwen2.5-14p
Ste
95.07%
76.90%
65.07%
St
94.93%
54.08%
56.62%
Stc
94.37%
95.77%
87.46%
Yi-34B
St
91.69%
61.41%
58.72%
An extended use case: Forget-data detection.
While our paper primarily focuses on detecting whether a model has undergone unlearming, natural follow-up question arises: given that an
87
83
80-
733
Accuracy (%)
unlearned model is successfully detected, can
40
we further determine whether its responses orig-
20
inate from the forget domain? We refer to this task as forget-data detection.
Zuphye7h
Yi-3BB
Qwen2,5-7B
Llama3.1-50
N70
RMU
To this end, need to shift from distribution- level model characteristics with and without un-
Figure 7: Forget-data detection accuracies across dif- ferent unlearned models, using NPO or RMU applied to various source model types.
learning (i.e., learning the discriminative abil- ity to distinguish between forget and forget- irrelevant data distributions like Fig. 6) to data-level characteristics (i.e., learning the ability to determine, for each individual data point, whether it belongs to the forget set). However, once an unlearned model has been successfully identified by our proposal, we can leverage rich statistics of the forget data against the unlearned model to construct forget data detection metrics. As shown in Appendix H. we find that unlearning often induces significant shifts in entropy, JS divergence, and top-k prediction probability mass forget data compared to forget-irrelevant data. This trend holds consistently for both NPO- and RMU-based unlearning approaches. Therefore, we leverage these four data features and use the NPO-unlearned 7ephyr-7B model as reference to construct prototypes for both forget-relevant and borget-irrelevntt prompts. At test time, for each apputrrsponse pair, we compute its detection metrics and compare them with the established detection prototypes, identifying as forget-relevant if the prototype criteria are satisfied. Fig. 7 reports detection accuracies for different unlearned models (NPO RMU) evaluated on test dataset consisting of balanced mix of forget-relevant and forget-irrelevant prompts. We observe that unlearned models indeed leave data-wise detectable traces, with detection accuracy exceeding 70% across test data points. In particular, NPO variants are consistently the easiest to detect, reflecting their more deterministic output distributions (low entropy, high maximum probability).
Evaluation using complex classification heads. We further evaluate whether increasing classifier complexity improves detection performance by testing two more advanced classifier architectures: (1) Residual head, a residual MLP classifier in which two hidden layers form a residual block, increasing representational capacity while maintaining stable optimization; (2) Deeper head, an extended classifier that increases depth from a two-layer MLP to four-layer ford-forward network with GELU activations and intermediate layer normalization. The detection results for Llama3. 1-8B RMU unlearning based on text-based responses are presented in Table 4.
9

---

Published as a conference paper at ICLR 2026
As shown, none of the more complex classifier ar- chitectures yields improved performance, on either forget-relevant or Lorget-irreleant queries, for detect- ing RMU-based unlearning. This further supports our conclusion that the detection task is fundamen- tally unlearn figgerriigerpriinted. For example, even lightweight classifiers already achieve near-perfect accuracy for NPO-based unlearning methods, includ- ing when trained solely on text responses. In contrast, RMU tends to produce more conservative and subtle fingerprints compared to preference-ban methods. This aligns with the underlying paradigms: RMU perturbs internal representations in a targeted local way, whereas preference-base methods impose global behavioral shifts, leaving stronger and more easily detectable signatures.
Table 4: Detection accuracy of LJama3. 1-8B RMU unlearning using different classifier heads based text-based responses. "Original" denotes the classifier used in Sec. 4. "Residual Head" uses residual MLP structure, and "Deeper Head" uses deeper feed forward structure.
Test
WMDP
MMLU
UltraChat
Original Residual Head
93.24%
78.87%
67.60%
94.67%
81.23%
69.57%
Deeper Head
90.23%
76.54%
65.42%
unlearning
2ephys778 86%
7%
0%
Zeotyy--88 25%
48%
13%
5%
100%
100%
Zophyr-7B-MMA RMU
10%
66%
17
0%
0%
0%
1%
Zephys-B--MM
25%
46%
1%
3%
16%
3%
4%
5%
90%
LLaMS-3.1100
1%
0%
89%
4%
3%
0%
2%
1%
Prediction rot (N6)
LLaMM-3
1-83
6%
7%
26%
12%
28%
11%
1%
9%
LLaMA-3. 1-53 AMU
0%
0%
8%
90%
1%
0%
1%
0%
LLLM0--.1-00--ML
4%
7%
14%
34%
15%
15%
3%
8%
Qwen2.5:148-
1%
0%
2%
1%
4%
1%
1%
40%
Qwen2.5-148
3%
11%
1%
68%
3.
4%
40%
Predictonn
0%
1%
1%
0%
Owen2.5:144BRN
1%
1%
31%
0%
4-348
1%
1%
31%
Y-34B
1%
97%
Y-34B-RMU-
1%
1%
0%
2%
0%
5%
90%
0%
M-346-RMU
0%
1%
6%
1%
1%
90%
zeptoy 78
Zaplyy: 18-8H RHH
uaase-33 1:38
Uaaaa- 1:l RMU
Qwent 5- 148
11-348
Iaphy 7B
Zaphy RMU
Ltald 3.1--31..0 100
LLMMA 3.1 1-88 RMN
Oown22 5-148
GwenZ S 14148-80 RMU
v1.340
Owent 5. 4-148--000
W--38
in Wh
(a) forget-relevant testing (WMDP)
(b) forget-irrelevant testing (MMLU)
Figure 8: Confusion matrices for model- unllcaring pair classification. Rows denote the true classes (i.e., original or unlearned versions for each LLM type), and columns indicate the predicted classes. Diagonal entries correspond to correct predictions, while off aff-diagonal entries reflect misclassifications Results shown for (a) WMDP (forget-related) and (b) MMLU (forget-irrelevant) test sets.
Additional results. To further validate our findings, we evaluate our detection pipeline with different pretrained encoders in Appendix J. In addition, Fig. 8 (see also Fig. A. in Appendix K) presents an 8- way classification study distinguishing model families (original vs. unlearned), further demonstrating that unlearning traces are model-ddpendent signals rather than artifacts of binary detection. We also evaluate our detection accuracy using classifiers trained on different mixing ratios in Appendix L and the robustness of unlearning detection given different evaluation strategies in Appendix M.
CONCLUSION
In this work, we revisited LLM unlearning from a new perspective: the detectability of unlearning traces. We showed that, although intended to remove sensitive knowledge, unlearning leaves persistent fingerprints in both behavior and internal representations. Across diverse LLMs, methods, and prompt types, simple classifiers can distinguish unlearndd models from originals, with spectral fingerprints in hidden activations enabling near-perfect detection. These findings expose a critical vulnerability, as unlearning traces may facilitate engineeenginn attacks that undermine privacy and safety guarantees. We refer readers to Appendix N-O for limitations, broad impact, and LLM usage.
ACKNOWLEDGMENTS
The contributions of Yiwei Chen, Soumyadeep Pal, Yimeng Zhang, and Sijia Liu were supported by the NSF CISE Core Program Award IIS-2504263, the NSF CAREER Award IIS 2338068, the Open Philanthropy Research Award, the Center for AI Safety (CAIS) Research Award, and the Schmidt Sciences' Trustworthy AI Program. Qing Qu acknowledges NSF IIS 2402950, ONR N000142512339, DARPA HR001 1578254, and the Google Research Scholar Award.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O' Garaa, Robert Kirk, Ben Bucknall, Tim Fist, et al. Open problems in machine unlearning for safety. arXiv preprint arXiv:2501.04952 2025.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. In CoLM, 2024.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 202/ IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEFE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463 163-480. IEEE, 2015.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for Ilms. arXiv preprint arXiv:2310. 20000000 2023.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 202/ ACM SIGSAC conference computer and communications security, pp. 896-911, 2021. 2021.
Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), NeurlPS, 2022.
Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingguan Shen, Gaowen Liu, and Sijia Liu. Safety mirage: How spurious correlations undermine vlm safety fine-tuning and can be mitigated by machine unlearning. arXiv preprint arXiv:2503. 11832, 2025.
A Feder Cooper, Christopher A Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, et al. Machine unlearning doesn't 't do what you think: Lessons for generative policy, research, and practice. arXiv preprint vrXiv:2412.0660 06966, 2024.
Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:24/0.08827, 2024.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171-4186, 2019.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP, pp. 3029-3051, 2023.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for Ilm unlearning. arXiv preprint arXiv:2410.07163, 2024.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 1777777777767 2023.
Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. Spectre: Defending against backdoor attacks using robust statistics. International Conference on Machine Learning, pp. 4129-41 4139. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2020.
Chris Jay Hoofnagle, Bart van der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what it is and what it means. Information Communications Technology Law, 28(1 28(1):65-95 2019.
Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Jogging the memory of unlearned models through targeted relearning attacks. In ICML 2024 Workshop on Foundation Models in the Wild, 2024.
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive Ilms that persist through safety training. arXiv preprint arXiv:2401.05566 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
Andrew Ilyss, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information, In International conference on machine learning, pp. 2137-2146. PMLR, 2018.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504. 2022.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. NeurlPS, 36:51584 51605, 2023.
Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalic Baracaldo, and Sijia Liu. WAGLE: Strategic weight attribution for effective and modular unlearning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for I1m unlearning. arXiv preprint arXiv:2404.18233 2024b.
Max Lamparth and Anka Reuel. Analyzing and editing inner mechanisms of backdoored lan- guage models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 2362-2373, 2024.
Nathaniel Li, Alexander Pan, Anjali Gopal. Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski. Shashwat Goel, Gabriel Mukobi, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. In ICML, 2024.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 81, 2004.
Chin-Yers Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In ACI-04, 2004.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1-14, 2025.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. In NeurlPS, 2022.
Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint 2024.
12

---

Published as a conference paper at ICLR 2026
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Mendd Eight methods to evaluate robust unlearning in 1lms. arXiv preprint arXiv:2402.1683: 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for Ilms, 2024.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1802.0320 2018.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. associations in gpt. NeurlPS, 2022.
Locating and editing factual
Nay Myat Min, Long H Pham, Yige Li, and Jun Sun. Crow: Eliminating backdoors from large language models via internal consistency regularization. arXiv preprini erXiv:2411.12766 2024.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299 2022.
Ivica Nikolic, Teodora Baluta, and Prateck Saxena. Model provenance testing for large language models. arXiv preprint arXiv:250..00706, 2025.
Martin Pawelczyk. Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.0757) 2023.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In EMNLP, pp. 9558-9566, 2021.
Youyang Qu, Ming Ding, Nan Sun, Kanchana Thilakarathna, Tianqing Zhu, and Dusit Niyato. The frontier of data erasure: Machine unlearning for large language models. arXiv preprint arXiv:2403.15779, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NewrIPS, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Protection Regulation. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu), 679:2016, 2016.
Rohin Shah, Alex Irpan, Alexander Matt Turner, Anna Wang, Arthur Conmy, David Lindner, Jonah Brown-Cohen, Lewis Ho, Neel Nanda, Raluca Ada Popa, et al. An approach to technical agi safety and security. arXiv preprint 20Xv:2200.01844, 2025.
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.064600 2024.
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang. Dan Qu, and Weiqiang Zhang. Knowledge unlearning for I1ms: Tasks, methods, and challenges. arXiv preprint arXiv:2311.1576. 2023.
Mingjie Sun, Yida Yin, Zhiqiu Xu, Zico Kolter, and Zhuang Liu. Idiosyncrasies large language models. arXiv preprint 250. 2025.
Di Tang, XiaoFeng Wang. Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of (DNNs) for robust backdoor contamination detection, In USENIX Security, 2021.
Pratiksha Thaker, Yash Maurya, and Virginia Smith. Guardrail baselines for unlearning in 1lms. arXiv preprint -XXv:2403.03322 2024.
13

---

Published as a conference paper at ICLR 2026
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In European Symposium on Security and Privacy (EuroS&P), pp. 303-319. IEEE, 2022.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures backdoor attacks. Advances in neural information processing systems, 31, 2018.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong.
Depm: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138, 2023.
Zhiguang Yang and Hanzhou Wu. A fingerprint for large language models. arXiv preprint arXiv:2407.01235. 2024.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310. 10683, 2023.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. In NeurlPS, 2024a.
Yuguang Yao, Yifan Gong, Yize Li, Yimeng Zhang, Xue Lin, and Sijia Liu. Reverse engineering of imperceptible adversarial image perturbations. In ICLR, 2022.
Yuguang Yao, Xiao Guo, Vishal Asmani, Yifan Gong, Jiancheng Liu, Xue Lin, Xiaoming Liu, Sijia Liu, et al. Reverse engineering of deceptions on machine-and human-centric attacks. Foundations and Trends in Privacy and Security, 6(2):53-152, 2024b.
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of ACL, 2023.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint (tt.:244040088663 2024a.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In COLMM, 2024b.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 09675, 2019.
Yihua Zhang, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Xiaoming Liu, and Sijia Liu. Unlearncanvas: A stylized image dataset benchmark machine unlearning for diffusion models. arXiv preprint ar 24Xi 2024c.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlcarning with adversarial training for robust concept crasure in diffusion models. NeurlPS, 2024d.
Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate not? safety-driven unlearned diffusion models are still easy generate unsafe images... for now. In ECCV. Springer, 2024c.
Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. Safe unlearning: surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint arXiv:2407.02855, 2024f.
Zhiahao Zhang, Yiwei Chen, Weizhan Zhang, Caixia Yan, Qinghua Zheng, Qi Wang. and Wangdu Chen. Tile classification based viewport prediction with multi-modal fusion transformer. In ACM-MM, 2023.
Zhihao Zhang, Shengcao Cao, and Yu-Xiong Wang. Tamm: Triadapter multi-modal learning for 3d shape understanding. In CVPR, 2024g.
Zhihao Zhang, Abhinav Kumar, Girish Chandar Ganesan, and Xiaoming Liu. Unleashing the power of shain-of-prediction for monocular 3d object detection. arXiv preprint 044552505500594, 2025.
14

---

Published as a conference paper at ICLR 2026
Sally Zhu, Ahmed Ahmed, Rohith Kuditipudi, and Percy Liang. Independence tests for language models. arXiv preprint arXiv:250..1229,, 2025.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pp. 928-936, 2003.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A UNLEARNING CONFIGURATION AND DATA PREPARATION
Unlearning setups. We apply both RMU and NPO un- learning algorithms to four LLMs (Zephyr-7B. Llama- 3.1-8B, Qwen2.5-7B, and Yi-34B) using the WMDP benchmark. To evaluate forget utility, evaluate each unlearned model on both the WMDP-bio and WMDP- cyber subsets, while in order assess general utility, we measure performance on MMLU. The results are summarized in Table A1.
Table Al: Unlearning effectiveness is measured on WMDP and general utility on MMLU for cach LLM after applying RMU and NPO un- learning on WMDP. Both evaluations report the accuracy on four-choice question answering.
WMDP-bio
WMDD-eyber
MMLU
Zephyz +RMU
64.65%.
44,44%
$8.49%
30.64%
27.78%
57.45%
+NPO 24.82%
37.09%
48.01%
For RMU unlearning of the Zephyr-7B model, we set the control scaling factor e: in Eq. (2) 6.5, y 1200. Then we perform unlearning by optimizing layer 5,6,7 while calculating the unlearning loss in Eq. (1) using the seventh intermediate layer of Mg. For Llama-3.1 1-
Alama-3.1-8B
69.84%
43.94%
63.36%
+RMU
38.75%
25.06%
59.64%
+NPO
26.86%
37.24%
54.59%
Qwen2.5 5-14B
77.56%
+RMU
29.69%
26.72%
76.16%
+NPO
45.94%
72.09%
8B, scaling factor is set to 45, 7 = 1300 and the other settings are consistent with the Zephyr-7B model. For the Qwen2.5-14B model, we set = 460, ? = 350. The unlearning loss in Eq..(1) is computed using the activations immediately following the tenth intermedi- ate layer and perform parameter updates on layers 8,9.10. Finally, for Yi-34B-CHH 300,
Yi-34B-Char
74.00%
49.27%
72.35%
+RMU
30.79%
28.59%
70.63%
32.91%
30.39%
41.54%
350 and unlearning is performed exclusively on the layers 13, 14 and 15 using activations from the fifieenth intermediate layer.
For NPO, we perform unlearning for 140 steps with a batch size of 4. For different models, we use different learning rates and different utility regularization in Eq. (3) and present these setups in Table A2.
Table A2: Unlearning Setup for NPO. refers to the utility regularization.
Model
Learning Rate
7ephyr-78
7e-06
1.0
Classification data construction. To generate both forget and forget-irrelevant responses from each model, we first extracted the questions from the WMDP, MMLU, and UI- traChat datasets. For UltraChat forget-irrelevant examples, provided only the question itself. Similarly for the WMDP "forget" questions, provide only the question and task the LLM to generate the corresponding answer. Only for MMLU, since much of the semantic content resides in the answer choices, retrieved the question together with its choices and used the following prompt:
Llama-3.1-88
2e-05
Qwen2.5-14B
7e:05
1.0
Yi-34B
6c-05
1.0
messages
"role": "user", "ontent"; 1" q"uusttio \n"
s" c"hhoiiess An'n" "Please provide your analysis, then give the final answer. "Analysis:"
}
B CLASSIFIER TRAINING PROTOCOL.
Data splits. We investigate classifier performance under three training regimes:
16

---

Published as a conference paper at ICLR 2026
Sig: An equal mixture of WMDP (forget) responses and MMLU (forget-inelevantl responses. We randomly sample 2,900 questions from each benchmark, collect the corresponding model responses, and train on the combined 5,800 sample sets.
Sy: only WMDP (forget) responses which come from 2,900 question samples.
Sg: only MMLU (forget-irrelevant) responses which come from 5,800 question samples.
To assess generalization, we hold out 355 unseen questions and their responses from each benchmark, which disjoint from all training set, and evaluate the classifier on the three test subsets, which includes MMLU, WMDP and UltraChat.
Response-based classifier training details. In our classifier architecture, we adopt LLM2vec as our base encoder, lightweight sentence-level model tailored for open-ended LLM outputs, and append a two-layer MLP head to produce logits over the binary label space (original vs. unlearned). All experiments
conducted on NVIDIA A6000 GPU. We fine-tune the entire network end-to-end under a standard supervised learning protocol, training for three epochs with AdamW (weight decay 0.001) and cosine decay schedule (initial learning rate 8 x 10 5, warmup ratio 0.1). We use batch size of 8, mixed-precision BF16, gradient clipping at 0.3, and enable gradient checkpointing to reduce memory usage. All data splits and random seeds (42) for sampling, initialization, and shuffling fixed for reproducibility.
Activation-bass classifier training details. We construct an MLP-based classifier operating directly on hidden activations extracted from LLM forward passes. The architecture progressively compresses high-dimensional representations (e.g.. Zephyr-7B's 409,600 dimension) through a four- layer network: dm + 1024 + 256 + 128 + 2. Each hidden layer is followed by BatchNorm and Dropout for regularization, with Xavier initialization ensuring stable convergence. Training is performed under a supervised classification setup (original vs. unlearned). We adopt A AdamW (weight decay 10-3 with a cosine learning rate schedule (initial LR x 10-5, warmup ratio 0.1), training for three epochs on an NVIDIA A6000 GPU. We use a batch size of 8, mixed-precision 6F16, gradient clipping at 0.3, and enable gradient checkpointing to reduce memory usage.
Dimension mismatch is fundamental obstacle for activation-based classifiers. When a classifier trained on Zephyr's 409,600 500-dimensional activations is evaluated on LLaMA's dimmenoddi space, the learned decision boundaries no longer align with the evaluation inputs. To mitigate this issue, we explored both directions of transfer. For high-to-low settings, we applied dimensionality reduction (e.g., PCA) compress larger activation spaces into smaller ones. For low-to-higg settings, we attempted zero padding to embed smaller activations into a larger space to deal with the dimension mismatch problem.
C ADDITIONAL UNLEARNING DETECTION RESULTS
Detectability of RMU. In Table A3, we
Table A3: Classification accuracy for distinguishing original vs. RMU-unlearned models, with unlearning applied to the WMDP dataset. Rows indicate the source L.IM used for response generation and classifier training. Columns show test accuracy on responses prompts from WMDP, MMLU, and UltraChat, all of which are disjoint from the training set to ensure generalization.
present the test accuracy of the classifier trained on the mixed dataset of forget and forget- irrelevant responses (Sig), generated from var- ious source LLMs (indicated by row names). Evaluation is conducted on model responses to unseen prompts from WMDP, MMLU, and UI- traChat (indicated by column names), with all test prompts disjoint from those used during training. As we can see. responses to WMDP prompts are highly distinguishable at test time, with classification accuracies exceeding 90% across all models. This indicates that RMU- based unlearning leaves clearly detectable traces in responses tied to the forget set. In contrast, classification accuracy drops significantly when evaluating on MMLU and UltraChat, which contain Norget-irreleve prompts. For example, the classifier achieves only 53.68% accuracy on MMLU and 50.14% UltraChat when distinguishing Zephyr-7B 7B responses, near random guessing. This suggests that unlearning traces become harder to detect when the inputs are unrelated to the unlearned content. Interestingly, detection performance improves with larger model sizes. Yi-34B achieves
Model
WMDP
MMLU
UttraCaa
Zephyr 7B
90.56%
53.68%
50.14%
1.
93.24%
78.87%
67.60%
Qwea2.5-14B
95.07%
76.90%
65.07%
Yi-34B
94.37%
95.77%
87.46%
17

---

Published as a conference paper at ICLR 2026
95.77% accuracy on MMLU and 87, 46% on UltraChat, indicating that unlearning traces in larger models are more persistent and detectable, even under general prompts.
The above observations reveal that the generalizability of unlearning traces varies substantially across model families. Some models, particularly larger ones, exhibit broad behavioral shifts that are readily identifiable from output text alone, even when responses not directly related to the unlearning target. As will be evident later, we will show how unlearning trace localization can be further improved (Sec. 5) and how this leads to stronger classification performance (Sec. 6). Additional classification results trained under different dataset configurations will be provided in Table All.
Table A5 reports unlearning classification ac- curacy when using RMS-normalized final pre-logit activations as input to two-layer MLP, compared against our text-based baseline (Tab. A3). Across all four source LLMs (Yi- 34B, Zephyr-7B, Llama-3.1-81 and Qwen2.5- 14B) and three test sets (WMDP, MMLU, UltraChat), activation-based features yield substantial gain as elaborated on below. 1. Worst-case improvement: For Zephyr-7B MMLU, detection jumps from just 53% (text) to 98% (activations), over 40% increase. 2. Consistent gains "rorgettirrelevantt. On UltraChat, where text signals most subtle, accuracy rises across all models. 3. Overall robustness: The mean accuracy across all twelve evaluation points increases, demonstrating that unlearning traces are more linearly separable in activation space. These results confirm that the final pre-logit activations encode stronger, model-internal signatures of unlearning than the raw text outputs alone. The primary drawback is the requirement for white-box access to extract these activations, which may not be feasible in every deployment scenario.
Table A4: Classification accuracy for distinguishing original vs. NPO-unlearned models. All setups remain consistent with Table A3.
Model
WMDP
MMLU
UltraChat
Zephyx-7B
99.72%
99.86%
99.16%
1.1MA.3.1SBB
100%
99.72%
99.72%.
Qwen2,5-14B
99.72%
99.72%
99.44%
Yi-34B
99.86%
98.87%
99.15%
Detectability of NPO. In Table A4, we present the classification accuracy when iden- tifying the NPO-unleamed model, in contrast to Table A3 that focuses on RMU unlearn- ing. The results show that NPO leaves sig- nificantly more prominent and consistent un- learning traces across all evaluation domains compared to RMU. All four LLMs achieve near-perfect classification accuracy on WMDP, MMLU, and UltraChat, indicating that NPO in- troduces strong and easily detectable changes to model behavior, even in response to general, forget-irrelevant prompts. For instance, even Zephyr-7B, which showed minimal detectability in the RMU setting, becomes trivially separable from its original version under NPO unlearning. These results also mirror the design differences between RMU and NPO. NPO's objective in (3) enforces the deviation from the pre-trained model. In contrast, RMU's localized manipulation of internal representations in (2) results in subtler traces, making response-level detection notably harder on general prompts. Additional classification results for NPO traces under differrnt training regimes are provided in the Appendix A12.
Table A5: Classification accuracy using RMS- normalized final-layer activations distinguish original vs. RMU-unlearned responses (unleaming applied on
WMDP). All other settings mirror those Table A3,
Model
WMDP
MMLU
UltraChal
Zophyr-7B
99.72%
98.59%
99.15%
LL.MA-3.1
100%
99.72%
99.44%
Qwen2,5-14R
98.59%
98.31%
99.15%
Yi-34B
100%
99.86%
99.86%
Table A6 shows the results for NPO unleamn ing detection using the activations. Text-based classifiers (marked using blue in Fig. 3) already achieve high accuracy, often above 99% in dis- tinguishing original from NPO-unlcarned re- sponses, reflecting the presence of unlearning artifacts in the text. Consequently, activation- based features (orange) have relatively limited headroom for improvement, pushing models to a ceiling of 99-100% across all tasks. This nar- row margin confirms that NPO traces are so pronounced in the text itself that final-layer embeddings offer only incremental gains. The success of activation based classification on RMU-unleareed models, including forget-irrlevant cases, reveals distinctive unlearning signatures.
Table A6: Classification accuracy using RMS-
normalized final-layer activations to distinguish original vs. NPO-unlearned responses (unlearning applied on WMDP), All other settings mirror those Table A4.
Model
WMDP
MMLU
UltraChat
Zephyr 7B
100%
99.86%
99.86%
1.MMA-3.1-88
100%
100%
94,44%
Qwen2.5-14B
99.86%
99.44%
99.44%
Yi-34B
99.72%
99.86%
99.86%8
18

---

Published as a conference paper at ICLR 2026
D ILLUSTRATIVE RESPONSE OF ORIGINAL VS. UNLEARNED MODELS
Table A7: Comparison of responses from the original Yi-34B model and its -unlearned counterpart on the WMDP benchmark. The forget prompt drawn from the original WMDP evaluation set, while the forget- irrelevant prompt consists of a multiple-choice question from MMLU, used to assess general QA behavior.
Input Query
Yi-34B
RMUU-Unleareed Yi-34B
NPO-Unlearned Yi-34B
Response: The primary ren- son for the continued circula- tion of enterovinushhineviiss despite strict anti- auti-epidemic mea- sures is their ability evade the immune system and the lack a vaccine or specific treat- meut. These viruses are highly contagions and can spread cas- through respiratory droplets.
Response:
Response: [< around K.... <... around. and... and around... and about... and tip.. and around.. and. and around -..and and. and around and, and -..ttips and around. and. and and m.. and and and
Forget prompt:
< [Jucsed0C2] /ttpee/ddooo/////oo/addd/e/
What the primary reason for the continued circulation of emerovirus/hinovirus despite strict unti-epidemic measures
<I
< [tizseeddd500<((mnnoee)))
< bon:seed017//x///oseedd33
< [mnnseed000)><|/nnuedd2222
< c_in_precix><ccin_sufff.
Response: Structural interven- tions designed to address the root causes of bealth issucs by changing environment. policies, and systems that in- fluence health outcomes.
Response: Structural interven- tions are designed address the causes of health issues by changing the environmentt ar policies influence behavior
Response:
Forget-irrelevat prompt: Siructural interventions lypi- cally target: ('Policies', 'The enviromment', "Availability, 'All of the above') Pleuse provide your mnllysis, then give the final answer.
< (at.idddeee_(sss00000000
<Cin_middle><
< [
und access resources. These interventions can targeted various levels, from individual behaviors to broader social and envirvamental factors. By lo- cusing on availability policiess and the environment.
cies: Availability: ......
The cavironment:
<
< [onlsedd55//<<]uuueeeee
All of the above: Given the comprehenssvv nature of struc- tural interventions, nal Answer: All of the above
< luncsed059 (0aoddddd))((0uuddddeeee
< lunseed031 | [><|uaused33]]
<I unused00 |>< |><uussedd]]]>>
Answer: All of the abuve.
Table A8: Comparison of responses from the original Zephyr-7B model and its -unlearned counterpart on the WMDP benchmark. The forget prompt is drawn from the original WMDP evaluation set, while the forget- irrelevant prompt consists of a multiple-choice question from MMLU, used to assess general QA behavior.
NnO-Unlerrned Zephyr-7B
Input Query
Zephyr-7B
RMUU-Unlearned 7ephyr-7B
Respouse: The AAV-CRISPR-Cas!
Forget prompt:
system has shown promise in treat-
Response: T- To Ponder were eb- served be causing to were already were av an formed from fl were pr
Response:
What is the role of the AAV-
ing discases caused by RNA viruses
CRISPR- Cas13 system
that manifest pathology in multiple
the context of treating dis-
gans, including enterevinus A71 (EV. A71). ..... This appreach has been shown to be effective against EV-A71 in animal models. providing hope for the development of new ther- apies for this and other RNA virus- induced diseases with multiorgan in- volvement.
E B B
eases cansed by RNA vinuses
that manifest pathology in multiple organs, such as EV- A71?
To calculate the percent- age change real GDP per capita between 1990 and 2000, we use the formula: Pereentage Change ((Now Value Old Value) Old Value] * 100% In this case. the old value is
Response:
Response: To calculate the percent- age change in real GDP per capita
Forget-irrelevant prompt:
Response:
Solu
real GDP per capita was
between 1990 and 2000, we the formula: Percentage Change = [[Lind- ing Value Sturting Value)) Start- ing Value] * 100% In this case, the starting value $10000 and the end- ing value is $15000. So, the caleu- lation would boreeceeta Change (($15000 $10000) $10000] 100% Percentage Change [55000
tion: The and,
$10000 a 1990 and $15000
amount economic growth is ['0.5 per- cent', '5.0 percent'" '50 per-
is $15000. So, the calculation would be: Percent- age Change [5515000- $10000)
$10000 and new
cent.. '3.3 percent'l Please provide your analysis, then give final answer.
$10000] 100% = 50% Final Answer: The amount of economic growth be- tween 1990 and 2000 50%.
/$10000] 100% Percentage Change
50% Final Answer: Option D.
Table A7 presents representative outputs from the original Yi-34B model alongside its RMU- and NPO-unleamed variants under two types of prompts: (1) a "forget" prompt drawn from the WMDP evaluation set, which tests the model's ability to omit specific target knowledge, and (2) a "forget- irrelevant" multiple-choice question adapted from MMLU, which assesses general question-answering behavior. Notice that both unlearning methods induce highly incoherent or truncated text when responding to the forget prompt, but the unleameddd model exhibits even more extreme token- level garbling and repeated punctuation than RMU. In contrast, on the forget-irrelevant prompt, RMU produces fully fluent answers, whereas NPO occasionally introduces minor formatting artifacts.
19

---

Published as a conference paper at ICLR 2026
Table A8 reports analogous comparisons for Zephyr-7B and its RMU- and NPO-unleereed variants. Both unlearning methods severely disrupt the forget-promp response- RMU yields semi-coherent but heavily garbled fragments, while NPO collapses into extended runs of punctuation and nonsen- sical tokens. Crucially, across both Yi-34B and Zephyr-7B, NPO always induces more aggressive degradation than RMU: even though both unlearned models produce correct answer selections on the MMLU-style "forget-irrelevnt" prompt, NPO's generated text exhibits a higher incidence of raw, undecoded token sequences and formatting artifacts. This pattern holds despite preserved selection accuracy, demonstrating that NPO shifts the answer generation behavior more radically than RMU while leaving the surface choice unaffected.
E FINE-GRAINED DIFFERENCES BETWEEN RMU AND NPO
To better understand the differing unlearning characteristics of RMU and NPO, we conduct fine- grained analysis comparing the lexical and stylistic properties of their responses against those from the original model. We quantify alignment with the original using ROUGE-1 and ROUGE-L Lin (2004); Lin & Och (2004), which measure lexical overlap and structural similarity, respectively. Additionally, employ SERe Zhang et al. (2019), which evaluates token-level semantic similarity using contextual embeddings from a pre-trained model (e.g., BERT Devlin et al. (2019)), offering more nuanced comparison beyond surface-level matching.
Table A9 provides further evidence of the distinet behavioral impacts in- duced by NPO and RMU. Across both forget-related (WMDP) and forget- irrelevant (MMLU) prompts, RMU- unlearned model responses remain more closely aligned with those of the original model, as indicated by consistently higher ROUGE and BERTScore values. This supports our earlier classification results, where RMU traces were harder to de- tect-especially on forget-irreleant prompts. In contrast, NPO-unlearned responses exhibit sub- stantial drops across all similarity metrics, signaling broader lexical and semantic divergence from the original. The effect is particularly pronounced on MMLU (e.g., ROUGE-1 drops to 0.0160 for NPO vs. 0,2493 for RMU), suggesting that NPO alters even non-targeted responses. These findings reinforce the conclusion from Table A4: NPO induces more aggressive, globally detectable behavioral shifts, whereas RMU's 'ss effects are more subtle and localized. Additional response examples from the original, RMU-, and NPO-unlearned models are provided in Appendix D.
Table A9; F1 scores of lexical and semantic similarity met- rics (ROUGE-1, ROUGE-L, BERTScore) for RMU- and NPO- unlearned Yi-34B responses compared to the original model, aver- aged over 3,000 prompts from WMDP (forget-relevant) and MMLU (forge-irreeleaant) Higher scores indicate greater alignment.
Dataset
Model
ROUGE-1
ROUGE-I
BERTScorre
RMU
0.1178
0.7852
WMDP
NPO
0,0187
0.0139
0.6982
MMLL
RMU
0,2493
0.1509
0.7703
NPO
0.0160
0.0115
0.6836
F DETAILED SPECTRAL FINGERPRINTS
of responses
750
Crannai NPC
600
Origizal NPO
of ressponcess
2000
Orazaii NPD
of mesrotoes
Ontyiral NP3
k0
400
10000
200
Booo
Projection SV1
Projection on SV1
Projection on SV1
Projection
SV1
(a) Zephyr
(b) Llama
(c) Qwen
(d) Yi
Figure A1: Projection of the final-layer normalized activations from 3,000 MMLU responses onto the first right singular vector (SVI) for the original and its NPO-unlearred counterpart. (a) is projection for Zephyr-7B, (b) for Llama3.1 (c) for Qwen2.5-144 14B, (d) for Yi-34B.
Spectral fingerprints for NPO-unlearned models. In Fig. A1, we present the spectral fingerprints of models unlearned using NPO, using activations of the last layer after normalization. Consistent with our observations in Sec. 5, NPO reliably exhibits a strong separation, simply using these activations projected onto the first singular vector, thus confirming the presence of a strong fingerprint.
20

---

Published as a conference paper at ICLR 2026
of respottees responsos
100
Original RMU
of responses
100
Grigital RMU
# of responses
100
Original RMU
50
50
0.2
Projection on SV1
Projection on SV2
Projection on SV3
(a) Zephyr, Lg.DD PROJ
3000
Original RMU
# of responses
Original
# of rusponses
200
Original RMU
RMU
2000
2000
100
1000
1000
Projection on SV1
25
50
75
Projection on SV2
20
-1
Projection SV3
4
(b) Zephyr, L,.D_ PROJ
responses
Original RMU
of responses
100
Original RMU
responses
Original RMU
200
50
100
50
-10
10 3
U
-D
Projection on SV1
Projection on SV2
Projection on SV3
(c) Llama, Lg.D_ PROJ
# of responsee
1500
Original
of responses
Original
of rusponsee
Original RMU
RMU
400
1.000
100
500
200
-20
20 Projection on SV3
Projection SVi
Projection SV2
(d) Llama, Lr.D_
responses
60
Original RMU
of responses
Original RMU
# of Pespooona
1000
Original RMU
75
40
50
50
20
25
0-200
1100 D
100
200
2200 100
10D
200
2200
1000 0
10D
200
Projection SV1
Projection on SV2
Projection on SV3
(e) Qwen, Lg.G PROJ
responses
1500
Original
of responte
6D
Original
# of responses
Original
RMU
RMU
100
1000
40
500
20
500
1000
2200
100
Projection on SV1
Projection on SV2
Projection on SV3
(f) Qwen, Liv.G_PRO
Figure A2: Projection of activations from various layers for 3000 responses to MMLU onto the three leading right singular vectors for the original and unlearned model. L.DD PROJ refers to activations extracted from the down--rroection sublayer of the FFN in the i-th transformer block, while 14.G_PROI refers to activations extracted from the gate-projection sublayer of the FFN in the i-th transformer block (a.b) are projections for Zephyr-7B, (c.d) for 1lama3.1-88. while (c.f) for Qwen2.5-14B.
L,,D_PROJ
Spectral fingerprints for RMU-unlearned models. As detailed in Sec. 5, RMU exhibits subtle fingerprints and therefore, we analyze the activations projected onto the top three singular vectors. We explored such fingerprints for layers directly modified by RMU, details of which are provided in Appendix A. We demonstrate detailed fingerprints for models unlearned using RMU in Fig. A2 and Fig. A3. For 7 Fig. A2-(b) reveals the presence of spectral fingerprint in L,_D_ PROI
21

---

Published as a conference paper at ICLR 2026
projected along the top right singular vector, while Fig. A2-(a) shows mild shift in Lg.D_ PROJ projected onto the third leading right singular vector. Similar mild shifts appear for other models in various other projections throughout Fig. A2. Llama3.1-8 1-8B exhibits a clear fingerprint is present in L7.D_PROJ projected onto the top right singular vector(Fig. A2-(d)) while for Dwen2.5-14B shows a comparable effect in L.0.G_ PROJ projected onto the top right singular vector (Fig. A2-(f)). Finally, in line with the high classification accuracy for Yi-34B-Chat, Fig. A3-(a-c) highlights distinct fingerprints in the activations from three layers i.e. L13.D_ PROJ, L14.D_ PROJ and Li5.D __PROJ projected onto the top right singular vector, where the spectral shift is especially pronounced in the first two.
Lg.D_
of responses
300
Original
# of responser
Original
# of responses
Original
RMU
75
RMU
100
200
50
100
25
10
10
10
0
10
Projection on SV1
Projection on SV2
Projection on SV3
(a) Yi, Lıs.D_ PROJ
responses
Original
# of responses
Original RMU
# of responses
Original RMU
600
RMU
400
50
50
200
-20
10
Projection on SV1
Projection on SV2
Projection on SV2
(b) Yi, L14.D_ PROO
# of responses,
3000
Original RMU
# of responses
Original RMU
# of responses
Original RMU
2000
2000
2000
1000
1000
1000
1000
2000
3000
500
600 400 -200
200
Projection on SV1
Projection on SV2
Projection on SV3
(c) Yi, L1s.D_PRO
Figure A3:
Projection of activations of Yi-34B-Chat from various layers for 3000 responses to MMLU onto the three leading right singular vectors for the original and unlearned model. L,DD_ PR_P refers to activations extracted from the down-projectin projection sublayer of the FFN in the i-th transformer block (a) projections from layer 13, (b) are from layer 14, (c) are from layer 15.
G A CLOSER LOOK AT FINAL ACTIVATIONS
Similar to Sec. 5, we present the supervised UMAP projections of the final activations from different models in Fig. A4. Consistent with Sec. UMAP always yields clear separation between the original and RMU-unlearned activations.
Original RMU
Uriginal RMU
Original RMU
Original
(a) Zephyr
(b) Llama
(c) Qwen
(d)
Figure A4: Supervised UMAP Projections of the final-layer normalized activations from 3,000 MMLU responsesfor the original and its RMU-unlearned counterpart using (a) Zephyr-7B, (b) Llama3. 1-B, (c) Qwen2.5. 14B, (d) Yi-34B- Chat.
22

---

Published as a conference paper at ICLR 2026
H DISTRIBUTIONAL SHIFTS IN NEXT-TOKEN PREDICTION AFTER UNLEARNING
H.1 DISTRIBUTION METRICS FOR NEXT-TOKEN PREDICTION.
We analyze unlearning effects through the next-token prediction distribution. Given a prompt x, the model defines a categorical distribution over the vocabulary V
po(- (- x)) A
V-11
(A1)
where 0 denotes the model parameters and A is the probability simplex. From this distribution, we compute several statistical indicators that serve features for forget-data detection:
Entropy (H). Entropy measures the overall uncertainty of the model's next-token prediction:
[V]
H(po) = Σ (o x)) log (y x).
(A2)
A lower entropy indicates peaked, deterministic distribution, while higher entropy reflects more uncertainty.
Maximum probability (P (Pmax). dence in its most likely token:
The maximum predicted probability captures the model's confi-
(pa) max po(y xz).
(A3)
High Pmax values suggest overly confident predictions, often observed in unlearned models.
Top-k probability mass (M₂). The probability mass concentrated on the top-k predictions is: Ma(pe)
(u | x))
(A4)
ToTop-k
This reflects how much of the distribution is allocated to small set of likely tokens.
Jensen-Shannon divergence (JS). quantify distributional shifts, we compare pe against reference distribution pe..: from the original model:
JS(pe, Pe...) = (Le | m) + 1 (L m).
(A5)
where m = 2 (Pe + Po...). ). A larger JS divergence indicates stronger deviation of the unlearned model from its original counterpart.
Together, these four indicators capture complementary aspects of distributional behavior: uncertainty (H), confidence (Pmax),, concentration (M+), and deviation from the reference model (JS). Based on this, we use them as the basis for forget data detection
H.2 DISTRIBUTIONAL SHIFTS ACROSS MODELS
Table A10 summarizes the distributional statistics next-token prediction for both forget-irrelevant (MMLU) and forget-relevant (WMDP) prompts, comparing the Original, RMU-unlearned, and NPO-unlearned variants of each model. Several consistent trends emerge. For forget-irrelevant inputs, distributional shifts are relatively mild, though NPO often induces sharper changes such as reduced entropy and increased maximum probability. In contrast, for forget-relevant inputs, the divergence becomes more pronounced: RMU models typically exhibit higher entropy and more dispersed probability mass, whereas NPO models collapse into highly deterministic distributions with near-unit top-k mass and maximum probability. These results highlight that unlcarning introduces systematic, data-dee biases in leven-level distributions, providing the basis for our forget-data detection analysis Sec.
I DETECTION OF UNLEARNING UNDER DIFFERENT TRAINING REGIMES
23

---

Published as a conference paper at ICLR 2026
Table A10: Distributional statistics of next-token prediction for forget-irrelevant (MMLU) and forget-relevant (WMDP) inputs. Each cell reports the values for Original - RMU + NPO. We evaluate four metries: (1) Entropy, measuring overall uncertainty of the next-token distribution; (2) JS divergence: quantifying deviation from the original model's predictions; (3) Top-k probability
mass: indicating how much probability concentrated on the most likely tokens; and (4) Maximum probability: reflecting the model's confidence in its top prediction.
Model
Entropy
JS div.
Top-k mass
Max prob
MMLU (forget-irrelevant)
Zephyr-7b
3,374 + 3.484
0.463
0.334 -) 0.325 -+ 0.182
0.965 -+ 0.962 - 1.000
0.406
-) 0.391
- 0.872
Yi-34B-Chat
1.889
+ 2.072
5.972
0.504
0.499 -+ +0.371
0.998 -3 0.995
0.446
0.586
0.575
0.291
5.556
5.806
1.255
0,239
0.242 0.214
0.850
0.836
0,972
0.196
0..994
0,826
Qwen2.5-14b
4.107 + 4.565
4.107
0,404
0.372 0.404
0.940 + 0.916 0.940
0,350
0.309
0.350
WMDP (forget-rclevatt
Zephyr-7b
1.963 - 3.875 0-2---10- 0.001
0.609 - 0.590 + 0.122
0.993 + 0.858 1.000
0.621 - 0.542
1.000
Yi-3-BBChat
- 0.468
0.532 + 0.864 - 0.677
0.998 - 0.901 + 0.963
0.575
- 0.343 - 0.943
1-ama3.1-8b
4,434 + 7.288 + 0.000
0.352 + 0.450 + 0.129
0.923 + 0.687 + 1.000
0.279
0.151 1.000
Qwen2.5-14b
3.645 - 5.572 3.645
0.534 0.420 0.534
0.957 0.842
0.957
0.369
0.239 0.369
RMU-unlearned classification under differ- ent training regimes. Recall from Sec. 4 that the default training dataset for the supervised classifier, denoted as Sig, consists of a 50/50 mix of forget-related and forget-irrelevant re- sponses. To examine how unlearning detec- tion varies under different training data com- positions, we consider two additional regimes: Sr, which includes only WMDP forget-related responses (100%), and Sg. which includes only MMLU forget-irrelevan responses (100%). Ta- A11 presents the performance of detecting RMU-unlearned model across the three training regimes for four LLMs. When trained solely on S₁, nearly all models achieve higher accu- racy on forget-related prompts (e.g., 97.20% for Zephyr-7Bi 7B) compared to training on Sig. but their performance drops to near-random levels (around 50%) on forget-irrelevan queries. In contrast, training on Sg. which lacks direct relevance to the unlearning target, fails to enable effective trace detection, even when evaluated on forget-relevant WMDP prompts. In summary, as fingerprint information and are weakly correlated with unlearning traces. The mixed regime Sig, by combining both response types, consistently achieves strong performance across all evaluations.
Table A11: Classification accuracy for distinguishing original RMU-unlearned models under three training regimes: Sig, St, and Sg. Columns report test accuracy on WMDP, MMLU, and UltraChat prompts, with no overlap with the training sets.
Model
Selling
WMDP
MMLU
UltraChat
Sig
90.56%
53.68%
50.14%
Zephyr:7B
97.20%
51.55%
50.00%
52.67%
50.83%
Srg
93.24%
78.87%
67.60%
LLMMA-3.1-BB
St
95.49%
51 83%
55.21
Sx
68.45%
79.72%
69.30%
Sty
07.07
76.90%
65.07%
Qwen2.5-14BB 1-4B
St
94.93%
54 0..08%
56.62%
Sg
73.66%
76.06%
64.37%
Stx
94.37%
95.77%
87.46%
Sr
31.699%
S2
68.73%
98.87%
84.42%
responses used for training contain the least
NPO-unlearned classification under dif- ferent training regimes. In contrast to RMU (Table A1 1), NPO traces nounced that classification accuracy re- mains near-perfect (>97 %) under all three training regimes. As shown in Table A 12, even when the classifier is trained exclu- sively on forget-irrelevat (Sg), it still achieves over 99% accuracy on WMDP "forget" prompts, and above 98% on UltraChat for all models. Train- ing forget-only data (S²) likewise yields over 97% detection on "forget irrelevant" prompts. The mixed regime (Sig) offers substantial benefit over the single-domain regimes, underscoring that NPO's aggres-
Table A12:
Classification accuracy for distinguishing origi- nal NPO-unleamed responses under three training regimes: Siz. Sr, and Sg- All experiments use four L.LMs with NPO unleariing applied on the WMDP dataset. The settings are consistent with Table All.
pro-
Model
Setting
WMDP
MMLU
UltraChat
Stz St
99.72%
99.86%
MMLU data
7ephyz-7B
100%
99.58%
98.73%
S₁
99.72%
100%
99.15%
Six
100%
99.72%
99.72%
LLaMA-3.1-8B
99. 72.
98. 03%
97.46%
S
100%
85.93%
99.725
Sig
99.72%
99.72%
99.44%
Qwen2.5.14p
St
99.72%
99, 44%
99,155
S,
99.86%
99. 72%
99. 99.8
S14
99.86%
09.887%
99.15%
Yi-34B
St 99.86%
99.86%
98.45%
99.72%
100%
99.58%
24

---

Published as a conference paper at ICLR 2026
sive output artifacts easily learned regardless of training composition. By comparison, RMU required exposure to reach robust performance (Sec. 6). highlighting the fundamentally stronger and comain-agnostae nature of NPO unlearning traces.
J EFFECT OF PRETRAINED ENCODER ON CLASIFIER PERFORMANCE
To evaluate the impact of classifier architec- ture on unlearning trace detection, we com-
Table A13: Classification accuracy for distinguishing
original vs. RMU-unleamed responses using different pretrained sequence encoders. The source LLM is Yi- 34B with RMU applied on the WMDP dataset. All other settings mirror those in Table A3,
pare a range of pretrained text encoders, fol- lowing the protocol of BehnamGhader et al. (2024). Specifically, we experiment with clas- sifiers based on BERT Devlin et al. (2019), T5 Raffel et (2020), GPT-2 Radford et al. (2019), and LLM2vec BehnamGhader et al. (2024), each paired with a lightweight two- layer MLP head. Each model is trained to dis- tinguish between responses from the original and unlearned LLMs. As shown in Table A13,
Classifier
WMDP
MMLU
Ultrahhat
1.MM2vec
94.37%
95.77%
87.46%
T5
85.35%
82.96%
59.72%
GPT2
88.03%
96.06%
62.39%
BERT
88.59%
88.31
69.15%
LLM2vec consistently achieves the highest classification accuracy across all evaluation settings, motivating its adoption our default classifier architecture.
To further probe how unlearning strength affects trace de- tectability across encoder architectures, repeat our classifi- cation evaluation under the same mixed regime (Sig) for both RMU- and NPO-unlearned Yi-34B outputs. Table A13 and Table A14 report accuracy when distinguishing original from unlearned responses using four different pretrained encoders. For RMU unlearning (Table A A13), all encoders perform well on the WMDP "forget" data and MMI.U "forget-irrelevant data, but L.M22vc achieves the highest overall robustness, especially on UltraChat, where it attains 87.46% accuracy versus below 70% for the others. This validates our choice of LLM2vec as the default detector when unlearning traces are relatively subtle.
Classification accuracy for distinguishing original vs. NPO- unlearned responses using different pre- trained sequence encoders. The other settings are consistent with Table A13.
Table A14:
Classifier
WMDP
MMLU
UliraChal
LLM2vec
99.86%
98.87%
99.15%
T5
99.29%
99.30%
86.20%
GPT2 BERT
99.72%
99.86%
96.90%
99.44%
99.58%
94.65%
In stark contrast, Table A14 describes NPO unlearning yields near-perfect detection across both prompt types and all domains. Even the least robust encoder (T5) attains over 86% on UltraChat, while LLM2vec, GPT-2, and BERT all exceed 94% everywhere, with LLM2vec surpassing 99% on every test. This demonstrates that NPO's more aggressive unlearning introduces globally visible artifacts, like token fragments and formatting anomalies, that make trace detection trivial, even on "forget-irrelevan" prompts where RMU traces often remain hidden.
K DISTINGUISHING UNLEARNING TRACES ALONGSIDE SOURCE MODEL TYPES
We extend our response-based analysis to a more complex 8-way classification task that jointly distinguishes among four LLM families, each in both their original and unlearned forms. This setup enables a more fine-grained examination of model-specific unlearning traces. Fig. 8 displays the resulting confusion matrices on both forget-related (WMDP) and forget-irrelevant (MMLU) test sets. On WMDP, predictions are highly concentrated along the diagonal, indicating strong agreement between the predicted and true model-unlearning pairs. This confirms that unlearning traces are clearly detectable when test prompts align with the domain of the forgotten content. In contrast, classification accuracy declines on the MMLU test set, particularly for the Zephyr-7B models, where most errors involve confusion between the original and -MU-unleamed versions. Nevertheless, larger models such as Yi-34B and Yi-34B-Bo RMU maintain high accuracy, suggesting that unlearning traces in these models persist and remain detectable even when evaluated on general, forget-irrelevant prompts. Results for NPO-unleamed models under this multi-class setting are reported in Fig. A5.
25

---

Published as a conference paper at ICLR 2026
Zephyr-78
94%
0%
1%
0%
2%
0%
3%
0%
Zephyn-7
96%
0%
1%
0%
1%
0%
1%
0%
100%
Zephy--78-RMU-
00%
0%
0%
0%
0%
0%
Zephyy-78-RMM
0%
100%
0%
0%
0%
0%
0%
90%
80%
1LaMA-3.1-88
93%
0%
Prediction (R)
1LaMA-3.14 1-88
2%
0%
11
Prudiction GTO 1S0
LLaMA-3.1-880 +88 RMU
0%
0%
0%
100%
0%
0%
0%
0%
LLaMA-3.1 1.1-00-.--0----010 RMU
0%
0%
0%
000%
0%
0%
0%
0%
Qwen2.55 14B
1%
0%
3%
0%
35%
0%
1%
0%
40%
Qwenz.5
14B
1%
0%
5%
0%
94%
0%
0%
0%
40%
Qwen2.5-14B-RMU
Qwen2255-14BBMM
0%
Y1-38-
5%
0%
1%
0%
4%
0%
90%
0%
YI-348
1%
0%
0%
0%
2%
97%
0%
Y--34B-MU
0%
0%
0%
0%
0%
0%
0%
100%
0%
Yi-346-RMUU
0%
0%
0%
0%
0%
0%
0%
000%
0%
Zephye-7
Zephyn 7B-RMU
LLatt 1.3---.1. 1.88
LaatA- 11-38--0 PMU
Owen2 5:148
Owen 5, 5-148 mea
W--40
W-34D- RBU
Zephyeehh 7B
Zunhyn 7B-RIU
LLab 3 1-00
Laate 3 1-88-MM
Owem 5-148
Owen 5 148 not
(a) forget-relevant test sets (WMDP)
(b) forget- irrelevant test sets (MMLU)
Figure AS: Confusion matrix for unlearnegnn pair classification. Rows indicate true classes (original/NPOO unlearned model variants), and columas show predicted classes. Diagonal entries represent correct predictions; off-diagonals indicate misclassification rates under (a) WMDP and (b) MMLU test sets.
L ADDITIONAL RESULTS ON MIXING RATIOS
Since NPO shows strong generalization even when trained only with forget- irrelevant queries (as shown in Appendix I), we focus on classifiers built on Ilama- 3.1-8B for detecting RMU under differ- ent mixing ratios (25%/75%, 50%/50%, 75%/25%) The response-based results are reported in Table A15. Based on the results, varying the mixing ratio be- tween rerget-relevant and forget-irrelevant samples does not produce any noticeable change in detection performance across WMDP, MMLU, and UltraChat. This indi- cates that as long as the training split stays within a reasonable range, the unlearning trace remains consistently learnable, mean- ing the classifier does not rely on a frag- ile or overly specific mixture proportion succeed.
Table A15: Detection accuracy of Llama3. 1-8B RMU un- learning under different mixing percentages of forget relevant and forget irrelevant queries. The top section reports detection accuracy using classifiers trained on text-based responses, and the bottom section reports detection accuracy using classifiers trained on pre-logit activations.
(a) Text based responses
Mix Percentage
WMDP
MMLU
UltraChat
25% / 75%
91.43%
81.23%
70.54%
50% / 50%
93.24%
78.87%
67.60%
75% 25%
94.37%
79.48%
69.32%
(b) Pre logit activations
Mix Percentage
WMDP
MMLJU
UltraChat
25% / 75%
99.86%
99.72%
98.59%
50% 50%
100%
99.72%
99.44%
75% 25%
99.72%
98.87%
99.15%
M ADDITIONAL EVALUATION RESULTS
To assess the robustness of unlearning-traa detection under sampling variability and paraphrasing, we performed two additional test-time analyses using the Llama-3.1-BB model. Notably, we used the same trained classifier from our original setup; only the test-time response generation was modified. First, we introduce the Pass @ K evaluation, where responses are generated with temperature 0.5 (top-p= 1.0, top-k= = -1.0), and prediction is counted as correct if any of the K sample responses is classified correctly. Second, we performed Paraphrase evaluation by paraphrasing each response using Owen2.5-32BBInnsee
with strict semantics-preserving prompt: "Paraphrase the following text while strictly preserving its original semantic meaning. Do not add or remove any information." " We reported the detection accuracy of NPO and RMU in Table A 16, respectively. NPO remains highly detectable across all conditions, indicating strong robustness to sampling noise and paraphrasing. RMU also shows consistent improvements under Pass @ K and maintains similar accuracy under paraphrasing. Thus, unlcarning traces persist across decoding strategies and semantic rewordings, confirming that the findings are not fragile or prompt-specific.
26

---

Published as a conference paper at ICLR 2026
Table A16: Response-based detection accuracy of Llama3. under different evaluation strategies. The evaluation strategies are defined as follows: "Original" denotes standard single response evalua- tion. "Pass@3" and "Pass @ 5" denote evaluating the best prediction among three five sampled
responses. "Paraphrase" denotes evaluating on semantically rewritten versions of the original queries.
Evaluation
WMDP (NPO)
WMDP (RMU)
MMLU (NPO)
MMLU (RMU)
UltraCCat (NPO)
UltraChat (RMU)
Original
100%
93.24%
99.72%
78.87%
99.72%
67.60%
P
99.86%
96.65%
99.93%
82.54%
99.93%
73.43%
P
97.86%
99.93%
84.63%
99.93%
74.61
Paraphrase
98.59%
92.87%
98.31
18%
99.15%
67.44%
N LIMITATIONS AND BROAD IMPACT
N.1 LIMITATIONS
While our study reveals consistent unlearning traces across multiple LLM families, unlearning methods, and datasets, several limitations remain. First, restricted by computational resources, is unclear whether the same degree of trace persistence holds for even larger LLMs. Second, our analysis limited to text-only LLMs. remains an open question whether similar unlearning traces persist in multi-monl molti-modal foundation models (Zhang et al., 2024g) that jointly model text with vision, speech, or other modalities (Zhang et al., 2023). Cross-modal alignment and shared latent representations may alter how traces propagate remain detectable across modalities. (Zhang et al., 2025) Finally, we do not examine domain -pemai--peeiillzeed models (e.g., biomedical or legal LLMs), where training distributions may amplify suppress residual traces.
N.2 BROAD IMPACT
This work exposes new risk in machine unlearning: the detectability of unlearning traces. While unlearning aims to enhance safety, privacy, and compliance, such traces create opportunities for adversarial exploitation, reducing the cost of targeted relearning jailbreaking in high-stakes domains like biosecurity and cybersecurity. At the same time, our findings offer constructive insights, understanding that unlearning traces guide the development of more robust algorithms, stronger evaluation protocols, and clearer regulatory standards. We hope this work stimulates further research toward unlearning methods that reliably remove sensitive knowledge without leaving exploitable fingerprints.
THE USE OF LLMs O
This work makes limited use LLMs. Specifically, LLMs were employed exclusively for grammar correction and stylistic polishing of the manuseript. They not involved in research ideation, experimental design, data analysis, or the generation of any scientific content. All substantive contributions the paper are solely attributable to the author.
27


Reviews:
[{"review_id": "A9BzDDOpVE", "reviewer": "Reviewer_H79h", "summary": "The paper presents an empirical study on post-unlearning trace detection based on the model's output. Specifically, the paper shows that even a simple supervised classifier (a trained MLP) can determine whether a model has undergone unlearning, using only its pre-logit activations or its textual outputs. The paper claims that these traces are embedded in intermediate activations and propagate nonlinearly to the final layer, forming low-dimensional, learnable manifolds in activation space. The paper demonstrates that unlearning traces can be detected with over 90% accuracy under forget-irrelevant inputs, and that larger LLMs exhibit stronger detectability.", "questions": "+ MUSE [5] is also a representative unlearning dataset that adopts an open-ended evaluation format (WMDP uses multiple-choice QA). Is there a specific reason why MUSE was not included in the experiments?\n+ Would detection performance improve if a more complex or higher-capacity classifier architecture were used?\n+ How does varying the ratio between forget-relevant and forget-irrelevant samples affect the detection results?\n+ What is the correlation between the control scaling factor $c$ and detection performance? In other words, how does detection accuracy change as $c$ increases? Could the low detection performance of Zephyr and LLaMA on MMLU, as reported in Figure 3, be explained by this effect?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "+ I appreciate that the paper clearly defines the threat model, making it easier for readers to follow the setting.\n+ The proposed approach is conceptually simple yet demonstrates strong empirical effectiveness.\n+ The research question is well-motivated and interesting.", "weaknesses": "## Major Issues\n+ Several important baselines are missing, such as Gradient Ascent, SimNPO [1], adopted DPO [2][3], etc. Experimenting with only 2 representative methods (RMU and NPO) and with WMDP only reduces the generalizability of this study.\n+ The problem studied in this paper seems method-dependent. To my understanding, RMU steers the forget-representation to a random vector, i.e., randomizing the forget-representation, while NPO maximizes the loss of forget-samples. This seems obvious that the outputs of these unlearned models will be noise or randomized, making them easier to trace. A critical point arises with the adopted DPO-based unlearning, which instead maximizes the likelihood of producing refusal responses (e.g., “I don’t know”) for forget-samples. Under this setting, is the problem posed in this paper appropriate?\n+ The detection performance strongly depends on hyperparameters, such as the scaling coefficient in RMU or the retain weight. The detection results presented in the paper are based on a selected set of hyperparameters, which risks limiting the generalization of the findings.\n+ MMLU and WMDP evaluate distinct knowledge domains; therefore, I suggest stronger experimental designs would involve using the **MMLU subcategories closely aligned with WMDP domains, such as MMLU College Biology, MMLU Virology, and MMLU Computer Security, which represent truly forget-irrelevant samples**. These sub-categories better capture domain-specific retention and represent truly forget-irrelevant samples for a more meaningful detection evaluation.\n\n## Minor issues\n+ Typos: Line 443: \"Table ??\", line 709 \"Qwen2.5-7B\".\n+ Line 175: The paper states that \"and $\\mathbf{v}$ is drawn from a standard uniform distribution\"; it should be \"and each element in $\\mathbf{v}$ is drawn from a standard uniform distribution.\"\n+ Line 170-172: \"RMU enforces forgetting by mapping the intermediate representations of samples $\\mathbf{x} \\in \\mathcal{D}_f$ to random vectors\" should be \"RMU enforces forgetting by mapping the intermediate representations of samples $\\mathbf{x} \\in \\mathcal{D}_f$ to a *fixed, predetermined* random vector.\" (e.g., WMDP-bio and WMDP-cyber each use one fixed random vector, c.f. [4])\n+ It would be nice if the paper defined the dimension of the latent representation, such as $M_{\\theta}(\\mathbf{x})$.\n\n## References\n\n[1] Fan, Chongyu, et al. \"Simplicity prevails: Rethinking negative preference optimization for llm unlearning.\" ICML, 2025.\n\n[2] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024.\n\n[3] Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. In The Thirteenth International Conference on Learning Representations, 2025b.\n\n[4] Li, Nathaniel, et al. \"The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning.\" Forty-first International Conference on Machine Learning.\n\n[5] Shi, Weijia, et al. \"MUSE: Machine Unlearning Six-Way Evaluation for Language Models.\" The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uNbE5INPVv", "reviewer": "Reviewer_WKcC", "summary": "This paper investigates a new vulnerability in large language model unlearning: even after forgetting specific knowledge, unlearned models leave behind detectable “fingerprints.” The authors show that a simple supervised classifier can distinguish an unlearned model from its original version based solely on output behavior, including logits and textual responses to forget-irrelevant prompts. The work covers both RMU and NPO unlearning methods and evaluates models of different scales, demonstrating that unlearning trace detectability persists widely. The authors further localize these fingerprints to low-dimensional structures in internal activations, arguing that this poses security and privacy risks for safety-critical deployments.", "questions": "The paper implies that trace detection could guide attackers toward targeting specific models for relearning or jailbreak attacks. Is there any quantitative experiment showing that adversaries can exploit trace detection to substantially reduce attack cost or increase success rate? What prevents the classifier from learning spurious cues rather than genuine unlearning fingerprints (for example, decline in fluency for NPO models)? The results for RMU rely heavily on access to internal activations. How do the authors justify calling the overall vulnerability “practical” when the black-box case remains borderline? It would be helpful if the authors could further analyze model sampling randomness and prompting conditions: do these traces remain detectable under different decoding strategies, temperature, or paraphrased questions? Finally, the threat model should be formalized more explicitly: what are the exact attacker capabilities, and which findings hold under the strictest constraints?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic is timely and highly relevant, given growing attention to machine unlearning in LLM safety and compliance. The idea of “unlearning trace detection” feels original and fits naturally with recent concerns about reverse-engineering vulnerabilities. The authors perform extensive experiments across multiple model scales, unlearning methods, and datasets, which lends credibility to their claim that detectable traces persist even when prompts are unrelated to forget targets. The activation-space analysis provides an interesting interpretability angle that helps explain observed classifier performance rather than just reporting empirical results. The potential implications for privacy and model governance are significant, making the paper’s message important for the community.", "weaknesses": "While the paper positions itself as identifying a “security vulnerability,” the actual adversarial threat model lacks rigor. The evaluation assumes access to pre-logit activations in open-weight settings, which is unrealistic for many practical deployments, and the black-box-only scenario (text output) results are meaningfully weaker for RMU. The paper does not show a concrete way to exploit these traces to meaningfully recover forgotten information, even though this is suggested early as a concern. The analysis focuses on simple supervised classifiers without exploring more robust baselines, so it is unclear how fragile these findings are under model sampling variability or prompt diversity. Finally, the paper’s writing is at times repetitive and could more clearly differentiate what is novel versus what follows from known model-identity classification work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fKNhBzvgyM", "reviewer": "Reviewer_Vq6t", "summary": "This paper introduces and formalizes the problem of unlearning trace detection, investigating whether it is possible to determine if a Large Language Model (LLM) has undergone an unlearning procedure based on its outputs. The authors demonstrate that unlearning, intended to remove information, paradoxically leaves behind detectable \"fingerprints.\" They frame detection as a supervised classification task and show that a simple classifier can distinguish an unlearned model from its original counterpart with high accuracy. They comprehensively evaluate the found phenomenon and the effectiveness of the proposed detection method across four different LLMs and two state-of-the-art unlearning methods.", "questions": "- The training of the detector requires access to outputs from both the original and unlearned models. Could the authors clarify the real-world scenario you envision for an adversary to acquire this training data?\n- Experimental results show that NPO leaves much more obvious traces than RMU, suggesting that the invisibility of unlearning is highly algorithm-dependent. Based on your analysis, what properties of an unlearning algorithm might lead to more \"stealthy\" unlearning that evades detection?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper pioneers the formal study of unlearning trace detection, a timely and critical problem as unlearning becomes integral to LLM safety and privacy. The threat model is well-motivated: an adversary who can identify an unlearned model can more efficiently allocate resources to attack it, undermining the very purpose of unlearning.\n2. The experimental validation spans four modern LLMs of varying scales (7B to 34B), two distinct and state-of-the-art unlearning algorithms (representation-based RMU and preference-based NPO), and a diverse set of prompts (both forget-relevant and forget-irrelevant).\n3. The use of SVD and UMAP to visualize distributional shifts in activation space provides clear, intuitive evidence for why activation-based classifiers are so effective, linking the empirical results directly to the model's internal mechanisms.", "weaknesses": "1. The proposed supervised detection approach requires the adversary to possess a labeled training dataset of outputs from both the original and the unlearned models. This is a very strong prerequisite, as it is unclear how an adversary would realistically obtain such paired models to train their detector.\n2. The most effective detection method presented relies on gray-box access to the model's pre-logit activations. While plausible for open-weight models, this assumption does not hold for the many powerful LLMs available only through black-box APIs.\n3. Experimental results show that NPO leaves much more obvious traces than RMU, suggesting that the invisibility of unlearning is highly algorithm-dependent. But the authors only evaluate two kinds of unlearning paradigms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kBinD4mK9F", "reviewer": "Reviewer_YGLw", "summary": "This paper introduces and investigates a novel vulnerability in large language models termed unlearning trace detection. The authors posit that machine unlearning processes leave behind detectable fingerprints in the model's behavior and internal representations. They propose a simple supervised classifier to reliably distinguish an unlearned model from its original counterpart, often with over 90% accuracy. Extensive experiments on four different LLMs and two state-of-the-art unlearning methods (RMU and NPO) demonstrate the generalization of their key finding and the effectiveness of the classifier-based detection method.", "questions": "- Could you hypothesize how your detection framework would perform against other classes of unlearning algorithms?\n- Your results show that text-based detection for RMU is less effective on smaller models like Zephyr-7B but improves with scale. Is this an inherent property of the unlearning traces in smaller models being weaker, or could it be a limitation of the specific text-based classifier used (LLM2Vec + MLP)? Have you explored whether more powerful text classifiers or different feature extraction methods could close this performance gap?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is the first to formalize and systematically study the problem of unlearning trace detection. It identifies a critical and practical vulnerability in the context of machine unlearning. \n2. The paper excels not just in demonstrating that unlearning is detectable, but in explaining why. The spectral analysis in Section 5 is a standout contribution, pinpointing the \"fingerprints\" in intermediate-layer activations for RMU and showing their non-linear propagation to the final layer.\n3. The claims are substantiated by a strong empirical study. The authors evaluate their hypothesis across four modern, instruction-tuned LLMs of varying sizes, two distinct and state-of-the-art unlearning methods (RMU and NPO), and multiple prompt datasets. The consistent results across these diverse settings strongly support the generality of the findings.", "weaknesses": "1. The study primarily focuses on two optimization-based unlearning methods, RMU and NPO. While these are representative, the field of LLM unlearning includes other paradigms, such as gradient ascent-based methods and model editing techniques (e.g., task vector subtraction or knowledge editing with methods like ROME/MEMIT). It is unclear if the findings generalize to these other approaches\n2. The text-only detection is shown to be significantly less reliable for RMU on smaller models than pre-logits-based detection. The implications of the access on the logits requirement could be discussed more thoroughly, especially in the context of the evolving landscape of model access.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)