The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning
Authors: Raj Sanjay Shah
Jing Huang
Keerthiram Murugesan
Nathalie Baracaldo
Diyi Yang
Venue: COLM 2025 (2025)
Keywords: Unlearning evaluation
Multi-hop reasoning
TL;DR: A Dynamic Framework for Evaluating LLM Unlearning

Abstract: Unlearning in Large Language Models (LLMs) aims to enhance safety, mitigate biases, and comply with legal mandates, such as the right to be forgotten. However, existing unlearning methods are brittle: minor query modifications, such as multi-hop reasoning and entity aliasing, can recover supposedly forgotten information. As a result, current evaluation metrics often create an illusion of effectiveness, failing to detect these vulnerabilities due to reliance on static, unstructured benchmarks. We propose a dynamic framework that stress tests unlearning robustness using complex structured queries. Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains, allowing precise control over query difficulty. Our experiments show that the framework: (1) shows comparable coverage to existing benchmarks by automatically generating semantically equivalent Q&A probes, (2) aligns with prior evaluations, and (3) uncovers new unlearning failures missed by other benchmarks, particularly in multi-hop settings. Furthermore, activation analyses show that single-hop queries typically follow dominant computation pathways, which are more likely to be disrupted by unlearning methods. In contrast, multi-hop queries tend to use alternative pathways that often remain intact, explaining the brittleness of unlearning techniques in multi-hop settings. Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets, enabling easier adoption for real-world applications. We release the pip package and the code at https://sites.google.com/view/unlearningmirage/home.

Published as a conference paper at COLM 2025
The Unlearning Mirage: A Dynamic Framework for Evaluat- ing LLM Unlearning
Raj Sanjay Shah Georgia Institute of Technology
Jing Huang Stanford University
Keerthiram Murugesan IBM Research
Nathalie Baracaldo IBM Research
Diyi Yang
Stanford University
Abstract
Unlearning in Large Language Models (LLMs) aims to enhance safety, mitigate biases, and comply with legal mandates, such as the right to be forgotten. However, existing unlearning methods are brittle: minor query modifications, such as multi-hop reasoning and entity aliasing, can recover supposedly forgotten information. As a result, current evaluation metrics often create an illusion of effectiveness, failing to detect these vulnerabilities due reliance on static, unstructured benchmarks. We propose a dynamic framework that stress tests unlearning robustness using complex structured queries. Our approach first elicits knowledge from the target model (pre- unlearning) and constructs targeted probes, ranging from simple queries multi-hop chains, allowing precise control over query difficulty. Our experiments show that the framework (1) shows comparable coverage to existing benchmarks by automatically generating semantically equivalent Q&A probes, (2) aligns with prior evaluations, and (3) uncovers new un- learning failures missed by other benchmarks, particularly in multi-hop settings. Furthermore, activation analyses show that single-hop queries typically follow dominant computation pathways, which are more likely be disrupted by unlearning methods. In contrast, multi-hop queries tend to use alternative pathways that often remain intact, explaining the brittleness of unlearning techniques in multi-hop settings. Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets, enabling easier adop- tion for real-world applications. We release the pip package and the code at https: //sites, google. www/wwww///ccccctcott..cc..... www./viwwwunlearnnngn... ragg/mimaae///mmme
1 Introduction
Selective unlearning in Large Language Models (LLMs) is an important capability for model safety (Liu et al., 2023), fairness (Gallegos et al., 2024), and legal compliance (Yao et al., 2025). As LLMs are integrated into real-world applications, removing specific knowledge, such as harmful, biased, or private information, has become important (Li et al., 2024; Ashuach et al., 2024). Regulatory frameworks such the General Data Protection Regulation (GDPR) or California Consumer Privacy Act (CCPA) may enforce the "right to be forgotten," " necessitating that LLMs comply with user requests for data removal (Rosen, 2011; Zhang et al., 2024a). Consequently, model owners must develop mechanisms to erase specific data while preserving the model's general capabilities (Liu et al., 2025).
Despite progress in unlearning methods, typically involving gradient reversal localized weight updates (Jang et al., 2022; Eldan Russinovich, 2023; Lee et al., 2024; Zhang et al., 2024b), the distributed and redundant nature of knowledge representation in LLMs makes targeted forgetting difficult. Recent studies show that even after unlearning, models retain
Emails: (mmiils(rajsaajjyyshh) heansanjjayyaa@@@gtteeh.... @gatech.edu, (hij, diyiy) @stanford.edu, ( KKeerthiram.Murugeee baracald (eracald)@ibb..om

---

Published as a conference paper at COLM 2025
Our Evaluation Framework
Target Model
Forget Stephen King
-ginge Hopp
shrking
-Snng Hasp Name the outhor populrtty known the King of Homor
LLM
Unlearming Operation
LLM O
protagenat
Evaluate
namnd Jack Tomance
Score
-Twe Hep Name the author of the host wheee artagoeest
named Perrgwise.
Graph
20 Knowledge
Queries
kaston the winser canmaloer the Over took horel.
Three Hop: Name astnor book whose protagorists
Knowledge graph creatlon expansion using BFS
N-Hop
Degth 0
Depth:
Death 2
Single Hop. Name the author of the book "Th Sniring".
C
Jack
ILM
wrote
Shining
wrote
Shining
Torrance
Single Hop: Name author papularty as the King of Horror
Stepher King_
Stephen
lives in
author
Stephen King
lives in
author
Hop: Name the author book protagenist was namied Jack Torrance.
King
Maine
relevant
Maine
JAI Flistation (BI Atomic rel. creation
book
(A) Elicitation
(8) Atomic rel. creation
(C) Alias Resolution
antagoniat namndd Pernywiss
Who
Suppeer Ring (aec browe for ling
(Pepees 11 sireed
The Shiing, a writen by Stypher 4 Speniting
"Stypee
whose prosagonist known as the winter X
Stephen
caretaiker of the Overtook hotel.
Figure 1: Overview of our framework: Our evaluation framework constructs a knowledge graph from pre-unlearning model outputs, enabling the automatic generation of structured single-hop, multi-hop, and alias-based queries. After applying unlearning, we probe the model assess residual knowledge. The framework is dynamic, instantiable for any entity, and structured, providing fine-grained control over query complexity.
subtle traces of the supposedly erased information (Lynch et al., 2024; Thaker et al., 2024). A major challenge lies in evaluating unlearning. Existing benchmarks primarily on simple retrieval tasks and static Q&A datasets, which often fail to detect residual knowledge when queries are rephrased, aliased, or composed into multi-hop reasoning chains (see figure 2; Maini et al. (2024); Choi et al. (2024); Jin et al. (2025)). As a result, current metrics can create a misleading impression of unlearning success, often missing failure modes. We argue that robust unlearning requires a more systematic evaluation metric, one that explores residual knowledge through structured variations in query form and reasoning depth.
To address these shortcomings in existing eval- uation methods, we introduce dynamic evalu- ation framework that stress tests unlearning us- ing structured, model-informed probes. Unlike previous approaches that rely on manually con- structed datasets or external commercial LLMs like GPT-4 to generate probes, our approach elic- its knowledge directly from the model before un- learning, capturing what the model initially knew about the target entity. This extracted knowledge is then used to generate probes of varying com- plexity, ranging from simple single-hop retrievals to multi-hop reasoning chains, allowing to pre- cisely control query difficulty and evaluate how well different unlearning methods prevent access to residual information.
Goal: Forget Stephen King
Single Query Yrmme
Provious metries potneriing wodke!
Mati Hop Query
Did the method actually work?
Name surooe clire bocc whos purt was
Figure 2: Limitations of existing "single- hop" evaluation metrics in assessing LLM unlearning robustness. Single- hop queries might suggest successful forgetting, but minor variations, such as multi-hop reasoning or entity alias- ing, can still recover the supposedly for- gotten information.
Central our approach is the use of knowledge graphs, which we dynamically construct for any given entity through a breadth-first querying process over the target model's s inter- nal knowledge. By recursively querying the model about the entity, its related concepts, attributes, and relationships, we generate a structured view of its per-unlearning knowl- edge (refer section 4.1). Using this graph, we generate a variety of queries, from single-hop queries, such as Name the author of the book "The Shining", (Answer: Stephen King) to multi-hop queries, like Name the author of the book whose protagonist was named Jack Torrance, (Answer: Stephen King) as well as alternative phrasings using aliases ( (e.g., Stephen Edwin King). Thus, our evaluation process is structured, as it captures semantic relationships and dynamic, as it can be automatically constructed for any target entity without manual data curation.
2

---

Published as a conference paper at COLM 2025
We evaluate our framework several unlearning methods and compare them against existing benchmarks. Our results show that: (1) it achieves comparable coverage to prior datasets without requiring human annotations (e.g. ~78% of RWKU Q&A pairs), (2) it aligns with prior rankings of unlearning effectiveness across methods, and (3) it exposes new failure modes, particularly in multi-hop and alias-based queries, that previous static evaluations overlook. Finally, we analyze model activations using PatchScopes (Ghandeharioun et al., 2024) and find that unlearning primarily disrupts dominant activation pathways used in direct queries. In contrast, multi-hop queries often route through alternate pathways that remain unaffected, explaining the brittleness of current unlearning techniques.
A visual overview of our framework is shown in Figure 1, illustrating how we extract entity- specific knowledge from the model, construct dynamic knowledge graph, and generate structured probes to evaluate unlearning robustness across varying query complexitinn
Benchmark
WHP (Elidan Russmorich 2023)
WMDP
MUSE
TOFU
RWKU
Ours
et al. 2:24)
(Shi al, 2024)
(Maini et 2024)
(Inn et al,, 2025)
Unlearning Targets * Forget Probes
200 1,000
200 13,131
Any Entity Dynamic
300
4,157
220
Fonget Corpus
Harry Potter series
PubMed, Github
Booke/ News
Syn. QA pairs
N/A
N/A
Retain Corpus
N/A
Wikitext
Fan pages/ News
Syn. QA pairs
N/A
N/A
Furget Assessment
Knowledge Memorization probes
Knowledge Manipulation probes Multi-hop Eval.
x
X
X
X
x
x
Retain Assessment
Neighbour Pe Perturbation Relationship retention
Table 1: A comparison between existing unlearning benchmarks and our benchmark. Our benchmark allows us to evaluate any entity, allowing us to automatically generate forget probes. Knowledge Memorization probes: these are cloze style probes ("Capital of France is -"); Knowledge Manipulation probes: these are MCQ style Q&A probes; O: While we do not cover Knowledge Manipulation probes, our framework can be easily modified for this probe style. "We cover a subset of adversarial attacks.
2 Related works
Evaluating unlearning in LLMs Despite advancements in unlearning methods, recent literature has identified many failure modes of unlearning methods (Thaker et al., 2024). These include catastrophic forgetting where excessive unlearning leads to unintended knowledge loss, often affecting structurally related concepts beyond the intended forget set (Zhang et al., 2024b; Yao et al., 2025), ability of an LLM to relearn through few shot tuning (Jin et al., 2025), cross-lingual and multimodal generalization failures (Si et al., 2023), and lastly, recovering unlearnt information by semantic perturbations and adversarial probing (Maini et al., 2024; Liu et al., 2025; Jin et al., 2025). Several benchmarks have been proposed to evaluate unlearning in LLMs, each focusing on aspects such as knowledge removal (Eldan & Russinovich, 2023; Li et al., 2024; Jin et al., 2025), robustness to input variation (Lynch et al., 2024), or retention capability (Maini et al., 2024; Shi et al., 2024).
While these benchmarks evaluate methods along several axes, they rely on static sets for testing data removal. There are two ways to construct the static sets: (1) Manual curation of datasets (WHP,, WMDP): These require significant human effort and are resource-intensive for example, the creation of WMDP required expert-level knowledge and cost dataset creators upwards of $200,000 (as reported by authors). (2) LLM-assisted probe generation (TOFU, MUSE, RWKU): These approaches automate test set generation using systems like GPT-4, followed by human validation or filtering. As a result, the probes may fail to capture model-specific knowledge representations. Lastly, to ensure removal does not affect other model capabilities, most benchmarks test generic tasks (like MMLU, Big-bench-hard, etc.) instead of focused evaluations on semantically close knowledge. In contrast, our framework
3

---

Published as a conference paper at COLM 2025
explicitly constructs entity-specific knowledge graphs from model-internal representations, enabling precise, targeted evaluations of semantically related knowledge. Our evaluation framework reveals failure modes not captured by existing benchmarks.
Dynamic graph-based evaluation LLMs Our evaluation methodology is closely related to the idea of building adaptive and dynamic benchmarks. Recent works have proposed graph-based approaches to dynamically assess LLM capabilities across complex reasoning and knowledge tasks Zhang et al. (2024c); Zhu et al. (2023); Feng et al. (2025). However, they focus primarily on external task graphs used for general reasoning, rather than model- specific representations. Recent literature in model editing leverages multi-hop reasoning benchmarks to evaluate the effectiveness of edits, focusing on how "fact" updates propagate through related knowledge chains Zhong et al. (2023); Cohen et al. (2024); Yang et al. (2024). In contrast, our framework constructs a knowledge graph from the model's unlearning outputs, offering structured and entity-specific snapshot of internal knowledge. This graph scaffolds the generation of semantically controlled probes that vary in reasoning depth (e.g., single-hop vs. multi-hop) and surface form (e.g., paraphrases or aliases), enabling targeted stress testing of residual knowledge. Unlike prior works that build graphs independent of the target model, our method is tightly coupled to the model's own knowledge structure, allowing for dynamic evaluation tailored to each unlearning target entity.
pre-
3 Preliminaries
3.1 Unlearning Objectives
We formalize unlearning as selectively removing the influence of specific data points from trained LLM. Given original training set D and an unlearning set Du D, aim to update the model parameters to meet two criteria removal and retention. To ground this discussion, consider running example where D, includes facts about Stephen King, like:
Dp = {("Who wrote The Shining?", "Stephen King"),
("Who is Stephen King's "Tabith" King"))-
"Tabitha
The objective is to remove the model's knowledge of Stephen King while preserving its general language capabilities and knowledge of unrelated topics.
Removal Criterion: The model should behave as though it never saw Du- Formally, for all examples in D., the updated model should be indistinguishable from a model trained without D.. If Du contains ("Who wrote The Shining?" "Stephen King"); then, after unlearning, the model should fail to answer this query correctly.
Retention Criterion: The updated model must preserve its performance on unrelated data, ie., 0(x;0*) F(x;0) for all x E D D,- Post-unlearning, the model should still answer unrelated queries correctly, e.g., "Who wrote 1984?" or "Define the term 'protagonist."
Trade-off Considerations:
These two criteria conflict; aggressive unlearning may cause unintended loss of knowledge, whereas insufficient unlearning leaves residual informa- tion. Effective unlearning strategies balance these factors to selectively remove targeted knowledge while retaining overall model capabilities.
3.2 Unlearning Methods
We evaluate popular unlearning approaches, consisting of both optimization--asse and prompting- techniques. These methods differ in how they suppress knowledge from the forget set Diorget and whether they explicitly preserve utility on the retain set Dretain- We include the following optimization-based approaches: Gradient Ascent (GA) (Jang et al., 2022), Direct Preference Optimization (DPO) (Rafailov et al., 2023), Negative Preference Optimization (NPO) (Zhang et al., 2024b), Task Vectors (TV) (Ilharco et al., 2023), Unlearning via Logit Difference (ULD) (Ji et al., 2024). We test In-Context Unlearning (ICU) (Pawelczyk et al., 2023) as the prampt-based unlearning method. To improve utility preservation
4

---

Published as a conference paper at COLM 2025
Dretain, we follow previous work (Shi et al., 2024; Maini et al., 2024) and combine GA, DPO, and NPO with two commonly used regularization techniques: (1) Gradient Descent on the Retain Set (GDR) (Maini et al., 2024), which jointly trains on D during unlearning. (2) KI. Divergence Minimization (KLR) (Zhang et al., 2024b), which constrains the unlearned model's output distribution to remain close to the original model. This results in 12 candi- date methods: GA, GAGDR, GÄKLR, DPO, GDO DPOKLR, NPO, GPO NPOKLR, ULD, Task Vector, and ICU. Full implementation details are included in appendix A.5.
4 A Dynamic Evaluation Framework for Unlearning
4.1 Knowledge Graph Construction
Understanding the knowledge an LLM encodes, and how it is retrieved, is central to our unlearning efficacy evaluation. To systematically probe "what the model encodes", we construct knowledge graph (KG) that represents factual relationships encoded by the model before unlearning. This graph serves as a structured model-specific representation of the entity and its associated knowledge. It allows to control and probe the accessibility of knowledge from a model after unlearning.
We propose three-step process for knowledge graph creation, shown in Figure 1. Our goal is to ensure the test set reflects the model's internal representations. Thus, we extract knowledge directly by querying the model for attributes, relationships, and context, without external sources.
1. Entity-Centric Extraction: Starting from a seed entity (e.g,, "Stephen King"), we elicit facts about the entity and express model responses as set of atomic triplets (et, r, e2), such as ("Stephen King", "wrote", "wroee"" "The Shining"). Triplet extraction is performed using an LLM with structured conversion prompt (appendix A.3).
2. Graph Expansion via BFS with Decay: We recursively expand the KG by querying for facts about newly discovered nodes using breadth-first search. To avoid combinatorial growth, we apply an exponential decay factor to limit the number of expansions per depth level.
3. Relevance Filtering and Alias Resolution: We filter generic or irrelevant nodes (for example, "books") to the seed entity from our expansion set. We also resolve entity aliases (e.g. "Stephen Edwin King" vs. "Stephen King") to support surface-level variation during evaluation. While identifying relevant or irrelevant nodes to the seed entity, we use the target model as a judge (Zheng et al., 2023), marking an edge it is expected to be forgotten when the seed entity is unlearned.
This process ensures that the knowledge graph is constructed on the basis of the LLM's internal representation of the entity rather than requiring external knowledge sources.
Graph Expansion We use a breadth-first search (BFS) strategy with exponential decay to expand the knowledge graph. The graph defined as a directed structure = EV,E), where nodes V represent entities or concepts, and edges E correspond to factual relations.
BFS with Decay for Expansion Control Let by be the ini- tial number direct relationships extracted from the seed entity at depth O. At depth i, the number of expanded nodes b, is given by: b, bo a'; where a is decay factor that limits the exponential growth of the graph. Thus, the total number of nodes up depth dmax is shown in equation 1. To balance exploration breadth vs. computational efficiency, we impose constraints on (1) Maximum graph depth dmaxi (2) Total node count Nootali (3) API call budget Arotal As- suming each node expansion requires k API calls, the total API usage is given in equation 2; Complexity: O(k- Nootal).
1 1 am +1
Niotal bo
(1)
a
a mmaxt +1
A = k kbb
1
(2)
5

---

Published as a conference paper at COLM 2025
Alias Resolution via LLM Calls Since LLMs may encode the same entity with different names, we incorporate alias detection to prevent redundant nodes. Given two nodes i; and 0j E V, we query the LLM: for example, "Is 'Stephen King' the same as 'Stephen Edwin King'?". If the model confirms aliasing with high confidence, we merge the nodes, keeping only one canonical representation.
Additional considerations We note that the knowledge graph needs to be constructed only once per (model, seed entity) pair. Once built, can be reused to evaluate multiple unlearning methods, making the associated API cost/ model call a one-time overhead rather than a recurring burden. Specifically, each node in the graph requires a minimum of three LLM queries: one for entity elicitation, one for extracting atomic facts, and one for alias resolution. Since the graph is expanded via a breadth-first search with an exponential decay factor (a), the number of nodes, and consequently, the number of model calls, grows sub-exponentially, as shown in equations 1 and 2. For example, under a decay factor of a = 0.8, we empirically observe that the average number of nodes per seed entity in the RWKU dataset at depths i, 2, and 3 is approximately 57.6, 103.7, and 140.5, respectively. This results in total model call count ranging from 228 to 1,942 per entity, depending on graph density, alias resolution needs, and retries due to API response exceptions.
In totality, our constraints ensure that our graph remains tractable while preserving com- pleteness, enabling unlearning evaluation across single-hop retrieval and multi-hop reason- ing chains.
4.2 Structured Probe Generation
The constructed graph allows us to represent the knowledge about an entity as a set of atomic triplets (2), Following previous work (Petroni et al., 2019), we consider fact to be retained post-unlearning if the model can correctly predict e2 given a query composed of e and r. We generate three types of probes: conventional single-hop, multi-hop, and alias-based. An example of a single-hop query that targets depth-1 facts would be "Who wrote The Shining?" for the tuple (The Shining, written by, Stephen King). Similarly, to construct multi-hop queries, we traverse graph paths over chain of facts leading to an entity to brittleness to compositional reasoning (e.g., "Who wrote the book whose protagonist was Jack lorrance?"). Alias-based probes test robustness to surface form variation (e.g., "Who wrote The Shining?" "Stephen Edwin King" " instead of "Stephen King"). The exact prompts to "hop" over the constructed graph to construct probes are given in appendix A.3. We randomly sample 100 "searches" for each kind to compute scores.
To ensure evaluation reliability, we only probe the post-unlearning model if the pre- unlearning model can correctly answer it, verifying that the fact can be retrieved from the target model, which follows previous approaches for unlearning evaluation (Jin et al., 2025). Additionally, we assess retention beyond the target entity by probing the model's ability to answer questions about related facts and popular relations. For the former, we identify facts that are 1-hop and 2-hops away from forgotten facts and use them to test whether knowledge suppression propagates to semantically nearby concepts. For example, suppose the unlearning target is the entity, Stephen King. In that case, we probe the model with facts that are related but distinct, such as: (The Shining, prolagonist, Jack Torrance) (1-hop awayy) and (Jack Torrance, occupation, writer) (2-hop away). This allows us to quantify the unintended effects of unlearning on related but non-targeted entities. For the latter, we sample high-frequency relations from the graph (e.g., lives in, has spouse, is a) and evaluate whether the model continues to answer these correctly for unrelated or distant entities (for instance, Who is the spouse of Jack Torrance?). Together, these evaluations allow us to measure both unintended forgetting and the model's ability to retain general relational knowledge following unlearning.
4.3 Evaluation Protocol
After constructing the knowledge graph and constructing probes, we evaluate unlearning efficacy in the following manner:
6

---

Published as a conference paper at COLM 2025
Multi-hop Forgetting Score (Avg. Multi-hop): We define the removal effectiveness as the average accuracy across multi-hop queries ( 1 Accuracyn-hupp A lower score indicates more effective removal of targeted knowledge. We choose = 3 to limit computa- tional overhead and ensure benchmark accessibility.
Accuracy + Accuracy2-hop Accuracy h-h
Avg. Multi-hop
Retention Score We define the Aug. Retention Score as the average accuracy across 1-hop fact retention, 2-hop fact retention, and relationship retention queries. A higher Aug. Retention Score indicates better preservation of related or unrelated knowledge.
Accuracy 1-hop retention
Accuuaccy-2au 2-hop retention
Accuracy rel. retention
Avg. Retention Score
3
Overall score: To succinctly summarize the trade-off between effective knowledge removal and retention, we propose a combined harmonic mean score between (1 - -Ay. Multi-hop) and Avg. Retention Score This metric penalizes methods that either insufficiently erase targeted knowledge or overly disrupt unrelated knowledge.
Experiments
We benchmark unlearning methods using the proposed dynamic framework and compare our results with existing unlearning benchmarks. We find that (1) our dynamic evaluation framework has comparable coverage to existing benchmarks by automatically generating semantically equivalent probes, (2) our benchmark method produces rankings that are comparable with existing benchmarks, and (3) we uncover new unlearning failure modes, particularly in multi-hop settings.
5.1
Setup
We evaluate various unlearning methods using our framework the entities present in the RWKU and TOFU benchmarks, respectively, using the LLaMA-3.1-Istrruct (8B) model. Our choice of research (Bhaila et al., 2024; Shi et al., 2024; Maini et al., 2024; Jin et al., 2025), providing consistent basis for comparison across different evaluation strategies.
is driven by its widespread use in existing unlearning
5.2 Results
Our automatically constructed benchmark has comparable query coverage with existing benchmarks. To validate the generality of our framework, we first measure its coverage against existing entity-centric unlearning benchmarks. Our structured probe generation recovers approximately 78% of RWKU and 66% of TOFU queries without using benchmark templates or external corpora. This demonstrates that our method captures substantial por- tion of established benchmark content. The full methodology is provided in appendix A.6.
We choose = 3 to limit computational overhead.
Our metric shows the same relative efficacy of methods as previous unlearning evaluation methods. Results for RWKU are summarized in Table 2, showing each unlearning method's performance across the multi-hop forgetting criterion and the retention criterion. Additional results on TOFU can be found in the Appendix.
Despite our evaluation requiring manual annotation external knowledge sources, we successfully captured relative differences between methods. We calculate Spearman's rank correlation between previously used metrics and our evaluations and see significant correlation between both criteria (Removal Criteria, Spearman's rank correlation: RWKU 0.87*** TOFU (-) 0.79*** Retention Criteria, Spearman's 's rank correlation RWKU
7

---

Published as a conference paper at COLM 2025
Multi-hop Queries. 2-hop
Retention criteriat 2-facts away
Multi-hop Forget Score
Avg. Retain
Overall
Method
1-hop
3-hop
1-fact away
Rel. Ret.
Score
Target model
98.6
97.2
84.1
98.9
98.1
99.1
98.7
12.5
ICL
14.7
19.2
28.5
34.2
52.5
93.4
20.8
60.0
68.3
GA
19.3
23.8
31.2
44.6
59.3
55.5
24.8
53.1
62.3
GDR
21.8
25.7
32.5
73.8
70.5
76.2
26.7
73.5
73.4
GAKLR
22.3
26.2
33.0
74.5
71.2
76,4
27.2
74.0
73.4
DPO
22.1
30.9
34.6
49.7
58.4
58.4
29.2
55.5
62.2
DPOGDR
25.2
32.5
35.8
65.1
67.8
79.6
31.2
70.8
69.8
DPOKLR
26.4
32.7
36.1
65.8
68.4
80.2
31.7
71.5
69.8
NPO
16.2
22.9
30.7
47.1
59.9
60.5
23.3
55.8
64.6
NPOGDR
16.3
24.8
31.9
65.3
71.1
81.4
24.3
72.6
74.1
NPOKLR
17.8
22.3
31.4
69.6
72.5
82.3
23.8
74.8
75.5
ULD
11.2
18.7
28.1
74.2
78.8
86.1
19.3
79.7
80.2
TV
28.3
44.5
54.1
772
81.7
87.9
42.3
82.3
67.8
Avg
20.1
27.0
33,9
61.7
67.7
76.5
27.0
68.7
70.7
Table 2: Scores from our evaluation metric instantiated with the seed entities in RWKU for Llama 3.1-Instruct (8B). Values indicate means higher is better, and J means lower is better. Methods as described in section 3.2
0.75 TOFU = 0,58 "*p 0.01, *** 0.005). In addition to LLaMa 3.1, we also test our framework on Phi-4-mmni-instruct (3.8B) and Saanit-3.2-BB-Instrucc on RWKU; see tables 4 and 5 in the Appendix. While the model generally achieves higher residual knowledge retrieval scores for multi-hop queries compared to LLaMA 3.1 (8B), we see similar relative efficacy scores for different unlearning methods (Spearman's rank correlation RWKU: Removal Criteria 0.88*** Retention Criteria = 0.77 ""p 0.005).
In terms of relative differences between methods, general relationship knowledge effectively Rel. Ret. RWKU=93.47 TOFU=87.2%% but shows a substantial decline the ability to retain facts close (1-hop away) from the targeted unlearning entities (RWKU: 34.2%; TOFU: 31.5%). Optimization-based methods, i.e., GA, DPO, NPO, have substantial retention performance drops when applied without regulariza- tion. However, these methods improve with regularization on retention (increasing average retention score by approximately 18 to 20% for RWKU and 10 15% for TOFU), showing the advantages of explicit regularization strategies. Among all methods, ULD presents the optimal balance between effective forgetting and retaining general knowledge, achieving the highest overall score (RWKU: 80.2%, TOFU: 78.5%).
benchmark shows that ICL retains
Multi-hop queries expose new failure modes. Multi-hop queries consistently succeed in finding residual knowledge. Averaged across all methods, multi-hop query accuracy remains notably high (1-hop: 20.1%, 2-hop: 27.0%, and 3-hop: 33.9%, highlighted in red, table 2). Furthermore, the evidence of residual information increases with query complexity, from single-hop to multi-hop, indicating that compositional queries are adversarial to unlearning methods.
Moreover, we find that aliasing further exacerbates residual knowledge recovery, and de- composing queries via thain-of-thought does not prevent recovering residual knowledge. Table 3 shows residual knowledge retrieval on 2-hop queries under different evaluation settings to test surface-level perturbations for RWKU. Aliasing intermediate entities in two-hop queries leads to an additional average increase of 2.4% in residual knowledge re- covery, highlighting vulnerabilities minor surface perturbations. Decomposing multi-hop queries step-by-step via few-shot examples showed negligible improvement in unlearning effectiveness. Models displayed similar residual knowledge regardless of whether queries were parsed in Chain-of-thoughe manner (Nguyen et al., 2024).
Moreover, proximity to the unlearning target shows a drop unintended forgetting scores. Retention performance decreases for facts directly adjacent to the unlearning target (Avg. fact retention for facts that are one hop away RWKU: 61.7%; TOFU: 61.7% ), with accuracy improving distance increases (2-hop away RWKU: 67.7%; TOFU: 66.9%) This confirms that proximity to the target entity in the knowledge graph is predictive of unintended knowledge removal. This score is akin to neighbor set scores in RWKU ( et al., 2025).
8

---

Published as a conference paper at COLM 2025
Single-Hop Queries
10.0
Singe-Hop
Queries) Decompositin
Frequan
Method
Default
Allasing
Target model
972
96.7
97.
5.0
JCL
192
19.9
22.5
GA
25.8
22.4
76.2
GAqut
GDR
262
257
254
36.6
29.3
76.9
0.0
7
23
15
17
19
21
23
25
27
29
31
DPQ
314
32.1
32.
Layers
DPOcoR
32.5
316
344
Two-Hop Queries
DP0
NPO
2244
327
22.3
337
218
36
15
First Hop
NPO
218
25.1
26.1
Frequency
Second Hop
NPOKLN
22.3
21.5
21.1
ULD
187
19.9
20.9
Avg.
27.0
26.9
24 24%)
Table 3: For RWKU, we compare de- fault 2-hop queries with two vari- ants: (+ Decomposition) prompting the model to solve the query step-by- step, and (+ Aliasing) substituting in- termediate entities with known aliases.
Layers
Figure 3: Localizing entity resolutions in the tar- get LLM: Single-hop queries are most resolved in intermediate layers. In contrast, two-hop queries demonstrate a two-stage resolution pattern, with the first hop resolved early (layers 1-11) and the second hop resolved later (layers 12-32).
5.3 Analysis on the Multi-hop Failures
We further analyze the multi-hop failures in unlearning methods. We hypothesize that the failures are due to unlearning methods only targeting dominant pathways for single-hop entity resolutions, i.e. middle layers in transformer-based LLMs, in the gradient updates.
We analyze internal transformer-layer activations using PatchScopes (Ghandeharioun et al., 2024), which decodes hidden activations into interpretable language to precisely identify where entities are internally resolved during inference. We compare activations from single-hop query ("The author of The Shining is _..") and a two-hop query ("The author of the (book with protagonist Jack Torrance) is ".."). Single-hop entities predominantly resolve clearly in intermediate layers, enabling effective disruption by unlearning, Alternatively, two-hop queries show bifurcated resolutions: the first-hop entity ("The Shiming") resolves in early layers, while the second-hop entity ("Stephen King") resolves distinctly later (see Figure 3). This layered resolution provides candidate explanation for why current unlearning methods fail: effectively removing direct single-hop knowledge, yet struggling to eliminate indirect multi-hop knowledge.
Discussion and Limitations
Given the growing importance of unlearning in LLMs, anticipate an increased research focus on building robust evaluations and benchmarks for unlearning methodologies. Cur- rent evaluation strategies rely on manually curated, static test sets, which are hard to scale. To address this shortcoming, we advocate shifting toward dynamic evaluation frameworks that enable the automatic generation of test cases to systematically probe for evidence of successful/ / failed unlearning. Ideally, evaluation frameworks would not require the construction of fixed hold-out sets but instead generate evaluation queries dynamically and possibly adaptively. Furthermore, these evaluations should allow precise control over their complexity, including perturbations and multi-hop reasoning, enabling more rigorous stress testing of unlearning methods.
We argue that multi-hop unlearning is not a theoretical corner case but a practical require- ment. In real-world applications, users interact with LLMs through indirect, compositional, or paraphrased queries, whether via search assistants, RAG pipelines, or conversational agents. For example, rather than explicitly asking about "The Hunger Games", user might ask, Who wrote the book whose main character is Katniss Everdeen? Our experiments show that even when direct (single-hop) queries appear successfully unlearned, residual

---

Published as a conference paper at COLM 2025
knowledge often remains accessible through such multi-hop rephrased queries, revealing vulnerabilities in current unlearning techniques.
From the standpoint of users and regulators, looking at phrasing specific success is insuffi- cient; what matters is whether the sensitive or protected content is fully inaccessible. If a model can reproduce forgotten information under minor variations in question form, then the unlearning mechanism has failed its real-world obligation. Thus, we believe users, regu- lators, and stakeholders, care about outcome-level guarantees, not phrasing-specific ones. Our method reflects this risk by constructing multi-hop and alias-based probes directly from the model's 's own knowledge structure, avoiding arbitrary synthetic templates.
Despite these advantages, it is important to consider the limitations of our approach. The primary challenge lies with knowledge elicitation. While eliciting information from LLMs about well-known entities (e.g., "Tell me about Stephen King") is straightforward, eliciting knowledge in low-salience domains is tough. An example is WMDP (Li et al., 2024), where unlearning is tested on expert-level knowledge, such as novel protein compounds or cybersecurity threats, models often struggle to produce consistent outputs for complex or low-frequency information. Elicitation and Multi-hop queries, the two central ideas of our evaluation, create a paradoxical scenario (a "chicken-and-egg" problem) where we demonstrate unlearning failures through effective elicitation of residual knowledge using Multi-hop queries, yet, elicitation itself is difficult for certain kinds of information. The second limitation lies in cases where the forget and retain sets overlap significantly (e-g., MUSE-Books (Shi et al., 2024): distinguishing copyrighted material from derivative works; separating Harry Potter books from fan pages), elicitation alone becomes insufficient, and external knowledge sources or additional manual intervention is often required for accurate disambiguation for information be removed and retained. Another limitation is that the metrics derived from our methods, like any evaluation measure, only approximate true unlearning efficacy. Next, the created knowledge graph is non-deterministic, and moreover, once knowledge graph is created, it remains static for given evaluation. Future work can explore evolving evaluation paradigms and graph construction for true adversarial testing. Lastly, we use the knowledge graph to test unintended forgetting, which may not represent model utility on generic tasks.
7 Conclusion
We propose a dynamic, graph-based framework for evaluating unlearning in large language models. In contrast to prior benchmarks that rely on static, manually curated, or externally sourced queries, our approach builds structured knowledge graphs from the model's own pre-unlearning outputs and generates semantically controlled probes of varying complexity. Our experiments show that this method not only matches the coverage of existing bench- marks like RWKU and TOFU but also uncovers new failure modes, particularly through multi-hop queries that previously used static evaluations miss. One such case involves an entity where unlearning appears successful for a single-hop query but fails under multi-hop reasoning. We find that single-hop queries often align with dominant computation path- ways, which are more likely to be disrupted by unlearning interventions. By grounding the evaluation in the model's own knowledge structure, our method enables scalable, entity- specific assessments of unlearning robustness without manual curation. Our work exposes limitations in current benchmarks and, yet again, provides additional evidence challenging the completeness of forgetting guarantees.
References
Tomer Ashuach, Martin Tutek, and Yonatan Belinkov. Revs: Unlearning sensitive infor- mation in language models via rank editing in the vocabulary space. arXiv preprint arXiv:2406.09335, 2024.
Karuna Bhaila, Minh-Hao Van, and Xintao Wu. Soft prompting for unlearning in large language models. Xiv preprint ar arXim:2406.1200 2024.
10

---

Published as a conference paper at COLM 2025
Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXio preprint ar Safi::20061277777 2024.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nu- anced metrics for measuring unintended bias with real data for text classification, 2019.
Minseok Choi, ChaeHun Park, Dohyun Lee, and Jaegul Choo. Breaking chains: Unraveling the links in multi-hop knowledge unlearning. arviv preprint ar axXto:2410.13274, 2024.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the rip- ple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-296 298, 2024.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in lims. 2023.
Xiv preprint arXiv:2310002238, 2023.
Tao Feng, Yihang Sun, and Jiaxuan You. Grapheval: A lightweight graph-based I1m frame- work for idea evaluation. Xiv preprint ar ArXiv:2503.1 12600, 2025.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Compulational Linguistics, 50(3):1097-1179, 2024.
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patch- scopes: A unifying framework for inspecting hidden representations of language models. In Forty-first International Conference on Machine Learning, 2024. URL https: barxiv.org// /000s//221..2 20002040000..0 06102.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. aiv preprint arXiv:2210.01504 2022.
Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Kompella, Sijia Liu, and Shiyu Chang. Reversing the forget-retain objectives: An efficient Ilm unlearning framework from logit difference. Advances in Neural Information Processing Systems, 37:12581-12611, 2024.
Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. Advances in Neural Information Processing Systems, 37:98213- 98263, 2025.
Bryan Klimt and Yiming Yang. The enron corpus: A new dataset for email classification research. In European conference on machine learning, pp. 217-226. Springer, 2004.
Dohyun Lee, Daniel Rim, Minseok Choi, and Jaegul Choo. Protecting privacy through approximating optimal parameters for sequence unlearning in language models. arXiv preprint ar Arxi::224661444 14091, 2024.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Knn-rn Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm- Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert- Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Liu, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, lan Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula,
11

---

Published as a conference paper at COLM 2025
Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The wmdp benchmark: Measuring and reducing malicious use with unlearning, 2024.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1-14, 2025.
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. ar Xrv preprint ar arXi::2305.1333 2023.
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods evaluate robust unlearning in llms. arv preprint ar XrXiv:2402.16855, 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and Zico Kolter. Tofu: A task of fictitious unleariin for llms. ar preprint ar arXiv:2401.0612 2024.
Anmol Mekala, Vineeth Shrea, Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, and Elita Lobo. Alternate preference optimization for unlearning factual knowledge in large language models, 2024. URL 13474.
Minh-Vuony Nguyen, Linhao Luo, Fatemeh Shiri, Dinh Phung, Yuan-Fang Li, Thuy-Trang Vu, and Gholamreza Haffari. Direct evaluation of chain-of-thought in multi-hop reasoning with knowledge graphs. arv preprint ar Xrxe:2402.11199, 2024.
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language
models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Language Processing (EMNLP-IJCNNP") pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653 00/6653/v1/D1----1 v1/D19-1250. URL http:://acccaatttoooooooo///////
Natural
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Xiv preprint ar XrXto22305.18290, 2023.
Jeffrey Rosen. The right be forgotten. Stan. L. Rev. Online, 64:88, 2011.
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint ar rXtw:2407.06460 2024.
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowl- edge unlearning for 1lms: Tasks, methods, and challenges. ar Xiv preprint arXio22311.15766, 2023.
Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, and Virginia Smith. Position: Llm unlearning benchmarks are weak measures of progress. aiv preprint ar arXio:2410.02879, 2024.
Sohee Yang, Nora Kassner, Elena Gribovskaya, Sebastian Riedel, and Mor Geva. Do large language models perform latent multi-hop reasoning without exploiting shortculs? ar preprint 2rX1:92411.16679, 2024.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 17:105425-105475, 2025.
12

---

Published as a conference paper at COLM 2025
Dawen Zhang, Pamela Finckenberg Brooen,,,,,,,,,,, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right to be forgotten the era of large language models: Implications, challenges, and solutions. AI and Ethics, pp. 1-10, 2024a.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint ar drXio:2444.05868, 2024b.
Zhehao Zhang, Jiaao Chen, and Diyi Yang. Darg: Dynamic evaluation of large language models via adaptive reasoning graph. arXiv preprint arXit:2406.17271, 2024c.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonza- lez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, Т. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 46595-46623. Curran Associates, Inc,, 2023. URL https: wwwprrooceeeeiiinnieiirrs....o.e http:/pprrrieeeeeeee///// fttee97fh222o000e222222000170-15
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing language models via multi-hop questions. Xrvi preprint ar arXt::2305.1795, 2023.
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. arviv preprint ar ArXio23399.1716, 2023.
13

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Unlearning Objectives
We formalize the unlearning problem as follows. Let D be the original training set for an LLM parameterized by 0, and let D, = y1 "= C D be designated "unlearning set" of и examples whose influence we aim to remove. Unlearning methods need to satisfy both the Removal and the Retention criteria.
To ground this discussion, consider a running example where Dy includes facts about Stephen King, such as:
Du {("Who wrote The Shining?", "Stephen King"),
("Who is Stephen King's spouse?", "Tabitha King")).
The objective is to remove the model's knowledge of Stephen King while preserving its general language capabilities and knowledge of unrelated topics.
Removal Criterion A model should behave as if it never saw the unlearning data in the first place, i.e., the updated model should no longer encode or reproduce knowledge from D,- Formally, for any E D,, the output distribution should be statistically indistinguishable from that of a model trained without D,. That is:
F(x;0") R F(x;
where PDD, denotes the parameters learned by training D \ Du-
Example. After unlearning, the model should fail to answer questions like "Who wrote The Shining?" or "Who is Stephen King's wife?", just model trained without that data would.
Retention Criterion The updated model should preserve its performance unrelated data. That is, for any x E D Du, the model's output should remain close to the original: F(x;0") F(x;0).
Example. The model should still correctly answer questions such as "Who wrote 1984?" or "Define the term "protagonist."
Trade-Off Considerations In practice, these two criteria are in direct tension: stronger forgetting often leads unintended degradation in performance retained knowledge. Unlearning methods must balance these competing objectives by controlling the scope and intensity of forgetting. Aggressive interventions (e.g,, gradient ascent on D₂) may lead to unintended unlearning, where knowledge beyond D, is also lost. Conversely, conservative approaches may leave (many) residual traces of Du, making unlearning incomplete. This trade-off is central to evaluating the efficacy of unlearning methods.
LLM Unlearning aims selectively remove the knowledge and influence of specific unlearning targets from LLM, ensuring that it no longer reinforces undesired outputs while preserving its overall performance and capabilities.
14

---

Published as a conference paper at COLM 2025
A.2
Comparison to previous metrics
A.2.1 Data Preparation and Implementation
RWKU does not provide an explicit retain corpus, which makes it challenging to evaluate unlearning methods that incorporate regularization or aim to preserve surrounding knowl- edge. To address this, we construct synthetic retain corpus by leveraging the Wikipedia pages of RWKU unlearning targets. Specifically, we extract all outbound hyperlinks from each target's page and retrieve the full content of the linked pages. These linked pages represent semantically neighboring knowledge that should remain unaffected by the un- learning process. This design choice to build the retain corpus follows the pseudo-forget corpus creation process in RWKU.
Previous experiments with unlearning evaluations show model utility collapse for batch- target unlearning, i.e., simultaneously unlearning too many targets leads to collapse in any model utility. Thus, RWKU's main experiment unlearns single entity, which is resource- intensive. Therefore, inspired by the TOFU task settings and experimentation setup by Ji et al. (2024), we report an average of three runs, each with 1% of all unlearming targets for batch unlearning for both RWKU and TOFU.
Final hyperparameters: TV, (GA, DPO, NPO), Variants: The training hyperparameters are consistent across all baseline methods: the batch size 32, a learning rate of 10 5 weight decay of 0.01, and retain weight of 1. We use the AdamW optimizer with ß1 = 0.9 and $2 = 0.99. We use a consistent assistant LLM configuration for all experiments and utilize K = 8 for assistant LLM construction. Training hyperparameters for ULD are: batch size 32, the learning rate of 10-3, weight decay of 0.01, and retain weight of 6.5. At inference time, we apply greedy decoding for all unlearned LLMs, following previous work (Jin et al., 2025).
15

---

Published as a conference paper at COLM 2025
A.2.2
Real World Knowledge Unlearning
Multi-hop Queries.
Retention criteriat 2-facts away
Multi-hop Forget Score
Avg. Retain
Overall
Method
1-hop
2-hop
3-hop
awayy
Rel. Ret.
Score
Targel model
94.1
92.6
78.4
94.2
93.8
95.0
88.4
94.3
20.7
ICL
17.7
22.2
31.5
31.2
49.4
89.6
23.8
56.7
65.0
GA
21.6
27.4
34.3
42.1
56.8
52.3
27.8
50.4
59.4
GDR
24.5
29.6
35.6
70.5
67.2
72.5
29.9
70.1
70.1
GKL
24.8
30.2
36.1
71.1
68.3
73,4
30.4
70.9
70.3
DPO
24.3
33.2
37.6
46.8
55.2
55.1
31.7
52.4
59.7
DPOD
27.5
35.1
39.1
63.2
65.3
77.3
33.9
68.6
67.1
DPOKL
28.8
35.5
39.5
64.4
66.1
77.9
34.6
69.5
67.3
NPO
19.4
25.9
33.8
44.3
57.1
57.2
52.9
616
NPOD
19.8
27.1
35.2
62.7
68.8
79.5
27.4
70.3
71.5
NPOKL
21.2
26.7
35.0
67.4
69.7
80.6
27.6
72.6
72.5
ULD
15.0
22.4
32.1
72.0
76.9
84.1
23.2
77.7
77.2
TV
32.2
47.8
57.
74.6
79.3
85.8
45.8
79.9
64.5
Avg.
23.1
30.3
37.3
59.2
65.1
73.8
30.2
66.0
67.8
Table 4: Scores from evaluation metric instantiated with the seed entities in RWKU for Phi-4-mini-instruct (3.8B). Values indicate means higher is better, and 4 means lower is better. Methods as described in section 3.2
Multi-hop Queries.
Retention criteria 2-facts away
Multi-hop Forget Score
Avg. Retain
Overall Score
Method
I-hop
2-hop
3-hop
1-fact away
Rel. Ret.
Target model
80.4
97.3
97.5
97.6
15.1
ICL
13.1
17.6
27.9
35.6
50.3
91.8
59.2
68.2
GA
18.2
24.7
30.1
43.1
60.8
54.2
24.3
52.7
62.1
GDR
22.6
26.9
30.6
72.2
69.1
74.1
26.7
71.8
72.5
GKL
22,8
27.9
31.4
72.6
69.7
75.2
27.4
72.5
72.6
DPO
21.4
29.4
33.3
51.1
56.5
56.2
28.0
54.6
62.1
DPOD
24.3
33.7
34.5
63.2
69.2
81.4
30.8
71.3
70.2
DPOKL
27.5
31.9
35.0
64.7
66.5
81.8
31.5
71.0
69.7
NPO
14.4
24.1
29.6
48.7
59.3
59,2
22.7
55.7
64.8
NPOD
17.2
23.7
30.2
68.7
72.6
82.6
23.7
74.6
75.5
NPOKL
21.1
30.0
70.2
83.5
22.4
74.0
75.7
ULD
12.6
19.1
26.9
72.7
76.6
85,4
19.5
78.2
79.3
TV
26.8
45.9
52.6
75.4
79.5
89.2
41.8
81.4
67.9
Avg.
19.8
27.2
32.7
61.4
66.7
76.2
26.5
68.1
70.1
Table 3: Scores from our evaluation metric instantiated with the seed entities in RWKU for IBM Granite
Values indicate t means higher is better, and means lower is better. Methods as described section 3.2
Previous Metrics
Our Metric
Method
Fonget Set (All) Neighbor Set (All)
Multi-hp Forget Scote
Avg. Ret Score
Target model
77.3
GA
39.6
24.8
53.1
GDR
47.3
74.1
26.7
73.5
GAgLR
50.6
70.
27.2
74.0
DPQ
39,8
24.2
55.5
DPO
DPOKLR
44.5
71.5
NPO
29.8
73.3
23.3
NPOGDR
30,2
77.8
24.3
72.6
NPOxLR
31.1
7488
23.8
74,8
ULD
23,8
Spearman's Rank Corr.
0,870
0,75
Table 6: RWKU: Comparison of unlearning effectiveness using previous static metrics vs. our dynamic evaluation framework when using Llama 3.1-Instruct (8B). Spearman's rank correlation shows strong agreement with prior rankings while revealing new vulnerabilities missed by static benchmarks, Significance levels: "*p < 0.005
16

---

Published as a conference paper at COLM 2025
A.2.3 TOFU: Task of Fictitious Unlearning
Multi-hop Queries
Retention criteriat
Multi-hop Forget Score
Avg. Retain
Overall
Method
1-hop
2-hop
3-hop
1-fact away
2-facts away
Rel. Ret.
Score
Targel model
92.8
86.3
78.6
94.0
88.7
92.7
85.9
91.8
24.4
ICL
13.8
20.3
30.5
31.5
56.2
87.2
21.5
58.3
66.9
GA
20.5
25.3
28.9
47.9
62.9
59.5
24.8
56.8
64.7
GDR
23.2
24.7
30.7
67.9
74.6
70.6
25.9
71.0
72.5
GAKLR
22.0
26.3
30.6
69.7
65.6
70.5
26.3
68.6
71.1
DPO
20.6
33.3
32.1
52.5
51.9
61.9
28.7
56.4
63.0
DPOGDR
27.1
34.5
33.3
69.5
648
75.1
31.6
69.8
69.1
DPOKLR
28.3
30.2
36.9
61.5
63.1
76,2
31.8
66.9
67.6
NPO
15.1
24.1
32.7
49.6
63.9
64.2
24.0
59,2
66.6
NPOGOR
15.5
23.5
34.2
60.7
75.9
75.3
24.4
70.6
73.0
NPOKLR
16.8
21.2
33.5
63.6
69.9
77.4
23.8
20.3
73.1
ULD
11.8
20.1
30.2
78.6
73.2
81.4
20.7
77.7
78.5
TV
29.7
41.1
51.1
714
77.9
77.3
40.6
72.2
65.1
Avg.
20.6
26.8
33.7
60.4
66.9
73.1
27.0
66.6
69.7
Table 7: Our Metric the TOFU benchmark for Llama 3.1-Instruct (8B). Values indicate means higher is better, and + means lower better. Methods: GA- Gradient Ascent; GDR- Gradient Diff (Gradient ascent of forget set with gradient descent of retain set); GAKLR- Gradient ascent of forget set with KL. Divergence minimization on the retain set; DPO- Direct Preference Optimization; DPO Direct Preference Optimization with Gradient descent retention; DPOKLR- Direct Preference Optimization with KL Divergence minimization on the retain set; NPO- Negative Preference Optimization; NPOGDR- Negative Preference Optimization with Gradient descent retention; NPOKLR- Negative Preference Optimization with KL Divergence minimization the retain set; ICL Incontext Unlearning; TV Task vectors; ULD Unlearning via Logit Difference.
Metrics Model Ltility
Method
Forget Quality
Multi-hop Forget Score
Avg. Ret, Score
Target model
0.00
0.68
83.9
91.8
0.41
0.54
24.8
56.8
GDR
0,26
0.55
25.9
71.0
GAKUR
0.45
68.6
DPO
0,27
28.7
56.4
DPOcDR
0.26
0.58
31.6
69.8
DPOKLR
0.26
0.59
31.8
66.9
NPO
NPOgDR
0.59
0.57
24.4
70.6
NPOKLA
0151
0.56
23.8
70.3
ULD
77.7
0.33
0.60
40.6
72.2
Spearman's 's Rank Corz.
-0.79
0
Table 8: TOFU: Comparison unlearning effectiveness using previous static metrics vs. our dynamic evaluation framework (LLaMa 3.1 (8B)). Spearman's rank correlation shows strong agreement with prior rankings while highlighting differences missed by static benchmarks. Significance levels: **p 0.01, ***p < 0.005
17

---

Published as a conference paper at COLM 2025
A.3 Prompts used in Knowledge Graph Creation
A.3.1 Eliciting information about an entity
Generate list of diverse questions regarding the entity '(entity)". Each question should cover different aspect:
1. Basic introduction: Who is (entity)?
Key concepts related to (entity): what are the nain concepts or characteristics associated with (entity)?
3, Connections to related antities: What are the most significant relationships between ( entityy other related entities?
4. Functional roles: What is the role or importance of (entity) in its field or domain?
Lesser-knawn facts:
sone resser-known
moi-mainst details controversies or debates surrounding [entity)?
(entity)?
6. Controversies or debates: Are there
Future trends: How could (entity) evelve or influence future developments in its field? Historical significance: What has been the historical impact of (entity)?
Comparison to sinilar entities: How does (entity) compare to similar entities in the same different fields?
18. Missing information: What information is missing under-researched about (entity) that help understand better?
Input: "[entity)"
Provide the output list of questions.
A.3.2 Obtaining relationships from text
In a knowledge graph, entities represent real -world objects, concepts, or things.
Valid entities Specific and identifiable (e.a., names, places, distinct itens). Not overly abstract, repetitive, or general. knowledge graph structure.
Relevant
Extract all stanic facts from the input text.
Dutput each atomic fact in the format: (subject, relationship, object), where:
Relationships and objects
concise, meaningful, and specific.
Longer pieces of text can be broken into muitiple relationships. for fact, if applicable,
relationships (el, rl, e2) (e2, e1).
Text: "[text)"
A.3.3 Finding irrelevant facts
Rate the relevance of the following triple to the initial query on scale from to 18.
Query: "[Seed Entity)' Triple: ("(entity)" "(relation)". "(obi}") Pravide only the
response.
A.3.4 Alias Resolution
"(node)" the "(visited_node)""
A.4 Popular unlearning benchmarks
Several benchmarks have been proposed to evaluate unlearning in LLMs, each focusing on different aspects such as knowledge removal, adversarial robustness, and model retention capability. Below, we summarize key benchmarks and their evaluation methodologies.
1. Who's Harry Potter? (WHP) Benchmark Eldan & Russinovich (2023): The WHP benchmark tests unlearning on a single entity, the Harry Potter book series. The benchmark evaluates forgetting through 300 manually curated Q&A probes target- ing knowledge about the Harry Potter universe.
2. Weapons of Mass Destruction Proxy (WMDP) Benchmark et al. (2024): The WMDP benchmark simulates unlearning high-risk expert-level knowledge related to bioweapons and cybersecurity threats. The forget set consists of multiple-choice questions biology, virology, cybersecurity, and chemistry, while the retain set is
18

---

Published as a conference paper at COLM 2025
drawn from MMLU college-level question sets. Unlike WHP, WMDP includes 4,157 forget probes, allowing for a more extensive evaluation of knowledge removal.
3. TOFU Benchmark Maini et al. (2024): TOFU evaluates unlearning on fictional entities using a synthetic dataset of 4,000 Q&A pairs about fictional authors. The benchmark uses fine-tuned version of LLaMA-2-7B-chat, with the goal of un- learning subset of 1%, 5%, or 10% of the authors' information. Unlike WHP and WMDP, TOFU incorporates neighbor perturbation testing, making one of the first benchmarks to assess whether unlearning affects related entities. However, TOFU does not include adversarial attacks, knowledge memorization tests, or multi-hop reasoning, limiting its effectiveness in evaluating unlearning robustness.
4. Machine Unlearning Six-Way Evaluation (MUSE) benchmark Shi et al. (2024): The MUSE benchmark introduces a six-way evaluation framework focused on data owner and deployer expectations, including verbatim and knowledge mem- orization, privacy leakage, utility retention, scalability, and sustainability, It uses real-world corpora (e.g., news, books) and evaluates unlearning effectiveness under practical constraints.
5. Real-World Knowledge Unlearning (RWKUU Benchmark Jin et al. (2025): RWKU is the largest benchmark to date, containing 13,131 synthetic Q&A pairs about 200 real-world celebrities. Unlike previous benchmarks, RWKU incorporates adversar- ial probing techniques such as knowledge manipulation, knowledge memorization, and membership inference attacks to stress test unlearning effectiveness. Addition- ally, RWKU assesses model utility on five capabilities, including reasoning ability (measured using Big- Brg-BBnch-Harde and truthfulness (measured on TruthfulQA).
A.5 Popular unlearning methods
We evaluate various popular unlearning methods, including optimization--asel and prompt-based approaches. Several of these can be combined with regularization techniques designed to preserve model utility the retain set. This leads to a total of 12 candidate methods evaluated in our framework: GA, GAGDR, GAKLR, DPO, DPOGDR DPOKLR, NPO, NPOGDR, NPOKLR, ICL, ULD, and Task Vector.
Let faaget denote the original (target) model, Piorget the forget set, Dretain the retain set, and funleam the model after unlearning. Below, we summarize each method.
Gradient Ascent (GA) minimizes the likelihood of correct predictions on Piorget by performing gradient ascent on the cross-entropy loss (the opposite of conven- tional learning with gradient descent). GA has achieved mixed results: while Jang et al. (2022) found it effective for unlearning examples from the Enron email dataset (Klimt & Yang, 2004) with minimal performance degradation, Ilharco et al. (2023) reported that GA significantly harms general model utility when unlearning high-toxicity subset of the Civil Comments dataset (Borkan et al., 2019).
Direct Preference Optimization(DPO; Rafailov et al., 2023): DPO frames unlearning a preference learning task, where the model is trained prefer "I don't know" responses over correct ones for inputs in Diorget- It modifies the conventional pref- erence loss to discourage high likelihood on the forget set, typically without explicit supervision on the retain set. This implementation of Direct Preference Optimiza- tion is sometimes known as Rejection Tuning (Maini et al., 2024). Alternative implementations of DPO generate counterfactual positive samples (Mekala et al., 2024) for tuning.
Negative Preference Optimization (NPO; Zhang et al., 2024b) treats the forget set negative preference data and adapts the offline DPO objective (Rafailov al., 2023) to tune the model to assign low likelihood to the forget set without straying too far from the original model flarget-
(0) -E Druges [oogo (-Alog fo(x)
fuaret(()) )}-
F.
19

---

Published as a conference paper at COLM 2025
where fe refers to the model that undergoes unlearning, is the sigmoid function, and f is a hyperparameter that controls the allowed divergence of fa from its initialization ftarget- Following Rafailov et al. (2023); Zhang et al. (2024b), we fix f ==00 0.1 in our experiments.
Task Vectors (Ilharco et al., 2023) derived from straightforward arithmetic on the model weights can effectively steer neural network behavior. We adapt task vectors to perform unlearning in two stages. First, we train
ftarget on D forget until the model overfits, yielding a reinforced model freinforce- We then obtain a task vector related to D forget by calculating the weight difference between ftarget and freinforce. To achieve unlearning, we subtract this task vector from ftarget's weights, intuitively moving the model away from the direction it used to adapt to Poonget - iee, funleam
ftarget - (freinforce - ftarget).
Unlearning via Logit Difference (ULD) Ji et al. (2024): ULD fine-tunes an assistant model on the forget set D Piorget while simultaneously training the main model to
differ from the assistant. This ensures that unlearned logits move away from correct predictions by computing:
Forget (Y = (( a lassist (Y (x)))
Here, a controls the forgetting strength. This method is particularly effective for token-level unlearning in LLMs.
In-Context Learning (ICL)-Based Unlearning Pawelczyk et al. (2023): Rather than modifying model weights, this approach suppresses recall through prompting. The model is given context such as: Youu are an Al assistant that no longer knows about [Entity). Please respond accordingly." " This method is efficient and lightweight but knowledge can resurface once the prompt is removed.
non-persistent--frrae
Two regularizers for utility preservation. GA, DPO, and NPO are not explicitly designed for utility preservation, so we discuss several regularization strategies that either improve the performance on the retain set ensure the unlearned model remains close to the target model during unlearning.
A.6 Coverage of Our metric compared to previous metrics
To evaluate the comprehensiveness of our dynamic evaluation framework, we assess its coverage with respect to two entity-centric unlearning benchmarks: TOFU and RWKU. We aim to determine whether our automatically generated probes capture the same factual content existing benchmark queries. Despite relying solely model-informed knowl- edge graphs, without manually constructed templates or external corpora, we achieve 66% coverage of TOFU and 78% coverage of RWKU (refer table 10). Figure 4 shows the metric coverage under different graph expansion constraints.
We define semantic equivalence between probes as the ability match the underlying fact tested by benchmark query, regardless of surface phrasing. To identify equivalence, we first use commercial LLM (GPT-4o-mini)
extract the key entity and relation from each benchmark probe. We then check whether the corresponding triplet r,e2 exists in our generated knowledge graph. If a probe generated from that triplet leads the same answer as the original query, we consider it as semantically equivalent. Table 9 provides a few examples of semantic equivalence.
In addition to exact matches, we allow for partial matches in open-ended queries. For example, TOFU includes prompts such as "What are some of the books Hina Ameen has written?" with multiple expected answers. If our framework extracts any subset of these facts (e.g., (Hina Ameen, wrote, Granite Glossary)), we consider the probe covered, as it verifies that the model retains part of the knowledge.
20

---

Published as a conference paper at COLM 2025
Benchmark
Benchmark Probe
Equivalent Probe
Answer
TOFU
In which city Hina Ameen born?
Where Hina Ameen born?
Karachi, Pakistan Portland
RWKU
Stephen King was born in - Maine.
Where Stephen King born?
Table 9: Examples of semantic equivalence between benchmark probes and our framework. Partial matches are accepted for open-ended queries.
Benchmark
Total probes
Overlapping probes
Coverage (%)
Avg. graph size
RWKU
13,131
10.256
78.1%
1OFU
4,000
2,636
65.9%
Table 10: Coverage of our evaluation framework with respect to existing unlearning bench- marks. Total Probes refers to the number of queries in the original benchmark. # Overlap- ping probes counts how many of those probes are semantically matched by our automat- ically generated probe set. Coverage (%) indicates the proportion of probes reproduced. Avg. graph size the average number of nodes needed in our knowledge graph reach maximum overlap with each benchmark.
9.1.
0w10
2-11
c-033
0H0.7
0.64
Coverage
Coverage
6:5.1
4-03
0.60
0.58
50
100
150
200
250
40
60
100
120
140
Graph Size (Nodes)
Graph Size (Nodes)
(a) RWKU
(b) TOFU
Figure 4: Coverage of existing benchmarks at different graph expansion rates.
A.7 Examples of Queries
Single-hop Queries:"Win Who wrote the book Thhe Shining'?" Answer: Stephen King (expected to be forgotten).
Multi-hop Queries (2-hop, 3-hop):
2-hop Query:
"Who wrote the book whose protagonist is Jack Torrance?"
Answer: Stephen King.
3-hop Query:
"Who is married the author of the book whose protagonist is Jack Tor-
rance?"
Answer: Tabitha King.
Fact Retention (1-hop, 2-hop):
1-hop Retention Example: "Who the protagonist of 'The Shining'? ?"
Answer: Jack Torrance.
2-hop Retention Example: "What was the occupation of Jack Torrance?"
Answer: Writer.
Relationship Retention: "Who is the spouse of Jack Torrance?" Expected retained answer: Wendy Torrance.
21

---

Published as a conference paper at COLM 2025
Methods
Previous Metries
Our metric
Score
Removal Crit.
Ret. Crit.
Multi-hop Queries
Fact Ret.
Rel.
Avg.
Aug-
Vblim.
Know. 4
U
I-hop
2-hop 1 2-hop Seq.
3-hop
1-hop
2-hop
Ret
Kenn.
Ret. 1
Muse-Rooks
Fig Model
91.4
1911
62.
99.2%%
18.5%
12..%%
12.4%
1954%%
58.3%
191.3%
16.35%
99777
NA
GA
0
92%
20155%
20.7%
26.4%
65.8%
09.7%
64.3%
19.25%
73.35
76.8%
GDR
10.3
10.8%
22.4%
22.1%
32.5%
71.1%
76.3%
85.2% 2%
21.9%
77 59.
78 0%
GAVER
26.1
11.2%
24.8%
34.9%
73.1%
96.4%
24.2%
77.5%
76.8%
DPQ
38.4
17.8%%
75.2%
DPOKIR
28.3%
73.1%
NPG
717%
26.1%
615%
77.3%
NPOCDR
9.8
66.7%
67.8%
20.7%
73.7%
NPOKLA
18.1
32.7
13.2%
23.1%
25.7%
33.5%
67.5%
72.9%
87.3%
75.9%
76.0%
ICL
10.5
7.
25.3
4.3%
8.2%
8.9%
16.7%
53.7%
57 1%
72
9.8%
62.4%
72 6%
TV
31.2
42.3
11.6%
23.5%
24.3%
36.1%
75.1%
78.3%
88
23.9%
8%
78
ULD
31.8
29.4
51.4
120%
23.5%
20.8%
33.2%
73.6%
78.2%
87.8%
23.4%
79.%
78.9%
Table 11: Comparison of Unlearning Methods Various Metrics the MUSE-books benchmark Shi et al. (2024). The target model here is LLama 3B. Methods: GA- Gradient Ascent; GDR- Gradient Diff (Gradient ascent of forget set with gradient descent of retain set); GAKLR- Gradient ascent of forget set with KL Divergence minimization on the retain set; DPO- Direct Preefernce Optimization; DPO Direct Preference Optimization with Gradient descent retention; DPOKLR- Direct Preference Optimization with KL. Divergence minimization on the retain set; NPO- Negative Preference Optimization; NPOGDR-Negative Preference Optimization with Gradient descent retention; NPOKLR- Negative Preference Optimization with KI. Divergence minimization on the retain set; ICL Incontext Unlearn- ing; TV Task vectors; ULD Unlearning via Logit Difference.
A.8 MUSE: Machine Unlearning Six-way Evaluation A case study
MUSE, introduced by Shi et al. (2024), presents unique challenge to our framework. It consists of two datasets: Books and News. For MUSE-Books, the goal is to forget all the Harry Potter books but retain the Harry Potter-related content obtained from the WinWiki. For Muse-News, the goal is to forget BBC news articles published before August 2023 and to retain articles published after. Our framework is ill-equipped to handle both of these datasets: (1) MUSE-Books, where there is an overlap between the forget and the retain set; (2) MUSE-News, where the goal is to forget the verbatim for the article but not to forget the actual news. Our metric, as described in the paper, is ill-equipped to handle both of these setups. Thus, to inquire if our metric can give useful signals about unlearning efficacy, we modify the evaluation protocol for the case of MUSE-Bood Books.
Modified Evaluation Protocol:
We extract an initial set of entities from test sets constructed by the authors of MUSE. We consider those entities as unlearning targets mentioned in the forget set probes but not the retain set. Additionally, we also mark the ten most frequently mentioned entities in the book to also be part of forget queries. Afterward, we create knowledge graph with multiple seed entities and follow the graph expansion steps described above.
Key highlights: Table 11 shows our metric and previous metrics on MUSE-Booke Books. prior metrics (Verbatim, Knowledge, and Utility) show near-perfect unlearning scores (e-g., gradient ascent-based methods such as GA and NPO indicating complete removal), our evaluation reveals significant residual knowledge accessible via multi-hop queries. For instance, Gradient Ascent (GA), despite showing perfect removal by previous metrics, yields minimum multi-hop accuracy of 9.6%, indicating residual information retention. Meth- ods incorporating retention regularization (e.g., GDR, GAKLR and variants of DPO/NPO) similarly reveal vulnerabilities under multi-hop querying.
B Activation Pathway Analysis with PatchScopes
We further investigate why unlearning methods show limited efficacy multi-hop queries, by using PatchScopes to investigate intermediate layers (Ghandeharioun et al., 2024). PatchScopes decodes hidden transformer-layer activations into interpretable natural lan- guage, enabling us to pinpoint precisely which layers resolve specific entities during model inference.
22

---

Published as a conference paper at COLM 2025
Experimental Setup. We follow the methodology and the experimental design used by (Biran et al., 2024). Specifically, we analyze activation pathways for one useful case, samples where unlearning achieves knowledge removal for single-hop queries, yet fails to generalize to related multi-hop queries. We specifically focus on our running example involving knowledge about Stephen King:
Single-hop query (direct retrieval): "The author of The Shining __"
Two-hop query (indirect retrieval): "The author of the (book with protagonist Jack Torrance is "."
Representation Extraction and Decoding. Our procedure involves the following detailed steps:
1. Hidden Representation Extraction: We pass each query through the original (pre- unlearning) model, recording hidden activations at every transformer layer, specifically the token positions corresponding the query's final answer.
2. Identity-based Decoding: To interpret these hidden activations, we employ an identity decoding prompt designed to explicitly surface the encoded semantic information:
"cat is cat, table is table, blue is blue, Xis
Here, we insert hidden representations extracted from the query in place of "X," allow- ing us to explicitly decode and identify the resolved entity at each layer.
3. Layer-wise Analysis of Entity Resolution: We systematically track entity decoding across all transformer layers separately for single-hop and two-hop queries.
Observations. Our analysis reveals an interesting internal activation patterns:
For single-hop queries (e.g., "The author of The Shining is -."", we observe the queried entity ("Stephen King") clearly resolved within intermediate (middle) transformer layers.
This indicates reliance on a dominant, direct internal activation pathway.
For two-hop queries (e.g,, "The author of the (book with protagonist Jack Torrance is _"), we observe a two-stage resolution: the first-hop entity ("The Shining") is resolved early in the model's transformer layers, while the second-hop entity ("Stephen King") emerges distinctly only in the deeper layers. This indicates multi-hop queries inherently depend on alternate, distributed activation pathways.
Post unlearning: We observe a nearly complete inability to resolve single hop queries. For two hop queries, the unlearning model always resolved the first hop in the early layers and the second hop is resolved in later layers.
Interpretation. Our results qualitatively paint a story: unlearning seems to work when the target entity is resolved in the middle layers and not when it resolved much later in the model. This analysis hopes to build an intuition on why unlearning may fail, however, concrete quantitative analysis is out of scope for this paper.
23


Reviews:
[{"review_id": "b9G71byxYw", "reviewer": "Reviewer_zv4e", "summary": "The paper proposes an evaluation framework for LLM Unlearning which is claimed to be superior to existing unlearning evaluations. The focus is on dynamically creating multi-hop queries which are able to retrieve information which are missed by existing benchmarks. The multi-hop queries are created by probing the LLM's internal knowledge about unlearning targets before unlearning, and constructing knowledge graphs. These graphs can be automatically created without manual data curation.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a strong reasoning and supporting arguments as to why multi-hop queries might expose flaws in existing unlearning methods. Furthermore, it goes on to provide a framework to automatically probe LLMs to create multi-hop queries. It then does an extensive evaluation of most popular unlearning methods to show performance of unlearning. There is also a detailed discussion on how the proposed benchmark compares to existing benchmarks like TOFU and MUSE. The limitations sections is extremely transparent and provides ideas on how to drive unlearning research forward.", "reasons_to_reject": "No strong reasons for rejection.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYuKqDL0fL", "reviewer": "Reviewer_cADi", "summary": "- This paper introduces a dynamic benchmark to evaluate unlearning robustness in LLMs.\n- The core idea is to construct a knowledge graph by querying the target model *before* unlearning, capturing what the model “knows” about a specific entity.\n    - This involves querying the model to extract facts and build a graph of single-hop relations.\n    - The graph is then expanded to include multi-hop paths and aliases.\n    - Using this graph, the framework generates various probing questions—single-hop, multi-hop, and alias-based—to evaluate whether unlearning was successful.\n- The key insight is that while most current unlearning methods pass basic single-hop tests, they often fail under multi-hop or alias-based queries—indicating that the knowledge isn't fully removed.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed benchmark is more rigorous than existing ones. Most prior work evaluates unlearning using static, single-hop questions, which are easy for current methods to pass. This framework exposes brittleness by generating multi-hop and alias-based queries that surface residual knowledge.\n- The framework is dynamic and model-specific, meaning it constructs the benchmark by extracting knowledge directly from the model. This avoids the need for manual construction or external LLMs like GPT-4, and helps ensure that the probing queries are aligned with what the model actually encoded pre-unlearning.\n- The empirical results are thorough. The authors show that their metric aligns with existing benchmarks on method ranking, but also captures failure cases that others miss (e.g., multi-hop alias queries where residual knowledge is still accessible).", "reasons_to_reject": "- The graph construction cost is not well discussed. Since the graph is built by repeatedly querying the model and expanding nodes, how many model calls are needed per entity? This affects scalability, especially if this method is to be used in practice. There's mention of API budget and decay factor, but no concrete numbers or analysis of resource usage.\n- There’s a potential issue with using model outputs as ground truth for what should be forgotten. For instance, prompting (A, wrote, B) may succeed, but the reverse (B, was written by, A) may fail even before unlearning. This asymmetry (reversal curse) can introduce noise into the evaluation: are we really measuring unlearning failure, or just limits in how the model responds to different phrasings?\n- It’s not entirely clear whether multi-hop unlearning is practically necessary. Is this a real-world issue, or is it mainly an academic edge case? If we're constructing synthetic multi-hop queries, we may just end up with unlearning methods that overfit to synthetic query styles without actually improving robustness in realistic settings. Some discussion around the practical importance of this failure mode would be helpful.", "questions_to_authors": "- What is the “LLaMA-3.3-Instruct 8B” model? To my knowledge, Meta has only released 8B models on LLaMA-3.1. Is 3.3 an internal/private version, or a typo?", "extra_scores": {}}, {"review_id": "V70uGBvLBs", "reviewer": "Reviewer_sfKb", "summary": "This work introduces a dynamic framework for benchmarking LLM unlearning. It begins by extracting seed knowledge from the model and constructs a graph-based expansion to discover related information. The proposed benchmark is then used to evaluate existing unlearning methods, showing comparable coverage and effectiveness to static evaluation methods. Moreover, it reveals new failure modes, particularly in multi-hop reasoning settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written, clearly structured, and easy to follow.\n\n2. This work presents a unified, dynamic evaluation framework for model unlearning that demonstrates flexibility and adaptability across different query types and reasoning complexities.", "reasons_to_reject": "1. The evaluation is primarily conducted on the Llama 3.3 8B model, limiting the generalizability of the findings.\n\n2. Since the benchmark relies on knowledge extracted from each model prior to unlearning, comparisons across models become difficult. Each model is evaluated on a potentially different set of knowledge. This raises concerns about the consistency and fairness of cross-model comparisons, as variations in the constructed knowledge graphs may lead to differences in evaluation difficulty rather than reflecting true differences in unlearning performance.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Di9t1u6ywA", "reviewer": "Reviewer_6xWi", "summary": "The paper proposes a framework to automatically construct automatically a benchmark to assess unlearning efficiency. The framework elicits the knowledge from the baseline large language model to construct a knowledge graph. Using the knowledge graph information such as aliases, relations, etc. The framework constructs benchmarks to assess unlearning efficiency. \n\nBy testing to a large number of methods on the benchmark, the authors show that the automatically constructed benchmark agrees higly with high quality manual construction. The framework also enables discovery of new issues for unlearning.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has clearly exposed its motivation and contributions. \n\nThe extensive experiments shown in Table 2 and the appendix demonstrates the viability of the method. Not only the method shows its efficiency when compared with existing benchmarks, the papers show that such automatic construction might reveal new issues for the unlearning research, which makes the paper have potentially good impact.\n\nThe paper discusses the limitation of the proposed approach transparently and in an insightful way. The automatic construction has the limitation from the elicitation part and can give future research directions.", "reasons_to_reject": "I personally found Table 2 and the corresponding description in 5.2 a bit confusing. After reading carefully, I believe the results are produced under their constructed benchmark but seed with RWKU entities. If this is true, it needs to be clarified at several places:\n- Table 2 caption, it is then not on RWKU benchmark\n- line 250 as an example, these are not results for RWKU\n\nThe authors have more results in the Appendix and I think authors could refer to it more appropriately for curious readers.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously found this paper to be a valuable and timely contribution to the growing literature on LLM unlearning, praising its clear motivation, rigorous methodology, and thorough experiments (scores = 8, 7, 7, 6). They especially appreciated the dynamic model-specific benchmark that uncovers residual knowledge via multi-hop and alias-based probes, a weakness that static tests miss, while still aligning well with prior evaluations. All critical points raised during discussion (graph-construction cost, asymmetry in prompting, real-world relevance of multi-hop queries) were addressed convincingly. I think this is an impactful paper that should be accepted.
Decision: Accept