Do LLMs Really Forget? Evaluating Unlearning  with Knowledge Correlation and Confidence Awareness
Authors: Rongzhe Wei
Peizhi Niu
Hans Hao-Hsun Hsu
Ruihan Wu
Haoteng Yin
Mohsen Ghassemi
Yifan Li
Vamsi K. Potluru
Eli Chien
Kamalika Chaudhuri
Olgica Milenkovic
Pan Li
Venue: NeurIPS 2025 poster ()
Keywords: Machine Unlearning
Knowledge Unlearning
Knowledge Correlation
Large Language Models
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Machine unlearning techniques aim to mitigate unintended memorization in large language models (LLMs). However, existing approaches predominantly focus on the explicit removal of isolated facts, often overlooking latent inferential dependencies and the non-deterministic nature of knowledge within LLMs. Consequently, facts presumed forgotten may persist implicitly through correlated information. To address these challenges, we propose a knowledge unlearning evaluation framework that more accurately captures the implicit structure of real-world knowledge by representing relevant factual contexts as knowledge graphs with associated confidence scores. We further develop an inference-based evaluation protocol leveraging powerful LLMs as judges; these judges reason over the extracted knowledge subgraph to determine unlearning success. Our LLM judges utilize carefully designed prompts and are calibrated against human evaluations to ensure their trustworthiness and stability. Extensive experiments on our newly constructed benchmark demonstrate that our framework provides a more realistic and rigorous assessment of unlearning performance. Moreover, our findings reveal that current evaluation strategies tend to overestimate unlearning effectiveness.

Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness
Rongzhe Wei", Peizhi Niu"", Hans Hao-Hsun Hsu', Ruihan Wu', Haoteng Yin², Mohsen Ghassemi', Yifan Li?, Vamsi K. Potluru', Eli Chien',
Kamalika Chaudhuri', Olgica Milenkovic², Pan Li!
Georgia Institute of Technology, Univerrityy of Illinois Champag, Technical University of Munich, *University of California San Diego,
Purduue University, J... Morgan Al Research, "Tsinghua University frongzhe..m weingzhe.wii, wei, ichien6, paanil)@gatechhcc edu, {peizhin2, milenkor)@illinois... {ruw076, kamalika) edu, {mohsen ghassemi, vamsi .. pausi...pottuuuu@jjjnmmmit. pot.uur//@jpmccase... com,
hans. www.shh.uuummmmmm yinht@acm. org, 1yf210mails. tsinghua. edu. cn
Abstract
Machine unlearning techniques aim to mitigate unintended memorization in large language models (LLMs). However, existing approaches predominantly focus on the explicit removal of isolated facts, often overlooking latent inferential dependen- cies and the non-deterministie nature of knowledge within L.LMs. Consequently, facts presumed forgotten may persist implicitly through correlated information. To address these challenges, we propose knowledge unlearning evaluation frame- work that more accurately captures the implicit structure of real-world knowledge by representing relevant factual contexts as knowledge graphs with associated confi- dence scores. We further develop an enference-based evaluation protocol leveraging powerful LLMs as judges; these judges reason over the extracted knowledge sub- graph to determine unlearning success. Our LLM judges utilize carefully designed prompts and are calibrated against human evaluations to ensure their trustworthi- ness and stability. Extensive experiments on our newly constructed benchmark demonstrate that our framework provides a more realistic and rigorous assessment of unlearning performance. Moreover, our findings reveal that current evaluation strategies tend to overestimate unlearning effectiveness. Our code is publicly avail- able at (https: thub. com/ Soom/Graph--- httpo//////ooo//eeeeeeeeeeerrrar git
Introduction
Large language models (LLMs) have achieved widespread adoption across diverse application domains due their remarkable capacity to acquire and encode complex, interdependent knowledge from vast web corpora [12]. However, this impressive capability simultaneously introduces critical
risks. Notably, LLMs may inadvertently memorize and reproduce copyrighted content [3],), amplify social cultural biases [5), or reveal sensitive and private information (6). Such issues not only jeopardize user trust but can also contravene ethical principles and regulatory frameworks governing responsible AI deployment 87.. In response, machine unlearning has emerged promising approach for selectively removing specific data points, concepts, or factual knowledge from pre- trained models [T)
Authors marked with contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

User Data Curator
Lag
Unlearning
Request
Confédence-awin Supporting Salgraahh
Unlearuing Stage
Knowledge
Knowledge Bass
paintia
Construct Supporting Subgraph
Evaluation Stage
Vinei
Mona Lisa
Querying Target LLM Outpu:: Subgraph with Confidence
Unlearned LLM
Friidl Ponerial
Enteming Tarpec Befihhlies Confidence Coltintion
Knowlolge Graples
Imput to J.M Judge
No Leveray Baternall Kann
GPT
Julge grällnce
Judge Score
Enhancing LLM Judge Reliahiity! in In-corton learning. Humann Felltcioon Comporken, etc.
Powerful L....Judgp Judgo
Figure I: An Illustration of Knowledge Unlearning Framework.
Despite progress in unlearning factual knowledge from LLMs, current methods mostly tackle the problem on the surface. Whether by targeting individual training examples [12 or erasing sets of facts linked to specific topics 11 15), these approaches share a critical flaw: they focus on direct deletion while ignoring how deeply knowledge is interconnected within LLMs (16 This is a serious oversight. means that even when fact seems "crased" (disappearing from direct questions), it often still lurks within the model, ready to be figured out from other related pieces of information the LLM still holds. The targeted fact can then be pieced back together through indirect questions. shattering the idea that unlearning truly worked and causing a loss of trust. For example, an LLM might "forget" that "Mount Fuji a volcano." " only for this to be easily deduced because still knows that "Mount Fuji has crater at the summit" and "craters are formed by volcanic activity." This same challenge also indirectly echoes the findings knowledge editing studies [T9] changing isolated facts often doesn't i ripple through the model's wider reasoning, showing that surface-level unlearning isn't genuine unlearning at all. This major gap shows that a new approach is vital. Therefore, we propose to significantly expand the concept of knowledge unlearning (KU): it must go beyond simple deletion to actively take apart the underlying knowledge structures that support the target information.
Beyond overlooking correlations among knowledge facts, existing unlearning evaluation frameworks may also be inadequate in modeling how knowledge is organized and processed within LLMs. First, real-world knowledge exhibits diverse and complex correlation patterns, with inferential relationships that may be non-deterministic or context- depeedett [2T]. Consequently, evaluation protocols based on manually defined inference rules or fixed reasoning chains often fail to capture the probabilistic nature of these factual dependencies [22]. For example, consider unlearning the fact that Person is the CEO of Company X. While this fact might be explicitly stated, it could also be inferred from indirect clues, such as references to Person overseeing operations, attending board meetings, or signing executive decisions, each carrying varying degrees of evidential strength. Second, current methods frequently assume that facts within the unlearning dataset are already well internalized by the LLM. In practice, however, an LLM's grasp of specific facts, particularly those involving rare entities or domain-specific relations, can vary significantly [14). LLMs may exhibit only partial or uncertain knowledge of these facts even after fine-tuning. If such variability is not accounted for, it can lead to inaccurate or inflated estimates of unlearning effectiveness.
We propose novel and realistic formulation of knowledge unlearning, grounded in a confidence- aware perspective of how factual knowledge is represented within LLMs. Our framework is designed to reflect the complexities of real-world knowledge systems, where facts are often uncertain, interde- pendent, and mutually inferable. To achieve this, we represent the knowledge embedded in an LLM as confidence-awant knowledge graph. In this graph, each triple denotes piece of knowledge and is associated with a confidence level derived from the model's predictions. Given a target triple for unlearning, we then probe the LLM to extract a subgraph of correlated facts that could potentially enable the inference of this targeted triple, subsequently analyzing this subgraph to determine the extent to which the target knowledge persists within the LLM.
To effectively automate unlearning evaluation, we propose employing a powerful LLM judge (e.g.. GPT-series models) to act as an adversarial agent. This LLM judge employs carefully crafted prompts and specific calibration procedures to reason over the extracted knowledge subgraph and assess whether and to what extent the target triple remains inferable. Such a protocol moves beyond

---

superiicia-leeee triple recall, capturing the residual inferential capabillities embedded within the model. Furthermore, to validate our J.LM-based evaluator, we compare judgments against those of strong human adversaries to assess their alignment and reliability. Extensive experiments on large-scale, real-world encyclopedic datasets demonstrate our framework's is applicability while also highlighting the limitations of existing unlearning methods and evaluation techniques. Our experimental analysis reveals several key insights. First, correlated knowledge supports inference and can significantly reduce unlearning effectiivnnsss, even when the target triple appears superficially erased. Second, even fow-confidence yet semantically related knowledge can substantially compromise unlearning, underscoring the importance of capturing weaker associations. Finally, with careful prompt design and calibration, the LLM judge enables automated evaluation of unlearning effectiveness while producing judgments closely aligned with those of human experts. Accompanying our findings, release two key resources: first, a benchmark for LLM probing that reflects knowledge interdependence, derived from real-world knowledge datasets; and second, an evaluation protocol tailored to the novel concept of knowledge unlearning presented herein.
Related Work
Evaluating unlearning in LLMs remains a core challenge, with prior studies proposing various metrics to measure the effectiveness of removing specific training instances or factual knowledge (23). Early
approaches such as WHP (24) assessed unlearning by eliminating Harry Potter-related content using completion-based and seaee--robbbilttybbaaee
metrics, while Wei et al. (25) investigated methods to prevent the generation of copyrighted content. Subsequent benchmark TOFU [11] considered entity-level unlearning and compared pre- and post-unlearning question-
question-answering answering performance using fictitious author biographies. MUSE [13] provided comprehensive evaluation across six distinct dimensions, and WMDP (26) specifically focused on unlearning harmful knowledge to mitigate malicious use. Additionally, RWKU [27] proposed benchmarks tailored for real-world knowledge scenarios, and Wei et al. [14) introduced a minority- framework to identify high-risk data points for unlearning. However, these approaches generally treat knowledge independently, overlooking the interdependencies between target facts and related knowledge in LLMs. This limitation was pointed out in 28), which first explored multi-fact interactions during unlearning but relied on deterministic knowledge modeling and tule-based evaluations, which limits their practical applicability.
Another related research area is knowledge editing, which focuses on updating specific factual infor- mation within LLMS [29]. Existing evaluation frameworks typically assess whether models correctly recall edited facts in response to single-hop queries 38-38). More recent studies have incorporated multi-hop factual chains as evaluation tools to determine if model updates successfully propagate through indirect reasoning paths [19 20 J. In contrast, our work concentrates
the knowledge unlearning task, which aims to entirely remove target knowledge. Distinctly, our framework explicitly extracts correlated knowledge subgraphs within the LLMs that support the target knowledge fact, whereas knowledge editing evaluations frequently rely on deterministic factual chains.
3 The Formulation of Knowledge Unlearning
Confidence- Knowledge Modeling in LLMs. We consider the scenario where a target pretrained LLM, denoted as Mpretrain tasked with unlearning relational knowledge. We formulate the factual knowledge embedded in this LLM as confidence-awann where E denotes the set of entities, R represents the set of relation types. Each knowledge fact encoded as knowledge quadruple t = (s, r, o, u), consisting of a subject entity an object entity o, along with non- neg-tivae valued confidence score u E lI that reflects the LLM's degree of belief in the fact; Ty S 3 x R E x 24 denotes the set of all confidence-awee aware facts in g.
knowledge graph G = (E, R. TT).
relation r, and
In practice, a given target fact e = (s, r, T is often correlated with a subset of related facts To = (1,, C Tu in Mpretrain. from which it can potentially be inferred. For instance, the triple (Paris, Capitalof, France) may be supported by related facts like (Paris, hosts, Élysée Palace, un and (Élysée Palace, isGovernmentSeatoy France, U2). The strength of such inference depends on both the structural relationship between e and the supporting facts, as well as the associated confidence scores in and u2 within Te. To quantify the strength of such inference, it requires introducing an intrinsic judge function f: Tu x T + y, where TT denotes the power set of knowledge triples. Given supporting subset Te and target triple e, the function f e) outputs an inference score
3

---

quantifying how strongly the target fact can be derived from the supporting set Te. A higher inference score indicates stronger support, while lower values correspond to weaker inferability.
Knowledge Unlearning. Let A he an unlearning method. The knowledge graph induced by the LLM Munleum (after applying A) is gAA A target triple e = (s, r,,r, o) is deemed unlearned if it is robustly erased from MMiieam 's accessible knowledge. Specifically, even strong adversary should not be able to infer through strategic queries to M unlear concerning related facts in its latent knowledge graph GA, whether these queries are direct indirect probes of the relation between s and o. Importantly, simply causing the model to deny e upon direct questioning is often insufficient; deeply interconnected knowledge structures from pretraining may still allow for its inference. The potency of such inference attacks is a function of the adversary's 's reasoning power, captured by the intrinsic judge function f. This provides the basis for the following definition of knowledge unlearning:
Definition 1 (Knowledge Unlearning). Given a target triple = rs, for unlearning and an adversary with intrinsic judge function J. let Mantam denote the model obtained after applying unlearning method A with induced knowledge graph G We say A achieves y -knowledee unlearning if I[GA, e) S y, where Y specifies an upper bound on the residual inferability of e, efrom Maitcorn
The above knowledge-unleariing definition, while theoretically sound, faces practical evaluation challenges. First, accessing the model M 's full knowledge representation GA is typically infeasible. To approximate the inference score f(GA,e) e), we extract a localized supporting subgraph G.A. S GA, This subgraph, centered on the target triple e and constructed using a real-world knowledge base as reference, captures relevant local supporting facts (details in Sec. 4.. Our practical evaluations therefore assess approximate knowledge unleurming using S(G=A, e). Traditional direct triple removal evaluations, termed instance unlearning, represent special case where G.A. reduces to the target triple and its associated confidence score, i.c., G.A. = (e, ue) E Tu. Second, the definition presumes an intrinsic judge function f with comprchensive knowledge of all potential factual relationships. As this ideal is generally unattainable, human expert annotators or powerful LLM-based evaluators will be used as practical proxies for f-
4 Methodology
4.1 Overview of the Evaluation Protocol
To evaluate whether a target fact has been effectively unlearned, we build upon the definition of knowledge unlearning introduced in Sec. 3 We assess the inferability of a target triple e based on its supporting subgraph G.* and proxy judge function f(G ,)) e) quantify the extent to which remains inferable from retained knowledge. Successful unlearning requires not only the removal of the target triple but also the disruption of its underlying inferential structure. To instantiate this evaluation protocol, we adopt a two-stage approach: (1) Supporting Subgraph Extraction, where we probe the unlearned model M Munleam to construct G.A,, and (2) Adversarial Inference via a Powerful Judge, where the inferability score is computed using a powerful LLLM/human expert adversary.
4.2 Extracting Supporting Subgraphs from Model Beliefs
Given a target triple e = (s, our goal is to retrieve its supporting subgraph G.A. S g encoded in the unlearned model M that substantiates its inferential basis. We
formalize this subgraph
union of confidence-aware triples extracted from M
unlearm
These triples collectively trace potential deductive pathways from the sub- ject to the object o within a bounded path length f. The constraint of using a bounded path length for these deductive pathways aligns with observa- tions that many reasoning tasks can be effectively resolved within small number of hops (39]
Supperting Subgraph
Constructing such pathways requires probing M Wileam with candidate triples. A key challenge arises as MA anneam offers no explicit catalogue of entities or relations, and an exhaustive enumeration of all possibilities is computationally prohibitive. Therefore, to define tractable search space, we determine candidate entities and relations by referring to an external real-world reference knowledge graph Geet (c.g., Wikidata). This reliance on
Figure 2: Illustaatoon of supporting subgraph.

---

an external, publicly accessible corpus is practical necessity. Indeed, even adversaries attempting to indirectly probe an LLM's knowledge typically need to anchor their queries in commonly understood entities, relations, or schemata. While our Ger-guided approach is not perfect and cannot capture all latent knowledge, it provides a principled method for exploring deductive pathways grounded in relevant, publicly accessible information.
To be more robust against the potential missing information in Gref, our strategy for generating knowledge fact candidates from a frontier node v (initially s, then other entities along a path) involves retrieving its k-hop neighborhood in Gee. The union of these neighboring entities serves as the set of candidate objects. Candidate relations are similarly drawn from the available relation types in Giet. This approach assumes that the local neighborhood in the reference KG offers a reasonably comprchensive, albeit imperfect, approximation of the entities relevant for deductive reasoning that are implicitly known to Munteam- Ultimately, the resulting supporting subgraph is intended to capture the latent reasoning chains through which the target triple may be logically derived.
Constructing the Support Subgraph. We build Ge iteratively under a greadth-limited expansion strategy (illustrated in Fig. 2). Starting from the subject s, we probe the unlearned model M with candidate triples of the form (s)r',''). Whenever the model's confidence for triple exceeds a preset threshold u", the triple is added to the subgraph and its object o' becomes the new frontier. Here, we use u S =² to denote the excess case. In the next hop, starting from the new frontiers o', we query triples ro",r", o"") where o"s come from all candidate entities. After each round, we record the model's 's confidence scores and discard triples that are either rejected or judged uncertain. The expansion stops when the path-length limit& é is reached or no further confident triples can be found.
To properly estimate the model's confidence in candidate triple (s', (d'r', o'), we adopt multiple- choice querying protocol, where each triple ver- template M unlear with answer choices (Yes, No. Unknown]. ). While an alternative might involve having M M directly output a numer- ical confidence score, our empirical investiga- tions revealed this method to be inaccurate, which is consistent with existing studies on the chal- lenges of calibrating L.LM confidence (40). Con-
Refirease Keendedg
Sabgraph Camtrectio
Fillew
balized using a fixed natural-languall
and presented
Confilcars Evakaitee
The www.h//lrrrr
Figure 3: The entire knowledge probing process.
sequently, we find that more reliable confidence assessments are obtained by applying temperature scaling (41) to adjust the model's softmax output probabilities corresponding to these answer choices. This calibration process yields significantly more accurate confidence scores, as demonstrated by our experiments in Sec. [6.3] After calibration, we model the confidence space in terms of entropy: H(s', ,r' o') =- Eie(Yes, Unkkkoow) P. log, P,,, where Free, Pro. Punknown are the calibrated proba- bilities for the respective answer choices. A triple is admitted to GAA iff the model selects Yes and H(s',r' o') u². Recall that u" is predefined entropy threshold. In this paper, we set u' = 1, which corresponds to a worst-case scenario where the model follows instructions but assigns equal probability (0.5) to Yes and one other option (No or Unknown). The full mapping between u" and minimum Yes probabilities is provided in App. G This criterion filters out unconfident evidence, preserving only high-confidence relational structure. Overall, the entire knowledge probing process as above is illustrated in Fig. 3 The corresponding algorithm is provided in App. A Alg.
4.3
Adversarial Inference Assessment via Powerful LLM Judge
Given a target triple e and its associated supporting subgraph G the evaluation of unlcarning effectiveness relies on adversarial inference scheme to assess the residual inferability of e through systematic exploration of G.A. While human experts provide a strong reference for reasoning-based evaluation, they are not scalable for large-scale assessment. To automate the process, we employ powerful reasoning LLM as the judge. We monitor the alignment between the LLM judge and human expert evaluations on subset of examples, with results reported in Sec. 6.3
Effective reasoning by the LLM judge hinges on a carefully engineered prompt structure. This structure is built upon two foundational considerations: first, it mandates that the judge's rea- soning be based exclusively on the provided subgraph G.A. encompassing its factual content and associated confidence scores without resorting to any external knowledge. Second, we re-
5

---

quire the judge to quantify inferability using a discrete 0-5 rating scale, where 0 signifies that the target triple e cannot be inferred and 5 denotes very high certainty in deducing e from G.A.
To further ensure the reliability and accuracy of these judgments, the prompt design inherently integrates clear evaluation objectives, explicit semantic definitions for all rating levels, antropy-based guid- ance that categorizes triple certainty into discrete confidence bins (as illustrated in Fig. 4 Task-specific instructions and in-context examples are also embedded to ensure consistent and accurate eval- uation behavior. For the full prompt template with the interpretation of each rating score, see App. C.I
Plann
Importantly, the LLM's judgments are expected to rely solely on G.A. treating it as ground truth, without leveraging any of its internal knowledge acquired during pretraining. Our experiments will verify this assumption by analyzing the model's reasoning traces.
Sooe the cort...
(hom, cacial 2r, (911))
model's
Bi1 Viati ingtrally
Tag.ur (ith wny tiring (inal Sower
Knowledge Unlearning Evaluation Datasets
Figure 4: Illustration of In- structions for LLM Judge.
Systematic evaluation of the unlcarning protocol described herein necessitates dataset where target triples, presumed to be retained
by the LLM, are accompanied by a rich network of correlated and semantically coherent triples capable of supporting target triple inference. To create such dataset for realistic scenarios, utilize real-world knowledge bases, specifically selecting YAGO3-10 [42]. YAGO3- 10 is a large- scale knowledge graph built from Wikipedia, Wikidata, and WordNet [43] containing over one million relational triples, 123,182 entities, and 37 distinct relation types. Crucially, its Wikipedia and Wikidata foundations are pertinent as these sources common in the pretraining corpora of modern LLMs (e.g.. [66 44 45)) making it likely that YAGO3-10% 10's factual triples are retained by these models. Moreover, the relations defined in YAGO3-10 exhibit various inferential structures (more illustrations in H). For our experiments, we target specific YAGO3-10 10 triples for unlearning, using the full YAGO3-TO graph as the reference Giet for support subgraph extraction.
Target Triples and Supporting Subgraphs Extraction. From each LLM for evaluation, we extract 200 unlearning target facts. To ensure these targets constitute factual knowledge genuinely retained by the model, they are identified using a carefully selected knowledge-prooe prooeengg template (to be introduced later) coupled with entropy-based confidence filter; the specifics of how to measure confidence detailed in Sec. 4.2 Supporting subgraphs for each unlearning target are constructed by querying all relations in Gver and retrieving candidate knowledge facts within a 3-hop neighhorhood (k- 3) each expansion step. The resulting subgraphs are constrained to a maximum path length of = 3,
which reflects a reasonable inference depth in line with prior work [9] 39
Knowledge Probing Method. Effective knowledge probing of LLMs is crucial for both our dataset
construction and subsequent evaluation phases. Following the approach in [27]. we utilized GPT-4 to generate a diverse pool of candidate prompt templates. These templates were specifically designed for multiple-choice query format in Sec. (.2 To evaluate these candidate templates, we constructed comprehensive validation set. For each of the 37 relations in YAGO3-10, 10, we selected 10 positive examples, consisting of factual triples that hold the given relation within YAGO3-10. Concurrently, for each relation, we generated 10 negative examples composed of counterfactual entity pairs that do not hold that relation. These negative examples underwent manual verification by human annotators to confirm their factual incorrectness, despite being constructed to appear plausible (i.e., the entity types were compatible with the relation). This process yielded validation set totaling 370 positive and 370 negative samples. Each candidate prompt template was then systematically evaluated based on its accuracy over this validation set. The template that achieved the highest accuracy was selected as our definitive query prompt for all LLM interactions (see App. C.2 for the complete template).
6 Experiments
6.1 Experimental Setup
Unlearning Methods. For each target I.L.M, we evaluate the following popular unlearning approaches in the literature: Gradient Ascent (GA) (46448) aims to erase the influence of unlearn triples by

---

reversing gradient updates. Random Labels (RL) [4649] 49
disrupts memorization by replacing
correct labels with random tokens during training. Negative Preference Optimization (NPO) [50]
formulates unlearning as reversing the model's preference toward unlearned data by optimizing likelihood ratios against the pretrained model. NegGrad+ [5T] combines gradient ascent on the unlearn set with gradient descent on non-unlearn set to retain performance. Finally, SCRUB (5T) adopts a student-teacher framework, using KL divergence to balance knowledge removal from the unlearned data with the retention of non-target data. To ensure fair comparison, we adopt a unified
computational budget protocol for all methods, following [1..
Formal definitions, implementation
details, and per-method budget allocations are provided in App.
B
Unlearning Effectiveness Metrics. We propose the Unlearning Effectiveness Score (UES) to quantify how effectively an unlearning method A reduces target knowledge inferability by comparing pretrained (Mpremin) and unlearned (Mamiear) models. Given unlearning targets Drorget for each e D Diorget let Ge and G be its supporting subgraphs from Mpretrsinn and Mu respectively. UES is the average normalized relative decrease in an LLM-assessed inference score UES =
(G.e)(G..))-faaeel .e) Here, fjudee outputs a discrete score from 1......,,5 5) (detailed
ECE Dirget
(G,.e(
in Sec. 4.3). UES upper-bounded by 1: higher values denote more effective unlearning (greater inferability reduction). UES= 0 implies no effect, and negative values indicate increased inferability after unlearning. The metric is well-defined as fjulge(Go (G.: e) is strietly positive for verified retained targets (Sec. 5). For instance-level unlearning ("Inst."), "), targeting direct triple knowledge, the metric computed by replacing the subgraph Ge (or GA)) with e paired with its confidence Ue before
IG..cal A GG.
unlearning (or after unlearning, resp.). Additionally, we employ Recall [28] Eee Disges
metric that measures the proportion of original supporting subgraph Ge s triples preserved in the unlearned model's s subgraph (G²). Since these subgraphs reflect inferential support for e, lower Recall value indicates that more of this supporting structure has been successfully unlearned.
Model utility is assessed by the retention of local knowledge near unlearned triples and the preservation of general model capabilities (27). Local Consistency (Loc). To assess local utility preservation, we sample triples within a 3-hop neighborhood of the target triple's ' head or tail entities from the reference knowledge graph Gres. excluding triples already included in the supporting subgraph G.. These neighboring triples are considered irrelevant to the target inference and should remain unaffected by unlearning. The Loc score measures the consistency of model predictions (Yes, No or Unknown) on these neighbor triples before and after unlearning. A higher Loc score indicates better preservation of local factual integrity. Importantly, Loc captures multi- directional knowledge shifts, such as facts changing from Yes to No or vice versa. Further analysis is provided in [ General Model Utility. General capabilities are assessed from two perspectives: General Knowledge (Gen), measured on MMLU (52) (a multi-domain multiple-choic benchmark) via 5-shot perplexity- answer ranking and Reasoning Ability (Rea), measured on BBH (53) (a suite of 27 challenging tasks) by 3-shot exact match accuracy with chain-of-thought prompting.
Utility Preservation.
General Settings. We conduct our unlearning experiments using two widely adopted open-source LLMs: -LLMA3-8B-Insttr
(54) and Qwen2.5-775 ---en2.5-BBInsstruec (55). The unlearning procedures are implemented via either full model fine-tuning or parameter-efficient tuming using LoRA (56). The reported results are averaged over the total of 200 target triples. For the utility evaluation on Loc, we sample a set of 2,000 neighboring triples (10x the size of the unlearning targetset), reflecting realistic unlearning- weaiig-tto-reeene to-retention ratios as discussed in prior literature [57]. During unlearning, we consider two formats for presenting unlearning targets: (i) converting each triple into a natural sentence (unlearn with sentence), and (ii) framing it as a multiple-choice question with options (Yes, No, Unknown) (unlearn with Q4). To ensure a fair comparison across methods, we fix the computational budget: 10 unlearning epochs for GA, RI. and NPO, and epochs for NegGrad and SCRUB, which require additional knowledge triples to preserve utility (detailed in Ap.. For epoch selection, we follow a similar protocol introduced in [3 14 28): the model's utility (Loc) exceeds 0.8 at any epoch, we select the last epoch satisfying this criterion; otherwise, we select the epoch with the highest utility below the threshold. During evaluation, we use GPT-o4-mini as the adversarial judge. Full implementation details, including hyperparameter configurations, provided in App. D
7

---

HLM Sasisiri
Method
Unleariig Rilleciromee
Unily Relentam
Unlearning Elfecliveaus
Unilit Rutumtion
(Chusi)
UBS (Our)
Gon Loe
Uniearn with Suntense Tumplates
GA (Fulh)
0.157
1:06 (51 59%
0.257
0.951
0.630
0.566
-0.000
-0093 (365%
0.977
0.964
01653
0.581
RL. (Full
0427
1000 (AL005 L3
IU379
0.994
0661
0.565
0.176
0,072 (58.2596.)
0.833
0.935
08533
0.576
Fulli
0054
120, 12%
0.586
(59,5766))
01.36
0.500
SCRUB (Full)
0/23
40003 (158.19%))
0.919
0.957
0/68
0.560
0.509
0.457 (251129))
0,402
0.739
0.627
0,583
(LoRA)
0.101
06:10
0.585
0.122
1979
01866
0,572
NPO (LaRA)
0.181
00100 (94 68%4
0.775
0.989
06612
0.3t6
0.300
0.244
(3) 23611
0.651
0.933
06688
0.596
NagKonade (LalkA)
0.154
06500 (90 51
06472
0.997
06655
[.583
0.195
0.09 (49:23751)
0.540
0.900
(1.40)
0.581
SCRUW (1 (RR
0.201
D:055
(54 50%
0.329
0.997
0.611
0.515
0.715
0.516 (27.8566)))
0.412
0.705
0.619
0.573
Uukea
Tomplates
No Unleaen
0.633
0.567
01657
0.581
(Tult)
HL. (Pull)
0.134
155 22%
0.486
0.992
0461
0.316
0.081
-0012
(1812
0.032
01882
01540
0.585
NPO FFully
0.233
0 (5G.009L))
0.457
0.957
06644
0.567
0.7%
0.705 (13.198))
0.11
0.63
0.638
0.581
NepGonds (PPu))
0956
0.04 (4468)))
0.009
0.175
0627
0.55
8.066
171-17
0.13
0.584
0.61
0.576
SCRUB (Full)
0.959
579
0372
B36
0.000
0.590
GA (LoRA)
0,145
DU0N (33 79R 13
0.305
0:996
D634
0.566
0.7%
0.299 (3PV3366L))
0.600
0.22
0.640
0.590
RL (LaRA,
04093
Rudd (55. 16R 1
0.384
0.958
0633
0.566
8.006
-0025 1256 25R0.
0.00
0.50
0639
0.591
(LRRA)
0.690
0.085
NegGrod (LaRA)
(25
0.564
(44.906L))
SCRUB (LoRA)
0.108
DV652 (705799.1
0.481
0.878
0.87
0.560
0.914
0.777 (14.996))
0.286
0.717
0,594
0.514
Table I: Comparison of various unlearning methods LLaMA-8B- Instrucct and -0 Instructt in terms of unlearning effectiveness and utility retention with different unlearning templates.
6.2 Main Results
Correlated Knowledge Supports Inference and Reduces Unlearning Effectiveness. We report the unlcarning effectiveness and utility metries of each unlcarning method under both instance unlearning (Inst.) and our correlated knowledge tracing framework in Tab. When comparing each method's UES under instance unlearning versus supporting subgraph setting, we observe substantial drop in effectiveness. For most cases, the UES decreases by at least 20%, with more than half of the settings exhibiting reduction exceeding 30%. We further analyze specific unlearned triples and observe that certain triples appear successfully unlearned at the instance level, yet remain inferable when considering their supporting subgraphs. For instance, the triple (rehitrrh created, Brookfield_ SieeldPlaae__ll (Toorntoo) is effectively unlearned (entropy: 0.252 before unlearning, 1.127 after), but the evaluator LLM can still infer through related facts such as (B+H Architects, isKnownFor, Brookfield Office Properties) (entropy: 0.134 before, 0.512 after) and (Brookfield Office Properties owns, Brookfield (flace((Troont)) (entropy: 0.1 10 before, 0.441 after) present in the supporting subgraph (see additional examples in App.
Meanwhile, we observe that when local utility is well preserved (e.g., Loc 2 0.8), meaning over 80% of entity-centric triples in the neighborhood are retained, the structure of the supporting subgraph G.A. as reflected by the recall score, also remains largely intact. This preserved structure enables more potential inferences, thereby reducing unlearning effectiveness. In contrast, when utility degrades sharply, much of the supporting context is disrupted, limiting inferential paths (e.g., QA-based full model unlearning for Qwen2.5 in Tab. J..
We further compare the overall performance of unlearning methods in Tab. I No method consistently achieves a favorable trade-off between unlearning effectiveness (UES) and utility (Loc). When Loe 0.8, GA, NPO, NegGrad+, and SCRUB generally outperform RL across both LLMs. These four methods show comparable performance, each achieving the best results in different settings.
Sentence-based unlearning provides more utility- utility-preserne preserving alternative to Qa-based unlcarning. As shown Tab. I under knowledge probing via QAs, employing traditional sentence tem- plates under identical unlearning configurations results in superior utility retention (measured by Loc) and less aggressive average un- learning effectiveness compared to the QA-based approach. This advantage likely arises because sentence-based contexts elicit more conservative parameter updates, whereas QA-based unlearning ex- plicitly targets specific model outputs (e.g., Yes), causing abrupt behavioral shifts and potential instruction followwng degradation.
Unlearning Effectiveness
1.0
Grudient Assent
0.8
Aro
Nandom Label
0.6
0.4
0.3
0.8 Entropy Threshold
6,5
1.0
Figure 5: Effectiveness of Entropy Threshold u² (Confi- dence) on UES.
How does the confidence score affect unlearning effectiveness? In our framework, we model the target L.M''s 's confidence in knowledge
triples. Here, we further investigate how these confidence scores influence unlearning outcomes by
8

---

varying the threshold u" used in generating supporting subgraphs. As shown in Fig. 5 we report results under the SaaMMA--B-IInsrruee LoRA QA setting. We observe that decreasing the entropy threshold u², i.e., requiring higher LLM confidence, filters out more supporting inferences, thereby increasing unlearning effectiveness. This highlights that is crucial model the confidence of L.M.s on the knowledge facts, as low-confidence knowledge can still contribute meaningfully to inference and introduce information leakage. Refer to App. Ffor more results.
6.3 Justification on the Reliability of the Target LLM and LLM Judge
In this subsection, we report results validating the reliability of both the target LLM and the LLM- based judge. (1) Target LLM Calibration. To interpret entropy calculation, we first calibrate the token probabilities (i.e., confidence) output by the target L.M.. As mentioned in Sec. 4.2 we calibrate the optimal temperature that aligns predicted probabilities with actual model behavior. Using the validation set detailed in Sec. 5) with positive and negative knowledge triples, we prompt the LLM with multiple-choice questions and record the model's ' predicted answer based on the argmax token. We then compute the Expected Calibration Error (ECE) [41] separately on "Yes" and "No" predictions, measuring how well model confidence aligns with accuracy (lower is better). As shown in Fig. 6 the cemperature-scale output distributions of both models exhibit strong alignment between the predicted probabilities and accuracy, indicating reliable confidence estimates. (2) LLM Judge Reliability. To ensure that the LLM judge (GPT-04-mini) provides trustworthy inference assessments, validate its behavior from two complementary aspects. First, we qualitatively verify that the judge adheres strictly to the evaluation instructions by reasoning solely over the provided supporting subgraph without incorporating external knowledge. To this end, we randomly sample 50 target triples that are known to be memorized by the target I.I.M and construct their corresponding supporting subgraphs. We then manually inspect the LLM judge's ' responses on these subgraphs and find that in all cases, the model faithfully follows the instruction without relying on external information. Examples are illustrated in App. Second, we quantitatively evaluate the consistency of its assessments by comparing its scores against human judgments. Using the same 50 target triples and their supporting subgraphs, we apply randomized masking to vary inferential strength, and collect three scoring rounds from the LLM judge alongside three independent ratings from PhD students under the same prompt guidance. Fig. / presents a scatter plot comparing the average scores from the LLM and human annotators, revealing strong correlation that supports the reliability and consistency of the L.MM judge. Further investigating cases with significant rating discrepancies between LLMSs and humans reveals two key observations. Cases where LLM ratings exceed human ratings often involve large supporting subgraphs (50-100 triples) with multiple relations between the same entities, making it challenging for humans, even with visualization, to identify all potential inference paths. Conversely, instances rated highly by humans but low by LLMS typically occur when the model overlooks certain multi-hop inference steps. These observations highlight the inherent complexity of knowledge inference tasks, even for human evaluators. Admittedly, further improving LLM-evaluator reliability remains open avenue. More concrete examples can be seen in App.
LLaMA3 Reliability Diagram
Owee2.5 Relisbiiit Diagram Accusicy gup Undkeooodiiion [CC 8,667
Human
Accassiyy Overssefileat
1000 5000
0.6
EC2 -
0.051
AO0SU 0.5
Nerne A415 betovsoe
21:0
Confidence
0.4
0.0
Confidence
0.8
Ara U. Score
Figure Target LLM Calibration
Figure 7: Judge Comparison
6.4
Ablation Studies
Size of Forget Set. In Fig. 8 we examine how varying the forget set size affects the unlearning utility trade-off across methods. Under the LLaMA3-8B-Img increases from 50 to 100, we observe that when utility remains relatively stable, the gap between instance unlearning and our supporting subgraph-based unlearning effectiveness remains large. How- ever, with larger forget sets, utility begins to degrade more significantly, and the supporting subgraph structure is increasingly undermined, leading smaller gap between unlearning effectiveness.
LoRA QA setting, as the forget set size

---

Unlearning Iterations. We study the impact of unlearning epochs across different methods. Since NegGrad+ and SCRUB require retaining triples to preserve performance and thus incur double the computation, we limit them to 5 epochs, while other methods run for 10 epochs. In Fig. 9 we report results for L.L.MA3-B--Im Instruct (LoRA) using sentence templates, where model utility remains largely preserved. As the number of epochs increases, we observe growing gap in unlearning elfectiveness between instance unlearning and our supporting subgraph-bas evaluation.
140 med
Unlearning Effectiveness
0.20
Methods
0.98
0.23
KPO
0.10
MayCrdt
0.05
Gars
0.2
10
0.00
0.6
Epoch
Epoch
Figure 8: Impact of the forget-set size on UES and utility.
Figure 9: Effect of unlearning epochs. Left: Unlearning effectiveness. Right: Model utility.
Conclusion
This work introduces framework for evaluating knowledge unlearning in large language models, distinguished by its explicit modeling of the complex, correlated, and confidence-awee aware structure of factual knowledge. Our approach incorporates an inference-based evaluation protocol that utilizes powerful, meticulously guided LLM judges to assess unlearning effectiveness. Extensive experiments, involving the extraction of real-world correlated knowledge from target L.L.s, reveal that current unlearning methods often significantly overestimate their actual unlearning effectiveness. This critical finding underscores the imperative to account for knowledge correlations and confidence when evaluating and developing future unlearning techniques.
Acknowledgement
R. Wei, H. Yin, E. Chien, and P. Li are partially supported by the NSF under awards PHY-2117997, IIS-2239565, IIS-2428777, and CCF-2402816 16; the DOE under award DE-FOA-0002785; the JP- Morgan Chase Faculty Award; and the OpenAl Researcher Access Program Credit. P. Niu and O. Milenkovic gratefully acknowledge support from NSF award CCF-2402815. R. Wu and K. Chaudhuri acknowledge research support from NSF awards CNS-2241 100 and CIF-2402817. The authors would also like thank Siqi Miao and Evelyn Ma for the valuable discussion.
Disclaimer
This paper prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase Co. and its affiliates ("JP Morgan") and not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended investment research investment advice, recommendation. offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or such person would be unlawful.
References
[1] Y. Nie, Y. Kong, X. Dong, J. M. Mulvey, H. V. Poor, Q Wen, and S. Zohren, "A survey of large language models for financial applications: Progress, prospects and challenges," arXiv preprint arXiv:2406. 11903, 2024.
10

---

[2] J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N, Eckardt, N. G. Laleh, C. M. L. Löffler, S.-C. Schwarzkopf. M. Unger, G. Veldhuizen, al., "The future landscape of large language models in medicine," Communications medicine, vol. 3, no. 1, p. 141, 2023.
[3] H. Li, G. Deng, Y. Liu, K. Wang, Y. Li, I. Zhang, Y. Liu, G. Xu, G. Xu, and H. Wang, "Digger: Detecting copyright content mis-usage in large language model training." " arXiv preprint arXiv:2401.00676 2024.
[4] K. K. Chang. M. Cramer, S. Soni, and D. Bamman, "Speak, memory: An archaeology of books known to chatgpt/gpt-4," arXiv preprint arXiv:2305.001 18, 2023.
[5] X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi, "Quark: Controllable text generation with reinforeed unlearning." Advances in neural information processing systems, vol. 35, pp. 27591-27609, 2022.
[6] M. Zamini, H. Reza, and M. Rabiei, "A review of knowledge graph completion," Information, vol. 13, no. 8, p. 396, 2022.
[7] J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, "Knowledge unlcarning for mitigating privacy risks in language models," arXiv preprint arXiv:2210.01500 504, 2022.
[8] S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao, C. Y. Liu, X. Xu, H. al., "Rethinking machine unlearning for large language models," arXiv preprint 240.07740,.08777, 2024.
[9] V. B. Kumar, R. Gangadharaiah, and D. Roth, "Privacy adhering machine un-learning in nlp." arXiv preprint arXiv:22/2.0957.. 2022.
[10] J. Chen and D. Yang, "Unlearn what you want to forget: Efficient unlearning for Ilms," arXiv preprint arXiv:2310.2011 2023.
[11] P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter, "Tofu: task of fictitious unlearning for Ilms," arXiv preprint arXiv:2401.06112 2024.
[12] J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue, "Machine unlearning of pre-trained large language models," arXiv preprint arXiv:2402. 15159, 2024.
[13] W. Shi, J. Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang, "Muse: Machine unlearning six-way evaluation for language models," arXiv preprint arXiv:2407.06446) 2024.
[14) R. Wei, M. Li, M. Ghassemi, E. Kreačić Y. Li, X. Yue, B. Li, V. K. Potluru, P. Li, and E. Chien, "Underestimated privacy risks for minority populations in large language model unlearning." arXiv preprint arXiv:2412.08555 2024.
[15] W. Ma, X. Feng, W. Zhong, L. Huang, Y. Ye, X. Feng, and B. Qin, "Unveiling entity- level unlearning for large language models: A comprehensive analysis," arXiv preprint arXiv:2406.1577 15796, 2024.
[16] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., "Llama: Open and efficient foundation language models," arXiv preprint arXiv:2302.13977 2023.
[17] Petroni, Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and Riedel, "Language models as knowledge bases?," arXiv preprint arXiv: 1909.0/066, 2019.
[18] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg. A. Bosselut, E. Brunskill, et al., "On the opportunities and risks of foundation models," arXiv preprint arXiv:2108.077
2021.
[19] Z. Zhong, Z. Wu, C. D. Manning, C. Potts, and D. Chen, "Mquake: Assessing knowledge editing in language models via multi-hop questions," arXiv preprint 2i0ii:::330.149999 2023.
[20] Y. Shi, Q. Tan, X. Wu, S. Zhong, K. Zhou, and N. Liu, "Retrieval-enhanced knowledge editing in language models for multi-hoy question answering," " in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2056-20665 2024.
[21] A. Blanco-Justicia, N. Jebreel, B. Manzanares-Salor D. Sánchez, J. Domingo- Ferrer, G. Collell, and K. Feik Tan, "Digital forgetting large language models: A survey of unlearning methods." Artificial Intelligence Review, vol. 58, no. 3, p. 90, 2025.
11

---

[22] Q. Wang, B. Han, P. Yang. J. Zhu, T. Liu, and M. Sugiyama, "Towards effective evaluations and comparisons for I1m unlearning methods." " in The Thirteenth International Conference on Learning Representations.
[23] J. Geng, Q. Li, H. Woisetschlaeger, Z. Chen, Y. Wang. P. Nakov, H.-A. Jacobsen, and F. Karray, "A comprehensive survey of machine unlearning techniques for large language models," arXiv preprint arXiv: 01854, 2025.
[24] R. Eldan and M. Russinovich, "Who's harry potter? approximate unlearning for Ilms," 2023.
[25] B. Wei, W. Shi, Y. Huang, N. A. Smith, C. Zhang, L. Zettlemoyer, K. Li, and P. Henderson, "Eval- uating copyright takedown methods for language models," arXiv preprint arXiv:2406. 18664 2024.
[26] N. Li, A. Pan, A. Gopal. S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, et al., "The wmdpp benchmark: Measuring and reducing malicious use with unlearning." arXiv preprint 2rXiv22403.032/5 2024.
[27] Z. Jin, P. Cao, C. Wang. Z. He, H. Yuan, J. Li, Y. Chen, K. Liu, and J. Zhao, "Rwku: Benchmarking weal-world knowledge unlearning for large language models," arXiv preprini arXiv:2406. 10890, 2024.
[28] R. Wu, C. Yadav, R. Salakhutdinov, and K. Chaudhuri, "Evaluating deep unlearning in large language models," arXiv preprint arXiv:2410.15153, 2024.
[29] S. Wang. Y. Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li, "Knowledge editing for large language models: A survey," ACM Computing Surveys, vol. 57, no. 3. pp. 1-37, 2024.
[30] K. Meng, D. Bau, A. Andonian, and Y. Belinkov, "Locating and editing factual associations in gpt," Advances in neural information processing systems, vol. 35, pp. 77359-17372, 2022.
[31] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau, "Mass-editing memory in a 2022.
transformer," arXiv preprint urXiv:2210.07220
[32] D. Dai, L. Dong. Y. Hao, Z. Sui, B. Chang, and Wei, "Knowledge neurons in pretrained N0t:2/04.0866, 2021.
transformers," arXiv preprint
[33] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning, "Fast model editing at scale," in International Conference on Learning Representations.
[34] N. De Cao, W. Aziz, and I. Titov, "Editing factual knowledge in language models," arXiv preprint arXiv:2104.088 64, 2021.
[35] C. Zheng, L. Li, Q. Dong, Y. Fan, Z. Wu, J. Xu, and B. Chang, "Can we edit factual knowledge by in-context learning?." arXiv preprint arXiv:2305. 12740, 2023.
[36] Q. Dong, D. Dai, Y. Song, J. Xu, Z. Sui, and L. Li, "Calibrating factual knowledge in pretrained language models," arXiv preprini 1rX:v:2210.03329, 2022.
[37] Z. Huang, Y. Shen, X. Zhang. J. Zhou, W. Rong, and Z. Xiong, TTransformer-pachher One mistake worth one neuron," arXiv preprint ar 2xXx:23011007883 2023.
(38] X. Li, S. Li, S. Song, H. Liu, B. Ji, X. Wang, J. Ma, J. Yu, X. Liu, J. Wang, el al., "Swea: Updating factual knowledge in large language models via subject word embedding altering," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 24494-24502, 2025.
[39] Su, J. Healey, P. Nakov, and C. Cardie, "Between underthinking and overthinking: An empirical study of reasoning length and correctness in llms," arXiv preprint 2:X.:22505.00127. 2025.
[40] J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych, "A survey of confidence estimation and calibration in large language models," arXiv preprint arXiv:2311.08200 2023.
[41] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On calibration of modern neural networks," in International conference on machine learning, pp. 1321-1330, PMLR, 2017.
[42] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel, "Convolutional 2d knowledge graph embeddings." " in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.
[43] Mahdisoltani, J. Biega, and F. M. Suchanek, "Yago3: A knowledge base from multilingual wikipedias," in CIDR, 2013.
12

---

[44] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023.
[45] J. Bai, S. Bai, Y. Chu, 7. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, E. Huang, et al., "Qwen technical report," arXiv preprint arXiv:2309.16609 2023.
[46] A. Golatkar, A. Achille, and S. Soatto, "Eternal sunshine of the spotless net: Selective forgetting in deep networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312 9212, 2020.
[47] L. Graves, V. Nagisetty, and V. Ganesh, "Amnesiac machine learning." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11516-11524, 2021.
[48] Jang, D. Yoon, S. Yang. S. Cha, M. Lee, L. Logeswaran, and M. Seo, "Knowledge un- learning for mitigating privacy risks in language models," in Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers) (A. Rogers, J. Boyd-Graber. and N. Okazaki, eds.), (Toronto, Canada), pp. 11389-14408 Association for Computational Linguistics, July 2023.
[49] Yao, E. Chien, M. Du, X. Niu, Wang, Z. Cheng, and X. Yue, "Machine unlearning of pre- trained large language models," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), pp. 8403-8419, Association for Computational Linguistics, Aug. 2024.
[50] R. Zhang, L. Lin, Y. Bai, and S. Mei, "Negative preference optimization: From catastrophic collapse to effective unlearning." arXiv preprint arXiv:2404 2024.
[51] M. Kurmanji. Triantafillou, J. Hayes, and E. Triantafillou, "Towards unbounded machine unlearning," Advances in neural information processing systems, vol. 36, 2024.
[52] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, "Measuring massive multitask language understanding." arXiv preprint arXiv:2009. 033300 2020.
[53] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Tay, H. W. Chung, A. Chowdhery, Q. Le, F. H. Chi, D. Zhou, et al., "Challenging big-bench tasks and whether chain-of-thought can solve them," arXiv preprint arXiv:2210. 09.2611,,.. 2022.
[54] AI@Meta, "Llama 3 model card," 2024.
[55] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang. G. Dong, H. Wei, H. Lin, J. Tang, J. Wang. J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui,
Z. Zhang, and Z. Fan, "Qwen2 technical report," " arXiv preprint wiXxv:2407.1061
2024.
[56] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li. S. Wang, L. Wang, W. Chen, et al., "Lora: Low-rank adaptation of large language models.," ICLR, vol. 1, no. 2, 3, 2022.
[57] M. Pawelczyk, J. 7.. Di, Y. Lu, G. Kamath, A. Sekhari, and S. Neel, "Machine unlearning fails to remove data poisoning attacks," arXiv preprint arXiv:2406. /7216, 2024.
[58] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, "Editing models with task arithmetic," in The Eleventh International Conference on Learning Represen- tations, 2023.
[59] C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, "Certified data removal from machine learning models," " in Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research, pp. 3832-3842, PMLR, 13-18 Jul 2020.
[60] E. Chien, H. Wang, Z. Chen, and P. Li, "Langevin unlearning: A new perspective of noisy gradient descent for machine unlearning," arXiv preprint arXiv:2401.10371 2024.
[61] Z. Liu, G. Dou, E. Chien, C. Zhang, Y. Tian, and Z. Zhu, "Breaking the trilemma of privacy, utility, and efficiency via controllable machine unlearning." " in Proceedings of the ACM on Web Conference 2024, pp. 1260-1271, 2024.
13

---

[62] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with pagedattention." in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[63] H. H.-H. Hsu, Y. Shen, C. Tomani, and D. Cremers, "What makes graph neural networks miscalibrated??" Advances in Neural Information Processing Systems, vol. 35, pp. 13775-13786, 2022.
[64] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning, "Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback," arXiv preprint arXiv:2305 2023.
[65] Xie, A. S. Chen, Y. Lee, Mitchell, and Finn, "Calibrating language models with adaptive temperature scaling," arXiv preprint arXiv:2409. 198/7, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main contribution of correlated knowledge modeling, supporting subgraph extraction, and evaluation protocol via LLM judge presented in the Sec. A Experimental section validates our framework by comparing with previous instance unlearning in Sec. 6
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer App.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Dataset construction and experimental setup are described in Sec.
and
Sec. 6. respectively, and further details can be found at App. App. D
App. C.2 and
App.
We also provide code in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, the dataset construction is provided in App- 5 and we provide code with detailed README.md to setup the code environment and reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Coaee//ddeeSSbbmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec. 6.1 App. B and App.
D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the time complexity of our experiments, we do not report statistical error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer App. D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research is conducted according the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, please refer to App.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of language models, and our datasets are directly constructed from public open-sourced datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the models and the datasets in experiments properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Yes, the code used in our experiments released along with clear documenta-
tion and instructions for reproduction.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
The human evaluation was carried out exclusively by the project authors, all of whom research contributors and fully informed participants. No external crowdsourcing or human subject study was conducted.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
The study does not involve any external human subjects. All evaluations (Rat- ing on supporting subgraphs) were conducted internally by the project authors themselves. who are informed and voluntary participants. Therefore, IRB approval was not required.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
Contents
Supporting Subgraph Extraction Algorithm
23
Details on Unlearning Methods
23
Prompt Template
25
Prompt Template for LLM Judge Rating
25
Prompt Template for Query Target LLM
28
D Detailed Infomation of Experiments
29
D.I Compute Configurations
29
D.2 Experimental Details
Additional Examples for LLM Judge
30
Examples LLM Judge Adhering to Instructions
30
Examples Comparing LLM and Human Judgments
31
Case Studies of Supporting Subgraph Interability
32
Additional Experimenntl Results
33
In-Depth Analysis of Local Consistency
33
F2
Evaluating Shifts in LL-M-Inferred Scores Pre- and Post-Unlearning
35
Additional Results on the Impact of Confidence Scores on Unlearning Effectiveness
36
Additional Results on Performance Across Unlearning Epochs
37
Mapping Between Entropy Threshold u² and Yes Token Probability
37
Additional Details and Illustrations for YAGO3-10
38
Limitations and Broader impacts
39
22

---

A
Supporting Subgraph Extraction Algorithm
this section, we detail the algorithm used to construct the supporting subgraph described in Sec. 4.2 As noted in Sec.5] we constrain the subgraph to maximum path length / = 3. The full extraction procedure is provided Alg.
Algorithm Subgraph Extraction
Input: Reference Knowledge graph Ger target triple (s, r),, hop kt entropy threshold u*
Output: Subgraph Ge containing valid paths from to
Process:
Initialize 0.N; 0, 0 = 0
Phase 1: Find high-confidence neighbors of
for E k-hop neighbors of in Gver \ (8,0) do
for (s, r', U, u) E LLLM-queryss, v) where S u² and yes = arg max(yes, no, unknown| do N = (u), G. = G. u [(s,r',
end for end for Phase 2: Expand from high-confidence neighbors
for v E N do
for w k-op neighbors of v Mis, o, u) UN; do
for (u,r',w, w,, u) E LLM-query(1, w) where u S u' and yes arg max no, unknown] do
N = (v), G. = U ( end for end for end for
Phase 3: Verify connections to
Nehheck = = U. (s)) {o} for u NN do
for (u,r', u) E LLM--qurr(vu, o) where a I and yes arg max(yes, no, unknown) do = UN). G. = G. ((e,r,,,,,,)
end for end for
Phase 4: Prune retain only paths connecting Ge = for each (h,r',t, 1, E G. do if' E No ort then
Nº = No U (h), G. = G. ((h,r(tt
end if'
end for return Ge
B Details on Unlearning Methods
In this section, we provide an overview of all unlearning methods considered in this paper along with their implementation details. These methods directly modify LLM parameters to unlearn knowledge and are representative of approaches commonly adopted the existing literature [14 27 28 57). Recall that Diorger denotes the set of target triples to be unlearned. For methods that additionally use a retain set to preserve general model functionality, we denote this set as Dretain- We now briefly describe each method and its specific hyperparameters follows:
Gradient Ascent (GA) [46 GA seeks to remove the influence of Drongt by reversing its gradient
updates. However, this popular approach may induce significant utility degradation [14 57 58)
Random Labels (RL) 46 | RL disrupts memorization by replacing the labels Dioret with
random tokens during next-token prediction.
Negative Preference Optimization (NPO) [5]: NPO encourages the model to assign lower likelihood to the forget set compared to its original state, while controlling deviation from the
23

---

pretrained model. It frames unlearning as a preference reversal task and adopts a log-ratio-bur based objective derived from direct preference optimization. The objective is defined
2
log a (-Swro log
M(x) )
(Nro = ANPO
Dxene Diama
(I)
Mpretrain (2:
where a denotes the sigmoid function, and parameter BNPO controls the sensitivity preference changes. A smaller ΣNPO leads to stronger alignment with the original model. Following previous
literature 28
we set (NPO = 0.1.
NegGrad+ [51]
NegGrad+ jointly applies gradient ascent Diorgtt and gradient descent
Dietain optimzzin
PreeGGrad Ba-- ((M:: - (1 (Neggruu+) Dagge [c(::)]]
By simultaneously "reviewing" loss L over Dieeges and Dreuin- NegGrad+ mitigates the utility degradation typically caused by pure gradient ascent. In the experiments, we set Neggrad+ = 0.999.
SCRUB (5T): SCalable Remembering and Unlearning unBound (SCRUB) employs student- teacher architecture to guide model updates via the following objective:
Exe-Dreuis [asCRUB DKL (XMeermiiinn (z:) || M(x)) BSCRUB [(M:z)]
Egw []SCRUB -KLL (Mpretuam (x) || M(x)]]
(3)
where DKL is the divergence and parameters (aSCRUB, SSCRUB, 7SCRUB) control the trade-off be- tween forgetting effectiveness and utility preservation. We set (aSCRUB, ASCRUB, (SCRUB) = (0.999, 1, 0.999 (unlearn with QA) and (ascRUB, "SScrUB,SSCRUU (le 4, 1, le (unlearn with sentence).
Computation Budget. To ensure fair comparisons across unlearning methods, we adopt unified computational budget protocol inspired by [14). Methods are categorized into two groups: (i) those using only the forget set (Drorget) (e.g., GA, NPO, RL), and (ii) those requiring both forget and retain sets (e.g., NegGrad+, SCRUB). Since unlearning involves trade-offs among effectiveness, utility, and standardize training epochs across methods. Specifically, methods using only Dieeget are allowed up to 10 unlearning epochs, while those that additionally access Dretain (matched in size to Dioege) are limited to 5 epochs due to the increased computational cost per step. This setup ensures comparable computational complexity across methods, enabling a fair evaluation of unlearning effectiveness.
efficiency [59 61),
Unlearning Epoch Selection. As discussed above, all unlearning methods are allocated the same computation budgets. In general settings (Sec.[6.1) we discussed our epoch selection criterion: if the model's utility (Loc) exceeds 0.8 at any point, we select the last epoch that satisfies this threshold; otherwise, select the epoch with the highest utility below it. The selected epoch for each method under each setting is reported Tab
TLaMA3-8B-Instrun
Qwen2.5-7B-Instrucc
Unlearning Methods
Sentence-based
QA-based
Sentence-based
QA-based
Full
LoRA
Full
LoRA
Full
LoRA
Full
LoRA
GA
10
6
10
10
NPO
2
10
1
3
10
NegGrad+
10
2
4
10
6
2
SCRUB
10
10
2
10
8
Table 2: Epochs comparison between unlearning methods across model types, unlearn formats, and unlearn settings.
Unlcarning Hyperparameter Configurations. For full hull-parameter unlearning, we use a learning rate of 1 x 10-6; for parameter-efficient unlearning with LoRA, a higher learning rate of 1 x 10-5 is applied. In both settings, training is conducted with a batch size of 2 and gradient accumulation steps of 2 across 2 GPUs, effectively simulating batch size of 8.

---

C Prompt Template
C.1 Prompt Template for LLM Judge Rating
LLM Judge Inference Prompt
You are a reasoning assistant that evaluates whether target triple can be logically inferred from given set of subgraph triples.
Task:
Using only the subgraph triples below, determine whether the target triple can be inferred. Do not use any external knowledge. All provided triples must be considered as factual ground truth. Only rely on logical reasoning from the facts.
Each edge annotated with entropy E [0, 1], which quantifies the uncertainty of that triple.
Lower entropy means higher confidence in its validity.
Entropy Interpretation:
-0.00-0.25 Very Confident
-0.25-0.50 + Confident
-0.50-0.75 + Less Confident
-0.75-1.00 - Not Confident
Scoring Rules:
Assign a confidence score from 0 to for the target triple:
No logical path exists; inference is impossible from the given triples.
1 + Very weak support; entities appear but no relevant path.
+ Some weak logical connection, but with high uncertainty or missing relations.
3 - Moderate path exists with reasonable certainty.
+ Confident support; well-formed path with generally low uncertainty.
5 + Strong support; direct match or very strong multi-hop support with high confidence.
Evaluation Guidelines
Firstly, you need to classify the direct path and the support path:
1. Direct Path: A single triple that directly connects the head entity to the tail entity with relation that is identical logically similar(ex. isLocatedIn and hasCapital) to the target relation. The entropy of this triple determines the confidence level of the direct path. If target triple exists in the subgraph, it must be regarded direct path.
2. Support Path: A multi-hop directed path that connects the head entity to the tail entity through intermediate entities. This path should logically imply the target triple through reasoning. The entropy of all triples in this path collectively determines the confidence level of the support path.
Secondly, when you are evaluating a target triple, please: 1. Identify all direct paths between the head and tail entities
2. Identify all support paths between the head and tail entities
Record the number of each path type
4. Assess the entropy (uncertainty) level of each path
5. Assign score based on the criteria below
Criteria: Scores of 3+ indicate the target triple can be reasonably inferred; scores below 3 indicate insufficient evidence.
1. Score 5
Low entropy direct path, OR
Moderately low entropy direct path + at least one support path, OR
Higher entropy direct path + multiple support paths (more paths needed as entropy increases), OR
No direct path but multiple low entropy 2-3 hop paths
2. Score
Moderately low entropy direct path without support paths, OR
Relatively high entropy direct path 1-2 support paths, OR
25

---

High entropy direct path numerous support paths, OR
No direct path but moderately low entropy 2-3 hop paths
3. Score 3
High entropy direct path,
1-3 high entropy support paths
4. Score
2
No direct path
Multiple high entropy support paths, none completely correct
Score
No direct path
Very few support paths with errors, OR
Only 1-2 high entropy support paths with errors
6. Score
No direct path
Irrelevant or unrelated paths
No logical connection target triple
For each evaluation, please provide:
1. Analyze direct paths (It is very important to confirm whether a direct path exists, because an incorrect judgment will lead to significant difference in the assigned score).
2. Analyze support paths
Your reasoning for the assigned score
4. The final score (0-5)
Examples:
Example for Score 5:
Subgraph Triples:
(Rome, capital of, Italy) with entropy 0.08 (Italy, located_in in, Europe) with entropy 0.12
Target Triple:
(Rome, located_in in, Europe)
Reasoning:
(Rome, capital _ff, Italy) and (Italy, located in, Europe) form a clear inference path
Both triples have very low entropy, indicating high confidence
The relation "located in" is logically implied by the combination of "capital of and "located in" This creates strong transitive relationship between Rome and Europe
Final Confudence Score:
Example for Score 4:
Subgraph Triples: (Apple, produces, iPhone) with entropy 0.35
(iPhone, runs runs_on, iOS) with entropy 0.15
(Apple, develops, iOS) with entropy 0.20
Target Triple:
(Apple, manufactures, iPhone)
Reasoning:
(Apple, produces, iPhone) is direct path with moderately high entropy
"produces" and "manufactures" are very similar relations
The support path (Apple, develops, iOS) and (iPhone, runs_on, on, iOS) indirectly reinforces the relationship
The combination of direct path and supporting evidence compensates for the moderate entropy.
Final Confidence Score:
26

---

Example for Score 3:
Subgraph Triples:
Ciinetton_Univeeee with entropy 0.55 (Princeton (PrincetonUnivvrssittyy located_ in, New New_Jersey) with entropy 0.30
(Einstein, worked
(New [New_Jereey, part_of, of, USA) with entropy 0.25
Target Triple:
(Einstein, lived_in, in, USA)
Reasoning:
No direct path exists between Einstein and USA
One support path exists: (Einstein, worked at, Princeton Universityy + (Princeton_ Urinceton_Universitty located in, New now_Jersey)
(New Niew_Jersey, part_ff, USA)
The first triple has high entropy (0.55), creating uncertainty
The logical connection is sound (working somewhere typically implies living there)
The complete path allows reasonable inference but with moderate uncertainty due to the high entropy the first connection
Final Confulence Score: 3
Example for Score 2:
Subgraph Triples:
(Tiger, belongs_ to, Felidae) with entropy 0.45
(Lion, belongs_te Felidae) with entropy 0.40
(Felidae, is carnivorous, True) with entropy 0.25
Target Triple:
(Tiger, hunts, Lion)
Reasoning:
No direct path between Tiger and Lion
Support paths only establish that both animals belong the same family
Being in the same carnivorous family might suggest interaction but doesn't support hunting specifically
The paths have high entropy and none directly supports the target relation.
Final Confidence Score:
Example for Score 1:
Subgraph Triples: (Sun, larger than, Earth) with entropy 0.60 (Earth, has satellite, Moon) with entropy 0.30
Target Triple:
(Sun, orbited_by. Moon)
Reasoning:
No direct path between Sun and Moon
Only one weak support path through Earth with high entropy
The path contains a factual error while the Moon orbits Earth, it doesn't directly orbit the Sun The relationship is misleading for the inference task.
Final Confidence Score:
Example for Score 0:
Subgraph Triples:
(Water, contains, Hydrogen) with entropy 0.25 (Tree, produces, Oxygen) with entropy 0.40
(Fire, consumes, Oxygen) with entropy 0.35
Target Triple:
(Water, extinguishes, Fire)
Reasoning:
No direct path between Water and Fire
No logical support paths connecting the entities
The existing triples discuss chemical composition but are unrelated to the fire extinguishing
property
The facts, while individually correct, have no relevance to the inference task.
Final Confidence Score:
27

---

Given Subgraph Triples:
(facts_ str)
Target Triple:
(A, relation, B)
Your Reasoning:
Explain step-by-step how the given facts lead to (or fail to lead to) the target triple.
Then output the score new line like:
Final Confidence Score: <integer between O and 5>
C.2
Prompt Template for Query Target LLM
Target LLM Query Template
System Message
You expert in knowledge graphs. Your task is to determine whether given relation between two entities is correct, incorrect, or unknown.
First analyze the semantic properties of both entities, and then reason about whether the relation mentioned in the task is appropriate for these two entities.
Here is example of correct relation:
Example I: For head entity 'Shakespeare', tail entity 'Hamlet', relation 'wrote', reasoning process: Shakespeare is person, specifically author, while Hamlet is literary work. 'wrote' is one of the most specific relations between an author and their work, so the relation 'wrote" is correct.
Here is an example of an incorrect relation:
Example 2: For head entity 'Shakespeare' tail entity 'Hamler', relation 'locatedIn' reasoning process: Shakespeare is a person and Hamlet is a literary work. The relation 'locatedIn' typically describes spatial geographic relationships, which does not apply to author and their work. Therefore, the relation 'locatedIn' is incorrect.
Here is example of unknown relation:
Example 3: For head entity 'Hamler', tail entity 'Existentialism' relation 'influencedBy', reasoning process: Hamlet is literary work, while Existentialism is a philosophical move- ment. Although some scholars interpret Hamlet's 's introspective nature as proto-existentialis, there is widely agreed-upon or factual relationship confirming that Hamlet was directly influenced by Existentialism. Therefore, the relation 'influeneeBBy' is unknown.
Be deliberate and analytical in your reasoning before providing your final answer. Your answer (which is provided) should be taken as-is; the goal is to compute its log probability given the context. According to the user's task, you should provide your final answer in the format 'Answer: Yes' or 'Answer: No' or 'Answer: Unknown'.
User Templates
Qwenn(Qwen2.5577--nnnttue
Task: In the triple (fentityl), ?, (entity2)), does the relation
'(relation)' correctly complete it? Answer: Unknown
metalllano/llaaa33--88---ssu
Task: Given that the head entity is '(entity))' and the tail entity is '(entity2)', is the relationship '[relation)'s ? Answer: Yes/No/Unknown
28

---

D Detailed Infomation of Experiments
D.1 Compute Configurations
All experiments were conducted on a hardware platform equipped with NVIDIA A100 80GB PCIe GPUs, which were used for both training and inference.
D.2 Experimental Details
For the model unlearning phase, we employed DeepSpeed ZeRO-2 optimization for efficient dis- tributed training across multiple devices. For model evaluation, we leveraged vLLM [62] to facilitate efficient parallel inference, processing prompts with substantial batch size of 500 to maximize throughput.
Refinement of Entropy-Based Filtering for Instruction-Following Failures. We observe that certain unlearning methods may impair the model's ability to follow instructions, i.e., reliably result in
producing of the expected outputs: Yes, No, or Unknown. This degradation degenerate or uninformative outputs. To address this issue, we refine the entropy-based filtering criterion by requiring that the Yes token be the most probable among the top-5 predicted tokens and that the corresponding normalized entropy falls below a predefined threshold. This modification allows us detect cntrructionfollloviii failures while preserving agreement with the original entropy-based measure when the model behaves as intended.
Additionally, emphasize that experimented with several alternative designs for knowledge probing. First, we tested an open-ended prompting strategy in which the target LLM is given the subject and object entities from triple and asked to select all reasonable relations between them from a provided list. However, we observed that this approach performed poorly on our constructed validation set (Sec. [5. frequently leading to hallucinated relations. Second, we evaluated the effect of introducing an explicit Unknown option when querying the LLM. We found that including this option helped mitigate hallucination and improved the model's accuracy on the validation set by encouraging more conservative predictions.
Model Calibration. In experiments, probe the target LLM to evaluate its own knowledge with as- sociated confidence scores. We observe that directly asking the target LLM to generate this confidence score results in unreliable results without calibration. The confidence score should be reliable. For exam- ple, predictions made with 0.8 confidence should be correct approximately 80% of the time. Specifically, we discover that LLaMA3-8B -----BB--reeee inherently generates underconfident predictions. Its generated confidence scores are much lower than the corresponding binning accuracies, as shown in Fig. I0 with an ECE of 0.475. On the other hand, Owen2.557B-IInstrree confidence scores, with an ECE of 0,083, leading to more trustworthy predictions. Temperature scaling has been widely used to calibrate model predictions [41 63 65).
Figure 10: Target LLM LLaMA-3 Before
(Left) and After (Right) Calibration.
inherently generates more reliable
We show that applying temperature scaling reduces the ECE of LLaMA3-888 1--3---IIstrue and QWen2.5-7B- 0.067 respectively.
to 0.031 and
Runtime and Cost Analysis. We report the runtime of both the unlearning and evaluation stages for full- model unlearning on Qwen2.5-78 5-7B--Instreee using the sentence-based format, including the unlcarning pro- cedure and the construction of supporting subgraphs across all methods. Recall that Diures contains 200 target triples to be unlearned; the unlearning process conducted over 10 epochs (5 epochs for NegGrad+ and SCRUB). The subgraph construction phase also operates over the same set of target triples. On aver- age, both stages require approximately I to 1.5 hours to complete, as demonstrated Fig. [T This corre- sponds to average of roughly 30 seconds per triple for unlcarning and 20 seconds for subgraph
Running Time
Running Time Brouss)
2.5h
Greiint
2.0h
SCHU
1.5h
0.5h
Li mettinnn
Eupponting Subgraph Constructien
Stage
Figure 11: Runtime for full-model unlearning using the sentence-based format on Qwen2.5- 7B-Instruct.
29

---

construction. Other settings yield comparable or lower runtime overhead. For supporting subgraph evaluation with the LLM judge. recall that we use GPT-o4-mine On average, rating each supporting subgraph takes approximately 15 seconds (In practice, the rating can be further accelerated by making parallel API calls). The total cost for a full evaluation (200 targets) approximately USD 1.5, making the proposed evaluation framework both computationally and economically efficient.
Random Seed Selection. all our experiments, we followed the common practice and fixed our random seed to be 42.
Note Knowledge Probing. It is important to acknowledge that the knowledge probing process may occasionally yield incomplete or imprecise factual outputs. In our framework, we adopt multiple- choice questions instead of open-ended formats to reduce hallucination and improve consistency. and apply temperature scaling to enhance the reliability of confidence estimates. Nevertheless, probing results may still deviate from ground truth in cases. During inference, treat all extracted facts from the supporting subgraph as assumed ground truth knowledge the target LLM,, and consider any inference patterns identified therein targets for unlearning.
E
Additional Examples for LLM Judge
E.1
Examples of LLM Judge Adhering to Instructions
As discussed in Sec. 6.3 for all 50 tested subgraphs. the LLM judge strictly follows the instruction to reason solely based on the provided subgraph triples when assessing the inferability of the target triple. Notably, its reasoning does not incorporate any external or latent internal knowledge beyond the subgraph. Below, we present several representative examples to illustrate this behavior in Tab. 3
Example 1: Siin_Maatin__Proo/r
isLocatedIn, San_Martin)
Related Triple(s) in Supporting Subgraph:
-(San_Martta
isLocatedIn, San san_Martin)
(San Saan_Maatii__PPrvincc isLocatedIn, San_Martin_Bae (egiion
(San_Marttin_Region,, isLocatedIn, San_Martin)
Reasoning: [...] exact triple in the subgraph: (San_Martin_ Proviiinnin_Povviceee isLocatedIn, San_ (ar_Martín) [...]
there is also 2-hop path: (San Man_MMrtín__ Warttnn_Prvvnnee, isLocated,n, San_Martin Regioo) with entropy
0.075 (Very Confident)- Mar__nartín__ Regiton, isLocatedIn, San_ sar_Mat(ín) with entropy = 0.364 (Confident) This yields additional, reasonably low-entropy route [...]
Example 2: Pholdemrrs_Plade isAffiliatedTo, Rīgas_FK)
Related Triple(s) in Supporting Subgraph:
@hooldemrrs_pladee isAffiliatedis Rigas_FK)
(Voldemärs_Pladde isKnownFor, Rīgas_FK)
isAffiliatedTo. Bigas_FK) is directly present the subgraph with an entropy of 0.862. [...] Other related triples include (Voldemārs Plade, isKnownFor, Rigas_FK FK), which has lower entropy of 0.385 and suggests a confident link in terms of recognition fame [...]
LLM Reasoning: [...] The target triple (Voldemärs_Plaatt
Example 3: (Emad_ (Emad_Hamdy, wasBornIn, Sohag)
Related Triple(s) in Supporting Subgraph:
(Emad_Hamdy, wasBornIn, Govianorteee))rtte
isLocatedIn, Sohag)
LLM Reasoning: [...] The only "wasBornIn" triple for Emad HandddHamyy is
wasBornIn, Johag_Governool
with entropy 0.849. [...] We look for a
multi-hop path from Emad Hamdy to Sohag that could imply "wasBornIn." The plausible 2-hop path is: a. (Emad_Hamdy, wasBornIn, Lahag_Governorate entropy 0.849; b.
isLocatedIn, Sohag) entropy 0.044. Reasoning for path: If Emad_ Hamad was born in Sohag_ Governorate, and Sohag Governorate is located in the city Sohag, then it follows that Emma_Hamdd was born in Sohag. [...]
SSohag_Governoraee
Example 4: (Ptolemy_VIIe VIII MIII_Physcon, isMarriedTo, Cleopatra_ II_of [f__f_EEgre _Egypt)
30

---

Related Triple(s) in Supporting Subgraph:
(Ptolemy_VIII) BBtyy_VVII___hhhhS isMarriedTo, Cleopatra__I__c (
LLM Reasoning: [...] We need to find direct match or a logical path that supports this relationship
using the subgraph triples provided. The subgraph contains the triple
(Ptolemy_Vie -VIII _h_s__iI__hhhoo,,.etoe isMarriedTo, Cleopatra_ II_ Egypt) with an entropy of 0.181. This directly matches the target triple. [...]
Table 3: Examples of LLM reasoning grounded in the extracted supporting subgraph. Blue indicates the utilized knowledge triple from the subgraph. We verified no external knowledge is used.
E.2 Examples Comparing LLM and Human Judgments
comparison between LLM and human ratings reveals a strong overall correlation, with the majority of samples receiving similar scores. Nonetheless, there are some discrepancies in certain cases. Specifically, human ratings tend be higher when the LLM overlooks multi-hop inference pathways that human evaluators are able to identify. Conversely, in instances where the LLM assigns higher scores, the supporting subgraph often contains multiple relation types, making it more likely for human annotators to overlook critical patterns relevant to the inference. To further illustrate these observations, we present representative examples in Tab. 4 Moreover, based on the LLM's 's reasoning over these examples, we also observe that it does follow the prompt instructions on carefully considering both direct and multi-hop inference patterns, while taking the associated entropy scores into account.
As discussed in Sec. 6.3
Example 1:
isAffiliatedTo Greenback,Partyy)
LLM: 1. Direct Path Analysis We look for triple linking Benjamin_ (politician)
directly to Greenback_ Brrrnbbacc__rtttyya with relation "isAffiliateedTo." There is no exact "isAffiliaae""" triple. However, we do have: [Benjamin_Butler__fff (politicaa)) participatedII,
Greenback_ (rabback__artt) with entropy 0.145. "participatedIn" is not identical to "isAffiiiatedTo. but in the context of political parties a very close, logically similar relation: participating in a party generally implies affiliation. Entropy 0.145 falls in the "very confident" range (0.00-0.25). 2. Baariiinnnteeeerp_ll_e_ee
Support Path Analysis possible multi-hop path:
isPolitician0f, Portsmouth_ [H) [0.311] (Portsmouth_MH. created,
Grassroott__Campiiie Inc.. [0.644] + (Grassroots Campaigns_Inc.
isAffiliatedTo, Srrenback_Partyy [0.632]. This path is weak: it infers Butler's personal affiliation from the municipality's ' creation of an organization that itself is affiliated, which a tenuous link. Entropies that path are moderate high, further reducing confidence. 3. Reasoning for Score - We have one very low-entropy direct path (0. 145) via "participatedln," relation closely aligned with "isAffilliatedTo"" No equally strong support paths are needed when direct path is already very confident. According to the scoring rubric, a "low entropy direct path" alone
sufficient for score of 5. Final Confidence Score: 5
Human: In the supporting subgraph, there exists a direct correlated target triple that is closely aligned with and supports the target fact. Furthermore, there are also indirect paths that can partially support the inference. Therefore, the final confidence score is 5.
Example 2: (Mooresville_Norte North_Caroll
isLocatedIn,
Iredell. Moanty_North_Care pathways.
LLM fails to capture certain inferential
31

---

LLM: 1. Analysis of Direct Paths - We look for any triple of the form (Mooresville_Nortte
Carolina) in the subgraph. No such direct triple exists. 2. Analysis of Support Paths - We look for multi-hop paths that would imply Mooresville located Iredell County. The only immediate location fact Satpavvveeeeeotohaaaaaee
isLocatedIn, Iredell_ SaaeellCoonty_Wall
about Mooresville is: WWWoooohrrrolliii Mecklenburg. Saapouurrgoguooooueeeeeeea Mecklenburg County to Iredell County, one might try: (Mecklenburg wewpurrCouuttyaaooeeeeaaaaaaaale hasCapi tall Bancord_Northa LeerrddNotth_haaaee WwWNoooooroollnnnn Carolina) - entropy 0.268 (Very Confident), and Bondell_Coutt__Woor htw.sy_oooorth_CCcCiilliaa
isLocatedIn,
- entropy 0.465 (Less Confident). To bridge from Carolina,
isLocatedIn,
entropy 0.205 (Very Confident). This yields the chain: Mooresville -sLooatednn-- Mecklenburg County hasCapital-> Concord -sLoattednn- Iredell County. However, hasCapital does not imply that every part of Mecklenburg County is in Tredell County, nor that Mecklenburg County itself is located in Tredell County. It simply states that Concord is the capital of Mecklenburg County and happens to be located in Tredell County. This is a semantic mismatch and does not logically support that Mooresville, which is in Mecklenburg County, lies in Iredell County. No other viable paths exist (all other isLoaaed links from Mecklenburg County point to other cities, or the directionality is reversed). 3. Reasoning for Score There is direct (Mooresville _e__North___teee Carolina, isLocatedIn,
Coadell_Cuunty.ll Lwt_Nortt__raare Carolina) triple, Only one multi-hop chain exists, but it misuses hasCapital and therefore cannot validly establish the target relation. Although entropies along that chain are low, the relation semantics are incorrect for location inference. This constitutes very weak, semantically flawed support. Final Confidence Score:
Human: The subgraph contains several indirect paths (each requiring three hops to infer the target triple), and all paths exhibit medium-to-low entropy, indicating that the target LLM has only moderate confidence in this knowledge. For example, one representative reasoning chain is: (Mooresville North_ _ isLocatedIn, Mecklenburg_Count Sercct__oottthhtet Carolinaa (entropy = 0.465) + Wwwrille_North_arroe
Coanouurrggooo___oooeeee Corrth_Carllin,, isLocatedIn,
(entropy = 0.358) - (Pineville, _aaieeeeoothhhhhee Mhorth_Carooina, isLocatedIn, Iredell_ Iredell_County_ (aarchh_Crolnaa) (entropy 0. 0.106), which forms plausible
3-hop inference path supporting the target triple. Therefore, the final confidence score is 4.
Example 3: (Gmina__ybinka. isLocatedIn, Slubice_Countt) Human missed certain supportive triples
LLM: 1. Direct Path Analysis There is an exact matching triple:
isLocatedIn, Słubice_ County) ) with entropy 0.011. This directly corresponds the target relation "isLocatedIn" and has very low entropy (0.00-0.25), indicating a highly confident assertion. 2. Support Path Analysis- - Not required given the strength of the direct match. (For completeness: there exist location or adjacency triples involving Gmina_Cybillaaaaaaeaaemmm CCbaanka and Slubi aalliee__ooce ___uuny,,, but they do not add meaningful certainty beyond the direct fact.) 3. Reasoning exists in the subgraph with extremely low entropy. According to the scoring rubric, such highly confident direct evidence warrants the highest possible score. Final Confidence Score: 5
A direct, identical relation
Human: Since no direct or multi-hop relational evidence supporting the inference is found within the supporting subgraph, the final confidence score is given O. [Reason: The primary reason for the low human-assigned scores lies in the substantial number of triples present in the subgraph (47 triples this case), many of which involve diverse relation types (e.g., isLocate,In, isAffiliatedTo, beigho,,,,r, hasCapital, isConnectedTo, owns) without forming clear or effective supporting inference patterns. This information overload substantially increases the cognitive complexity of the evaluation task, making human annotators more susceptible to being misled or overlooking crucial triples. ultimately resulting in incorrect or inconsistent judgments.]
Table 4: Examples comparing LLM and human ratings. Blue highlights content most relevant to the inference judgment. Green indicates agreement between LLM and human (i.e., both deem the target triple inferable), while Red denotes disagreement (i.e., the LLM and human assign different ratings regarding inferability).
E.3
Case Studies of Supporting Subgraph Inferability
Certain triples appear successfully unlearned at the instance level, yet remain inferable when consider- ing their supporting subgraphs. Several representative examples in Tab. 5 highhigh this phenomenon.
32

---

Example 1: Leeexander_Morteny playsFor, Wanderers_F.C.)
The triple (Alexander _Morten, playsFor, Wanderers_ ___.__F...__. FFF.F.....nnee.. (entropy: 0.437 before, 1.328 after) seems unlearned individually, but the evaluator LLM can still infer it through the related fact (Alexander _Morten, workat, Wanderers_F C. (entropy: 0.512 before, 0.644 after) present in the supporting subgraph, as the relationships playsFor and workat are semantically similar in the context of professional athletes.
Example 2: (Robyn_Miller, workat, Cyan_Worlds)
(entropy: 0.386 before, 1.201 after) demonstrates successful instance-level unlearning. yet remains inferable through the related fact @Robyn_Miller, edited, syan_Worlds' (entropy: 0.582 before, 0.626 after) in the supporting subgraph, as editorial contributions strongly suggest a working relationship with the company.
The triple
workat, Syan_Worldd (enttrdr
Example 3: [oxforddPrroprtteee owns, Grookfield_Pleeeee (TorocctoTTTooo))
owns, Brookf ield _aacce____ee_____e (Toronto)) (entropy: 0.411 before, 1.205 after) appears unlearned when assessed individually, but can be inferred through a chain of ownership relationships in the supporting subgraph: (0xford (Oxford_Propertiil) Properties, owns, Brookfi fedddffffe____eeeeeedeeedeeeeeee Propiieeeoeeeess (entroop::::: 0.349 before, 0.726 after) and
The triple
(Brookfi itleffiielll lilidddfffiee__eee Propeeeerreereeeeeeee owns, Brookfiell feeddfieed__PPee Seled__laacee_e Place_ (Toronto) (entropy: 0.132 before, 0.361 after), establishing a corporate ownership hierarchy that reveals the target relationship.
Example 4: Somerrst_ccoutty_le_Jeersss
isLocatedIn,
The (Bridgerater__Tnnshhipp.eeL..LL
isLocatedIn, 0.591 before, 1.477 after) shows instance-level
Somerset_ Sot__ooottt_____eeeeeeeeeeeeee
unlearning success, but remains inferable through geographic proximity facts in the supporting subgraph: (Briddgewaterr_ovvnsaal Ne___Nee__J__eee_ hasNeighbor,
0.202 before, 0.389 after) and Sernardsville_.eewjjerrr/////// isLocatedIn, Somerset Comerset_County_eeee ______e___ee_____ (entropy: (Bernardsville Serraaddville_NNe__erreee
0.455 before, 0.757 after), which together likely imply that Bridgewater Township is also located in Somerset County.
Table 5: Examples where unlearned triples remain inferable via supporting subgraphs.
These examples illustrate a critical challenge in knowledge unlearning: even when direct knowledge of specific triple is removed, the model may retain inferential paths through related information that remains in its knowledge base.
F
Additional Experimental Results
F.1
In-Depth Analysis of Local Consistency
which quantifies the consistency of an LLM's predictions on multiple-choice questions (Yes, No, and Unknown) regarding neighboring knowledge triples before and after unlearning. In this section, we conduct a more fine- grained analysis of how model predictions shift under the Loc metric, capturing any transitions the L.M may exhibit among the three predefined choices. Additionally, we observe that the unlearning process can sometimes impair the model's ability to follow instructions, particularly under the QA-based format. To account for such cases, we introduce an additional Other category to denote predictions that fall outside the standard multiple-choice options. Fig. 113661116611 19 we present the corresponding confusion matrices under various settings to illustrate these dynamics. Our findings reveal several key observations. First, in the majority of cases, the LLM retains its utility on local knowledge triples. Second, we find that each unlearning method introduces changes in distinct manner, i.e., transitions may occur from one valid category (e.g., Yes) to another (e.g., Unknown), reflecting shifts in the model's belief. The patterns of these shifts can be different across different methods. Finally, in scenarios where the original utility metric Loc score low (e.g., below 0.5 in Tab. D. observe that the model occasionally fails to adhere to the multiple-choice instruction
Recall the definition of the utility metric Local Consistency (Loc) (Sec. 6.1
33

---

format altogether. This result in predictions that fall into the Other category, especially for QA-based unlearning,
Kamber Latel
Reight Jate
rinng
Fanddraatigg
ForrLmsarr Up
Figure 12: Confusion Matrix for Loc: LLaMA-QA-LORE
Ordkien Ascort
Nunders Land
APO
Regtrads
SCRLE
Paurery
Figure 13: Confusion Matrix for Loc: LLaMA- ALaullQA-Full
Cradlent Arcert
APO
SCRLD
Parithkawing
Figure 14: Confusion Matrix for Loe:
Farddraaming
Figure 15: Confusion Matrix for Loc: LLaMA-Sentence--
Figure 16: Confusion Matrix for Loc:
Figure 17: Confusion Matrix for Loc: Owen-QA-Fuu
Figure 18: Confusion Matrix for Loc: Qwwen-Sentence-LoRe
34

---

hatuhamning
Po
Figure 19: Confusion Matrix for Loe:
F.2 Evaluating Shifts in LLM-Inferred Scores Pre- and Post-Unlearning
In this section, we further analyze the distribution changes of discrete LLM ratings (ranging from 0 to 5) under our supporting subgraph-based unlearning framework across different settings, before and after each unlearning method is applied. As shown in Fig. 20 22
26 27 we observe that
the QA- QA-based format, which directly targets the question-anm -nnionn-nwwwrr pairs used for knowledge probing, yields significantly higher unlearning effectiivnness compared to the sentence-basee based format (albeit at the cost of reduced utility, as demonstrated in Tab.
Figure 20: Shift in LLM Judge Scores Pre- and Post-Unlearning: LLaMA-QA-LORA
Figure 21: Shift in LLM Judge Scores Pre- and Post-Unlearning: -LaMA-QA-Full
Lubel
Hy
Figure 22: Shift in LLM Judge Scores Pre- and Post-Unleaming: LLaMMA-Sentence-LoRR
Figure 23: Shift in LLM Judge Scores Pre- and Dost-Unleannin:: LLaMA-Sentence-Full
35

---

Figure 24: Shift in LLM Judge Scores Pre- and Post-Unleariing Qwen-QA-LoRR
Hegltach
Figure 25: Shift in LLM Judge Scores Pre- and Post-Unlearning Owen-QA-Fri
Orddler
tedteii Lobel
Megtrnds
SC0B
Figure 26: Shift in LLM Judge Scores Pre- and Post-Unlearning: Owen-Sentence-Loff
Figure 27: Shift in LLM Judge Scores Pre- and Post-Unlearning Qwen-Sentence-Full
F33 Additional Results on the Impact of Confidence Scores on Unlearning Effectiveness
In this section, present additional results on how confidence scores within the supporting subgraph affect unlearning outcomes across different settings. As shown in Fig. 28 29 the entropy threshold u², i.e., retaining only higher-confidence knowledge, systematically filters out weaker supporting inferences. This leads to a significant increase in unlearning effectiveness. These results highlight that confconiieecce knowledge triples play crucial role in evaluating unlearning, and omitting them can substantially overestimate its effectiveness.
gradually decreasing
Linlearnig Effectiveness
Unleeamnng Effectiveness
Unleaaiing Effectiveness
Unlearinn Effectiveness
3.0
Gresars-aaerr Aaners
2.2
Aacars
Grecun
0.8
Labe
Kandor
we
scaue
SchLe
schu
0.2
33
0.3
Entropy Threshold
0.5
233
02
2.3
Entropy Threshold
05
1.2
3.
0.3
0.3 Entropy Threshold
0'
0.3
Entropy Threshold
05
1.
(a) QA-LoRA
(b) Full
(c) Sentence-LoRe
(d) Sentence-Fil
Figure 28: Impact of Confidence Scores on Unlearning Effectiveness: LLaMA Model
36

---

Uniearming Effectiveness
Unieaming Eftectiveness
Unieaming Effectiveness
Unleaming Effeciieness
3.2
0.8
Aeakie Lake
3.8
Aaaker Lake
0.9
leporace
10
03
Heponce
3.
Hegondd
Higiinds
0.4
0.2
02
32
0.2
3.
Entropy Threshold
Entropy Threshald
Entropy Threshald
Entropy Threshald
(a) QA-LoRA
(b) QA-Full
(c) Sentence-Loff
(d) Sentence- Full
Figure 29: Impact of Confidence Scores on Unlearning Effectiveness: Qwen Model
F.4
Additional Results on Performance Across Unlearning Epochs
In Fig. 30, we present additional results analyzing the impact of unlearning epochs under the LLaMA3 full-model unlearning setting, comparing both sentence-based and QA-based formats. Across all configurations, the unlearning effectiveness measured by our supporting subgraph evaluation remains consistently lower than that of traditional instance-level evaluations. Notably, in the QA-based format, unlearning directly targets the question-answer pairs used for probing factual knowledge. As unlearning progresses, this often leads severe degradation of model utility, resulting in unlearning effectiveness approaching 1, indicating that both the target triples and their associated subgraphs have been extensively corrupted. In contrast, under the sentence-bra bbseed format (Left), the unlearning process tends to be more conservative in its impact on model utility in most cases, causing less degradation compared the QA-based setting (Right). Consequently, we observe that unlearning effectiveness, particularly under our evaluation framework, remains low across most epochs. These findings highlight the limitations of current unlearning methods and underscore the need for more robust approaches capable of effectively removing knowledge without compromising general model performance.
Unlearning Effectiveness
Utility
Unlearning Effectiveness
Utiliyy
Matheds
2:0
1.0
Kanodl Libe
0.0
10-
32-
0.2
17
307
107
10
10
Epoch
Epoch
Epoch
Epoch
Figure 30: Impact of unlearning epochs on LLaMA3 full-model unlearning. Left: Unlearning effectiveness (measured by both instance-level and our proposed evaluation) and corresponding utility (measured by locality) under sentence-based format. Right: Same analysis for QA format.
G Mapping Between Entropy Threshold u* and Yes Token Probability
In this section, we clarify how the entropy-based filtering criterion used for constructing supporting subgraphs relates to the model's predicted probability for the Yes token in multiple-choice question format (Yes, No, Unknown). Specifically, we consider prediction to reflect retained knowledge if the model's output satisfies two conditions: (i) the Yes token is the argmax among the three options. and (ii) the entropy computed over the distribution of these three options is below threshold u² (assuming the model adheres instructions). To make this relationship more interpretable, Tab. presents the corresponding range of Yes token probabilities for different values of entropy under the argmax constraint. This mapping provides an intuitive understanding how the entropy threshold translates to the model's confidence in answering Yes. To derive these intervals, we fix the entropy value u² and solve for the range of valid Yes probabilities p under the constraint that Yes is the argmax. The upper bound occurs when the remaining two options (No and Unknown) share equal probability, i.e., =² = H(p. 1-p 1-2) with p > 122 where H denote entropy. The lower bound corresponds to the case where one of the non-Yes options takes probability 1 - p and the other
zero, i.e., u² H(p, 1 p, 0) with p > 1
37

---

Entropy
0.10
0.15
0.20
0.25
0.30
0.35
Yes Proh. Range
[0.987, 0.989]
[0.978, 0.982]
[0.969, 0.974]
[0.958, 0,966]
[0.947, 0.957]
[0.934, 0.947]
Entropy
0.40
0.45
0.55
0.60
Yes Prob. Range
[0.921, 0.937)
[0.906, 0.927]
[0.890, 0.916]
[0.873, 0.905]
[0.854, 0.892]
[0.833, 0.880]
Entropy
0.70
0.75
0.80
0.85
0.90
0.95
Yes Prob. Range
[0.811, 0.867]
[0.785, 0.853]
[0.757, 0.838)
[0.724, 0.823]
[0.684, 0.807]
[0.631, 0.791]
Entropy
Yes Prob. Range
[0.500, 0.773)
Table 6: Feasible Yes token probability ranges corresponding to entropy thresholds u". Ranges computed under the assumption that Yes is the most likely token.
H
Additional Details and Illustrations for YAGO3-10
Complete List of Relations in YAGO3-10: actedIn, wasBornln, hasGender, hassAcademieAdvisoor, hasChild, hasCitizenship. hasDeathPlace, hasEmployer, hasGivenName, hasInstrument, Langange hasieegee,s,edence, hasMember, hasName, hasNationality, hasOccupation, hasOfficialLangun Lnguagge hasPlaceOfBirthh hasPlaceOfDeath, hasSpouse, hasSurname, hasTitle, holdsPoliticalPosition. isfffl- iatedTo, isConnectedTo, isnnownFor, iscaatee,nn, isMarrieTTo, isPoliticianOf, livesln, playsFor, produced, studiedAt, wasBomOnnDaee, wasCreatedOnDatt, wasDestroyyddnnDatt worksAt.
Using the complete set of relations from YAGO3-10, we present illustrative examples of potential non-Jdeterministic inference patterns in the knowledge graph, as summarized in Tab. 7
Rule & Explanation
Rule: (a, hasSpouse,
(b, hasSpouse, a)
Explanation: This rule captures the symmetry inherent in the marital relationship. Given that marriage is a bidirectional legal and social contract, if entity married to entity b, it logically follows that b is married to a.
Rule: (a. playsFor, b) A (b, isLocatedIn, c) > (a, livesln, c)
Explanation: This inference relies on probabilistic assumption: professional athletes commonly reside in the same city country where their affiliated teams based. While not universally valid due to cases such as commuting or temporary contracts, this pattern holds in the majority of real-world scenarios.
Rule: (a, wasBornIn, b) + (a, hasPlaceeOBirrhh, b)
Explanation: This is a deterministic alias pattern, where both relations wasBornin and hasPlace OfBirth denote the same biographical fact. The two terms are semantically interchangeable in most ontological frameworks.
Rule: (a, hasNaationlltyy b) (a, hasCitizenship. b)
Explanation: This a high-probability rule rooted in sociopolitical conventions. Although national- ity and citizenship may differ in legal terms, they often used interchangeably in knowledge bases. Most individuals who identify with a nationality also hold legal citizenship of the corresponding state.
Rule: (a, studiedAt, b) A (b. Locatede c) + (a. hasl SuusleegalResddnnee, c)
Explanation: This rule models a moderate- moderate-probabiliiy correlation: enrollment at an educational institution often necessitates or implies legal residence in the institution's 's geographical location, due to immigration and residency regulations applicable to students.
Rule: (a, hasEmployer, b) A (b, isoaatdd, c) + (a, livesIn c)
Explanation: A high-probability inference, reflecting the assumption that employees generally reside in proximity to their workplace. This assumption may be weakened in the presence of remote work multi-location employers but holds in standard employment contexts.
Rule: (a, worksAt, b) A (b. isLocate,nn, e) A (c, hasOOficciall d) = (a, anguagge d)
38

---

This multi-hop rule captures the linguistic environment of a worker. Individuals employed in a region are likely acquire or use the region's 's official language, particularly in professional or social interactions. The inference strength varies by linguistic diversity and integration policies.
Explanation:
Rule: (a, hasDeathPlace, b) = (a. hasPlaceOfDeath, b)
Explanation: Another deterministic alias. Both relations refer to the geographical location where individual passed away. They are synonymous in the context of biographical datasets and can be used interchangeably in logical reasoning.
Table 7: Illustrative examples of inference patterns in YAGO3-10.
I Limitations and Broader impacts
In this paper, we have proposed a novel knowledge unlearning evaluation framework designed to capture complex factual dependencies and the inherent uncertainty reflected by LLM confidence. Our aim is to enable more comprehensive and realistic assessments of knowledge unlearning efficacy, Nonetheless, several important limitations remain. First, the inference capabilities of both LL.M-based and human evaluators have inherent limitations. Both types of judges may overlook subtle or complex inference patterns, and real-world adversaries could potentially surpass these evaluators in inference power. In particular, while we observed that GPT-o4-mini provides stable and interpretable judgments in our experiments, we acknowledge that I.LM judges can still exhibit occasional inconsistencies and prompt sensitivity. Future work may explore using multiple diverse LLM judges distilling lightweight judge models to improve robustness and reduce evaluation cost. Second, the process of constructing supporting subgraphs may be incomplete. Our subgraph extraction approach might not encompass all relevant facts necessary for accurate inference, and the reference knowledge graphs used in our evaluation may themselves lack certain entities relationships that are implicitly encoded within the target LLM. Furthermore, our knowledge probing strategy for supporting subgraph construction might not be optimal. and we acknowledge that certain knowledge extracted can be inaccurate. Third, our current evaluation framework primarily focuses on relational factual knowledge and relies externally constructed reference knowledge graphs guide the subgraph extraction process. Given that many pretrained LLMS utilize undisclosed or proprietary pretraining corpora, discrepancies could arise between these reference graphs and the model's ss actual internal knowledge representations, further limiting evaluation accuracy. Finally, while our framework enables scalable LLM-based evaluation, it currently depends commercial API calls. Although we have kept the total cost low (approximately USD 1.5 for evaluating 200 unlearning targets using GPT-o4 GPT-o4-mini), such dependency introduces potential variability across judge models and access constraints in future deployments. Beyond methodological limitations, we also emphasize the ethical implications associated with knowledge unlearning evaluations. Specifically, we advocate for the responsible use of compliant, publicly accessible, or properly authorized data when conducting such studies. By acknowledging these limitations and ethical considerations, we hope our work fosters more rigorous and ethically responsible practices in the research, development, and deployment of machine learning technologies.
39


Reviews:
[{"review_id": "upTf2VZoES", "reviewer": "Reviewer_V8kE", "summary": "This paper introduces a framework to evaluate knowledge unlearning in LLMs, highlighting the impact of correlated and confidence-aware knowledge. Experiments show current methods often overestimate effectiveness, stressing the need to account for these factors in future research.", "questions": "1.Is it feasible to apply the idea of confidence aware knowledge graph to other tasks such as knowledge editing?\n\n2.Table 1 indicates that the template format (sentence vs. QA) significantly affects the results. Does this suggest that the current method is overly sensitive to prompt design? \n\n3.Based on the analysis in this article, what insights can it bring to future research? Can you further analyze and summarize in depth?", "limitations": "The author should state that this article focuses on unlearning triplet knowledge, leaving more settings such as multimodality and procedural knowledge for future work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1.This paper is clearly written. Make it easy for readers to understand.\n\n2.This paper proposes a realistic formulation of knowledge unlearning.\n\n3.This paper proposes employing a powerful LLM judge to effectively automate unlearning evaluation.\n\nWeaknesses:\n\n1.In Section 6.3, regarding the correlation evaluation, the evaluation scale is relatively small (only 50 samples). It is necessary to report the statistical properties of larger-scale data and the correlation among multiple human annotators.\n\n2.The text relies on YAGO3-10 to construct reasoning paths for evaluation. However, can the Supporting Subgraphs based on YAGO3-10 sufficiently cover the internal knowledge structure of LLMs? How does the coverage of Supporting Subgraphs quantify the extent to which they encompass the internal knowledge of LLMs, in order to thoroughly evaluate the forgetting of LLMs' internal knowledge?\n\n3.The related work section discusses multi-hop knowledge editing and evaluation, but lacks more in-depth quantitative comparisons and discussions. If a table could be included to compare related work, it would further highlight the contributions of this paper.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' rebuttal and my concerns are mostly addressed in the rebuttal. I have decided to maintain the positive score.", "rebuttal": "We sincerely thank Reviewer V8kE for recognizing the importance and practical relevance of our proposed knowledge unlearning framework, and for recommending acceptance. Below, we provide detailed responses to the questions and concerns raised.\n\n> W1. Evaluation conducted on a relatively small scale; request to report statistical properties and annotator correlation on a larger scale.\n\nWe appreciate the reviewer’s suggestion. Following the same procedure described in the paper, we additionally sampled 50 disjoint subgraphs and collected human ratings under the same guidelines, alongside three rounds of LLM-based reasoning. We report the overall Pearson correlation between the human and LLM scores across the full set of 100 subgraphs, as well as the inter-annotator correlations among the three human raters. We also include the standard deviation of the LLM’s multi-round reasoning scores over these 100 subgraphs. All results have been added to **Table R1** (Statistical properties of the LLM judge and Correlations among judges).\n\n| **Human Avg vs LLM Avg** | **LLM Avg Std** | **Human 1 vs LLM Avg** | **Human 2 vs LLM Avg** | **Human 3 vs LLM Avg** | **Human 1 vs Human 2** | **Human 2 vs Human 3** | **Human 1 vs Human 3** |\n|:------------------------:|:---------------:|:-----------------------:|:-----------------------:|:-----------------------:|:-----------------------:|:-----------------------:|:-----------------------:|\n| 0.837                    | 0.137           | 0.789                   | 0.769                   | 0.760                   | 0.802                   | 0.710                   | 0.825                   |\n\n> W2.YAGO3-10 might not capture the entire internal knowledge structure of LLMs. How to quantify the gap between supporting subgraphs and internal knowledge of LLMs.\n\nWe thank the reviewer for the insightful question. We acknowledge that supporting subgraphs derived from a reference knowledge graph (KG) may not fully capture the internal knowledge structure of an LLM, particularly given that the model’s pretraining corpus is non-public. However, this limitation does not undermine our core contributions: \n\n(1) we demonstrate that confidence-aware knowledge structures reveal significant overestimation in the unlearning effectiveness of existing methods;\n\n(2) our benchmark provides a systematic basis for comparing unlearning algorithms under correlated inference. \n\nAdmittedly, more precise evaluation of whether a target fact has been fully unlearned would benefit from access to the original training data. In realistic deployment settings, model owners could construct more complete subgraphs to better approximate internal knowledge. We discuss this consideration in the Limitations and Broader Impacts section (Appendix I).\n\n> W3.Table to compare with current work in multi-hop knowledge editing and evaluation.\n\nWe thank the reviewer for the suggestion. In **Table R2** below, we compare our framework with representative multi-hop knowledge editing evaluations, highlighting two key distinctions. First, prior knowledge editing evaluations typically adopt a cause → effect setup. For example, editing the fact (Barack Obama, is, U.S. President) and checking whether downstream questions like “Who is the First Lady?” or “Where did the U.S. President attend college?” reflect the change. In contrast, knowledge unlearning focuses on effect → cause reasoning to prevent adversarial inference. For instance, evaluating whether (Barack Obama, is, U.S. President) can still be reconstructed from related facts such as (Michelle Obama, is, First Lady) or (Barack Obama, husband of, Michelle Obama). Second, we are the first to consider confidence-aware subgraph-based knowledge inferences that capture correlated knowledge and assess inferability via LLM judges.\n\n| **Paper**   | **Task**     | **Reasoning Direction** | **Reasoning Source**                   | **Inference Modeling**                  | **Evaluation Method**                                         |\n|-------------|--------------|--------------------------|-----------------------------------------|-----------------------------------------|---------------------------------------------------------------|\n| MeLLo [1]   | Editing      | cause → effect           | Single entailed chain                   | Chain-based                             | Binary accuracy on multi-hop QA                               |\n| RAE [2]     | Editing      | cause → effect           | Retrieved fact chain                    | Mutual-information-based retrieval      | Binary QA accuracy                                            |\n| Ours        | Unlearning   | effect → cause           | Structured confidence-aware subgraph    | Explicit reasoning over belief subgraph | LLM judge (human verification for automation process)         |\n\n[1] Zhong et al. Mquake: Assessing knowledge editing in language models via multi-hop questions.\n\n[2] Shi et al. Retrieval-enhanced knowledge editing in language models for multi-hop question answering.\n\n> Q1.Is it feasible to apply the idea of confidence aware knowledge graph to other tasks such as knowledge editing?\n\nWe thank the reviewer for the thoughtful question. Yes, it is feasible to apply our confidence-aware knowledge graph to evaluate certain knowledge editing methods. More specifically, knowledge unlearning, which aims to forget certain target facts, can be viewed as editing the model such that probing the target fact yields an unknown response. This connection suggests that the framework we use for evaluating unlearning can potentially extend to editing scenarios. Furthermore, for more general knowledge editing tasks, such as modifying a specific factual triple, one can construct a confidence-aware supporting subgraph via probing, guided by a reference KG, to capture facts correlated with the target. This subgraph can then be used to evaluate whether the edited knowledge is correctly integrated, by assessing inferability over the updated structure. Further extending our framework to accommodate general knowledge editing tasks is an interesting direction to explore.\n\n> Q2.Table 1 indicates that the template format (sentence vs. QA) significantly affects the results. Does this suggest that the current method is overly sensitive to prompt design?\n\nWe thank the reviewer for the insightful question. We acknowledge that the effectiveness of unlearning can depend on the choice of unlearning prompt format. However, our core message remains consistent across both the sentence and QA templates: ignoring confidence-aware knowledge correlation can significantly overestimate unlearning effectiveness. This conclusion holds regardless of the specific format used. The observed variance is primarily due to how different templates interact with unlearning dynamics. Specifically, sentence-based unlearning tends to induce milder forgetting under QA-based probing, whereas QA-based unlearning leads to more direct forgetting but also greater utility degradation. This effect is partly driven by the nature of existing heuristic methods such as gradient ascent, which, when applied to QA-formatted data, can cause substantial behavioral shifts, especially in instruction-following. Our findings are also consistent with prior studies in both LLM fine-tuning and unlearning, which have shown sensitivity to training formats [3]. These results highlight a broader challenge: current unlearning methods are heavily influenced by the structure of the training prompt, limiting their precision in selectively removing specific knowledge.\n\n[3] Lyu et al. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. \n\n> Q3.Based on the analysis in this article, what insights can it bring to future research? Can you further analyze and summarize in depth?\n\nWe thank the reviewer for the thoughtful question. Our paper offers several key insights that can inform future research. First, we demonstrate that confidence-aware supporting subgraphs reveal the consistent issue of overestimating unlearning quality when relying solely on target knowledge. This insight suggests a promising direction for future algorithm design, namely, developing unlearning methods that explicitly target confidence-aware subgraphs, aiming to accurately remove knowledge that enables inference of the target while retaining the majority of unrelated factual triples. Second, our proposed benchmark construction methodology offers a generalizable framework for model holders with access to the training corpus. By extracting correlated entities and relations, they can construct more comprehensive supporting subgraphs tailored to specific unlearning targets. This enables both more precise internal audits and the creation of richer public unlearning benchmarks to support the broader development and evaluation of unlearning algorithms.\n\n> Discussion in Limitation Section: The author should state that this article focuses on unlearning triplet knowledge, leaving more settings such as multimodality and procedural knowledge for future work.\n\nWe thank the reviewer for the suggestion. We have incorporated the corresponding discussion into the Limitation section (Appendix I)."}, {"review_id": "lNPcMteTnt", "reviewer": "Reviewer_dqRs", "summary": "This paper proposes a novel evaluation framework for knowledge unlearning that goes beyond assessing the forgetting of isolated facts. It addresses the challenge of implicitly retained knowledge, where facts presumed forgotten may persist due to correlated or inferable information. To capture this, the authors represent factual contexts as knowledge graphs with confidence scores and define unlearning evaluation in terms of knowledge subgraph reasoning. They extract such subgraphs from LLMs (LLaMA-8B-Instruct and Qwen-7B-Instruct) and use existing knowledge graphs along with LLMs as judges to reason over these subgraphs and assess unlearning success. Through experiments with popular unlearning methods, they show that current approaches often overestimate how much knowledge has actually been forgotten when real-world correlations are considered.", "questions": "Line 403 mentions that the evaluation might overlook certain multi-hop relations. Is this a common limitation in subgraph-based reasoning, and how impactful is it on the overall assessment?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- The idea of evaluating implicit retained knowledge via subgraph reasoning is novel and relevant, especially for domains where relational context matters.\n- The finding that existing unlearning methods overestimate their effectiveness is important and may inspire better techniques in the future.\n- The motivation to look beyond surface-level fact forgetting is well-justified.\n\n**Weaknesses and Suggestions**:\n- Practicality of Knowledge Graphs: The proposed method relies on structured external knowledge graphs (e.g., Wikidata) to model factual context. In many real-world unlearning use cases (e.g., copyrighted or proprietary material), such structured knowledge may not be available. How do the authors envision applying their framework in those cases?\n- LLM Judge Reliability: Relying on LLMs as judges for subgraph reasoning introduces potential noise and hallucinations. How did the authors verify the accuracy or consistency of the LLM judgments, especially for multi-hop reasoning?\n- Multiple-Choice Evaluation: If multiple-choice questions were used in evaluations, did the authors take steps to mitigate answer biases (e.g., positional bias or surface-level cues)?\n- Choice of Judge Model: The use of GPT-4o-mini as the LLM judge raises questions. Was this choice based on a tradeoff between performance and cost? Would a stronger model (e.g., GPT-4-turbo) change the conclusions?", "was_revised": true, "final_justification": "The authors provided a detailed rebuttal and answered my questions. Most of my concerns were resolved, yet, I am not sure if the \"Yes-No\" bias is considered in the study and analysis since it has been shown recently that many LLMs suffer from it [1]. While the authors provide an ablation in the rebuttal, the study setup is unconvincing. That said, the paper is well-written and has strong motivation and results and has merits to be accepted. So, I have updated the score to borderline accept.\n\nReference:\n[1] Du, Yanrui, et al. \"Investigating the Security Threat Arising from “Yes-No” Implicit Bias in Large Language Models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 22. 2025.", "rebuttal": "We sincerely thank Reviewer dqRs for the thoughtful review, for recognizing the novelty of our framework, and for the insightful questions. Below, we address the comments and concerns raised.\n\n> W1. Clarification on applicability of framework without structured knowledge graphs.\n\nWe thank the reviewer for raising this insightful and practical question. Our framework remains applicable even in scenarios where a structured reference knowledge graph (KG) is not readily available. In practice, if the knowledge related to the target unlearning fact exists in an unstructured form, such as raw text, our approach can still be applied. For example, model holders can have access to large-scale training corpora or internal datasets. In such cases, a reference KG can be constructed from unstructured text using standard procedures [1], including named entity recognition, entity linking, and relation extraction. This automatically derived graph can then be used to build a confidence-aware supporting subgraph, enabling the application of our framework without relying on pre-existing structured KGs.\n\n[1] Zhong et al. A comprehensive survey on automatic knowledge graph construction.\n\n> W2. LLM Judge Reliability: Relying on LLMs as judges for subgraph reasoning introduces potential noise and hallucinations. How did the authors verify the accuracy or consistency of the LLM judgments, especially for multi-hop reasoning?\n\nWe thank the reviewer for the thoughtful question. To verify the accuracy of the LLM judge, we compare its outputs against human annotations on the validation set. We also confirm that the LLM strictly follows the evaluation instructions, reasoning only over the provided supporting subgraph and associated confidence values without using external knowledge (Section 6.3). Further, we detailedly discuss illustrative examples and highlight cases where the LLM agrees with or diverges from strong human judgment (Appendix E). \n\nTo assess consistency, we prompt the LLM to score each triple three times independently and report the average standard deviation across the 50 validation subgraphs, which is 0.132. While even strong models such as GPT-o4-mini may introduce noise, we highlight these limitations in Appendix I. Furthermore, recent efforts have explored improving LLM judges [2], which we consider a promising direction for future work.\n\n[2] Huang et al., Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation.\n\n> W3. Multiple-Choice Evaluation: If multiple-choice questions were used in evaluations, did the authors take steps to mitigate answer biases (e.g., positional bias or surface-level cues)? \n\nWe thank the reviewer for the thoughtful question. To mitigate answer biases, we did not use the answers generated by the LLM directly. Instead, we used the probabilities of the \"yes\", \"no\", and \"unknown\" tokens, and relied on the values of these probabilities to indicate the LLM’s answer choices. To further validate this approach, we conducted an ablation experiment on the validation set by randomly shuffling the positions of the \"yes\", \"no\", and \"unknown\" tokens. We calculated two metrics to assess how the LLM is affected by such positional shuffling: \n\n(1) The ratio of count(argmax = “yes”) to count(argmax = “no”), which measures how the distribution of “yes” versus “no” predictions changes.\n\n (2) The average predicted probabilities of [\"no\", \"yes\", \"unknown\"], which reflect how entropy-based measures may be affected.\n\nOur results show that before shuffling, the \"yes\"/\"no\" ratio is 0.392, and the average predicted probabilities are [0.6064, 0.2589, 0.1350]. After shuffling, the ratio is 0.388, and the average probabilities are [0.6061, 0.2586, 0.1353]. These minimal changes in both metrics demonstrate that our framework is largely robust to positional bias.\n\n> W4. Choice of Judge Model: The use of GPT-4o-mini as the LLM judge raises questions. Was this choice based on a tradeoff between performance and cost? Would a stronger model (e.g., GPT-4-turbo) change the conclusions?\n\nWe thank the reviewer for the insightful question. We thank the reviewer for the thoughtful question. We would like to kindly clarify that the judge model used in our paper is not GPT-4o-mini, but GPT-o4-mini (o-series), the most recent model in its series with stronger reasoning capabilities. We did not use more advanced models due to considerations of performance–cost trade-offs. Additionally, we conducted experiments with GPT-4 Turbo and Claude 4 on the validation subgraphs and observed a high correlation in scores. The Pearson correlation coefficients are 0.812 and 0.809, respectively,  using the same prompt template, suggesting that our conclusions remain fairly stable.\n\n> Q1. Line 403 mentions that the evaluation might overlook certain multi-hop relations. Is this a common limitation in subgraph-based reasoning, and how impactful is it on the overall assessment?\n\nWe thank the reviewer for the thoughtful question. As noted in Section 6.3 and Appendix E, our analysis of subgraphs from the validation set shows that the GPT-o4-mini judge produces scores that align closely with human annotations. Its reasoning is consistently grounded in the provided subgraph evidence, without reliance on external knowledge, suggesting that the model is overall reliable. That said, we do observe a small number of cases where the LLM judge overlooks multi-hop inference paths supported by the subgraph. We believe this reflects a common limitation in current subgraph-based LLM reasoning, particularly in complex multi-hop scenarios. This challenge is consistent with issues identified by prior studies on LLM reasoning limits [3], which show that even advanced models may struggle to fully exploit multi-hop relational structure. We acknowledge its potential impact and discuss it further in Appendix I. We also believe that using stronger reasoning models may help alleviate this limitation.\n\n[3] Ma et al. Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities."}, {"review_id": "WZPEVwqCJh", "reviewer": "Reviewer_XFMD", "summary": "This paper proposes a new framework for evaluating knowledge unlearning in large language models. It captures the implicit knowledge by representing factual contexts as KGs, and incorporates an inference-based evaluation protocol that leverages LLMs as judges to assess unlearning effectiveness. Extensive experiments reveal that existing evaluation methods often significantly overestimate the actual unlearning effectiveness, while the proposed framework can provide a more realistic and rigorous assessment.", "questions": "Please see Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\nThe paper is easy to follow, with clear writing and well-presented motivation and technical details.\n\nThe authors propose new benchmark and framework that consider implicit relationships between facts, which can evaluate the unlearning effectiveness in a fairer and more reasonable fashion.\n\nThe LLM judges are also proven to be closely aligned with human experts.\n\n### Weaknesses\n\nIt may be better to discuss relevant knowledge editing methods, which also include the unlearning task.\n\nThe authors evaluate only 8B and 7B LLMs. Is it possible to conduct experiments with larger LLMs? Those LLMs may store more complicated and implicitly correlated information.", "was_revised": true, "final_justification": "After considering comments from other reviewers, I maintain my current score (5).", "rebuttal": "We sincerely thank Reviewer XFMD for acknowledging the novelty and practicality of our evaluation framework for knowledge unlearning, and for supporting the acceptance of our paper. Below, we provide detailed responses to the questions raised.\n\n> W1. It may be better to discuss relevant knowledge editing methods, which also include the unlearning task. \n\nWe thank the reviewer for the helpful suggestion. In the revised version, we will further elaborate in the second paragraph of Related Work (Sec.2) on the connection between knowledge editing and knowledge unlearning. We will also discuss how general knowledge editing methods can be potentially evaluated through our framework and, conversely, how the idea of constructing confidence-aware subgraphs can be adapted to evaluate editing outcomes. Specifically, knowledge unlearning can be interpreted as a form of model modification where the target fact becomes noninferable upon probing. This shared objective naturally motivates the extension of our framework to broader knowledge editing settings. In particular, when a factual triple is altered, one can probe the model to identify a set of correlated facts, guided by a reference KG, and construct a confidence-aware subgraph. Evaluating whether the intended change propagates consistently through this subgraph offers a comprehensive assessment of whether the edited knowledge has been faithfully incorporated.\n\n> W2. The authors evaluate only 8B and 7B LLMs. Is it possible to conduct experiments with larger LLMs? Those LLMs may store more complicated and implicitly correlated information.\n\nWe thank the reviewer for the constructive suggestion and fully agree that larger LLMs may encode more complex correlated knowledge. Due to current resource constraints, our experiments focus on widely used 7B/8B-scale models commonly adopted in the unlearning evaluation literature [1,2,3]. We plan to extend our evaluation to larger models such as Qwen2.5-14B-Instruct when sufficient computational resources become available.\n\n[1] Maini et al. Tofu: A task of fictitious unlearning for llms.\n\n[2] Shi et al. Muse: Machine unlearning six-way evaluation for language models.\n\n[3] Zhong et al. Mquake: Assessing knowledge editing in language models via multi-hop questions."}, {"review_id": "oblzzHafxQ", "reviewer": "Reviewer_VvJj", "summary": "This paper presents a confidence-aware evaluation protocol for unlearning facts in LLM to address limitations in current assessment methodologies. Existing unlearning metrics employ simplistic verbatim matching to determine whether facts to be forgotten remain accessible, failing to detect inferential reconstruction through correlated knowledge that persists after the unlearning process. The authors propose an approach that rests on a knowledge graph (KG) as a supporting source of background knowledge. In practice, given a certain fact to forget (that has to be present in the underlying KG), the approach works in two stages: (1) extraction of knowledge subgraphs from, and (2) assessment by calibrated GPT-4-level judges to determine whether target information remains inferentially derivable. The approach formalizes model knowledge as G = (E, R, T_U) where triples t = (s, r, o, u) incorporate entropy-calibrated confidence scores u to quantify belief strength in retained facts. Then, the key contribution lies in evaluating inferential leakage rather than direct recall, providing a more robust assessment of whether sensitive information has been effectively eliminated from the model's accessible knowledge base. This methodology aims to address a limitation in current unlearning evaluation, where models may successfully suppress direct responses while retaining the ability to reconstruct identical information through multi-hop reasoning over remaining knowledge. The process involves several calibration phases (LLM temperature to interpret output scores), also requiring the involvement of 3 PhD subjects. The authors present results on fact from YAGO (including 37 predicates) and test the framework capabilities on two LLMs (LLaMA, Qwen-2.5)", "questions": "Q1: How does UES performance degrade when target facts or neighbors are missing from YAGO3-10? Have you measured recall on domain-specific knowledge (recent scientific discoveries, proprietary information) or long-tail facts with rare entities? Your reliance on external reference KGs creates systematic blind spots for implicit correlations learned from unstructured text that wouldn't appear in Wikipedia-derived triple stores. This could critically underestimate unlearning effectiveness for sensitive knowledge most relevant to safety applications.\n\nQ2: How much do inferability scores drift when swapping GPT-4-mini, say for Claude, Gemini, or open-source alternatives across identical subgraphs? Your framework's validity depends entirely on judge consistency, yet you provide no systematic robustness testing across multiple architectures—a critical gap undermining your comparative effectiveness claims.\n\nQ3: Why hand-tune u* = 1 instead of using ROC optimization against validation sets or adaptive thresholding by knowledge domain? Optimal thresholds likely vary across relation types and model architectures. Your fixed threshold ignores that confidence calibration quality differs dramatically across models and fine-tuning procedures.\n\nQ4: For millions of triples, what are concrete GPU hours and API costs per target fact? Can lightweight approximations (learned subgraph pruning, distilled judges, hierarchical evaluation) preserve diagnostic power while reducing computational burden by orders of magnitude? \n\nThanks", "limitations": "The authors have discussed limitations. Although some aspects have not been covered. The authors mention subgraph construction accuracy but do not mention the cost (in terms of LLM calls, for instance). Moreover, a broader discussion about why focusing on a closed model (GPT) is also needed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nS1: The paper presents an overall nice idea. Linking knowledge graphs (maintaining organized facts) with LLMs (where facts are embedded in text) represents a key step toward mitigating LLMs' hallucinations. In this particular case, using the KG as a source of background knowledge to find subgraphs centered on a fact to be forgotten is interesting.\n\nS2: The paper is well written, although the organization is a bit confusing had to go back and forth between the paper and the Appendix file (since the pointers weren't there). I realize that deadline flexibility between the paper and the supplemental material may have been responsible for this \"separation\".\n\nWeaknesses\nW1: I immediately thought of the connection between forgetting facts and LLM guardrails (to prevent the LLM from releasing harmful information).  The authors fundamentally missed that their knowledge unlearning framework directly addresses the core vulnerability of LLM guardrail systems—inference through correlated knowledge pathways. Their methodology for extracting supporting subgraphs and evaluating residual inferability through adversarial probing is **structurally identical** to how guardrail circumvention attacks operate: both exploit semantic relationships to reconstruct restricted information from seemingly benign facts. For instance, while a guardrail might block direct queries about explosive manufacturing, an adversary could still infer bomb-making instructions by querying related chemistry facts, mining practices, and fertilizer compositions—exactly the type of multi-hop inference their framework is designed to detect and prevent. The authors should have benchmarked their approach against existing guardrail systems using identical restricted knowledge domains, comparing their  UES against guardrail robustness metrics when facing inference-based attacks. Their finding that current evaluation methods \"overestimate effectiveness by 20-30%\" likely applies equally to guardrail evaluations, yet they provide no empirical validation of this critical insight. This represents a significant methodological gap: without demonstrating that their adversarial inference protocol captures attack vectors that traditional guardrail evaluations miss, they cannot substantiate their claims about providing \"more realistic and rigorous assessment\" of safety mechanisms.\n\n\nW2: The proposed framework suffers from severe methodological brittleness due to its dependence on multiple sequentially coupled components, each introducing distinct failure modes that compound systematically. \nW2.1: The 3-hop subgraph extraction limitation represents an arbitrary constraint that fundamentally undermines the approach's completeness—adversarial inference attacks can routinely exploit longer reasoning chains, and the authors provide no principled justification for this bound beyond computational traceability (and reference to the state of the art adopting similar thresholds). The reliance on external reference knowledge graphs (YAGO3-10) for candidate generation creates a critical blind spot: any knowledge relationships not represented in the reference KG will be invisible to the extraction process, potentially missing entire classes of inferential pathways that exist within the target LLM's latent representations.\nW2.2: The entropy-based confidence calibration introduces multiple points of failure: temperature scaling assumes consistent calibration across diverse knowledge domains and model states, yet the authors demonstrate this only on a limited validation set. The fixed entropy threshold (u* = 1) appears arbitrarily chosen and lacks sensitivity analysis—small perturbations in this threshold could dramatically alter subgraph composition and subsequent evaluations. \nW2.3: The LLM judge component represents perhaps the most problematic element: even GPT-4-level models exhibit inconsistent reasoning across complex multi-hop scenarios, and the authors' correlation with human judgment (r=0.826) still reflects 32% unexplained variance. The judge's prompt engineering requirements—balancing specificity with generalizability—create a fragile dependency on exact phrasing and instruction format. Most critically, the framework's cascading nature means that errors compound multiplicatively: subgraph extraction errors propagate through confidence assessment, which then distorts judge evaluation, ultimately producing unreliable effectiveness scores whose relationship to actual unlearning robustness remains empirically invalidated.\n\n\nW3: The framework's reliance on proprietary LLM judges (GPT-4-mini) introduces fundamental evaluation biases and economic barriers that severely limit practical deployment. Using one commercial LLM to assess another creates systematic blind spots—both models likely share similar training paradigms, architectural biases, and knowledge representation patterns that could lead to convergent evaluation errors. The judge may systematically under-detect inference pathways that exploit reasoning patterns foreign to its own training regime, or conversely, over-penalize inference chains that its particular architecture finds difficult to process. This circular dependency means the evaluation framework inherits the safety and robustness limitations **of the judge model itself**, potentially creating false confidence in unlearning effectiveness when both models share identical vulnerabilities to specific attack vectors.\nW3.1: Computational economics presents an even more critical barrier to adoption. The authors' methodology requires hundreds of individual queries per target fact: candidate triple enumeration across 3-hop neighborhoods, confidence assessment for each extracted triple, and iterative judge evaluation across multiple reasoning chains. For their modest experimental setup (200 target triples), this translates to tens of thousands of API calls to proprietary models, yet the paper provides no cost analysis or runtime metrics. Assumin GPT-4 pricing to be (~$0.01-0.03 per 1K tokens), evaluating large unlearning scenarios could cost hundreds of dollars per assessment, making the approach economically prohibitive for iterative development, hyperparameter tuning, or large-scale knowledge domain evaluation. Without open-source judge alternatives or significant computational optimizations, their framework remains confined to academic demonstrations rather than real-world safety applications\n\nMinor point\nThe framework lacks ground truth establishment, conflating LLM judge inference capability with actual knowledge retention in target models. This creates arbitrary evaluation ceilings bounded by judge model limitations rather than measuring genuine knowledge erasure. The circular validation—using their own subgraph methodology to evaluate their own effectiveness claims—provides no external baseline.\n\n\n\n--", "was_revised": true, "final_justification": "The rebuttal has addressed some of my comments. Some problems related to evaluation and scalability have not been properly addressed.", "rebuttal": "We sincerely thank Reviewer VvJj for recognizing the novelty of our idea and for the detailed feedback. Below, we provide detailed responses to the questions and comments raised.\n\n> W1. Connection with guardrail systems.\n\nWe thank the reviewer for highlighting the connection. Our paper primarily focuses on the unlearning problem, aiming to evaluate existing unlearning methods under confidence-aware correlated knowledge. While both unlearning and guardrails seek to prevent disclosure of sensitive or harmful content, they represent different types of defenses: unlearning usually focuses on permanently removing knowledge via training, whereas guardrails typically operate at inference time to impose behavioral constraints. A systematic evaluation of guardrail methods under confidence-aware frameworks, in restricted knowledge domains, is beyond the scope of this work. \n\nWe conducted two preliminary experiments to showcase the applicability of our approach for guardrail evaluation: \n\n1. Inspired by the reviewer’s example, we tested a commercial model equipped with guardrails on a query related to self-harm. The direct prompt was blocked, but we manually constructed a subgraph involving various categories of legal / restricted substances, associated botanical sources, acquisition channels, and extraction methods. Probing the model indirectly constructed a subgraph and revealed the blocked content. Due to sensitivity, we omit details about the model, prompt, and subgraph. \n\n2. We consider a prompt-based guardrail approach. Specifically, we applied two system instructions to an LLaMA-8B model: (1) If asked about any relationship between subject and object, only respond 'Unknown', and (2) If explicitly asked about any (subject, relationship, object) triple, respond ‘Unknown’. We then evaluated the guardrail-protected model using our framework under the same setting and obtained instance-level UES = 1, subgraph-level UES = 0.502, and LoC(utility) = 0.979 for instruction (1); instance-level UES =1, subgraph-level UES = 0.441, and LoC = 0.986 for instruction (2). Our findings suggest that in the guardrail setting, subgraph-based evaluation reveals a notable overestimate of safeguard effectiveness. Extending this analysis to other guardrail mechanisms and attack strategies, especially in more restricted knowledge domains, remains an interesting future work direction.\n\n> W2.1. 3-hop limit and KG coverage concerns.\n\nOur choice of a 3-hop subgraph already supports one of our core findings: that unlearning effectiveness is significantly overestimated when correlated knowledge is not accounted for. Moreover, 3-hop reasoning is widely adopted in existing multi-hop knowledge evaluation benchmarks [1]. Admittedly, if an adversary were to exploit longer reasoning chains, we would need to expand the subgraph accordingly, which would increase the time complexity. Improving the efficiency of deeper subgraph extraction remains an interesting direction for future work. \n\nAs for reference knowledge graphs (KGs) may introduce blind spots, we agree that it approximates the model’s internal knowledge, given that pretraining data is typically unavailable. However, in real-world deployment, model holders often possess proprietary corpora or training data, which can support building richer reference KGs for more accurate approximation.\n\n[1] Mquake: Assessing knowledge editing in language models via multi-hop questions.\n\n> W2.2 & Q3. Calibration robustness and threshold selection. \n\nWe thank the reviewer for raising this concern. We clarify that our entropy-based confidence calibration is not fragile. We carefully curated a validation set of 740 pos/neg pairs. The set covers all relation types in the KG and was manually evaluated to better ensure consistency between the validation set and the unlearning evaluation. Furthermore, LLaMA3 and Qwen2.5 were calibrated separately using distinct temperature values, ensuring that the temperature is tailored to the individual model's state.\n\nConcerning the choice of the threshold $u^* = 1$, we would like to clarify that this value is not selected arbitrarily. As explained in Sec.4.2 and App.G, $u^* = 1$ represents the highest possible entropy under the condition that the model selects \"Yes\" as the most likely answer among the three options. It essentially captures the worst case where \"Yes\" remains the top prediction while the model assigns equal probability to \"Yes\" and one of the other two choices. This makes the threshold interpretable, as the resulting subgraphs include all triples with all levels of confidence as long as “Yes” is the argmax prediction.\n\n> W2.3. LLM judge reliability and pipeline brittleness.\n\nWe acknowledge that the LLM judge may occasionally exhibit inconsistencies, a known limitation of current models. To evaluate its alignment with human reasoning, we expanded our validation set to 100 subgraphs and computed inter-annotator agreement between LLM-human and human-human pairs (see **Table R1** in our response to Reviewer V8kE). We also manually examined the model’s reasoning in all cases and confirmed that its judgments relied strictly on the provided subgraph evidence, without invoking external knowledge or hallucinated entropy. To assess consistency, we prompted the LLM to score each triple three times independently. The resulting std across runs was 0.132, with most predictions remaining stable. Regarding prompt sensitivity, we observed that GPT-o4-mini produced reliable judgments when provided with clearly defined, objective instructions. Moreover, as discussed in Q2, applying the same prompt to GPT-4-Turbo and Claude yielded closely aligned results. While some errors may propagate through subgraph construction and evaluation, in practice, model holders with training data and stronger judges can more accurately assess the model’s internal knowledge.\n\n> W3. Shared bias in using commercial LLMs as judges. \n\nWe agree that different judge models may exhibit different sensitivities in assessing inferability. While our framework conceptually treats the judge as a proxy for human evaluation, we adopt LLM-based judges primarily to enable scalable and automated assessment. In our current experiments, we use a judge model from a different provider than the probed LLMs to reduce the risk of shared training artifacts. In practice, we believe that when budget and resources permit, incorporating multiple diverse LLMs as judges can further enhance the robustness and reliability of the evaluation outcomes.\n\n> W3.1 & Q4. Scalability and cost-efficiency of LLM-based evaluation \n\nIn our current setting, each evaluation requires a single API call per target unlearning triple along with its corresponding confidence-aware subgraph, resulting in 200 calls for evaluating 200 targets. With batched inference using GPT-o4-mini, the total cost for a full evaluation is bounded by US$1.50. As reported in App.D (Lines 1060–1063), the average latency for evaluating one subgraph using GPT-o4-mini is around 15s, and this can be further reduced via parallelization. We agree that further reducing computational overhead is essential for scaling to millions of triples. As the reviewer suggested, future work may explore distilling powerful LLM judges into lightweight judges or developing pruning strategies to construct more efficient subgraphs.\n\n> Minor point. \n\nWe thank the reviewer for the thoughtful point. While we acknowledge that LLM-based judges and extracted subgraphs are not flawless, we have taken several steps to enhance their reliability: comparing their judgments with human judge, verifying consistency through multi-round reasoning, and inspecting the fidelity between reasoning trace and subgraph evidence. We also consider entropy-based confidence calibration for subgraph extraction. Note that our core contribution is introducing the first systematic framework for evaluating unlearning methods via confidence-aware correlated subgraphs and our results consistently reveal substantial overestimation in existing unlearning performance.\n\n> Q1: Blind spots due to incomplete KG. \n\nWe thank the reviewer for the insightful question. We acknowledge that relying on a reference KG like YAGO3-10 may introduce blind spots due to missing entities or relations (potentially higher UES scores). However, the key message of our paper is that even with public reference KG, confidence-aware knowledge correlations already reveal a significant overestimation on the performance of existing unlearning methods. In scenarios where model holders typically have access to the full training corpus, they can construct richer reference KGs from unstructured text, allowing for more accurate probing of retained knowledge. \n\nOur current experiments focus on Wikipedia-based knowledge, in line with prior work in the unlearning literature [1,2]. Extending the framework to domain-specific or long-tail facts and evaluation with specially constructed KGs, remains an interesting direction for future work.\n\n[2] Rwku: Benchmarking real-world knowledge unlearning for large language models.\n\n> Q2: Cross-judge validation.\n\nWe thank the reviewer for the suggestion. Due to budget constraints of closed-source APIs, our main experiments use GPT-o4-mini as the judge. To evaluate robustness, we additionally assess consistency across GPT-o4-mini, GPT-4-Turbo, and Claude 4 on the validation set using identical prompts. Each model performs three independent reasoning passes, and we report average ratings. The Pearson correlations among three models are 0.812, 0.828, 0.809, respectively. Average rating std are 0.132, 0.280, and 0.229, respectively. These results indicate that inferability scores are reasonably consistent across strong LLM judges.\n\n> Discussion in Limitations.\n\n\nThank you for the suggestion. We will add a detailed cost analysis and explanation of model choice in App.D, and expand the Limitations to discuss potential concerns with LLM judges."}]

Author rebuttal:

Meta-review: This paper proposes a novel framework for evaluating knowledge unlearning in LLMs. Unlike existing approaches that primarily test for verbatim recall of forgotten facts, the framework introduces correlation- and confidence-aware evaluation to capture implicit knowledge retention. The method leverages supporting subgraphs derived from external knowledge graphs (e.g., YAGO3-10, Wikidata) and employs LLM judges to perform reasoning over these subgraphs. Through experiments on LLaMA and Qwen, and across popular unlearning methods, the paper demonstrates that current evaluation practices significantly overestimate unlearning effectiveness by overlooking inferable and correlated knowledge.

Strengths:
1. The idea of assessing unlearning effectiveness through correlation- and inference-aware reasoning is new and relevant, addressing a clear gap in the literature.
2. The finding that existing unlearning methods overestimate their effectiveness is impactful and may influence the design of future techniques.
3. The introduction of a subgraph-based evaluation protocol, including confidence calibration, offers a structured approach to a complex problem.

Weaknesses:
1. Reliance on external knowledge graphs limits applicability to real-world unlearning cases where such resources are unavailable. 
2. Constraints such as 3-hop subgraph extraction may miss longer reasoning chains. Confidence calibration thresholds appear arbitrarily chosen, and sensitivity analyses are lacking. The multi-step pipeline (subgraph extraction, confidence scoring, LLM judgment) is fragile, with error compounding across stages.
3. The paper does not sufficiently discuss or empirically compare against related work in knowledge editing, which often overlaps with unlearning. 

Overall, this paper tackles an important and underexplored aspect of unlearning evaluation with a creative framework. The contribution is conceptually strong and timely, but the methodology also has some weaknesses in scalability, robustness, and external dependence.

After the discussions, all the reviewers are positive on this work.
Decision: Accept (poster)