Title: Do LLMs Really Forget? Evaluating Unlearning  with Knowledge Correlation and Confidence Awareness
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine Unlearning

Knowledge Unlearning

Knowledge Correlation

Large Language Models
Abstract: Machine unlearning techniques aim to mitigate unintended memorization in large language models (LLMs). However, existing approaches predominantly focus on the explicit removal of isolated facts, often overlooking latent inferential dependencies and the non-deterministic nature of knowledge within LLMs. Consequently, facts presumed forgotten may persist implicitly through correlated information. To address these challenges, we propose a knowledge unlearning evaluation framework that more accurately captures the implicit structure of real-world knowledge by representing relevant factual contexts as knowledge graphs with associated confidence scores. We further develop an inference-based evaluation protocol leveraging powerful LLMs as judges; these judges reason over the extracted knowledge subgraph to determine unlearning success. Our LLM judges utilize carefully designed prompts and are calibrated against human evaluations to ensure their trustworthiness and stability. Extensive experiments on our newly constructed benchmark demonstrate that our framework provides a more realistic and rigorous assessment of unlearning performance. Moreover, our findings reveal that current evaluation strategies tend to overestimate unlearning effectiveness.

Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness

Abstract

Machine unlearning techniques aim to mitigate unintended memorization in large language models (LLMs). However, existing approaches predominantly focus on the explicit removal of isolated facts, often overlooking latent inferential dependencies and the non-deterministic nature of knowledge within LLMs. Consequently, facts presumed forgotten may persist implicitly through correlated information. To address these challenges, we propose a knowledge unlearning evaluation framework that more accurately captures the implicit structure of real-world knowledge by representing relevant factual contexts as knowledge graphs with associated confidence scores. We further develop an inference-based evaluation protocol leveraging powerful LLMs as judges; these judges reason over the extracted knowledge subgraph to determine unlearning success. Our LLM judges utilize carefully designed prompts and are calibrated against human evaluations to ensure their trustworthiness and stability. Extensive experiments on our newly constructed benchmark demonstrate that our framework provides a more realistic and rigorous assessment of unlearning performance. Moreover, our findings reveal that current evaluation strategies tend to overestimate unlearning effectiveness. Our code is publicly available at https://github.com/Graph-COM/Knowledge_Unlearning.git .

1 Introduction

Large language models (LLMs) have achieved widespread adoption across diverse application domains due to their remarkable capacity to acquire and encode complex, interdependent knowledge from vast web corpora [1,2]. However, this impressive capability simultaneously introduces critical risks. Notably, LLMs may inadvertently memorize and reproduce copyrighted content [3,4], amplify social or cultural biases [5], or reveal sensitive and private information [6]. Such issues not only jeopardize user trust but can also contravene ethical principles and regulatory frameworks governing responsible AI deployment [7, 8]. In response, machine unlearning has emerged as a promising approach for selectively removing specific data points, concepts, or factual knowledge from pretrained models [9-11].

User / Data Curator

Unlearning

Request

painted

Target LLM

Unlearned LLM

Unlearning

Stage

Milan

Gran Caval o

Florence

Paris

Renaissance

LLM Confidence Level

Confident

Somewhat Confident

Uncertain Confident

Da Vinci

Knowledge Base

Evaluation

Stage

Construct Supporting Subgraph

Querying Target LLM

Output: Subgraph with Confidence

Provide Potential

Entities & Relations

Da Vinci

Louvre Museum

Paris

Renaissance

influenced

Confidence-aware

Supporting Subgraph

Target Unlearn Knowledge

In-context learning,

Human Evaluation Comparison, etc.

Input to LLM Judge

Powerful LLM Judge

No Leverage External Knowledge

Judge guidance

Pre-unlearning

Post-unlearning

Enhancing Target LLM Reliability:

Confidence Calibration

Unlearned Facts

Pre-unlearning

Post-unlearning

Confident

Somewhat Confident

Uncertain Confident

LLM Confidence Level

Despite progress in unlearning factual knowledge from LLMs, current methods mostly tackle the problem on the surface. Whether by targeting individual training examples [12-14] or erasing sets of facts linked to specific topics [11, 15], these approaches share a critical flaw: they focus on direct deletion while ignoring how deeply knowledge is interconnected within LLMs [16-18]. This is a serious oversight. It means that even when a fact seems 'erased' (disappearing from direct questions), it often still lurks within the model, ready to be figured out from other related pieces of information the LLM still holds. The targeted fact can then be pieced back together through indirect questions, shattering the idea that unlearning truly worked and causing a loss of trust. For example, an LLM might 'forget' that 'Mount Fuji is a volcano,' only for this to be easily deduced because it still knows that 'Mount Fuji has a crater at the summit' and 'craters are formed by volcanic activity.' This same challenge also indirectly echoes the findings in knowledge editing studies [19, 20]: changing isolated facts often doesn't ripple through the model's wider reasoning, showing that surface-level unlearning isn't genuine unlearning at all. This major gap shows that a new approach is vital. Therefore, we propose to significantly expand the concept of knowledge unlearning (KU): it must go beyond simple deletion to actively take apart the underlying knowledge structures that support the target information.

Beyond overlooking correlations among knowledge facts, existing unlearning evaluation frameworks may also be inadequate in modeling how knowledge is organized and processed within LLMs. First, real-world knowledge exhibits diverse and complex correlation patterns, with inferential relationships that may be non-deterministic or context-dependent [21]. Consequently, evaluation protocols based on manually defined inference rules or fixed reasoning chains often fail to capture the probabilistic nature of these factual dependencies [22]. For example, consider unlearning the fact that Person A is the CEO of Company X . While this fact might be explicitly stated, it could also be inferred from indirect clues, such as references to Person A overseeing operations, attending board meetings, or signing executive decisions, each carrying varying degrees of evidential strength. Second, current methods frequently assume that facts within the unlearning dataset are already well internalized by the LLM. In practice, however, an LLM's grasp of specific facts, particularly those involving rare entities or domain-specific relations, can vary significantly [14]. LLMs may exhibit only partial or uncertain knowledge of these facts even after fine-tuning. If such variability is not accounted for, it can lead to inaccurate or inflated estimates of unlearning effectiveness.

We propose a novel and realistic formulation of knowledge unlearning, grounded in a confidenceaware perspective of how factual knowledge is represented within LLMs. Our framework is designed to reflect the complexities of real-world knowledge systems, where facts are often uncertain, interdependent, and mutually inferable. To achieve this, we represent the knowledge embedded in an LLM as a confidence-aware knowledge graph. In this graph, each triple denotes a piece of knowledge and is associated with a confidence level derived from the model's predictions. Given a target triple for unlearning, we then probe the LLM to extract a subgraph of correlated facts that could potentially enable the inference of this targeted triple, subsequently analyzing this subgraph to determine the extent to which the target knowledge persists within the LLM.

To effectively automate unlearning evaluation, we propose employing a powerful LLM judge (e.g., GPT-series models) to act as an adversarial agent. This LLM judge employs carefully crafted prompts and specific calibration procedures to reason over the extracted knowledge subgraph and assess whether and to what extent the target triple remains inferable. Such a protocol moves beyond

superficial-level triple recall, capturing the residual inferential capabilities embedded within the model. Furthermore, to validate our LLM-based evaluator, we compare its judgments against those of strong human adversaries to assess their alignment and reliability. Extensive experiments on large-scale, real-world encyclopedic datasets demonstrate our framework's applicability while also highlighting the limitations of existing unlearning methods and evaluation techniques. Our experimental analysis reveals several key insights. First, correlated knowledge supports inference and can significantly reduce unlearning effectiveness, even when the target triple appears superficially erased. Second, even low-confidence yet semantically related knowledge can substantially compromise unlearning, underscoring the importance of capturing weaker associations. Finally, with careful prompt design and calibration, the LLM judge enables automated evaluation of unlearning effectiveness while producing judgments closely aligned with those of human experts. Accompanying our findings, we release two key resources: first, a benchmark for LLM probing that reflects knowledge interdependence, derived from real-world knowledge datasets; and second, an evaluation protocol tailored to the novel concept of knowledge unlearning presented herein.

2 Related Work

Evaluating unlearning in LLMs remains a core challenge, with prior studies proposing various metrics to measure the effectiveness of removing specific training instances or factual knowledge [23]. Early approaches such as WHP [24] assessed unlearning by eliminating Harry Potter-related content using completion-based and token-probability-based metrics, while Wei et al. [25] investigated methods to prevent the generation of copyrighted content. Subsequent benchmark TOFU [11] considered entity-level unlearning and compared pre- and post-unlearning question-answering performance using fictitious author biographies. MUSE [13] provided a comprehensive evaluation across six distinct dimensions, and WMDP [26] specifically focused on unlearning harmful knowledge to mitigate malicious use. Additionally, RWKU [27] proposed benchmarks tailored for real-world knowledge scenarios, and Wei et al. [14] introduced a minority-aware framework to identify high-risk data points for unlearning. However, these approaches generally treat knowledge independently, overlooking the interdependencies between target facts and related knowledge in LLMs. This limitation was pointed out in [28], which first explored multi-fact interactions during unlearning but relied on deterministic knowledge modeling and rule-based evaluations, which limits their practical applicability.

Another related research area is knowledge editing, which focuses on updating specific factual information within LLMs [29]. Existing evaluation frameworks typically assess whether models correctly recall edited facts in response to single-hop queries [30-38]. More recent studies have incorporated multi-hop factual chains as evaluation tools to determine if model updates successfully propagate through indirect reasoning paths [19, 20]. In contrast, our work concentrates on the knowledge unlearning task, which aims to entirely remove target knowledge. Distinctly, our framework explicitly extracts correlated knowledge subgraphs within the LLMs that support the target knowledge fact, whereas knowledge editing evaluations frequently rely on deterministic factual chains.

3 The Formulation of Knowledge Unlearning

Confidence-Aware Knowledge Modeling in LLMs. We consider the scenario where a target pretrained LLM, denoted as M pretrain, is tasked with unlearning relational knowledge. We formulate the factual knowledge embedded in this LLM as a confidence-aware knowledge graph G = ( E , R , T U ) , where E denotes the set of entities, R represents the set of relation types. Each knowledge fact is encoded as a knowledge quadruple t = ( s, r, o, u ) , consisting of a subject entity s , a relation r , and an object entity o , along with a non-negative valued confidence score u ∈ U that reflects the LLM's degree of belief in the fact; T U ⊆ E × R × E × U denotes the set of all confidence-aware facts in G .

In practice, a given target fact e = ( s, r, o ) ∈ T is often correlated with a subset of related facts T e = { t 1 , t 2 , . . . } ⊆ T U in M pretrain, from which it can potentially be inferred. For instance, the triple ( Paris , Capitalof , France ) may be supported by related facts like ( Paris , hosts , Élysée Palace , u 1 ) and ( Élysée Palace , isGovernmentSeatof , France , u 2 ) . The strength of such inference depends on both the structural relationship between e and the supporting facts, as well as the associated confidence scores u 1 and u 2 within T e . To quantify the strength of such inference, it requires introducing an intrinsic judge function f : 2 T U ×T → Y , where 2 T U denotes the power set of knowledge triples. Given a supporting subset T e and a target triple e , the function f ( T e , e ) outputs an inference score

quantifying how strongly the target fact e can be derived from the supporting set T e . A higher inference score indicates stronger support, while lower values correspond to weaker inferability.

Knowledge Unlearning. Let A be an unlearning method. The knowledge graph induced by the LLM M A unlearn (after applying A ) is G A . A target triple e = ( s, r, o ) is deemed unlearned if it is robustly erased from M A unlearn 's accessible knowledge. Specifically, even a strong adversary should not be able to infer e through strategic queries to M A unlearn concerning related facts in its latent knowledge graph G A , whether these queries are direct or indirect probes of the relation r between s and o . Importantly, simply causing the model to deny e upon direct questioning is often insufficient; deeply interconnected knowledge structures from pretraining may still allow for its inference. The potency of such inference attacks is a function of the adversary's reasoning power, captured by the intrinsic judge function f . This provides the basis for the following definition of knowledge unlearning :

Definition 1 (Knowledge Unlearning) . Given a target triple e = ( s, r, o ) for unlearning and an adversary with intrinsic judge function f , let M A unlearn denote the model obtained after applying unlearning method A with induced knowledge graph G A . We say A achieves γ -knowledge unlearning if f ( G A , e ) ≤ γ , where γ specifies an upper bound on the residual inferability of e from M A unlearn .

The above knowledge-unlearning definition, while theoretically sound, faces practical evaluation challenges. First, accessing the model M A unlearn 's full knowledge representation G A is typically infeasible. To approximate the inference score f ( G A , e ) , we extract a localized supporting subgraph G A e ⊆ G A . This subgraph, centered on the target triple e and constructed using a real-world knowledge base as reference, captures relevant local supporting facts (details in Sec. 4). Our practical evaluations therefore assess approximate knowledge unlearning using f ( G A e , e ) . Traditional direct triple removal evaluations, termed instance unlearning , represent a special case where G A e reduces to the target triple and its associated confidence score, i.e., G A e = ( e, u e ) ∈ T U . Second, the definition presumes an intrinsic judge function f with comprehensive knowledge of all potential factual relationships. As this ideal is generally unattainable, human expert annotators or powerful LLM-based evaluators will be used as practical proxies for f .

4 Methodology

4.1 Overview of the Evaluation Protocol

To evaluate whether a target fact has been effectively unlearned, we build upon the definition of knowledge unlearning introduced in Sec. 3. We assess the inferability of a target triple e based on its supporting subgraph G A e and a proxy judge function f ( G A e , e ) to quantify the extent to which e remains inferable from retained knowledge. Successful unlearning requires not only the removal of the target triple but also the disruption of its underlying inferential structure. To instantiate this evaluation protocol, we adopt a two-stage approach: (1) Supporting Subgraph Extraction , where we probe the unlearned model M A unlearn to construct G A e ; and (2) Adversarial Inference via a Powerful Judge , where the inferability score is computed using a powerful LLM/human expert adversary.

4.2 Extracting Supporting Subgraphs from Model Beliefs

Given a target triple e = ( s, r, o ) , our goal is to retrieve its supporting subgraph G A e ⊆ G A encoded in the unlearned model M A unlearn that substantiates its inferential basis. We formalize this subgraph as a union of confidence-aware triples extracted from M A unlearn .

These triples collectively trace potential deductive pathways from the subject s to the object o within a bounded path length ℓ . The constraint of using a bounded path length for these deductive pathways aligns with observations that many reasoning tasks can be effectively resolved within a small number of hops [39].

Constructing such pathways requires probing M A unlearn with candidate triples. A key challenge arises as M A unlearn offers no explicit catalogue of entities or relations, and an exhaustive enumeration of all possibilities is computationally prohibitive. Therefore, to define a tractable search space, we determine candidate entities and relations by referring to an external real-world reference knowledge graph G ref (e.g., Wikidata). This reliance on

Filtered Edge

Retain Edge

an external, publicly accessible corpus is a practical necessity. Indeed, even adversaries attempting to indirectly probe an LLM's knowledge typically need to anchor their queries in commonly understood entities, relations, or schemata. While our G ref-guided approach is not perfect and cannot capture all latent knowledge, it provides a principled method for exploring deductive pathways grounded in relevant, publicly accessible information.

To be more robust against the potential missing information in G ref, our strategy for generating knowledge fact candidates from a frontier node v (initially s , then other entities along a path) involves retrieving its k -hop neighborhood in G ref. The union of these neighboring entities serves as the set of candidate objects. Candidate relations are similarly drawn from the available relation types in G ref . This approach assumes that the local neighborhood in the reference KG offers a reasonably comprehensive, albeit imperfect, approximation of the entities relevant for deductive reasoning that are implicitly known to M A unlearn . Ultimately, the resulting supporting subgraph is intended to capture the latent reasoning chains through which the target triple e may be logically derived.

Constructing the Support Subgraph. We build G e iteratively under a breadth-limited expansion strategy (illustrated in Fig. 2). Starting from the subject s , we probe the unlearned model M A unlearn with candidate triples of the form ( s, r ′ , o ′ ) . Whenever the model's confidence u for a triple exceeds a preset threshold u ∗ , the triple is added to the subgraph and its object o ′ becomes the new frontier. Here, we use u ≤ u ∗ to denote the excess case. In the next hop, starting from the new frontiers o ′ , we query triples ( o ′ , r ′′ , o ′′ ) where o ′′ s come from all candidate entities. After each round, we record the model's confidence scores and discard triples that are either rejected or judged uncertain. The expansion stops when the path-length limit ℓ is reached or no further confident triples can be found.

To properly estimate the model's confidence in a candidate triple ( s ′ , r ′ , o ′ ) , we adopt a multiplechoice querying protocol, where each triple is verbalized using a fixed natural-language template and presented to M A unlearn with answer choices { Yes , No , Unknown } . While an alternative might involve having M A unlearn directly output a numerical confidence score, our empirical investigations revealed this method to be inaccurate, which is consistent with existing studies on the challenges of calibrating LLM confidence [40]. Con-

Confidence Evaluation

Multiple-Choice

Query

Entropy

Calculation

Admission

Criteria

Supporting Subgraph Construction

All Possible Triples

k-hop Neighbors Extraction (Eg. k=2)

relation set

neighbors set:

Combine

Search

in Ref KG

2-hop Neighbors

Filtered edge

Retain edge

sequently, we find that more reliable confidence assessments are obtained by applying temperature scaling [41] to adjust the model's softmax output probabilities corresponding to these answer choices. This calibration process yields significantly more accurate confidence scores, as demonstrated by our experiments in Sec. 6.3. After calibration, we model the confidence space U in terms of entropy: H ( s ′ , r ′ , o ′ ) = -∑ i ∈{ Yes , No , Unknown } P i log 2 P i , where P Yes , P No , P Unknown are the calibrated probabilities for the respective answer choices. A triple is admitted to G A e iff the model selects Yes and H ( s ′ , r ′ , o ′ ) ≤ u ∗ . Recall that u ∗ is a predefined entropy threshold. In this paper, we set u ∗ = 1 , which corresponds to a worst-case scenario where the model follows instructions but assigns equal probability (0.5) to Yes and one other option ( No or Unknown ). The full mapping between u ∗ and minimum Yes probabilities is provided in App. G. This criterion filters out unconfident evidence, preserving only high-confidence relational structure. Overall, the entire knowledge probing process as above is illustrated in Fig. 3. The corresponding algorithm is provided in App. A, Alg. 1.

4.3 Adversarial Inference Assessment via Powerful LLM Judge

Given a target triple e and its associated supporting subgraph G A e , the evaluation of unlearning effectiveness relies on an adversarial inference scheme to assess the residual inferability of e through systematic exploration of G A e . While human experts provide a strong reference for reasoning-based evaluation, they are not scalable for large-scale assessment. To automate the process, we employ a powerful reasoning LLM as the judge. We monitor the alignment between the LLM judge and human expert evaluations on a subset of examples, with results reported in Sec. 6.3.

Effective reasoning by the LLM judge hinges on a carefully engineered prompt structure. This structure is built upon two foundational considerations: first, it mandates that the judge's reasoning be based exclusively on the provided subgraph G A e , encompassing its factual content and associated confidence scores without resorting to any external knowledge. Second, we re-

quire the judge to quantify inferability using a discrete 0-5 rating scale, where 0 signifies that the target triple e cannot be inferred and 5 denotes very high certainty in deducing e from G A e .

To further ensure the reliability and accuracy of these judgments, the prompt design inherently integrates clear evaluation objectives, explicit semantic definitions for all rating levels, entropy-based guidance that categorizes triple certainty into discrete confidence bins (as illustrated in Fig. 4. Task-specific instructions and in-context examples are also embedded to ensure consistent and accurate evaluation behavior. For the full prompt template with the interpretation of each rating score, see App. C.1.

Importantly, the LLM's judgments are expected to rely solely on G A e , treating it as ground truth, without leveraging any of its internal knowledge acquired during pretraining. Our experiments will verify this assumption by analyzing the model's reasoning traces.

5 Knowledge Unlearning Evaluation Datasets

Systematic evaluation of the unlearning protocol described herein necessitates a dataset where target triples, presumed to be retained

Task Description:

Given subgraph triples, determine whether

the target triple can be inferred...

Entropy

Interpretation:

0.00-0.25

→

→

Confident

0.50-0.75

→

Less Confident  0.75-1.00

→

Not Confident

0 → No logical path exists...

1 → Very weak support...

2 → Some weak logical connection...

3 → Moderate path exists...

4 → Confident support...

5 → Strong support...

Scoring

Rules:

Evaluation

Guidelines:

1. Analyze direct paths..

2. Analyze support paths...

3. Assess the uncertainty of each path...

4. Give the score...

Input:

Target Triples:

Subgraph

Triples:

Several

Examples:

Example for Score 5:

(Rome, capital_of, Italy),

(Italy, located_in, Europe) form support path...

Both triples have very low entropy...

located_in can be logically

implied by capital_of and located_in...

(Rome, located_in, Europe)

(Rome, capital_of, Italy) entropy 0.08

(Italy, located_in, Europe) entropy 0.12

Example for Score 0:

Example for Score 4:

...

Support path very strong → Final Score: 5

by the LLM, are accompanied by a rich network of correlated and semantically coherent triples capable of supporting target triple inference. To create such a dataset for realistic scenarios, we utilize real-world knowledge bases, specifically selecting YAGO3-10 [42]. YAGO3-10 is a largescale knowledge graph built from Wikipedia, Wikidata, and WordNet [43], containing over one million relational triples, 123,182 entities, and 37 distinct relation types. Crucially, its Wikipedia and Wikidata foundations are pertinent as these sources are common in the pretraining corpora of modern LLMs (e.g., [16,44,45]), making it likely that YAGO3-10's factual triples are retained by these models. Moreover, the relations defined in YAGO3-10 exhibit various inferential structures (more illustrations in H). For our experiments, we target specific YAGO3-10 triples for unlearning, using the full YAGO3-10 graph as the reference G ref for support subgraph extraction.

Target Triples and Supporting Subgraphs Extraction. From each LLM for evaluation, we extract 200 unlearning target facts. To ensure these targets constitute factual knowledge genuinely retained by the model, they are identified using a carefully selected knowledge-probing template (to be introduced later) coupled with an entropy-based confidence filter; the specifics of how to measure confidence are detailed in Sec. 4.2. Supporting subgraphs for each unlearning target are constructed by querying all relations in G ref and retrieving candidate knowledge facts within a 3-hop neighborhood ( k = 3 ) at each expansion step. The resulting subgraphs are constrained to a maximum path length of l = 3 , which reflects a reasonable inference depth in line with prior work [19,39].

Knowledge Probing Method. Effective knowledge probing of LLMs is crucial for both our dataset construction and subsequent evaluation phases. Following the approach in [27], we utilized GPT-4 to generate a diverse pool of candidate prompt templates. These templates were specifically designed for our multiple-choice query format in Sec. 4.2. To evaluate these candidate templates, we constructed a comprehensive validation set. For each of the 37 relations in YAGO3-10, we selected 10 positive examples, consisting of factual triples that hold the given relation within YAGO3-10. Concurrently, for each relation, we generated 10 negative examples composed of counterfactual entity pairs that do not hold that relation. These negative examples underwent manual verification by human annotators to confirm their factual incorrectness, despite being constructed to appear plausible (i.e., the entity types were compatible with the relation). This process yielded a validation set totaling 370 positive and 370 negative samples. Each candidate prompt template was then systematically evaluated based on its accuracy over this validation set. The template that achieved the highest accuracy was selected as our definitive query prompt for all LLM interactions (see App. C.2 for the complete template).

6 Experiments

6.1 Experimental Setup

Unlearning Methods. For each target LLM, we evaluate the following popular unlearning approaches in the literature: Gradient Ascent (GA) [46-48] aims to erase the influence of unlearn triples by

reversing gradient updates. Random Labels (RL) [46, 49] disrupts memorization by replacing correct labels with random tokens during training. Negative Preference Optimization (NPO) [50] formulates unlearning as reversing the model's preference toward unlearned data by optimizing likelihood ratios against the pretrained model. NegGrad + [51] combines gradient ascent on the unlearn set with gradient descent on a non-unlearn set to retain performance. Finally, SCRUB [51] adopts a student-teacher framework, using KL divergence to balance knowledge removal from the unlearned data with the retention of non-target data. To ensure fair comparison, we adopt a unified computational budget protocol for all methods, following [14]. Formal definitions, implementation details, and per-method budget allocations are provided in App. B.

Unlearning Effectiveness Metrics. We propose the Unlearning Effectiveness Score (UES) to quantify how effectively an unlearning method A reduces target knowledge inferability by comparing pretrained ( M pretrain) and unlearned ( M A unlearn ) models. Given unlearning targets D forget, for each e ∈ D forget , let G e and G A e be its supporting subgraphs from M pretrain and M A unlearn respectively. UES is the average normalized relative decrease in an LLM-assessed inference score UES = E e ∈ D forget [ f judge ( G e ,e ) -f judge ( G A e ,e ) f judge ( G e ,e ) ] . Here, f judge outputs a discrete score from { 0 , 1 , ..., 5 } (detailed in Sec. 4.3). UES is upper-bounded by 1; higher values denote more effective unlearning (greater inferability reduction). UES = 0 implies no effect, and negative values indicate increased inferability after unlearning. The metric is well-defined as f judge ( G e , e ) is strictly positive for verified retained targets (Sec. 5). For instance-level unlearning ("Inst."), targeting direct triple knowledge, the metric is computed by replacing the subgraph G e (or G A e ) with e paired with its confidence u e before unlearning (or after unlearning, resp.). Additionally, we employ Recall [28] E e ∈ D forget [ | G e ∩ G A e | | G e | ] . A metric that measures the proportion of original supporting subgraph G e 's triples preserved in the unlearned model's subgraph ( G A e ). Since these subgraphs reflect inferential support for e , a lower Recall value indicates that more of this supporting structure has been successfully unlearned.

Utility Preservation. Model utility is assessed by the retention of local knowledge near unlearned triples and the preservation of general model capabilities [27]. Local Consistency (Loc). To assess local utility preservation, we sample triples within a 3-hop neighborhood of the target triple's head or tail entities from the reference knowledge graph G ref, excluding triples already included in the supporting subgraph G e . These neighboring triples are considered irrelevant to the target inference and should remain unaffected by unlearning. The Loc score measures the consistency of model predictions ( Yes , No , or Unknown ) on these neighbor triples before and after unlearning. A higher Loc score indicates better preservation of local factual integrity. Importantly, Loc captures multidirectional knowledge shifts, such as facts changing from Yes to No or vice versa. Further analysis is provided in App. F. General Model Utility. General capabilities are assessed from two perspectives: General Knowledge (Gen) , measured on MMLU [52] (a multi-domain multiple-choice benchmark) via 5-shot perplexity-based answer ranking and Reasoning Ability (Rea) , measured on BBH [53] (a suite of 27 challenging tasks) by 3-shot exact match accuracy with chain-of-thought prompting.

General Settings. We conduct our unlearning experiments using two widely adopted open-source LLMs: LLaMA3-8B-Instruct [54] and Qwen2.5-7B-Instruct [55]. The unlearning procedures are implemented via either full model fine-tuning or parameter-efficient tuning using LoRA [56]. The reported results are averaged over the total of 200 target triples. For the utility evaluation on Loc, we sample a set of 2,000 neighboring triples (10 × the size of the unlearning target set), reflecting realistic unlearning-to-retention ratios as discussed in prior literature [57]. During unlearning, we consider two formats for presenting unlearning targets: (i) converting each triple into a natural sentence ( unlearn with sentence ), and (ii) framing it as a multiple-choice question with options (Yes, No, Unknown) ( unlearn with QA ). To ensure a fair comparison across methods, we fix the computational budget: 10 unlearning epochs for GA, RL, and NPO, and 5 epochs for NegGrad + and SCRUB, which require additional knowledge triples to preserve utility (detailed in App. B). For epoch selection, we follow a similar protocol introduced in [13, 14, 28]: if the model's utility (Loc) exceeds 0.8 at any epoch, we select the last epoch satisfying this criterion; otherwise, we select the epoch with the highest utility below the threshold. During evaluation, we use GPT-o4-mini as the adversarial judge. Full implementation details, including hyperparameter configurations, are provided in App. D.

Method

LLaMA-8B-Instruct

Qwen2.5-7B-Instruct

Unlearning Effectiveness

Utility Retention

Unlearning Effectiveness

Utility Retention

UES (Inst.) (

↑

)

UES (Ours) (

↑

)

Recall (

↓

)

↑

)

↑

)

Rea (

↑

)

UES (Inst.) (

↑

)

UES (Ours) (

↑

)

Recall (

↓

)

↑

)

↑

)

Rea (

↑

)

No Unlearn

-

-

-

-

0.633

0.567

-

-

-

-

0.637

0.581

GA (Full)

0.157

0.076 (51.59%

↓

)

0.257

0.951

0.630

0.568

-0.020

-0.093 (365%

↓

)

0.977

0.964

0.632

0.581

RL (Full)

0.027

0.007 (74.07%

↓

)

0.779

0.994

0.631

0.565

0.176

0.077 (56.25%

↓

)

0.883

0.975

0.633

0.576

NPO (Full)

0.064

0.046 (28.12%

↓

)

0.890

0.956

0.629

0.566

0.140

0.058 (58.57%

↓

)

0.880

0.882

0.636

0.580

NegGrad+ (Full)

0.006

0.001 (83.33%

↓

)

0.649

0.977

0.629

0.564

0.702

0.534 (23.94%

↓

)

0.489

0.848

0.630

0.578

SCRUB (Full)

0.037

-0.003 (108.10%

↓

)

0.919

0.957

0.628

0.569

0.609

0.457 (25.12%

↓

)

0.402

0.739

0.627

0.583

GA (LoRA)

0.107

0.059 (44.85%

↓

)

0.231

0.960

0.630

0.565

0.122

0.022 (81.97%

↓

)

0.802

0.827

0.636

0.577

RL (LoRA)

0.027

0.017 (37.03%

↓

)

0.618

0.997

0.633

0.563

0.026

-0.052 (300.00%

↓

)

0.928

0.934

0.634

0.580

NPO (LoRA)

0.181

0.010 (94.48%

↓

)

0.575

0.989

0.632

0.566

0.389

0.244 (37.25%

↓

)

0.651

0.943

0.638

0.576

NegGrad+ (LoRA)

0.154

0.030 (80.51%

↓

)

0.472

0.997

0.635

0.563

0.195

0.099 (49.23%

↓

)

0.840

0.967

0.640

0.581

SCRUB (LoRA)

0.211

0.033 (84.36%

↓

)

0.329

0.997

0.631

0.565

0.715

0.516 (27.83%

↓

)

0.412

0.746

0.619

0.573

Unlearn with QA Templates

No Unlearn

-

-

-

-

0.633

0.567

-

-

-

-

0.637

0.581

GA (Full)

0.796

0.622 (21.86%

↓

)

0.270

0.262

0.633

0.565

0.890

0.816 (8.31%

↓

)

0.164

0.733

0.638

0.580

RL (Full)

0.134

0.060 (55.22%

↓

)

0.486

0.992

0.633

0.568

0.051

-0.012 (123.53%

↓

)

0.932

0.882

0.640

0.583

NPO (Full)

0.232

0.102 (56.03%

↓

)

0.457

0.957

0.634

0.567

0.794

0.705 (11.19%

↓

)

0.211

0.683

0.638

0.581

NegGrad+ (Full)

0.956

0.904 (5.44%

↓

)

0.009

0.175

0.627

0.553

0.666

0.614 (7.81%

↓

)

0.139

0.584

0.631

0.576

SCRUB (Full)

0.959

0.658 (31.37%

↓

)

0.236

0.145

0.638

0.572

0.963

0.941 (2.29%

↓

)

0.043

0.607

0.610

0.580

GA (LoRA)

0.145

0.096 (33.79%

↓

)

0.305

0.996

0.634

0.566

0.474

0.289 (39.03%

↓

)

0.630

0.929

0.640

0.580

RL (LoRA)

0.091

0.059 (35.16%

↓

)

0.384

0.968

0.633

0.566

0.016

-0.025 (256.25%

↓

)

0.971

0.961

0.639

0.581

NPO (LoRA)

0.690

0.585 (15.22%

↓

)

0.063

0.936

0.633

0.565

0.055

0.004 (92.7%

↓

)

0.965

0.929

0.641

0.578

NegGrad+ (LoRA)

0.433

0.324 (25.17%

↓

)

0.664

0.993

0.631

0.564

0.098

0.054 (44.90%

↓

)

0.853

0.943

0.647

0.581

SCRUB (LoRA)

0.108

0.032 (70.37%

↓

)

0.481

0.878

0.637

0.569

0.914

0.777 (14.99%

↓

)

0.286

0.717

0.594

0.574

6.2 Main Results

Correlated Knowledge Supports Inference and Reduces Unlearning Effectiveness. We report the unlearning effectiveness and utility metrics of each unlearning method under both instance unlearning (Inst.) and our correlated knowledge tracing framework in Tab. 1. When comparing each method's UES under instance unlearning versus our supporting subgraph setting, we observe a substantial drop in effectiveness. For most cases, the UES decreases by at least 20%, with more than half of the settings exhibiting a reduction exceeding 30%. We further analyze specific unlearned triples and observe that certain triples appear successfully unlearned at the instance level, yet remain inferable when considering their supporting subgraphs. For instance, the triple (B+H_Architects, created, Brookfield_Place_(Toronto)) is effectively unlearned (entropy: 0.252 before unlearning, 1.127 after), but the evaluator LLM can still infer it through related facts such as (B+H_Architects, isKnownFor, Brookfield_Office_Properties) (entropy: 0.134 before, 0.512 after) and (Brookfield_Office_Properties, owns, Brookfield_Place_(Toronto)) (entropy: 0.110 before, 0.441 after) present in the supporting subgraph (see additional examples in App. E.3).

Meanwhile, we observe that when local utility is well preserved (e.g., Loc ≥ 0 . 8 ), meaning over 80% of entity-centric triples in the neighborhood are retained, the structure of the supporting subgraph G A e , as reflected by the recall score, also remains largely intact. This preserved structure enables more potential inferences, thereby reducing unlearning effectiveness. In contrast, when utility degrades sharply, much of the supporting context is disrupted, limiting inferential paths (e.g., QA-based full model unlearning for Qwen2.5 in Tab. 1).

We further compare the overall performance of unlearning methods in Tab. 1. No method consistently achieves a favorable trade-off between unlearning effectiveness (UES) and utility (Loc). When Loc ≥ 0 . 8 , GA, NPO, NegGrad+, and SCRUB generally outperform RL across both LLMs. These four methods show comparable performance, each achieving the best results in different settings.

Sentence-based unlearning provides a more utility-preserving alternative to QA-based unlearning. As shown in Tab. 1, under knowledge probing via QAs, employing traditional sentence templates under identical unlearning configurations results in superior utility retention (measured by Loc) and less aggressive average unlearning effectiveness compared to the QA-based approach. This advantage likely arises because sentence-based contexts elicit more conservative parameter updates, whereas QA-based unlearning explicitly targets specific model outputs (e.g., Yes ), causing abrupt behavioral shifts and potential instruction-following degradation.

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

dence) on UES. Howdoes the confidence score affect unlearning effectiveness? In our framework, we model the target LLM's confidence in knowledge triples. Here, we further investigate how these confidence scores influence unlearning outcomes by

varying the threshold u ∗ used in generating supporting subgraphs. As shown in Fig. 5, we report results under the LLaMA3-8B-Instruct LoRA QA setting. We observe that decreasing the entropy threshold u ∗ , i.e., requiring higher LLM confidence, filters out more supporting inferences, thereby increasing unlearning effectiveness. This highlights that it is crucial to model the confidence of LLMs on the knowledge facts, as low-confidence knowledge can still contribute meaningfully to inference and introduce information leakage. Refer to App. F for more results.

6.3 Justification on the Reliability of the Target LLM and LLM Judge

In this subsection, we report results validating the reliability of both the target LLM and the LLMbased judge. (1) Target LLM Calibration. To interpret entropy calculation, we first calibrate the token probabilities (i.e., confidence) output by the target LLM. As mentioned in Sec. 4.2, we calibrate the optimal temperature that aligns predicted probabilities with actual model behavior. Using the validation set detailed in Sec. 5) with positive and negative knowledge triples, we prompt the LLM with multiple-choice questions and record the model's predicted answer based on the argmax token. We then compute the Expected Calibration Error (ECE) [41] separately on 'Yes' and 'No' predictions, measuring how well model confidence aligns with accuracy (lower is better). As shown in Fig. 6, the temperature-scaled output distributions of both models exhibit strong alignment between the predicted probabilities and accuracy, indicating reliable confidence estimates. (2) LLM Judge Reliability. To ensure that the LLM judge (GPT-o4-mini) provides trustworthy inference assessments, we validate its behavior from two complementary aspects. First, we qualitatively verify that the judge adheres strictly to the evaluation instructions by reasoning solely over the provided supporting subgraph without incorporating external knowledge. To this end, we randomly sample 50 target triples that are known to be memorized by the target LLM and construct their corresponding supporting subgraphs. We then manually inspect the LLM judge's responses on these subgraphs and find that in all cases, the model faithfully follows the instruction without relying on external information. Examples are illustrated in App. E. Second, we quantitatively evaluate the consistency of its assessments by comparing its scores against human judgments. Using the same 50 target triples and their supporting subgraphs, we apply randomized masking to vary inferential strength, and collect three scoring rounds from the LLM judge alongside three independent ratings from PhD students under the same prompt guidance. Fig. 7 presents a scatter plot comparing the average scores from the LLM and human annotators, revealing a strong correlation that supports the reliability and consistency of the LLM judge. Further investigating cases with significant rating discrepancies between LLMs and humans reveals two key observations. Cases where LLM ratings exceed human ratings often involve large supporting subgraphs (50-100 triples) with multiple relations between the same entities, making it challenging for humans, even with visualization, to identify all potential inference paths. Conversely, instances rated highly by humans but low by LLMs typically occur when the model overlooks certain multi-hop inference steps. These observations highlight the inherent complexity of knowledge inference tasks, even for human evaluators. Admittedly, further improving LLM-evaluator reliability remains an open avenue. More concrete examples can be seen in App. E.

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

ECE = 0.031

LLaMA3

Reliability Diagram

Over confident gap

Under confident gap

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

ECE = 0.067

Qwen2.5

Reliability Diagram

Over confident gap

Under confident gap

6.4 Ablation Studies

Size of Forget Set. In Fig. 8, we examine how varying the forget set size affects the unlearning-utility trade-off across methods. Under the LLaMA3-8B-Instruct LoRA QA setting, as the forget set size increases from 50 to 100, we observe that when utility remains relatively stable, the gap between instance unlearning and our supporting subgraph-based unlearning effectiveness remains large. However, with larger forget sets, utility begins to degrade more significantly, and the supporting subgraph structure is increasingly undermined, leading to a smaller gap between unlearning effectiveness.

0

1

2

3

4

5

Avg LLM Judge Score

0

1

2

3

4

5

Avg Human Judge Score

Human vs LLM

Pearson Coefficient: 0.826

Supporting Subgraphs: 50

Unlearning Iterations. We study the impact of unlearning epochs across different methods. Since NegGrad+ and SCRUB require retaining triples to preserve performance and thus incur double the computation, we limit them to 5 epochs, while other methods run for 10 epochs. In Fig. 9, we report results for LLaMA3-8B-Instruct (LoRA) using sentence templates, where model utility remains largely preserved. As the number of epochs increases, we observe a growing gap in unlearning effectiveness between instance unlearning and our supporting subgraph-based evaluation.

0.8

Gradient Ascent

NPO

SCRUB

0.6

UES (Inst.)

Random Label

NegGrad+

ZZZ UES (Ours)

0.4

0.2

0.0

50

100

200

1.0

CLoC

0.9

50

100

200

Forget Set Size

7 Conclusion

This work introduces a framework for evaluating knowledge unlearning in large language models, distinguished by its explicit modeling of the complex, correlated, and confidence-aware structure of factual knowledge. Our approach incorporates an inference-based evaluation protocol that utilizes powerful, meticulously guided LLM judges to assess unlearning effectiveness. Extensive experiments, involving the extraction of real-world correlated knowledge from target LLMs, reveal that current unlearning methods often significantly overestimate their actual unlearning effectiveness. This critical finding underscores the imperative to account for knowledge correlations and confidence when evaluating and developing future unlearning techniques.

Disclaimer

This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates ('JP Morgan') and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.

References

Y. Nie, Y. Kong, X. Dong, J. M. Mulvey, H. V. Poor, Q. Wen, and S. Zohren, 'A survey of large language models for financial applications: Progress, prospects and challenges,' arXiv preprint arXiv:2406.11903 , 2024.

UnlearningEffectiveness

Utility

1.00

0.20

Methods

GradientAscent

0.98

0.15

Random Label

S

E

5

0.10

NPO

NegGrad+

SCRUB

9

0.96

0.94

Inst.

0.05

Ours.

0.92

0.00

100

101

0.90

1

2

3

4

56

7

8

6

10

Epoch

Epoch

J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N. Eckardt, N. G. Laleh, C. M. L. Löffler, S.-C. Schwarzkopf, M. Unger, G. P. Veldhuizen, et al. , 'The future landscape of large language models in medicine,' Communications medicine , vol. 3, no. 1, p. 141, 2023.

H. Li, G. Deng, Y. Liu, K. Wang, Y. Li, T. Zhang, Y. Liu, G. Xu, G. Xu, and H. Wang, 'Digger: Detecting copyright content mis-usage in large language model training,' arXiv preprint arXiv:2401.00676 , 2024.

K. K. Chang, M. Cramer, S. Soni, and D. Bamman, 'Speak, memory: An archaeology of books known to chatgpt/gpt-4,' arXiv preprint arXiv:2305.00118 , 2023.

X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi, 'Quark: Controllable text generation with reinforced unlearning,' Advances in neural information processing systems , vol. 35, pp. 27591-27609, 2022.

M. Zamini, H. Reza, and M. Rabiei, 'A review of knowledge graph completion,' Information , vol. 13, no. 8, p. 396, 2022.

J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, 'Knowledge unlearning for mitigating privacy risks in language models,' arXiv preprint arXiv:2210.01504 , 2022.

S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao, C. Y. Liu, X. Xu, H. Li, et al. , 'Rethinking machine unlearning for large language models,' arXiv preprint arXiv:2402.08787 , 2024.

V. B. Kumar, R. Gangadharaiah, and D. Roth, 'Privacy adhering machine un-learning in nlp,' arXiv preprint arXiv:2212.09573 , 2022.

J. Chen and D. Yang, 'Unlearn what you want to forget: Efficient unlearning for llms,' arXiv preprint arXiv:2310.20150 , 2023.

P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter, 'Tofu: A task of fictitious unlearning for llms,' arXiv preprint arXiv:2401.06121 , 2024.

J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue, 'Machine unlearning of pre-trained large language models,' arXiv preprint arXiv:2402.15159 , 2024.

W. Shi, J. Lee, Y . Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang, 'Muse: Machine unlearning six-way evaluation for language models,' arXiv preprint arXiv:2407.06460 , 2024.

R. Wei, M. Li, M. Ghassemi, E. Kreaˇ ci´ c, Y . Li, X. Yue, B. Li, V . K. Potluru, P. Li, and E. Chien, 'Underestimated privacy risks for minority populations in large language model unlearning,' arXiv preprint arXiv:2412.08559 , 2024.

W. Ma, X. Feng, W. Zhong, L. Huang, Y. Ye, X. Feng, and B. Qin, 'Unveiling entitylevel unlearning for large language models: A comprehensive analysis,' arXiv preprint arXiv:2406.15796 , 2024.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. , 'Llama: Open and efficient foundation language models,' arXiv preprint arXiv:2302.13971 , 2023.

F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, 'Language models as knowledge bases?,' arXiv preprint arXiv:1909.01066 , 2019.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. , 'On the opportunities and risks of foundation models,' arXiv preprint arXiv:2108.07258 , 2021.

Z. Zhong, Z. Wu, C. D. Manning, C. Potts, and D. Chen, 'Mquake: Assessing knowledge editing in language models via multi-hop questions,' arXiv preprint arXiv:2305.14795 , 2023.

Y. Shi, Q. Tan, X. Wu, S. Zhong, K. Zhou, and N. Liu, 'Retrieval-enhanced knowledge editing in language models for multi-hop question answering,' in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pp. 2056-2066, 2024.

A. Blanco-Justicia, N. Jebreel, B. Manzanares-Salor, D. Sánchez, J. Domingo-Ferrer, G. Collell, and K. Eeik Tan, 'Digital forgetting in large language models: A survey of unlearning methods,' Artificial Intelligence Review , vol. 58, no. 3, p. 90, 2025.

Q. Wang, B. Han, P. Yang, J. Zhu, T. Liu, and M. Sugiyama, 'Towards effective evaluations and comparisons for llm unlearning methods,' in The Thirteenth International Conference on Learning Representations .

J. Geng, Q. Li, H. Woisetschlaeger, Z. Chen, Y. Wang, P. Nakov, H.-A. Jacobsen, and F. Karray, 'A comprehensive survey of machine unlearning techniques for large language models,' arXiv preprint arXiv:2503.01854 , 2025.

R. Eldan and M. Russinovich, 'Who's harry potter? approximate unlearning for llms,' 2023.

B. Wei, W. Shi, Y. Huang, N. A. Smith, C. Zhang, L. Zettlemoyer, K. Li, and P. Henderson, 'Evaluating copyright takedown methods for language models,' arXiv preprint arXiv:2406.18664 , 2024.

N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, et al. , 'The wmdp benchmark: Measuring and reducing malicious use with unlearning,' arXiv preprint arXiv:2403.03218 , 2024.

Z. Jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y. Chen, K. Liu, and J. Zhao, 'Rwku: Benchmarking real-world knowledge unlearning for large language models,' arXiv preprint arXiv:2406.10890 , 2024.

R. Wu, C. Yadav, R. Salakhutdinov, and K. Chaudhuri, 'Evaluating deep unlearning in large language models,' arXiv preprint arXiv:2410.15153 , 2024.

S. Wang, Y. Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li, 'Knowledge editing for large language models: A survey,' ACM Computing Surveys , vol. 57, no. 3, pp. 1-37, 2024.

K. Meng, D. Bau, A. Andonian, and Y. Belinkov, 'Locating and editing factual associations in gpt,' Advances in neural information processing systems , vol. 35, pp. 17359-17372, 2022.

K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau, 'Mass-editing memory in a transformer,' arXiv preprint arXiv:2210.07229 , 2022.

D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei, 'Knowledge neurons in pretrained transformers,' arXiv preprint arXiv:2104.08696 , 2021.

E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning, 'Fast model editing at scale,' in International Conference on Learning Representations .

N. De Cao, W. Aziz, and I. Titov, 'Editing factual knowledge in language models,' arXiv preprint arXiv:2104.08164 , 2021.

C. Zheng, L. Li, Q. Dong, Y. Fan, Z. Wu, J. Xu, and B. Chang, 'Can we edit factual knowledge by in-context learning?,' arXiv preprint arXiv:2305.12740 , 2023.

Q. Dong, D. Dai, Y. Song, J. Xu, Z. Sui, and L. Li, 'Calibrating factual knowledge in pretrained language models,' arXiv preprint arXiv:2210.03329 , 2022.

Z. Huang, Y. Shen, X. Zhang, J. Zhou, W. Rong, and Z. Xiong, 'Transformer-patcher: One mistake worth one neuron,' arXiv preprint arXiv:2301.09785 , 2023.

X. Li, S. Li, S. Song, H. Liu, B. Ji, X. Wang, J. Ma, J. Yu, X. Liu, J. Wang, et al. , 'Swea: Updating factual knowledge in large language models via subject word embedding altering,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 39, pp. 24494-24502, 2025.

J. Su, J. Healey, P. Nakov, and C. Cardie, 'Between underthinking and overthinking: An empirical study of reasoning length and correctness in llms,' arXiv preprint arXiv:2505.00127 , 2025.

J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych, 'A survey of confidence estimation and calibration in large language models,' arXiv preprint arXiv:2311.08298 , 2023.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, 'On calibration of modern neural networks,' in International conference on machine learning , pp. 1321-1330, PMLR, 2017.

T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel, 'Convolutional 2d knowledge graph embeddings,' in Proceedings of the AAAI conference on artificial intelligence , vol. 32, 2018.

F. Mahdisoltani, J. Biega, and F. M. Suchanek, 'Yago3: A knowledge base from multilingual wikipedias,' in CIDR , 2013.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. , 'Llama 2: Open foundation and fine-tuned chat models,' arXiv preprint arXiv:2307.09288 , 2023.

J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. , 'Qwen technical report,' arXiv preprint arXiv:2309.16609 , 2023.

A. Golatkar, A. Achille, and S. Soatto, 'Eternal sunshine of the spotless net: Selective forgetting in deep networks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9304-9312, 2020.

L. Graves, V. Nagisetty, and V. Ganesh, 'Amnesiac machine learning,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, pp. 11516-11524, 2021.

J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, 'Knowledge unlearning for mitigating privacy risks in language models,' in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (A. Rogers, J. Boyd-Graber, and N. Okazaki, eds.), (Toronto, Canada), pp. 14389-14408, Association for Computational Linguistics, July 2023.

J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue, 'Machine unlearning of pretrained large language models,' in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), pp. 8403-8419, Association for Computational Linguistics, Aug. 2024.

R. Zhang, L. Lin, Y. Bai, and S. Mei, 'Negative preference optimization: From catastrophic collapse to effective unlearning,' arXiv preprint arXiv:2404.05868 , 2024.

M. Kurmanji, P. Triantafillou, J. Hayes, and E. Triantafillou, 'Towards unbounded machine unlearning,' Advances in neural information processing systems , vol. 36, 2024.

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, 'Measuring massive multitask language understanding,' arXiv preprint arXiv:2009.03300 , 2020.

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. , 'Challenging big-bench tasks and whether chain-of-thought can solve them,' arXiv preprint arXiv:2210.09261 , 2022.

AI@Meta, 'Llama 3 model card,' 2024.

A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan, 'Qwen2 technical report,' arXiv preprint arXiv:2407.10671 , 2024.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. , 'Lora: Low-rank adaptation of large language models.,' ICLR , vol. 1, no. 2, p. 3, 2022.

M. Pawelczyk, J. Z. Di, Y. Lu, G. Kamath, A. Sekhari, and S. Neel, 'Machine unlearning fails to remove data poisoning attacks,' arXiv preprint arXiv:2406.17216 , 2024.

G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, 'Editing models with task arithmetic,' in The Eleventh International Conference on Learning Representations , 2023.

C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, 'Certified data removal from machine learning models,' in Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research , pp. 3832-3842, PMLR, 13-18 Jul 2020.

E. Chien, H. Wang, Z. Chen, and P. Li, 'Langevin unlearning: A new perspective of noisy gradient descent for machine unlearning,' arXiv preprint arXiv:2401.10371 , 2024.

Z. Liu, G. Dou, E. Chien, C. Zhang, Y. Tian, and Z. Zhu, 'Breaking the trilemma of privacy, utility, and efficiency via controllable machine unlearning,' in Proceedings of the ACM on Web Conference 2024 , pp. 1260-1271, 2024.

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, 'Efficient memory management for large language model serving with pagedattention,' in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023.

H. H.-H. Hsu, Y. Shen, C. Tomani, and D. Cremers, 'What makes graph neural networks miscalibrated?,' Advances in Neural Information Processing Systems , vol. 35, pp. 13775-13786, 2022.

K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning, 'Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,' arXiv preprint arXiv:2305.14975 , 2023.

J. Xie, A. S. Chen, Y . Lee, E. Mitchell, and C. Finn, 'Calibrating language models with adaptive temperature scaling,' arXiv preprint arXiv:2409.19817 , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main contribution of correlated knowledge modeling, supporting subgraph extraction, and evaluation protocol via LLM judge are presented in the Sec. 4. Experimental section validates our framework by comparing with previous instance unlearning in Sec. 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to App. I.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Dataset construction and experimental setup are described in Sec. 5 and Sec. 6.1 respectively, and further details can be found at App. B, App. D, App. C.2, and App. C.1. We also provide code in supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, the dataset construction is provided in App. 5, and we provide code with detailed README.md to setup the code environment and reproduce the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Sec. 6.1, App. B and App. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to the time complexity of our experiments, we do not report statistical error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to App. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research is conducted according to the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, please refer to App. I.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not involve the release of language models, and our datasets are directly constructed from public open-sourced datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, the models and the datasets in our experiments are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Yes, the code used in our experiments is released along with clear documentation and instructions for reproduction.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The human evaluation was carried out exclusively by the project authors, all of whom are research contributors and fully informed participants. No external crowdsourcing or human subject study was conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The study does not involve any external human subjects. All evaluations (Rating on supporting subgraphs) were conducted internally by the project authors themselves, who are informed and voluntary participants. Therefore, IRB approval was not required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

Contents

A Supporting Subgraph Extraction Algorithm

In this section, we detail the algorithm used to construct the supporting subgraph described in Sec. 4.2. As noted in Sec. 5, we constrain the subgraph to a maximum path length l = 3 . The full extraction procedure is provided in Alg. 1.

Algorithm 1 Subgraph Extraction

B Details on Unlearning Methods

In this section, we provide an overview of all unlearning methods considered in this paper along with their implementation details. These methods directly modify LLM parameters to unlearn knowledge and are representative of approaches commonly adopted in the existing literature [14, 27, 28, 57]. Recall that D forget denotes the set of target triples to be unlearned. For methods that additionally use a retain set to preserve general model functionality, we denote this set as D retain. We now briefly describe each method and its specific hyperparameters as follows:

Gradient Ascent (GA) [46-48]: GA seeks to remove the influence of D forget by reversing its gradient updates. However, this popular approach may induce significant utility degradation [14,57,58].

Random Labels (RL) [46,49]: RL disrupts memorization by replacing the labels in D forget with random tokens during next-token prediction.

Negative Preference Optimization (NPO) [50]: NPO encourages the model to assign lower likelihood to the forget set compared to its original state, while controlling deviation from the

pretrained model. It frames unlearning as a preference reversal task and adopts a log-ratio-based objective derived from direct preference optimization. The objective is defined as

where σ denotes the sigmoid function, and parameter β NPO controls the sensitivity to preference changes. A smaller β NPO leads to stronger alignment with the original model. Following previous literature [28, 50], we set β NPO = 0 . 1 .

NegGrad + [51]: NegGrad + jointly applies gradient ascent on D forget and gradient descent on D retain, optimizing

By simultaneously 'reviewing' loss L over D forget and D retain, NegGrad + mitigates the utility degradation typically caused by pure gradient ascent. In the experiments, we set β NegGrad + = 0 . 999 .

SCRUB [51]: SCalable Remembering and Unlearning unBound (SCRUB) employs a studentteacher architecture to guide model updates via the following objective:

where D KL is the divergence and parameters ( α SCRUB , β SCRUB , γ SCRUB ) control the trade-off between forgetting effectiveness and utility preservation. We set ( α SCRUB , β SCRUB , γ SCRUB ) = (0 . 999 , 1 , 0 . 99) ( unlearn with QA ) and ( α SCRUB , β SCRUB , γ SCRUB ) = (1 e -4 , 1 , 1 e -4) ( unlearn with sentence ).

Computation Budget. To ensure fair comparisons across unlearning methods, we adopt a unified computational budget protocol inspired by [14]. Methods are categorized into two groups: (i) those using only the forget set ( D forget) (e.g., GA, NPO, RL), and (ii) those requiring both forget and retain sets (e.g., NegGrad+, SCRUB). Since unlearning involves trade-offs among effectiveness, utility, and efficiency [59-61], we standardize training epochs across methods. Specifically, methods using only D forget are allowed up to 10 unlearning epochs, while those that additionally access D retain (matched in size to D forget) are limited to 5 epochs due to the increased computational cost per step. This setup ensures comparable computational complexity across methods, enabling a fair evaluation of unlearning effectiveness.

Unlearning Epoch Selection. As discussed above, all unlearning methods are allocated the same computation budgets. In general settings (Sec. 6.1), we discussed our epoch selection criterion: if the model's utility (Loc) exceeds 0.8 at any point, we select the last epoch that satisfies this threshold; otherwise, we select the epoch with the highest utility below it. The selected epoch for each method under each setting is reported in Tab.2.

Unlearning Hyperparameter Configurations. For full-parameter unlearning, we use a learning rate of 1 × 10 -6 ; for parameter-efficient unlearning with LoRA, a higher learning rate of 1 × 10 -5 is applied. In both settings, training is conducted with a batch size of 2 and gradient accumulation steps of 2 across 2 GPUs, effectively simulating a batch size of 8.

C Prompt Template

C.1 Prompt Template for LLM Judge Rating

LLMJudge Inference Prompt

You are a reasoning assistant that evaluates whether a target triple can be logically inferred from a given set of subgraph triples .

Task:

Using only the subgraph triples below, determine whether the target triple can be inferred. Do not use any external knowledge. All provided triples must be considered as factual ground truth. Only rely on logical reasoning from the facts.

Each edge is annotated with entropy ∈ [0 , 1] , which quantifies the uncertainty of that triple. Lower entropy means higher confidence in its validity.

Entropy Interpretation:

0.25-0.50 → Confident

0.00-0.25 → Very Confident

0.50-0.75 → Less Confident

0.75-1.00 → Not Confident

Scoring Rules:

Assign a confidence score from 0 to 5 for the target triple:

1 → Very weak support; entities appear but no relevant path.

0 → No logical path exists; inference is impossible from the given triples.

2 → Some weak logical connection, but with high uncertainty or missing relations.

4 → Confident support; well-formed path with generally low uncertainty.

3 → Moderate path exists with reasonable certainty.

5 → Strong support; direct match or very strong multi-hop support with high confidence.

Evaluation Guidelines

Firstly, you need to classify the direct path and the support path:

1. Direct Path : A single triple that directly connects the head entity to the tail entity with a relation that is identical or logically similar(ex. isLocatedIn and hasCapital) to the target relation. The entropy of this triple determines the confidence level of the direct path. If target triple exists in the subgraph, it must be regarded as a direct path.

2. Support Path : A multi-hop directed path that connects the head entity to the tail entity through intermediate entities. This path should logically imply the target triple through reasoning. The entropy of all triples in this path collectively determines the confidence level of the support path.

Secondly, when you are evaluating a target triple, please: 1. Identify all direct paths between the head and tail entities

Identify all support paths between the head and tail entities

Record the number of each path type

Assess the entropy (uncertainty) level of each path

Assign a score based on the criteria below

Criteria: Scores of 3+ indicate the target triple can be reasonably inferred; scores below 3 indicate insufficient evidence.

Score 5

Low entropy direct path, OR

Moderately low entropy direct path + at least one support path, OR

Higher entropy direct path + multiple support paths (more paths needed as entropy increases), OR

No direct path but multiple low entropy 2-3 hop paths

2. Score 4

Moderately low entropy direct path without support paths, OR

Relatively high entropy direct path + 1-2 support paths, OR

High entropy direct path + numerous support paths, OR

No direct path but moderately low entropy 2-3 hop paths

3. Score 3

High entropy direct path, OR

1-3 high entropy support paths

4. Score 2

No direct path

Multiple high entropy support paths, none completely correct

5. Score 1

No direct path

Very few support paths with errors, OR

Only 1-2 high entropy support paths with errors

6. Score 0

No direct path

Irrelevant or unrelated paths

No logical connection to target triple

For each evaluation, please provide:

Analyze direct paths (It is very important to confirm whether a direct path exists, because an incorrect judgment will lead to a significant difference in the assigned score).

Analyze support paths

Your reasoning for the assigned score

The final score (0-5)

Examples:

Example for Score 5:

(Rome, capital_of, Italy) with entropy 0.08

(Italy, located_in, Europe) with entropy 0.12

Target Triple:

(Rome, located_in, Europe)

Reasoning:

(Rome, capital_of, Italy) and (Italy, located_in, Europe) form a clear inference path

Both triples have very low entropy, indicating high confidence

The relation "located_in" is logically implied by the combination of "capital_of" and "located_in"

This creates a strong transitive relationship between Rome and Europe

Final Confidence Score: 5

Example for Score 4:

(Apple, produces, iPhone) with entropy 0.35

(iPhone, runs_on, iOS) with entropy 0.15

(Apple, develops, iOS) with entropy 0.20

Target Triple:

(Apple, manufactures, iPhone)

Reasoning:

(Apple, produces, iPhone) is a direct path with moderately high entropy

"produces" and "manufactures" are very similar relations

The support path (Apple, develops, iOS) and (iPhone, runs_on, iOS) indirectly reinforces the relationship

The combination of a direct path and supporting evidence compensates for the moderate entropy. Final Confidence Score: 4

Given Subgraph Triples:

{facts_str}

Target Triple:

(A, relation, B)

Your Reasoning:

Explain step-by-step how the given facts lead to (or fail to lead to) the target triple. Then output the score on a new line like:

Final Confidence Score: <integer between 0 and 5>

C.2 Prompt Template for Query Target LLM

Target LLM Query Template

System Message

You are an expert in knowledge graphs. Your task is to determine whether a given relation between two entities is correct, incorrect, or unknown.

First analyze the semantic properties of both entities, and then reason about whether the relation mentioned in the task is appropriate for these two entities.

Here is an example of a correct relation:

Example 1: For head entity 'Shakespeare', tail entity 'Hamlet', relation 'wrote', reasoning process: Shakespeare is a person, specifically an author, while Hamlet is a literary work. 'wrote' is one of the most specific relations between an author and their work, so the relation 'wrote' is correct.

Here is an example of an incorrect relation:

Example 2: For head entity 'Shakespeare', tail entity 'Hamlet', relation 'locatedIn', reasoning process: Shakespeare is a person and Hamlet is a literary work. The relation 'locatedIn' typically describes spatial or geographic relationships, which does not apply to an author and their work. Therefore, the relation 'locatedIn' is incorrect.

Here is an example of an unknown relation:

Example 3: For head entity 'Hamlet', tail entity 'Existentialism', relation 'influencedBy', reasoning process: Hamlet is a literary work, while Existentialism is a philosophical movement. Although some scholars interpret Hamlet's introspective nature as proto-existentialist, there is no widely agreed-upon or factual relationship confirming that Hamlet was directly influenced by Existentialism. Therefore, the relation 'influencedBy' is unknown.

Be deliberate and analytical in your reasoning before providing your final answer. Your answer (which is provided) should be taken as-is; the goal is to compute its log probability given the context. According to the user's task, you should provide your final answer in the format 'Answer: Yes' or 'Answer: No' or 'Answer: Unknown'.

User Templates

Qwen/Qwen2.5-7B-Instruct:

Task: In the triple ({entity1}, ?, {entity2}), does the relation

'{relation}' correctly complete it? Answer: Yes/No/Unknown

meta-llama/Llama-3.1-8B-Instruct:

Task: Given that the head entity is '{entity1}' and the tail entity is '{entity2}', is the relationship '{relation}'? Answer: Yes/No/Unknown

D Detailed Infomation of Experiments

D.1 Compute Configurations

All experiments were conducted on a hardware platform equipped with NVIDIA A100 80GB PCIe GPUs, which were used for both training and inference.

D.2 Experimental Details

For the model unlearning phase, we employed DeepSpeed ZeRO-2 optimization for efficient distributed training across multiple devices. For model evaluation, we leveraged vLLM [62] to facilitate efficient parallel inference, processing prompts with a substantial batch size of 500 to maximize throughput.

Refinement of Entropy-Based Filtering for Instruction-Following Failures. We observe that certain unlearning methods may impair the model's ability to follow instructions, i.e., reliably producing one of the expected outputs: Yes , No , or Unknown . This degradation can result in degenerate or uninformative outputs. To address this issue, we refine the entropy-based filtering criterion by requiring that the Yes token be the most probable among the top-5 predicted tokens and that the corresponding normalized entropy falls below a predefined threshold. This modification allows us to detect instruction-following failures while preserving agreement with the original entropy-based measure when the model behaves as intended.

Additionally, we emphasize that we experimented with several alternative designs for knowledge probing. First, we tested an open-ended prompting strategy in which the target LLM is given the subject and object entities from a triple and asked to select all reasonable relations between them from a provided list. However, we observed that this approach performed poorly on our constructed validation set (Sec. 5), frequently leading to hallucinated relations. Second, we evaluated the effect of introducing an explicit Unknown option when querying the LLM. We found that including this option helped mitigate hallucination and improved the model's accuracy on the validation set by encouraging more conservative predictions.

Model Calibration. In our experiments, we probe the target LLM to evaluate its own knowledge with associated confidence scores. We observe that directly asking the target LLM to generate this confidence score results in unreliable results without calibration. The confidence score should be reliable. For example, predictions made with 0.8 confidence should be correct approximately 80% of the time. Specifically,

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

ECE = 0.475

LLaMA3 - Before Calibration

Over confident gap

Under confident gap

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

ECE = 0.031

LLaMA3

After Calibration

Over confident gap

Under confident gap

Confidence

Confidence

we discover that LLaMA3-8B-Instruct inherently generates underconfident predictions. Its generated confidence scores are much lower than the corresponding binning accuracies, as shown in Fig. 10, with an ECE of 0.475. On the other hand, Qwen2.5-7B-Instruct inherently generates more reliable confidence scores, with an ECE of 0.083, leading to more trustworthy predictions. Temperature scaling has been widely used to calibrate model predictions [41, 63-65]. We show that applying temperature scaling reduces the ECE of LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct to 0.031 and 0.067 respectively.

Unlearning Supporting Subgraph Construction Stage 0.0h 0.5h 1.0h 1.5h 2.0h 2.5h Running Time (Hours) 1h 23min 1h 23min 1h 40min 1h 21min 2h 2min 1h 4min 1h 30min 1h 6min 1h 35min 56min Running Time NegGrad+ Gradient_ascent NPO RL SCRUB Figure 11: Runtime for full-model unlearning using the sentence-based format on Qwen2.57B-Instruct. Runtime and Cost Analysis. We report the runtime of both the unlearning and evaluation stages for fullmodel unlearning on Qwen2.5-7B-Instruct using the sentence-based format, including the unlearning procedure and the construction of supporting subgraphs across all methods. Recall that D forget contains 200 target triples to be unlearned; the unlearning process is conducted over 10 epochs (5 epochs for NegGrad+ and SCRUB). The subgraph construction phase also operates over the same set of target triples. On average, both stages require approximately 1 to 1.5 hours to complete, as demonstrated in Fig. 11. This corresponds to an average of roughly 30 seconds per triple for unlearning and 20 seconds for subgraph

construction. Other settings yield comparable or lower runtime overhead. For supporting subgraph evaluation with the LLM judge, recall that we use GPT-o4-mini. On average, rating each supporting subgraph takes approximately 15 seconds (In practice, the rating can be further accelerated by making parallel API calls). The total cost for a full evaluation (200 targets) is approximately USD 1.5, making the proposed evaluation framework both computationally and economically efficient.

Random Seed Selection. In all our experiments, we followed the common practice and fixed our random seed to be 42.

Note on Knowledge Probing. It is important to acknowledge that the knowledge probing process may occasionally yield incomplete or imprecise factual outputs. In our framework, we adopt multiplechoice questions instead of open-ended formats to reduce hallucination and improve consistency, and apply temperature scaling to enhance the reliability of confidence estimates. Nevertheless, probing results may still deviate from ground truth in some cases. During inference, we treat all extracted facts from the supporting subgraph as assumed ground truth knowledge in the target LLM, and consider any inference patterns identified therein as targets for unlearning.

E Additional Examples for LLM Judge

E.1 Examples of LLM Judge Adhering to Instructions

As discussed in Sec. 6.3, for all 50 tested subgraphs, the LLM judge strictly follows the instruction to reason solely based on the provided subgraph triples when assessing the inferability of the target triple. Notably, its reasoning does not incorporate any external or latent internal knowledge beyond the subgraph. Below, we present several representative examples to illustrate this behavior in Tab. 3.

Related Triple(s) in Supporting Subgraph:

-(San_Martín_Province, isLocatedIn, San_Martín)

-(San_Martín_Province, isLocatedIn, San_Martín_Region)

-(San_Martín_Region, isLocatedIn, San_Martín)

Reasoning: [...] exact triple in the subgraph: (San_Martín_Province, isLocatedIn, San_Martín) [...] there is also a 2-hop path: - (San_Martín_Province, isLocatedIn, San_Martín_Region) with entropy = 0.075 (Very Confident) - (San_Martín_Region, isLocatedIn, San_Martín) with entropy = 0.364 (Confident) This yields an additional, reasonably low-entropy route [...]

Example 2: (Voldem¯ ars_Plade, isAffiliatedTo, R¯ ıgas_FK)

Related Triple(s) in Supporting Subgraph:

-(Voldem¯ ars_Plade, isAffiliatedTo, R¯ ıgas_FK)

-(Voldem¯ ars_Plade, isKnownFor, R¯ ıgas_FK)

LLMReasoning: [...] The target triple (Voldem¯ ars_Plade, isAffiliatedTo, R¯ ıgas_FK) is directly present in the subgraph with an entropy of 0.862. [...] Other related triples include (Voldem¯ ars_Plade, isKnownFor, R¯ ıgas_FK) , which has a lower entropy of 0.385 and suggests a confident link in terms of recognition or fame [...]

Example 3: (Emad_Hamdy, wasBornIn, Sohag)

Related Triple(s) in Supporting Subgraph:

-(Emad_Hamdy, wasBornIn, Sohag_Governorate)

-(Sohag_Governorate, isLocatedIn, Sohag)

LLMReasoning: [...] The only 'wasBornIn' triple for Emad_Hamdy is

(Emad_Hamdy, wasBornIn, Sohag_Governorate) with entropy 0.849. [...] We look for a multi-hop path from Emad_Hamdy to Sohag that could imply 'wasBornIn.' The plausible 2-hop path is: a. (Emad_Hamdy, wasBornIn, Sohag_Governorate) entropy = 0.849; b.

(Sohag_Governorate, isLocatedIn, Sohag) entropy = 0.044. Reasoning for path: If Emad_Hamdy was born in Sohag_Governorate, and Sohag_Governorate is located in the city Sohag, then it follows that Emad_Hamdy was born in Sohag. [...]

Example 4: (Ptolemy_VIII_Physcon, isMarriedTo, Cleopatra_II_of_Egypt)

Related Triple(s) in Supporting Subgraph:

-(Ptolemy_VIII_Physcon, isMarriedTo, Cleopatra_II_of_Egypt)

LLM Reasoning: [...] We need to find a direct match or a logical path that supports this relationship using the subgraph triples provided. The subgraph contains the triple

(Ptolemy_VIII_Physcon, isMarriedTo, Cleopatra_II_of_Egypt) with an entropy of 0.181. This directly matches the target triple. [...]

Table 3: Examples of LLM reasoning grounded in the extracted supporting subgraph. Blue indicates the utilized knowledge triple from the subgraph. We verified no external knowledge is used.

E.2 Examples Comparing LLM and Human Judgments

As discussed in Sec. 6.3, our comparison between LLM and human ratings reveals a strong overall correlation, with the majority of samples receiving similar scores. Nonetheless, there are some discrepancies in certain cases. Specifically, human ratings tend to be higher when the LLM overlooks multi-hop inference pathways that human evaluators are able to identify. Conversely, in instances where the LLM assigns higher scores, the supporting subgraph often contains multiple relation types, making it more likely for human annotators to overlook critical patterns relevant to the inference. To further illustrate these observations, we present representative examples in Tab. 4. Moreover, based on the LLM's reasoning over these examples, we also observe that it does follow the prompt instructions on carefully considering both direct and multi-hop inference patterns, while taking the associated entropy scores into account.

Example 1: (Benjamin_Butler_(politician), isAffiliatedTo, Greenback_Party)

LLM: 1. Direct Path Analysis - We look for a triple linking Benjamin_Butler_(politician) directly to Greenback_Party with relation 'isAffiliatedTo.' There is no exact 'isAffiliatedTo' triple. However, we do have: (Benjamin_Butler_(politician), participatedIn, Greenback_Party) with entropy 0.145. 'participatedIn' is not identical to 'isAffiliatedTo,' but in the context of political parties it is a very close, logically similar relation: participating in a party generally implies affiliation. Entropy 0.145 falls in the 'very confident' range (0.00-0.25). 2. Support Path Analysis - A possible multi-hop path: (Benjamin_Butler_(politician), isPoliticianOf, Portsmouth_NH) [0.311] → (Portsmouth_NH, created, Grassroots_Campaigns_Inc.) [0.644] → (Grassroots_Campaigns_Inc., isAffiliatedTo, Greenback_Party) [0.632]. This path is weak: it infers Butler's personal affiliation from the municipality's creation of an organization that itself is affiliated, which is a tenuous link. Entropies on that path are moderate to high, further reducing confidence. 3. Reasoning for Score - We have one very low-entropy direct path (0.145) via 'participatedIn,' a relation closely aligned with 'isAffiliatedTo.' No equally strong support paths are needed when a direct path is already very confident. According to the scoring rubric, a 'low entropy direct path' alone is sufficient for a score of 5. Final Confidence Score: 5

Human: In the supporting subgraph, there exists a direct correlated target triple that is closely aligned with and supports the target fact. Furthermore, there are also indirect paths that can partially support the inference. Therefore, the final confidence score is 5.

Example 2: (Mooresville_North_Carolina, isLocatedIn,

Iredell_County_North_Carolina) -LLM fails to capture certain inferential pathways.

LLM: 1. Analysis of Direct Paths - We look for any triple of the form

(Mooresville_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) in the subgraph. No such direct triple exists. 2. Analysis of Support Paths - We look for multi-hop paths that would imply Mooresville is located in Iredell County. The only immediate location fact about Mooresville is: (Mooresville_North_Carolina, isLocatedIn, Mecklenburg_County_North_Carolina) -entropy 0.465 (Less Confident). To bridge from Mecklenburg County to Iredell County, one might try: (Mecklenburg_County_North_Carolina, hasCapital, Concord_North_Carolina) -entropy 0.268 (Very Confident), and (Concord_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) -entropy 0.205 (Very Confident). This yields the chain: Mooresville -isLocatedIn → Mecklenburg County -hasCapital → Concord -isLocatedIn → Iredell County. However, hasCapital does not imply that every part of Mecklenburg County is in Iredell County, nor that Mecklenburg County itself is located in Iredell County. It simply states that Concord is the capital of Mecklenburg County and happens to be located in Iredell County. This is a semantic mismatch and does not logically support that Mooresville, which is in Mecklenburg County, lies in Iredell County. No other viable paths exist (all other isLocatedIn links from Mecklenburg County point to other cities, or the directionality is reversed). 3. Reasoning for Score - There is no direct (Mooresville_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) triple. Only one multi-hop chain exists, but it misuses hasCapital and therefore cannot validly establish the target relation. Although entropies along that chain are low, the relation semantics are incorrect for location inference. This constitutes very weak, semantically flawed support. Final Confidence Score: 1 Human: The subgraph contains several indirect paths (each requiring three hops to infer the target triple), and all paths exhibit medium-to-low entropy, indicating that the target LLM has only moderate confidence in this knowledge. For example, one representative reasoning chain is: (Mooresville_North_Carolina, isLocatedIn, Mecklenburg_County_North_Carolina) (entropy = 0.465) → (Mecklenburg_County_North_Carolina, isLocatedIn, Pineville_North_Carolina) (entropy = 0.358) → (Pineville_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) (entropy = 0.106), which forms a plausible 3-hop inference path supporting the target triple. Therefore, the final confidence score is 4.

Example 3: (Gmina_Cybinka, isLocatedIn, Słubice_County) - Human missed certain supportive triples

LLM: 1. Direct Path Analysis - There is an exact matching triple: (Gmina_Cybinka, isLocatedIn, Słubice_County) with entropy 0.011. This directly corresponds to the target relation 'isLocatedIn' and has very low entropy (0.00-0.25), indicating a highly confident assertion. 2. Support Path Analysis - Not required given the strength of the direct match. (For completeness: there exist location or adjacency triples involving Gmina_Cybinka and Słubice_County , but they do not add meaningful certainty beyond the direct fact.) 3. Reasoning - A direct, identical relation exists in the subgraph with extremely low entropy. According to the scoring rubric, such highly confident direct evidence warrants the highest possible score. Final Confidence Score: 5 Human: Since no direct or multi-hop relational evidence supporting the inference is found within the supporting subgraph, the final confidence score is given 0. [Reason: The primary reason for the low human-assigned scores lies in the substantial number of triples present in the subgraph (47 triples in this case), many of which involve diverse relation types (e.g., isLocatedIn, isAffiliatedTo, hasNeighbor, hasCapital, isConnectedTo, owns) without forming clear or effective supporting inference patterns. This information overload substantially increases the cognitive complexity of the evaluation task, making human annotators more susceptible to being misled or overlooking crucial triples, ultimately resulting in incorrect or inconsistent judgments. ]

Table 4: Examples comparing LLM and human ratings. Blue highlights content most relevant to the inference judgment. Green indicates agreement between LLM and human (i.e., both deem the target triple inferable), while Red denotes disagreement (i.e., the LLM and human assign different ratings regarding inferability).

E.3 Case Studies of Supporting Subgraph Inferability

Certain triples appear successfully unlearned at the instance level, yet remain inferable when considering their supporting subgraphs. Several representative examples in Tab. 5 highlight this phenomenon.

Example 1: (Alexander_Morten, playsFor, Wanderers_F.C.)

The triple (Alexander_Morten, playsFor, Wanderers_F.C.) (entropy: 0.437 before, 1.328 after) seems unlearned individually, but the evaluator LLM can still infer it through the related fact (Alexander_Morten, workat, Wanderers_F.C.) (entropy: 0.512 before, 0.644 after) present in the supporting subgraph, as the relationships playsFor and workat are semantically similar in the context of professional athletes.

Example 2: (Robyn_Miller, workat, Cyan_Worlds)

The triple (Robyn_Miller, workat, Cyan_Worlds) (entropy: 0.386 before, 1.201 after) demonstrates successful instance-level unlearning, yet remains inferable through the related fact (Robyn_Miller, edited, Cyan_Worlds) (entropy: 0.582 before, 0.626 after) in the supporting subgraph, as editorial contributions strongly suggest a working relationship with the company.

Example 3: (Oxford_Properties, owns, Brookfield_Place_(Toronto))

The triple (Oxford_Properties, owns, Brookfield_Place_(Toronto)) (entropy: 0.411 before, 1.205 after) appears unlearned when assessed individually, but can be inferred through a chain of ownership relationships in the supporting subgraph: (Oxford_Properties, owns, Brookfield_Office_Properties) (entropy: 0.349 before, 0.726 after) and

(Brookfield_Office_Properties, owns, Brookfield_Place_(Toronto)) (entropy: 0.132 before, 0.361 after), establishing a corporate ownership hierarchy that reveals the target relationship.

Example 4: (Bridgewater_Township_New_Jersey, isLocatedIn, Somerset_County_New_Jersey)

The triple (Bridgewater_Township_New_Jersey, isLocatedIn,

Somerset_County_New_Jersey) (entropy: 0.591 before, 1.477 after) shows instance-level unlearning success, but remains inferable through geographic proximity facts in the supporting subgraph: (Bridgewater_Township_New_Jersey, hasNeighbor,

Bernardsville_New_Jersey) (entropy: 0.202 before, 0.389 after) and

(Bernardsville_New_Jersey, isLocatedIn, Somerset_County_New_Jersey) (entropy: 0.455 before, 0.757 after), which together likely imply that Bridgewater Township is also located in Somerset County.

These examples illustrate a critical challenge in knowledge unlearning: even when direct knowledge of a specific triple is removed, the model may retain inferential paths through related information that remains in its knowledge base.

F Additional Experimental Results

F.1 In-Depth Analysis of Local Consistency

Recall the definition of the utility metric Local Consistency (Loc) (Sec. 6.1), which quantifies the consistency of an LLM's predictions on multiple-choice questions ( Yes , No , and Unknown ) regarding neighboring knowledge triples before and after unlearning. In this section, we conduct a more finegrained analysis of how model predictions shift under the Loc metric, capturing any transitions the LLM may exhibit among the three predefined choices. Additionally, we observe that the unlearning process can sometimes impair the model's ability to follow instructions, particularly under the QA-based format. To account for such cases, we introduce an additional Other category to denote predictions that fall outside the standard multiple-choice options. In Fig. 12 13 14 15 16 17 18 19, we present the corresponding confusion matrices under various settings to illustrate these dynamics. Our findings reveal several key observations. First, in the majority of cases, the LLM retains its utility on local knowledge triples. Second, we find that each unlearning method introduces changes in a distinct manner, i.e., transitions may occur from one valid category (e.g., Yes ) to another (e.g., Unknown ), reflecting shifts in the model's belief. The patterns of these shifts can be different across different methods. Finally, in scenarios where the original utility metric Loc score is low (e.g., below 0.5 in Tab. 1), we observe that the model occasionally fails to adhere to the multiple-choice instruction

format altogether. This result in predictions that fall into the Other category, especially for QA-based unlearning.

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.64

0.12

0.00

0.24

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

96.78

3.22

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

93.56

5.96

0.00

0.48

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.52

0.00

0.00

0.48

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

87.85

10.36

1.33

0.46

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

SCRUB

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

26.24

0.12

0.00

73.64

0.00

50.00

0.00

50.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

10

20

30

40

50

60

70

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.28

0.00

0.00

0.72

50.00

50.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

96.19

0.00

0.00

3.81

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

17.52

0.00

0.00

82.48

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

95.71

4.29

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

SCRUB

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

95.11

4.89

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.40

0.60

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

95.64

4.36

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

97.74

2.26

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

95.71

4.29

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

SCRUB

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

96.07

0.83

0.00

3.10

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.76

0.24

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

98.93

1.07

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.76

0.24

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.76

0.24

0.00

0.00

0.00

100.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

SCRUB

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

86.76

8.35

4.89

0.00

0.14

99.86

0.00

0.00

0.00

3.68

96.32

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.90

0.10

0.00

0.00

5.06

94.94

0.00

0.00

10.43

2.45

87.12

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

100.00

0.00

0.00

0.00

8.82

91.18

0.00

0.00

21.47

3.68

74.85

0.00

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

97.86

0.10

2.04

0.00

9.54

86.99

3.47

0.00

0.92

0.00

99.08

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

60.18

36.25

3.56

0.00

0.43

99.57

0.00

0.00

9.20

43.87

46.93

0.00

0.00

0.00

0.00

0.00

SCRUB

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

48.88

31.57

19.55

0.00

0.00

100.00

0.00

0.00

0.00

9.82

90.18

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

100.00

0.00

0.00

0.00

19.22

80.20

0.14

0.43

29.45

1.23

69.33

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

98.37

0.00

0.20

1.43

13.01

34.10

1.16

51.73

18.71

0.00

50.61

30.67

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

82.18

0.00

17.82

0.00

19.80

5.92

74.28

0.00

2.15

0.00

97.85

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

52.04

46.84

1.12

0.00

0.58

99.42

0.00

0.00

29.75

65.64

4.60

0.00

0.00

0.00

0.00

0.00

SCRUB

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

72.20

6.21

21.59

0.00

0.58

89.45

9.97

0.00

0.00

0.31

99.69

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

89.00

6.01

4.99

0.00

0.87

97.11

2.02

0.00

0.31

0.92

98.77

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

90.53

6.82

2.65

0.00

0.29

99.28

0.43

0.00

3.07

1.84

95.09

0.00

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

99.69

0.31

0.00

0.00

4.48

95.38

0.14

0.00

7.67

1.84

90.49

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

92.46

0.00

0.00

7.54

2.17

64.88

0.00

32.95

16.87

8.28

41.10

33.74

0.00

0.00

0.00

0.00

SCRUB

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

100.00

0.00

0.00

0.00

4.62

95.23

0.14

0.00

9.20

3.07

87.73

0.00

0.00

0.00

0.00

0.00

Gradient Ascent

0

20

40

60

80

100

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

97.66

0.81

1.53

0.00

1.88

96.68

1.45

0.00

1.53

0.00

98.47

0.00

0.00

0.00

0.00

0.00

Random Label

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

90.94

8.76

0.31

0.00

0.29

99.71

0.00

0.00

13.19

31.60

55.21

0.00

0.00

0.00

0.00

0.00

NPO

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

82.28

16.29

1.43

0.00

2.31

97.69

0.00

0.00

23.31

11.66

65.03

0.00

0.00

0.00

0.00

0.00

NegGrad+

0

20

40

60

80

Percentage (%)

Yes

No

Unknown

Other

Post-Unlearning

Yes

No

Unknown

Other

Pre-Unlearning

89.00

0.31

0.31

10.39

2.46

56.50

0.00

41.04

19.33

1.23

65.03

14.42

0.00

0.00

0.00

0.00

SCRUB

0

10

20

30

40

50

60

70

80

Percentage (%)

F.2 Evaluating Shifts in LLM-Inferred Scores Pre- and Post-Unlearning

In this section, we further analyze the distribution changes of discrete LLM ratings (ranging from 0 to 5) under our supporting subgraph-based unlearning framework across different settings, before and after each unlearning method is applied. As shown in Fig. 20 21 22 23 24 25 26 27, we observe that the QA-based format, which directly targets the question-answer pairs used for knowledge probing, yields significantly higher unlearning effectiveness compared to the sentence-based format (albeit at the cost of reduced utility, as demonstrated in Tab. 1).

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

SCRUB

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

SCRUB

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

SCRUB

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.2

0.4

0.6

0.8

1.0

Density

SCRUB

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Density

SCRUB

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

SCRUB

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

SCRUB

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

Gradient Ascent

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

Random Label

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NPO

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

NegGrad+

Pre-Unlearning

Post-Unlearning

0

1

2

3

4

5

Score

0.0

0.1

0.2

0.3

0.4

0.5

Density

SCRUB

Pre-Unlearning

Post-Unlearning

F.3 Additional Results on the Impact of Confidence Scores on Unlearning Effectiveness

In this section, we present additional results on how confidence scores within the supporting subgraph affect unlearning outcomes across different settings. As shown in Fig. 28 29, gradually decreasing the entropy threshold u ∗ , i.e., retaining only higher-confidence knowledge, systematically filters out weaker supporting inferences. This leads to a significant increase in unlearning effectiveness. These results highlight that low-confidence knowledge triples play a crucial role in evaluating unlearning, and omitting them can substantially overestimate its effectiveness.

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(a) QA-LoRA

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(b) QA-Full

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(c) Sentence-LoRA

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(d) Sentence-Full

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(a) QA-LoRA

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(b) QA-Full

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(c) Sentence-LoRA

0.3

0.5

0.8

1.0

Entropy Threshold

0.0

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

(d) Sentence-Full

F.4 Additional Results on Performance Across Unlearning Epochs

In Fig. 30, we present additional results analyzing the impact of unlearning epochs under the LLaMA3 full-model unlearning setting, comparing both sentence-based and QA-based formats. Across all configurations, the unlearning effectiveness measured by our supporting subgraph evaluation remains consistently lower than that of traditional instance-level evaluations. Notably, in the QA-based format, unlearning directly targets the question-answer pairs used for probing factual knowledge. As unlearning progresses, this often leads to severe degradation of model utility, resulting in unlearning effectiveness approaching 1 , indicating that both the target triples and their associated subgraphs have been extensively corrupted. In contrast, under the sentence-based format (Left), the unlearning process tends to be more conservative in its impact on model utility in most cases, causing less degradation compared to the QA-based setting (Right). Consequently, we observe that unlearning effectiveness, particularly under our evaluation framework, remains low across most epochs. These findings highlight the limitations of current unlearning methods and underscore the need for more robust approaches capable of effectively removing knowledge without compromising general model performance.

10 0

10 1

Epoch

10

2

0

10

2

10

1

10 0

UES

Unlearning Effectiveness

Methods

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

Inst.

Ours.

10 0

10 1

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Utility

10 0

10 1

Epoch

0.2

0.4

0.6

0.8

1.0

UES

Unlearning Effectiveness

Methods

Gradient Ascent

Random Label

NPO

NegGrad+

SCRUB

Inst.

Ours.

10 0

10 1

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Utility

G Mapping Between Entropy Threshold u ∗ and Yes Token Probability

In this section, we clarify how the entropy-based filtering criterion used for constructing supporting subgraphs relates to the model's predicted probability for the Yes token in a multiple-choice question format ( Yes , No , Unknown ). Specifically, we consider a prediction to reflect retained knowledge if the model's output satisfies two conditions: (i) the Yes token is the argmax among the three options, and (ii) the entropy computed over the distribution of these three options is below a threshold u ∗ (assuming the model adheres to instructions). To make this relationship more interpretable, Tab. 6 presents the corresponding range of Yes token probabilities for different values of entropy under the argmax constraint. This mapping provides an intuitive understanding of how the entropy threshold u ∗ translates to the model's confidence in answering Yes . To derive these intervals, we fix the entropy value u ∗ and solve for the range of valid Yes probabilities p under the constraint that Yes is the argmax. The upper bound occurs when the remaining two options ( No and Unknown ) share equal probability, i.e., u ∗ = H ( p, 1 -p 2 , 1 -p 2 ) with p ≥ 1 -p 2 , where H denote entropy. The lower bound corresponds to the case where one of the nonYes options takes probability 1 -p and the other is zero, i.e., u ∗ = H ( p, 1 -p, 0) with p ≥ 1 -p .

H Additional Details and Illustrations for YAGO3-10

Complete List of Relations in YAGO3-10: actedIn, wasBornIn, hasGender, hasAcademicAdvisor, hasChild, hasCitizenship, hasDeathPlace, hasEmployer, hasGivenName, hasInstrument, hasLanguage, hasLegalResidence, hasMember, hasName, hasNationality, hasOccupation, hasOfficialLanguage, hasPlaceOfBirth, hasPlaceOfDeath, hasSpouse, hasSurname, hasTitle, holdsPoliticalPosition, isAffiliatedTo, isConnectedTo, isKnownFor, isLocatedIn, isMarriedTo, isPoliticianOf, livesIn, playsFor, produced, studiedAt, wasBornOnDate, wasCreatedOnDate, wasDestroyedOnDate, worksAt.

Using the complete set of relations from YAGO3-10, we present illustrative examples of potential (non-)deterministic inference patterns in the knowledge graph, as summarized in Tab. 7.

Explanation: This multi-hop rule captures the linguistic environment of a worker. Individuals employed in a region are likely to acquire or use the region's official language, particularly in professional or social interactions. The inference strength varies by linguistic diversity and integration policies.

Rule: ( a, hasDeathPlace , b ) ⇒ ( a, hasPlaceOfDeath , b )

Explanation: Another deterministic alias. Both relations refer to the geographical location where an individual passed away. They are synonymous in the context of biographical datasets and can be used interchangeably in logical reasoning.

I Limitations and Broader impacts

In this paper, we have proposed a novel knowledge unlearning evaluation framework designed to capture complex factual dependencies and the inherent uncertainty reflected by LLM confidence. Our aim is to enable more comprehensive and realistic assessments of knowledge unlearning efficacy. Nonetheless, several important limitations remain. First, the inference capabilities of both LLM-based and human evaluators have inherent limitations. Both types of judges may overlook subtle or complex inference patterns, and real-world adversaries could potentially surpass these evaluators in inference power. In particular, while we observed that GPT-o4-mini provides stable and interpretable judgments in our experiments, we acknowledge that LLM judges can still exhibit occasional inconsistencies and prompt sensitivity. Future work may explore using multiple diverse LLM judges or distilling lightweight judge models to improve robustness and reduce evaluation cost. Second, the process of constructing supporting subgraphs may be incomplete. Our subgraph extraction approach might not encompass all relevant facts necessary for accurate inference, and the reference knowledge graphs used in our evaluation may themselves lack certain entities or relationships that are implicitly encoded within the target LLM. Furthermore, our knowledge probing strategy for supporting subgraph construction might not be optimal, and we acknowledge that certain knowledge extracted can be inaccurate. Third, our current evaluation framework primarily focuses on relational factual knowledge and relies on externally constructed reference knowledge graphs to guide the subgraph extraction process. Given that many pretrained LLMs utilize undisclosed or proprietary pretraining corpora, discrepancies could arise between these reference graphs and the model's actual internal knowledge representations, further limiting evaluation accuracy. Finally, while our framework enables scalable LLM-based evaluation, it currently depends on commercial API calls. Although we have kept the total cost low (approximately USD 1.5 for evaluating 200 unlearning targets using GPT-o4-mini), such dependency introduces potential variability across judge models and access constraints in future deployments. Beyond methodological limitations, we also emphasize the ethical implications associated with knowledge unlearning evaluations. Specifically, we advocate for the responsible use of compliant, publicly accessible, or properly authorized data when conducting such studies. By acknowledging these limitations and ethical considerations, we hope our work fosters more rigorous and ethically responsible practices in the research, development, and deployment of machine learning technologies.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a confidence-aware evaluation protocol for unlearning facts in LLM to address limitations in current assessment methodologies. Existing unlearning metrics employ simplistic verbatim matching to determine whether facts to be forgotten remain accessible, failing to detect inferential reconstruction through correlated knowledge that persists after the unlearning process. The authors propose an approach that rests on a knowledge graph (KG) as a supporting source of background knowledge. In practice, given a certain fact to forget (that has to be present in the underlying KG), the approach works in two stages: (1) extraction of knowledge subgraphs from, and (2) assessment by calibrated GPT-4-level judges to determine whether target information remains inferentially derivable. The approach formalizes model knowledge as G = (E, R, T_U) where triples t = (s, r, o, u) incorporate entropy-calibrated confidence scores u to quantify belief strength in retained facts. Then, the key contribution lies in evaluating inferential leakage rather than direct recall, providing a more robust assessment of whether sensitive information has been effectively eliminated from the model's accessible knowledge base. This methodology aims to address a limitation in current unlearning evaluation, where models may successfully suppress direct responses while retaining the ability to reconstruct identical information through multi-hop reasoning over remaining knowledge. The process involves several calibration phases (LLM temperature to interpret output scores), also requiring the involvement of 3 PhD subjects. The authors present results on fact from YAGO (including 37 predicates) and test the framework capabilities on two LLMs (LLaMA, Qwen-2.5)\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nS1: The paper presents an overall nice idea. Linking knowledge graphs (maintaining organized facts) with LLMs (where facts are embedded in text) represents a key step toward mitigating LLMs' hallucinations. In this particular case, using the KG as a source of background knowledge to find subgraphs centered on a fact to be forgotten is interesting.\n\nS2: The paper is well written, although the organization is a bit confusing had to go back and forth between the paper and the Appendix file (since the pointers weren't there). I realize that deadline flexibility between the paper and the supplemental material may have been responsible for this \"separation\".\n\nWeaknesses\nW1: I immediately thought of the connection between forgetting facts and LLM guardrails (to prevent the LLM from releasing harmful information).  The authors fundamentally missed that their knowledge unlearning framework directly addresses the core vulnerability of LLM guardrail systems—inference through correlated knowledge pathways. Their methodology for extracting supporting subgraphs and evaluating residual inferability through adversarial probing is **structurally identical** to how guardrail circumvention attacks operate: both exploit semantic relationships to reconstruct restricted information from seemingly benign facts. For instance, while a guardrail might block direct queries about explosive manufacturing, an adversary could still infer bomb-making instructions by querying related chemistry facts, mining practices, and fertilizer compositions—exactly the type of multi-hop inference their framework is designed to detect and prevent. The authors should have benchmarked their approach against existing guardrail systems using identical restricted knowledge domains, comparing their  UES against guardrail robustness metrics when facing inference-based attacks. Their finding that current evaluation methods \"overestimate effectiveness by 20-30%\" likely applies equally to guardrail evaluations, yet they provide no empirical validation of this critical insight. This represents a significant methodological gap: without demonstrating that their adversarial inference protocol captures attack vectors that traditional guardrail evaluations miss, they cannot substantiate their claims about providing \"more realistic and rigorous assessment\" of safety mechanisms.\n\n\nW2: The proposed framework suffers from severe methodological brittleness due to its dependence on multiple sequentially coupled components, each introducing distinct failure modes that compound systematically. \nW2.1: The 3-hop subgraph extraction limitation represents an arbitrary constraint that fundamentally undermines the approach's completeness—adversarial inference attacks can routinely exploit longer reasoning chains, and the authors provide no principled justification for this bound beyond computational traceability (and reference to the state of the art adopting similar thresholds). The reliance on external reference knowledge graphs (YAGO3-10) for candidate generation creates a critical blind spot: any knowledge relationships not represented in the reference KG will be invisible to the extraction process, potentially missing entire classes of inferential pathways that exist within the target LLM's latent representations.\nW2.2: The entropy-based confidence calibration introduces multiple points of failure: temperature scaling assumes consistent calibration across diverse knowledge domains and model states, yet the authors demonstrate this only on a limited validation set. The fixed entropy threshold (u* = 1) appears arbitrarily chosen and lacks sensitivity analysis—small perturbations in this threshold could dramatically alter subgraph composition and subsequent evaluations. \nW2.3: The LLM judge component represents perhaps the most problematic element: even GPT-4-level models exhibit inconsistent reasoning across complex multi-hop scenarios, and the authors' correlation with human judgment (r=0.826) still reflects 32% unexplained variance. The judge's prompt engineering requirements—balancing specificity with generalizability—create a fragile dependency on exact phrasing and instruction format. Most critically, the framework's cascading nature means that errors compound multiplicatively: subgraph extraction errors propagate through confidence assessment, which then distorts judge evaluation, ultimately producing unreliable effectiveness scores whose relationship to actual unlearning robustness remains empirically invalidated.\n\n\nW3: The framework's reliance on proprietary LLM judges (GPT-4-mini) introduces fundamental evaluation biases and economic barriers that severely limit practical deployment. Using one commercial LLM to assess another creates systematic blind spots—both models likely share similar training paradigms, architectural biases, and knowledge representation patterns that could lead to convergent evaluation errors. The judge may systematically under-detect inference pathways that exploit reasoning patterns foreign to its own training regime, or conversely, over-penalize inference chains that its particular architecture finds difficult to process. This circular dependency means the evaluation framework inherits the safety and robustness limitations **of the judge model itself**, potentially creating false confidence in unlearning effectiveness when both models share identical vulnerabilities to specific attack vectors.\nW3.1: Computational economics presents an even more critical barrier to adoption. The authors' methodology requires hundreds of individual queries per target fact: candidate triple enumeration across 3-hop neighborhoods, confidence assessment for each extracted triple, and iterative judge evaluation across multiple reasoning chains. For their modest experimental setup (200 target triples), this translates to tens of thousands of API calls to proprietary models, yet the paper provides no cost analysis or runtime metrics. Assumin GPT-4 pricing to be (~$0.01-0.03 per 1K tokens), evaluating large unlearning scenarios could cost hundreds of dollars per assessment, making the approach economically prohibitive for iterative development, hyperparameter tuning, or large-scale knowledge domain evaluation. Without open-source judge alternatives or significant computational optimizations, their framework remains confined to academic demonstrations rather than real-world safety applications\n\nMinor point\nThe framework lacks ground truth establishment, conflating LLM judge inference capability with actual knowledge retention in target models. This creates arbitrary evaluation ceilings bounded by judge model limitations rather than measuring genuine knowledge erasure. The circular validation—using their own subgraph methodology to evaluate their own effectiveness claims—provides no external baseline.\n\n\n\n--\n\nQUESTIONS:\nQ1: How does UES performance degrade when target facts or neighbors are missing from YAGO3-10? Have you measured recall on domain-specific knowledge (recent scientific discoveries, proprietary information) or long-tail facts with rare entities? Your reliance on external reference KGs creates systematic blind spots for implicit correlations learned from unstructured text that wouldn't appear in Wikipedia-derived triple stores. This could critically underestimate unlearning effectiveness for sensitive knowledge most relevant to safety applications.\n\nQ2: How much do inferability scores drift when swapping GPT-4-mini, say for Claude, Gemini, or open-source alternatives across identical subgraphs? Your framework's validity depends entirely on judge consistency, yet you provide no systematic robustness testing across multiple architectures—a critical gap undermining your comparative effectiveness claims.\n\nQ3: Why hand-tune u* = 1 instead of using ROC optimization against validation sets or adaptive thresholding by knowledge domain? Optimal thresholds likely vary across relation types and model architectures. Your fixed threshold ignores that confidence calibration quality differs dramatically across models and fine-tuning procedures.\n\nQ4: For millions of triples, what are concrete GPU hours and API costs per target fact? Can lightweight approximations (learned subgraph pruning, distilled judges, hierarchical evaluation) preserve diagnostic power while reducing computational burden by orders of magnitude? \n\nThanks\n\nLIMITATIONS:\nThe authors have discussed limitations. Although some aspects have not been covered. The authors mention subgraph construction accuracy but do not mention the cost (in terms of LLM calls, for instance). Moreover, a broader discussion about why focusing on a closed model (GPT) is also needed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper proposes a new framework for evaluating knowledge unlearning in large language models. It captures the implicit knowledge by representing factual contexts as KGs, and incorporates an inference-based evaluation protocol that leverages LLMs as judges to assess unlearning effectiveness. Extensive experiments reveal that existing evaluation methods often significantly overestimate the actual unlearning effectiveness, while the proposed framework can provide a more realistic and rigorous assessment.

STRENGTHS AND WEAKNESSES:
### Strengths
The paper is easy to follow, with clear writing and well-presented motivation and technical details.

The authors propose new benchmark and framework that consider implicit relationships between facts, which can evaluate the unlearning effectiveness in a fairer and more reasonable fashion.

The LLM judges are also proven to be closely aligned with human experts.

### Weaknesses

It may be better to discuss relevant knowledge editing methods, which also include the unlearning task.

The authors evaluate only 8B and 7B LLMs. Is it possible to conduct experiments with larger LLMs? Those LLMs may store more complicated and implicitly correlated information.

QUESTIONS:
Please see Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel evaluation framework for knowledge unlearning that goes beyond assessing the forgetting of isolated facts. It addresses the challenge of implicitly retained knowledge, where facts presumed forgotten may persist due to correlated or inferable information. To capture this, the authors represent factual contexts as knowledge graphs with confidence scores and define unlearning evaluation in terms of knowledge subgraph reasoning. They extract such subgraphs from LLMs (LLaMA-8B-Instruct and Qwen-7B-Instruct) and use existing knowledge graphs along with LLMs as judges to reason over these subgraphs and assess unlearning success. Through experiments with popular unlearning methods, they show that current approaches often overestimate how much knowledge has actually been forgotten when real-world correlations are considered.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The idea of evaluating implicit retained knowledge via subgraph reasoning is novel and relevant, especially for domains where relational context matters.\n- The finding that existing unlearning methods overestimate their effectiveness is important and may inspire better techniques in the future.\n- The motivation to look beyond surface-level fact forgetting is well-justified.\n\n**Weaknesses and Suggestions**:\n- Practicality of Knowledge Graphs: The proposed method relies on structured external knowledge graphs (e.g., Wikidata) to model factual context. In many real-world unlearning use cases (e.g., copyrighted or proprietary material), such structured knowledge may not be available. How do the authors envision applying their framework in those cases?\n- LLM Judge Reliability: Relying on LLMs as judges for subgraph reasoning introduces potential noise and hallucinations. How did the authors verify the accuracy or consistency of the LLM judgments, especially for multi-hop reasoning?\n- Multiple-Choice Evaluation: If multiple-choice questions were used in evaluations, did the authors take steps to mitigate answer biases (e.g., positional bias or surface-level cues)?\n- Choice of Judge Model: The use of GPT-4o-mini as the LLM judge raises questions. Was this choice based on a tradeoff between performance and cost? Would a stronger model (e.g., GPT-4-turbo) change the conclusions?\n\nQUESTIONS:\nLine 403 mentions that the evaluation might overlook certain multi-hop relations. Is this a common limitation in subgraph-based reasoning, and how impactful is it on the overall assessment?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework to evaluate knowledge unlearning in LLMs, highlighting the impact of correlated and confidence-aware knowledge. Experiments show current methods often overestimate effectiveness, stressing the need to account for these factors in future research.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1.This paper is clearly written. Make it easy for readers to understand.\n\n2.This paper proposes a realistic formulation of knowledge unlearning.\n\n3.This paper proposes employing a powerful LLM judge to effectively automate unlearning evaluation.\n\nWeaknesses:\n\n1.In Section 6.3, regarding the correlation evaluation, the evaluation scale is relatively small (only 50 samples). It is necessary to report the statistical properties of larger-scale data and the correlation among multiple human annotators.\n\n2.The text relies on YAGO3-10 to construct reasoning paths for evaluation. However, can the Supporting Subgraphs based on YAGO3-10 sufficiently cover the internal knowledge structure of LLMs? How does the coverage of Supporting Subgraphs quantify the extent to which they encompass the internal knowledge of LLMs, in order to thoroughly evaluate the forgetting of LLMs' internal knowledge?\n\n3.The related work section discusses multi-hop knowledge editing and evaluation, but lacks more in-depth quantitative comparisons and discussions. If a table could be included to compare related work, it would further highlight the contributions of this paper.\n\nQUESTIONS:\n1.Is it feasible to apply the idea of confidence aware knowledge graph to other tasks such as knowledge editing?\n\n2.Table 1 indicates that the template format (sentence vs. QA) significantly affects the results. Does this suggest that the current method is overly sensitive to prompt design? \n\n3.Based on the analysis in this article, what insights can it bring to future research? Can you further analyze and summarize in depth?\n\nLIMITATIONS:\nThe author should state that this article focuses on unlearning triplet knowledge, leaving more settings such as multimodality and procedural knowledge for future work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}