ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs
Authors: Manit Baser
Dinil Mon Divakaran
Mohan Gurusamy
Venue: Accepted by TMLR ()

Abstract: Robust model-editing techniques are essential for deploying large language models (LLMs) in practical applications, as they enable cost-effective ways to deal with challenges such as privacy breaches, bias mitigation and misinformation spread. For example, an LLM-based healthcare assistance may need to update out-dated or incorrect knowledge to prevent harmful recommendations. However, many editing techniques focus on isolated facts, which critically fail to prevent indirect knowledge leakage---the unintended reconstruction of edited-out information through persistent causal links and contextual relationships. To assist users in selecting the right editing technique, we develop and present ThinkEval, a framework to systematically quantify indirect knowledge leakage and ripple effects in model-editing. ThinkEval builds and employs specialized knowledge graphs to analyze the causal structure of facts before and after editing. To support this approach, we present KnowGIC, a benchmark dataset comprising multi-step reasoning paths that precisely measure these complex knowledge transformation effects. We evaluate five editing techniques: AlphaEdit, RECT, ROME, MEMIT, and PRUNE across multiple LLMs. Our results show that these techniques struggle to balance indirect fact suppression with the preservation of related knowledge, compromising the contextual integrity of a model's knowledge. Our dataset is available at: https://github.com/manitbaser/KnowGIC.

Published in Transactions on Machine Learning Research (01/2026)
ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs
Manit. Baser
monit. munit.baser&u.nus.com
Electrical and Computer Engineering
National University of Singapore
Dinil Mon Divakaran
dinal dioakaran@a-star.ccmm
* Institute for Infocomon Research (A (A*TAAR PR), Singapore
Mohan Gurusamy
gmohan www.s.duu...u.......
Electrical and Computer Engineering
National University of Singapore
Reviewed on OpenReview: https: // openreview. net/ forum? id= IR2GAW90BB
Abstract
Robust model-editing techniques are essential for deploying large language models (LLMs) in practical applications, as they enable cost-effective ways to deal with challenges such privacy breaches, bias mitigation and misinformation spread. For example, an LLM- based healthcare assistance may need to update out-dated or incorrect knowledge to pre- vent harmful recommendations. However, many editing techniques focus on isolated facts, which critically fail to prevent indirect knowledge leakage- the unintended reconstruction of edited-out information through persistent causal links and contextual relationships. To assist users selecting the right editing technique, we develop and present THINKEVAL, framework systematically quantify indirect knowledge leakage and ripple effects in model- editing. THINKEVAL builds and employs specialized knowledge graphs to analyze the causal structure of facts before and after editing. To support this approach, we present KnowGIC, a benchmark dataset comprising multi-step reasoning paths that precisely measure these complex knowledge transformation effects. We evaluate five editing techniques: AlphaFdit, RECT, ROME, MEMIT, and PRUNE across multiple LLMs. Our results show that these techniques struggle balance indirect fact suppression with the preservation of related knowledge, compromising the contextual integrity of model's knowledge. Our dataset. is GIC
available
1 Introduction
Large Language Models (LLMs) are increasingly getting adopted in various domains such as health-
care (Goyal et al. (2024] : Yang et al. (022] Qiu et al. (2024)), ), cybersecurity (Divakaran & Peddinti (0024)
Zhang et (2025) ). legal sector (Yao al. (2024a); Cheong et al. (2024); Zhang et al. (2024a)), etc. Despite
the impressive capabilitics, L.LMs often retain outdated, sensitive or incorrect information, raising concerns
about privacy, bias, and propagation of misinformation (Sallami al. (2024); Lin et al. (2024a); Zhao &
Song 2024)). ). These models lack inherent mechanism to selectively update knowledge without under-
going costly retraining, leading to the development of model-editing techniques (Hase et al. (2024): Zhong
et al. 2023): Fang et al. (2025) Users ranging from news organizations correcting misinformation, to policy researchers mitigating social bias and platform developers ensuring compliance with privacy laws all require reliable editing techniques tailored to their domain and deployment context.
While recent rescarch works have emphasized preserving the integrity of broader contextual knowledge (Co hen al. (2024): Qin et. al. (2024)), model-editing techniques often overlook the persistence of deducible

---

Published in Transactions on Machine Learning Research (01/2026)
original knowledge (which is supposedly "edited out"). When causally connected facts remain unchanged, the edited information can still be reconstructed through multi-step inference, leading to inconsistencies and undermining the reliability of the model. For example, as shown in Fig. la editing Harry Potter's s school to Ilvermoryy without adjusting his Gryffindor affiliation enables an inference that he studied at Hogwarts. At the same time, broader context, such as Slytherin's 's association with Hogwarts, may become incoherent, weakening the model's contextual integrity. For users, it crucial to make an informed decision for selecting the right editing technique as well as the right model to prevent indirect leakage and maintain contextual integrity before deployment, preserving reliability across diverse use cases.
schoolmate
schoolmate
Harry Potter
Reacher.
Where did Harry Potter study?
Ilvermorny
Ron Weasley
bouss
Drado Maltoy
(b) A direct query.
seitool
Professor McGonagal
house
achool
headmosstess
sichne
house
Who are Harry Potter's 's schoolmates?
Draco Malfoy, Ron Weasley ...
Gryffindor
belonge to
belongs to
Which house did Draco Malfoy belong to?
Slytherin
Hogwarts
Stytherin
(a) A subgraph showing deducibility of the fact "Harry Potter studied at Hogwarts" via connected relation- ships. Nodes represent entities and edges denote rela- tionships. Green solid edges, part of broader contextual knowledge, should be preserved, while red dashed edges, tied to the primary subject, should be edited.
Which school does Slytherin belong to?
Hogwarts!
(c) A 3-step implication chain.
Figure 1: Example of extracting the original fact post-edit. Prompting with a direct query may fail, but a 3-step sequential inference may extract the original fact.
To assist users to select the right model-oditing technique for their model, we propose and develop ThinkEval, framework that quantitatively and systematically evaluates model-editing techniques using sequential prompting, to consequently help with reliable LLM deployment. THINKEVAL utilizes Chain- of Thought (CoT') reasoning (Wei et al. (2022)) to construct model-specific knowledge graphs to quantify implicit associations and reasoning pathways. This enables analysis of how edits affect the model's internal knowledge. CoT reveals chains of connected facts that may either be disrupted by edits or contribute to the unintended recovery of the original fact. These specialized knowledge graphs facilitate analysis of indirect fact recovery, as cach subject-to-object path reflects underlying relationship. To improve their accuracy, human oversight verifies and prunes hallucinated misinterpreted triplets and supplements missing ones that the model may fail to express. Since L.MSs encode and express knowledge differently, THINkFvval also highlights the limitations of model-agnostic evaluation datasets, as discussed in Appendix A Addition- ally. THinKEVAL serves as a -agnantttngggnninngggggg tool, enabling more comprehensive evaluations to support informed decision-making before meployment.
Inspired by recent advances in machine unlearning (Wu al. (2024); Sinha et al. (2024); ; Sanyal & Mandal (2025),, we introduce deep editing (Section 5) new evaluation setting within THINKEVAL. With deep editing, we (1) quantify the extent to which the edited facts can be empirically deduced through multi- step reasoning, and (2) the ripple effect propagation in the broader context. We propose Indirect Fact Recovery (IFR) as a new metric for deep editing (Section 5.1p. IFR measures original fact deducibility via the reasoning paths post-edit. Along with IFR, we utilise Preservation (Cohen et al. (2024)) for deep editing evaluation. This exposes a critical limitation in existing model-oditing techniques and evaluation strategios, which often overlook indirect fact leakage. To illustrate, we present a detailed case study on AlphaEdit in Section 6.2 where over 80% of samples still leak the original fact through indirect paths, underscoring
2

---

Published in Transactions on Machine Learning Research (01/2026)
the need for systematic evaluation before deployment. We further discuss knowledge graph completion and human involvement in Appendix B
Using THINKEVAL, we develop the KnowGIC (Knowledge Graph Implication Chains) benchmark dataset to evaluate model-oditing techniques under the deep editing setting. Unlike prior benchmarks that rely on single-query evaluations, or condense multi-hop reasoning into a single complex query, KnowGIC decomposes reasoning into multi-step implication chains. Each chain is sequence of linked queries, where the answer to one step becomes the premise for the next. This design directly tests whether an "edited-out" fact can still be reconstructed through chained reasoning. KnowGIC consists of 1,406 samples, draws from two sources: (1) diverse samples from the categorics in the MQuAKE datasct (Zhong et al. (2023)) ) ensuring broad relational coverage, and (2) a targeted sample: (Harry Potter, school, Hogwarts), selected to demonstrate the complexity of deeply embedded causal relationships in LLMs. As illustrated in Fig. [ even
when direct queries for the target fact fail, multi-step reasoning can reconstruct the original fact.
We evaluate five parameler-modilying model-editing lechniques: AlphaEdit (Fang et al.
2022 D, RECT (uu
et al. (2024)), ROME (Meng et al. (2022)), MEMIT (Meng et al. (2023)), ) and PRUNE (Ma et al (2025)
Qwen2.5-775 II--truuttt (Team (2024)) ),
(Grattafiori et al. (2024) and GPT2-
XL (1.5B) (Radford et al. (2019) using IFR and Preservation. Our results reveal notable shortcomings in existing techniques that undermine model reliability, highlighting the need for advanced methods to update knowledge holistically while preserving broader contextual integrity.
Contributions. (I) We develop THINKEVAL for systematic evaluation of model-editing techniques, us- ing CoT reasoning to build specialised knowledge graphs to reveal implicit fact associations. (TT) Within THINKEVAL, we introduce deep editing, new evaluation setting that quantifies indirect knowledge Icakage through multi-step reasoning and measures ripple-effect propagation across broader contextual knowledge. We propose a new metric Indirect Fact Recovery (IFR) tailored to this setting. (III) Using THINKEVAL, we construct the KIOwGIC benchmark dataset of 1,406 multi-step sequential-prompting samples for deep edit- ing evaluation. (IV) We systematically evaluate five model-editing techniques across three LLMs, revealing trade-offs between indirect knowledge leakage and contextual integrity.
2 Related Work
Model-editing techniques. Recent works classify model-editing techniques into two broad categories:
1 parameter-modifyinng and @ parameter-preserving techniques (Fang ct al. (2025)) ). Parameter-modifyingg
methods, like fine-tuning, meta-learning (e-g., MEND (Mitchell et al. (2021) }), and locate-then-edit (e.g., ROME (Meng et al. (2022)), MEMIT (Meng et al. (2023), AlphaEdit (Fang et al. (2025)), )), directly alter
model weights. Fine-tuning often causes catastrophic forgetting (Hsueh et al. (022) ), while meta-learning mitigates it but scales poorly (Hsuch et al. ). Locate-then-cdit mothods target specific knowledge, with AlphaEdit. reducing interference via null-space projections (Fang et al. (2025 ), yet sequential edits
still lead to gradual forgetting (Gupta et al. (2024)) )
preeeerrrr-.se methods, such as retrieval-
augmented (o.g., SERAC (Mitchell et al. (002) IKE (Zheng ct al.] 2023))) and memory-basod approaches
(e.g., T-Patcher (Huang et al. (2023)) GRACE Hartvigsen et al. (023 )), use external modules. They
better retain performance and reduce forgetting. T-Patcher adds neurons per mistake (Yao et al. (2024b)
and GRACE uses key-value codebooks (Hartvigsen et al. (2023) ). But these techniques face scalability, generalization, and computational challenges (Zhang et al. (2024b Hsuch et al. (2024); Lin et al. (022bb) ).
GLAME (Zhang et al. (2024b)) uses knowledge graphs for multi-hop reasoning via graph augmentation and graph-based odit modules. However, it overlooks model-specific internal associations, missing latent paths (e.g,, deducing Hogwarts via Slytherin), and relies on external knowledge, incurring more computational costs. RippleCOT (Zhao et al. (2024)) utilises In- Context Learning with CoT reasoning to incorporate a thought component to decompose the multi-hop logic within questions. However, CoT reasoning integration increases computational complexity during inference. ChainEdit (Dong et al. (025 dynamically generates and edits logically connected knowledge clusters to improve consistency, but it relies on pre-defined graph- derived rules and doesn't discuss indirect fact reconstruction.
Model-editing evaluations. KnowEdit ZZhng et al. (2024c) ). LEME (Rosati et al. (024)): ) Counter- Fact Meng et al. (2022) ) and CounterFactPhig THoelscher-Oeemmaae et al. 2023)) focus on direct fact
3

---

Published in Transactions on Machine Learning Research (01/2026)
edits but overlook related knowledge impacts and original fact deducibility. KnowEdit and LEME assess specific lact. changes, while CounterFact and CounterFactPlus test. adaptability to counterfactuals, over- looking chained reasoning effects. MQUAKE (Zhong et al. (2023) incorporates multi-hop query answering. Though valuable for probing broader knowledge, it does not test sequential recovery of facts, which is how users may naturally query LLMs. These evaluations primarily evaluate direct fact accuracy and multi-hop reasoning, overlooking implicit knowledge deducibility or contextual integrity. UnKE.Bench evaluates un- structured knowledge editing, where knowledge is represented in complex, free-form text (Deng al. (2025)) ). However, it doesn't 't consider sequential reasoning to uncover "edited-out" facts. While prior works address ripple effects (Cohen et al. )2024); ; Qin et al. [2024] Chen et al. [2025); ); Wang et al. (2024)), event-level con- sistency (Peng et al. (2024). concept-level and instance-level consistency (Niu et al. (2025)) none evaluate whether edited facts can still be recovered through multi-slep reasoning. THINKEVAL fills this gap through the deep editing setting and the IFR metric.
3 Preliminary
A fact represented as a triplet (,,,r,0), where vi, U2 are entities (subject and object, respectively) and r T relationship connecting them. For example, (Harry Potter, school, Hogwarts) can rep- resent the statement "Harry Potter studied at Hogwarts". A knowledge graph g (V, is directed graph where & VVVXXT is set of edges representing facts that connect entities in V.
3.1 Graph-based reasoning
Unlike rule-based systems (Wu et al. (2024), ) the relationships in G are not derived from predefined set of logical rules. G is designed such that traversing its edges naturally implics relationships between entities. We do not assume transitive closure over G; instead, each path in 9 is retained only when it plausibly implies a meaningful relationship between the base subject and object. path p = (V1, T1, u1, 72, u2..... U2-1 Tni U2) in with u, E V and r; E T captures a chained relationship r between the endpoints, even (V,, r, 02) $ For example, path (Harry Potter, house, Gryffindor, belongs to, Hogwarts) implics Harry Potter studied at Hogwarts.
3.2 Knowledge graphs for model editing
After editing LLM, the internal knowledge graph G (V,E) is updated with set of changes Ag. The updated graph g' = (V","" incorporates:
Adding new edges:
AG. {(v1.r, 02) (c1, 02) $8}
(1)
Removing existing edges:
AG. [(v,,,, 02) (0),7, E E}
(2)
Modifying edges:
AGm. ((i,, 02) (2 E
(3)
The updated edge set is:
E' = (EUAGG UAG \ AG.
(4)
and
V' VV=V De (vi,,,0)) AG, U AGm)
(5)
For this study, we limit our experimentation to analyze modifying edges (Agm) in LLMs.
Definition 1 Deductive Closure: Given a knowledge graph G = (V,8) with E C V x x V as the set of triplets, the deductive closure of g, denoted ((G), the set of all relationships implied by G such that:
1. 3 C R(G), i.e., every triplet (Ui,r, v2) & is direct relationship between in and 02;
4

---

Published in Transactions on Machine Learning Research (01/2026)
Indirect Fact Recovery
Base Tripict [6, r..r.oo 0l
Original Model
Edited Model
Dotaset
Connected Knawiedge Preservation
LLM Response Generation
Chain-of-Though Answering
Facts Segmentation
Triplets Extraction
Human-aesisted Triplet Pruning Addition
?
Triplet r, 0)
Knowledge Validation
Generation
Human-assisted Query Refinerenn
(4p 4g
Knowledge Graph Synthesis
Chain Sequancing
Dataset
18 Tap Sy, Eo S.)
Figure 2: THINKEVAL framework. Initially, ! a base triplct and 2 an LLM are utilized to generate a tailored datasel reflective of the LLM's internal knowledge structure. Nexl, ① the LLM is edited using an editing technique. The 5 the edited model is evaluated over the constructed dataset, yielding insights into the effectiveness of the editing process from deep editing perspective, (6 mcasured via IFR and Proservation.
2. ((G) is deductively closed with respect to path traversal, i.e., for every pair of entities U1, E V. there exists a path = (V1, includes the implied relationship (Wi,Tp: Tp: un), where Up is the composite relationship inferred from the sequence of relationships (F1, (2.:225..,57-11
T1, T1,
1/, :n...,.,,n-1........
9 where each (V,, U++1) E E, then (g))
((G) captures all explicit and implicit relationships derivable from g through its paths.
4 ThinkEval
We introduce THINKEVAL, a framework designed to assist users by constructing specialised knowledge graphs to quantiratively and systematically evaluate model-editing techniques via sequential prompting, as illus- trated in Fig. 2 THiKEVval operates through three interwoven components to create the dataset: Query Validation and Refinement, Automated Triplet Generation, and Graph Synthesis and Chain Sequencing. These components function in a cyclical manner, iteratively refining the knowledge representa- tion. All the respective prompt templates are provided in Appendix P Additionally, THINKEVAL serves dataset augmentation tool, capable of enhancing existing datasets by adding structured implication chains or triplets for a more comprehensive evaluation. Algorithms! [ and showease the key processes utilised in THIKKEVAL. For users, choosing the right model-editing technique is vital for preventing knowledge leakage and ripple effects, thereby safeguarding reliability across diverse applications.
4.1 Query validation and refinement
The Query Validation and Refinement component initiates knowledge extraction by validating a triplet (8,r,0) or a chain (51, .,,..,,,,)))) evaluating if the LLM 'recognizes' the relationship. The process begins by generating targeted queries from a triplet or a chain. THINkEvAl. synthesizes a knowledge graph beginning with a single triplet, and employs two prompt types:
Triplet-based Query Generation prompt. Targets a single triplet elicit direct response. For example, (Harry Potter, student, Hogwarts), forms the query "Where did Harry Potter study?".
5

---

Published in Transactions on Machine Learning Research (01/2026)
Chain-based Query Generation prompt. Probes multi-hop relationships in a chain, targeting the terminal entity, For instance, from (Harry Potter, subject, Transfiguration taught by, Minerva McGonagall), "Who teaches Transfiguration to Harry Potter?" is crafted.
The LLM generates a response to the query, which is analyzed to determine the expected object's presence, accounting for phrasing variations or synonyms. For triplets, the object aligns, the triplet and query are passed to the second and third components; for chains, they proceed only to the second component. If validation fails, human expertise refines the query triplet, re-entering the cycle. After several failed cycles, indicating the model's lack of knowledge, the triplet is discarded.
4.2 Automated triplet generation
The Automated Triplet Generation component extracts further knowledge from the LLM using its reasoning capabilitics, structuring the output into new triplets. These are cycled back to the previous component for validation. This component's 's functionality is demonstrated with an illustrative example in Appendix D
The validated query is submitted to the LLM with CoT prompt, prompting step-by-step reasoning to reveal intermediate associations alongside the final answer. The multi-sentence CoT response is segmented into discrete facts, parsing it into individual statements for triplet extraction. This segmentation simplifies the response into manageable units to preserve extracted information's 's integrity.
Each fact processed by the LLM with a fact extraction prompt to generate atomic triplcts, like (Harry Potter, school, Hogwarts) or (Transfiguration, taught by, Minerva McGonagall). To handle er- from hallucination or context misinterpretation, human experts prune triplets, verifying them against known facts. Authentic triplets are cycled back to the prior component, while human expertise also adds missing triplets to account for the facts overlooked by the LLM.
4.3 Graph synthesis and chain sequencing
The Graph Synthosis and Chain Sequencing component integrates validated triplets into a knowledge graph and structures sequences of query for inferring relationships for the dataset. The constructed chain of triplets (s1, 82, .n)) is cycled back to the first component.
The validated triplets are organized into the directed knowledge graph G. g evolves with each itcration, incorporating new triplets, reflecting the growing complexity of the extracted knowledge.
G is processed to form n-step chains ($1, T1, 82, T2...., ,,),, to serve two purposes: (1) chains are sent to first component for multi-hop query generation, and (2) their query sequences are compiled into a dataset. These are capped at five steps, as longer chains tend to diminish the impact of associating the subject with the object. The effectiveness of extracting meaningful relationships decreases with increasing number of queries, making shorter chains more impactful for establishing clear associations.
5
Deep Editing
Existing model editing techniques often modify a target fact in isolation, often overlooking logically connected facts and thereby introducing inconsistencies. We introduce deep editing as a new evaluation setting that characterizes whether an odit has been performed consistently with respoct to related knowledge. Unlike event-level editing (Deng et al.] (2025)), which focuses on updating facts tied to specific event, deep editing evaluates whether edits propagate appropriately through network of logically connected facts, while preserving unrelated knowledge. A fact is deep edited up to n-steps if it cannot be empirically deduced from the retained knowledge through any sequence of reasoning steps of length S where n is user-defined parameter reflecting computational or application constraints. We provide a comparison of deep editing with other model-editing settings in Appendix C
Suppose an editing technique A modifics target fact t (v1,r,0)) 02) E E, resulting in modified triplets E.7 E to form an updated graph g' = (V" 8'). Here, E' = (E ((4)) AgA, where AEA, {(v,, 03)
6

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm
Query Validation and Refinement
Input:
Knowledge input (either triplct (s, r, o) or chain c (81, f1, 82, 82, , sn)), LLM M, max
iterations kmax
Output: Validated triplet E' chain d' or (discarded)
q + GenerateQuery(Z)
Triplet chain-based prompt
4: k ++0
5: while k: < kimax do
+ +M(())
> LLM response
7:
E then
Object present response
if is chain c then
c
10:
cycle to Algorithm
11:
else
12:
t end if
13:
14:
return y Algorithm 2
15: else
16:
1, t- Humannefinegg, R,
D Refine based input type
17: kek+1
18: end if
19: end while
20: discard
b No validation after kmax
21: return
Algorithm Automated Triplet Generation
Input: Validated query q, LLM M
Output: Set of new triplets Tnew R MMg, CoT)
CoT response D Set of facts
F+ Segmen(Facts(R)
5: Tremp +
6: for fE do
7:
Tj M Extracttromppt)
Extract triplets
Teemp + Tremp UTF T
end for
10: Tnew +- HumanPrmneAntAddl? Ttemp)
Prune and augment
11: return Tnew to Algorithm
Algorithm Graph Synthesis and Chain Sequencing
1: Input: Validated triplet (s, r, o), corresponding validated query
Output: Knowledge graph G, dataset D
V VU (s,o)
E +-= E EU ((s())),))}
5: G + (V,E)
b Initialize graph
6: C - GenerateChains((]
Chains from G Initialize dataset
7: D
for c = (81, 82, r2, ..., 3n) ) do
O. GetCnerySequencccc Imax)
b Query sequence with maximum length 4max
10: D DU
11:
cycle to Algorithm 1
12: end for
13: return G,D

---

Published in Transactions on Machine Learning Research (01/2026)
(Us, rai 11)) EA1) represents the modified triplets in E, with the new relationships rf E T. If technique A deep edits the fact I, then & must not be implied by any path in 9", iee,, i should not belong to li(").
Definition 2 Deep Editing: An editing technique A deep edits the fact t == (V1, U2) with respect to the knowledge graph G = 8) if does not belong to the deductive closure 2(g'), where G' = (V",") , is the updated graph, and E (E EX1) AEM, reflects the modified triplets. Formally, 0, & (g'"), i.e. no path in G implies the original relationship r between U1 and v2.
5.1 Evaluation metrics for deep editing
We outline the Counterfact (Meng et al. (2022) metrics to evaluate model editing Appendix D For doep editing evaluation, we propose new evaluation metric Indirect Fact Recovery (IFR).
IFR. measures the extent to which the original fact remains deducible despite the edit. It is implied by paths in g= (V,E) after editing to g' = =V",'') Let D = {Si..... Sm3 be sequences of queries corresponding to chains from 9, where S, = (1a....,, Usn: J has length n2, and each 4s probes a step in a path implying a target fact. Let Ps = .s.....,s... and Pe = "'.....,P'ss. P's.., P's. = (P'1....,,P,, are the probabilities of the original target output occurring in the model response to each qiy in 9 and G', respectively.
where Ps, = (Pa,...,,Pa) and
Definition 3 Indirect Fact Recovery:
R's, Rss.
VT4
S
if D' + 0,
IFR(G, 9', Pg)
(6)
S
0,
otherwise,
where Rs.,- = [T;= Pij. R's. -1 p. and D' (8, E
Rs. #3.
IFR computes the weighted average of retention ratios R's, R's,/Rs, RR.,, normalized by yng, across all the sequences to emphasize the higher implication strength of the shorter paths. Here, Pi and p'y reflect probability of the original target output (object in triplet) in the model's response di before and after editing. A high IFR indicates significant deducibility of the original target fact, suggesting incomplete deep editing, A low IFR reflects that original fact hardly deducible after editing.
We use Preservation (Cohen et al. (2024) to evaluate the the broader contextual knowledge integrity deep editing. While RippleEdits utilises it to evaluate other subjeectrelatiionsiiss,, we utilise it to measure retention of accuracy across facts in = EV,E) after editing to g' (V' E"). excluding those which are not part of the broader contextual knowledge (links directly linked to the original subject unless specified otherwise). Let to = (su. To. ou) be the original triplet, and End = {(s, o) E E s * so} as the set of triplets where the subject is not 80- For each 1 = ()) E &idd, let pt E [0, 1] and P E [0. 1] be probabilities of the original target output o occurring in the model's response to a query 4: in 9 and g', respectively.
Definition 4 Preservation:
PLe Pt
Eld * 0,
Te
Preservation g', Eind)
(7)
REE!
otherwise,
where End = (1 E Eind Pt 0}-
Preservation averages the ratios PL/P: over all triplets in E' ind, where Pt and p represent probabilities of the original object o in response qi before and after editing. high Preservation indicates strong preserva- tion of broader contextual knowledge, suggesting that the editing process successfully avoided catastrophic
8

---

Published in Transactions on Machine Learning Research (01/2026)
forgetting over them. A low Preservation reflects loss of accuracy in these facts, implying unintended knowledge degradation within the model.
6 Performance Evaluation
Using the KnowGIC benchmark, we evaluated five model-editing techniques: AlphaEdit, RECT, ROME, MEMIT and PRUNE via IFR, Preservation and Efficacy on three models, Qwen2.5 (7B), Llama3 (8B) and GPT2-XL (1.BB.. All implementations are sourced from Jiang 2024). All experiments are performed on an NVIDIA A100 80GB GPU.
6.1 KnowGIC Benchmark
To evaluate in the deep editing sctting, we introduce the KnowGIC benchmark, which is built using THIKEEVAL Extending the multi-hop reasoning datasets, KnowGIC consists of n-step implication chains, or sequences of n prompts to probe specific relationships via multiple reasoning steps. These chains capture direct and implied relationships for fine-grained evaluation. Unlike MQuAKE's n-hop queries, which en- capsulate multiple reasoning hops in single query, each implication chain step probes a single hop in the knowledge graph. If the original fact can be logically reconst through such sequential prompting, it signals incomplete editing. Conversely, breaks in the chain that disrupt unrelated but connected facts reveal unintended ripple effects. Table presents the distribution of chain lengths in the dataset. All chains have been manually reviewed lo remove redundancy, factual inaccuracies, and irrelevant triplets, ensuring high quality and relevance for deep editing evaluation.
Table 1: Distribution of multi-step implication chains in the KnowGIC benchmark.
Chain length (steps)
1
2
3
4
5
Number of Chains
24
108
227
428
619
We selected various base templates from the MQUAKE dataset across diverse relational domains, generating knowledge graphs using THINKFVAl for their samples. To illustrate the depth of connected facts and how interlinked facts can help extract the original relationship, we include an additional sample (Harry Potter, school, Hogwarts). Using THINKEVAL, we grow its knowledge graph till achieves 100 implication chains from Harry Potter to Hogwarts. The KnowGIC benchmark, including this sample, comprises 1,406 n-step reasoning paths. Extensive details of the dataset are in Appendix
6.2 Motivation for new metrics
Existing metrics like Efficacy and Specificity provide valuable insights into diffcrent aspects of model editing, but are insufficient for deep editing evaluation. Efficacy focuses on immediate effect of the edit, overlooking whether the original fact can be inferred via sequential reasoning. Specificity focuses on the correctness of the LLM's response in terms of higher probability assigned to the correct response. Similarly, Consistency focuses only on factual coherence. Deep editing quantifies how much the original fact can be deduced even via sequential reasoning. Hence, a metric is essential to evaluate this dimension comprehensively.
Table 2: Harry Potter casc-study setup and results.
Relationship triplet
(Harry Potter, school, Hogwarts)
Edit request
Hogwarts - lveeroryy
Model
Llama3-8B-Instruet
Edit technique
AlphaEdit
Paths identified
100
Active paths post edit
80
Efficacy (f)
1.000
IFR. (L)
0.780
9

---

Published in Transactions on Machine Learning Research (01/2026)
Harry Potter
Hogwarts
Where did Harry Potter study?
0.2
Who teaches Trensfigurstion Harry Potter?
Which school is Professor McGonagall the headmistress of?
1.0
Professor McConegall
1.0
Who Harry Potter's schoelmates?
Which Thomas
school does Gryftindor belong
Dean
Gryffindor
Who were Harry Potter's
Who taught Pations to Hermione Granger?
Which house is Professor Snape the head
Which school does Slytherin balong to?
schoolmates?
Hermione Granger
Professor Snape
Slytherin
Who were Harry
Who were Hermione Granger's schoolmates?
Who teaches Transfiguration Which house is Professor
Which school does Gryffindor belong to?
Potter's as achoolmates?
Ron Weasley?
McGonagall the of?
1.0
Hermione Granger
Weasley
0.8
Professor McGonagall
Gryffindor
Figure 3: Samples of n-step chains from Harry Potter case-study, leading to original fact leakage even after editing. The number below each link represents the ratio of responses (out of five generations) that retain the original output, quantifying the extent to which the LLM reveals the initial fact via indirect reasoning.
We conduct casc-study on the sample (Harry Potter, school, Hogwarts) from KnowGIC to evaluate the need for new metrics in deep editing, aiming lo edit the fact (Hogwarts Ilvermorny). The selup is detailed in Table 2 For each query in the implication chains, we generate five distinct responses from both the original and edited models. An Efficacy of 1.000 shows that the LLM responds with Ilvermorny when queried directly. However, it does not account for original fact recovery through indirect reasoning paths. An overall IFR of 0.780 suggests that the original fact still remains deducible to significant extent. Figure g shows samples from these 80 active chains.
Table 3: n-step IFR for the Harry Potter case-study.
n
n-step chains
Active chains post edit
n-step IFR
0.200
2
20
14
0.814
3
30
23
0.736
35
30
0.484
5 14
12
0.402
As shown in Table 8 the 1-step chain with IFR of 0.200 shows effective suppression of direct inference. However, the 2-step and 3-step chains exhibit high IFR values of 0.814 and 0.736 respectively, demonstrating high indirect fact. leakage through short reasoning paths. The 4-step and 5-step chains show moderate IFR values of 0.484 and 0.402, further demonstrating persistent leakage across longer paths. Despite an Efficacy of 1.00, the original fact persists through multi-stop rcasoning, indicating AlphaEdit's surface-level success but failure in deeper knowledge updates. For an editing method to qualify successful under the deep editing paradigm, the original fact must be eliminated across all reasoning paths. This case-study also highlights that traditional metrics like Efficacy are insufficient for deep editing. IFR's focus multi-step chains provides a more robust evaluation. Further details of the case-study are in Appendix H
6.3 Results and discussions
Table 4 presents the overall IFR, Preservation and Efficacy values for different editing techniques across GPT2-XI, Llama3-8B, and Qwen2.5-7B. For Llama3-8B, AlphaEdit achieves the highest Efficacy of 0.958, followed by PRUNE at 0.875, indicating strong performance in updating target facts. For Qwen2.5-7B, AlphaFdit achieves perfect Efficacy of 1.000, with MEMIT also performing strongly 0.958. For GPT2-
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Comparison of model-editing techniques for various models using IFR, Preservation (Pres.), and
Efficacy (Eff.). Values are color-coded from red (lower performance) green (higher performance).
IFR (L)
Model
Technique
Pres. (t)
Eff. (t)
Overall
1-step
2-step
3-step
4-step
5-step
GPT2-XL Qwen2.5-777 2-BBB Llama3 8B
AlphaEdit
0,509
0.413
0.557
0.641
0.626
0.351
0.890
0.958
MEMIT
0.550
0.434
0.628
0.634
0.599
0.461
0.857
0.792
RECT
0.487
0.454
0.465
0.552
0.564
0.406
0.823
0.708
PRUNE
0.187
0.178
0.187
0.195
0.195
0.178
0.696
0.875
ROME
0.134
0.173
0.116
0.222
0.177
0.060
0.685
0.833
AlphaEdit
0.496
0.108
0.269
0.563
0.579
0.497
0.869
1.000
MEMIT
0.577
0.097
0.287
0.535
0.549
0.741
0.859
0.958
RECT
0.569
0.304
0.561
0.599
0.679
0,494
0,851
0.750
PRUNE
0.514
0.378
0.427
0.521
0.496
0,561
0.765
0.792
ROME
0.332
0.177
0.212
0.173
0.381
0,417
0.669
0.708
Edit
0.461
0.199
0.547
0.639
0.565
0.294
0.663
1.000
MEMIT
0.484
0.271
0.518
0.654
0.549
0.362
0.683
0.875
RECT
0.516
0.482
0.539
0.673
0.548
0.413
0.689
0.708
PRUNE
0.211
0.238
0.208
0.209
0.219
0.203
0.587
0.917
ROME
0.196
0.299
0.237
0.263
0.278
0.158
0.500
0.792
Techniques
1.0
Techniques
Techniques
-- AlphaEdit
MEMIT PRUNE
RECT
+ AlphaEdit
MEMIT PRUNE
RECT
- AlphaEdt
+ MEMIT
- RECT
0.8
RDME
0.8
-B- ROME
0.8
-BI ROME
PRUNE
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0 1-step
2:step
3-step
4-step
S-step
0.0
1-step
2-step
3sstep
4-step
5-step
-L-step
2-step
3-step
4-step
5:step
n-Step Implication Chains
n-Step Implication Chains
n-Step Implication Chains
(a) Llama3-8B
(b) Qwen2.5-7B
(c) GPT2-XL
Figure 4: IFR for n-step samples. A lower IFR implics lower doducibility of the supposodly-edited fact.
XL, both AlphaEdit and PRUNE achieve high Efficacy scores of 1.000 and 0.917, respectively. Effficccy, however, provides limited insights into the performance of these techniques by focusing direct edit success while neglecting the broader implications of these edits.
In contrast, IFR scores reveal significant variability across models, as shown in Fig. 4 For Llama3-88 AlphaEdit cxhibits an IFR increase from 0.413 (1-step) to 0.641 (3-step) before declining to 0.351 (5-step), indicating that the original fact remains highly deducible, particularly via mid-range chains (3-step). On Qwen2.5-7B, AlphaEdit's IFR rises steadily from 0.108 (1-step) to 0.497 (5-step), showing persistent fact leakage. For GPT2-XL, AlphaEdit's ss IFR increases from 0.199 (1-step) to 0.639 (3-step) before dropping to 0.294 (5-step), following a similar fise-then-fall pattern. MEMIT Qwen2.5-7B shows a sharp IFR. increase from 0.097 (1-step) to 0.741 (5-step), while on Llama3-8B, it follows comparable rise-then-fall trend (0.434 at 1-step, 0.634 at 3-step, 0.461 at 5-step). For GPT2-XL, MEMIT's IFR rises from 0.271 (1-step) to 0.654 (3-step) before declining to 0.362 (5-step), highlighting contrasting leakage patterns. ROME and PRUNE maintain low TFR across all models, with ROME on Llama3-8B dropping 0.060 at 5-step, and on GPT2- XL to 0.158, indicating strong reduction in original fact leakage. However, this may suggest ripple effects related knowledge. These techniques are lypically designed to edit direct facts (I-step chains), expecting their impact to diminish as chain length increases, ideally resulting in increasing IFR curve. Most techniques on Qwen2.5-7B align with this expectation, but deviate on other models, showing a peak before declining.
The rise-then-fall IFR pattern in AlphaEdit, MEMIT, and RECT on Llama3-8B and GPT2-XL is attributed to the interplay between their editing strategies and ripple effects. As IFR relies on the product of link strengths in an implication chain, longer chains (4-5 steps) are more likely to encounter weakened links, reducing deducibility due croded related knowledge. These techniques disrupt broader contextual con- nections, causing a decline in IFR beyond mid-range chains. In contrast, ROME, and PRUNE show lower
11

---

Published in Transactions on Machine Learning Research (01/2026)
Modes
GFT2
Lanaa 88
Preservation (11
Dwen2.5 40-
0.4
0.2
0.0
AlphaEdit
MEMIT
RECT Technique
PRUNE
ROME
Figure 5: Preservation for different model-editing techniques. retention of broader context integrity.
higher Preservation indicates stronger
0.6
0.6
0.6
Techniques
Tec techeiquas
Techniquws
0.5
AphaEdt
0.5
AighaEst:
0.5
AphaEd:
ROME
ROME
ROME
0.4
MENIT
0.4
NEVIT
0.4
MENIT
PRUIEE
PRUNE
PRUREE
0.3
RECT
0.3
RECT
0.3
RECT
0.2
0.2
0.1
0.1
0.1
0.44
0.4.
0.8.4
0.9
1.0
Preservation (t)
Preservation (1)
Preservation (t)
(a) Llama3-8B
(b) Qwen2.5-7B
(c)
CPT2-XL
Figure 6: IFR vs. Preservation for different model-editing techniques, illustrating trade-offs in indirect fact recovery and related knowledge preservation.
IFR across all models, risking over-editing, reflected by their Preservation scores. As illustrated in Fig. 5 ROME's Preservation is relatively low (0.685 Llama3-8B, 0.669 on Qwen2.5-7B, 0.500 on GPT2-XT.), indi- cating poor preservation of related facts, while AlphaEdit achieves higher Preservation (0.890 on Llama3-8B, 0.869 on Qwen2.5-7B, 0.663 on GPT2-XL). GPT2-XL models show low preservation across all techniques due to high entanglement facts stemmiin from its smaller size (Qin et al. (2024]). For Qwen2.5-7B. the predominantly increasing IFR curves, such as AlphaEdit's rise from 0.108 (1-step) to 0.497 (5-step), prompt additional investigations explore its robustness over extended implication chains, detailed in Appendix [A] Fig. ( highlights the varying performance of these techniques. For users, IFR delivers critical insights into the subtle impacts of edits across implication chains, revealing strengths and weaknesses missed by existing metrics and underscoring the importance of systematic evaluation frameworks such as THINKEVAL.
7 Conclusion
In this study, we discuss the challenge of maintaining consistent knowledge in LLMs under model editing, We introduced THINKEVAL, a framework which offers users involved in updating and deploying a model a systematic evaluation editing techniques. Within this framework, we defined the deep editing evaluation setting to quantify the original knowledge leakage and ripple effects caused by edits, and proposed Indirect Fact Recovery (IFR) as metric to quantify such leakage, thus assisting these model and the right editing technique for their case. For evaluation, we constructed the KnowGIC benchmark, dataset of multi-step reasoning paths. Our experiments expose significant limitations in five state-of-the-art editing techniques across three models. While these techniques effectively deal with directly queried facts, they frequently fail to eliminate indirect knowledge leakage, allowing the original information lo persist. Additionally, edits often introduce unintended ripple effects that disrupt the coherence of surrounding contextual knowledge. These findings highlight the need for evaluation frameworks like THINKEVAL to guide users in selecting editing methods and for the development of more holistic techniques that update knowledge reliably without compromising contextual integrity.
in choosing the suitable
12

---

Published in Transactions on Machine Learning Research (01/2026)
References
Qizhou Chen, Dakan Wang, Taolin Zhang, Zaoming Yan, Chengsong You, Chengyu Wang, and Xi- aofeng He. Uniedit: A unified knowledge editing benchmark for large language models. arXio preprint arXiv:2505.1280 12345, 2025.
Inyoung Choong, King Xia, KJ Kevin Feng, Quan Ze Chen, and Amy X Zhang. (a) i am not a lawyer, but...: engaging legal experts towards responsible Ilm policies for legal advice. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transporency, pp. 2454 24699 2024.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects knowl- edge editing in language models. Transactions of the Association for Computational Lònguistics, 12: 283-298, 2024.
Jingeheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. Everything is editable: Extend knowledge editing to unstructured data in large language models. In Proceedings of the International Conference Learning Representations (ICLR 2025), 2025. URL httpe://openreeveei het/paffid-K6r00vy3
Dinil Mon Divakaran and Sai Teja Peddinti. Large language models for cybersccurity; New opportunities.
IEEE Securily& & Privacy, 2024.
Zilu Dong, Xiangqing Shen, Zinong Yang, and Rui Xia. ChainEdit: Propagating ripplc effects in LLM knowl- edge editing through logical rule-guided chains. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13558-13571, Vienna, Austria, July 2025. As- sociation for Computational Linguistics. ISBN 979-8-89176-251-00 doi: 10. 111663/w1/2025.aclll URL http::://aclaathhologg org/ htt.2/20....c1ooogg.a 665/
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat- Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. In Proceedings of the International Conference on Learning Representations, 2025, 2025. accessed 2025-12-06.
net/
Jianqi Gao, Hao Wu, Yiu-ming Cheung, Jian Cao, Hang Yu, and Yonggang Zhang. Mitigating forgetting in adapting pre-trained language models to text processing tasks via consistency aligument. In Proceedings of the ACM on Web Conference 2025, pp.
2025a.
Jianqi Gao, Hang Yu, Yiu-ming Cheung, Jian Cao, Raymond Chi-Wing Wong, and Yonggang Zhang. Shaping pre-trained language models for lask-specific embedding generation via consistency calibration. Neural Networks, pp. 107754, 2025b.
Sagar Goyal, ELi Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward. Healai: A healtheare 11m for effective medical documentation. In Proceedings of the 1 17th ACM International Conference on Web Search and Data Mining, pp. 1167-1168, 2024.
Sagar Goyal, Eti Rastogi, Fen Zhao, Dong Yuan, and Andrew Beinstein. Specialtyscribe: Enhancing soap note scribing for medical specialties using llm's. In Proceedings of the Second Workshop on Patient- Oriented Language Processing (CL4Heulth), pp. 34 45, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint 2rXio:2407.21188 2024.
Jia- Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regularization to the rescue. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16801-16819, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/vv /v1/////////// /tt/224.2.mmppp... mwin.aannnn........ URL https://aclantholl.g/ org/ 2024. emnlp-main 934/
13

---

Published in Transactions on Machine Learning Research (01/2026)
Akshat Gupta, Amurag Rao, and Gopala Amumanchipalli, Model editing at scale leads to gradual and catastrophic forgetting. ar preprint or #rXii2440.007455 2024.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36:47934 09644994-47959, 2023.
Peter Hase, Thomas Hofweber, Xiang Zhou, Flias Stengel-Fskin, and Mohit Bansal. Fundamental problems with model oditing: How should rational belief revision work in llms? arXiv preprint arxiv:2406. 19354. 2024.
Jason Hoelscher-Oberrmaicc Julia Persson, Esben Kran, Toannis Konstas, and Faal Barez. Detecting edit failures in large language models: An improved specificity benchmark. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Compulational Linguistics: ACL 2023, pp. 11548-11558 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1 /202. findings-acl.735 URI. htt.://pp//acaaatooooooo///////2 winniinns-ac..... 733/
Cheng-Hsur Hsun Hsueh, Paul Kuo-Ming Huang, Tzu-Han Lin, Che-Wei Liao, Hung- ChggChhhhhhP Fang, Chao-Wei Huang, and Yun-Nung Chen. Editing the mind of giants: An in-depth exploration of pitfalls of knowledge editing in large language models. arXiv preprint or Xiv:24000.00 01486, 2024.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong. and Zhang Xiong, Transformer-patelan One mistake worth one neuron. In Proceedings of the International Conference on Learning Representations (ICLR 2023), 2023. URL https://openreviier wttpe//eeee..eee///ddddde/oooea
Houcheng Jiang. Alphaedit: Null-space constrained knowledge editing for language models, 2024. URL
GitHub repository.
Lena John, Tim Wittenborg, Sören Auer, and Oliver Karras. Human-in-the-looe workflow for neuro-symbolic scholarly knowledge organization. arXiv preprint Stl:::2506032222 2025.
Prashank Kadam. Enhancing financial fraud detection with human-in -nop feedback and feedback prop- agation. arXiv preprint arXiv:2411.05852 2024.
Luyang Lin, Lingzhi Wang, Jinsong Guo, and Kam-Fai Wong. Investigating bias in Ilm-based bias detection: Disparities between Ilms and human perception. arXie preprint arXiv:2403. 1 14896, 2024a.
Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wempeng Yin, and Lifu Huang. Navigating the dual facets: A comprehensive evaluation of sequential memory editing in large language models. arXiv preprint arXim:2402. 1100002..11111 11122, 2024b.
Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zue Ling, and Jia-Chen Gu. Perturbation-restraaned sequen- tial model editing. In Proceedings of the International Conference on Learning Representations (ICLR 2025). 2025. URL
Kevin Meng, David Bau, Alex Andonian, and Yonalan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359 17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In Proceedings of the International Conference on Leurning Represenlations, 2023, 2023. Available at http:////ccc.....vvtteeee 201///222////////// /ral// 12726.
Eric Mitchell, Charles Lin, Antoîne Bosselut, Chelsea Finn, and Christopher Manning. Fast model editing scale. Xive preprint arxiv:1100. 1000.11000 11309, 2021.
Eric Mitchell, Charles Lin, Antoine Bossclut, Christopher D Manning, and Chelsea Finn. Memory- based model editing scale. In International Conference on Machine Learning, pp. 15817-15831. PMI.R, 2022.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Yifan Niu, Miao Peng, Nuo Chen, Yatao Bian, Tingvang Xu, and Jia Li. Reledit: Evaluating concep- tual knowledge editing in language models via relational reasoning. In Findings of the Associalion for Computational Linguistics: ACL 2025, pp. 10220 10220-10238, 2025.
Ollama Team. Ollama: Run large language models locally. Ittpp:///ollama com 2023. Accessed: 2025-04- 29.
Ciyuan Peng, Feng Xia, Mehdi Naseriparsa, and Francesco Osborne. Knowledge graphs: Opportunities and challenges. Artificial Intelligence Review, 56(11):13071-11100, 2023.
Hao Peng, Xiaozhi Wang, Chunyang Li, Kaisheng Zeng, Jiangshan Duo, Yixin Cao, Lei Hou, and Juanzi Li. Event-level knowledge editing. ar preprint or Xrii:22402. 13093, 2024.
Jiaxin Qin, Zixuan Zhang, Chi Han, Penglei Yu, Manling Li, and Heng Ji. Why does new knowledge create messy ripple effects in LLMs? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12602 12609, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1 m1/2024.emmll-main.. 700. URL httpps:/aaclantholog) org/ 2024. emnlp main. 700/
Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. LIm-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6(12):1418 1420, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Yahya Kayani, Frank Rudzicz, and Hassan Sajjad. Long-form evaluation of model editing. Tn Kevin Duh, Helena Gomez, and Steven Bethard (eds.); Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3749 3780, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1 2004 maw.. long. 208. URL https: /aaclanthology.oo org. naacl-long... 208/
Dorsaf Sallami, Yuan-Chen Chang, and Esma Aimeur. From deception to detection: The dual roles of large language models in fake news. arXiv preprint arXiv:2409.17416, 2024.
Debdoop Sanyal and Murari Mandal. Alu: Agentic I1m unlcarning. arXie preprint arXiv:2502.00406 2025.
Yash Sinha, Murari Mandal, and Mohan Kankanhalli. Unstar: Unlcarning with self-taught anti-sample reasoning for llms. Xrxiv preprint ar arXiv:2410 10. 17050, 2024.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL httts:://wwenlm.ggt http://wwww...igu....m. io/ bllgg/qwen..5
Stefani Tsaneva, Danilo Dessi, Francesco Osborne, and Marta Sabou. Knowledge graph validation by inte- grating lims and human-in cemman-in-the-er in-the-loop. Information Processing & Management, 62(5):104 2025.
Changyue Wang, Weihang Su, Qingyao Ai, Yichen Tang, and Yiqun Liu. Knowledge editing through chain- of-thought. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 10684-10704, 2025a.
Jianchen Wang, Zhouhong Gu, Xiaoxuan Zhu, Lin Zhang, Haoning Yo, Zhuozhi Xiong, Hongwei Feng, and Yanghua Xiao. Efficienlly quantifying and mitigating ripple effects in model editing. arXin preprinl 0:XXv:2403.07855, 2024.
Jianchen Wang, Zhouhong Gu, Xiaoxuan Zhu, Lin Zhang, Haoning Ye, Zhuozhi Xiong, Sihang Jiang, Hong- wei Feng, and Yanghua Xiao. The missing piece model oditing: A deep dive into the hidden damage brought by model editing. In ICASSP 2025-2023 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASPP), pp. 1-5. IEFE, 2025b.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural in information processing systems, 35:24824 24837, 2022.
Ruihan Wu, Chhavi Yadav, Russ Salakhutdinov, and Kamalika Chaudhuri. Evaluating deep unlearning in large language models. arXiv preprint arXiv:2410.15155, 2024.
Haotong Yang, Zhouchen Lin, and Muhan Zhang. Rethinking knowledge graph evaluation under the open- world assumption. Advonces in Neural Information Processing Systems, 35:8374 8385, 2022.
Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. Talk2care: An Ilm-based voice assistant for communication between healthcare providers and older adults. Proceedings of the ACM on Interactive, Mobile, Weorable and Ubiquitous Technologies, 8(2):1 35, 2024.
Shunyu Yao, Qingging Ke, Qiwei Wang, Kangtong Li, and Jie Hu. Lawyer gpt: A legal large language model with enhanced domain knowledge and reasoning capabilities. In Proceedings of the 2024 3rd International Symposium on Robotics, Artificial Intelligence and Information Engineering, pp. 108 112, 2024a.
Zihan Yao, Yu He, Tianyu Qi, and Ming Li. Scalable model editing via customized expert networks. arxiv preprint arXiv:2404. 026699, 2024b.
Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, and Dan Meng. When lims meet cybersecurity: systematic literature review. Cybersecurity, 8(1):1144 41, 2025.
Kepu Zhang, Weijie Yu, Sunhao Dai, and Jun Xu. Citalaw: Enhancing Ilm with citations in legal domain. arXiv preprint Xiv:2244 2. 14556, 2024a.
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. Knowledge graph enhanced large language model editing. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 22647- 22647, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/vV
/v1 htt///1112222/2 //2//2222.2mnppp 20022.cmnlpmmmm..11. 1261. URL
1261/
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint orXiv:2401.01956 2024c.
Guoshenghui Zhao and Eric Song. Privacy-prcserving large language models: Mechanisms, applications, and future directions. arXiv preprini vrXiv:2412.06113 2024.
Zihao Zhao, Yuchen Yang, Yijiang Li, and Yinzhi Cao. RippleCOT: Amplifying ripple effect of knowledge editing in language models via chain-of-thoughtt in-context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 6337-6345 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/Ve /v /2/224. fini.ing-eemmp..3..... URL
findings- emnlp.368
Ce Zheng, Lei Li, Qingxin Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint Xiv:::2305.1 12740 2023.
Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Polts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15686-15702, Singapore, December 2023. Association for Computational Linguistics. doi: 10.1 11633/61//233./mm//// m911 URL htttps://aclantees wtt.://awatooooooooo//2./33.... o02/2023, mmnlp-main.977
16

---

Qwen2.5-7B accurately answered only 81% of the queries included across all implication chains, causing inference from a significant number of chains infeasible. This may be due to the fact that Llama3-8B was utilised for fact extraction for the KnowGIC dataset. To explore this further, we conduct additional tests using Ollama {Ollama Team (2023)) ). These tests are run a sample of the queries which failed to answer correctly while inferencing with the original unedited model. This is to determine whether existing queries Published in Transactions on Machine Learning Research (01/2026)
Target Triplet: (Hermione Granger, taught Ancient Runes by, Bathsheda Babbling)
Prompt Who teaches Ancient R unes Hermione Granger?
Response: In the Harry Potter series, Anclent R is not actually course taught at Hogwarts School of Witchcraft and Wizardry, Instead, Hermione Granger leams about ancient languages and scripts on her own. She shows an interest in Ancient Runes and spends time studying them independently.
The subject that comes closest to Ancient Runes within the wizarding curriculum at Hogwarts is Muggle Studies, which deals with the study of non-magical cultures and their inventions, including writing systems and historical texts, However, this not directly related to Hermione's interestin ancient languages from her studies.
In the books, Hermione's knowledge of ancientsrript such as Old Norse Runes and other similar similar scripas comes from her independent reading and research outside of class.
Prompt What does Bathsheda Babbling teach to Hermione Granger? Response: It seems there might be mix-up Rowling, nor do they teach anyone directly.
names titles There no known character named 'Bathsheda Babbling" in the Harry
P Poter serie written However, can help clarify If you meant "Bathilda Bagshot, who appears in some of the supplementary materials and books related to the Harry ooter universe. Bathilda Bagshot character known for her magical history books, but she doesn't specifically teach Hermione Granger anything in the main series.
If you're referring to a specific lesson or teaching that Hermione received from another character (like a professor), could you please provide more deta This would help me give you more accurate and detailed answer.
Prompt Did Bathsheda Babbling teach H Hermione Granger Ancient Runes?
Response: No, Bathsheda Babbling not teach Hermione G Granger Ancient Runes. The character you might thinking of is Bathilda Bagshot, who wrote "A History of Magic,' a book that Hermione used to leam about Ancient Runes. However, Hermione's primary teacher for Ancient R unnes was Professor Barty Crouch r., his guise as Mad-Eye Moody during her third year at Hogwarts.
Hermione studied Ancient Runes alongside other subjects and deived into the subject with great interest due to its magical and historical significance. She often relled on books like "A History of Magic" for additional reading and knowledge, but her main instruction came from Professor C rouch/Moody in class.
Figure 7: Qwwen255-7B' Responses to various prompts for the target triplet (Hermione Granger, taught Ancient Runcs by, Bathshoda Babbling)
A Original Model Performance on KnowGIC Queries
in the dataset are not able to fetch the expected responses, or the knowledge itself is not there in the model.
Fig 7 illustrates Qwen2.5-7B's responses to three prompts designed to test its knowledge of the triplet (Hermione Granger, laught Ancient Runes by, Bathsheda Babbling). These include a direct query where the object is not present within the prompt ("Who teaches Ancient Runes to Hermione Granger?"), a query where the relationship is not present within the prompt ("What does Bathsheda Babbling teach to Hermione Granger?"), and prompt which contains all the three elements of the triplet ("Did Bathsheda Babbling teach Hermione Granger Ancient Runes?"). The model's responses reveal significant knowledge gaps, such as denying Ancient Runes as a Hogwarts course, confusing Bathsheda Babbling with Bathilda Bagshot, and incorrectly attributing other entities. Thus, we can conclude there are gaps in Qwen2.5-7B's knowledge with respect to a few queries in KnowGIC.
Fig 8 illustrates Qwen2.5-7B's responses to three prompts designed to test its knowledge of the triplet (Nicole Richie, spouse, Joel Madden). The prompts include direct query where the object is not present within the prompt ("h is Nicole Richie's 's spouse?"), query where the relationship is not present within the prompt ("Who is Joel Madden's spouse?"), and prompt which contains all the three elements of the triplct ("Is Joel Madden's spouse Nicole Richie?"). The model's responses are inconsistent: it incorrectly names Blake Shelton as Nicole Richie's spouse and Ashlee Simpson as Joel Madden's 's spouse, despite later confirming that Joel Madden is married to Nicole Richie.
Evaluating another triplet shown in Fig Don Rich was a key member of The Buckaroos and not of The Jordanaires. The Buckaroos is the backing band for Buck Owens, not Flvis Presley or Charlie Rich. Don
17

---

Published in Transactions on Machine Learning Research (01/2026)
Target Triplet: (Nicole Richie, spouse, Joel Madden)
Prompt: is Richie's spouse?
Blake Shelton. They married on September 25, 2016. Blake Shelton is American singer and musician, best known for being coach on seasons 8-13 of "The Voice" and for his successful career in country music.
Response: Nicole Richie's
Prompt: Who is Joel Madden's spouse?
Response: Joel Madden, the lead singer of the band Good Charlotte and formerly of No Doubt (not be confused with Tony Kanal who was Doubt),
Ashlee Simpson. They got married on April 15, 2006, and had daughter named Wilder Amour Madden. However, Joel and Ashlee/ divoroad 2019.
Prompt: Is Joel Madden's spouse Nicole Richie?
Response: Yes, Joel Madden is married to Nicole Richie. They got married on September 29, 2007, and have two children logether: Jasmine Joy Madden (born 2010) and Haven James Madden (born 2014). Joel the singer the band Good Charlotte, while Nicole Richie well-known fashion designer, model, television personality.
Figure Qwen2.5-7B's Responses to various prompts for the largel triplet (Nicole Richie, spouse, Joel Madden)
Target Triplet: (Don Rich, band, The Buckaroos)
Prompt Which band was Don Rich a part of?
J ordanaires. He joined group 1957 was known smooth baritone vocals and his ha harnonies, which contributed significantly to the sound of many famous artists including Elvis Presley, P Patsy Cline, and Iim R Revess. The ordanaires are one of the most successful vocal groups country music history, backing numerous chart-topping hits throughout their long
Response: Don Rich
member
country music
Prompt Was Don Rlich a member of The Buckaroos music band?
Response: Don Rich indeed associated The Buckaroos, but role wasn'ta member band itself. Don Rich guitarist who played for the Charlie Rich Band, which supported The Buckaroos during their performances in the mid-1960s. Charlie Rich was famous American country and rockabill billy singer, and Don Rich's guitar wark significantly contributed to the sound that made Charlie Rich successful. The Buckaroos were influential band associated with Eivis resley. They played at resley's home studio, Graceland, and were known for their energetic performances and distinctive sound in rockabilly and country music
Figure 9: Qwen2.5-7B's Responses to various prompts for the target triplet (Don Rich, band, The Buckaroos)
Target Triplet: (Benjamin Keough, mother, Lisa Marie Presley)
Prompt Who is the mother Benjamin Keough?
Response: I'm sorry but don't have any specific information a abot who the mother Benjamin Keough might be. Without additional context or details, it's difficult to provide accurate information. The "B "Benjamin Keough" does not pondespond widely known historical figure or public personality that can confidently ide neni as needing such information, If you could provide more context or specify which Benja min Keough you're referring (e.g.. the time period, location, or any other relevant details), might be able assist you better.
Prompt is Lisa Marie P resley the mother of Benjamin Keough? Response: Lisa Marie Presley 2023, and there
the mother B Benjamin Keough. Benjamin Keough's biological parents are ustin and Keough. Lisa
Marie Presley passed is no public information suggesting she has child named Benjamin Keaugh. you have more specific details ar context abuutoour question, feel free provide for further clarification!
Figure 10: Qwen2.3-7B's Responses to various prompts for the target triplet (Benjamin Keough, mother, Lisa Marie Presley)
18

---

Published in Transactions on Machine Learning Research (01/2026)
Rich wasn't just supporting guitarist for another act; he was the lead guitarist and part of the band from early 1960s until 1974. The Buckaroos were not associated with Elvis Presley Graceland. They were Buck Owens' band, formed in Bakersfield, California. Charlie Rich, a separate artist had his own career and band, but Don Rich had no notable connection to him.
Evaluating the triplet shown Fig 10 Lisa Marie Presley mother of Benjamin Keough. Benjamin Keough was born on October 21, 1992, Lisa Marie Presley and her husband Danny Keough. There's no evidence of Benjamin Keough born to Justin and Erin Keough. Hence, for evaluating model editing, there is need to customize the prompts which may be specific to the model itself.
IFR trends for Qwen2.5-7hP differ significantly from those of Llama3-8B, which may be due to the reason men- tioned above. In Qwen2.5-7B, a higher number of implication chains remain inactive, leading to devialions from the observed trends in Llama3-8B. These errors highlight the challenges in model editing, as Qwen2.5- 7B fails to accurately reflect the target triplets, underscoring the need for frameworks like THINKENAL that enable model-specific prompt customization and identification of model-speeific relationships to address such deficiencies effectively.
B On Human Involvement and Graph Completion
Human involvement is essential in current LLM research due to the well-known issue of hallucination. Models often produce plausible but incorrect information. Several existing knowledge editing datasets also incorpo- rate human involvement (Deng et al. (2025): : Pong et al. (2024); Tsaneva et al. (2025)). Similarly, various
recent. LLM studies leverage HITL leedback (John et al. (2025); Kadam (2024)) to ensure factual accuracy, address data sparsity, and improve robustness. Our method follows this standard practice prioritize precision and interpretability.
The core objective of THINKEVAL is not to oblain complete knowledge graphs but to operationalize deep editing. Because of how knowledge is ever-evolving, it is extremely difficult to empirically establish that any knowledge graph is complete. Authors in Yang et al. (2022) and Peng et al. (2023) discuss the theory of Knowledge Graph Completion. Although there are many methods for constructing knowledge graphs, is still infeasible create comprehensive representations of all the knowledge in a field. They further dive into the open-world problem and the incompleteness of knowledge graphs (the open-world assumption).
For the Harry Potter case-study in Section 6.2 THINKEVAL generates the knowledge graph up-to 100 chains with minimal human intervention. This is not hard limit but design choice for our evaluation. These graphs can be expanded further using THINKEVAL, making it flexible and scalable tool for probing deep relational entanglements in LLMs.
C Comparison of Model-Editing Settings
Table 5 provides a detailed comparison between various model-editing settings. The use of specialised knowledge graphs in deep editing to infer over implication chains enables us to uncover subtle editing failures is something which other benchmarks or settings may miss. We measure whether the original facts remain deducible post-edit through reasoning paths, and also test how edits propagate through related facts. Either of the other two model-editing settings don't consider sequential reasoning to uncover "edited-out." facts.
D Counterfact Metrics for Model Editing
In evaluating model editing for LLMs, variety of metrics assess different aspects of edit quality and impact. Based on prior work (Meng et al. (002 ) we outline the Counterfact evaluation metrics for assessing model oditing in LLMs. Existing metrics are defined for an LIM fe, a knowledge fact prompt rs,, an edited target output o,, and the model's original output of, providing a comprehensive framework evaluate the effectiveness and robustness of edits:
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Comparison of Model-Editing Settings
Model Edit- ing Setting
Unstructured
Event-based
Deep Editing (Ours)
Focus
Evaluates unstructured knowledge editing, where knowledge is reprosented in complex, free-form text rather than structured formats.
Edits event-driven knowl- edge affecting multiple facts and future ten- dencies; mirrors how knowledge evolves through events rather than isolated fact changes.
Evaluates deep editing to prevent fact deducibility and preserve broader con- lext; assesses editing tech- niques via multi-step se- quential inference.
Primary Benchmark
UnKEBench
ELKEN
KnowGIC
Dataset Creation
1,000 counterfactual un- structured texts soureed from ConflictQA.
Built on Wikidata for 1,515 event, edits; GPT-3.5 para- phrasing and human verifi- cation for 6,449 factual and 10,150 tendency questions. Reliability and Locality
1,406 reasoning chains; each sample includes 1-5 multi-stpp questions.
Metrics Used
FactScore, MMLU, Rouge Scores, BERT BLEU
IFR, Preservation
Scores,
Efficacy. This metric quantifies the proportion of cases where the edited output or is more likely than the original output of when the model is queried with the prompt rs):
Ec(Ps, (o))())))))) Pj. Prllo)((ssr)))))
Generalization. Generalization measures the proportion of cases where or remains more probable than of in paraphrased prompts (()s,, which rephrase the original statement while preserving its meaning:
(o > ((s.)())((s
Specificity. Specificity evaluates the model's ability to preserve correct facts in neighborhood prompts O(ss,,))), which are prompts about related but distinct subjects:
E, ((((Polal(((ssrr))) (p)))))((ss,rtt)))
(10)
Fluency. Fluency assesses the quality of the model's generated text by measuring excessive repetition through the entropy n-gram distributions. computed as:
4 Σ gs(k) gs(k)
Σ 92(k) logz (k) 3
(11)
where (.)(.) represents the frequency distribution of n-grams in the output.
Consistency. Consistency evaluates the alignment between the model's generated text and a ground-truth reference. Given subject 8, the model fe generates text, and the cosine similarity computed between the TF-IDF vectors of this output and reference Wikipodia text about the target o, ensuring factual coherence.
E Various Prompts used in ThinkEval
The THINKEVAL framework employs structured set of prompts to extract, evaluate, and edit the internal knowledge of LLMs, illustrated in Figures 11
12
and
13
The Fact Extraction Prompt systematically
20

---

Published in Transactions on Machine Learning Research (01/2026)
Triplet-based Question Generation Prompt
Given relation triplet which contains the entity, relationship and attribute, generate a question about the entity that would help elicit relationship with the attribute. Strictly generate the output only.
Example:
Harry Potter. studied at. Hogwarts
School of
Witchoraft and Wizardry
Output:
Where did Harry Potter study?
Triplet: <target triplet>
Figure 11: The template of the triplct-basee query generation prompt.
Chain-based Question Generation Prompt
Given a chain of alternating entities and relationships, generate question based all the entities and their relationships so that the answer the last entity in the chain. Strictly generate the output only.
Example:
Harry Potter, subject, Transfiguration, taught by.
Minerva McGonagall
Question:
Who teaches Transfiguration to Harry Potter?
Chain: <target chain>
Figure 12: The template of the chain-based query generation prompt.
Fact Extraction Prompt
You are fact extraction assistant specialized in parsing Q&A data. Your task to analyze the input extract all fact triples (subject, relationship. object) a precise manner from the given Question and Answer. Each triple should encapsulate a single atomic fact expressed in the text. Focus characters, events, places. Give a but a numbered list.
Example:
Benedetto Varchi, profession, poet Benedetto Varchi, work, Storia Fiorentina
question prompt> Response: <model response>
Question:
Figure 13: The template of the fact extraction prompt.
21

---

Published in Transactions on Machine Learning Research (01/2026)
Question
Where did Harry Potter study?
2
Harry Potter studied at Hogwarts. Let's break it down step by step:
Chain-of-Thought Answering
Background Context: Harry Potter
Wizarding Education: In the wizarding Harry's Journey: After receiving his Confirmation: Hogwarts is explicitly..
Facts
Segmentation
(Harry Potter, studied at, Hogwarts)
(Harry Potter, house, Gryffindor)
Triplets
(Harry Potter's education, covered in, series)
Extraction
(Harry Potter, studied at, Hogwarts)
(Harry Potter, house, Gryffindor) (Draco Malfoy, house, Slytherin) (Harry Potter's education, covered in, series)
Human-assisted Triplet Pruning and Addition
Figure 14: Automated Triplet Generation.
retrieves factual relationships from the model, enabling the construction of its internal knowledge graph by identifying entities and their connections (e.g., mapping relationships like "Harry Potter, school, Hogwarts"). The Triplet-Based Query Generation Prompt creates targeted queries from (subject, rela- tion, object) triplets to evaluate factual consistency, such as "Who teaches Ancient Runes to Hermione Granger?" for (Hermione Granger, taught Ancient Runes by, Bathsheda Babbling). Similarly, the Chain-Based Query Generation Prompt does the same for a relationship chain. By leveraging these prompts, TunkKEval effectively maps model-specific knowledge structures, identifies inconsistencies, and facilitates precise evaluation of model editing, thereby addressing deficiencies in LLMs with greater accuracy and reliability.
F Explanation of Automated Triplet Generation Process
Figure 14 illustrates the Automated Triplet Generation process, which constructs knowledge graphs by transforming query like "Where did Harry Potter study?" into structured triplets. It be- gins with query, followed by CoT reasoning to systematically answer it, such as covering back- ground context, wizarding education, Harry's journey, and confirmation that Hogwarts is his school. From this, facts are extracted (e.g., "Harry Potter studied at Hogwarts") and converted into triplets: (Harry Potter, studied at, Hogwarts), (Harry Potter, house, Gryffindor), and (Harry Potter's education, covered in, series). Human-assisted pruning removes inaccuracies and irrele- vant facts like (Harry Potter's education, covered in, series), retaining the relevant set: (Harry Potter, studied at, Hogwarts), (Harry Potter, house, Gryffindor). Human-assisted triplet addi- tion utilised to add triplets which may be missod by the automated process even after few itcrations (such as (Draco Malfoy, house, Slytherin)). This process, combining CoT reasoning and human over- sight, ensures accurate triplets for evaluating fact deducibility and contextual knowledge preservation.
22

---

Published in Transactions on Machine Learning Research (01/2026)
G
KnowGIC Benchmark Details
Table 6: Various base query templates, triplets, and categories in the KnowGIC benchmark.
Category
Base query templates is citizen of child was created by is married to is associated with the sport
Sample triplet
Country of Citizenship
(Byron Dorgan, country of citizenship, USA)
Parental Relationship
(Elvis Presley, child, Lisa Marie Presley)
Creation and Origin
(Miss Piggy, created by, Jim Henson)
Marital Relationship
(Victoria Beckham, spouse, David Beckham)
Sport Affiliation
(Mark Teixeira, sport, baseball)
Country of Origin
was created in the coun- try of
(basketball, country of origin, USA)
Development Origin
was developed by
(Atari Jaguar, developer, Atari Corporation)
Work Location
worked the city of
(Vincent Auriol, work location, Paris)
Employers
was employed by
(Ward Kimball, emploger, The Walt Disney Company)
Production Entity
The company that produced is
(Buick LaCrosse, produced by, General Motors)
Study Location
studied at
(Harry Potter, school, Hogwarts)
To provide an overview of the KnowGIC benchmark employed in our evaluation of model-editing techniques, Table O presents the base queries, their corresponding triplet samples, and their respective categories. Our evaluation includes total of 83 distinct relationship types, derived from extending these base relationships through controlled logical transformations. The dataset encompasses 1,406 samples, each being an implica- tion chain of varying lengths (1 to 5 steps), as detailed in Table [ Table 7 provides a few samples of the implication chains of varying lengths present in the dataset. It spans diverse knowledge domains, including country of citizenship (e.g., "What the country of citizenship of Byron Dorgan?"), parental and marital relationships (e.g., "Who is Hillary Clinton's child?" and 'Who is Nicole Richie married to?"), creation origins (e.g., 'Who was Miss Piggy created by?"), sport affiliations (e.g., "Which sport is Mark Teixeira associated with?"), country of origin (e.g,, "Which country was Bakersfield sound created in?"), work locations (e.g.. 'Which city did Petronius work in?"), production entities (e.g., 'Which company is Buick LaCrosse produced by?"), and study locations (c.g., 'Where did Harry Potter study?"), among others.
Each triplet. represents a factual relationship that serves as the foundation for both direct queries and multi-step implication chains, which are designed to test fact deducibility and broader contextual knowledge prescrvation post-edit. For example, (Harry Potter, school, Hogwarts) is evaluated through the direct query "Where did Harry Potter study?" and extended into implication chains such those exploring related entities like schoolmates and teachers. This structured representation enables systematic evaluation of knowledge consistency across diverse domains, facilitating the evaluation of editing techniques' effectiveness in reducing deducibility (via IFR) and preserving connected knowledge (via Preservation). thus supporting the development and refinement of frameworks like THiNKFval for robust editing LLMs. Figures 15 16 21 and 22 present few of the knowledge graphs present in KnowGIC, which can be further extended using TVALL
18
19 20
H Harry Potter Study Details
Table S provides an overview of comparison of the existing metrics with our proposed metrics IFR, focussing on capability to evaluate indirect fact extraction. We conduct a casc-study on the sample (Harry Potter, school, Hogwarts) from KnowGIC evaluate the need for new metrics in deep editing, aiming to edit the fact. (Hogwarts lvermorny). We evaluated the edit using Effieacy and proposed IFR metric. This
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Multi-step implication chains samples from KnowGIC. Each row presents an implication chain with the request integrated into the sample queries evaluate fact deducibility and broader contextual integrity post-edit.
n-step
Implication chain with knowledge triplet
Triplet: (Petronius, work location, Rome)
In which city did Petronius work?
2
Triplet: (Hillary Clinton, child, Chelsen Clinton)
Who is Hillary Clinton's son-in-law? - Who is Mare Mezvinsky's 's wife?
3
Triplet: (Word Kimball, employer, Walt Disney)
Which character Ward Kimball animate? which film does Jiminy Cricket appear?
Which company produced Pinocchio?
4
Triplet: (Harry Poller, school, Hogwarts)
Who were Harry Polter's schoolmates? - Who were Ron Weasley's schoolmates? -
Who laught Transiguration to Hermione Granger?
Which school Professor McGonagall the headmistress of?
5
Triplet: (Bakersfield sound, country of origin, USA)
Which city is known the creation of Bakersfield sound? +
In which valley is Bakersfield situated? In which county is the San Joaquin Valley located?
+ In which state Kern County situated? + What country is California located in?
Table 8: Comparison of model editing metrics. This table compares existing metrics with our proposed IFR,
highlighting its focus on addressing indirect fact extraction.
Metric
What It Measures
Quantifies Indirect Fact Extraction
Efficacy
Success in outputting the edited fact for direct queries. Performance on replrased versions of the edited query.
X
Gemeralization
X
Specificity
Ensures unrelated queries unaffected by the edit.
X
Fluency
Naturalness and coherence of the model's responses post-edit. Logical coherence across related facts.
Consistency
(Partially focuses
on direct consistency)
IFR
Extent to which the original fact remains deducible through multi-step reasoning.
Preservation
Retention of facts related to the edited subject but not directly targeted.
X
case-study highlights that traditional motrics like Efficacy are insufficient for deep oditing tasks. IFR's focus on multi-step chains provides a more robust evaluation. These findings stress the need for advanced metrics and better editing methods for comprehensive knowledge updates in language models.
Generalization beyond factual model editing
While our work focuses factual model edits, which provide well-defined and controllable setting for evaluating indirect knowledge leakage, the core design of THINKEVAL is not inherently limited to factual knowledge alone. THINKEVAL relies on CoT-reasoning and graph-based causal modeling, both of which can, in principle, be extended to other forms of knowledge.
For example, commonscnse or procedural reasoning scenarios, graph nodes may represent abstract con- cepts, actions, or intermediate states rather than factual entities. Edges extracted from CoT traces can encode causal, temporal, conditional (fif-then") relationships, enabling the construction of thought-based
24

---

Published in Transactions on Machine Learning Research (01/2026)
Dest ray
Pehoo
Figure 15: Sample of a constructed knowledge graph from the KnowGTC implication chains for the relation- ship triplet (Harry Potter, school, Hogwarts). The graph was expanded with the generated triplets until 100 implications paths from the initial subject lo object were achieved. The red node represents the initial triplet subject, and the green node represents the initial triplet object.
Brock Jo ph Backham
Harper s en Bechbam
Cruz David cockham
David Beckha
Victoria Beckham
Romeo James Beckham
Figure 16: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Victoria Beckham, spouse, David Beckham).
25

---

Published in Transactions on Machine Learning Research (01/2026)
Madhya Pradesh
Iida
Lok Sabha
Bharatiya Janata Party
Uma Bharti
Figure 17: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Uma Bharti, country of citizenship, India).
Priscilla Beaulish Presley
Rssee
Ben mir Reough
Elvis Presley
Ale eough
Harper Vivienne Ann Lackwpoo
Finley Lockwood
Figure 18: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation-
ship triplet (Elvis Presley, child, Lisa Marie Presley).
26

---

Published in Transactions on Machine Learning Research (01/2026)
Texas R gers
All Ai-s Seleciio
Major Looso Baseball
Basetal
Mark Tebxeiaa
Siiver Slugger
Los Angeles Angels
Atlanta
Figure 19: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation-
ship triplet (Mark Teixeira, sport, baseball).
Mafc Meawnsyy
Aidan Ciimto Midwinsky
Bill dento
Chels Clinton
Hillary Clinton
Jasper Clinton Mezvesky
Charlotte Clinton Mezvinsky
Figure 20: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation-
ship triplet (Hillary Clinton, child, Chelsea Clinton).
27

---

Published in Transactions on Machine Learning Research (01/2026)
Farbsia
Dasset
Ole Wision
Chidensla
Nine fed
Ward he
nccom
Whitgrng Reiharman
The Theee Cabaliers eros
Dunboo
Figure 21: Sample of a constructed knowledge graph from the KnowGTC implication chains for the relation- ship triplet (Ward Kimball, employer, The Walt Disney Company).
pon Ruch
the Buckaroos
Bapentedd
Haggerts
Control Valley
Sap-bazule Valley
Kam Courtyy
Figure 22: Sample of constructed knowledge graph from the Know implication chains for the relation- ship triplet (Bakersfield sound, country of origin, USA).
28

---

Published in Transactions on Machine Learning Research (01/2026)
graphs non-factual knowledge structures. Similarly, for mule-based logical knowledge, nodes may correspond to predicates or rules, with edges capturing inference dependencies.
Under such extensions, the same evaluation paradigm (measuring the persistence or disruption of reasoning paths after targeted edits) can be used to assess indirect leakage and preservation behavior beyond factual settings. We leave the systematic instantiation and benchmarking of EvaL for these broader knowledge types as an important direction for future work.
J Illustrative example of IFR calculation
To provide clarity on the IFR metric, we present a step-by-step calculation using representative 3-step reasoning chain. This example illustrates how the metric quantifies the persistence of an original fact through indirect reasoning paths after a model edit has been performed.
J.1 Scenario Setup
Consider a scenario where we altempt to edit a specilic fact the model's knowledge base:
Target Edit: Harry Potter studied at "Hogwarts" + "IIvermoryy..
Despite this edit, the original fact (Herry Potter + Hogwarts) may still be recoverable the model retains related knowledge that allows it to reconstruct the original connection through multi-step reasoning.
J.2 Sample Reasoning Chain
We define a chain of questions that logically lead back to the original fact:
1. Q₁= Who were Harry Potter's schoolmates? - A₁ Ron Weasley.
Q2: Which house is Ron Weasley in? + Ag: Gryffindor.
Q3: Which school does Gryffindor belong to? As: Hogwarts.
J.3 Chain Recovery and IFR Computation
We measure the model's confidence (probability) in producing the correct intermediate facts both before and after the edit:
Question
Pre-Edit Probability (Ppre) Post-Edit Probability (Ppost)
QiQ
0.90
0.70
C2Q2
0.85
0.80
C1Q3
0.90
0.85
Table 9: Pro-odit and post-edit probabilities for 3-step reasoning chain.
Step Calculate Chain Recovery (R)
The chain recovery represents the overall likelihood that the original fact can be reconstructed through the reasoning path, calculated the product of the link probabilities:
R = 0.9 x 0.85 x 0.9 = 0.69
R Rpost 0.7 0.8 0.85 0.48
Step 2: Apply the IFR Formula
The IFR metric normalizes the change in recovery relative to the chain length n:
TFR (Rpost/ Rproo/
0.48/v3
0.70
1//n
0.69/V3
29

---

Published in Transactions on Machine Learning Research (01/2026)
J.4 Interpretation
In this instance, IFR of 0.70 indicates that even after the model was explicitly edited to believe Harry Potter attended Ilvermorny, the original association with Hogwarts remains 70% recoverable with respect to the original model. This demonstrates the knowledge leakage that THIKEVVLL is designed to systematically detect. The (1/yn) weighting normalizes for chain length by mitigating the compounding uncertainty of multi-step reasoning. This prevents long, noisy chains from dominating the metric and emphasizes shorter, higher-confidence reasoning paths.
Furthermore, both IFR and Preservation are inherently normalized with respect to each model's pre-edit factual accuracy. Specifically, the pre-edit performance terms in the denominators of each serve as a cal- ibration factor, ensuring that post-edit scores are interpreted relative to each model's original knowledge baseline. This design miligates model-specific bias and enables fair comparison across models with differing pre-edit accuracies.
K
Scalability and automated validation study
To evaluate the practical deployment of TINKEVALL, we conducted a scalability study measuring the end- to-end runtime and the efficacy of an automated LLM-based judge to reduce human overhead. We have provided all the samples our repository.
K.1 Computational Efficiency
We measured the average end-to-end runtime of the THINKEVAL process over incremental iterations until a threshold at least 100 samples was reached. The initial setup began with a single triplet. The growth of the reasoning graph and associated costs are detailed in Table 10
Iteration
Triplets
Paths (Subject + Target)
CoT Queries
12
2
15
3
43
3
23
124
4
51
27
405
5
83
133
654
Table 10: Growth reasoning paths and computational queries across iterations.
The empirical results confirm the efficiency of the framework:
Average CoT Runtime: 4.15 seconds per prompt.
Total Computational Cost: ~8.4 minutes of CoT GPU time for 133 samples.
End-to-End Latency: ~5 minutes total, including human verification (~4-5 min) and graph synthesis.
These findings suggest that while CoT-prompting is the primary computational driver, the validation and graph synthesis stages remain lightweight and highly parallelizable.
K.2 LLM-based Automated Judging
To minimize human dependency, we implemented an LLM-based judge (using GPT-4o) to validate and prune triplets. The judge filters misinformation to ensure only high-quality triplets are used for graph construction. We evaluated the judge's performance across five iterations, as summarized Table II
The LLM-judge achieved precision of 88%, recall of 86%, and an Fi-score of 87% (based on aggregate totals of TP 71, FP 10, FN 12). Notably, the automated judge reached the 100-reasoning-chain
30

---

Published in Transactions on Machine Learning Research (01/2026)
Iteration
True Positives (TP)
False Positives (FP) False Negatives (FN)
Correct Paths
0
2
13
2
20
4
48
2
35
109
Table 11: Performance of the GPT-4o judge in identifying valid triplets and reasoning paths.
threshold in the same number of iterations (five) as the human-verified baseline. Our results indicate that reserving hurman effort for final verification while using the LLM-judge for intermediate pruning can reduce human validation effort by 70 75% with only marginal loss in accuracy.
L Evaluation of parameter-preserving model editing
To assess the robustness of indirect knowledge preservation in non non- conammrmetricc editing scenarios, we evaluated the In-Context Knowledge Editing (TKF) technique (Zheng et al. 2023 on the KnowGTC dataset. Unlike traditional parameter-modifyye techniques, IKE relies on in-context learning to update model behavior without altering underlying parameters. The results of this evaluation are summarized in Table I2
Model
IFR Preservation
GPT22XL
0.680
0.542
Llama3-8B
0.729
0.756
Qwen2.5-7B
0.623
0.247
Table 12: Evaluation of IKE across different LLMs using the KnowGIC dataset.
The empirical findings indicate that IFR remains significant factor even within parameter-ppe preeservgggg techniques. For instance, Llama3-8B exhibits an IFR of 0.756, suggesting that substantial portion of the original knowledge remains accessible through reasoning paths despite the presence of in-context edit. These results suggest that in-context editing techniques do not fully prevent hidden factual leakage, highlighting the need for more rigorous editing strategies.
M Responsible model editing and future directions
Responsible model editing in LLMs necessitates a rigorous ethical framework to balance utility with safety, Techniques like THInKEVAL primarily serve enhance model reliability by facilitating privacy protection, such implementing the "right to be forgotten" for sensitive data, and the rapid correction of misinfor- mation. However, model editing could be leveraged for malicious injection of bias or the suppression of legitimate information. To mitigate these risks, we advocate for the implementation of transparent edit logging and strict human-in-tie numan-in-the-loop verification for all high-stakes or sensitive modifications. Furthermore, as our results demonstrate that single-query evaluations often fail to reveal hidden inference pathways, we propose that robust soquential testing be a mandatory component of the safcty audit process for any edited model to prevent the unintended leakage of suppressed facts.
Achieving low IFR while maintaining high Preservation remains central challenge for model editing, Fu- ture editing techniques may benefit from explicitly optimizing this tradc-off by incorporating constraints regularizers that minimize indirect leakage while bounding degradation on unrelated knowledge. In addition, integrating multi-step and adversarial reasoning probes during the editing process can help proactively close indirect inference pathways that singlo-query checks often miss. In this sense, THINKEVAL may serve not only as an evaluation framework, but also as practical guide for developing editing techniques that balance factual precision with contextual retention, supporting more robust and responsible model editing.
31

---

Published in Transactions on Machine Learning Research (01/2026)
N Reproducibility and transparency
To support reproducibility and transparency, we release all the model versions and editing configurations NVIDIA A100 80GB GPU.
utilised in experiments. All experiments performed
Model Versions. All language models used in experiments are publicly available on HuggingFace, Unless otherwise stated, models are used in their official, unmodified form as released by their respective authors, and no additional fine-tuning is performed prior to editing.
In our experiments, we experiment over the following models from HuggingFace:
meta-llama/Meet--lieee ---.-------aa----------- B-nsst--------tttrett
Qwon//Qnen223-7--IIstruce
opena--commuuityygptt....
Editing Configurations. For each editing technique, utilise the implementation given in AlphaEdit's repository (Jiang (024)).. We follow the hyperparameter configurations provided in the publicly released implementation, including the choice of layers to edit, thus ensuring faithful comparison.
Dataset reliability
To ensure the robustness of our evaluation framework and address potential biases in human validation, conducted pilot annotation study with three research group members. A random sample of 250 implication chains was reviewed to verify the presence of logical implications. While the strength of implication may decrease as chains grow longer, we mitigate this by scaling results by 1/V7, where n represents the number of links the chain; this weighting strategy was unanimously agreed upon by the annotators.
Furthermore, all factual triples were reviewed. To quantify consistency, we report raw agreement and Gwet's AC1, the latter provides more stable reliability metric in the presence of high agreement.
Table 13: Inter-annotate agreement statistics for human validation.
Metric
Value
Raw Agreement
977
Gwet's AC1
~98%
As shown Table 13 the results demonstrate high inter-annotator consistency and minimal bias.
P Automation and human-in-the-loop integration
To clarify the operational mechanics of THINKEVAL, we define the role of LLM and human involvement within our algorithm. THINKEVAL utilizes a single, unedited model; no secondary LLM agents or external models are employed for meta-processing. The workflow categorized into LLM-driven automation, rule- based logic, and humen-in-theellop (HITL) refinement, as detailed in Table 14
Comparison: ThinkEval vs. MQuAKE
While MQIAKE focuses on multi-hop reasoning within static datasot, THINKEVAL provides a dynamic framework to evaluate fact leakage and preservation through sequential queries. The fundamental differences summarized in Table 15
32

---

Published in Transactions on Machine Learning Research (01/2026)
Table 14: THINKEVAL Automation and HITL Components
Automation Type
Component
Where in Figure
Comment
Query generation
Right before blue region !
Similar to LLM-driven query generation (0023
Using unedited original LLM
Zhong et
LLM response generation Clain-of-Thougee answering
Blue region !
Pink region (
Triplets extraction
Pink region
Similar to Wang et al. (2025a) and other benchmarks
Knowledge validation
Blue region (
Rule-baser way Zhong et al. 2023 (using alias list) Segmented
No LLM human involvement
Fact segmentation
Pink region Z
into individual
statements
Chain sequencing
Green region
Query refinement
Blue region
Hemman-in-the-looy
Triplet pruning and addition Final sanity checks
Pink region
@
(HITL)
Dataset (Green region
Table 15: Comparison between MQuAKE and THINKEVAL.
Dimension
MQUAKE
ThinkEval
Fundamental differ-
Dataset
Framework to create datasets
ence
What they evaluate
Multi-hop reasoning in edited LLMs; however, multi-hop rcasoning re- mains beyond current parameter- editing techniques, which still strug- gle with ripple effects (Wang et al. (0025b))
Shows edited-out fact leakage via se- quential queries; bridges the gap be- tween reasoning and ripple effects by mapping paths in knowledge graphs, thereby highlighting which additional facts require editing and which must be preserved
How they evaluate
Multi-hop reasoning questions where multiple relations are compressed into single query
Multi-step sequential queries; eval- uates reasoning across sequences of queries, extending beyond single- query formats to reflect how users may naturally interact with LLMs Mitigales this issue by decompos- ing reasoning into simpler, sequential steps
Adaptability
Unedited models correctly answer only 30-40% of MQuAKE's multi-hop querics (as reported in Zhong et al. (2023 ), restricting the usable portion of the dataset.
Metrics
Singlo-query metrics
IFR (Section 5.1 and Prescrva- Lion CCohen et. al. (0224)
R Automated workflow robustness and results stability
We conduct a controlled stability experiment to test whether THinEEvAL's key findings remain robust when secondary LLM is introduced. In particular, examine whether (i) editing technique rankings and (ii) the characteristic rise-then-fall behavior of IFR persist under automated paraphrasing.
Experimental Setup. We select a 100-chain subset from KnowGIC and introduce a secondary model into the pipeline for automated paraphrase generation. Specifically, GPT-4o to generate three paraphrases
33

---

Published in Transactions on Machine Learning Research (01/2026)
Table 16: Stability test results across models and techniques using paraphrased query chains.
Model
Technique
IFR
Pres.
Overall
1-step
2-step
3-step
4-step
GPT2-XI,
AlphaEdit
0.736
0.903
0.736
0.433
0.773
0.824
0.652
MEMIT
0.727
0.880
0.727
0.433
0.903
0.714
0.671
PRUNE
0.250
0.782
0.250
0.311
0.280
0.248
0.232
RECT
0.754
0.850
0.754
0.667
0.752
0.829
0.686
ROME
0.279
0.752
0.279
0.233
0.250
0.282
0.301
Llama-3-8B
AlphaEdit
0.678
0.900
0.678
0.342
0.604
0.683
0.781
MEMIT
0.800
0.911
0.800
0.615
0.867
0.814
0.767
PRUNE
0.309
0.647
0.309
0.111
0.434
0.363
0.170
RECT
0.894
0.880
0.894
0.615
0.897
0.925
0.875
ROME
0.217
0.571
0.217
0.081
0.350
0.167
0,187
Qwen2.5-7B
AlphaEdit.
0.391
0.917
0.391
0.285
0.315
0.512
0.313
MEMIT
0.394
0.865
0.394
0.297
0.392
0.458
0.323
PRUNE
0.411
0.587
0.411
0.316
0.320
0.442
0.445
RECT
0.525
0.823
0.525
0.387
0.492
0.558
0.520
ROME
0.261
0.537
0.261
0.198
0.237
0.312
0.221
for each atomic factual triple (e.g., (Horry Potter, school, Hogworts) + "Harry Potter studied at", "Harry Potter's school is", "Harry Potter went to school at").
All five editing techniques are evaluated across three base models: GPT-2-XL, Llama-3-8B, and Qwen2.5-1 To incorporate paraphrasing into the evaluation, each link in reasoning chain is scored by averaging model correctness probabilities across its three paraphrased queries. IFR and Preservation are then recomputed using these averaged scores, ensuring that each chain reflects robustness across surface-level linguistic varia- tion. For transparency and reproducibility, we release the full paraphrased subset along with resultant plots in our repository.
Table 16 summarizes IFR and Preservation scores under paraphrasing across all models and editing tech- niques. We observe strong stability across all dimensions:
Preservation of relative rankings. Across all three models, the relative ordering of editing techniques remains consistent with the original evaluation. AlphaFdit, MEMIT, and RECT continue to exhibit higher IFR alongside strong Preservation, while ROME and PRUNE consistently achieve lower IFR at the cost of reduced Preservation. This mirrors the same trade-off pattern reported in the main experiments.
Persistence of IFR dynamics. The characteristic rise-then-fall behavior of IFR as chain length increases persists under paraphrasing, indicating that this effect is not an artifact specific prompt formulations but reflects stable reasoning behavior in edited models.
Robustness to automated workflow choices. Introducing a secondary L.M for paraphrase generation does not alter the qualitative conclusions drawn from THIKEVVAL. IFR, Preservation, and editing-techniqqe comparisons remain stable, suggesting that THINKEVAL robust to reasonable automation choices and workflow variations.
These results demonstrate that THINKEVAL is not sensitive surface-level phrasing or evaluation pipeline details, reinforcing its reliability evaluation framework for model editing.
The design of THINKEVAL is deeply informed by the broader literature on maintaining representational consistency and miligating catastrophic forgetting. Specifically, our focus on minimizing IFR while maxi- mizing Preservation aligns with the principles of consistency alignment and calibration (Gao et al. 2025a b)). These works emphasize that as LLMs are specialized or odited, maintaining the underlying structural co- herence of their embedding space is critical to preventing the degradation of previously learned knowledge. THINK EVAL extends this concept by providing evaluation framework that explicitly tests whether this
34

---

Published in Transactions on Machine Learning Research (01/2026)
coherence survives multi-step, sequential query chains. Furthermore, the practical utility of our framework extends into high-stakes, specialized domains. As demonstrated in applications like medical scribing for specialized clinical notes (Goyal et al. (2025)), the ability to edit models while ensuring they do not leak suppressed or outdated medical facts is paramount. By integrating THINKEVAL into the lifecycle of domain- specific model adaptation, practitioners can better assess the robust of their edits in environments where factual precision and logical consistency are non-negotiable.
35


Reviews:
[{"review_id": "1zDlwSe053", "reviewer": "Reviewer_B1vK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\n- The authors consider the complex task of selectively modifying facts after LLM training without costly re-training from scratch. The authors highlight that this task is difficult because editing out specific facts does not preclude the ability of a user to recover the removed fact using multi-step logical inferences, given access to the model; the authors have an example about recovering information about where Harry Potter goes to school, which they reference throughout the text and incorporate into their dataset. \n- The authors propose formalizing the set of deductions as the set of all paths in a knowledge graph G, where nodes represent entities and edges represent relationships between entities. The intuition they propose is that “traversing [the edges of such a graph] naturally implies relationships between entities.” Thus, they propose that the set of all paths in G “captures all explicit and implicit relationships derivable from G through its paths.”\n- The authors propose a framework for evaluating the effectiveness of different model editing techniques, which they call ThinkEval. ThinkEval is a meta-algorithm, in which chain-of-thought prompts are used to generate knowledge graphs modeling a set of relationships known to an LLM. The LLM is then modified using the editing technique under investigation. After editing, the knowledge graphs generated pre-edit can then be used to generate multi-hop queries, to assess whether the edited model can actually be prompted to recall the selected deleted fact(s) based on other relationships retained within the knowledge graph. \n- The authors formulate “deep editing” as a new target task. Unlike previous editing task definitions, deep editing is only considered successful if an entity-relationship-entity triplet does not belong to the closure of the knowledge graph of the LLM after editing.\n- In order to quantify deep editing performance, the authors introduce Indirect Fact Recovery (IFR). This metric computes the weighted average of retained path lengths, normalized to place a higher penalty on shorter paths.\n\n**Strengths:**\nThe authors tackle a very ambitious and relevant problem. Even their flagship example (Harry Potter) calls attention to the extreme importance of model editing in pursuit of respecting copyright.\nThe specific concern about recovering edited facts using multi-step logical inference is also very well-motivated, as the authors point out how brittle targeted fact-forgetting can be.\n\n**Weaknesses:**\n- The proposed framework (ThinkEval) seems difficult to automate and thus to scale. \n   - Consider the first step, “Query Validation and Refinement.” This requires turning a triplet or chain of relationships into a natural language query. The natural language query requires specific background knowledge to formulate. In their example, the authors state that (Harry Potter, subject, Transfiguration, taught by, Minerva McGonagall) becomes “Who teaches Transfiguration to Harry Potter?” It seems that generically triplets/chains may have many interpretations: why not “By whom was Harry Potter subjected to Transfiguration, as taught by Minerva McGonagall?” a prompt with the same entities/relationships but which is nonsensical and does not align with background knowledge from the Harry Potter series. This causes me to question how effectively the process of converting triplet/chains into prompts can be automated.\n   - How do the authors propose to automate such a process? My understanding from the manuscript is that in practiece they deploy a secondary LLM agent for converting triples/chains into prompts, and which then “analyzes [the target model’s response] to determine the expected object’s presence.” Is this correct?  \n   - At other steps, the authors explicitly acknowledge that human users will be required to perform large amounts of work as part of ThinkEval. At a minimum, the proposed framework requires that human users manually (a) refine queries/triplets that fail, (b) verify and prune triplets propuced during Automated Triplet Generation.\n- Changing any part of the ThinkEval meta-algorithm (e.g., the relative workload of human versus LLM agents in generating and refining queries) seems like it would significantly change important aspects of the evaluation. Thus it seems difficult to interpret any comparison between results obtained by different groups when using ThinkEval, which seems to limit its applicability. Do the authors see any avenues that would mitigate this difficulty?\n   - E.g., even given the same set of base triplets, the procedure of query generation/refinement involves significant qualitative steps, deciding when responses constitute “authentic triplets” versus misinterpretations. \n- Overlap with existing benchmarks: the proposed benchmark framework has significant overlap with MQuAKE, but the authors include minimal comparison with MQuAke and/or motivation for the creation of a new knowledge leakage framework.\n   - Do the authors have any examples of model editing techniques for which ThinkEval detects knowledge leakage but MQuAKE does not? Does ThinkEval suggest any new facts about the set of editing techniques evaluated compared to evaluation with MQuAKE? At least two of the techniques evaluated (MEMIT and ROME) also seem to be evaluated in the MQuAKE initial manuscript, so the authors could comment on comparisons of those methods without needing to run new experiments.\n   - The authors claim that “Though [MQuAKE is] valuable for probing broader knowledge, it doesn’t consider scenarios where unmodified connected facts logically reconstruct the original fact.”\n   - However, the example on the first page of MQuAKE (British Prime Minister) seems analogous to the Harry Potter example. I.e. the model’s failure in incorrectly responding “Carrie Johnson” when asked “Who is married to the British Prime Minister” seems analogous to answering “Draco Malfoy” as part of the Harry Potter knowledge leakage reasoning chain. I.e. one could consider the fact that MQuAKE reveals that “The Brittish Prime Minister is married to Carrie Johnson” as a scenario where the model [erroneously] reconstructs that “Borris Johnson is the British Prime Minister.” Thus MQuAKE seems very conceptually related to ThinkEval, and it is not clear what benefit the authors propose over MQuAKE. \n\n**Questions:**\n- See my questions about comparisons with MQuAKE in “Weaknesses.”\n- See my questions about the difficulty of comparing ThinkEval results across different research groups. \n- How stable are ThinkEval results to the choice of base-query datasets? The knowledge graph of modern LLMs is so massive that I’m concerned about whether it is feasible using this human-in-the-loop procedure to sample a sufficiently large fraction of the knowledge graph as to be able to reliably compare different editing techniques.\n- I’m confused about the validity of the intuition: “G is designed such that traversing its edges naturally implies relationships between entities.” This doesn’t seem generically true unless “relationships” are constrained to have transitive properties. E.g. the path (Snake, predator, mouse, smaller than, elephant) doesn’t imply any relationship between snake and elephant. Is the idea that the closure of G is a superset of all reasonable relationships, even if it includes vacant/erroneous relationships?", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors claim that they introduce a new framework (ThinkEval), a new editing task (deep editing), and a new evaluation metric (IFR). These claims are correct. I have concerns about the utility of the evaluation procedure that they introduce, but these are not correctness issues per se.", "audience_yn": "Yes", "audience_explanation": "The task identified by the authors of ensuring that edits to LLMs are robust to chains of logical inference is extremely challenging and relevant.", "requested_changes": "- The authors must include some comparison with other editing evaluation frameworks in their experiments. MQuAKE seems natural as it is conceptually related and because KnowGIC is built on MQuAKE’s dataset.\n- The authors must make explicit which parts of their meta-algorithm they intend to be automated with LLMs. Currently, such as “turning base triplets into queries” are left ambiguous as to whether they are intended for human-in-the-loop work versus automation with yet another LLM.\n- The authors need to discuss replicability and stability. To scale ThinkEval, it seems necessary to use a second LLM in many parts of the meta algorithm (query generation, query refinement, query validation, triplet extraction, triplet pruning, etc). It seems very difficult to control for the impact that this secondary LLM will have on the evaluation results. Empirically, how significantly do the results provided by ThinkEval change when different automation workflows are used? Is the relative ranking of editing methods (Figure 5) or counterintuitive phenomena (“rise-then-fall” in Figure 4) preserved under different automation workflows? How do the authors propose that the community compare different results obtained using ThinkEval as these workflows change?", "broader_impact_concerns": "I am unconvinced of the stability and replicability of the proposed evaluation framework. Thus, if this framework were deployed, I am concerned that it could yield misleading or unreplicable results to practitioners. Given the heightened current interest in LLMs and editing methods, this seems like a substantial potential for harm.", "additional_comments": "I find it misleading to refer to ThinkEval as an “architecture” in Fig. 2. ThinkEval is a framework or meta-algorithm, not a parameterized model.\n\nTypos:\nIn the opening sentence of the abstract, writing “to enable…” feels grammatically awkward. I might suggest “as they enable…”\nPage 5, grammatical error: “For users, choosing the right model-editing technique for their model is vital by evaluating…”", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fdmylc4KHQ", "reviewer": "Reviewer_jLqF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on Model Editing on LLMs - specifically, they attempt to study the phenomena of indirect knowledge leakage (also known as the \"ripple effect\") in case of such models. The basic idea is - once an LLM is edited with new information, can it reflect that new information in response to queries that have only indirect relation to such information? For that, they present ThinkEval - a novel framework to systematically evaluate such model editing techniques. This framework involves a detailed approach of building a knowledge graph using entity-relationship triplets, identifying sequences in such knowledge graphs, which are developed using Chain-of-thought reasoning from LLMs followed by \"fact segmentation\". They introduce Deep Editing to quantify indirect knowledge leakage through multi-step reasoning, and propose a metric called Indirect Fact Recovery (IFR) tailored to this setting. They use ThinkEval to develop the KnowGIC dataset of 1,406 multi-step sequential-prompting samples, which they use for evaluating combinations of LLMs and model editing techniques using the deep editing evaluation", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper systematically describes the components of ThinkEval to systematically explain the knowledge graph structure, and present clear algorithms for query evaluation and refinement, automated triplet generation and developing chains, and quantify the amount of knowledge leakage by comparing the knowledge graph structure before and after model editing. They illustrate these concept with many illustrative examples from various domains. Experiments on LLM+editing technique comparison using IFR are very clearly explained.", "audience_yn": "Yes", "audience_explanation": "LLM evaluation is a super-active area of research. Model editing is one of the few new topics in this field, and ThinkEval provides a systematic framework to evaluate model editing techniques by quantifying the indirect propagation of knowledge. Additionally, they also contribute a benchmark dataset for this purpose. Hence, the LLM researchers in the TMLR audience will definitely be interested. In fact, researchers working on Knowledge Modeling will be interested to understand the entity-relation knowledge graph and associated algorithms presented here,", "requested_changes": "I am generally satisfied with the paper, and have no changes to suggest. However, most of the examples illustrated in the paper are related to popular culture. Sometimes, query generation and knowledge propagation may be harder in scientific domains. I would love to see the how model editing works, and how ThinkEval can evaluate its performance on scientific fields like Physics, or even Machine Learning.", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "rTJt99dE6e", "reviewer": "Reviewer_jMMQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Paper Summary\n\nThis paper introduces ThinkEval, a framework for systematically evaluating how well model-editing techniques update and preserve knowledge in large language models.\nThe authors identify an important problem indirect fact leakage, where “edited-out” knowledge can still be inferred through multi-step reasoning.\nThinkEval's core contributions are:\n1. Deep Editing: A new evaluation standard that requires an edit to be successful not just for direct queries, but also against inference through chains of related facts.\n2. Indirect Fact Recovery (IFR): A new metric that quantifies how much of the original fact can be recovered via multi-step reasoning after an edit. \n3. The KnowGIC Benchmark: A dataset of 1406 multi-step reasoning paths built to test editing techniques under the deep editing setting.\nThe study provides detailed experiments across five state-of-the-art editing methods (ROME, MEMIT, PRUNE, RECT, AlphaEdit) and three LLMs (Llama3-8B, Qwen2.5-7B, GPT2-XL), analyzing the trade-offs between IFR, preservation, and efficacy.\n\nSummary Of Strengths\n1. Novel and Important Problem Formulation: The paper successfully identifies and formalizes a significant, overlooked issue in model editing: indirect fact recovery via multi-step reasoning. The \"Harry Potter\" case study is a compelling and clear illustration of the problem.\n2. Comprehensive Framework (ThinkEval): ThinkEval is a well-designed, multi-component framework that integrates CoT reasoning, automated triplet generation, human-in-the-loop validation, and graph synthesis. Its cyclical, iterative nature is a strength for building robust, model-specific evaluations.\n3. Innovative \"Deep Editing\" Paradigm and IFR Metric: The concepts of \"deep editing\" and the IFR metric are major contributions. They provide a principled way to measure editing success beyond direct query performance, addressing a clear gap in existing evaluation methodologies.\n\nSummary Of Weaknesses\n1. Scalability and Computational Cost: The ThinkEval process involves multiple rounds of CoT prompting, human validation, and graph synthesis, which could become computationally expensive. However, the paper provides no quantitative data on runtime, GPU hours, or human effort.\n2. Model-Specific Knowledge Gaps: Appendix A highlights a significant issue: the KnowGIC benchmark, built using Llama3-8B, contains facts that Qwen2.5-7B does not know. This raises questions about the fairness and universality of the benchmark when evaluating different models and suggests that a perfectly model-agnostic benchmark might be infeasible, though the authors rightly frame ThinkEval as a model-specific tool.\n3. Limited Scope of Editing Techniques: The evaluation focuses exclusively on parameter-modifying techniques. Including and contrasting with prominent parameter-preserving methods (e.g., SERAC, IKE) would have provided a more complete picture of the editing landscape and how different architectural choices impact indirect leakage.\n4. Clarity on IFR Calculation: The IFR formula (Equation 6) is complex. While the intuition is clear, a more detailed explanation or a simplified illustrative example in the main text would improve accessibility for a broader audience. The weighting by 1/√n_i is justified but could be further motivated.\n5. Lack of Discussion on Generalization Beyond Factual Edits: The paper primarily focuses on factual edits involving entity-level knowledge. It remains unclear whether ThinkEval can generalize to other types of edits, such as commonsense reasoning, procedural knowledge, or logical rules, which limits the framework’s perceived versatility.\n6. Human Evaluation Bias and Consistency: Given that human reviewers are involved in validating and pruning reasoning chains, the paper lacks an analysis of inter-annotator agreement or measures to ensure consistency. Potential human bias or subjectivity could influence the benchmark’s reliability.\n7. Absence of Ethical and Responsible Editing Discussion: The paper does not address the ethical implications of model editing, such as the potential misuse of editing techniques to distort factual knowledge or introduce bias. A short discussion on responsible use and mitigation strategies would enhance the paper’s completeness and societal relevance.\n8. Reproducibility and Transparency Limitations: The paper provides limited details on experimental reproducibility. Key implementation factors such as model versions, prompt templates, or hyperparameter settings are not described in sufficient depth, which may hinder replication and fair comparison by other researchers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper presents a well-structured framework, ThinkEval, which is validated through extensive experiments involving five prominent model-editing techniques (ROME, MEMIT, PRUNE, RECT, and AlphaEdit) and three large language models. The proposed Indirect Fact Recovery metric and Preservation measure are mathematically defined and consistently applied throughout the evaluation. The results are well-illustrated with both quantitative tables and qualitative case studies, which effectively substantiate the authors’ core claim that existing model-editing techniques fail to suppress indirect knowledge leakage.", "audience_yn": "Yes", "audience_explanation": "The paper introduces ThinkEval, a novel framework for assessing indirect knowledge leakage—a problem that is both timely and underexplored within the LLM community. By defining the deep editing paradigm and proposing the Indirect Fact Recovery metric, the study contributes a new and practical perspective on how to evaluate editing consistency and knowledge preservation beyond direct factual correctness. Additionally, the construction of the KnowGIC benchmark, which incorporates multi-step reasoning chains, extends current evaluation methodologies in a way that bridges causal reasoning and factual consistency—topics that are central to TMLR’s readership.", "requested_changes": "1. Automation for Scalability(Critical): Investigate ways to reduce human dependency in the ThinkEval loop, perhaps by using a more powerful \"judge\" LLM for initial triplet validation and pruning, reserving human effort for final verification. A study on the trade-off between automation and accuracy would be valuable.\n2. Benchmark Robustness and Model-Specific Bias(Critical): To address model-specific knowledge discrepancies (as observed between Llama3-8B and Qwen2.5-7B), consider constructing a core benchmark subset verified to be known by multiple base models. Alternatively, introduce a calibration or normalization step that adjusts IFR and Preservation scores based on each model’s pre-edit factual accuracy. This would make comparisons across models more interpretable and fair.\n3. Discussion on Mitigation Strategies(Recommended): The paper excellently diagnoses the problem. A discussion or preliminary ideas on how future editing techniques could be designed to achieve low IFR and high Preservation simultaneously would be a valuable addition, positioning ThinkEval not just as an evaluator but as a guide for future development.\n4. Broaden Generalization Discussion(Recommended): To increase the framework’s perceived versatility, the authors should discuss whether ThinkEval can be generalized beyond factual edits, such as those involving commonsense reasoning, procedural knowledge, or logical rules. Addressing how the framework can be applied to other knowledge types would demonstrate its broader applicability and strengthen its potential use in various real-world scenarios, including those requiring higher-level cognitive reasoning.\n5. Human Consistency and Ethical Considerations(Recommended): The paper would benefit from a more detailed discussion of human reviewer bias and consistency, especially given the role of human validation in the ThinkEval process. The authors should explore inter-annotator agreement and how human subjectivity might influence the evaluation outcomes. Additionally, a brief section addressing the ethical implications of model editing—such as the risk of misuse for manipulating knowledge—would add value. Including responsible usage guidelines or mitigation strategies would align the paper with growing concerns over ethical AI development.\n6. (Recommended)To strengthen the theoretical foundation and situate this work more clearly within the literature on editing robustness and consistency preservation, the authors are encouraged to cite related studies such as “Mitigating Forgetting in Adapting Pre-trained Language Models to Text Processing Tasks via Consistency Alignment” and “Shaping Pre-trained Language Models for Task-specific Embedding Generation via Consistency Calibration.” These works address the challenges of catastrophic forgetting and representation inconsistency during model adaptation, which are conceptually aligned with ThinkEval’s focus on maintaining knowledge preservation and reducing IFR after model editing. In addition, the authors are also encouraged to reference “SpecialtyScribe: Enhancing SOAP note Scribing for Medical Specialties using LLMs,” which demonstrates the application of LLMs in domain-specific knowledge editing and contextual reasoning. Including this citation would further emphasize the practical relevance of ThinkEval for evaluating editing robustness in specialized, high-stakes domains.", "broader_impact_concerns": "While the paper makes a valuable methodological contribution, there are several broader impact concerns that warrant deeper discussion. Specifically, model editing—the central focus of this work—has the potential for both beneficial and harmful applications. The ability to systematically modify knowledge in large language models could be misused to distort factual information, reinforce biases, or erase historically or socially significant content. Although the paper focuses on evaluating editing accuracy and preservation, it does not sufficiently address the ethical responsibilities and safeguards that should accompany such powerful interventions.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)