KnowledgeSmith: Uncovering Knowledge Updating in LLMs with Model Editing and Unlearning
Authors: Yinyi Luo
Zhexian Zhou
Hao Chen
Kai Qiu
Marios Savvides
Sharon Li
Jindong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Editing
Machine Unlearning
Knowledge Graph
Primary area: foundation or frontier models, including LLMs

Abstract: Knowledge editing and machine unlearning are two popular approaches for large language models (LLMs) to stay up-to-date. However, the knowledge updating mechanism of LLMs remains largely unexplored due to insufficient, isolated, and small-scale evaluation. For instance, are LLMs similar to humans in modifying certain knowledge? What differs editing and unlearning as training data increases? This paper proposes KnowledgeSmith, a unified framework to systematically understand the updating mechanism of LLMs. We first cast editing and unlearning as instances of one constrained optimization problem. Then, we propose an automatic dataset generator that provides structured interventions across multiple graph levels and data scales, enabling controlled studies of how different modification strategies propagate through model knowledge. Extensive experiments demonstrate nuanced insights over knowledge propagation, plasticity scaling, consistency, and robustness. For instance, our results show that LLMs do not exhibit similar updating as humans for different levels of knowledge, and there exists consistency-capacity trade-off. We hope our findings can offer suggestions to the design of more reliable and scalable strategies. Code: https://github.com/AIFrontierLab/KnowledgeSmith

Published as a conference paper at ICLR 2026
KNOWLEDGESMITH: UNCOVERING KNOWLEDGE UPDATING IN LLMS WITH MODEL EDITING AND UN- LEARNING
Yinyi Luo Zhexian Zhou', Hao Chen', Kai Qiu', Marios Savvides', Sharon Li,, Jindong Wanga1
Carnegie Mellon University William & Mary UUniversity of Iisconsin-Madison
ABSTRACT
Knowledge editing and machine unlearning are two popular approaches for large language models (LLMs) stay up-to-date. However, the knowledge updating mechanism of LLMs remains largely unexplored due to insufficient, isolated, and small-scale evaluation. For instance, are LLMs similar to humans in modify- ing certain knowledge? What differs editing and unlearning as training data in- creases? This paper proposes KnowledgeSmith, unified framework system- atically understand the updating mechanism of LLMs. We first cast editing and unlearning as instances of one constrained optimization problem. Then, pro- pose automatic dataset generator that provides structured interventions across multiple graph levels and data scales, enabling controlled studies of how differ- ent modification strategies propagate through model knowledge. Extensive ex- periments demonstrate nuanced insights over knowledge propagation, plasticity scaling, consistency, and robustness. For instance, our results show that LLMs do not exhibit similar updating as humans for different levels of knowledge, and there exists omsistency-capacee fer suggestions to the design of more reliable and scalable strategies. Code: hattp:://gtthut www.com/m.a.... Lab Knwwledge
trade-off. We hope our findings of-
INTRODUCTION
Human knowledge not stored as isolated facts but as a vast, interconnected web (Liu et al. 2024). From early encyclopedias to modern knowledge graphs, represent knowledge structured lations (Yang et al., 2025): concepts (nodes) linked by semantic or causal connections (edges). This networked organization enables humans reason flexibly (Mark et al. 2020), update beliefs (Paulheim 2016) when new evidence arises, and propagate changes across related domains (Flouris) al.) 2008). For instance, when scientists revised the classification of Pluto from planet dwarf planet, the update did not merely alter one fact but cascaded through textbooks, curricula, and related scientific explanations.
Do Large language models (LLMs) exhibit similar properties? Zhang et al. 2024) showed that they store and retrieve information scale, generating answers that span diverse domains; Yet, unlike human knowledge graphs, the internal structure of LLM knowledge remains opaque (Zhang et al.) 2023). Fine-tuning can overwrite large swaths of parameters but resource-intensive and imprecise [Balne et al. 2024 Gekhman et al. 2024), often introducing instability hallucinations (Khan (ct al., 2025] Ovadia et al. 2024). Researchers have recently shifted attention toward knowledge editing (Wei et al.J 2024 Markowitz et al. 2025 Wang et al. (024) and unlearning (Yao et al. 2024 Pawelczyk et al. 2024) Hong et al. 2024). where editing offers targeted modifications and unlearning aims broadly remove specific information. Both are valuable, yet they are typically studied in isolation and without grounding in structured knowledge representations.
How to understand the knowledge updating mechanism in LLMs? Recent efforts show that editing techniques can be adapted for forgetting by redirecting or suppressing knowledge representations (Li al. 2025b) Jung al. 2025), while unlearning methods sometimes resemble -oarse-gaained editing at the dataset level (Guo et al. 2019). Other works investigate continual or compositional
"ook done with Prof. Jindong Wang at William & Mary.
1 Contactt: yi ny @andrew ew. edd; Corresponding author: jdw@wm wwm.eweedd

---

Published as a conference paper at ICLR 2026
Subjects Knowledge Graph
Biology (e.g. DNA double helix)
[Root]
Economics (egg, Philips curve)
[Intermediate] How did double helix change the molecular biology?
History (e-g, Civil War)
Physics (eg, Theory of Gravity)
[Leaf] When was double helix discovered?
Automatic Benchmark Generation
-Eatiys disevereed
-Yeul -Yeu2>
diseevery sEaitto"
1994
1955 D. 1956
Entitt-relatiio Selection
Templatc-bosed Question
Mulliplo-choice Generation
External Validation
Generation
KnowledgeSmith
arg min Cookk Q") Apre (f:; Q Ace
When the DNA deoble helis discoverad?
L.M
When the DMA doalee hellx disomenec
Upder
Knowledge Editing
Machine Unlearning
Figure 1: KnowledgeSmithh pipeline. Starting from static KG, we generate dynamic probes at root, intermediate, and leaf levels, enabling evaluation of direct and propagated effects.
settings, where localized edits may interfere with broader forgetting objectives vice versa (Gupta iet al. 2024] Chen et al. (0024). A parallel strand examines the tension between specificity and gen- eralization: editing often prioritizes precision but risks side effects, whereas unlearning emphasizes removal but may fail to incorporate new or corrected knowledge (Yao et al., 2023a).
Despite recent progress, there are still three critical challenges. First, most evaluations target isolated facts. neglecting the structured and interconnected nature of real-world knowledge (Thede et al. 2025 For example, we update the fact that "Lyon is the capital of France" instead of Paris, coherent system should also adjust related knowledge such as "the Eiffel Tower is located in France's capital," which otherwise becomes inconsistent. Second, the role of data scale in editing unlearning remains unclear, with small data often sufficing for edits but not for forgetting(Zhong (et al. 2023 Meng et al. 2022a). Third, there is no unified framework to jointly understand editing and un unlearning, leaving their trade-off's in propagation, stability, and generalization unclear.
In this paper, we introduce KnowledgeSmith (Figure [.. a unified framework to understand the knowledge updating mechanisms in LLMs Theoretically. learning as complementary forms of constrained optimization. Empirically, building on the intuition that human knowledge is naturally structured as knowledge graphs (KGs), framework can tomatically transform any existing KG-related dataset into a benchmark for knowledge intervention evaluation, enabling systematic and scalable assessment without the need for hand-crafted test sets. For instance, more insights can be gained through interventions across hierarchical levels (root, in- termediate, leaf) and data scales (from single instances to millions). Then, we conduct an extensive evaluation of editing and unlearning on different LLM families to explore knowledge propagation, scaling laws, representation shifts, and robustness under stress tests. Our key findings are:
framework casts editing and un-
1. Propagation Asymmetry and Plasticity Limits: Editing can oversprread(uiintenionnnl altering related nodes), especially at higher nodes, while unlearning mostly under- spreads(forgetting failing to propagate beyond the target node). Hierarchical branch structure imposes intrinsic ceilings on update effectiveness, with higher or more central nodes limiting achievable knowledge modifications 55.21 $.22.2..
2. Consistency-Capacity Tradeoff and jubject-Depender Update: Increasing data can trig- ger consistency collapse, where local updates contradict other knowledge; editing prioritizes local enforcement, unlearning preserves broader consistency. Some domains, like history, resist updates more than others, highlighting the need for subject-aware evaluation ($...3 5.2.44.
Other approaches can also update knowledge in L.LMs; we focus on editing and unlearning in this paper.
2

---

Published as a conference paper at ICLR 2026
3. Model Robustness:
Editing improves in-domain accuracy but harms OOD and adversarial stability, while unlearning preserves global robustness at the cost of weaker local (
4. Method-level Trade-offs: Editing balances integration and preservation with strong low-data efficiency, unlearning is conservative but stable, while LoRA fine-tuning is unstable and prone to drift, making it unreliable for continual updates ($5.4).
5. Unified Failure Modes and Stress Testing: By observing model behavior on open-ended questions. we identify six main failure modes and find that unlearning preserves general task integrity better, whereas editing is more aggressive but effective in low-data regimes ($5.5)
Contributions. (I) We introduce KnowledgeSmith as a unified framework to understand knowledge updating in LLMs with editing and unlearning. (2) We present automatic data generation pipeline for LLM evaluation with scalable KG-structured interventions.
(3) Our experiments demonstrate several insightful findings towards LLM knowledge updating that could inspire future research.
2 RELATED WORK
Other than fine-tuning which is expensive and requires large amount of training data, knowledge editing and machine unlearning are two popular and effective approaches to update LLMs' knowl- edge.
Knowledge editing modifies L.MMs' internal parameters update its predictions on specific factual associations while ideally preserving unrelated knowledge Yao et al. 2023b] Cao et al, Sinitsin et al. (020). Existing approaches include gradient-based fine-tuning (Sinitsin et al,
2021
2020
Zhu et al. (020), localized weight modifications such ROME (Meng et al. (022a), MEMIT methods that externalize edits (Mitceell al.) (022). However, most prior evaluations restricted to small benchmarks (Levy et al. Meng et al. 2022a) and do not examine how edits propagate through
MMeng et al.) 2022b). and SERAC (Mitchell et al.) 2021]. and memory-augment
structured knowledge dependencies.
On the other hand, motivated by ethical, legal, or safety considerations, machine unlearning seeks selectively erase information linked dataset, (Izzo et al., [021]; Thudi et al. [022]] Xu al.) 2025). Methods include retraining- based approaches (Ginart et al. 2019). negative-gradien fine- tuning (Thudi et al. 2022), regularization- based constraints (Golatkar al. 2020), and approximate removal via influence functions or Fisher-weighted updates (Guo et al. 2019 Baumhauer et al.) 2022]. Yet, unlearning has largely been studied isolation from editing, without systematic com- parisons or evaluation in structured knowledge contexts.
In short, existing research highlights strong methodological advances but leaves two key gaps: (1) editing and unlcarning are often treated disjoint problems despite their conceptual overlap, and (2) evaluations rely on narrow datasets that fail to capture scaling behavior or structured propagation effects. Our work tries to establish unified view of them and present an extensive analysis towards understanding LLM knowledge updating.
KNOWLEDGESMITH
In this section, we propose KnowledgeSmith, unified framework to view editing and unlearning as complementary interventions.
3.1 PROBLEM DEFINITION
Let So denote a language model parameterized by 0, defining a conditional distribution po x)) over output y given input r. We study targeted interventions that modify or remove specific knowl- edge while preserving the model's general behavior.
An update request is given by an item e (e.g., a factual triple, a prompt-response pair, or a small dataset), optionally accompanied by a scope that defines locality or related probes. For example, e is the fact "Paris the capital of France", c could include all prompts asking about European capitals such as "What is the capital of France?" or "Name the capital of European countries" while excluding unrelated prompts like "Who is the president of the United States?", ensuring that only related knowledge is affected while leaving unrelated knowledge untouched. Applying update
3

---

Published as a conference paper at ICLR 2026
operator T (e.g., editing or unlearning) yields updated parameters:
6' = T(O;e,
A =
where A is the parameter update.
The objective is therefore update the targeted knowledge while preserving unrelated knowledge. To facilitate analysis, we define two prohe sets: (1) Positive probes O+ are inputs where the model's predictions should change; and (2) Preservation probes 2 are inputs where predictions should remain unchanged. Formally, for an input x, denote p-(- x) and p (- x) as the output distribution of the model before and after KnowledgeSmith intervention, respectively, we have:
d(po (+ x). Aurgel (- x)) S snn
Vo º+
d(pe (- p x)) E.
where .): is a divergence or distance measure between distributions (e.g.. KL divergence, cross- entropy, lz distance over logits), Quarger(' x) is the desired post-intervention distribution on positive probes, the constant 7f specifies a tolerance threshold for successful edits, reflecting that editing algorithms may only approximate the target distribution rather than match it exactly, and is a stability threshold controlling how much drift is allowed on Q
3.2 A UNIFIED FRAMEWORK FOR ANALYZING EDITING AND UNLEARNING
While Equation 2 formalizes the objectives using tolerance thresholds n¹ and E. in practice we implement these constraints by relaxing them into loss terms over probes. Specifically, Lusk (6'; Q+) penalizes deviations from the target distribution on Q+ Lpres(';; (8'; Q penalizes drift on Q and R(O', 0) regularizes the overall update. Thus, both model editing and unlearning can be cast as a constrained optimization over model parameters:
= arg min (oak 2-) + Appees Lpes ( + Aseg R(O', 0).
(3)
6'
where Crask enforces the desired behavior on Qt Epres penalizes drift Q and R(0', 0 regular- 1/ (gg 2004). Fisher norm (Gu et al. 2012), or others (Hu et al. 2022 )
izes the update (e.g., ||
Editing as targeted aligmment. Knowledge editing can be viewed as minimizing Caac toward distribution Quargt that encodes corrected knowledge. For example, ROME (Meng et al., 2022a) and MEMIT (Meng et al. 2022b) locate and modify specific MLP weights to enforce new facts, while MEND (itchell al. 2 trains auxiliary
to redirect predictions on edited queries. Other approaches apply gradient-basee updates on Q+ while regularizing drift, such as GRACE (Hartvigsen al. 2023). Even parameter-efficient methods like LoRA-based editing (Hu et al.
2022 Zheng et al. 2023) fit this form, with R(0', .0) enforcing low-rank adaptation.
Unlearning as neutral alignment. Unlearning corresponds to the same objective but with Qarget chosen as a neutral distribution Queutral that suppresses unwanted associations. This cap- tures approaches that erase knowledge through gradient descent (Thudi et al.) (022), influence- bunction-based forgetting (Golatkar et al.) 22200] Guo et al. (211)). or certified removal in convex models (Ginart et al., 2019). Recent work on unlcarning in deep networks (Jagielski et al., (002)) also fits: their objectives penalize predictive alignment with sensitive data while constraining per- formance on Q exactly corresponding to the Epee and R(O', 0) terms above.
A unifying lens. In this view, the distinction between editing and unlearning reduces to the choice of Iurrett Editing: Qtarget encodes a factual correction (e.g., "Paris is the capital of Germany"). Un- learning: Qtaget is neutral, erasing prior associations (e.g.. "Paris is the capital of [MASK]"). This framework subsumes methods across the spectrum: localized weight modifications (Meng et al., 2022b a), memory- based editors (Mitchell et al., 2021). parfmeter-efffcientt adaptations (Hu et al, 2022 Zheng et al.] 2023), influence- forgetting (Golatkar et al.) 2020), and certified removal (Giartt et al. 2019). Despite methodological differences, all can be interpreted solving the same constrained optimization problem with different instantiations of Lunks Lpess and R(6', 0).
Our formulation provides principled and generalized lens for analyzing parameter modifications in LLMs, enabling fair comparison of editing and unlearning on their trade-offs in plasticity, stability, and generalization. However, to rigorously measure these effects in practice, we need benchmarks
4

---

Published as a conference paper at ICLR 2026
that capture hierarchical dependencies, e.g., local versus global changes, and multilevel propaga- tion of updates, which are largely missing from existing datasets. This motivates our automated benchmark construction in the following.
CONSTRUCTING EVALUATION BENCHMARK
Existing benchmarks Meng et al. 2022a Levy et al. 2017) for knowledge intervention evaluation suffer from two major limitations. First, they are largely static, testing only isolated facts without accounting for how updates might affect related knowledge. Second, they fail to capture dependen- cies across facts, which are crucial for understanding how changes propagate through the model and for revealing trade-off's between editing and unlearning.
We leverage knowledge graphs (KGs) to address these gaps, which dynamically encode hierarchical and relational dependencies among facts. Anchoring probes in curated KG enables us to gener- ate both local edits and their downstream consequences, transforming a single KG into dynamic benchmark. Specifically, by targeting root, intermediate, and leaf nodes, our framework systemat- ically tests how interventions propagate across multiple levels of dependency, thus providing a rig- orous way to evaluate whether models can coherently update, forget, or preserve knowledge while maintaining global consistency. Concretely speaking, our data generation method can automatically transform any existing knowledge-related benchmarks such as MMLU (Hendrycks et al., 2021) into new ones, providing domain coverage and standardized multiple-choice QA format for easy eval- uation. Our pipeline consists of three stages (Figure I). ensuring both quality and flcxibility:
1. Entity-Relation Selection:
We begin by prompting GPT-4o generate KG where enti- ties and relations are organized hierarchically. The model is then asked to categorize nodes into three levels: rool (broad, domain-level concepts), intermediate (mid-level categories or subtopies), and leaf (specific entities or instances). Sampling nodes from all three categories preserves the KG's hierarchical structure, ensuring evaluation goes beyond isolated facts to capture how edits or deletions propagate across different levels of related knowledge.
2. Template- -Base Question Generation: Multiple question forms are generated for each triple, varying in directness and context. All templates are manually verified for grammaticality and factual alignment, preserving unambiguous mapping back to the KG. Six categories of probes are constructed (direct, reverse, conflict, multi-hop, comparison and contextual), each tied to different aspect of model behavior under intervention.
3. Multiple-Choice Construction: Each probe is cast as a four-choice QA item, consistent with the MMI.U-inspired format, ensuring that evaluation reflects true knowledge states rather than guesswork. Entity substitution and paraphrasing yield over one million samples across do- mains. All items are validated against the KG, with manual spot checks for quality assurance.
Connection to KG-Based Evaluation. Our generation pipeline is organized around two comple- mentary families of probes: (1) Positive probes Q+, which directly test the edited or redirected knowledge, including its hierarchical propagation across root, intermediate, and leaf nodes. (2) Preservation probes º, which ensure that unrelated or out-of-scope knowledge remains intact, guarding against collateral damage.
To operationalize these two families, we instantiate six probe types. Direct probes (Q+) test whether the target fact itself recalled or updated different hierarchical levels. Reverse probes (Q+) ex- amine whether knowledge updates preserve relation directionality. Conflict probes (Q*1Q expose residual beliefs and adversarial robustness by checking for contradictions after intervention. Multi- hop probes (Q+) evaluate whether interventions correctly propagate through chained relations in the KG. Comparison probes (Q+) assess whether the updated knowledge is consistently preferred when contrasted with alternatives distractors. Finally, Contextual probes (Q-) test whether unrelated in-domain or OOD knowledge remains preserved in naturalistic settings. This design aligns directly with experimental analyses: By explicitly embedding these probe types into the KG's hierar- chical structure, the benchmark enables analyses that go beyond isolated fact checking, revealing whether interventions cascade consistently across levels of related knowledge.
Generated Benchmark Dataset. Our method allows flexible data generation across domains. In this paper, instantiate the benchmark in four domains: economics, physics, history, and biology.
5

---

Published as a conference paper at ICLR 2026
We restricted our evaluation to four domains balance diversity and feasibility Each domain yields paired pre-edit and post-edit datasets that preserve entities but differ in factual content. Probes span root, intermediate, and leaf nodes, with conflict, propagation, comparative, and reverse variants, and include multiple paraphrased realizations. For each branch within every domain, we generate 10, 000 samples each for editing and unlearning, plus 100 evaluation probe sets, leading 360, 000 training samples in total. This design creates a benchmark that is both large-scale and structurally sensitive, allowing systematic evaluation of edits and unlearning not just the point of intervention but throughout the knowledge hierarchy. Dataset examples are in Appendix A
EXPERIMENTS
5.1 SETUP
Models. Our evaluation covers families of LLMs with 1B 123B parameters, leading to a total of 13 models: LLaMA-3 (1B, 3B, 8B, 70B) (Meta 2024). Qwen-3 (1.7B, 14B, 32B) (Team] 2025b),
QwQ-32B (Team 2025a), Mistral (24B, 123B) (Jiang et al. 2023). Gemma (2B, 7B) (Team) 2024).
and DeepSeek-R1 R -0520 15588-Qee33-8h (DeepSeek-Ai) 2022). This broad coverage enables us to study
whether scaling behaviors and editing/umlearning performance generalize across architectures.
Implementation Details. We adopted AlphaEdit (Fang al. 2025) and ReLearn (Xu et al. 2025 AlphaEdit is a state-of-the-an editor that has been shown to outperform prior methods such as MEMIT et al.) (2022b) and ROME(Meng et al.) (2022a) in editing tasks, while ReLearn repre- sents a leading approach to unlearning. Importantly, our framework is method- -agnosic and directly extensible other baselines, making it straightffrward integrate additional methods. Unlike tra- ditional unlearning approaches where the retain set corresponds to the original knowledge, in our redirection-base based setup the retain set is defined as the post-updated knowledge, ensuring that the model preserves the rewritten fact rather than reverting to its prior belief. This redirection-base formulation aligns better with real-world scenarios where knowledge updated rather than erased. Editing and unlearning were applied separately to leaf, intermediate, and root nodes of the knowl- edge graph, with training data sizes ranging from 10, 000 samples. This setup allowed us to systematically analyze the effect of both hierarchy depth and data scale on the success of editing and unlearning. For evaluation, since each knowledge probing question is multiple-choice, we re- port accuracy the proportion of questions for which the model selects the correct choice. This metric directly reflects the model's correctness in retrieving or updating the intended knowledge.
5.2
COMPARATIVE ANALYSIS OF EDITING AND UNLEARNING
5.2.1
PROPAGATION ASYMMETRY: OVER- VS. UNDER-SPREADING
Human learners expect hierarchical consis- tency:
updating a root concept should cas- cade to its descendants, while modifying leaf should remain localized. We evaluate this in LLMs by applying editing unlearning at three hierarchy levels (root, intermediate, leaf) and measuring performance both targeted and structurally related nodes. We quantify these effects using direct vs. multi-hop accu-
Number exampler
Number of (Editing
(Edking)
(Unlearmin))
(Urleemingg)
Figure 2: Propagation asymmetry metrics.
racy (Figure 2 as a proxy for propagation metrics: the Collateral Change Ratio (CCR) captures over-spreading for editing, and the Residual Retention (RR) captures under-spreading for unlearning (For the complete definitions of CCR and RR, see Appendix B).
Our results reveal clear asymmetry: editing tends to over-spread, unintentionally altering related nodes, especially in lower hierarchy levels, whereas unlearning often under-spreads, failing to propagate forgetting beyond the target. These simple, interpretable metries allow us to visualize propagation behavior across hierarchical branches.
"These subjects span both STEM and humanities, offering representative testhed. Our pipeline is directly extensible to other domains such as law and medicine. We also conduct experiments on other methods with similar performance. Reported Appendix

---

Published as a conference paper at ICLR 2026
21.3
23.0
22.3
ma--5 -
30
mon-rey
suld
Intand)
LL
23.5
21.0
root
20
23.5
Interedats
25.0
Number of training examples
Number of training examples
Number training examples
Number of training examples
(a) Scaling (editing)
(b) Scaling (unlearn)
(c) Propagation Limit
(d) Tradeoff
Figure 3: Plasticity scaling of the LLaMA3 family under (a) editing and (b) unlearning. (c) Propa-
gation limits across three branches. (d) Consistency capacity tradeoff.
5.2.2 PLASTICITY SCALING AND BRANCH-DEPENDENT LIMITS
Plasticity captures how readily a model can update knowledge response to limited training data, balancing the optimization of Lask on positive probes Q against preservation constraints Epes on º We extend this notion to plasticity scaling, examining systematically how model size, data scale, and hierarchical branch jointly influence the effectiveness of editing and unlearning.
Our main observations are as follows. First, as shown in Figures Ba and 3b smaller models exhibit higher immediate plasticity, rapidly adapting to few-shot interventions and achieving strong in- domain performance on - but their changes are often unstable, leading to degraded preservation on Q Larger models require more data to register updates, reflecting lower short-term plas- ticity, yet once modified they maintain stronger out-of-domain consistency, indicating more reliable preservation. Second, branch-dependent upper bounds. As shown in Figure [3c, different hier- archical branches exhibit distinct ceilings for achievable accuracy. Root-level edits/unlearning face lower ceiling due to structural complexity and the need for coherent propagation across descen- dants. Intermediate-level branches achieve moderate ceilings. Leaf-level edits/unlearning can reach near-perfect in-domain accuracy with fewer examples, reflecting minimal propagation constraints. This reveals the effectiveness of updates is not uniform across the hierarchy: higher or more central nodes constrain achievable plasticity, while lower nodes allow maximal update with limited data.
5.2.3 CONSISTENCY-CAPACOO TRADE-OFF
Most prior work (Zhong et al. 2023 Park et al. 2025 Shi et al. 2024 Li et al. 2025a primarily
assess whether the target fact is updated successfully, without probing inverse relations. To our knowledge, no prior work explicitly quantifies this type of cross-relation or hierarchical consistency. In this work, we define consistency as the model's ability to maintain logical coherence across related knowledge after an intervention. Specifically, we test consistency by probing both the direct relation (e.g.. "Paris is the capital of France") and the inverse or complementary relation (e.g.. "France has capital Paris"), as well as across hierarchical semantically related branches. A consistent update should correctly modify the target knowledge while preserving these related facts.
We uncover once data scale surpasses the model capacity. We term this the cewiiittecy-caaaccii trade-off. observed both in relation-inverse relation pairs (e.g., capital-of vs. has- capital) and across hierarchical branches. As shown in Figure 3d direct probes initially respond to interventions but plateau degrade training scale grows, whereas reverse probes remain stably high, indicating preserva- tion of contradictory knowledge. The divergence defines consistency collapse point, occuring earlier in lower branches (intermediate, leaf) than root. Editing typically achieves stronger local updates but triggers earlier global inconsistency; unlearning preserves broader consistency but rarely removes the targeted knowledge completely.
phenomenon: consistency collapses
Table 1:
Similarity scores for each model are independently normalized via a log-min- me--mminmmaa transformation: a small positive offset e added, log10 is ap- plied, and the resulting values are lin- early scaled the [0, 1] range.
Metrie
Setting
10
100
1000
10000
KL.
Edit
Unleam
0.014
0.522
0.392
0.805
0.838
0.883
Unlam
0.286
0.368
0.507
Fisher
Edir
Unleam
01001
0.14
0.438
0.352
0.553
0.781
0.64
0347
0.647
0.919
CKA
Edit
Unizam
0014
0.958
0.80
0.852
0.801
01506
0.144
033%
0.714
Representation and Efficiency. Table T shows the analysis of internal representations via Cen-
tered Kernel Alignment (CKA) (Kormblihh et al.] 2019). KL divergence, L2 distance and Fisher score (Zhang et al., 2022). The results show that unlearning exhibits abrupt phase transitions be- yond critical data scale, while editing induces smoother, localized adjustments (details in Ap-
7

---

Published as a conference paper at ICLR 2026
Scclpy Hittryy Econorric
rost-ed roppuu
mise
10
24
Physis
Ratt
leaful
Ed Ergg -i
Ed Eig cod
bigiing Coro--rrrlllll ACe
Ealting -og--Aa Aiversa Auc
Uritam ng-
Jnkammig Ofginal Acc
Urleaming ng- sod
unlesming Miteror al Acc
Number of training examples
Number of training examples
Number of training examples
Number of training examples
(a) Acc across subjects
(b) Conflict Rate
(c) OOD robustness
(d) Adv. robustness
Figure 4: Robustness evaluation under multiple stress tests. (a) Out-of-distribution (OOD) vs, in- domain accuracy. (b) Adversarial robustness relative original accuracy. (c) easructioo-following
accuracy in free generation, judged by an LLM. (d) Hallucination tendency across interventions.
pendix H.. Computationally, unlearning is faster (e.g.. 00.2 vs ~6h for editing on 1,000) samples on an NVIDIA H100), reflecting its focus on stability over precise enforcement.
Consistency collapse is not only evident in output accuracy but also mirrored in representation dy- namics and computational cost: editing maximizes factual enforcement at the expense of broader consistency and resources, whereas unlearning prioritizes stability and efficiency.
5.2.4 SUBJECT-DEPENDERR KNOWLEDGE UPDATE
At the subject level, Figure 4a reveal that knowledge updating is strongly subject-dependent Among the four subjects (biology, economies, history, and physics), history consistently exhibits the lowest update accuracy, sometimes remaining nearly unchanged even with large numbers of training examples. Other subjects update, contrast, propagate more efficiently. This highlights a critical insight: evaluation benchmarks must account for subject-specife difficulty. Standard datasets (e.g., CounterFact (Meng et al. 2022a). ZsRE (Levy et al. 2017)) treat all domains equiv- alently, but our results indicate that certain knowledge domains, such as history, are significantly more resistant to modification. Consequently, subject-awand evaluation is essential for accurately assessing editing and unlearning performance in LLMs.
5.2.5 CONTRADICTIONS AND CONFLICT RATE
While residual belief (Elidan et al. 2012) is commonly used to evaluate whether interventions suc- ceed in suppressing prior knowledge, it does not capture critical failure mode: the emergence of contradictions. We therefore introduce a complementary metric, conflict rate, which measures the proportion of queries where the model simultaneously supports mutually inconsistent statements af- ter intervention. For instance, a model may assert both "Paris is the capital of Germany" and "Paris the capital of France" under different contexts. Figure 4b shows this metric exposes patterns that residual belief alone cannot: editing often leads to higher conflict in related branches (over- spreading), whereas unlearning tends to leave contradictions unresolved in upstream nodes (under-spreading). By explicitly quantifying such inconsistencies, conflict rate provides fuller view of hidden instabilities and unintended side effects.
5.3 ANALYSIS ON ROBUSTNESS
OOD robustness is tested using MMLU (Hendrycks et al., 2021). In the unified framework, in- domain probes o consist of questions from the same subject (e.g., updating facts about geography using geography questions), reflecting alignment with Oarget- In contrast, out-of-domain (OOD) probes Q are drawn from unrelated subjects (e.g., updating geography facts but measuring perfor- mance on economics, history, or law), testing the model's ability to preserve unrelated knowledge after the intervention. As shown Figure 4cc these objectives often conflict. Unlearning pre- serves strong OOD accuracy (63-82% but yields modest in-domain gains (330%), while editing substantially boosts in-domain accuracy (up to 50-60% in economics) at the cost OOD stability, especially in mid-sized models. Larger models reduce but do not eliminate this trade-off. Increasing training examples improves in-domaín performance until gains plateau, and disciplines vary, with economies generalizing better and history proving more resistant. This trade-off reflects the bal- ance between Luss and Cpres: stronger enforcement on Q+ tends to destabilize preservation on º highlighting the challenge of achieving both local fidelity and global robustness together.
8

---

Published as a conference paper at ICLR 2026
We then measure adversarial robustness by exposing the model misleading deceptive inputs, such as probes combining unrelated concepts (Figure 4d). This assesses whether the optimization constraints maintain stability on preservation probes Q under stress (details in Appendix D.I)
5.4 ANALYSIS ON FINE-TUNING
We further compare editing and unlearning with LoRA fine-tuning on Llama3-8B-Instee late method-level tradeoffs. Figure [6a] shows LoRA yields unstable ID accuracy, sometimes dropping to 12.5% at k- = 1000. Scarce data lead to poor enforcement target updates (Q+) while undermining preservation (Q ). Figure 6b Srrrroobshhoo shows OOD accuracy declining from 63.0% (k 1) to 61.6% (k = 1000), indicating drift risks. Un- learning remains stable around 63%, preserving prior knowledge but limiting target success, Edit- ing combines stability with low-data efficiency, boosting ID accuracy to 25% at k 10 compared to 16.7% for LoRA and unlearning. In summary. editing balances new knowledge integration and preservation, LoRA risks drift, and unlearning is conservative but stable, explaining why we prefer editing/unlcamiing for continual updates.
LOBA
iso-
04.27
01.1
k5thg
Leborsr
12" trainng examples
Namber tranning examples
(a) ID
(b) OOD
Figure 6: LORA, Editing and Unlearning.
5.5 FAILURE MODE AND STRESS TESTING
Existing studies describe enrors such as incomplete for- getting knowledge pollution in a fragmented way, with- out systematically characterizing the underlying mecha- open-ended ques-
Table 2: Percentage (%) of observed
failures in editing and unlearning.
Failure Mode
Editing Unlearning
nisms. Through experiments tion answering. we observed that models fail for differ- ent reasons under editing and unlearning interventions. To capture these patterns, we propose a Unified Fail- ure Mode Taxonomy that organizes observed errors into six categories (examples of each type in Appendix D.2):
Under forgetting (RR) Over-spreading (CCR)
20
35
35
Conflict emergence
30
Knowledge drift
10
instructionffolooiin drop
22
18
Hallucination increase
under-forgetting (RR), over-spreading (CCR), conflict emergence (contradictions between updated and related knowledge), knowledge drift (performance degradation on unrelated tasks), instruction- following drop (reduced ability to follow complex instructions), and hallucination increase.
Stress-testing evaluates the failure modes with open generation tasks, making the model show prac- tical robustness and use gpt-4o to evaluate. Our results show that hallucination (evaluated on Truth- fulQA (Lin et al. 2022)) remains stable, instruction-following (open generation) drops moderately, and CoT reasoning can improve edit generalization but may increase residual knowledge, compli- cating unlearning (details in Appendix C).
Sequential update experiments, reported in Appendix F further illustrate how multiple consecutive edits affect these behaviors and highlight potential cumulative effects residual knowledge.
5.6 THEORETICAL ANALYSIS
Our theoretical perspective connects the observed behaviors to their geometric effects on model representations. Let W E R= denote parameter matrix (e.g., attention or MLP projection). with singular value decomposition W = U.VT. An intervention updates W to W "' The difference between W and W can be decomposed into two interpretable components:
Scaling effects. Changes in singular values ///Σ indicate amplification or attenuation of certain representational directions.
Rotational effects. Differences in subspaces SU, V) vs. span(U', V') reflect reorientation of features while preserving their magnitude.
Editing as local rotation with mild rescaling. As shown in Figure 7a editing primarily induces moderate rescaling of singular values while maintaining high orthogonal similarity between (U, V) and (U", V') across layers. This implies that editing preserves most of the representational geometry, redirecting specific factual directions through controlled rotations. Consequently, editing behaves
9

---

Published as a conference paper at ICLR 2026
(a) Editing
(b) Unlearning
Figure 7: SVD-based geometric analysis of interventions. (a) Editing adjusts knowledge by gently rotating and slightly rescaling the representation space, preserving overall geometry while redi- recting specific directions. (b) Unlearning, in contrast, acts by shrinking certain dimensions more aggressively, reducing the model's 's capacity in those directions rather than rotating them.
like a iiation--lus-scalinng operator: it reallocates emphasis toward new factual associations while retaining global coherence. This explains why editing achieves strong local enforcement but often over-spreads changes to nearby branches (high CCR in (3.2)
Unlearning anisotropic scaling. By contrast, Figure 7b shows that unlearning produces sharper downscaling of singular values with less stable alignment of U, V across layers. This indicates sup- pression of capacity in certain subspaces rather than simple rotation. Thus, unlearning resembles an attenuation operator: it removes the ability to encode certain directions but does not reliably ro- tate them into new ones. This mechanism aligns with the observed under-spreading behavior (high RR in Section 5.2). where forgetting is localized and fails to propagate fully across related nodes.
diferarchy-depende dynamics. Leaf-level interventions concentrate changes in later layers, sup- porting near-pertee near-perfect local adaptation. Root-level interventions require distributed rotations and scal- ings across the network, introducing stricter ceilings on achievable accuracy. Intermediate nodes combine aspects of both. These theoretical patterns mirror our empirical findings on branch- dependent plasticity limits (Section 5.2.2 3.2.2)
5.7 DISCUSSION
Our findings offer several potential directions for future research. (1) Model updating: Updates should employ dynamic, hierarchical control such as level- and relation-aware algorithms. Branch- specific strategies can also improve effectiveness: for leaf nodes, updates can use more data for higher accuracy, while root nodes may require less data. Data size should be carefully calibrated for global consistency. Moreover, models exhibit subject-dependett sensitivity, hence, update methods should account for differences across domains. (2) Evaluation metrics: The conflict rate offers a more nuanced assessment of models, capturing hidden inconsistencies and ensuring that updates improve the model more holistically rather than just for specific tasks. This mirrors human reasoning in the sense that humans also monitor for contradictions and coherence, but the analogy is descriptive rather than mechanistie. (3) Foundation models: Future models could be designed with layer-wise wise or tensor-wise modularity, enabling finer- mner-grained control when applying updates. By building update- friendly architectures, such models would allow interventions to target specific branches layers more effectively, improving both efficiency and consistency knowledge updates.
Our work has several limitations. First, our experiments based four domains due to limited compute budget and could be expanded to more domains and multimodal models. Second, our unified framework does give theoretical bound for propagation and consistency remains open. Third, the analysis is based on recent editing and unlearning approaches, which could be extended other algorithms to gain more insights.
6 CONCLUSION
We introduced KnowledgeSmith to understand the knowledge updating mechanism in LLMs by unifying editing and unlearning. Our experiments highlight fundamental trade-offs, e.g., unlearning prioritizes stability and efficiency but yields modest enforcement, while editing enforces knowledge updates more effectively at the risk destabilization and higher computational cost. We hope our benchmark and analysis shed light future research LLM knowledge updating.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This paper is partially supported by The Commonwealth Cyber Initiative (CCI) program (H-2Q25- 020), William & Mary Faculty Research Award, and Modal Academic Compute Award. The au- thors acknowledge William & Mary Research Computing for providing computational resources and/or technical support that have contributed to the results reported within this paper. URL: http://wwww.w.cch/i/te.
ETHICAL AND REPRODUCIBILITY STATEMENT
ETHICS STATEMENT
This work investigates knowledge editing and unlearning in large language models with the goal of improving our understanding of how models update and forget factual information. Our experiments restricted to controlled benchmarks, including publicly available datasets and synthetic data that we release. We do not use sensitive, private, or personally identifiable information. While the methods studied could, in principle, be misused to manipulate model knowledge for harmful purposes, our intention is purely scientific, and we have limited our scope to safe, non-sensitive settings. All pretrained models used in this study are publicly available and used in accordance with their licenses. We believe our work contributes to safer, more transparent, and more responsible approaches to model editing and unlearning.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our results. All datasets used are publicly available or synthetically generated; details of dataset construction, splits, and preprocessing are provided in Appendix [ Model architectures, and evaluation metrics are fully described. Our imple- mentation builds on open-source frameworks (c.g.. PyTorch, HuggingFace Transformers, vLLM), and we will release the configuration files and synthetic benchmark data upon publication.
REFERENCES
Charith Chandra Sai Balne, Sreyoshi Bhaduri, Tamoghna Roy, Vinija Jain, and Aman Chadha. Parameter efficient fine tuming: A comprehensive analysis across applications, 2024. URL
httpss///axxvv.or/////
13506
Thomas Baumhauer, Pascal Schöttle, and Matthias Zeppelzauer. Machine unlearning: Linear filtra- tion for logit-based classifiers. Machine Learning, 11109):203-3226, 2022.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models, 2021. URL htt.pss://axii..rrrrsrsses http:////.aai...rraaa//000...0 08164
Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xue'. Lifelong knowledge editing for LLMs with retrieval-augmentedd continuous prompt learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13565-13580 Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1/2024.emn) 172024.emnlp-main.755
emnlp URL https: httpss:/acllatthooooyy..es 1aw.::mammmmmmmmmmmmmmmmmmmmmmaa main.
7517]
DeepSeek-AL. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. JRLLattts://aaxiiv http://axxi...or////////..0.....
Gal Elidan, Ian McGraw, and Daphne Koller. Residual belief propagation: Informed scheduling for asynchronous message passing. arXiv preprint arXiv: 1206.622666660 6837, 2012.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He. and Tat seng Chua. Alphaedit: Null-space constrained knowledge editing for language models, 2025.
URL http:httt//aaaaa www.aawwwwvoroorgeee wwwg/abb//aaa 02355

---

Published as a conference paper at ICLR 2026
Giorgos Flouris, Dimitris Manakanatas, Haridimos Kondylakis, Dimitris Plexousakis, and Grigoris Antoniou. Ontology change: classification and survey. The Knowledge Engineering Review, 23 (2):117-152, 2008.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations?, 2024. URL httpps::
www.iioorg/aaaa......o...a.. http://ab//220.0000 05904
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. European Conference on Computer Vision, pp. 383-398. Springer, 2020.
Quanquan Gu, Zhenhui Li, and Jiawei Han. Generalized fisher score for feature selection. arXiv preprint TXXiv:/202.372 2012.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. urXiv preprint arXiv:1911.0300 2019.
Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting, 2024.
http:p////aaxv..oor///////////// 27433
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
7/ 7/arxiv.oxe wwtp:/o/g/abbbooooe 20090/3000000300
Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, and Haiqin Yang. Dissecting fine- tuming unlearning in large language models, 2024. URLhttpss:// http://aarxiv..ooo/a//////2
06606
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International conference artificial intelligence and statistics, pp. 2008-2016 PMLR, 2021.
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lec, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.0009) 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv.orq/abs//3311
Dahyun Jung, Jaehyung Seo, Jaewook Lee, Chanjun Park, and Heuiseok Lim. Come: An unlearning- approach to conflict-free model editing. arXiv preprint arXiv:2502.15822 2025.
Kabir Khan, Priya Sharma, Arjun Mehta, Neha Gupta, and Ravi Narayanan. Dysk-attn: A frame- work for efficient, real-time knowledge updating in large language models via dynamic sparse knowledge attention, 2025. CRLLattps::/// http:///axvv.vv. org/ abs/ 220////00.... 07185
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
12

---

Published as a conference paper at ICLR 2026
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprchension. arXiv preprint orXiv:1706.04115, 2017.
Xiang Li, Wenqi Wei, and Bhavani Thuraisingham. Mubox: A critical evaluation framework of deep machine unlearning Isystematization of knowledge paper]. In Proceedings of the 30th ACM Symposium on Access Control Models and Technologies, pp. 175-188, 2025a.
Zexi Li, Xiangzhu Wang, William F Shen, Meghdad Kurmanji, Xinchi Qiu, Dongqi Cai, Chao Wu, and Nicholas D Lane. Editing as unlearning: Are knowledge editing methods strong baselines for large language model unlearning? arXiv preprint arXiv:2505. 19855, 2025b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human 07958
falsehoods, 2022. httttps://axxi..orr///aaass
Lihui Liu, Zihao Wang, Jiaxin Bai, Yangqiu Song, and Hanghang Tong. New frontiers of knowledge graph reasoning: Recent advances and futurre trends. In Companion Proceedings of the ACM Web Conference 2024, pp. 1294-1222 1297, 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for Ilms, 2024.
Shirley Mark, Rani Moran, Thomas Parr, Steve W Kennerley, and Timothy EJ Behrens. Transferring structural knowledge across cognitive maps in humans and models. Nature communications, (1):4783, 2020.
Elan Markowitz, Anil Ramakrishna, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai- Wei Chang, and Aram Galstyan. K-edit: Language model editing with contextual knowledge awareness, 2025. URL wttptttts///xiiv..tttt
/ 102...1655 10626
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359-17372. 2022a.
Kevin Meng, Amnb Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-cditing memory transformer. arXiv preprint arXiv:2210.07229; 2022b.
Meta. The llama 3 herd of models, 2024. URL
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. arXiv preprint arXiv:21 10. /309, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Con- ference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15817-15831. PMLR, 17-23 Jul 2022. URL htttps://proceeeingg www.ggssm................. press/ v1 62 / mi wwwethheelllle 122aa html
Andrew Y Ng. Feature selection, 1 I vs. 12 regularization, and rotational invariance, In Proceedings of the twenty-firs international conference on Machine learning, pp. 78, 2004.
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? com- paring knowledge injection in llms, 2024. URL http:ttsss/......org/.//////o//
Haewon Park, Gyubin Choi, Minjun Kim, and Yohan Jo. Context-robust knowledge editing for language models, 2025. htRL:ttts:/// h/appss::///rivvv// http:///wwwxv...////eeas//// 2albbs//555..55
Heiko Paulheim. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web, 8(3):489-500) 2016.
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few-shot unlearners. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st Inter- national Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 40034 40050. PMLR, 21-27 Jul 2024. URL https: pectps://procee httpas////occeeeigg...e mlr. preas/v235/pawe pawe czyk24a html
13

---

Published as a conference paper at ICLR 2026
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. CoRR, 2024.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks, 2020. http::ts:// arxiv. org/ httgg////////// 2004. 00345
Gemma Team. Gemma: Open models based on gemini research and technology, 2024. URL
https://arxis: wttp://ww....a///2.3...0.2.
Qwen Team. Qwg-32b: Embracing the power of reinforcement learning, March 2025a.
URL
wtt..ggihthtbb...ooo.ooooo haoplllggggg-oooooo 32b/
Qwen Team. Qwen3 technical report, 2025b. URL /tt.attps//aax....//////////....
Lukas Thede, Karsten Roth, Matthias Bethge, Zeynep Akata, and Tom Hartvigsen. Understanding the limits of lifelong knowledge editing in Ilms, 2025. URL https: https::/arriv..rr//abs
2503.0555 05683
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Un- derstanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 303-319. IEEE, 2022.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint S.Kiv:2400.147660 2024.
Zihao Wei, Liang Pang, Hanxing Ding, Jingcheng Deng, Huawei Shen, and Xueqi Cheng. Stable knowledge editing in large language models, 2024.
13048
Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, and Ningyu Zhang. Relear:: Unlearning via learning for large language models. arXiv preprint 15Xi::5502.119990 2025.
Wenli Yang, Lilian Some, Michael Bain, and Byeong Kang. A comprehensive survey on integrating large language models with knowledge-bart based methods. Knowledge-Baud Based Systems, pp. 113503, 2025.
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Ma- chine unlearning of pre-trained large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pp. 8403-8419, Bangkok, Thailand, August URL 2024. Association for Computational Linguistics. doi: 10. 12..866///0022...lllooo.gg
https://aclanthol.oo
200p:/2222..cc11omn 4577]
Yunzhi Yao, Peng Wang. Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10222-10240, Singapore, December URL
2023a. Association for Computational Linguistics. doi: 10.
wwwph/llloo.....//222/ 2023, emn in. 632/
Yunzhi Yao, Peng Wang. Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. arXiv preprint ar 23155 2023b.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286 2024.
Ruixiang Zhang, Shuangfei Zhai, Etai Littwin, and Josh Susskind. Learning representation from neural fisher kernel with low-rank approximation. arXiv preprint urXiv:2202.0/94/ 2022.
14

---

Published as a conference paper at ICLR 2026
Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Rezz Reza Namazi-Rade and Jun Wang. How do large language models capture the ever-changing world knowledge? a review of recent advances. arXiv preprint drXtv:2310.07343, 2023.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprini arXiv:2305. 12740, 2023.
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795. 2023.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
15

---

Published as a conference paper at ICLR 2026
Appendix Uncovering Knowledge Updating in LLMs with Model Editing and Unlearning
KnowledgeSmith:
CONTENTS
Data Generation Example and Pipeline
17
Knowledge Point and Knowledge Graph (KG)
17
A.2
Template Generation
17
Prompting GPT for Question Generation
17
A.4
Probe Types
19
A.5
Multiple-Chan Choice Formatting and Data Records
20
A.6
Quality Control
20
Domain and Sample Granularity
20
B
Propagation Asymmetry Metrics and Algorithm
20
Stress Testing
21
Robustness and Failure Mode
D.I Adversarial Robustness Analysis
21
D.2 Failure Mode Examples
22
E Additional Methods
23
Unlearning
E.2 Editing
23
Sequential Update
24
Accuracy Result
25
H Model Similarity Result
25
LLM usage
25
16

---

Published as a conference paper at ICLR 2026
A DATA GENERATION EXAMPLE AND PIPELINE
To make our pipeline transparent, we provide end-to-end example showing how single knowl- edge point expands into a large set of evaluation items, emphasizing hierarchical structure and con- trolled fact editing.
A.1 KNOWLEDGE POINT AND KNOWLEDGE GRAPH (KG)
We illustrate how a knowledge point can be represented as a triple and anchored at different levels of the knowledge graph. Table 3 shows one example from each domain.
Table 3: Examples of knowledge triples and anchoring across different levels of the KG hierarchy.
Domuin
Example Triple
KG (Root Intermediate
Leuf)
Biology
(DNA double holix, discovered.inn, 1953)
Root: concept of DNA structure - role mokocular biology and geneties link genericstmediinchhicss applications
Economics
(Phillips curve. deseribes, intlation-
Reot: cconomic trade-offs + macroconemic models of intlation and unemployment
unemployment relationship)
relitiooi
policy dehates stagflation and monetary policy
History
(Declaration of Independence, signed.in, in, 1776)
Root: revolutions and independence movements -> American Revolutionary era
specific events such as the Continental Congress early U.S. govemance
Physics
(Theory lished 1915)
Relativity, pob-
Root: fundamental physics theories spacetime gravitation framework appli.
cations such as black holes, gravitational waves, or GPS conrections
This fact is anchored at three levels of the knowledge graph:
Root:
broad, domain-leve understanding.
Intermediate: contextual understanding, including its role and implications.
Leaf: fine-grained, specific questions.
A.2 TEMPLATE GENERATION
For the selected fact, we generate multiple question templates per KG level, capturing different aspects of the fact (definition, role, context, and application).
Root-level templates: Broad factual conceptual questions.
Intermediate-level templates: Questions about domain implications, causal relationships, and contextual applications.
Specific, field-dependent scenarios where the fact influences out- comes knowledge in that domain.
Leaf-level templates:
example of generated templates is shown Table where leaf-level templates are instantiated with different fields (e.g., genetics, medicine).
A.3 PROMPTING GPT FOR QUESTION GENERATION
Our pipeline for generating evaluation questions follows these steps:
1. Knowledge Graph Generation: GPT is prompted generate structured KG for the target domain. Nodes represent root, intermediate, and leaf-level knowledge.
2. Fact Selection:
From the KG, single fact is selected (e.g.. (DNA double helix, discovered_in, 1953) ) to anchor all subsequent questions.
3. Template Generation: GPT is prompted to produce multiple templated question forms sur-
rounding the fact. Templates vary in phrasing, style, and emphasis, covering definition, context, role, and applications.
4. Level-Specific Question Generation: Each template is input to GPT with instructions speci- fying the desired KG level (root, intermediate, leaf). Example prompts:
17

---

Published as a conference paper at ICLR 2026
Table 4: QA templates for four knowledge points across Biology, Economics, History, and Physies.
Level
Bology: DNA double helix
Economies: Phillips owrve
History; Declaration of In- dependence (1776)
Physies: General Relativity (1915)
Roat-lovel
What the DNA double he- lix?
What is the Phillips curve? What relationship does the Phillips curve describe!! Who proposed the Phillips curve?
What the Declaration Independence?
What is the Theory of General Relativity? proposed Theory of General Relativity?
Who discovered the DNA doublle helix?
When was Declaration of Independence signed? Who signed the Declaration of Indepentence!
When was the DNA double belix, discovered??
When was the Theory General Relatiiit pub- lished?
When was the Phillips curve intraduced?
What does the DNA druble belix describe?
Why was the Declaration of Independence created?
Why is the Phillips curve important in economics?
Why is the Theory of Gen- cral Relativity important? What does the Theory of General Relativity de- scribe?
Why is the DNA double he- important biology?
W What does the Declaration of Independence proclaim??
How is Phillips curve used in macmeconomies?
What shape is the DNA
Which country declared in- dependence in 1776?
double helix?
What does the Phillips curve imply about inflation and unemployment?
What was learned from the
What historical context led to the Declaration of Inde- pendence?
How does General Relativ- ity differ from Newtonian physies?
DNA double helix?
Which scientists worked the DNA double helix?
Which countrics have applied the Phillips curve concept?
Why is the Declaration of Independence important history?
What are the key concepts in General Relativity?
Which experiments con- firmed General Relativity?
Intermdiate
Hew the DNA double helix change molecular bi- ology?
How does Phillips curve affect monctary policy?
How the Declaration Independence inffucnee the American Revolution?
How did General Relativity influence modern physies? What role does General Rel- ativity play in cosmollogy? How does General Relativ. ity explain gravity? How was General Relativ- ity reseived by the scientific community?
What criticisms exist for the Phillips curve?
What discoveries followed the DNA double helix?
What ideus from the En- lightenment are in the Declaration?
How did the Phillips curve shupe evonomic thought?
What role did DNA dou- ble helix play in genetics?
How does the Phillips curve relate to inflation targeting?
How did other countries react to Declaration?
Hew the DNA double helix influence medical seurch??
What data supports or con- tradicts the Phillips surve' How do economists inter- pret the Phillips curve over time?
What role did the Docla- ration play in listning the U.S. government?
How does General Relativ- ity relate black holes? How is General Relativity Laught universities? What mathematical tools are used in Gencral Relativ- Ht?
What techniques confirmed the DNA double helix?
How was the Declaration received by crown?
How is DNA double he-
British
lix taught in schools?
How does the Phillips curve influence labor market poli- cies?
What reaction did scientists have to the DNA double he- lix?
What dobates occurred curing the drafting of the Declaration?
How is the Phillips curve taught in universities?
How does General Relativ ity affect GPS technology? ?
How did the DNA druble
How did the Declaration impact colonial society? How is the Declaration taught in schools?
bolix affect other fields of science?
Leaf-level
How did the DNA double belix influence research in genetics?
How does the Phillips curve explain stagilation 1970<?
Which founding fathers were key authees of the Declaration?
How did General Relativity predict bending of light? How was General Relativity confirmed during 1919
What impact did the DNA double helix have in medicine?
How did the Phillips curve influence central bank doci- sions?
How did the Declaration affect slavery debatos in the U.S.?
solar celipse?
How does General Rela- tivity influcnce gravitational wave research?
How forensic seience affected by the DNAbleble belix?
How unemployment mea- sured in relation to the Phillips curve?
What role did the Dealara- tion play in the Revolution- ary War?
How did General Relativity impact quantum theory?
In evolutionary biology. what role did the DNA doublc helix play?
What role did the Phillips curve play New Keyne- sian cconomies?
How were the colonies mobilized after Declara- tion?
does General Relativ- ity affect modern cosmolog- ical models? How do black hole studies rely on Genneral Relativit? How does General Rela- tivity explain time dilation near massive objects?
Why did biotechmology change after the DNA double helix!
How de different coun- tries' experiences validate the Phillips curve?
How did newspapers and pamphlets spread the Dec- laration?
What does public bealth owe to the DNA drable belix?
What empirical models used to test the Phillips curve?
What influence did the Declaration have on other independence movements? How did international law view the Declaration at the time?
How did the DNA double helix, influence research in anthropology?
How does the Phillips curve relate to wage inflation?
How did General Relativ- ity change our undersiand- ing of space-time?
How did the Phillips inform fiscal policy during recessions?
What impact did the DNA double helix have hioin- formatics?
How did the Declaration inspire sabsequent U.S. legislation?
How dues General Relativ-
ity relate the expansion the universe?
How is the Phillips curve applied in modern macree-
How was drug dovelopment affected by the DNA double
How did the Doclaration affect Native Americun relations? How did the Declaration shape early U.S. politcaal parties?
How are relativistic effects measured particle accel- erators?
conomic forecasting?
helix?
How does the Phillips curve interact with supply shocks?
In agriculture, what role did the DNA double helix play?
18

---

Published as a conference paper at ICLR 2026
Root-level Prompt
Knowledge fact: "DNA double helix is fundamental concept in molecular biology." Generate 3 multiple-choice questions targeting broad, domain-level understanding (root-level). Each question should have 4 answer options (A, B, C, D), one correct answer, and 3 plausible distractors.
Intermediaae-level Prompt
Knowledge fact: "DNA double helix discovery influenced the field of genetics." Generate 3 multiple-choice questions targeting intermediate-level understanding us- ing the same format.
Leaf-level Prompt
Knowledge fact: "DNA double helix was discovered in 1953 by Watson and Crick." Generate 3 multiple-choice questions targeting leaf-level understanding (specific facts). Ensure 4 answer options, one correct answer, and plausible distractors.
A.4 PROBE TYPES
From each generated question template, we derive six probe types to evaluate different aspects of model behavior:
Direct Prohe: Queries the target fact in its canonical direction.
Reverse Probe: Queries the fact in the inverted relation to test bidirectional consistency,
Multi-hop Probe: Tests knowledge propagation by asking indirectly via intermediate nodes.
Contextual Probe: Embeds the fact in rich or distractor-laden context.
Conflict Probe: Presents contradictory or competing information assess resolution.
Comparison Probe: Forces a choice between multiple candidates to evaluate selective updat-
ing.
Example prompts for the four subjects are shown in Table
Table 5: Example probes across four subject domains, illustrating six probe types.
Subject
Example Probes
Biology
Direct:
DNA
discovered!
(DNA double
Reverse: Which molecule's 's structure was determined in 1953 as a double helix?
helix)
Multi-hop: Who were the key scientists whose discovery of the DNA structure influenced modemn geneties?
Contextual: The DNA double helix discovery transformed molecular biology. In which year this break- through made?
Conflict: Some sources
1953, Which year correct?
Comparison: Was the DNA double helix discovered 1953 or 1955?
Economics
Direct: What relationship does the Phillips curve describe?
(Phillips
Reverse: Which economic principle captures the
inflation unemployment? Multi-hop: Which macrrecnnomic models rely on understanding the inflation-unemplor mmmeloommmnmmmm trade-off?
curve)
Contextual: The Phillips curve has shaped monetary policy debates. What relationship does represent?
Conflict: Some argue holds only short-term, others claim long-term relevance. Which correct?
Comparison: Does the Phillips curve describe inflation-unempllymenn or wage-productivity trade-offi?
History (Dec-
Direct: what year was the Declaration of Independence signed?
laration In-
Reverse: Which historical document signed in 1776?
dependence)
Multi-hop: Which events or congresses led to the signing of the Declaration?
Contextual: Amid Revolutionary era, Declaration was signed. Which year did occur?
Conflict: Some accounts state July others July 4. Which is correct?
Comparison: Was the Declaration signed in 1776 or 1777?
Physics
Direct: year Einstein publish the theory General Relativity?
(General
Reverse: Which scientist published General Relativity in 1915?
Relativity)
Multi-hop: Which subsequent physics phenomena were explained following Einstein's 's publication?
Contextual: General Relativity transformed our understanding of space-time time. When published?
Coufliet: Some sources claim 1915, others 1916. Which is correct?
Comparison: Einstein publish General Relativity
1920?
19

---

Published as a conference paper at ICLR 2026
A.5 MULTIPLE-CHOICA FORMATTING AND DATA RECORDS
All probes are formatted as four-choice QA items consistent with MMLU. Distractors are created via entity substitution and paraphrasing. An example for the four subjects is shown in Table 6
Table 6: Compact multiple-choice probes across four subjects. Correct answers indicated.
Subject
Example Multiple Choice
Biology (DNA double helix)
Q: When was DNA double helix discovered?
A. 1953 (Correct)
B. 1955 C. 1962
D. 1947
Ecunomics (Phillips curve)
Q: What relationship does the Phillips curve describe?
Infiation unemployment (Corect) B. Wage productivity
Interest
rale vs. invesiment
D. Savings vs. consumption
History (Deelaration of In-
In what year
Declaration Independence signed?
dependence)
A. 1776 (Correc))
B. 1775
C. 1777
D. 1781
Physics (Gencral Relativity)
what year Einstein publish the theory of General Relativity?
A. 1915 (Correat)
B. 1920
C. 1912 D. 1918
A.6 QUALITY CONTROL
Items undergo:
1. Format validation (4 options, I correct answer)
2. Factual validation against the KG
3. Distractor validation (plausible yet incorrect)
Manual spot checks ensure grammaticality and factual correctness; GPT-generated distractors cross-checked with encyclopedic sources.
A.7 DOMAIN AND SAMPLE GRANULARITY
Domains include Biology. History, Physies, and Economics, each curated into a structured KG. Our study focuses on modifying one fact at time; all QA items are anchored on this fact. Multiple templates per node level, probe types, paraphrases, and varying data scales (1, 10, 100, 1,000, 10,000) allow single fact to generate up to millions of QA items for scale evaluation.
B PROPAGATION ASYMMETRY METRICS AND ALGORITHM
To quantify over- vs. under-spreading rigorously, we define:
Collateral Change Ratio (CCR) 1rolaned
Σ
d(por (: Po |x)),
(4)
a Queaend
Residual Retention (RR)
Σ a Queiss
1 | (x) = yo(2)),
(5)
@relaned
where Qrelated denotes structurally related probes, Pa and Par predictions before and after inter- vention, and .(...) is a distance metric (KI., label change, etc.).
Propagation Evaluation Algorithm:
1. Select a target node at hierarchy level
2. Apply editing or unlearning to the node.
3. Measure direct accuracy on target node (Accairec).
4. Measure multi-hop accuracy on related nodes (AcCmuti-hop).
5. Compute CCR and RR metrics:
Editing: AcCmulti-hop as proxy for over-spreading.
Unlearning: Accmulti-hep as proxy for under-spreading.
6. Repeat for all hierarchy levels and average domains.
20

---

Published as a conference paper at ICLR 2026
C
STRESS TESTING
625
12.0
Edlting ongrral
Edlting-s
Eiling 2l
Urleaming
10.5
Urleaming
75.
10.0
165
Number of training examples
Number training examples
Number training examples
(a) COT.
(b) LLM as a judge
(c) Hallucination
Figure 8: Stress testing.
We evaluate instractionnfollowinn ability (Figure 8b) and hallucination on the TruthfulQA (Lin al. 2022) dataset (Figure 8c), testing whether the parameter update + 0' preserves desired behavior when executing complex tasks. These evaluations provide comprehensive view of how the unified framework constrains model updates, ensuring both local alignment with target distributions and global reliability across diverse scenarios.
For hallucination, the average accuracy across data scales for unlearning is 76.0%, and for editing is 76. 1%, with standard deviations of 0.87 and 0.91 respectively. This indicates that both editing and unlearning maintain stable performance under ballucination tests, with significant increase in spurious behavior.
For instruction-following when measured using LLM judge, editing accuracy drops from 63.0% (original) to 48.6% on average, while unlearning drops from 62.9% to 49.1%. Although the absolute difference is small, editing shows slightly larger variability (standard deviation 0. 0.12%) compared to unlearning (0.10%). This suggests that editing more aggressive in updating targeted knowledge but may slightly perturb complex reasoning tasks, whereas unlearning better preserves general instruction-following ability.
D ROBUSTNESS AND FAILURE MODE
D.1 ADVERSARIAL ROBUSTNESS ANALYSIS
To complement our main text results, we provide a detailed analysis of adversarial robustness for editing and unlearning interventions. Adversarial robustness is evaluated by exposing the model to deliberately misleading or deceptive probes, which combine unrelated or conflicting concepts. This stresses the model's ability to maintain prior knowledge (Q while incorporating updates.
Experimental Setup We vary the number of training examples used for each intervention: 1, 10, 100, 1000, and 10,000. For each data scale, we measure two complementary performance metrics:
Original Accuracy: The model's performance standard in-domain probes (o+), reflect- ing whether the intended knowledge update was successfully incorporated without disrupting unrelated facts.
The model's performance on conflict probes, which contain contra- dictory or misleading information. These probes test the model's robustmess against adversar- ial perturbations, i.e., whether can resist adopting incorrect or conflicting knowledge while maintaining its updated and preserved facts.
Adversarial Accuracy:
By comparing original and adversarial accuracy across training scales and intervention types (editing vs. unlearning), we assess:
The sensitivity of each method to misleading inputs.
How stability and resistance conflicts evolve more examples are provided.
Differences in trade-offs between aggressive updates (editing) and conservative updates (un- learning).
21

---

Published as a conference paper at ICLR 2026
This setup allows to systematically quantify the adversarial robustness of interventions, linking conflict probe performance directly to practical model reliability under deceptive or contradictory inputs.
Observations
Our observations are:
Editing exhibits strong local updates but high adversarial sensitivity: Original accuracy remains stable around 63% across all data scales. However, adversarial accuracy drops sharply from 36.7% at example to 31.7% at 10,000 examples. This indicates that while editing successfully enforces target updates, it leaves models vulnerable to misleading inputs, with adversarial failure increasing slightly as data scale grows.
Unlearning maintains more stable adversarial performance: Original accuracy is similar to editing. Adversarial accuracy remains relatively constant around 33-35%, showing that un- learning prioritizes preservation over aggressive enforcement, making the model less sensitive to adversarially constructed probes.
Trade-off between update intensity and robustness: Comparing the two interventions, edit- ing maximizes immediate factual incorporation at the cost of susceptibility adversarial probes. whereas unlearning provides conservative updates that better preserve prior knowledge, yielding higher adversarial robustness.
Data scale effects: Increasing the number of examples slightly improves adversarial robustness for unlearning (e.g., from 33.3% example to 34.8% 1,000 examples), but the trend is less pronounced for editing. This suggests that adding more training data does not fully mitigate adversarial vulnerability for aggressive editing strategies.
Summary These results reinforce the broader trade-offss observed in our main text. Editing achieves stronger local adaptation and in-domain gains, but adversarial robustness is compromised. Unlearning is more conservative, achieving lower immediate gains but maintaining stability under adversarial stress. Together, these findings highlight the importance of considering both factual enforcement and robustness when designing knowledge update strategies in LLMs.
D.2 FAILURE MODE EXAMPLES
We provide examples of failure mode for each subject as shown in Table 7
Table 7: Representative examples of each failure mode for the four studied subjects. Each subject is listed in separate for readability.
Subject
Failure Mode
Example
Biology (DNA)
Under-forgetiin (RR)
update DNA update changes RNA discovery year
DNA year remains
Over-spreading (CCR)
Conflict Emergence
DNA reported 1953 and 1955
Knowledge Drift
DNA update causes cell structure errors
Instruction- InstractionFoolloiing Drop
Fails to explain multi-step DNA replication
Hallucination Increase
Invents molecule "X DNA"
Economics (Phillips curve)
Under-forgetting (RR)
Phillips curve still -ataiioonnnemmllymment after update
Over spreading (CCR)
Phillips update alters Laffer curve
Conflict Emergence
Links both une-ploymment and padds-produetivity
Knowledge Drift
Update mispredicts supply-demand
Instruction- Instructio-FFolloiing Drop
Misapplies multi-ster economic policy reasoning
Hallucination Increuse
Fabricates fictional "Y-Index"
History (Declaration)
Under-forgeting (RR)
Declaration year still 1776 after update to 1777 Declaration update changes Constitution year Declaration signed 1776 and 1777
Over-spreading (CCR)
Conflict Eieergence
Knowledge Drift
Update affects French Revolation facts
Entruction-Followinne Drop
Strugles with chronological sequencing of events Clains fake historical figure influenced Declaration
Hallucination Increase
Physics (General Relativity)
Under-forgetting (RR)
GR year remains 1915 after update to 1920
ever-spre (CCR)
GR update changes Special Relativity year
Conflict Emergence
GR dated 1915 and 1920
Knowledge Drift
Update reduces quantum mechanics accuracy
Instruction Following Drop
Cannot solve multi-step step relativity problems
Hallucination Increase
Reports spurious physics law "Relativistic Thermodnnamics Law"
22

---

Published as a conference paper at ICLR 2026
E ADDITIONAL METHODS
To further validate the generality of the propagation asymmetry phenomena reported in the main paper, we conducted an additional suite of experiments using multiple independent intervention algorithms, spanning both editing and unlearning paradigms. These experiments performed on the same four subject domains (biology, economic, physics, history), and evaluated at root-, intermediate-, and leaf-level nodes in conceptual hierarchies.
UNLEARNING
We applied the gradient ascent method base Tofu (Maini al. 2024) framework with varying numbers of updates across four subjects and multiple training set sizes. The results shown in Table replicate the core findings presented in the main paper:
propagation remains asymmetric across hierarchy levels,
leaf nodes experience weaker upward transfer,
root-level deletions continue to exhibit stronger downward effects.
Importantly, these consistency patterns persist regardless of the number of training examples and irrespective subject domain. suggesting that the structural behaviors we identified not artifacts of particular unlearning implementation.
Subject
Train Size
Root
Intermediate
Leaf
biology
16.67
16.67
16.67
biology
10
16.67
16.67
16.67
biology
100
25.00
25.00
25.00
biology
1000
29.17
25.00
16.67
biology
10000
16.67
29.17
25.00
economic
29.17
29.17
29.17
economic
10
29.17
29.17
33.33
economic
100
20.83
16.67
25.00
economic
1000
37.50
37.50
29.17
economic
10000
37.50
20.83
25.00
physics
25.00
25.00
25.00
physics
10
25.00
25.00
25.00
physics
100
16.67
20.83
20.83
physies
1000
12.50
16.67
20.83
physics
10000
8.33
16.67
12.50
history
16.67
16.67
16.67
history
10
16.67
16.67
16.67
history
100
12.50
16.67
12.50
history
1000
4.17
8.33
0.00
history
10000
0.00
12.50
0.00
Table 8: Unlearning experiments using Tofu across domains and hierarchy levels.
E.2 EDITING
We also evaluated the MEND editing method (Mitchell et al. 2021 on the same corpus of subjects, hierarchy depths, and training sizes. The results shown in Table demonstrate that:
editing accuracy follows the same hierarchy-depender plasticity structure observed in the main paper,
root-level edits continue to propagate downward more strongly than bottom-up corrections,
leaf nodes remain the easiest modify reliably.
23

---

Published as a conference paper at ICLR 2026
These findings reinforce that the asymmetry patterns report algorithm agnnostic emerging from the structure of the knowledge graph itself rather than any specific intervention technique.
Dataset
Train Size
Root
Intermediate
Leaf
biology
35.2
32.7
42.1
biology
36.1
33.2
43.3
biology
100
37.6
34.4
44.7
biology
1000
38.9
35.1
46.2
biology
10000
20.3
18.7
40.5
economic
45.3
42.6
50.7
economic
10
46.2
43.3
49.4
economic
100
47.7
44.6
52.9
economic
1000
41.3
45.7
54.1
economic
10000
30.2
28.3
53.2
physics
25.3
22.7
30.2
physics
10
26.1
23.1
31.3
physics
100
27.4
24.6
32.6
physics
1000
28.7
25.4
27.7
physics
10000
15.2
13.4
30.3
history
10.3
9.7
12.4
history
10
11.2
10.3
13.3
history
100
11.1
11.4
14.1
history
1000
12.6
12.8
15.3
history
10000
6.1
5.7
14.8
Table 9: Editing experiments using MEND across domains and hierarchy levels.
F SEQUENTIAL UPDATE
To further validate our claim that editing and unlearning behave fundamentally differently, additionally conducted multi-step sequential updates on multiple facts using Qwen3-14B, LLaMA3- 8B, and Gemma-7B This section reports the results illustrate the phenomenon clearly.
SEQUENTIAL EDITING BEHAVIOR
Across multiple sequential edits, the model retains previously edited knowledge with only minor drift. Even after five cumulative edits, the performance on earlier edited facts remains largely stable. This supports our claim that editing operations are robust and localized, even under sequential updates.
Acc
1 &
& 2 & 3
& & &
& & 3 & &
Edit Fact
55.0%
54.5% 48.0%
54.0%
53.8%
53.5%
Edit Fact
47.5%
47.0%
46.5%
Edit Fact 3
62.0%
61.5%
61.0%
Edit Fact
50.0%
49.5%
Edit Fact 5
57.0%
Table 10: Sequential editing performance.
SEQUENTIAL UNLEARNING BEHAVIOR
In contrast, unlearning shows clear cumulative degradation. When more facts are removed se- quentially, the model's 's performance earlier unlearned facts, well related queries, drops sharply. This supports our central claim: Unlearning inherently more disruptive than editing, because removing information often affects interconnected knowledge.
24

---

Published as a conference paper at ICLR 2026
Acc
1
& 2
1 & &
& & 4
2 & & 4 &
Unlearn Fact
54.2%
37.7%
40.5%
34.3%
27.2%
Unlearn Fact 2
45.1%
37.8%
31.5%
25.2%
Unlearn Fact 3
43.2%
36.0%
28.8%
Unlearn Fact 4
40.5%
31.5%
Unlearn Fact 5
34.2%
Table 11: Sequential unlearning performance.
G ACCURACY RESULT
Editing accuracy for the 13 model across Ilama3, qwen3, qwq, mistral, gemma and deepseek families are lists below in Table [2] Unlearning accuracy for the 13 model across Ilama3, qwen3, qwq. mistral, gemma and deepseek families are lists below in Table 13
H MODEL SIMILARITY RESULT
Representation Similarity Analysis Our unified framework models editing and unlearning as op- timizing Liss against Cpees While probe-based evaluation measures outcomes on ++ and Q does reveal how the internal representations change during this optimization. To capture these hidden dynamics. we analyze representational shifts from the original Pree-KnolledgeSmingg state the Koossknnwleeeeeiii state using Centered Kernel Alignment (CKA) (Kornblith al. 2019) KL divergence, L2 distance and Fisher score (Zhang et al. (202)).
For unlearning, these metrics expose sharp phase transition around 1000 samples: below this point, representations remain close to baseline, but beyond it they reorganize abruptly, suggesting capac- ity breakpoint where Cpes is overwhelmed by repeated optimization produces smoother trajectories. KI divergence and Fisher scores increase steadily with training size, indicating progressive local updates to representations rather than wholesale restructuring. For example, biology edits on DeepScek-88 show KL and Fisher growing from (KL*=20, Fisher=9.77 with a single sample to (KL*1172, 1.LL=172.Fishree-00 Fisher==93.7) at 1000 samples, after which growth plateaus as the optimization stabilizes.
@+ Editing, in contrast,
These results demonstrate that unlearning triggers abrupt phase transitions in representation space once data scale crosses threshold, while editing produces gradual, localized adjustments, underscoring the need for representation level analysis beyond probe accuracy.
Computationally Efficiency. For the same model target dataset of 10, 000 examples, learning typically completes in about 1.5 hours on an NVIDIA H100. Knowledge editing is more resource-intensive (roughly hours). This additional cost highlights the heavier computational de- mands of precise factual editing.
In summary, unlearning prioritizes stability and low computational cost, while editing max- imizes factual enforcement but risks destabilizing other knowledge and requires more re- sources. The choice between the two depends on whether minimizing collateral effects or maxi- mizing certainty of change is the primary goal.
Model similarity for Ilama3, qwen3, qwq, mistral, gemma and deepseek families lists below in Tables 14 to 19
I LLM USAGE
We use large language models (LLMs) only for grammar checking and correction.
25

---

Published as a conference paper at ICLR 2026
Table 12: Editing Accuracy
laama3 2: instruct
llamu3 Sh instract
Branch
Train Size
Test
Biology
History
Economic
Physies
Biology
History
Economic
Physies
ID
45,83
25
37.5
20.83
20.83
4.17
29.17
12.5
00D
44,05
44.07
44.11
44.05
63.08
63.13
63.22
63.05
1D
20.83
25
50
33.33
25
12.5
20,83
10
Intermediate
000
44,04
44.06
44.02
42.06
63.07
63.04
63.1
63
100
1D
45.83
25
45.83
0
20.83
12.5
20.83
41.67
26.88
63.05
63.12
24.72
1000
1D
4.17
25
45.83
0
20.83
12.5
20.83
12.5
25.22
43.98
26.85
63.06
63.03
63.09
24.3
10000
10
4.17
25
45.83
20.83
12.5
20.83
12.5
00D
25.22
43.98
44.17
26.85
63.06
63.03
63.00
24.3
1D
41.67
25
41.67
29.17
16.67
4.17
33333
16.67
00D
44.17
44.16
43.99
44.15
63.04
62.91
63.1
63.05
10
29.17
25
45.83
29.17
12.5
4.17
33.33
25
10
Root
000
44.1
44.28
44.12
44.07
63.01
62.9
63.12
63.1
10
29.17
25
4.17
20.83
12.5
4.17
33.33
25
100
000
44.12
44.22
26.34
44.2
62.97
63.15
63
29.17
25
0
12.5
4.17
33.33
16.67
1000
44.26
25.4
44.09
63.11
63.11
10
29.17
25
0
25
12.5
4.17
33.33
16.67
10000
00D
44.15
44.26
25.4
44.09
62.98
62,98
63.11
63.11
1D
41.67
25
33.33
25
16.67
4.17
37.
16.67
000
44.13
44.11
44.08
44.02
63.1
63.07
63.09
10
25
25
62.5
20.83
16.67
4.17
37.
25
10
Leaf
000
44,19
44.41
43.74
43.18
63.12
62.96
63.06
62.75
10
4.17
4.17
4.17
16.67
12.5
4.17
100
000
25.45
40.55
25.53
26.88
62.78
62.47
62.6
25.41
25
45.83
0
8.33
16.67
4.17
16.67
1000
000
25.78
23.44
26.63
24.84
62.77
59.56
62.59
24.25
25
45.83
0
16.67
4.17
16.67
10000
00D
25.78
23.44
26.63
24.84
62.77
59.56
62.59
24.25
Hama3 2-3b-instrutt
Iluma3 1amma3.3-70b---00 7.00----nstt
25
12.5
41.67
16.67
20.83
8.33
20.83
25
00D
59.17
59.24
59.36
59.22
81.44
81,42
81.39
81.42
12.5
375
37.5
20.83
62.5
41.67
29.17
10
Intermediate
00D
56.38
56.84
58.63
58.39
81,38
81.38
81.43
81.48
100
20.83
000
23.47
26.9
23.32
25.84
81.46
81.26
81.33
81.39
1000
000
25.45
25.12
25.31
25.08
81.39
81.35
81.38
81.31
4.17
45.83
41.67
29.17
10000
00D
25.45
25.12
25.31
25.08
81.39
81.35
$1.38
81.31
25
4.17
29.17
12.5
20.83
4.17
20.83
25
00D
59.2
59.28
59.24
59.34
81.41
81.46
81.46
81.43
41.67
29.17
16.67
58.33
45.83
37.5
33.33
10
Root
00D
58.76
58.73
58.37
58.72
81.41
81.46
81.39
81.51
37.5
41.67
29.17
58.33
25
41.67
33.33
100
000
23.3
26.86
24.4
25.34
81.39
81.41
81.4
81.42
4.17
20.83
25
58.33
20.83
41.67
33.33
1000
000
25.57
26.48
25.06
25.2
81.42
81.46
81.48
$1.44
10000
4.17
41.67
33.33
00D
25.57
26.48
25.06
25.2
81.42
81.46
$1.48
$1.44
20.83
33.33
20.83
20.83
8.33
20.83
25
00D
59.23
59.25
59.3
59.24
81.43
81.37
81.43
81.45
37.5
8.33
45.83
29.17
25
58.33
20.83
10
Leaf
00D
59.13
59.26
58.35
56996
81.43
81.38
81.41
81 34
8.33
4.17
54.17
20,83
20.83
58.33
20.83
100
00D
24.49
25.52
23.21
24.55
81.41
81.44
81.33
81.5
20.83
4.17
54.17
4.17
25
62.5
20.83
1000
00D
27.23
26.24
23.19
25.42
81.43
81.37
81.29
81 81.44
20.83
4.17
54.17
4.17
28
62.5
20,83
00D
27.23
26.24
23.19
25.42
81.43
81.37
81.29
81 14
26

---

Published as a conference paper at ICLR 2026
quen3-1.77 1.7b
qwen 32h
Branch
Train Size
Test
Biology
History
Economic
Physics
Biology
History
Economic
Physies
ID
20.83
12.5
33,33
29.17
12.5
25
12.5
00D
53
53,99
54.05
54.08
75.11
75.19
75.09
TD
25
12.5
33.33
29.17
20.83
0
20.83
8.33
10
Intermediate
53,55
53.95
54.08
75.02
1D
20.83
12.5
33.33
37.5
20.83
0
29.17
8.33
100
53
53,02
74.97
10
25
12.5
33.33
37.5
20.83
0
29.17
8.33
1000
00D
53.04
53,99
53.87
53.07
75.2
75.15
74.86
75.03
1D
25
12.5
33,33
37.5
20.83
0
29.17
8.33
10000
00D
53.04
53,99
53.07
75.2
75.15
74.86
75.03
ID
57,5
45.83
25
12.5
20.83
16.67
45.83
8.33
00D
53.81
53.7
75.2
75.12
75.21
75
1D
33,33
45.83
16.67
12.5
16.07
25
33.33
16.67
10
Root
53.78
53.78
75.17
75.02
75.07
10
29.17
45.83
25
20.83
20.83
12.5
12.5
16.67
100
53.7
53.65
10
29.17
45.83
25
20.83
20.83
16.67
12.5
16.67
1000
53.75
75.1
10
29.17
45.83
25
20.83
20.83
16.67
12.5
16.67
10000
00D
53.79
53.82
53.75
75.1
75.1
75.06
1D
16.67
37,5
37.5
25
20.83
0
25
8.33
00D
53.97
53.19
51.06
75.15
75.1
75.07
75.07
10
16.67
29.17
8.33
16.67
20.83
25
29.17
4.17
10
Leaf
53.86
53.92
74.9
75.15
75.07
10
16.67
33.33
29.17
25
29.17
16.67
54.17
4.17
100
54.2
51.66
75.1
10
16.67
37.5
37.5
33.33
25
4.17
41.67
4.17
1000
53.19
39.71
10
16.67
37.5
37.5
33.33
25
4.17
41.67
4.17
10000
39.71
qwen3-14b
quq-322
20.85
8.33
4.17
70.83
12.5
000
73.84
73.86
73.89
73.94
77.4
77.45
77.42
77.45
4.17
20.83
12.5
4.17
12.5
10
Intermediate
000
73.78
73.54
73.42
73.62
77.36
77.39
77.42
77.35
25
4.17
16.67
12.5
100
000
73.76
73.45
73.36
73.56
77.28
77.43
77.41
77.39
1000
25
4.17
4.17
16.67
16.67
37.5
12.5
00D
73.73
73.42
73.38
73.56
77,38
77.43
77.41
77.4
10000
25
4.17
4.17
16.67
37.5
12.5
00D
73.73
73.42
73.33
73.56
77.38
77.43
77.41
77.4
41.67
20.83
33.33
12.5
20.83
20.83
12.5
00D
73,86
73.9
73.86
73.79
77.
77.47
77.37
77.42
20.85
16.67
12.5
16.67
12.5
10
000
73,81
73.71
73.87
73.85
77.35
77,53
77.47
77.38
37.5
16.67
12.5
16.67
12.5
100
000
73.71
73.68
73.81
73.58
77.43
77.39
77.43
77.3
12.5
16.67
12.5
1000
000
73.73
73.64
73.84
73.55
77.44
77.39
77.45
77.35
16.67
12.5
12.5
10000
00D
73.73
73.64
73.84
73.55
77.44
77.39
77.45
77.35
25
20.83
00D
73.89
73.89
73.87
73,88
77.48
77.39
77.4
77.48
20.85
41.67
12.5
10
00D
73.91
73.63
73.63
73.69
77.5
77.37
77.33
77.42
16.67
29.17
12.5
100
000
73.66
73.39
73.42
73.6
77.4
77,52
77.33
77.4
16.67
1000
00D
65.94
73.39
39.3
73.5
77.27
77,52
68.74
77.28
16.67
12.5
10000
00D
65.94
73.39
39.3
73.5
77.27
77,52
68.74
77.28
27

---

Published as a conference paper at ICLR 2026
mistral Small 24B Instract 2501
gemma 2b
Branch
Train Size
Test
Biology
History
Economic
Physics
Biology
History
Economic
Physies
ID
16,67
50
12.5
20.33
8.33
4.17
12.5
12.5
73.4
73.34
73.39
30.53
TD
50
45.83
16.67
0
20.83
12.5
29.17
4.17
10
Intermediate
24,47
22.99
25.2
25.51
1D
29.17
45.83
54.17
58.33
16.67
45.83
37.5
29.17
100
24.22
22.95
10
45.83
45.83
54.17
58.33
16.67
45.83
54.17
58.33
1000
00D
22.87
22.95
24.16
25.81
22.95
22.95
22.95
1D
45.85
45.83
54.17
58.33
16.07
45.83
54.17
58.33
10000
000
22.87
22.95
24.16
22.95
22.98
22.95
ID
33,33
20.83
16.67
45.83
4.17
0.0
16.67
4.17
73,42
73.42
73.16
73.39
30.64
30.54
1D
45.83
54.17
4.17
58.33
8.33
8.33
12.5
33.33
10
Root
23.14
27.18
30.55
10
45.83
37.5
54.17
58.33
4.17
50.0
25.0
54.17
100
22.99
29.3
10
45.83
37.5
41.67
37.5
45.83
45.83
54.17
58.33
1000
10
45.83
37.5
41.67
37.5
45.83
45.83
54.17
58.33
10000
00D
23.14
24.4
24.76
25.12
22.95
22.95
22.95
1D
50
41,67
83,33
4.17
8.38
33,33
25.0
0.0
73.32
73.24
73.42
30.25
28.98
30.64
30.44
10
4,17
45.83
4.17
41.67
12.5
33,33
20.83
4.17
10
Leaf
25,47
22.95
25.19
27.7
24.6
10
8,33
45.83
54.17
8:33
25.0
37,5
37.5
41.67
100
22.96
10
41.67
45.83
54.17
8.33
45.83
45.83
54.17
58.33
1000
22.96
22.95
22.95
10
41.67
45.83
54.17
8.33
45.83
45.83
54.17
58.33
10000
Miistal-Lage--lssmott-eea
gommaa76
25.0
25.0
37.5
41.67
000
82.13
82.42
82.22
82.37
59.22
58.96
56.69
57.78
54.17
10
Intermediate
000
26,89
22.97
24.53
24.7
22.95
22.95
22.95
23.25
25.0
100
000
23.89
25.84
25.0
23.05
24.2
22.95
23.11
23.11
1000
16.67
62.5
25.0
50.0
29.17
54.17
8.33
66.67
000
23.89
25.84
25.0
23.05
24.9
25.59
25.22
24.68
16.67
25.0
29.17
54.17
66.67
10000
000
23.89
25.84
25.0
23.05
24.9
25.59
25.22
24.68
25.0
12.5
37.5
41.67
00D
82.25
82.22
82.24
82.25
59.7
59.56
57.63
59.74
10
0.0
4.17
58.33
41.67
000
26.19
26.89
25.41
22.95
28.43
22.97
29.24
22.97
37.5
58.33
100
000
26.86
22.95
24.64
23.0
22.95
23.07
22.95
24.13
37.5
58.33
1000
000
26.86
22.95
24.64
23.0
23.98
23.07
23.24
23.34
37.5
10000
00D
26.86
22.95
24.64
23.0
23.98
23.07
23.24
23.34
54.17
29.17
41.67
54.17
00D
82.19
82.25
82.07
82.08
22.82
22.97
22.95
59.29
10
4.17
0.0
58.33
00D
25.47
25.51
22.95
23.00
22.95
23.07
23.33
22.87
50.0
54.17
37.5
100
000
23.05
24.6
24.69
25.55
23:96
23.78
23.38
22.94
54.17
1000
000
23.05
24.6
24.69
25.55
25.48
24.49
25.52
25.54
4.17
10000
00D
23.05
24.6
24.69
25.55
25.48
24.49
25.52
25.54
28

---

Published as a conference paper at ICLR 2026
DeepSeek-RI-052--020
Branch
Train Size
Test Sct
Biology
History
Economic
Physics
ID
8.33
0.0
20.83
25.0
OOD
65.99
66.09
66.0
66.02
ID
25.0
0.0
33.33
29.17
10
Intermediate
0OD
65.94
65.93
66.01
65.95
ID
16.67
33.33
33.33
100
00D
65.95
65.94
10
16.67
33.33
37.5
1000
00D
65.9
65.94
66.07
66.02
11
16.67
0.0
33.33
37.5
00D
65.9
65.94
66.07
66.02
1D
12.5
8.33
0.0
45.83
00D
66.07
65.96
28.17
65.94
ID
12.5
8.33
16.67
45.83
10
Root
66.02
65.99
100
00D
65.97
66.1
66.02
66.0
1D
12.5
4.17
8.33
45.83
00D
65.92
66.02
65.99
65.92
10
12.5
4.17
8.33
45.83
10000
00D
65.92
66.02
65.99
65.92
11)
16.67
0.0
20.83
25.0
00D
65.9
65.96
65.92
ID
20.83
8.33
37.5
25.0
10
Leaf
00D
65.83
65.84
65.92
65.92
10
16.67
4.17
29.17
25.0
00D
65.95
65.77
65.92
65.76
10
16.67
4.17
25.0
20.83
1000
00D
66.02
65.77
64.29
65.8
ID
16.67
4.17
25.0
20.83
10000
65.77
65.8
29

---

Published as a conference paper at ICLR 2026
Table 13: Unlearning Accuracy
laama322 2 instruct
Ilamu3 Sh instract
Branch
Train Size
Test
Biology
History
Economic
Physies
Biology
History
Economic
Physies
ID
25.0
20.83
33.33
12.5
16.07
4,17
29.17
12.5
00D
32.69
32.69
32,69
32.09
63.01
63.01
63.01
63.01
ID
20,83
20.83
33.33
16.67
16.67
4.17
29.17
12.5
10
Intermediate
000
32.75
32.74
32.74
32.6
63.01
63.0
63.0
62.98
100
1D
29.17
20.83
33.33
16.67
16.67
4.17
37.5
12.5
32.93
62.8
62.84
62.75
1000
1D
33.33
4.17
57.5
16.67
16.67
4.17
33.33
12.5
32,67
33.91
32.72
32.51
62.95
62.91
62.85
10000
10
33.33
4.17
37.5
16.67
16.67
4.17
33.33
12.5
00D
32.67
33.91
32.72
32.51
62.95
62.82
62.91
62.85
1D
25.0
20.83
33.33
12.5
16.07
4.17
29.17
12.5
000
32.69
32.69
32.69
32.09
63.01
63.01
63.01
63.01
10
20.83
20.83
33.33
16.67
16.67
4.17
29.17
12.5
10
Root
000
32.63
32.61
32.55
32.09
63.0
63.0
63.02
63.0
10
29.17
16.67
33.33
37.5
16.67
4.17
33.33
12.5
100
000
32.97
32.74
32.84
32.94
62.91
62.98
62.87
62.89
16.67
8.33
37.5
4.17
16.67
4.17
33.33
12.5
1000
32,66
32.86
33.26
62.81
62.89
10000
10
16.67
8.33
37.5
4.17
16.67
4.17
33.33
12.5
00D
32,66
32.86
33.11
33.26
63.01
62.99
62.81
62.89
1D
25.0
20.83
33.33
12.5
16.67
4.17
29.17
12.5
00D
32.69
32.69
32.69
32.09
63.01
63.01
63.01
63.01
10
25.0
20.83
37.5
12.5
16.67
4.17
29.17
12.5
10
Leaf
000
32.69
32.58
32.69
32.74
62.98
63.02
62.99
62.98
10
25.0
20.83
25.0
16.67
16.67
4.17
33.33
12.5
100
000
32.57
32.85
32.73
32.73
62.75
62.75
63.02
62.75
20.83
12.5
20.83
20.83
16.67
4.17
33.33
12.5
1000
000
32.68
33.08
32.27
31.68
62.68
62.53
62.98
62.69
20.83
12.5
20.83
20.83
16.67
4.17
33.33
12.5
10000
00D
52.68
33.08
32.27
31,68
62.68
62.53
62.98
62.69
Nama3 2-30-instrctt
Iluma3 1aana3.3-70b---e 7.30--0-nnstt
20.83
4.17
41.67
16.67
20.83
8.33
20.83
20.83
00D
59.33
59.33
59.33
59.37
81,33
81.33
81.33
81.33
20.83
4.17
41.67
16.67
20.83
8.33
20.83
20.83
10
Intermediate
00D
59.17
59.29
59.27
59.24
81,33
81.33
81.33
81.33
100
16.67
000
59.07
59.05
59.51
59.06
81.35
81,36
81.35
81.35
1000
000
59,14
59.22
59.13
59.3
81,38
81.47
81.41
81.37
16.67
41.67
20.83
8.33
29.17
29.72
00D
59.14
59.22
59.13
59.3
81,38
81.47
$1.41
$1.37
20.83
4.17
41.67
16.67
20.83
8.33
20.83
20.83
00D
59.33
59.33
59.33
59.37
81,33
81.33
81.33
81.33
20.83
4.17
41.67
16.67
20.83
8.33
20.83
20.83
10
Root
00D
59.29
59.3
59.34
59.2
81,33
81.33
81.33
81.33
25.0
4.17
37.5
16.67
20.83
8.33
20.83
20.83
100
000
58.94
58.98
59.51
59.07
81.38
81.35
81.37
81.35
16.67
4.17
41.67
12.5
20.83
8.33
25.0
23.33
1000
000
58.96
58.99
59.41
59.12
81.39
81.33
$1.41
81.33
10000
16.67
8.33
23.33
00D
58.96
58.99
59.41
59.12
81.39
81.33
$1.41
$1.33
20.83
4.17
41.67
16.67
20.83
8.33
20.83
20.83
00D
59.33
59.33
59.33
59.37
81,53
81.33
81.33
81.33
20.83
4.17
41.67
16.67
20.83
8.33
20.83
20.83
10
Leaf
00D
59.29
59.27
59.16
59.26
81.33
81.33
81.33
81.33
20.83
4.17
41.67
16.67
20.83
8.33
20.83
20.83
100
00D
59.12
59.09
59.46
59.08
81.37
81.37
81.35
81.39
12.5
4.17
45.83
16.67
20.83
8.33
25.0
20.33
1000
00D
58.99
58.94
99.31
58.87
81.32
81.37
81.44
81.32
12.5
4.17
45.83
16.67
20.83
8.33
25.0
20.33
00D
58.99
58.94
99.31
58.87
81.32
81.37
81.44
81.32
30

---

Published as a conference paper at ICLR 2026
quen3-1.77 1.7b
qwen 32h
Branch
Train Size
Test
Biology
History
Economic
Physics
Biology
History
Economic
Physies
ID
16.67
16.67
33,33
16.67
16,07
0.0
16.67
12.5
0OD
53.9
53,92
53.92
53.93
75.13
75.13
75.13
TD
16.67
16.67
33.33
16.67
16.07
0.0
16.67
12.5
10
Intermediate
53.97
53.95
75.13
1D
8,33
16.67
20.83
8.33
16.67
0.0
16.67
12.5
100
53.51
53.25
75.07
10
25.0
20.83
25.0
25.0
16.67
0.0
25.0
12.5
1000
00D
52,66
52.36
53.6
52.64
75.07
75.26
74.98
10000
1D
25.0
20.83
25.0
25.0
16.07
0.0
25.0
12.5
00D
52,66
52.36
53.6
53.64
75.21
75.07
75.26
74.98
ID
12.5
16.67
33.33
15:33
16.07
0.0
16.67
12.5
00D
53,92
53,92
53.92
53.92
75.13
75.13
75.03
1D
16.67
16.67
33.33
16.67
16.07
0.0
16.67
12.5
10
Root
53.99
53.85
53.89
75.13
1D
8,33
16.67
20.83
16.67
16.67
0.0
16.67
12.5
100
53.35
75.12
10
33,33
29.17
37.5
29.33
16.67
0.0
25.0
12.5
1000
53.67
1D
33.33
29.17
37.5
33.33
16.67
0.0
25.0
12.5
10000
00D
52.67
51.83
53.68
52.67
75.16
75.05
75.23
1D
16.67
16.67
33,33
16.67
16.07
0.0
16.67
12.5
000
53.9
53.9
53.92
75.13
75.13
75.09
10
16.67
16.67
33.33
16.67
16.67
0.0
16.67
12.5
10
Leaf
54.0
54.02
54.0
54.05
75.13
75.13
75.03
10
16.67
16.67
25.0
16.67
16.67
0.0
20.83
12.5
100
53.68
10
25.0
29.17
25.0
25.0
16.67
0.0
25.0
12.5
1000
53.81
10
25.0
29.17
25.0
25.0
16.67
0.0
25.0
12.5
10000
53.73
Qwen3-14b
quo--32b
20.85
4.17
25.0
12.5
12.5
29.17
12.5
000
73,86
73.86
73.86
73,86
77.42
77.42
77.42
77.42
20.83
25.0
12.5
12.5
10
Intermediate
000
73.84
73.86
73.86
73,86
77.38
77.45
77.44
77.4
25.0
12.5
100
000
73.61
73.63
73.89
73.94
77.4
77.37
77.28
77.37
1000
20.83
4.17
20.83
16.67
12.5
29.17
12.5
00D
73.15
73.23
73.83
73.59
77.35
77.37
77.27
77.42
20.85
4.17
20.83
16.67
12.5
29.17
12.5
10000
00D
73.15
73.23
73.83
73.59
77.35
77.37
77.27
77.42
20.85
25.0
12.5
12.5
29.17
12.5
00D
73,86
73,86
73.86
73,86
77.42
77.42
77.42
77.42
20.83
25.0
12.5
12.5
29.17
12.5
10
000
73.88
73.86
73.86
73.85
77.45
77.47
77.44
77.48
100
0.0
25.0
29.17
12.5
000
73.84
73.66
73.86
73.62
77.35
77.3
77.38
77.45
29.17
12.5
1000
000
73.54
73.48
73.5
73.26
77.55
77.3
77.38
77.55
25.0
12.5
10000
00D
73.54
73.48
73.5
73.26
77.55
77.3
77.38
77.55
20.85
4.17
25.0
12.5
29.17
12.5
00D
73,86
73,86
73.86
73,86
77.42
77.42
77.42
77.42
10
20.85
25.0
12.5
12.5
29.17
12.5
00D
73,86
73.86
73.86
73,86
77.42
77.47
77.43
77.45
0.0
25.0
12.5
12.5
100
000
73.64
73.91
73.84
73,83
77.39
77.4
77.38
77.36
25.0
16.67
12.5
1000
00D
72.99
73.91
73.64
73.51
77.3
77.4
77.47
77.35
25.0
12.5
12.5
10000
00D
72.99
73.91
73.64
73.51
77.3
77.4
77.47
77.35
31

---

Published as a conference paper at ICLR 2026
mistral Small 24B Instract 2501
gemma 2b
Branch
Train Size
Test
Biology
History
Economic
Physics
Biology
History
Economic
Physies
ID
16,67
50
12.5
20.33
8.33
4.17
12.5
12.5
73.4
73.34
73.39
30.53
TD
50
45.83
16.67
0
20.83
12.5
29.17
4.17
10
Intermediate
24,47
22.99
25.2
25.51
29.4
1D
29.17
45.83
54.17
58.33
16.67
45.83
37.5
29.17
100
24.22
22.95
10
45.83
45.83
54.17
58.33
45.83
45.83
54.17
58.33
1000
00D
22.87
22.95
24.16
22.95
22.95
22.95
1D
45.85
45.83
54.17
58.33
45,83
45.83
54.17
58.33
10000
000
22.87
22.95
24.16
22.95
22.95
22.98
22.95
ID
33,33
20.83
16.67
45.83
4.17
0.0
16.67
4.17
73,42
73.42
73.16
73.39
30.64
30.54
1D
45.83
54.17
4.17
58.33
8.33
8.33
12.5
33.33
10
Root
23.14
30.55
10
45.83
37.5
54.17
58.33
4.17
50.0
25.0
54.17
100
22.99
29.3
10
45.83
37.5
41.67
37.5
45.83
45.83
54.17
58.33
1000
10
45.83
37.5
41.67
37.5
45.83
45.83
54.17
58.33
10000
00D
23.14
24.4
24.76
25.12
22.95
22.95
22.95
1D
50
41,67
83,33
4.17
8.38
33,33
25.0
0.0
73.32
73.24
73.42
30.25
28.98
30.64
30.44
10
4,17
45.83
4.17
41.67
12.5
33,33
20.83
4.17
10
Leaf
25,47
22.95
25.19
27.7
24.6
10
8,33
45.83
54.17
8:33
25.0
37,5
37.5
41.67
100
22.96
10
41.67
45.83
54.17
8.33
45.83
45.83
54.17
58.33
1000
22.96
22.95
22.95
10
41.67
45.83
54.17
8.33
45.83
45.83
54.17
58.33
10000
Miistal-Lage--lssmott-eea
gommaa76
25.0
25.0
37.5
41.67
000
82.13
82.42
82.22
82.37
59.22
58.96
56.69
57.78
54.17
10
Intermediate
000
26,89
22.97
24.53
24.7
22.95
22.95
22.95
23.25
25.0
100
000
23.89
25.84
25.0
23.05
24.2
22.95
23.11
23.11
1000
16.67
62.5
25.0
50.0
29.17
54.17
8.33
66.67
000
23.89
25.84
25.0
23.05
24.9
25.59
25.22
24.68
16.67
25.0
29.17
54.17
66.67
10000
000
23.89
25.84
25.0
23.05
24.9
25.59
25.22
24.68
25.0
12.5
37.5
41.67
00D
82.25
82.22
82.24
82.25
59.7
59.56
57.63
59.74
10
0.0
4.17
58.33
41.67
000
26.19
26.89
25.41
22.95
28.43
22.97
29.24
22.97
37.5
58.33
100
000
26.86
22.95
24.64
23.0
22.95
23.07
22.95
24.13
37.5
58.33
1000
000
26.86
22.95
24.64
23.0
23.98
23.07
23.24
23.34
37.5
58.33
10000
00D
26.86
22.95
24.64
23.0
23.98
23.07
23.24
23.34
54.17
29.17
41.67
54.17
00D
82.19
82.25
82.07
82.08
22.82
22.97
22.95
59.29
10
4.17
0.0
58.33
00D
25.47
25.51
22.95
23.00
22.95
23.07
23.33
22.87
50.0
54.17
37.5
100
000
23.05
24.6
24.69
25.55
23:96
23.78
23.38
22.94
54.17
1000
000
23.05
24.6
24.69
25.55
25.48
24.49
25.52
25.54
4.17
10000
00D
23.05
24.6
24.69
25.55
25.48
24.49
25.52
25.54
32

---

Published as a conference paper at ICLR 2026
DeepSeek-R1-0S28-020
Branch
Train Size
Test Sct
Biology
History
Economic
Physics
ID
12.5
0.0
12.5
25.0
OOD
65.99
65.99
65.99
65.96
ID
12,5
0.0
12,5
25.0
10
Intermediate
0OD
65.96
65.93
66.0
65.95
ID
12.5
12.5
25.0
100
00D
65.85
65.7
10
12.5
16.67
25.0
1000
00D
66.39
65.7
66.24
65.89
11
12.5
0.0
16.67
25.0
00D
66.39
65.7
66.24
65.89
1D
12,5
0.0
12,5
25.0
00D
65.99
65.99
65.99
65.96
ID
12.5
12.5
25.0
10
Root
65.98
12.5
25.0
100
00D
65.82
65.95
65.97
65.73
1D
12.5
0.0
16.67
25.0
00D
66.14
65.95
66.13
66.16
10
12.5
0.0
16.67
25.0
10000
00D
66.14
65.95
66.13
66.16
I1)
12.5
0.0
12.5
25.0
00D
65.99
65.96
ID
12.5
12.5
25.0
10
Leaf
00D
65.99
66.04
66.0
65.92
10
12.5
0.0
12.5
25.0
00D
65.66
65.78
66.07
65.9
10
12.5
0.0
16.67
25.0
1000
00D
65.68
65.78
66.03
66.01
ID
12.5
16.67
25.0
10000
65.78
33

---

Published as a conference paper at ICLR 2026
Table 14: Normalized model similarity scores for Llama3
Editing
Unicarning
Subject
Branch
Train
CKA
Fisher
Fisher
KL.
1
0.999
0.000
0.017
0.006
1,000
0.000
0.000
0.000
10
0.999
0.150
0.129
0.137
0.993
0.392
0.368
0.304
roul
100
0.999
0.150
0.129
0.137
0.001
0,949
0.984
0.741
1000
0.999
0.150
0.129
0.137
0.623
0.895
0.811
0.770
10000
0.999
0.150
0.129
0.137
0.825
0.958
0.874
0.977
biology
0.999
0.012
0.080
0.014
1.000
0.000
0.000
0.000
10
0.999
0.087
0.119
0.091
0.994
0.390
0.355
0.302
intermediatee
100
0.999
0.147
0.142
0.142
0.227
0.919
0.944
0.727
1000
0.999
0.147
0.142
0.612
0,921
0.909
0.795
10000
0.999
0.147
0.142
0.142
0,784
0.988
0.896
0.982
0.999
0.010
0.079
0.017
1,000
0.000
0.000
0.000
10
0.999
0.216
0.263
0.229
0.994
0.371
0.358
0.285
100
0.999
0.479
0.695
0.494
0.277
0.917
0.948
0.729
1000
0.379
0,982
0.997
0.992
0.687
0.877
0.920
0.803
10000
0.379
0.982
0.997
0.992
0.909
0.903
0.876
0.989
1
0.999
0.018
0.087
0.003
1,000
0.000
0.000
0.000
10
0.999
0,018
0.087
0.003
0,996
0,388
0.329
0.306
100
0.999
0.018
0.003
0.000
0.948
0.983
0.741
1000
0.999
0.018
0.087
0.003
0.623
0.893
0.806
0.767
10000
0.999
0.018
0.087
0.003
0.801
1.000
0.820
0.921
ecomomics
1
0.999
0.020
0.052
0.004
1.000
0.000
0.000
0.000
10
0.999
0.194
0.159
0.199
0.997
0.378
0.334
0.298
intemmediate
100
0.999
0.316
0.269
0.322
0.502
0.940
0.950
0.747
1000
0.999
0.316
0.269
0.322
0.686
0.895
0.828
0,788
10000
0,999
0.316
0.269
0.322
0.807
0.909
0.825
0,966
1
0.999
0,022
0.048
0.018
1.000
13000
0,000
0,000
Icaf
10
0.999
0.323
0.326
0.99%
0.284
0.289
100
0.999
0.363
0.362
0.371
0.794
0.912
0.915
0.724
1000
0.999
0.371
0.365
0.380
0.666
0.891
0.818
0.787
10000
0.999
0.371
0.365
0.380
0.851
0.891
0.815
0.942
1
0.999
0.006
0.038
0.003
1.000
0.000
0.000
0.000
10
0.999
0.124
0.152
0.134
0.994
0.390
0.365
0.298
root
100
0.999
0.124
0.152
0.134
| 0,282
0,950
0.954
0,730
1000
0.999
0.124
0.134
0.687
0.920
0.774
10000
0.999
0.124
0.152
0.134
0.895
0.957
0,878
0.981
history
0.999
0.011
0.067
0.004
1.000
0.000
0.000
intemmediate
10
0.999
0.138
0.161
0.154
0.995
0.791
0.354
0.300
100
0.999
0.138
0.161
0.154
0,230
0.925
0,970
0.723
1000
0.999
0.138
0.161
0.154
0.738
0.868
0,850
0.770
10000
0.999
0.138
0.161
0.154
0,888
0.991
0.842
0.973
1
0,001
0.070
0.000
1.000
0.000
0.000
0.000
10
0.999
0.217
0.235
0.230
0.994
0.361
0.370
0.269
leal
100
0.999
0.439
0.243
1,000
0.722
1000
0,232
0.963
0.985
0.965
0.673
0,900
0.988
0,805
10000
0,232
0.963
0.985
0.965
0.895
0.899
0.942
1.000
1
0.999
0.011
0.000
0.010
1.000
0.000
0.000
0.000
10
0.999
0.158
0.127
0.162
0.994
0.396
0.359
0.309
rool
100
0.999
0.158
0.127
0.162
0.437
0.920
0.940
0.723
1000
0.999
0.158
0.127
0.162
0.749
0.896
0.892
0.774
10000
0.999
0.158
0.127
0.162
0.909
0.932
0.847
0.978
physics
0.999
0.008
0.027
0.007
1.000
0.000
0.000
0.000
10
0.999
0.214
0.223
0.381
0.337
0,300
intermediate
100
0.425
0.751
1.000
0.755
0.469
0.945
0.980
0.745
1000
1.000
1.000
0.701
0.911
0.869
0.801
10000
0
1.000
0.990
1.000
0.910
0.904
0.831
0.910
0.999
0.023
0.043
0.020
1.000
0.000
0.000
0.000
10
0.999
0.249
0.240
0.255
0.994
0.379
0.360
0.283
100
0.45
0.743
0.987
0.741
0.959
0.497
0.483
0.504
1000
0.154
0.976
0.985
0.985
0.934
0.710
0.692
0.748
10000
0.154
0.976
0.985
0.985
0.907
0.899
0.879
0.986
34

---

Published as a conference paper at ICLR 2026
Table 15: Normalized model similarity scores for DeepSeek
Editing
Uakarning
Subject
Branch
Train
CKA
Fisher
Fisher
KL.
L.2
1
1.000
0.150
0.199
0.054
1.000
0,000
0.000
0.000
10
0.998
0.315
0.247
0.238
0.975
0.302
0.261
0.276
roul
100
0.997
0.366
0.417
0.275
0:457
0.739
0.895
0.745
1000
0.997
0.366
0.426
0.275
0.481
0.752
0.914
0.827
10000
0.997
0.366
0.426
0.275
0.674
0.777
0.713
0.995
biology
0.999
0.144
0.115
0.061
1.000
0.000
0.000
0.000
10
0.996
0.405
0.389
0.271
0.988
0.292
0.298
0.271
intermediatee
100
0.994
0.461
0.468
0.352
0..499
0.733
0.693
0.747
1000
0.994
0.461
0.352
0.579
0.769
0.829
10000
0.994
0.461
0.470
0.352
0.727
0.775
0.792
0.916
0.999
0.131
0.011
0.033
1,000
0.000
0.000
0.000
10
0.991
0.523
0.682
0.385
0,986
0.250
0.501
0.248
100
0,966
0.738
0.828
0.608
0.772
0.635
0.824
0.731
1000
0,960
0.758
0.877
0.638
0.641
0.708
0.777
0.856
10000
0.960
0.758
0.877
0.638
0.781
0.764
0.756
0.972
1
0.995
0.000
0.000
0.000
1.000
0.000
0.000
0.000
10
0.993
0.219
0.186
0.194
0,968
0.320
0.334
0.276
100
0.992
0.261
0.246
0.543
1.645
0.744
1000
0.992
0.261
0.221
0.246
00408
0.819
0.730
0.819
10000
0.992
0.261
0.221
0.246
0.000
0.980
1.000
0.880
ecomomics
1
0.992
0.102
0,250
0.053
1.000
0,000
0.000
0.000
10
0.976
0.480
0.615
0.387
0,988
0.308
0,264
0.265
intemmediate
100
0.972
0.500
0.634
0.412
0.675
0.746
0.857
0.754
1000
0.972
0,500
0.634
0.412
0.646
0.787
0,877
0.836
10000
0,972
0.500
0.634
0.412
0.645
0.869
0.919
0.917
1
0.996
0.028
0.045
0.030
1.000
0.000
0.000
0,000
Icaf
10
0,982
0.378
0.399
0.320
0.976
0.264
0,500
0.251
100
0.953
0.546
0.561
0.505
0,716
0,711
0.910
0.733
1000
0.000
1.000
1.000
1.000
0.582
0.744
0.806
0.847
10000
0.000
1.000
1.000
1,000
0,460
1.000
0.890
0.897
1
0.999
0.164
0.199
0.073
1.000
0.000
0.000
0.000
10
0.997
0.299
0.394
0.197
0,980
0.315
0.370
0.276
root
100
0,993
0.427
0.682
0.357
0,594
0.692
0.659
0,734
1000
0,993
0.427
0.898
0.814
10000
0.993
0.427
0.679
0.357
0,550
0.798
0.965
1.000
history
0.998
0.127
0.085
0.054
1.000
0.000
0.000
0.000
intemmediate
10
0.998
0.224
0.138
0.158
0.979
0.309
0.575
0.273
100
0.998
0.247
0.164
0.195
0.418
0.730
0.969
0.741
1000
0.998
0.247
0.168
0.195
0.732
0.724
0.762
0.811
10000
0.998
0.247
0.168
0.195
0,702
0.775
0.949
0.992
1
0.999
0.146
0.101
0.075
1.000
0.000
0.000
0.000
10
0.987
0.515
0.620
0.395
0,983
0.267
0.484
0,232
leal
100
0.971
0.644
0.691
0.698
0.835
0.722
1000
0.968
0.638
0.699
0,590
0,106
0.739
0,833
0,868
10000
0.968
0.658
0.699
0.590
0.610
0.743
0.853
0.991
1
0.999
0.182
0.087
0.076
1.000
0.000
0.000
0.000
10
0.998
0.287
0.231
0.157
0,980
0.300
0.320
0.250
rool
100
0.998
0.287
0.232
0.157
0.600
0,700
0.800
0.750
1000
0.998
0.287
0.239
0.157
0.650
0.740
0.850
0.820
10000
0.998
0.287
0.239
0.157
0.550
0.800
0.950
0.950
physics
0.999
0.172
0.174
0.031
1.000
0.000
0.000
0.000
10
0.986
0.451
0.323
0.970
0.280
0,300
0.250
intermediate
100
0.984
0.498
0.485
0.368
0.680
0.720
0.780
0.760
1000
0.983
0.498
0.368
0.630
0.750
0.830
0.830
10000
0.983
0.498
0.482
0.368
0.600
0.770
0.850
0.900
0.998
0.221
0.144
0.076
1.000
0.000
0.000
0.000
10
0.989
0.447
0.441
0.360
0.980
0.150
U.450
0.250
100
0.969
0.604
0.669
0.542
0,700
0.670
0.820
0.720
1000
0.965
0.625
0.660
0.567
0.650
0.720
0.850
0.850
10000
0.965
0.625
0.660
0.567
0.600
0.740
0.870
0.920
35

---

Published as a conference paper at ICLR 2026
Table 16: Normalized model similarity scores for Qwen3
Editing
Unicarning
Subject
Branch
Train
CKA
Fisher
Fisher
KL.
L.2
0.999
0.102
0.060
0.085
1,000
0,000
0.000
0.000
10
0.996
0.461
0.331
0.424
0.995
0.347
0.280
0.278
roul
100
0.996
0.482
0.341
0,442
0,773
0.882
0.744
0.745
1000
0.996
0.482
0.341
0,442
0.626
0.941
0.905
0.861
10000
0.996
0.482
0.341
0.442
0.836
0.999
0.931
0.990
biology
0.999
0.057
0.015
0.052
1.000
0.000
0.000
0.000
10
0.999
0.313
0.171
0.274
0.992
0.340
0.285
0.276
intermediatee
100
0.999
0.379
0.227
0.311
0.546
0.910
0.774
0.766
1000
0.999
0.379
0.227
0.311
0.304
0.949
0.854
0.864
10000
0.999
0.379
0.227
0.311
0,742
0.934
0.866
0.933
0.999
0.092
0.061
0.111
1.000
0.000
0.000
0.000
10
0.998
0.422
0.308
0.419
0.994
0.330
0.290
0.256
100
0.996
0.527
0.364
0.516
0.561
0.900
0.792
0.755
1000
0.843
0.888
0.759
0.889
0.386
0.962
0.895
0.883
10000
0.843
0.888
0.759
0.889
0.719
0.982
0.902
0.986
1
0,999
0.002
0.072
0.000
1,000
0.000
0.000
0.000
10
0.998
0.292
0.236
0.268
0,995
0.342
0,280
0.283
100
0.998
0.311
0.245
0.290
0.894
0.743
0.755
1000
0.998
0.311
0,246
0.290
0.542
0.924
0.796
0.840
10000
0.998
0.311
0.246
0.290
0.667
0.946
0.820
0.936
ecomomics
1
0.999
0.048
0.101
0.019
1.000
0.000
0.000
0.000
10
0.996
0.428
0.311
0.389
0.993
0.341
0,290
0.267
intemmediate
100
0.996
0.454
0.326
0.417
0.720
0.904
0.785
0.772
1000
0.996
0.454
0.326
0.417
0.668
0.943
0.825
0,864
10000
0,996
0.454
0.326
0.417
0.741
0.945
0.844
0.946
1
0.999
0.038
0.078
0.012
1,000
0.000
0.000
0,000
Icaf
10
0.998
0.376
0.273
0.340
0.993
0.334
0.277
0.260
100
0.986
0.635
0.461
0.620
0.798
0.879
0.721
0.746
1000
0.000
1.000
1.000
1.000
0.677
0.960
0.858
0.871
10000
0.000
1.000
1.000
1,000
0,748
0.979
0.858
0.968
1
0.999
0.037
0.094
0.015
1.000
0.000
0.000
0.000
10
0.998
0.415
0.345
0.393
0.987
0.337
0.291
0.271
root
100
0.998
0.423
0.347
0.401
0.656
0.890
0.765
0.754
1000
0.998
0.423
0.347
0.401
0.608
0.900
0.773
0.832
10000
0.998
0.423
0.347
0.401
0,731
0.979
0,827
0.954
history
0.999
0.014
0.098
0.002
1.000
0.000
0.000
0.000
intemmediate
10
0.997
0.445
0.368
0.419
0.991
0.341
0.281
0.272
100
0.996
0.500
0.389
0.482
0,301
0.910
0.765
0.764
1000
0.996
0.500
0.390
0.482
0.000
0.924
0.797
0.841
10000
0.996
0.500
0.390
0.482
0.691
0.979
0.847
0.971
1
1,000
0.041
0.122
0.049
1.000
0.000
0.000
0.000
10
0.999
0.401
0.325
0.406
0.987
0.323
0,299
0.238
leal
100
0.997
0.510
0.655
0.886
0.819
0.732
1000
0.997
0.510
0,386
0.515
0.036
0.994
1.000
0,897
10000
0.997
0.510
0.515
0.520
0.982
0.980
1.000
1
0.997
0.083
0.045
0.063
1.000
0.000
0.000
0.000
10
0.997
0.407
0,280
0.363
0.987
0.335
0,200
0.278
rool
100
0.996
0.462
0.329
0.428
0.688
0.887
U.767
0.751
1000
0.996
0.462
0.329
0.428
0.517
0.921
0.888
0.848
10000
0.996
0.462
0.329
0.428
0.764
0.993
0.871
0.938
physics
0.999
0.000
0.000
0.003
1.000
0.000
0.000
0.000
10
0.996
0.424
0.986
0.339
0.310
0.269
intermediate
100
0.993
0.493
0.363
0.485
0.733
0.897
0.803
0.755
1000
0.993
0.493
0.485
0.583
0.975
0.853
0.872
10000
0.993
0.493
0.363
0.485
0.723
0.891
0.827
0.906
0.999
0.051
0.053
0.048
1.000
0.000
0.000
0.000
10
0.996
0.402
0.325
0.375
0.595
0.335
0.289
0.259
100
0.992
0.509
0.384
0.493
0,740
0.892
0.787
0.739
1000
0.992
0.509
0.384
0.493
0.526
0.966
0.904
0.889
10000
0.992
0.509
0.384
0.493
0.642
1.000
0.853
0.984
36

---

Published as a conference paper at ICLR 2026
Table 17: Normalized model similarity scores for QwQ
Editing
Unicarning
Subject
Branch
Train
CKA
Fisher
Fisher
KL.
0.741
0.208
0.000
0.000
0,493
0.000
0.000
0.000
10
0.736
0,550
0.492
0.414
0.440
0.321
0.348
0.286
rout
100
0.987
0.613
0.521
0.478
0.490
0.799
0.826
0.737
1000
0.760
0.613
0.521
0.478
0,433
0.788
0.887
0.803
10000
0.760
0.613
0.521
0.478
0.491
0.934
0.977
0.989
biology
0.766
0.111
0.048
0.022
0..493
0.000
0.000
0.000
10
1.000
0,543
0.433
0.408
0,440
0.308
0.356
0.275
intermediate
100
0.561
0.547
0.434
0.415
00,489
0.790
0,850
0.744
1000
0.739
0.547
0.435
0.415
0,000
0.845
0.829
0.810
10000
0.739
0.547
0.435
0.415
0,437
0.889
0.887
0.943
0.741
0.108
0.019
0.021
0.493
0.000
0.000
0.000
10
0.764
0.555
0.435
0.437
0.440
0,300
0.342
0.249
100
0.760
0.675
0.551
0.583
0.448
0.789
0.797
0.733
1000
0.976
0.803
0.699
0.758
0.050
0.886
0.964
0.846
10000
0.976
0,803
0.599
0.758
0..45
1.000
0.932
0.983
0.741
0.063
0.097
0.066
0,499
0.000
0.000
0.000
10
0.765
0.303
0.305
0.274
014493
0,310
0.348
0.297
100
0.765
0.329
0.320
0.292
0.490
0.823
0.733
1000
0.996
0.329
0.320
0.292
0.433
0.883
0.869
0.808
10000
0.996
0.529
0.320
0.292
0.448
0.844
0.865
0.844
ecomomics
0.741
0.079
0.089
0.100
0.440
0.000
0.000
0.000
10
0.763
0.403
0.391
0.370
0.440
0.301
0.351
0,266
intermediate
100
0.735
0.532
0.502
0.500
0.490
0.797
0.895
0.753
1000
0.766
0.000
0.032
0.016
0,430
0.862
0.937
0.832
10000
0.738
0,427
0.402
0.389
0.435
0.878
0,914
0.917
1
0.766
0,000
0.032
0.016
0.440
0,000
0,000
0,000
lcaf
10
0.738
0.427
0.402
0.389
0,493
0,288
0.346
0.253
100
0.788
0,610
0.576
0.576
0.438
0,788
0.787
0.722
1000
0.000
1.000
1.000
1.000
0.487
0.886
0.962
0.834
10000
0.000
1.000
1,000
1,000
0.434
0.981
0.951
0.932
1
0.766
0.194
0.195
0.148
0.434
0.793
0.848
0.728
10
0.739
0.508
0.451
0.386
0.487
0.805
0.877
0.793
root
100
0.736
0.604
0.535
0.478
| 0,434
0,793
0.848
0,728
1000
0.761
0.604
0.535
0,487
0.877
0.793
10000
0.761
0.604
0.535
0.478
0.490
0,982
0.925
0.935
history
0.563
0.210
0.205
0.140
0.493
0.000
0.000
10
0.999
0.500
0.473
0.406
00.493
0.315
0.345
0.279
internediate
100
0.762
0.569
0.527
0.474
0,491
0.789
0.791
0.740
1000
0.992
0.569
0.527
0.474
0,490
0.802
0.839
0.797
10000
0.992
0.569
0.527
0.474
0,492
0.852
0.901
0.930
1
0.766
0.235
0.244
0.183
0,488
0.000
0.000
0.000
10
0.994
0.568
0.501
0.462
0,493
0.290
0.347
0.239
leal
100
0.748
0.740
0.666
0.765
0.829
0.723
1000
0.717
0.782
0.706
0.713
0,486
0.854
1.000
0,842
10000
0.717
0.782
0.706
0.713
0.488
0.955
1.000
1
0.998
0.092
0.044
0.050
1,000
0.000
0.000
0.000
10
0.998
0.284
0.213
0.227
0.987
0.344
0.323
0.279
rool
100
0.998
0.302
0.229
0.249
0.575
0.836
0.836
0.741
1000
0.998
0.302
0.232
0.249
0.639
0.852
0.867
0.814
10000
0.998
0.302
0.232
0.249
0.741
0.908
0.889
0.955
physics
0.999
0.060
0.067
0.014
1.000
0.000
0.000
0.000
10
0.994
0.363
0.327
0.318
0.984
0.333
0.316
0.273
intermediate
100
0.736
0.548
0.525
0.522
0.627
0.854
0.854
0.753
1000
0.761
0.548
0.525
0.522
0.638
0.879
0.847
0.834
10000
0.761
0.548
0.525
0.522
0.744
0.855
0.836
0.905
0.999
0.098
0.080
0.048
1.000
0.000
0.000
0.000
10
0.985
0.366
0.335
0.330
0.990
0.321
U.366
0.264
100
0.804
0.619
0.680
0.592
0,800
0.686
1.697
0.654
1000
0.704
0.703
0.676
0.682
0,703
0.799
0.815
0.829
10000
0.704
0.703
0.676
0.682
0.716
0.880
0.867
0.963
37

---

Published as a conference paper at ICLR 2026
Table 18: Normalized model similarity scores for Mistral
Editing
Unlcarning
Subject
Branch
Train
CKA
Fisher
Fisher
KL.
L.2
0.999
0.023
0.000
0.022
1.000
0,000
0.000
0.000
10
0.480
0.547
0.977
0.532
0.988
0.347
0.376
0.000
roul
100
0.351
0.757
0.958
0.749
0.410
0.857
1,000
0.000
1000
0,313
0.955
0.963
0.958
0.577
0.863
0.856
0.410
10000
0.313
0.955
0.963
0.958
0.778
0.911
0.837
0.978
biology
0.986
0.028
0.010
0.023
1.000
0.000
0.000
0.000
10
0.232
0.575
0.966
0.564
0.991
0.341
0.366
0.000
intermediatee
100
0.249
0.758
0.977
0.754
0.491
0.854
0.908
0,000
1000
0.385
0.972
0.994
0.972
0.498
0.884
0.820
0.319
10000
0.385
0.972
0.994
0.972
0,751
0.899
0,800
1.000
1.000
0.026
0.030
0.022
1,000
0.000
0.000
0.000
10
0.479
0.560
0.958
0.549
0.991
0.317
0.396
0.000
100
0.683
0.762
0.957
0.739
0.537
0.817
0.932
0,000
1000
0.578
0.952
0.963
0.953
0.571
0.849
0.873
0.327
10000
0.578
0.952
0.963
0.953
0.803
0.883
0.967
0.988
1
0,986
0.012
0.139
0.013
1,000
0.000
0.000
0.000
10
0,351
0.543
1.000
0,526
0,986
0.350
0.314
0.288
100
0.147
0.736
0.982
0.719
0,432
0.790
0.747
1000
0.197
0.898
0.966
0.894
0.524
0.879
0.737
0,809
10000
0.197
0.898
0.966
0.894
0.489
0.975
0.880
0.912
ecomomics
0.999
0.027
0.087
0.017
1.000
0,000
0.000
0.000
10
0.242
0.552
0.981
0.541
0.993
0.342
0.425
0.000
intemmediate
100
0.199
0.754
0.981
0.744
0.632
0,863
0.938
0.000
1000
0,143
0.971
0.995
0.973
0.667
0.875
0,774
0.329
10000
0.143
0.971
0.995
0.973
0.731
0.908
0.762
0.951
1
0.986
0.015
0.124
0.012
1.000
0.000
0.000
0,000
Icaf
10
0.523
0.569
0.556
0,989
0.321
0.354
0.267
100
0.373
0.788
0.964
0.773
0.636
0.834
0.849
0.734
1000
0.324
0.989
0.991
0.642
0.865
0.827
0.835
10000
0.324
0.989
0.974
0.991
0.686
0.957
0.854
0.936
1
0.999
0.044
0.214
0.019
1.000
0.000
0.000
0.000
10
0,285
0.560
0.960
0.546
0.987
0.347
0.342
0.282
root
100
0,163
0.767
0.969
0.760
0,511
0.844
0.793
0,739
1000
0.185
0.929
0.660
0.849
0.864
0.807
10000
0.185
0.929
0.959
0.930
0.725
0.911
0,890
0.978
history
0.999
0.035
0.021
1.000
0.000
0.000
0.000
intemmediate
10
0.448
0.568
0.956
0.551
0.988
0.347
0.403
0.282
100
0.136
0.751
0.957
0.743
0,316
0.855
0.901
0.743
1000
0.000
1,000
0.963
1,000
0,490
0.839
0.803
0.807
10000
0.000
1.000
0.963
1.000
0.760
0.915
0.879
0.979
1
0,986
0.051
0.211
0.034
1.000
0.000
0.000
0.000
10
0.626
0.552
0.954
0.537
0.988
0.317
0.384
0.246
leal
100
0,560
0.776
0.957
0.532
0.825
0.885
0.725
1000
0,375
0.960
0.954
0.959
0,273
0.878
0.940
0,857
10000
0.575
0.960
0.954
0.959
0.675
0.925
0.997
1
0.999
0.000
0.018
0.000
1.000
0.000
0.000
0.000
10
0.376
0.566
0.960
0.551
0.987
0.344
0.323
0.279
rool
100
0.234
0.775
0.969
0.763
0.575
0.836
0.836
0.741
1000
0.117
0.971
0.971
0.975
0.639
0.852
0.867
0.814
10000
0.117
0.971
0.971
0.975
0.741
0.908
0.889
0.955
physics
0.999
0.029
0.042
0.024
1.000
0.000
0.000
0.000
10
0.443
0.591
0.964
0.573
0.984
0.333
0.316
0.273
intermediate
100
0,349
0.788
0.962
0.780
0.627
0.854
0.854
0.753
1000
0.211
0.985
0.638
0.879
0.847
0.834
10000
0,211
0.985
0.970
0.983
0.744
0.855
0.836
0.905
0.986
0.065
0.048
0.025
1.000
0.000
0.000
0.000
10
0.333
0.566
0.966
0.548
0.990
0.321
U.366
0.264
100
0.647
0.775
0.967
0.764
0,800
0.686
11.997
0.654
1000
0.285
0.974
0.965
0.977
0,703
0.799
0.815
0.829
10000
0.285
0.974
0.965
0.977
0.716
0.880
0.867
0.963
38

---

Published as a conference paper at ICLR 2026
Table 19: Normalized model similarity scores for Gemma
odit
Unicarning
Subject
Branch
Train
CKA
Fisher
Fisher
KL.
L.2
1
1.000
0.152
0.000
0.000
0,791
0.000
0.013
0.052
10
0.961
0.377
0.652
0.299
0.806
0.461
0.561
0.394
roul
100
0.962
0.462
0.787
0.575
0,949
0.651
0.645
0.452
1000
0.653
0.692
0.954
0.813
0.875
0.757
0.725
0.547
10000
0.683
0.692
0.954
0.813
0.904
0.947
0.889
0.876
biology
1,000
0.000
0.197
0.040
1,000
0.000
0.000
0.000
10
0.225
0.686
0.944
0.452
0.588
0.370
0.550
0.354
intermediatee
100
0.183
0.809
0.981
0.573
0.596
0.542
0.642
0.462
1000
0.196
0.730
0.999
0.925
0.859
0.837
0.741
0.530
10000
0.196
0.730
0.999
0.925
0.888
1.000
0.906
0.859
0.544
0.673
0.932
0.446
0.853
0.000
0.000
0.000
10
0.169
0.795
0.959
0.586
0.544
0,466
0.610
0.435
100
0.115
0.877
0.954
0.699
0.508
0.561
0.666
0.511
1000
0.158
0.859
1.000
0.936
0.710
0.807
0.761
0.590
10000
0.158
0.859
1.000
0.936
0.739
0.997
0.925
0.920
1
0.994
0.273
0.548
0,041
0.870
0.000
0.000
0.000
10
0.943
0.378
0.753
0,330
0.874
0.441
0.545
0.347
100
0.703
0.504
0.595
0,889
0.645
0.639
0.438
1000
0.152
0.726
0.964
0.897
0.822
0.828
0.737
0.534
10000
0.152
0.726
0.964
0.897
0.851
1.000
0.901
0.863
ecomomics
1
0.999
0.191
0.324
0.009
0.886
0,000
0.000
0.000
10
0,283
0.661
0.873
0.435
0.578
0.397
0.562
0.382
intemmediate
100
0.174
0.798
0.970
0.638
0.580
0.515
0.630
0.451
1000
0,238
0.886
0.969
0.931
0.671
0.809
0,775
0.626
10000
0.238
0.886
0.969
0.931
0.699
1.000
0.939
0.956
1
0,300
0.635
0.944
0.440
0,866
0.000
0.000
0,000
Icaf
10
0.154
0.700
0.967
0.544
0.631
0.449
0.578
0.370
100
0.254
0.887
0.955
0.664
0.495
0.521
0.660
0.518
1000
0.000
1.000
0.979
1.000
0.582
0.839
0.804
0.673
10000
0.000
1,000
0.979
1,000
0,581
1,000
0.968
1,000
1
1,000
0.061
0.074
0.036
0.878
0.189
0.182
0.173
10
0.149
0.762
0.924
0.489
0.501
0.394
0.577
0.405
root
100
0,108
0.657
0.974
0.639
0,738
0.559
0.606
0,375
1000
0.106
0.870
0.807
0.764
0.603
10000
0,106
0.870
0.970
0.925
0.696
0.998
0.928
0.935
history
0.999
0.057
0.317
0.063
0.000
0.000
0.000
10
0.511
0.759
0.931
0.474
0.550
0.376
0.585
0.432
intemmediate
100
0.354
0.883
0.964
0.687
0.533
0.541
0.670
0.528
1000
0,401
0.869
0.955
0.942
0.713
0.833
0.788
0.646
10000
0.401
0.869
0.955
0.942
0.742
1.000
0.952
0.976
1
0.325
0.753
0.942
0.463
0.745
0.000
0.000
0.000
10
0.424
0.759
0.934
0.499
0.555
0.400
0.590
0.429
leal
100
0.317
0.863
0,518
0.479
0.639
0.490
1000
0,276
0.863
0.989
0.874
0.678
0.747
0.746
0,590
10000
0.276
0.863
0.989
0.874
0.707
0.938
0.910
0.920
1
1,000
0.094
0.106
0.025
1.000
0.000
0.000
0.000
10
0.135
0.847
0.946
0.535
0.440
0,403
0.600
0.447
rool
100
0.120
0.880
0.950
0.711
0.511
0.574
0.672
0.519
1000
0,059
0.880
0.959
0.907
0.633
0.791
0.759
0.603
10000
0.059
0.880
0.959
0.907
0.661
0.982
0.923
0.933
physics
0.998
0.087
0.342
0.063
1.000
0.000
0.000
0.000
10
0.759
0.414
0.787
0.363
0.446
0.545
0.337
intermediate
100
0,362
0.429
0.851
0.600
0.980
0.655
0.614
0,369
1000
0.159
0.675
0.995
0.846
0.867
0.775
0.702
0.477
10000
0.159
0.675
0.995
0.846
0.896
0.966
0,866
0,806
1.000
0.185
0.220
0.036
0.860
0.000
0.000
0.000
10
0.258
0.837
0.922
0.498
0.433
0.381
U.596
U.456
100
0.267
0.701
0.964
0.649
0,710
0.560
11626
0.421
1000
0.142
0.882
0.976
0.908
0.651
0.783
0.759
0.604
10000
0.142
0.882
0.976
0.908
0.680
0.973
0.923
0.933
39


Reviews:
[{"review_id": "ACRlYzYoYI", "reviewer": "Reviewer_7TGp", "summary": "This paper regards knowledge ”editing\" and \"forgetting\" as a unified constrained optimization problem and studies the performance of these two tasks by constructing multi-level data automatically generated based on a knowledge graph.\nThey mainly discovered phenomena such as Propagation Asymmetry, Plasticity Scaling and Branch-dependent Limits, and consistence-capacity Trade-off. And finally, a theoretical analysis was conducted from the perspective of singular value decomposition.", "questions": "1. Do you think different editing or unlearning methods have a significant impact on the analysis results? Why?\n2. What advantages does your framework have over the view that \"the model editing method is regarded as a strong baseline for unlearning\", or what's your opinion on this issue?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper treats knowledge \"editing\" and \"forgetting\" as a unified constrained optimization problem. \nThis perspective is not very novel since some prior work has viewed unlearning as a subset of editing. However, the author conducts extensive empirical analyses from this unified viewpoint.\nThe experiments cover 13 models across 6 families of LLMs, ranging from 1B to 123B parameters. \nThese analyses provide valuable insights and offer useful suggestions for the current chaotic development of these two fields.", "weaknesses": "This work presents a unified perspective, and the author conducts extensive comparative experiments on editing and unlearning.  However, it seems that the framework does not offer additional guiding functions, and after reading the paper, I find it hard to grasp the significance of this unified framework.  \nMoreover, this framework closely resembles the editing framework;  if we consider editing in a broader sense, forgetting could be seen as a special case of editing with the \"target empty.\"  \nLastly, the paper respectively employs one method for both editing and unlearning, but the paradigms of methods in these two areas differ significantly, and this difference is not mentioned in the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cJku5Od25o", "reviewer": "Reviewer_hQBy", "summary": "This paper proposed KnowledgeSmith, a unified framework and benchmark for studying knowledge updating in large language models via editing and unlearning. By framing both as constrained optimization problems and evaluating 13 LLMs on a hierarchical knowledge-graph benchmark, the study reveals asymmetric propagation, including editing over-spreads and unlearning under-spreads, a consistency–capacity trade-off, and shared failure modes. While the approach is conceptually rather than technically novel, it provides a valuable, systematic analysis of how LLMs modify and maintain knowledge.", "questions": "- Could the authors provide more details about the benchmark generation pipeline to improve reproducibility? For example, how consistent are the outputs across different runs? How extensive was the manual verification process mentioned in Appendix A — approximately what proportion of generated items were manually checked, and by how many annotators?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This is the 1st work to a unified theoretical formulation linking knowledge editing and unlearning under a single constrained optimization framework.\n- The paper proposes a structured, hierarchical benchmark based on knowledge graphs, enabling fine-grained evaluation across root, intermediate, and leaf concepts.\n- The paper provides a comprehensive empirical analysis of 13 LLMs and four domains, revealing consistent phenomena such as propagation asymmetry and the consistency–capacity trade-off.", "weaknesses": "The benchmark construction heavily relies on GPT-based data generation, which raises concerns about reproducibility and annotation reliability. Although Appendix A describes prompt structures and includes multi-stage validation with manual spot checks and cross-checks against encyclopedic sources, the extent of human verification and generation parameters is not fully specified, limiting strict reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y3Ie1mmPQ9", "reviewer": "Reviewer_68hr", "summary": "This article proposes a unified framework KnowledgeSmith to systematically study the knowledge update mechanism in Large Language Models (LLMs), with a particular focus on the performance and differences between knowledge editing and knowledge forgetting methods. The theoretical modeling of the paper is rigorous, the experimental design is meticulous, and the results are highly inspiring. The author not only proposed new evaluation metrics such as Consistency Collapse and Conflict Rate, but also constructed a structured and scalable knowledge intervention evaluation benchmark, filling the gap in current research on the lack of systematic understanding of knowledge update mechanisms.", "questions": "- Does the observed propagation asymmetry (over-spreading in editing vs. under-spreading in unlearning ) stem primarily from the specific algorithms chosen (AlphaEdit/ReLearn), or is it a more fundamental characteristic inherent to the editing versus unlearning tasks themselves, regardless of the method?\n- Could the exclusive reliance on multiple-choice questions mask certain effects of knowledge updates, and how might findings, particularly the consistency-capacity trade-off , differ if evaluated using open-ended generation tasks?\n- Can the authors provide quantitative results from human evaluations assessing the factual accuracy, clarity, and distractor plausibility of the GPT-4o generated benchmark components (KG relations, question templates, multiple-choice options) ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel Unified Framework: The paper introduces KnowledgeSmith, a conceptually novel and theoretically grounded framework that unifies knowledge editing and machine unlearning as instances of the same constrained optimization problem, offering a new lens for understanding and comparing these distinct but related knowledge updating mechanisms.\n- Innovative Benchmark Generation: It proposes a highly original and automatic pipeline for generating large-scale, structured evaluation benchmarks from existing knowledge graphs . This method allows for systematic probing of knowledge updates across different hierarchical levels (root, intermediate, leaf) and data scales, addressing major limitations of previous static and isolated fact-based evaluations.\n- Insightful Empirical Findings: The extensive experiments yield several significant and nuanced insights into LLM knowledge updating behavior, including propagation asymmetry (over-spreading vs. under-spreading) , plasticity limits related to hierarchy , the consistency-capacity trade-off , subject-dependent update resistance , and a unified taxonomy of failure modes.", "weaknesses": "- Limited Scope of Investigated Update Methods: The study primarily focuses on AlphaEdit for knowledge editing and ReLearn  for machine unlearning. While these represent state-of-the-art approaches, the conclusions drawn about knowledge updating mechanisms (e.g., propagation asymmetry , consistency collapse ) might be specific to these particular algorithmic paradigms (locate-then-edit vs. retraining-based unlearning). The framework's generalizability across a broader spectrum of editing techniques (e.g., memory-based methods like SERAC , gradient-based methods like MEND ) or different unlearning approaches (e.g., gradient ascent ) remains underexplored, potentially limiting the universality of the observed phenomena.\n- Reliance on Multiple-Choice Question Format: The automatic benchmark generation pipeline exclusively creates multiple-choice questions derived from knowledge graph triples . While this format facilitates standardized evaluation and leverages existing datasets like MMLU, it may not fully capture the nuances of knowledge representation and retrieval in LLMs. Evaluating updates solely through multiple-choice accuracy might overlook impacts on generative capabilities, reasoning chains, or the model's ability to handle ambiguity, potentially offering an incomplete picture of the update's true effect.\n- Over-reliance on GPT-4o for Benchmark Generation and Insufficient Validation : The KnowledgeSmith framework heavily relies on GPT-4o for crucial generation steps, including initial KG construction, template generation, and potentially probe/distractor creation . This dependency raises concerns regarding reproducibility and potential biases inherent in the generator model, which could influence the benchmark's structure and content. While Appendix A.6 mentions quality control including external validation and manual spot checks , the paper lacks quantitative analysis from systematic human studies assessing the quality, factual accuracy, and potential hallucination rates specifically within the GPT-4o generated components (e.g., generated KG relations, question templates, or multiple-choice distractors), making it difficult to fully gauge the reliability of the benchmark itself.\n- Focus on Static Knowledge Updates : The current experimental design primarily investigates the effects of single-step, isolated knowledge editing or unlearning interventions based on a static KG structure. It does not explicitly address the challenges of dynamic knowledge updating in a continual learning setting, where knowledge evolves over time, and updates arrive sequentially. Consequently, the framework does not evaluate crucial aspects such as catastrophic forgetting across multiple sequential updates, the interaction between consecutive edits/unlearning operations, or how the observed mechanisms (like propagation or consistency trade-offs) might compound or change in a lifelong learning scenario", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)