Disentangling Knowledge Representations for Large Language Model Editing
Authors: Mengqi Zhang
Zisheng Zhou
Xiaotian Ye
Qiang Liu
Zhaochun Ren
Zhumin Chen
Pengjie Ren
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Knowledge editing
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel locate-then-edit approach that disentangles knowledge representations for large language model editing to preserve fine-grained irrelevant knowledge.

Abstract: Knowledge Editing has emerged as a promising solution for efficiently updating embedded knowledge in large language models (LLMs). While existing approaches demonstrate effectiveness in integrating new knowledge and preserving the original capabilities of LLMs, they fail to maintain fine-grained irrelevant knowledge, namely facts that share the same subject as edited knowledge but differ in relation and object. This challenge arises because subject representations inherently encode multiple attributes, causing the target and fine-grained irrelevant knowledge to become entangled in the representation space, and thus vulnerable to unintended alterations during editing. To address this, we propose DiKE, a novel approach that Disentangles Knowledge representations for LLM Editing (DiKE). DiKE consists of two key components: a Knowledge Representation Disentanglement (KRD) module that decomposes the subject representation into target-knowledge-related and -unrelated components, and a Disentanglementbased Knowledge Edit (DKE) module that updates only the target-related component while explicitly preserving the unrelated one. We further derive a closedform, rank-one parameter update based on matrix theory to enable efficient and minimally invasive edits. To rigorously evaluate fine-grained irrelevant knowledge preservation, we construct FINE-KED, a new benchmark comprising fine-grained irrelevant knowledge at different levels of relational similarity to the edited knowledge. Extensive experiments across multiple LLMs demonstrate that DiKE substantially improves fine-grained irrelevant knowledge preservation while maintaining competitive general editing performance.

Published as a conference paper at ICLR 2026
DISENTANGLING KNOWLEDGE REPRESENTATIONS FOR LARGE LANGUAGE MODEL EDITING
Mengqi Zhang 2; Zisheng Zhou' Xiaotian Ye, Qiang Liu",
Zhaochun Ren'. Zhumin Chen', Pengjie Ren't I SSanndng University
Suzhou Research Institute of Shandong University
School of Computer Science, Beijing University of Posts and Telccommumications NNew Laboratory of Pattern Recognition (NLPR)
State Key Laboratory Multimodal Artificial Intelligence Systems (MAIS)
Institute of Automation, Chinese Academy of Sciences
LLeiden University
{mengqi. zhang, chenzhumin, renpengjie]@sdu. edu.cn cn zisheng. zhou@mail sdu edu.. on yexiaot edu, on qiang. www.lllulnl..... iaw.ueeelrr.aa......oom...... cn, ren@liacs leidemuniv.nl nl
ABSTRACT
Knowledge Editing has emerged as promising solution for efficiently updat- ing embedded knowledge in large language models (LLMs). While existing ap- proaches demonstrate effectiveness in integrating new knowledge and preserv- ing the original capabilities of LLMs, they fail to maintain ine-grained irrele- vant knowledge, namely facts that share the same subject as edited knowledge but differ in relation and object. This challenge arises because subject represen- tations inherently encode multiple attributes, causing the target and fine- ine-grained irrelevant knowledge to become entangled in the representation space, and thus vulnerable to unintended alterations during editing. To address this, we propose DiKE, a novel approach that Disentangles Knowledge representations for LLM Editing (DiKE). DiKE consists of two key components: Knowledge Represen- tation Disentanglement (KRD) module that decomposes the subject representation into frrget-knnolldge-reelne based Knowledge Edit (DKE) module that updates only the target-related compo- nent while explicitly preserving the unrelated one. We further derive closed- form, rank-one parameter update based on matrix theory to enable efficient and minimally invasive edits. To rigorously evaluate fine- mne-grained irrelevant knowl- edge preservation, we construct FINE-KED, new benchmark comprising fine- grained irrelevant knowledge edited knowledge. Extensive experiments across multiple LLMs demonstrate that DiKE substantially improves fine- fine-graine grained irrelevant knowledge preservation while maintaining competitive general editing performance.
and -unrelated components, and Disentanglement
different levels of relational similarity to the
INTRODUCTION
Large language models (LLMs) have garnered significant attention for their extensive knowledge storage and advanced reasoning capabilities (Zhao al. 2024). However, the inherent noise in their training data and the continuous evolution of world knowledge often lead to inaccuracies and outdated information (Cao al. 202I). To address these limitations. knowledge editing (Wang et al. 2024) has emerged as promising solution, enabling precise and efficient updates to the knowledge embedded within L.Mss. Among various approaches, parameter-modiffin methods that directly update the internal parameters of L.LMs are particularly appealing due to their ability to produce consistent outputs without requiring additional inference-time context or external memory.
"quual contribution. "orresponning author.

---

Published as a conference paper at ICLR 2026
Coarse
Fleeggained
Current Editing Methods
Fine-grained Irrelevant Knowledge
USA
Bidden
The President of USA
Capital
50
President
70
USA
Miami
Editing
Coarse-grained Irrelevant Knowledge
601
USA
Trump
President
USA
AW
Founder of
Trump
Microsoft
Bill
FT..L
ROMI
(a)
(b)
Figure 1: Knowledge editing can unintentionally affect fine- the-grained irrelevant knowledge. Figure (a): Preservation performance on fine-gras the-grained vs. coarse- coarse-grained irrelevant knowledge. Figure (b): Illustration of knowledge editing can unintentionally affect fine- ame-grained irrelevant knowledge.
Representative methods include fine-tuning-basee based techniques (e.g., g.,. FT-L Zhu et al. 2020)), meta- learning strategies (e.g.. KE (Wang al.] 2025), MEND (Mitchell et 2022)), and locate-then- edit methods (e.g.. ROME (Meng et al., (022), MEMIT (Meng et al., 2023), AlphaEdit (Fang et al., 2025). This work focuses specifically on parameter- modittyyin approaches.
A successful knowledge editing method should not only accurately inject the desired updates but also minimize unintended interference with existing irrelevant knowledge. Recent research (Geva et al. (2023) suggests that LLMs retrieve stored knowledge by recalling facts associated with spe- cific subjects. These retrieval processes center around subject representations, which encapsulate extensive attribute information related to that subject. Consequently, irrelevant knowledge can be classified into two categories based on semantic proximity to the subject of edited knowledge: fine- grained and coarse-grained irrelevant knowledge. Fine-grae aeneigrained irrelevant knowledge comprises facts that share the same subject as the target knowledge but differ entirely in relation and object, rendering them logically independent of the knowledge being edited. For example, when updat- ing the knowledge from "The President of USA is Biden" to "The President of USA is Trump", a fine-grained irrelevant fact such as "The capital of USA is Washington" " should remain unchanged despite sharing the subject USA. However, as they are encoded in the same subject representation, fine-grat fin-grained irrelevant knowledge often becomes deeply entangled with the target knowledge within the model's parameter and representation spaces (Hermandez et al.) 2024 Qin et al.) [024]. render- ing particularly susceptible to unintended alterations during editing. In contrast, coarse-graings irrelevant knowledge refers to facts that involve completely different subjects, relations, and objects, such as " "Microsoft was founded by Bill Gates", and is typically distant enough from the target edit that inadvertent alteration is unlikely.
To conduct a preliminary analysis of how editing affects different types of irrelevant knowledge, we randomly sampled 1, 000 edit instances. For each, selected one fine-grained and one coarse- grained irrelevant fact and evaluated the performance of existing editing methods, FT-L (Zhu et al,, 2020), MEND (Mitchell al. (022), ROME (Meng al. 2022), and AlphaEdit (Fang al. (025), on LlaMA3(8B) in preserving them. As illustrated in Figure Ia). i), the results show a marked discrep- ancy between the categories, with fine-grained irrelevant knowledge being more vulnerable to unintended alterations. This observation highlights the inherent challenge of preserving fine-grained irrelevant knowledge during the editing process.
While current editing methods have shown promise in accurately injecting new knowledge and broadly preserving unrelated facts, they frequently fail to adequately maintain fine-grained ir- relevant knowledge. As illustrated in Figure [bb, current methods that directly manipulate the subject representation to perform knowledge edits tend inadvertently degrade fine- ame-grained irrele- vant knowledge. Existing methods commonly attempt mitigate this interference by imposing con- straints derived from broadly sampled unrelated knowledge, such as text from Wikitext. However, these coarse- coarse-grained constraints are insufficiently precise and thus struggle to effectively prevent undesired alterations of fine-grained irrelevant knowledge.
To address the aforementioned challenge, we propose a novel locate-then-edit approach that Disentangles Knowledge representations for LLM Editing (DiKE) to effectively preserve fine- grained knowledge unrelated to the target edit, depicted in Figure 2 Specifically, we first in- troduce a Knowledge Representation Disentanglement (KRD) module (3.I)) that disentangles the subject representation into farget-knowledge-eeetee and langet-Knowledge-unelaeee components
2

---

Published as a conference paper at ICLR 2026
within the LLM's representation space. Subsequently, to inject target knowledge into LLMs with- out impacting tme-grained irrelevant knowledge, we develop a Disentanglement- based Knowledge Edit (DKE) 3$3.2 module. This module performs editing operations the lugee-knowledge- representation to remain
related representation while constraining the target-knowleeg--neeeie unchanged. Furthermore, leveraging matrix theory, derive a rank-one update formula that sat- isfies the these constraints, enabling the efficient update of model parameters. To comprehensively evaluate the effectiveness of our method in preserving fine-grained irrelevant knowledge, we con- struct new dataset, FINE-KED ($4.1) which categorizes test instances into three levels based on the relational semantic similarity between the edited knowledge and its fine-grat fine-grained irrelevant counterparts. Extensive experiments using GPT2-XL (1.5B), GPT-J (6B) and LLaMA-3 (8B) on FINE-KED and COUNTERFACT demonstrate that DiKE effectively preserves fine-grained unre- lated knowledge while achieving comparable general edit performance with other state-of-the-art editing methods. We summarize our contributions as follows:
We categorize irrelevant knowledge into coarse-graing and fine-grained types, and identify the unique challenge of preserving fine-grained irrelevant knowledge in knoweldge editing.
We propose DiKE, a novel knowledge editing method based on knowledge representation disen- tanglement, which enables precise editing of target knowledge while minimizing interference with fine-grained irrelevant knowledge. Furthermore, we derive a closed-form solution for parameter updates based on matrix theory, incorporating multiple constraints ensure effective editing.
We construct a new dataset FINE-KED, to thoroughly evaluate the ability of existing editing methods to preserve fine-grained irrelevant knowledge. Extensive experiments conducted on LLMs of varying sizes demonstrate the effectiveness of our proposed method.
PRELIMINARIES
2.1
AUTOREGRESSIVEE LANGUAGE MODEL
We focus on autoregressive LLMs that generate text by predicting the next token sequentially. Let F denote an LLM with transformer decoder layers, each consists of a multi-hear attention (MHA) module and feed-forward network (FFN). The hidden state representation at layer / is computed as:
h' = h1-l +
(1)
where a' and vt denote the outputs of the MHA and FFN modules, respectively. The FFN output is given by:
v = s(W, h h-1)) W
out?
where W and Wi out are the parameter matrices of the first and second layers of the FFN, spectively, and f()) is a non-lineat activation function. For brevity, we omit the layer index in subsequent sections and denote W
W out-
2.2 KNOWLEDGE EDITING VIA RANK-ONE UPDATES
Knowledge Editing aims to modify or inject single or multiple pieces of knowledge into LLMs without requiring full retraining. Each fact typically represented as a triple (s, (s, where denotes subject, relation, and o object, respectively. An edit sample is denoted as = (s,7,0, representing the update of the original knowledge (s, o) to a new knowledge (8, o,)).
Rank-one Knowledge Editing, exemplified by methods such ROME and MEMIT (Meng et al.) 2022 2023), which follow locate-then-edit paradigm. These methods assume that factual knowl- edge is encoded in the feed-forward networks (FFNs) of the model as key-value pairs, where the first FFN layer generates keys and the second layer uses these keys to produce values. To inject new knowledge, the FFN parameters W are updated to W, such that the model associates a new key-value pair (k., vx). To avoid corrupting unrelated knowledge, an auxiliary set of key-value straints K, = ; kz:..... :......;,, k,) and Vo = [V1; Vai...., vp is introduced, representing knowl- edge be preserved.

---

Published as a conference paper at ICLR 2026
Disentangler
h,, hy
h h
Disentangler
k
Irrelevant
Knowledge
Ze
Reconstruction
FFN
Atten
Ze
Knowledge
Disentanglement
Preserving
Rank-one
Udpate
Target
Knowledge
Knowledge
Constraint
Editing
LLM
Recomposer
Training Set
New Knowledge
Recomposer
Knowledge Representation Disentanglement (KRD)
Discntanglementt-oaee Knowledge Edit (DKE)
KRD Training
DKE Editing
The framework operates in two distinct phases. (Left) KRD Training: The module extracts subject and relation representations from the LLM and learns disentangle them into target-knowlege- wennoowwwg---eeteee and -unrelated components via optimiz- ing disentanglement, constraint, and reconstruction objectives. (Right) DKE Editing: During the editing phase, the pre-trained Disentangler is frozen. The DKE module utilizes the disentangled representations derive closed-form rank-one parameter update (Eq. (I9)), which injects knowledge into the target-related component while explicitly constraining the unrelated component preserve fine-grained irrelevant knowledge. Note that the KRD module is pre-trained the training set and does not require retraining for subsequent editing tasks.
Figure 2: Overview of the DiKE architecture.
Adopting the least-squares formulation from MEMIT (Meng et al. 2023). the updated weights w obtained by solving the following objective:
W argmin([ Wk. - v.1l + ||WK, - Voll²))
(3)
w
The closed-fon solution derived via the normal equation as in MEMIT, is given by:
W = W + (v. Wk. k, (KoK, +kkk)-!
(4)
where Kg is estimated based on sample of Wikipedia text Meng et al. 2023
3 METHODOLOGY
In this section, we present our proposed method, DiKE, with its overall architecture illustrated Figure? 2 The DiKE framework comprises two core components: (1) Knowledge Representa- tion Disentanglement (KRD), which explicitly disentangles the subject representation into target- knowledge
eewleddg-related related and -unrelated components within the LLM's representation space, and (2) Disentanglement lemet--bbbedd Knowledge Edit (DKE), which injects target knowledge into specific param- eters using the disentangled representations while preserving the fine-grained irrelevant knowledge.
3.1 KNOWLEDGE REPRESENTATION DISENTANGLEMENT
Target edit knowledge and its fine mna-grained irrelevant knowledge, such as "The President of the United States is Biden" and "The capital of the United States is Washington", are often deeply entangled within model representations. Ensuring the invariance of such fine-grained irrelevant knowledge during parameter updates presents significant challenge. To address this, design a Knowledge Representation Disentanglement (KRD) module, which decomposes the subject rep- resentation of each target knowledge instance into distinct components: capturing target- knowledge-related information, and the other encoding
untellted information.
Specifically. the KRD module consists of two components: the Knowledge Disentangler and the Knowledge Recomposer. For each edited knowledge triplet (s, r, o), the Knowledge Disentangler takes the representations of the subject and relation inputs and produces pair of disentangled

---

Published as a conference paper at ICLR 2026
vectors: the target-knowledge related representation and the weeete-kolledge-nneee unrelated represen- tation. The Knowledge Recomposer then reconstructs the original representation from the disentan- gled components, ensuring consistency and completeness in the knowledge representation.
Formally, the suggetknowllege tation z for each edit sample (s, o") computed
representation z and target-know burgt--noolledg--urrlatddd represen-
z = Dis, (h,, h,) = f(Wah, + Wyh,), (5) where ʃ(-) is the GELU Hendrycks & Gimpel (2016) activation function, and W; E Rdxd for :....: are trainable projection matrices.
E Dis, (h.: h,) J[W h, W2h,),
The Knowledge Recomposer then reconstructs subject representation h, from z and z
h Rec(z,, 2²) W sz, Wez",
where Ws and W6 Ra are learnable parameters. The subject representation h, and relation representation h, extracted from the hidden state of the last tokens in the subject and the prompt p(s, r) at the I-th layer, respectively.
To ensure that the Knowledge Disentangler effectively separates the grree-knowllege--eeeaeee related and the grgge--knowledge-unneleen unrelated representations, we introduce three complementary objectives:
Knowledge Disentangling Loss encourages the frnget-knowledge-rehie turget-knowledge- wer.unnewwwdgeeurrraeeeeeed unrelated representation z" to capture distinct attributes of the subject represen- tation hg. To achieve this, we adopt contrastive learning objective He al. (2020) that maximize the mutual information (MI) between each of z and h,, and z. and h,, while treating z and z as negative pair encourage their separation in the representation space.
representation z and the
Concretely, we define (z", h.) and (z", h.) as positive pairs, while treating (z z²) as an additional negative pair. This lead the following objective:
Cet = Info h, [ H.)) + InfoNCE(2". h, [2:;; H.))
where H. E R axd represents B negative subject representations sampled from the same batch. Here, [z.: H.] and (z"; H.] serve as the negative sample sets for z and zy respectively, The InfoNCE loss is formulated as:
exp(sim(s,s*)/5
InfoNCE(s,s' s =
log Σx Exe(((s)ss))) exp(sim(s,s')/1
where sim(... refers to the cosine similarity function, and is the temperature parameter.
Knowledge Constraint Loss is designed to ensure that the disentangled representations effectively encode meaningful attribute of the subject s: the wrgge--nowleege-relaae representation encodes the edited fact (s, r, o). while the large-Knnwlldge--nnlltee representation preserves the fine- grained irrelevant facts (s.r
Specifically, for a given knowledge triple (s, r. o). we sample set of fine-grained irrelevant fact N, where each (s, r', o') e N shares the same subject but differs in relation and object. We construct corresponding prompts p(s, r) and p(s, r'), which are separately fed into the LLM F. During forward computation, the original subject representation h, is replaced with z and z respectively, to predict the target object and unrelated object o'. The loss is defined as:
INV
Leon - log 1p00.,=)0000((..)1
Σ
log Pr lo'ssr)))))
(9)
(s.r",')EN
where F(h, :- denotes the forward computation with the subject representation h, replaced by the specified vector.
Knowledge Reconstruction Loss ensures that essential semantic information is retained during the disentangiement process by encouraging faithful reconstruction of the original subject representa- tion. is computed using the Mean Squared Error (MSE) between the original subject representatiion h, and its reconstructed representation h.:
Lracon lb, hell2
(10)
5

---

Published as a conference paper at ICLR 2026
Finally, the parameters of the Knowledge Disentangler and Knowledge Recomposer are jointly op- timized by minimizing the following overall objective:
C = Cetr aLeon + B.
(11)
where a and [ are weighting coefficients that balance the contributions of different loss functions.
Notably, the Knowledge Disentangler and Knowledge Recomposer, once trained, can be di- rectly applied during the editing process without requiring retraining for each individual edit sample. Detailed training procedures provided in Appendix F.
3.2 DISENTANGLEMENT-BASSE KNOWLEDGE EDIT
In this section, we discuss how to leverage disentangled representations to perform parameter up- dates for injecting target knowledge, while explicitly preserving fineggrined irrelevant knowledge.
Following MEMIT ct al. (2023), we formulate the editing process as the injection of key-value pairs into the FFN. For each edit instance (s, r, o²), the parameter update. The key k, is computed by averaging across N randomly generated prefixes attached prompts:
(, we compute key-value pair (k., v.) that guide
1 #,,, f(W),. in h k, N j=
(12)
Target Knowledge Editing. To compute the corresponding value v.. we first disentangle the sub- ject representation h, into z and z". Unlike MEMIT and ROME, which directly optimize the FFN output at specific layer to encode the new knowledge (,, o*), our approach instead optimizes a representation z-:
modification to the wnget-knowledger
h = Rec(z,) x.: + 5, )
8 arg min log ((b [0" p(ss r).
(13)
Based on the residual formulation of hidden states in Equation T the updated represetttttttt h at the editing layer can be expressed as:
h hn
1=1 V
(14)
where hς is the initial embedding of the subject, and a!, and v denotes the output of the attention module and FFN at layer 1, respectively.
Consequently, we compute the value v. to be injected at the FFN layer as:
h h". where h h
a!
v/.
(15)
Fine-grained Irrelevant Knowledge Preserving. To prevent unintended modifications to fine- grained irrelevant knowledge during the editing process, enforce that the disentangled unrelated representation remains invariant before and after editing. This is formalized as minimizing:
[Dis,, (h., h,) - Dis,, (h., h, )12 = Dis (hş + Wk., h,) -Dis -is, (h,,h,)
= =||5 (Ws + Wk.) W.h.) - (Wah, W.h,)
Here, W represents the updated weights, hy + Wk" denotes the subject representation extracted by the edited model. For simplicity, we omit the activation function and enforce consistency in the representation before activation, leading to the following constraint:
Ww he + Wk.) Wah,
W (Wk. - vo) 1|
(16)
where Vo = h2 h2 Wk. is the original output of the edited FFN module.
While MEMIT and ROME preserve existing knowledge by maintaining the correspondence between Kn and Vn. as described Section 2.2 our approach further ensures that the disentangled target- knowledge-unrelate representations of knowledge set (Ko, Vo) remain unchanged before and after

---

Published as a conference paper at ICLR 2026
editing. To achieve this, we minimize the following objective function:
Kol
Dis (h?, Wk;, h,,) Dis,, (b. br.)
Wa(WKo - Vo) F² (17)
where k; Ko. hs; and hr denote the corresponding representations of subject 8; and in the edited layer. The full derivation is provided in Appendix C.1
Rank-One Parameter Update. By composing Equations (16 and 17 parameter updating objective:
derive the final
W = argmin
\wk. Vik.-vv.. v.
WKo Voll
w
Target knowledge editing
Course-grained irrelevant knowledge presersing
(18)
Ws [k. vu)
W3 WK. (VVK-Vo) Vo)
Fine-grained irrelevant knowledge preserving
The closed-form solution to this optimization problem is:
W w + (W3 W;
A
(19)
where the identity matrix and MEMIT represents the parameter update derived from MEMIT (Equation 4h). The detailed derivation is provided in Appendix C.2
EXPERIMENTS
In this section, we first introduce our newly constructed dataset, FINE-KED, designed to evaluate the impact of editing fine-grained irrelevant knowledge. To provide a more comprehensive sessment of our DiKE's performance in gereral editing scenarios, we further evaluate DiKE on the widely used COUNTERFACT and MQUAKE- 3K benchmarks across GPT2-XL (1.5B), GPT-J (6B), and LLaMA3 (8B). More detailed experimental results are provided in Appendix
4.1 KINE-KED DATASET
FINE-KED is designed to evaluate the impact of editing methods on fine-grained irrelevant knowl- edge. For each edit sample (s, r, o), we construct fine-grained irrelevant knowledge (s, o') and cat- egorize the dataset into three levels: Easy, Middle, and Hard, based on the semantic relatedness tween relation and r' (examples of these levels are provided in Table T). We employ Efficacy to quan- tify the success rate of edits and Relational Locality to assess the preservation of fine-grained irrelevant knowledge. Detailed information is provided in Ap- pendix D.I
Table 1: Relation Examples in Different Lev- els of FINE-KED
Level
Relations
The name of the child of O is The name of the award { won is
Easy
The place of death of {} is
Middle
The place of birth of {} is
The name of the head of state of { is Hard
The name of the capital city of { is
4.2 EXPERIMENTAL. SETUPS
Datasets and Evaluation Metrics. In addition to FINE-KED, we conduct experiments on COUNTERFACT (Meng et al. 2022), a benchmark designed to evaluate insertion of counter- factual knowledge into LLMs. Performance is assessed using three key metrics: Efficacy Score, Paraphrase Score, and Neighborhood Score. Further details are in Appendix D.2 To further inves- tigate DiKE's generalization capability in more complex scenarios, also evaluate on the multi- hop reasoning dataset MQUAKE. Zhong et al.] (2023). Dataset descriptions are provided in Appendix D.3 with corresponding results reported in Appendix G.2
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on FINE-KED in terms of Efficacy (%) and Relational Locality (%). The best performance is highlighted in boldface, and the second-best is underlined.
R-Loc. (GPT2-XL)
R-Loc. (GPT-J)
R-Loc. (LLaMA-3)
Method
Eff.
Eff.
Eff.
Easy
Mid.
Hard
Avg.
Easy
Mid.
Hard
Avg.
Easy
Mid.
Hard
Avg.
BASE
20.4
62.7
65.7
71.9
65.8
21.7
86.9
87.6
89.2
87.6
23.9
81.1
80.3
81.3
80.9
FT
99.1
46.8
45.0
40.9
44.9
100
69.2
64.8
60.6
65.9
100
69.4
63.0
49.6
62.8
MEND
94.2
47.1
47.5
31.5
43.4
98.7
71.7
66.5
51.8
65.4
94.3
71.5
63.4
54.7
65.2
ROME
91.8
48.3
45.4
56.3
49.5
99.9
53.5
49.4
44.9
50.3
99.8
56.8
50.7
47.8
53.0
ROME-C
91.7
49.7
46.7
57.4
50.8
99.9
63.3
58.3
59.4
61.0
99.9
65.5
57.9
60.9
62.3
MEMIT
90.8
49.0
45.0
55.9
49.6
99.9
61.9
56.0
59.5
59.7
98.7
64.6
54.9
62.6
61.5
MEMIT-C
91.0
49.5
45.5
55.3
49.8
99.8
67.3
62.3
67.1
65.9
97.2
68.1
60.3
69.2
66.3
AlphaEdit
98.7
47.3
43.9
47.4
46.4
99.9
59.8
54.9
54.8
57.2
98.2
68.1
59.2
67.8
65.6
DiKE
97.4
54,2
51.8
60.2
55.0
99.1
72.0
67.4
74.1
71.3
99.1
72.7
65.3
72.4
70.6
Improve
9.1%
9.1%
4.9%
8.3%
0.4%
1.4%
10.4%
8.2%
1.7%
3.0%
4.6%
6.5%
Baselines. Our experiments are conducted three LLMs: GPT2-XL (1.5B) (Radford al. 2019),
GPT-J (6B) (Wang& & Komatsuzaki] 202T) and LLaMA3 (8B) (Dubey et al. 2024). We compare our method with number of knowledge editing methods: Fine-Tuning (FT) (Zhu et al.) 2020), MEND
(Mitchell et al. 2022), ROME (Meng et al.) 2022). MEMIT (Meng et al.] (203)), and AlphaEdit
Fang et al. 2025). To further validate the superiority of DiKE in preserving the fine-grained irrele- vant knowledge, we compare it with two variant models ROME-C and MEMIT-C on FINE-KED, which directly incorporate multiple additional relational constraints (s, rg. o.) into the LLM in each editing. The implementation details of baselines and DiKE in Appendix F.1
4.3 EXPERIMENT RESULTS
Through these experiments, we aim address the following key research questions:
How does DiKE Perform in Preserving Fine-grained knowledge? To avoid performance infla- tion due to data leakage and to verify the generalization capability of our KRD module, we ensure minimal subject overlap between the KRD training data and the evaluation datasets. Specifically, the subject overlap rates between the KRD module's training set and the edited samples in FINE- KED and COUNTERFACT are only 1.39% and 6.33%, respectively. A detailed investigation of the generalization behavior of the KRD module is provided in Appendix G.66
(i) Performance FINE-KED. Table presents the performance of all editors on FINE-KED From the results, we can draw the following observations: (1) DiKE excels in all evaluations of Relational Locality while maintaining high editing success rate. Specifically, DiKE performs hancements up to 8.3% on the Relational Locality metric over the baseline model. This demon- strates that fine-grained disentanglement and constraint of target knowledge can effectively mitigate negative impacts on fine-grained unrelated knowledge without compromising editing performance. (2) ROME, MEMIT and AlphaEdit generally perform worse on the Relational Locality compared to DiKE. MEND and FT, demonstrate better performance than ROME and MEMIT on the Easy and Middle levels of FINE-KED. However, their performance declines significantly on the Hard level of the dataset. We believe this is because these models directly manipulate the subject or relation representation to perform parameter updates, which can easily have negative impacts on fine-gre tree-graind unrelated knowledge that is entangled within the parameters and representations. Al- though AlphaEdit introduces null-space constraints to preserve unrelated knowledge, the constraint is applied at coarse-grained level and thus struggles to prevent interference with fine-grained irrel- evant knowledge. (3) ROME-C and MEMIT-C, which incorporate additional relational constraints, improve upon their original versions but still fall short of DiKE. This suggests that merely adding constraint samples is insufficient to reliably preserve fine-grained unrelated knowledge. Further- more, the requirement manually construct extra constraints for each editing instance introduces substantial overhead. In contrast, DiKE, with its KRD moudle, requires only one-time training pro- cess and can be directly applied during subsequent edits without retraining, significantly enhancing editing efficiency.
(ii) Performance on COUNTERFACT. Table 3 presents the performance of all edi- tors on COUNTERFACT The results show that DiKE achieves competitive results across other key editing evaluation metrics, such as Paraphrase Score and Neighborhood Score.
8

---

Published as a conference paper at ICLR 2026
DIKE (LLAMIA) 9/o CTR
w/o KC
w/o TRE
FT
ROME MEMIT
AlphaElit
w/o TIK
MEND
DIKE
GPT22XL
GPT4
LLaMA
100
100
100
Elllowy (36)
80-
40-
Vahic (96
Pelocionnl Localit (MS
60
40
22
20-
Ellinnyy
RL-Kay
8L-Mikkle
8L-Had
RL-AI
Edi Batch Size
Figure 3: Ablation studies on FINE-KED in terms of Efficacy and Relational Locality.
Figure 4: Performance of subject-consistent batch editing on the FINE-KED.
These findings suggest that the disentanglement- based approach, DiKE, not only effectively pre- serves fine-grained irrelevant knowledge but also maintains strong generalization ability and better re- tention of unrelated neighborhood knowledge.
Table 3:
Performance comparison on COUNTERFACT in terms of Efficacy Score, Paraphrase Score, and Neighborhood Score. The Avg. is the harmonic mean of the three metrics.
How do Different Components Affect the DiKE Performance? To investigate the effectiveness of each component DiKE, we conduct an ablation study LLaMA3 using the following variants: w/o CTR, which removes knowledge disentangling loss from the KRD module; w/o KC, which cludes knowledge constraint loss from KRD mod- ule; w/o TKE, which performs editing directly the original representations rather than the disentangled ones; and w/o FIK, which removes the constraint for preserving fine nne-grained irrelevant knowledge in the DKE module. Figure B presents the experimental re- sults on RED, leading to the following con- clusions: (1) DiKE outperforms w/o CTR and w/o
Model Method
Avg.
Effi.
Para.
Neigh.
BASE
23.8
16.2
19.2
81.0
GPT-J (6B)
FT
24.4
100.0
95.8
9.8
MEND
62.2
97.7
53.2
52.1
ROME
90.3
99.9
99.7
75.9
MEMIT
90.4
99.6
95.2
79.2
AlphaEdit
91.0
99.6
96.9
79.3
DIKE
90.8
99.8
96.1
79.3
LLaMA3 (8B)
BASE
14.1
9.2
10.9
85,9
FT
18.4
100.0
97.7
MEND
58.4
100.0
73.6
36.0
ROME
90.8
100.0
77.3
MEMIT
91.1
99.8
94.6
80.9
AlphaFdit
89.7
100.0
93.5
78.5
KC on most Relational Locality metrics, confirming that contrastive learning and knowledge constraint contribute to effective disentanglement. (2) Compared with DiKE, w/o TKE exhibits a substantial performance drop, underscoring the importance of disentangling and editing the target-knowledge- related representation rather than the original entangled one. (3) Compared with w/o FIK, DiKE further improves on Relational Locality, especially in middle and hard level, demonstrating the ben- efit of explicitly enforcing invariance the disentangled aange--nnollegge-unelatee representation. Experimental results on GPT2-XL and GPT-J are shown in Figure g
DiKE
92.4
99.9
82.8
Which Scenarios Benefit from the DiKE Approach? To further validate the importance of bbaad editing in preserving fine-grained irrelevant knowledge, we conduct subject-consisten batch editing experiment where all edit samples within batch share the same subject. This setup exerts stronger influence on the subject's representation. making the task significantly more challenging than conventional batch editing settings, in which edit samples are typically unrelated. We evaluate performance on FINE-KED under batch sizes of 1, 2, 4, and 8, with results shown in Figure
We observe that DiKE achieves an Efficacy Score close to 100% across most models and batch settings, while maintaining the highest Relational Locality Score. By contrast, ROME exhibits a sharp degradation batch size increases. Although MEMIT and AlphaEdit incorporate mech- anisms for batch editing and irrelevant knowledge preservation, respectively, their improvements in Relational Locality under larger batch settings remain significantly lower than those of DiKE. This advantage arises because DiKE explicitly disentangles subject representations across different relations, thereby isolating relation-specifie knowledge and minimizing interference. As result,

---

Published as a conference paper at ICLR 2026
edits to relation have little impact other knowledge associated with the same subject. By contrast, ROME, MEMIT, and AlphaEdit directly update entangled subject representations, where overlapping relational information leads to conflicts, imprecise updates, and reduced effectiveness. Furthermore, AlphaEdit constrains only coarse-grained irrelevant knowledge, making it less effec- tive preserving fine-grained relational distinctions within subject representations. We also find that MEND suffers from a significant performance drop in this setting, which may be attributed to its reliance on a hypernetwork to predict parameter updates based on gradients. When editing batch of knowledge instances sharing the same subject, the predicted gradients are more likely to conflict with each other, resulting editing failures.
5 RELATED WORK
In this section, we introduce the related work on knowledge editing, which aims to inject new knowledge into LLMs or modify their internal knowledge, while minimizing unintended changes to unrelated knowledge. This study focuses on parameter-modiffing methods, which can be broadly categorized into three groups:
fine-tuning-baser methods (Gu et al. 2024 Yu et al. 2024 Ni et al, 2024) utilize efficient
parameter-tuning techniques to update model knowledge. To alleviate issues such as overfitting. these methods typically introduce additional constraints to preserve unrelated knowledge. For ex- ample, RECT (Gu et al.) 2024p injects new knowledge by selecting and fine-tuning the top-k pa- rameters most relevant to the target, while simultaneously constraining the magnitude of updates to reduce interference with other knowledge.
Meta-learning--assed methods employ a hypernetwork to generate editing speciffc parameter up- dates. MEND (Mitchell et al.) 2022) uses a low-rank gradient decomposition and a lightweight hypernetwork to transform fine-tuning gradients into weight updates. MALMEN (Tan et al. 2024) extends MEND to batch editing by formulating the update process as a least-squares optimization problem.
Locate-then-edit Methods perform editing by first identifying model parameters associated with
the target knowledge and then applying targeted updates (Meng al. 2023 Gupta et al. 2024
Zhang et al. [2024) Fang et al. 2025 Zhang et al.) 2025b a). Early work such as Knowledge Neurons
(Dai et al. 2022), proposed knowledge attribution method to identify relevant neurons. However,
this approach exhibits limitations in precisely adjusting model weights. Subsequently, ROME (Meng
al. 2022) treats the weights of the FFN layers as a form of linear associative memory and updates specitic layers to encode new knowledge. MEMIT (Meng et al.] 2023) extends ROME by enabling large-scale knowledge editing through shared updates across multiple layers, thereby mitigating interference with previously edited layers. AlphaEdit (Fang et al. (025) extends this line of work by introducing null-space constraints to prevent updates from affecting unrelated knowledge. However, its constraints are applied at a coarse- -rrriie- level, making it less effective at preserving fine-grained irrelevant knowledge that may be entangled with the target in the representation space.
CONCLUSION
In this paper, we have proposed DiKE for knowledge editing in LLMs. DiKE leverages a Knowledge Representation Disentanglement module separate target-related knowledge from fine-grained un- related knowledge. Building on this, we introduce a Disentanglement-to based Knowledge Edit mod- ule that injects the edited knowledge while preserving the fine- mme-grained neighboring irrelevant knowl- edge. Experimental results across three LLMs on our constructed FINE- -ED and COUNTERFACT, demonstrate the effectiveness and superiority of DiKE in model editing tasks.
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (Nos. 62502286, 62472261, 62576339, 62372275). It was also Funded by Basic Research Program of Jiangsu (BK20250429). the Shandong Provincial Natural Science Foundation (2R2024QF203), the Tech- nology Innovation Guidance Program of Shandong Province (YDDXX022088), the Provincial Key
10

---

Published as a conference paper at ICLR 2026
R&D Program of Shandong Province (2024CXGC010108), and the Open Research Fund of the State Key Laboratory of Multimodal Artificial Intelligence Systems.
ETHICS STATEMENT
The goal of this work is to advance the field of knowledge editing in LLMs, aiming to improve the precision and reliability of information they contain. Our proposed method, DiKE, designed to update factual knowledge while minimizing unintended side effects on related information, which we believe is a step toward more controllable and trustworthy AI systems. Such improvements have positive applications in areas like correcting factual inaccuracies, removing harmful or biased content, and keeping models updated with the latest information without costly retraining.
However, we acknowledge that like any technology that allows for the modification of information, knowledge editing tools could be misused. A malicious actor could potentially leverage such tech- niques to insert subtle misinformation, propaganda, harmful biases into LLM. While our work focuses on improving the technical fidelity of edits, it does not in itself prevent such applications. We believe the broader research community must continue to develop robust detection mechanisms and safeguards in parallel with advancements in editing techniques. We encourage the responsible development and deployment of knowledge editing technologies, with strong emphasis ethical considerations and safeguards against misuse.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we provide detailed descriptions of our methodology and experimental setup throughout the paper. All experiments were conducted using publicly avail- able Large Language Models: GPT2-XL (1.5B), GPT-J (6B), and LLaMA3 (8B). We evaluate our method on the widely-used COUNTERFACT and MQUAKE-3K benchmarks. For evaluating the preservation of fine-grained irrelevant knowledge, we introduce new benchmark named FINE- KED, and we will release the dataset publicly upon publication.
Appendix F.I provides specific implementation details, including key hyperparameterss such as learning rates, batch sizes, optimizer settings, and the specific layers targeted for editing in each model. For baseline comparisons, we utilized the experimental framework provided by Meng al. (023) to ensure consistency. To further aid reproducibility, we will make our source code and the newly constructed FINE-KED dataset publicly available.
REFERENCES
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp- 1860-1874, 2021.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Trunsactions of the Association for Computational Linguistics, 12:283-2983 2024.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), ACI. 2022, pp- 8493-8502, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21778 2024.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat- Sat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference Learning Representations, 2025.

---

Published as a conference paper at ICLR 2026
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual asso- ciations in auto-regressive language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
2023.
Jia-Chen Gu, Hlao-iian Xu, Jun Yu Ma, Pan Lu, Zhen-Hua Ling, Kai- Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regularization to the rescue. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2024, 2024, pp. 16801-16819, 2024.
Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli, A unified framework for model editing. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15403-15418, 2024.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/ 605.0841,, 2016
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In The Seventh Interna- tional Conference on Learning Representations, 2019.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359- 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In The Tenth International Conference on Learning Representations, 2022.
Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang. Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024. pp. 57 9716-57311 2024.
Jiaxin Qin, Zixuan Zhang, Chi Han, Pengfei Yu, Manling Li, and Heng Ji. Why does new knowledge create messy ripple effects in Ilms? In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 12602-12609, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Chemmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. In The Twelfth International Conference on Learning Representattons, 2024.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, May 2021.
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Comput. Surv. 57(3):5::-55:33 2025.
Xiaohan Wang, Shengyu Mao, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu. Huajun Chen, and Ningyu Zhang. Editing conceptual knowledge for large language models. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 706-724, 2024.
12

---

Published as a conference paper at ICLR 2026
Lang Yu. Qin Chen, Jie Zhou, and Liang He. MELO: enhancing model editing with neuron- indexed dynamic lora. Thirty-Eightt AAAI Conference on Artificial Intelligence, AAAI 2024, pp. 19449-19457, 2024.
Mengqi Zhang, Xiaotian Ye. Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. Knowledge graph enhanced large language model editing. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024. pp. 22647-22666 2024.
Mengqi Zhang, Bowen Fang, Qiang Liu, Xiaotian Ye, Shu Wu, Pengjie Ren, Zhumin Chen, and Liang Wang. Kele: Residual knowledge erasure for enhanced multi-hop reasoning in knowledge editing. In Findings of the Association for Computational Linguistics; EMNLP 2025, pp. 24537- 24552, 2025a.
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Shu Wu, Pengjie Ren, and Zhumin Chen. Uncovering over- fitting in large language model editing. In The Thirteenth International Conference on Learning Representations, 2025b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. A survey of large language models, 2024.
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. Con- ference on Empirical Methods in Natural Language Processing, pp. 15686-15702, 2023.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
During the preparation of this work, we use a LLM for assistance with the following tasks: (I) refining the language and improving the clarity of the manuscript, particularly in the Methodology and Results sections; (2) correcting grammatical errors and ensuring stylistic consistency.
B LIMITATIONS & FUTURE DISCUSSION
We acknowledge several limitations of our work, which suggest promising directions for future research.
The first limitation of DiKE is its focus structured knowledge editing tasks, similar to those addressed by ROME, MEMIT, and AlphaEdit. Specifically, DiKE is best suited for scenarios where knowledge can be explicitly represented relational triples (s, T, o). This design choice may limit its applicability in broader knowledge editing contexts, especially those involving unstructured or semi-structured knowledge formats. Extending disentangled representation learning and editing to support more diverse and unstructured forms of knowledge remains an important direction for future work.
Second, the scope of our evaluation benchmark, FINE-KED, currently limited to measuring editing success and the preservation of fine-grained irrelevant knowledge. While provides tar- geted assessment of editing success and preservation of fine-grained irrelevant knowledge, these dimensions alone are insufficient to fully characterize the broader capabilities required in knowl- edge editing. In particular, many real-world editing scenarios involve additional challenges such as generalization to paraphrases, multi-hop -hop reasoning, and robustmess distribution shifts. Therefore, FINE-KED should be viewed as complementary benchmark rather than a standalone evaluation. In future work, we plan further expand FINE-KED to better capture the diverse challenges in model editing and provide more complete assessment of editing performance.
Finally, we focus specifically on fine-grained irrelevant knowledge that shares the same subject as the edited fact in this study. This choice is motivated by the observation that existing representative editing methods, such as AlphaEdit and ROME, perform poorly on this type of knowledge (as shown in Fig[I, DD. highlighting it as a particularly challenging and underexplored problem. We believe that addressing this category is both meaningful and necessary. Moreover, in structured knowledge edit- ing scenarios, defining fine-grained irrelevant knowledge via shared subjects but differing relations is both intuitive and formally tractable, which facilitates evaluation and methodological design. In future work, we plan explore other types of fine-grained irrelevant knowledge beyond subject overlap to further broaden the scope and generality of our framework.
C DERIVATION DETAILS FOR FINE- GRAINGRDN IRRELEVANT KNOWLEDGE PRESERVATION
C.I DERIVATION OF EQUATION
L7
We provide detailed derivation of Equation equation 17 from Section 3.2 which enforces the consistency the disentangled wagge--nnolledg--unneltte representations before and after editing:
Kol Dis, (h,, Wk,, h,.) ) -Dis, (h., h
W W Vo)
We begin by expanding the definition of Knowledge Disentangler:
Kol
Dis [b? + Wk;, hn.)) Disu (Berr br:) 1
(20)
K
Σ
(Wo he, + +Wk) + Wahr.) - -(Whh, W.h.
14

---

Published as a conference paper at ICLR 2026
Following the approximation in Equation (6). omit the nonlinearity activation function and enforce consistency at the pre-activation level:
(Wa(h), + Wk.) + Wab..) - (Wah., + Wabh
Kel
Wa(h", + Wk; ba))1 2
Kel
Ws(WWk - -va)
Kol
(21)
Tr (Wa(Wk: -v:(kwk -v) Tw)
Ko
Tr
Ws(Wkk vo(wk, TW
=Tr (Wa(WKo vol(WKo Vo) W3
-||wa(WKK- Vol F
where Tr(:) denotes the trace operator.
C.2 DERIVATION OF THE CLOSED-FORM SOLUTION FOR EQUATION
18
We provide a detailed derivation of the closed-form solution for the optimization problem defindd
Equation 18) from Section 3.2
|Wk. V.
2
WKr Voll2
W argmin W
Target knowledge editing
Course-grained knowledge preserving
+ W3 (Wk. - vo) + W3 (WK, Vo
Vo vo
Fine-grained irrelevant knowledge preserving
To start, we denote the update as W with W AW; and reformulate the objective as:
L(AW) I(( + AWIK. v. + W; ((W +AW]k, vo)
+ +[W + AW)Ko Vol + [: ((W + AWjK, - Vo)
(22)
=1AWK, (v. -k. | Wa(vo Wk.)
+ -AWK, - -Voo W + W,AWK, Wa(Vo W
To facilitate the derivation, recall a general form of Frobenius norm minimization:
L(W) JAWB- - CI22 C)
-Tr (AWBBBC)AWB - CC)T)
(23)
= Tr (AWBB W T AWBCT CB W AT +CC)).
15

---

Published as a conference paper at ICLR 2026
Next, compute the gradient of i(W) with respect to W:
VwL(W) Vw Tr (AWBB w A AWBC CB -
WT + cC.)
① Tr (AWBB W AT)
Tr (AWBCT)
OW
oW
(24)
aW Tr (CB W A
Tr TT(CC
OW
(A AWBB A CCC
This result serves as the foundation for deriving the gradient for L(AW) with respect AW:
VawL(AW) (AWk.k, -(v+ Wk.)k,)
+2 (W. WaAWk,k, W; W3(vo Wu,vo-Wk,1k)
(25)
+ 2 K, - -Vo WK.)K)
2 (W3 WaAWK,K, WW Wa WK.)K,))
Since W is assumed to be the optimal least-squares solution for memorizing a mapping from previous set of keys Kg to values Vo, it satisfies the normal equation (Meng al. 2022)
WKGKj VoK.
(26)
Moreover, as vo = Wk., we simplify the gradient expression as follows:
VAwL(AW) = WA - (v. Wk.)k + WT WaAWk.k
+ AWK, + W. W W,AWW,
(27)
2 W. + E) + k,k,) - -vv. -kk.k.)).)
Setting the gradient to zero yields the optimal update:
(W3 W. EJAW K + +k.,,) (v. W.,,,, =
AW W W. E) E Wk.)k, (KIKg +K
(28)
(W3 W. -
Finally, the updated weight matrix W is expressed as:
W = W + (W Wa + E) AMEMIT
(29)
D DETAILS OF DATASETS & EVALUATION METRICS
D.1 DETAILS OF FINE-KED
We construct FINE-KED to systematically evaluate the impact of knowledge editing methods on fine-grained irrelevant knowledge. Table summarizes the dataset statistics, and Table S illustrates an example from the dataset.
2024 we first sample set of subjects and their corresponding knowledge triples T, = [ T;, o;) ]
Following RippleEdits (Cohen et al.
1,2, Specifically, we select entities as subjects if their corresponding Wikipedia pages ranked within the top-1000 most viewed pages for at least one month during 2020 to 2022. For each relation r, we collect set of objects O,, comprising all objects from knowledge triples sharing the relation For every subject, randomly select triple (s, r, o) as the edit prompt and sample a target object o* # from O,, To construct the neighborhood prompt, the GPT-J model to filter out knowledge already correctly recalled by the model (i.e., triples where the model's accuracy exceeds than 80%) and then randomly select one remaining (s,r',0) the fine-graine' neighborhood prompt.
To comprehensively evaluate the effectiveness of editing methods preserving fine-grained irrel- evant knowledge, classify all neighborhood prompts into three difficulty levels based their
16

---

Published as a conference paper at ICLR 2026
Table 4: Composition statistics FINE-KED Dataset
Type
Total
Edit Prompts
3085
Neighborhood Prompts
3085
Easy Level
1501
Middle Level
Hard Level
757
Table 5: An Example of FINE-KED Dataset
Property
Value
Edit Prompt
The namc of the father of (Mitch McConnell) is Muhammad al- Jawad. The name of country of citizenship of Mitch McConnell U.S.A..
Neighborhood Prompt
relational similarity to the edit prompt: Easy, Middle and Hard. Specifically, we prompt Mistral- 7B- IB-Instruct-vO.33 (see Table F. for details) to evaluate relational similarities scale from 0 (com- pletely unrelated) to 10 (highly related). We define the categories as follows: Easy (0-3), Middle conducted
(4-6), and Hard (7-10). To ensure the reliability of our difficulty-Ieve categorization, a human evaluation with expert annotators from the field of LLMs. The results confirmed a high level of agreement between the labels generated by our method and the judgments of human experts.
For subject-consisten batch editing task, where all edits in a batch share the same subject, we expand the edit prompts by incorporating additional knowledge triples. Specifically, for each subject, extended the editing prompt by adding triples remaining after constructing the edit and fine-grained neighborhood prompts, and construct the prompts in the same way as the edit prompts. We then filtered out samples where the total number of editing prompts was fewer than 8, resulting in the construction of 605 samples for the task.
To evaluate performance on FINE- KED across all editors, we adopt two primary metrics, Efficacy and Relational Locality. Each metric is calculated as follows:
Efficacy measures the edited model's ability correctly recall the updated target entity given the edit prompt p(s, r). It is computed as E[[[[" argmax P (-l(sp((s r)]]]
Relational Locality assesses the edited model's capability to correctly recall the original entity with the fine fine-grained neighborhood prompts. It is computed as E[I[o' = argmax (--ppsx)
D.2 DETAILS OF COUNTERFACT
T presents an example from the COUNTERFACT dataset, which includes an edit prompt, two paraphrase prompts, and multiple neighborhood prompts. the given example, the edit prompt aims to update the model's 's knowledge of Selma Kurz was employed in from Vienna to London. Paraphrase prompts are semantically rephrased versions of the target edit prompt. Neighborhood prompts retain the same relational structure as the edit request but involve different subjects whose associated knowledge should remain unchanged by the edit. We randomly sample 1, 000 records to evaluate all editing methods.
To evaluate COUNTERFACT across all editors, we adopt three widely used metrics (Meng et al.] 2022 2023 Efficacy, Paraphrase Score and Neighborhood Score. Each metric is calculated as follows:
Efficacy Score measures whether the post-edit LLMss can correctly recall the new target entity when provided with the edit prompt p(s,r). Unlike the calculation FINE-KED, it is computed as E[[[[P pr (o" (p(s,r) > (pl(sp((s
Paraphrase Score evaluates the performance of the post-edit LLM rephase prompt set po derived from the edit prompt r). The calculation similar to Efficacy: Epppa [I[P p (o P Pr (olp)]]
17

---

Published as a conference paper at ICLR 2026
Table An Example of COUNTERFACT Dataset
Property
Value
Edit Prompt
[Selma Kurz} was employed in Vienna + London.
Paraphrasc Prompt
Selma Kurz took up work in London.
Neighborhood Prompt
Gottfried Wilhelm Leibniz worked the city of Vienna.
Neighborhood Score assesses whether the post-edit LLM assigns higher probability to the correct fact on prompt set pl, which consists of distinct prompts sharing the same relation and target object as edited knowledge but differing subject. This metrie calculated as Ppp "per<([[[P [[PP'(("lM)) < (plii]]]]
D.3 DETAILS OF MQUAKB-3K
MQUAKE-3K (Zhong et al. 2023) is a challenging benchmark for evaluating whether models perform multi-hop reasoning with newly edited knowledge. Each instance consists of mul- tiple single-hop edits, accompanied by questions that require multi-hop reasoning over the up- dated facts. This setup places stricter demands new information but also integrate it across reasoning chains. Table [ provides an example from MQUAKE-3K dataset. To fully exploit the reasoning capabilities of LLMs, we adopt zero-shot setting for answer generation. Following Zhong et al. (2023), we report the Efficacy Score to mea- sure the accuracy of the post-edit model on the multi-hop question P about the edit sample: E [I[PP]IIWPPWW answerlg) > P( Pooriiinal answer 4)]]-
edited L.LMs, they must not only memorize
Table 7: An Example of MQUAKE dataset
Property
Value
Edit Request
[Lou Pearlman a citizen of United States of America -> India The capital of {India] is New Delhi > Taloga
Edit Request 2
New Question
What the capital of the country to which Lou Pearlman belonged? (Lou Pearlman, citizen of, United States of America), (United States of America, the capital of, Washington) Washington
Original Relation
Original Answer
New Relation
(Lou Pearlman, a citizen of, India), (India, the capital of, Taloga) Taloga
New Answer
E BASELINES
Our experiments are conducted on GPT-2 XL (1.5B) (Radford et al. 2019), GPT-J (6B) (Wang & Komatsuzaki 202T) and LLaMA3 (8B) (Dubey al. 2024). We compare the DiKE against the following state-on of editing techniques:
Constrained Fine-Tuning (FT) (Zhu al. 2020), which directly fine-tunes specific layers of LLM's parameters using gradient descent while applying regularization constraints to prevent catastrophic forgetting:
MEND (Mitchell et al. 2022), a gradient-based low-rank decomposition method that employs
hypernetwork to perform edits;
ROME (Meng al. 2022). which assumes that knowledge in LLMS is stored in FFN modules
and performs edits by optimizing and updating specific FFN layers to insert knowledge;
MEMIT (Meng et al. by editing sequence of FFN layers.
2023), an extension of ROME designed specifically for batch editing tasks
AlphaEdit (Fang et al. 2025), a null-space projection method designed to better preserve unre-
lated knowledge during editing by constraining updates orthogonal preserved information.
To further verify the superiority of our disentanglementt-aaae knowledge editing method, we also compare method with variant models ROME-C and MEMIT-C. These baselines are de-
18

---

Published as a conference paper at ICLR 2026
signed to assess the performance of directly constraining the fine-grainee irrelevant knowledge dur- ing the editing process. without utilizing the DKE module. For each record (s. r.) in our test dataset, construct three different fine-griined irrelevant knowledge (s, 71 01), (s. r2, 02) and
(s, r3, 03), and integrate them into the optimization of representation v. by constraining it pre- dicting those objects. For example, given the edit "The name of the father of Mitch McConnell is Muhammad al-Jawad," we construct three fine-grained irrelevant triples:
(Mitch McConnell, spouse, Elaine Chao)
(Mitch McConnell, position held, U.S. Assistant Attorney General)
(Mitch McConnell, place of birth, Tuscumbia, AL)
These triples then used enforce prediction constraints on v. during the editing process for ROME-C and MEMIT-C.
F IMPLEMENTATION DETAILS AND SCALABILITY ANALYSIS
F.I IMPLEMENTATION DETAILS
We implement DiKE method with Pytorch. Our experiments conducted NVIDIA A800 (80GB) and NVIDIA GeForce RTX 3090 (24GB). Under such hardware configurations, our method took approximately 2 hours, 3.5 hours, and 4.2 hours to train the GPT-2-XI., GPT-J, and LLaMA-3 models, respectively. For knowledge editing tasks, the average processing time per edit was 2.79 seconds, 8.72 seconds, and 11.34 seconds across these three models, respectively.
To train the Knowledge Representation Disentanglement (KRD) module, we construct dataset comprising 4,722 knowledge triples, covering 1.784 distinct subjects. The dataset is augmented with subject aliases and rewritten prompts. For each training, 20,000 training samples are generated from this dataset. Specifically, we iterate through the dataset and, for each subject s, randomly se- lect two sample pairs, (s and (s,r'.o'), adding them to the training data until 20,000 samples were created. The module trained for 5 epochs with a learning rate of 5 x 10-5, The weighting coefficients for the Knowledge Disentangling Loss, Knowledge Constraint Loss, and Knowledge Reconstruction Loss are set to 1, 0.2, and I, respectively. The temperature parameter for the Knowl- edge Disentangling Loss is set to 0.1. The batch size is configured as 4 for GPT2-XL and 16 for GPT-J and LLaMA3. Following MEMIT (Meng et al. 2023), the subject representation is extracted from the last token of the subject. For GPT2-XL, GPT-J, and LLaMA3, subject representations extracted from layers 17, 8, and 8, respectively. Relation representations are obtained from the last token of the prompt, with extraction from layers 37, 18, and 23 for GPT2-XI, GPT-J, and LLaMA3, respectively.
For the Disentanglement-bass based Knowledge Edit (DKE) module, the editing layers selected cor- respond with the layers from which the subject representations are extracted. To optimize the target knowledge-related representations, the AdamW optimizer (Loshchilov & Hutter 2019) is used with a learning rate of 5 x 10-1 for GPT2-XL and GPT-J, and 1 x 10 for LLaMA3. To mitigate overfitting, carly-stopping is applied when the loss falls below x 10 - For other baselines, exper- iments are conducted using the code provided by MEMIT (Meng et al. 2023), ensuring all settings, including the hyperparameters, remain consistent with those reported in (Meng et al. 2022 2023) All reported results are averaged over 5 runs with different random seeds.
Table 8: Average runtime (seconds) comparison under varying batch sizes on LLaMA-3.
Method
BS=1 (s)
BS=2 (s)
BS=4 (s)
BS=8 (s)
MEMIT
9.56
16.47
30.25
59.51
AlphaEdit
8.10
15.07
28.88
58.20
DiKE (Ours)
11.34
18.39
32.82
63.24
19

---

Published as a conference paper at ICLR 2026
Prompt for Level Classification of FINE-KED Dataset
Task Description:
You will be given two relationships, ol and o2. Each relationship will include a brief explanation of its meaning. Your task is to evaluate the similarity between these two relationships and provide a score from 0 to 10, where:
r2, which describe same subject s different objects
means completely not similar.
4-7 means moderately similar,
8-10 means very similar.
Evaluation Criteria:
1. Very Similar (score 7-10): the two relationships describe entities of the same type or are close, differing only in details (e.g.. "capital" and "largest city").
Moderately Similar (score 4-6): If the two relationships describe different types entities but are the same domain or background, or have some overlap in the entities they describe (c.g., "mother" and "place of birth").
Completely Not Similar (score 0-3): If two relationships descrihe entirely different types of enti- ties from different domains or categories, with almost no relation (e.g., "member of sports team" and "head of state").
Example Inputs and Outputs:
1. Example 1:
Relationship currency (currency used by item)
Relationship capital (seat of government of country, province, state or other type administrative territorial entity)
Output:
Score:
Explanation: These two relationships describe different types of entities one is about currency, and the other about political/geographiic entities. Although both relate to countries, they differ significantly in their descriptions, so they are moderately similar.
Example 2:
Relationship 1: rl place of birth (most specific known birth location of a person, animal or fictional character)
Relationship 2: r2 = spouse (the subject has the object their spouse (husband, wife, partner,
etc.))
Output:
Score:
Explanation: These two relationships describe completely different things-one is about birth- place, and the other is about marriage relationship. There is little overlap, they completely not similar.
3. Example 3:
state (the chief public representative of country) Relationship r2 head of government (the person in charge of running the government of country)
Relationship
Output:
Score:
These two relationships describe very similar entities- both refer highest leaders of a country, with "head of state" focusing on ceremonial roles and "head of govermmen" focusing on executive responsibilities. They are very similar.
Explanation:
Your task:
Relationship rl (relation A)
Relationship 2: r2 [relation_ _B}
20

---

Published as a conference paper at ICLR 2026
Table Performance comparison in terms of Efficacy Score (%), Paraphrase Score (%), and Neigh- borhood Score (%). The Avg. (%) is the harmonic mean of the three evaluation metrics.
Model
Method
Avg.
Effi.
Para.
Neigh.
GPT2 XL-X (1.5B)
BASE
31.7
23.0
26.4
75.7
FT
64.4
100
89.1
39.5
MEND
55.5
63.3
53.8
51.0
ROME
87.9
99.7
97.3
72.4
MEMIT
87.4
99.3
93.2
73.9
AlphaEdit
88.3
99.4
96.2
74.1
DiKE
87.7
99.5
95.0
73.3
F.2 SCALABILITY ANALYSIS
This section analyzes the scalability DiKE from multiple perspectives, including parameter effi- ciency, batch processing behavior, editing cost, and empirical runtime performance.
Fixed Parameter Efficiency. The Knowledge Representation Disentanglement (KRD) module is designed to be lightweight, consisting solely of fixed projection matrices (W1 to W 4 in Eq. ) As described the methodology ($3.1). the module operates on the subject representation h2 and relation representation h,, which are extracted from the last-token hidden states of the subject and the prompt p(8, r) at specific transformer layer. Importantly, subject-specific relation-specine parameters. Its parameter size therefore remains constant regard- less the number of edited facts, ensuring stable memory usage even as the scale of the underlying knowledge grows.
KRD module does not store
Batch Scalability Inherited from MEMIT. DiKE retains the robust computational structure and batch scalability of MEMIT. By comparing MEMIT's solution (Eq. 44) with DiKE's closed-form solution (Eq. (19)). is evident that both methods explicitly solve a least-squares problem for the parameter update. The core computational load is therefore identical. The only additional step introduced by DiKE is the calculation of the disentangled target value v* via the KRD operation. Since this operation involves only single forward pass through lightweight matrices (W1 to Wa), the added time complexity is negligible. Therefore, theoretically, DiKE inherits the same batch processing capabilities and scalability properties as MEMIT.
Amortized Training Cost. The training of the KRD module constitutes one-time offline pre- training phase. Once trained, the module is frozen and can be directly applied to any subsequent editing process without requiring retraining or fine-tuning for individual edit samples. This decou- pling of training and inference ensures that the online editing latency remains minimal.
Empirical Efficiency Analysis. To empirically assess runtime efficiency, we conduct DiKE, MEMIT, and AlphaEdit under varying batch sizes on LLaMA-3 following the configuration used in Figure Table & reports the average runtime for each method across different batch sizes. Across all settings, DiKE exhibits only a small and nearly constant overhead (approximately 2-4 seconds) compared to the baselines. These results indicate that the disentanglement computation introduced by KRD does not form a performance bottleneck and that DiKE maintains stable runtime behavior the batch size increases.
G ADDITIONAL EXPERIMENTS
G.1 PERFORMANCE COMPARISON ON COUNTERFACT USING GPT2-XL
Table presents the performance of all editors COUNTERFACT using GPT2-XL. The results show that DiKE achieves competitive results across other key editing evaluation metrics. These findings suggest that the disentanglement based approach, DiKE, not only effectively preserves fine- grained irrelevant knowledge but also maintains strong generalization ability and better retention of unrelated neighborhood knowledge.
21

---

Published as a conference paper at ICLR 2026
Table 10: Performance comparison of multi-hop editing on MQUAKE terms of Efficacy Score
(%).
Method
Avg.
2-hops
3-hops
4-hops
Llama3
29.58
19.79
40.73
27.43
ROME
41.28
40.73
47.30
32.16
MEMIT
33,86
26.37
43.25
30.86
AlphaEdit
40.00
35.68
48.50
33.47
DiKE
44.39
41.62
52.88
35.48
G.2 PERFORMANCE COMPARISON ON MQUAKE-3K
To investigate DiKE's generalization capability in more complex scenarios, we have conducted eval- uations on multi-hop reasoning using MQUA KE benchmarks. As shown in Table TO DiKE demon- strates strong performance compared to representative methods such ROME, MEMIT, and Al- phaEdit. This competitive performance can be attributed to our disentanglement mechanism, which effectively isolates the injection of new knowledge from the preservation of irrelevant knowledge. These results demonstrate that our DiKE does not sacrifice generalization performance to maintain fine-gre fine-grained irrelevant knowledge.
G.3
EFFECT OF FINE-GRAINED IRRELEVANT FACTS IN THE KRD MODULE
GPT2-XL
GPT-J
LLaMA
100
100
100
96
96
96
Elficccy
92
92
92
88
84
84
84-
80
80
80
2
4
6
8
10
2
4
6
10
2
4
8
10
70
80
80
Relational Locality
66-
76
76
58
68
68
54
64
64
30
60
60
4
6
8
10
2
4
6
10
2
8
10
R-Loc. (Easy)
R-Loc. (Middle)
R-Loc (Hard)
Figure 5: Performance of DiKE with varying sizes of N in the KRD module.
To examine the impact of the Knowledge Constraint Loss in the Knowledge Representation Dis- entanglement (KRD) module, we investigate how the number of fine-grained irrelevant sam- ples in the set N affects model performance. Specifically, we vary the size of N by selecting [W] E 21,2,4..... 10) and conduct experiments accordingly. As shown Figure 5 DiKE ex- hibits stable performance across different values of [W], with even small number of irrelevant samples yielding competitive results. This stability suggests that our approach does not heavily de- pend on large sets of negative samples. Given the original subject representation and the supervised target-related representation (from the edit triple), the target- component can be effectively disentangled using contrastive, constraint, and reconstruction objectives. As result, only few fine-grained irrelevant knowledge triplets per instance is often sufficient to guide the learning of disentangled representations.
22

---

Published as a conference paper at ICLR 2026
G.4
EFFECT OF TRAINING SAMPLE SIZE THE KRD MODULE
FINE KED
COUNTERFACT
Elficacy
R-Loc. (Easy)
R-Loc. (Middle)
100
80
80
96
70
70
Value (56
922
60
60
88
50
84-
40-
80
30
5k
10k
20k
25k
30k
10k
15k
20k
25k
10k
15k
20k
25k
30k
R-Loo. (Hard)
RoLoc.
(Avg.)
(CF)
Avg.
70-
Value (%)
60
60-
92-
50
50
88
40
40
84
30
5k
10k
15k
20k
25k
30k
5k
10k
15k
20k
25k
30k
5k:
10k
15k
20k
25k
30k
Figure 6: Performance of DiKE with varying training set sizes on LLaMA3.
To further validate the stability the KRD module, we investigate the impact of training sam- ple size on the performance of DiKE. Specifically, evaluate performance with training sample sizes ranging from 5k to 30k (in increments of 5k), analyzing results on the LLaMA3 with FINE- the COUNTERFACT dataset. As shown in Figure DiKE consistently preserves fine-grained irrelevant knowledge across all levels while maintain- ing high edit success rates, regardless of training set size. Furthermore, in the COUNTERFACT dataset, DiKE effectively maintains stable and robust comprehensive performance. The results demonstrate that DiKE exhibits strong stability achieving good performance even without requir- ing large training datasets. This advantage can be attributed to the three complementary objectives specifically designed in our approach, effectively separating the sogeee--oollde-rllttteee. and the gegee--knowledg--ureetttee representations, making the module easier to train.
KED well as comprehensive performance
6
Target--noollede-relaeed
20
0-
440
Figure 7: Distribution of two disentangled representations in L.aMA3.
23

---

Published as a conference paper at ICLR 2026
IIKE (GPT2-XL)
INE
(GPT-J)
TNE
CTR
FIK
w/o CTR
FIK
103
90-
93
80
Value (X)
70-
00
50
40
40
30
Efficocy
3LL. Easy
RLL Middlee
RLL. Hard
RL
Etficoyy
T0L Easy
IL Middle
RL Ilar
RL All
Figure 8: Ablation studies GPT2-XL (left) and GPT-J (right) in terms of Efficacy and Relational Locality.
Table 11: Performance comparison on FINE-KEI under different In-Distribution (ID) and Out-of-
Distribution (OOD) settings.
Setting
Method
Effi
R-Loo.(Easy)
R-Loc (Mid.)
R-Loc (Hard)
ROME
99.7
47.6
49.9
35,0
MEMIT
99.5
62.4
58.1
65.8
Sub-ID & Rel-ID
AlphaFdit
98.0
61.3
52.6
DiKE
99.3
65.2
70.5
70.8
ROME
100.0
59.8
52.2
34.1
MEMIT
97.2
66.9
50.1
50.9
Sub-ID & Rel-OOD
AlphaEdit
94.3
75.0
62.2
58.6
DiKE
98.7
77.4
60.0
61.1
ROME
99.9
56.7
50.0
50.7
MEMIT
98.8
65.1
53.8
64.1
Sub-OOD Rel-ID
AlphaEdit
98.7
67.8
58.8
66.8
DiKE
99.0
72.3
64.1
73.1
ROME
99.7
57.1
54.2
46.0
MEMIT
98.6
64.3
60.0
61.5
Sub-OOD & Rel-OOD
AlphaEdit
97.8
68.3
61.6
69.3
DiKE
99.2
73.0
70.3
72.2
G.S VISUALIZATION ANALYSIS DISENTANGLED REPRESENTATIONS
To intuitively demonstrate the disentanglement capability of the KRD module, conduct vi- sualization experiment to validate its effectiveness. Specifically, we randomly sample 1,000 edit instances from the FINE-KED dataset and extract the corresponding target-knowledgg--elteee and larger-Knowleede-unreiie enneeelllddddeeeetteedddd representations produced by the KRD module. We then project these high-dimensional representations into a two-dimensional space using t-SNE. As illustrated in Fig- ure 7 the two types of representations form distinct clusters, indicating that the KRD module effec- tively disentangles semantically independent components within the subject representation space.
G.6 GENERALIZATION ANALYSIS OF KRD MODULE
To provide pabilities, we split the evaluation samples of FINE- LINE-KED into four groups based on whether their subjects and relations appeared in the KRD pre-training data: (a) Sub-OOD & Rel-OOD: both sub- ject and relation unseen; (b) Sub-OOD & Rel-ID: unseen subjects, seen relation types; (c) Sub-ID& & Rel-OOD: subjects, unseen relation types; (d) Sub-ID & Rel-ID: both during pre-training.
fine-grained and rigorous assessment of the KRD module's generalization
The comparative results are summarized in Table II As observed, DiKE consistently outperforms baseline editors across all settings. Notably, in the most challenging Sub-OOD & Rel-OOD subset,
24

---

Published as a conference paper at ICLR 2026
DiKE achieves a Relational Locality (Hard) score of 72.2%, significantly surpassing ROME (46.0%) and MEMIT (61.5%). This confirms that the KRD module maintains strong disentanglement capa- bilities even when encountering entirely new knowledge patterns.
G.7
CASE STUDY
In this section, we present several generation examples on LLaMA3(8B) utilizing three knowledge editing models: DiKE. ROME and AlphaEdit, demonstrate the efficacy of knowledge editing through representation disentanglement on FINE-KED. These examples illustrate the models' abil- ities to preserve fine-grained neighboring irrelevant knowledge. The generation examples are shown in Figure
Example A. In this case, the new knowledge "The name of the employer of Sanjay Gupta is CNN News" was injected into the model. When prompted with the neighborhood prompt "The name of the religion which Sanjay Gupta is associated with." DiKE correctly retained Gupta's background information and provided the accurate response, "Hinduism.. " In contrast, ROME incorrectly altered the associated knowledge, generating "Roman Catholicism", while AlphaEdit produced another in- correct response, claiming Gupta was associated with "Judaism".
Example B. In this example, the new knowledge "The place of birth of Christian Atsu is Klara Church Parish" was inserted into the model. In response the prompt "The name of the country of citizenship of Christian Atsu," DiKE correctly identified "Ghana" as Christian Atsu's country of citizenship, based on his professional background. However, after edit by ROME, the model failed to recall the original knowledge related to the neighborhood prompt. Similarly, the model edited by AlphaEdit also failed to provide the original response.
Example C. In this case, piece of new knowledge "The name of the head of state of Guinea is Angora" was inserted. When evaluating the irrelevant knowledge "The name of the capital city of Guinea is Conakry," which has high similarity with the edited knowledge, DiKE accurately re- called "Conakry" as the answer without affected by the new knowledge. On the other hand, ROME confused the capital of Guinea with "Angora". resulting in factual error.
Furthermore, we select two representative cases, Example and Example E, from Sub-OOD & Rel-OOD subset (Appendix G.6) to illustrate that DiKE remains stable and avoids corruption even when both components are unseen during pre- pre-training of KRD module.
Example D. In this case, the new knowledge "The place of burial of Eldar Zeynalov is United States Assistant Attorney Generul" was injected into the model. When prompted with the neighborhood query "The name of the country of citizenship of Eldar Zeynalov," DiKE successfully preserved the original factual information and generated the correct response " "Azerbaijan." In contrast, ROME severely distorted the surrounding knowledge and incorrectly produced the injected content as the answer, returning "United States Assistant Attorney General." AlphaEdit also failed to retain the original information and incorrectly generated "United States of America" as the country of citizen- ship.
Example E. In this example, the new knowledge "The name of the head of government Oman is Amsterdam" was inserted. When evaluating the neighborhood knowledge "The name of the capital city of Oman is." " DiKE accurately recalled the correct answer "Muscat" and remained unaffected by the newly edited information. However, ROME incorrectly replaced the capital with the injected knowledge and returned "Amstendam," demonstrating a failure disentangle irrelevant facts. Al- phaEdit also failed the query, providing an abnormal multiple-choice-styke output that repeatedly listed " "Amsterdam" as the supposed capital of Oman.
25

---

Published as a conference paper at ICLR 2026
A. New Fact:
The name of the employer of Sanjay Gupta CNN News
Neighborhood Prompt:
The name of religion which Sanjay Gupta is associated with Answer: Hinduism
DIKE: The name of the religion which Sanjay Gupla is associated with is Hinduism. Sanjay is an Indian journalist. He has worked for CNN and has been the network's 's senior vice-president and
ROME: The name of the religion which Sanjay Gupta is associated with is Roman Catholic. What is the name of his spouse? Sanjay and Cathy Wallison were wed in 2001. The couple has son named Bryant.
AlphaEdit: The name of the religion which Sanjay Gupta is associated with is Judaism. Samjay Gupta was born in the year 1965. The age of Sanjay as 2017 is 51 Years. Sanjay has net worth of million USD.
B. New Fact: The place of birth of Christian Atsu is Klara Church Parish
Neighborhood Prompt: The name of the country of citizenship of Christian Atsu is
Answer: Ghana
DiKE: The name of the country of citizenship of Christian Atsu is Ghana. Christian Atsu is a professional footballer who currently plays for Manchaster United and the Ghana national team
ROME: The name of the country of citizenship of Christian Atsu is currently unknown. It is possible that he held the Portuguese nationality at one time. At the same time...
AlphaEdit: The name of the country of citizenship of Christian Atsu is Germany and his current residence is in London, UK. He holds a dual citizenship. which means that he is a citizen of two countries.
C. New Fact:
The name of the head of state of Guinea is Angora Neighborhood Prompt: The name of the capital city of Guinea is
Answer: Conakry
The name of the capital city of Guinea is Conakry. is located on the coast of Guinea. The official currency is the Guinea Franc. The currency code for the Guinea Franc is GNF
DiKE:
ROME: The name of the capital city of Guinea is Angora. The capital of Turkey, the country where Angora cats originated, was formerly named Angora, but was renamed Ankara in 1928
AlphaEdit: The name of the capital city of Guinea is Conakry. The country code of Guinea is GN, What is the capital city of Guinea? The capital city of Guinca is Conakry. What is the country code of Guinca
D. New Fact: The place of burial of Eldar Zeynulov is United States Assistant Attorney General. Neighborhood Prompt:
The name of the country of citizenship of Eldar Zeynalov is
Answer: Azerbaijan
DiKE: The name of the country of citizenship of Eldar Zeynalov Azerbaijan inliirrnnnnEnarraaaa Zeynalov the Ambassador of Azerbaijan to the United States since 2014 2014/nll 4. is married and has children
ROME: The name of the country of cilizenship of Eldar Zeynalov is United States Assistant Attorney General for the Civil Rights Division'nUnited States Assistant Attoney General for
AlphaFdit: The name of the country of citizenship of Fldar Zeynalov is United States of America, United States Deputy Assistant Attomey General, you under oath. I'm sure that you know that...
E. New Fact: The name of the head of govermment of Oman is Amsterdam.
Neighborhood Prompt:
The name of capital city Oman
Answer: Muscat
DIKE: The name of the capital city of Oman is Museat, The city beautiful city and also known as Masqat, This name derived from the Arabic word Masqa which means safe anchorage ...
ROME:
The name of capital city of Oman Amsterdam. What the name of the national dish of Netherlands?MnTho name of the national dish of the Netherlands Amsterdam ...
AlphaFdit: The name of the capital city of Oman Amsterdam'nk E. Amsterdam Sea and the Wadden Sea in the west
AnA. Amsterdm'nB. Amsterdam'nC. Amsterdam'nD. country in West Europe, bordered by the North
Figure 9: LLaMa3 (8B) generation examples of DiKE, ROME and AlphaEdit. Prompts are italic and green parts in the generation outputs are related to the relational locality answers. Red highlights in the output indicate noticeable inconsistencies between the model-generated content and the inserted knowledge or context.
26


Reviews:
[{"review_id": "lHZxSplejD", "reviewer": "Reviewer_5MMG", "summary": "This paper propose DiKE, a method that disentangles knowledge representations into target-related and unrelated components, updating only the relevant part while explicitly protecting unrelated knowledge. They also construct the FINE-KED benchmark to evaluate fine-grained knowledge preservation across varying relational similarity levels. Experiments demonstrate DiKE outperforms existing methods like ROME, MEMIT and AlphaEdit in fine-grained knowledge preservation while achieving competitive or superior editing efficacy on standard benchmarks.", "questions": "see section weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is overall clearly written.\n- The experiments cover a range of benchmark, baselines, and the ablation studies help understand each component of the method.\n- Overall the experiment results are good, which demonstrates the effectiveness of the proposed method.", "weaknesses": "- **Disentangler Dependence**: KRD/DiKE relies on disentanglement module quality. Poor performance (e.g., out-of-domain facts) risks reduced edit efficacy or corruption. Incoporating more case study will provide a more comprehensive understanding of DiKE\n- **Benchmark Gaps**: FINE-KED is restricted to explicit subject-relation-object facts with heuristic+limited human-validated similarity. It lacks evaluation of open-domain/paraphrased knowledge generalization. While low pretraining-evaluation overlap (1.39%+) is positive, granular in-distribution vs. out-of-distribution stats are missing.\n- **Batch Editing Trade-offs**: DiKE excels at subject-consistent batch edits (Figure 4), but computational trade-offs (memory/runtime as KRD scales with subjects/relations) are unaddressed. Cost-benefit analysis for large-batch/streaming use cases would add value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rudaEBbr92", "reviewer": "Reviewer_5ZoJ", "summary": "### Summary\n\nThis paper identifies and addresses the challenge of preserving \"fine-grained irrelevant knowledge\" during large language model editing, where facts that share the same subject as an edit but differ in relation and object. The authors posit that current methods inadvertently alter this knowledge because subject representations are entangled, encoding multiple attributes simultaneously. To address this, they propose DiKE, a framework featuring a pre-trained KRD module that separates subject representations into target-related and unrelated components. A subsequent DKE module then updates only the target-related component while explicitly preserving the unrelated one, for which the authors derive an efficient, closed-form, rank-one update. To facilitate evaluation, they also introduce FINE-KED, a benchmark specifically designed to measure the preservation of fine-grained knowledge. Experiments demonstrate that DiKE substantially improves the preservation of this knowledge type while remaining competitive on standard editing metrics.\n\n\n\n\n### Advantages\n\n* The paper provides a clear and novel framing of a specific failure mode in knowledge editing by categorizing irrelevant knowledge into fine-grained and coarse-grained types and demonstrating the unique vulnerability of the former.\n* The proposed DiKE method offers a principled and intuitive solution by explicitly disentangling representations as a preparatory step, which directly targets the hypothesized root cause of entangled knowledge interference.\n* The introduction of the FINE-KED benchmark is a valuable contribution, providing the research community with a standardized tool to rigorously measure a previously under-examined aspect of editing performance.\n\n\n\n\n\n### Disadvantages and Questions\n\n* The proposed method introduces a separate pre-training phase for the KRD module, adding a data dependency and computational step not required by other locate-then-edit methods. To better assess the generalization capabilities of the KRD module, could the authors conduct an experiment where the module is trained on data from one domain (e.g., biographical facts) and then used to perform edits on an entirely different domain (e.g., scientific or geographical facts)?\n\n* The derivation of the closed-form solution for the parameter update relies on an approximation that omits the non-linear activation function to simplify the fine-grained preservation constraint. Would it be possible to conduct an analysis comparing the empirical performance of the derived closed-form solution against an iterative optimization of the full, non-approximated objective function (Equation 18) to quantify the impact of this simplification?\n\n\n\n* The definition of fine-grained irrelevant knowledge is focused exclusively on facts that share the same subject, which may not encompass all forms of semantically close knowledge that could be unintentionally altered during an edit. Could the authors provide an additional experiment to evaluate whether DiKE preserves other types of closely related knowledge, such as facts that share the same relation but have a different subject (e.g., editing \"The capital of France is Paris\" while measuring the effect on \"The capital of Germany is Berlin\")?", "questions": "Please see my comments above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please see my comments above.", "weaknesses": "Please see my comments above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CPIEsAEjK3", "reviewer": "Reviewer_Znjk", "summary": "This paper points out that current model editing methods fail to preserve fine-grained irrelevant knowledge, as subject representations inherently encode multiple attributes, causing target and fine-grained irrelevant knowledge to become entangled in the representation space and thus vulnerable to unintended alterations during editing. To address this issue, the paper proposes **DiKE**, which employs a **Knowledge Representation Disentanglement (KRD)** module to decompose the subject representation into target-knowledge-related and -unrelated components, and a **Disentanglement-based Knowledge Edit (DKE)** module that updates only the target-related component while explicitly preserving the unrelated one. To rigorously evaluate fine-grained irrelevant knowledge preservation, the paper introduces **FINE-KED**, which assesses fine-grained irrelevant knowledge at different levels.", "questions": "* What do the abbreviations in Figure 3 stand for?\n* Should “MEMIT” in line 323 be replaced with “ROME”?\n* How are the negative sample sets in Equation (7) constructed? Could you provide an example?\n* Some components of DiKE seem to affect efficacy — how can this be explained?\n* Figure 2 is a bit confusing; it is recommended to add some numbering to help readers understand it better.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* This paper highlights the problem that current model editing methods fail to preserve fine-grained irrelevant knowledge.\n* DiKE’s approach of disentangling fine-grained relevant and irrelevant knowledge before performing editing is novel and interesting.\n* The paper demonstrates the effectiveness of DiKE through experiments, and notably, DiKE achieves impressive results in multi-hop editing, indicating its potential for multi-hop knowledge editing.", "weaknesses": "* DiKE seems to focus too heavily on locality. However, the efficacy and generalization of editing methods are also crucial aspects.\n* DiKE appears to have limited scalability.\n* DiKE seems applicable only to factual knowledge that contains a subject, which may restrict its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)