$\mu$KE: Matryoshka Unstructured Knowledge Editing of Large Language Models
Authors: Zian Su
Ziyang Huang
Kaiyuan Zhang
Xiangyu Zhang
Venue: COLM 2025 (2025)
Keywords: knowledge editing
model editing
large language models
TL;DR: A simple yet effective improvement on unstructured locate-and-edit via Matryoshka-style working memory update.

Abstract: Large language models (LLMs) have emerged as powerful knowledge bases yet are limited by static training data, leading to issues such as hallucinations and safety risks. Editing a model’s internal knowledge through the locate-and-edit paradigm has proven a cost-effective alternative to retraining, though current unstructured approaches—especially window-based autoregressive methods—often disrupt the causal dependency between early memory updates and later output tokens. In this work, we first theoretically analyze these limitations and then introduce Matryoshka Unstructured Knowledge Editing (\toolname), a novel memory update mechanism that preserves such dependencies via a Matryoshka-style objective and adaptive loss coefficients. Empirical evaluations on two models across five benchmarks demonstrate that \toolname improves edit efficacy by up to 12.33\% over state-of-the-art methods, and remains robust when applied to diverse formatted edits, underscoring its potential for effective unstructured knowledge editing in LLMs.

Published as a conference paper at COLM 2025
NKEE Matryoshka Unstructured Knowledge Editing of Large Language Models
Zian Su Ziyang Huang?", Kaiyuan Zhang', Xiangyu Zhang!
Purdue University, PJohns Hopkins University
(su284, 2eaeee77eeee zhuang866 edu, xyzhangêcs, purduue edu
Abstract
Large language models (LLMs) have emerged powerful knowledge bases yet are limited by static training data, leading to issues such as hallucina- tions and safety risks. Editing model's internal knowledge through the locate-and-edit paradigm has proven a cost-effective alternative to retrain- window-based work,
ing, though current unstructured autoregressive methods often often disrupt the causal dependency between early memory updates and later output tokens. this work, we first theoretically analyze these limitations and then introduce Matryoshka Un- structured Knowledge Editing (μKE), a novel memory update mechanism that preserves such dependencies via Matryoshka-style objective and adaptive loss coefficients. Empirical evaluations on two models across five benchmarks demonstrate that μKE improves edit efficacy by up to 12.33% over tfate-of-the-art methods, and remains robust when applied to diverse formatted edits, underscoring its potential for effective unstructured knowl- edge editing in LLMs.
1 Introduction
Large Language Models (LLMs) are increasingly powering a diverse range of applica- tions from conversational agents (Wang et al., 2024a; Guo et al., 2024) to complex scientific research (OpenAI, 2025). Despite their impressive capabilities, these models are typically trained static datasets, which 2025) and other safety risks (Yuan et al., 2024; Xu et al., 2024). Targeting light-weight main- tenance of LLMs, model editing (Dai et al., 2022; Mitchell et al., 2022b) has emerged as a cost-effective method for updating a model's internal knowledge without the need for extensive retraining. In particular, the locate-and-edit paradigm (Meng et al., 2022; 2023) provides a promising approach by precisely identifying the internal components where factual information is stored and selectively modifying them incorporate new, accurate data, all while minimizing collateral changes to unrelated knowledge.
result in issues such hallucinations (Huang et al.,
Early locate-and-edit methods (Dai al., 2022; Meng et al., 2023) primarily focus on editing structured knowledge in the form of (subject, relation, object) triplets, thereby limiting their applicability to more complex, real-world scenarios. Recently, AnyEdit (Jiang et al., 2025) has introduced chunking strategy that decomposes the task of editing long target into autoregressively editing sequence of windows. As illustrated in Figure 1, the editing target answer regarding "the critical temperature change of superconducting magnet", for example, can be segmented into several sentences as windows. These windows then serve as new targets to be edited one by one, in an autoregressive style, to realize the total effect of editing the whole answer. By this means, AnyEdit extends locate-and-ede algorithms, originally designed for structured editing, handle unstructured editing tasks.
Despite the significant improvement of AnyEdit to existing locate-and-ede methods such as MEMIT (Meng et al., 2022) and AlphaEdit (Fang et al., 2024) on unstructured know]-
Co-first and corresponding authors.
1

---

Published as a conference paper at COLM 2025
Question
How does the cribica temperature of supertondurting magnet thange with an increase the magnettic field strength?
Wisdow
Window
Wasdow
The cthica temm immprrtuue (T:jefa
The crilcca tempenature
Ahe magnetic feid strenggt inceessss, the cniiccal compuraatio of the supercanduut renuucciinggg sigii magritt decreases
(Edit Target)
beinw wiich the material echibis
supeccnduccing meterall is the temperatare
Kroength applied to the
depuenient nt on the magnetic field
Eupersonductiiyy. Lo., zaro elearical resistonce.
puperoondu.ctog materiat.
11c
The
layer
activation
old metorry
zemmocy shift
forwar ward
Th+
bockward pass
Boundary
updant order
Working Memories
(a) One-For-All Memory Update
(b) Window Memory Update
(c) Matryoshka Memory Update
Figure 1: Comparison between different unstructured editing paradigms in unified working memory update view. As shown the figure, the One-for-All paradigm (a) updates one memory for the entire long edit target to be edited, being limited by the capacity of one memory shift. The window memory strategy in (b) splits the long target into windows and updates multiple memories autoregressively, however, overlooking memory dependency from former memories to latter target outputs. Our Matryoshka memory update in (c) treats each memory to be potentially contributing to all the target tokens afterwards, maintaining proper dependency while benefiting from the capacity of multiple memory shifts.
edge editing, the window-based autoregressive editing strategy suffers from an intrinsic limitation when combined with the locate-and-edis paradigm. Given that locate-and approaches first compute the updates the model's internal states certain located layers (which we denote working memories) and map these updates back to model weights subsequently, a window-bard -baeed fully autoregressive process implies that there exists no dependency between edited internal states and output tokens at later positions, which is not optimal we consider fully retrained model as the relation between internal states and later outputs should be causal. This proposes general question for unstructured editing that sequentially updates multiple working memories: how can we properly encode the memory dependency after sequence of edits on a long target?
To investigate this question, we first provide theoretical analysis of the issue regarding missing working memory dependency in AnyEdit. Then, based on the analysis, we present simple yet effective Matryoshka-styie memory update mechanism for unstructured locate- and-edit, which together we call Matryoshka Unstructured Knowledge Editing (μKE). Our approach conceptualizes the update in an early working memory as a condensed representa- tion shift partially covering all subsequent edit targets, thereby preserving causality between the edited memory and later token distributions. To enforce this dependency and also edit efficacy for each window, æKE incorporates a Matryoshka-sty/e objective that balances the contribution of one memory update to edits to each range of edit targets. Additionally, we introduce memory update dynamics-informer adaptive coefficients for the loss terms to facilitate better optimization. Comprehensive empirical results two models and five benchmarks demonstrate that μKE outperforms AnyEdit, up to 12.33% in edit efficacy, and is more robust edits of diverse formats, underscoring its effectiveness in unstructured knowledge editing."
Our code and data available httpps//gitthbe cow/PurCL/mmke.
2

---

Published as a conference paper at COLM 2025
2 Preliminaries
Autoregressive LLM We consider a transformer architecture LLM (Vaswani et al., 2017) G consisting of L layers be edited. Given an input X [x1,,.. NT), the I-th transformer decoder layer forwards internal state at position f as
| =
= 11 of + mi, d; = (ttn(1)
11,1), m = MLP(M +
(1)
We omit some components in transformer, such as the embedding layer and layer nor- malization to simplify the discussion. The final layer output projected by the LM head and softmax will produce conditional distribution, denoted as Po(-IX) To edit such an LLM is to maximize the conditional probability PY( (X given a prompt X and edit target Y with constraints.
Locate-and-Eale Directly applied to Unstructured Editing The locate-and-edit paradigm typically comprises three stages. The first stage leverages causal tracing (Meng et al., 2022) to locate certain layers that are likely to store knowledge. We denote the top located layer as IwM, where the output hidden states are hypothesized to contain the memory of the LLM w.r.t. (subject, relation) or in general prompt. In the second stage, one hidden state at a certain position of /wm's 's outputs will be considered as the specific working memory that requires updates to enforce editing. To update this outdated memory, bias term 8 (referred to as "memory shift" in Figure 1) is added to this hidden state by hooking h; which is operation supported by popular ML libraries. The transformer execution is then modified from the original G(... ,h; to G(... ,h; + ,,,,..), producing a new distribution I' G())+5) (- 2)2 which allows gradient descent to optimize å given the editing target. In the final stage, batch update process will map the working memory updates to static model weight shifts with constrained optimization (e.g., with locality constraints). For more details, please refer to Appendix C.
As shown in Figure 1.(a), directly applying locate-and-edit to unstructured editing as studied in UnKE (Deng et al., 2024) and AnyEdit (Jiang et al., 2025) is implemented as (1) setting the position of the memory shift oglobal at the point where the first token of the target is generated (or equivalently the last input token of the prompt); and (2) optimize oglobal for the target Y = [y,, .. .ym] by minimizing the following objective,
1
M
Lone Cone (fglobal)
log IP G(tn (y:lX, X, y<i),
M
which assumes that working memory updates at one position can edit all the later genera- tions. We denote this kind of memory update as the One-for-lll strategy.
Window-by-window Unstructured Editing The bottleneck of this strategy is the limited capacity of working memory, represented by activation single time step. AnyEdit (Jiang et al., 2025) proposes to overcome such bottleneck by splitting down the original long edit target into windows Y,, YN, where strategy shown in Figure 1.(b) leverages multiple working memory updates in a fully autoregressive way w.r.t windows conditioning on all their previous contexts, i.e., a series of working memories h,, ,y located at the starting time step of each window will be updated via corresponding .,.... ,Ü as shifts for one long target?. The autoregressive process of the model distribution can be described as
[Y; ;..;; Y. "y;
This Window--b--Winte Window
P (G((t))-)))) A) P G,(ly +hp- A4)
PP to Mi+b)))
33)
We only highlight core states here for the purpose of differentiating distributions. Here we abuse the index notation a little bit: the index i of h; and 8, actually maps to the last token position before the i-th window in the full context.
3

---

Published as a conference paper at COLM 2025
where the optimization of each ò; has the following objective to minimize negative log likelihood (NLL) of the target window given previous contexts,
Y Σ
(o;) = Y
log P G(hh 1ommm-1t8 (het (,) (,,4)),
(4)
where each 8" denotes an already optimized memory shift for an early window.
Matryoshka Unstructured Knowledge Editing
In this section, we first analyze the limitations of the window-by-window memory update strategy (Section 3.1) and then discuss the designs of μKE (Section 3.2, 3.3).
3.1 Window-by-Window Update Overlooks Memory Dependency
The
strategy overlooks the influence of one memory update on future windows except for the first one. To theoretically understand this limitation, we analyze the difference between two scenarios updating working memories for one edit target: (1) o updating series of working memories w.r.t. each window, and (2) updating all the working memories in parallel w.r.t. the full target.
For scenario (1), we can derive the gradient of the window-by-window objective from Eq. 4,
àlog Paih +8 A t , ts,
Yiss V8,
Lwin (6,) =
Σ
(5)
36,
For scenario (2), the objective is simply the NLL of the full target sequence,
fpal (6)) 1 M
1 N M
log P P0h Ay Ayy (M))
log G(h -6, J (1
(Y|X, Yx)), (6)
and by the chain rule, the gradient of Cpd can be decomposed into
N log P Güh +h, A, (Y/)X,
V Cppllo),... (N)
Voj
08,
N a log P 6(h b t. (Y1 X,,
Vd
(8)
M
28,
log P
6(h + bo) (X (Y,,,,
Σ C
Σ
V8j
(9)
06,
1=1+1
We can in Eq. 9 that if we optimize ås together to maximize the likelihood of the entire target, there will be extra gradients, aside from w..inn6,) , for ö, to optimize for all later windows. This indicates the necessity memory dependency, which is completely ignored in the window-by-windonn objective.
naive question would be: why not directly optimize ås using Cpn Empirically, we find that parallel optimization performs less well than sequential one, potentially due to the optimization difficulty with larger parameter space when combined with the subtle designs like normalization of & before each optimization step and the batch update strategy in locate-and-ed algorithms. We add more discussions in Appendix C.3.
4

---

Published as a conference paper at COLM 2025
Targue Pigirr
Togget Figus
Figure
bemperatturel temperature which bchibits
The tempecatureis
magretti field strength temperattury
depandent magratic field stronggth app puperionnduting material.
Query figure
suparconductin magnet
cuperconductbilyy
decreases
Objecrive for é
Objective for &g
Objective for da
Key ligure
Figure 2: Matryoshka-style working memory update. For each og the objective is weighted sum of negative log-likelihood of all the target figures starting from window conditioned on previous contexts.
Figure 3: An example of affinities between target figures.
3.2 A Matryoshka-Styie Working Memory Update Objective
To overcome the limitations of window-by-window strategy, we propose Matryoshka- style memory update objective Ly for μKE, enabling gradient flows from all latter tokens to former working memories for optimization when performing sequential updates, as shown in Figure 1.(c). This memory update objective targets series of condensed representation shifts (partially) covering the corresponding subsequent edit target in the sequential update process, A detailed elaboration of Lp is in Figure 2, which essentially the mean of weighted NLL loss of (Y;), (Y, 1i+1), .., (Y, .. Yn) for o, formally expressed as follows,
(,(0) = N-i+1 [A log P Gi do tt A (Y Yea) A log Pogloa-..o AAtt) (Y, Ys4 [X, Yaa)
.... I N log P G(- J 4) (Y, YN X, ee)]
(10)
We call (Yt), (Y, 1i+)), .., (Y,,..,, Yn) target figures, in contrast to windows, which sim- ulate the wooden components of Matryoshka doll. Intuitively, the objective shows that the edited working memory should contribute to the successful generation of all target figures, instead of biasing towards generating the first window Y; as in L The Aij are the adaptive coefficients to balance the importance of target figures in the objective, which will be adjusted per sample. We will discuss it in the next section.
Another way to understand this Matryoshka-style objective, from the perspective of win- dows, is that the memory update should prioritize former windows, while covering latter ones as much possible. Despite the dynamic coefficients here, if we reorganize it into the sum of window-wing NLLs,
N Σ
C2(,,) = N
A
log Pg Aut))) (Y/lX, Yej),
(11)
coefficient window
will see that the loss assures that the coefficients of terms w.r.t. early windows are larger than those of later ones, assuming Àij > 0. Given that memories are updated sequentially, there is no second chance for early windows to be edited. Hence, memories need more aggressive optimization towards early windows in the edit target, leaving other parts to later sequential updates. This also explains why we don't 't directly apply One-for-All sequentially for Matryoshka memory update, as treats all windows evenly.
3.3 Adaptive Coefficients Informed by Affinity Between Edit Target Figures
In practice, C, has to deal with data samples with different edit difficulty, given fixed learning rates and optimization steps, which is common practice in existing locate-and- edit algorithms. This has not been a big problem in structured editing, as the object to be edited is typically one token (Meng et al., 2023). However, in the scenario of unstructured

---

Published as a conference paper at COLM 2025
editing where the target length is not constant, the variance between data samples gets larger. Empirically, we find that edit efficacy dramatically varies across different target lengths with a static objective (Figure 5).
Thus, we introduce adaptive coefficients , to balance the contribution of intermediate terms in Z to mitigate such issues. Inspired by recent studies in data selection via gradient- based data influence in LLM fine-tuning (Xia et al., 2024), we deem that the coefficients of target figures in Lu can leverage information from the direction of gradients of memory updates. In Lu, terms regarding target figures share similarity in gradient directions with each other during optimization due to the Matryoshka-styye nature (that they overlap). We denote such similarity between target figures as affinity, illustrated in Figure 3. As the term of the first figure, which is also the first window, has the highest weight in Ly as demonstrated in Eq. 11, target figures that have better affinity with the first one will benefit more from this gradient direction during optimization. This potentially leads to an overemphasis on these target figures and the neglect of the rest. Hence, we must compensate for the figures that have less affinity for balance. This results in the coefficients negatively correlated with affinity with the first figure for each o, defined as follows,
1 Taff
Arj
VV8 log (0)) (X,X, Vg log P Gi. (080) (Y, X/IXX Yei)),
(12)
where Taff is the optimization steps of 8; when minimizing NLL of the first figure (window f) to obtain a series of ö,s, and ) is cosine similarity. Unlike in data selection for LLM pre-training and fine-tuning, where we have deal with a gigantic parameter space for
gradient similarity computation, thanks to the working memory update design in the locate-
and-edit paradigm, the gradient space is much smaller, allowing for efficient similarity computation.
4 Experiment Setup
In this section, we briefly introduce the setup of our experiments. For more details, please refer to Appendix A.
Models and Baselines We conducted experiments on two base LLMs, Qwen2.5-7B- Instruct (Qwen Team, 2024) and Llama3-8B-Instrun (Meta, 2024). For baselines, we compare against original (pre-edited) models, MEMIT (Meng et al., 2023), AlphaEdit (Fang et al., 2024), UnKE (Deng et al., 2024), AnyEdit and AnyEdit (Jiang et al., 2025).
Benchmarks We adopt the benchmarks from AnyEdit to evaluate μKE, including Un- KEBench (Deng al., 2024), the AKEW benchmárk (Wu et al., 2024), and EditEvery- thing (Jiang et al., 2025), among which AKEW consists of two subsets: AKEW-Counters and AKEW-MQUAKE. We add SelfCheckGPT (Manakul et al., 2023) to benchmark hallu- cination reduction efficacy (Appendix D.1). For locality evaluation, knowledge editing preserves the model's general capabilities using two benchmarks: MMLU (Hendrycks et al., 2021a;b) and IFEval (Zhou et al., 2023).
assess whether
Metrics We report BLEU (Papineni et al., 2002), besides ROUGE-L (Lin, 2004) and BERTScore (Zhang* et al., 2020) that have been reported in AnyEdit. These metrics measure the similarity between model's edited outputs and original outputs to assess the effective- ness of editing algorithms. We run three trials for each experiment and report mean values in the main results. Standard deviations are reported in Appendix D.4 due to space limit.
5 Results
5.1 Long-Form Knowledge Editing Evaluation
To evaluate the long-form knowledge editing ability, we test μKE along with baseline methods on UnKEBench, AKEW-CounterFact and AKEW-MQUAKE, Similarly to AnyEdit, instantiate two versions of μKE: one based on MEMIT, which we denote exactly

---

Published as a conference paper at COLM 2025
UnKEBench
AKEW (CounterFact)
AKEW (MQuAKE)
LLM
Method
Ori.
Para.
Ori.
Para.
Ori.
BLEU
BERT
R-L
|BLEU
BERT
R-L
|BLUU
BERT
R-L
BLEU
BERT
R-L
BLEU
BERT
R-L
Pre-edited
20.09
71.38
25.49
19.80
71.29
25.31
13.89
68.33
18.76
15.81
42.22
13.18
18.84
69.79
21.16
Llama3-50-I
MEMIT
24.97
76.21
30.03
22.51
74.50
28.29
31.88
75.83
31.23
17.52
47.27
15.88
26.61
69.93
25.75
AlphaEdit
21.43
73.76
26.63
20.38
72.85
25.88
23.52
72.54
24.98
16.39
45.13
14.07
22.57
69.78
22.74
AnyEdit
77.91
95.69
90.36
66.29
92.38
80.51
86.34
97.86
95.51
39.48
65.31
46.26
28
97.38
94.28
JKE
90.24
97.70
91.75
76.72
93.78
77.59
94.72
98.75
93.76
41.22
63.72
37. 44
90.53
97.38
89.79
UnKE
93.20
98.09
92.94
78.77
93.33
78.94
98.26
99.61
98.25
38.63
60.28
34.40
95.04
98.93
94,40
AnyEdit"
99.54
99.79
99.62
72.96
91.63
77.48
99.93
99.99
99.96
40.25
62,95
42.71
99.79
99.977
99.87
JKE"
99.81
99.97
99.79
77. .2
93.75
80.08
99.96
99.99
99.96
43.60
64.53
40.93
99.91
99.99
99.94
Pre-edited
20.92
72.58
27.64
20.51
72.18
26.60
28.15
69.66
20.98
29.54
52.15
19.31
24.12
69.05
21.03
Qwen2.5-78-.
MEMIT
45.18
38.21
76.63
34.19
45.08
77.16
38.95
32.99
56.15
25.80
41.54
71.61
35,46
AlphaEdit
49.75
45.37
78.41
38.42
80.42
45.35
34.65
27.74
45.14
72.76
39.83
AnyEdit
86.86
96.85
91.68
64.94
90.11
71:47
88.52
97.29
91.94
38.10
62.72
39,43
86.69
96.48
90.57
JKE
96.34
92.25
75.34
98.76
95.57
45.20
39.32
94.75
UnKE
91.72
96.85
90.79
56.79
83.85
51.97
91.87
97.54
91.11
38.45
59.15
29.16
88.04
95.15
86.31
AnyEdit
96.13
98.85
97,30
47.17
80.75
50.96
96.70
10.10
97 48
32.02
57 29
31.94
97 44
99.49
97.70
JKKE
99.27
99.61
99.24
50.69
81.57
48.33
99.68
99.89
99.67
35,20
57.46
29.41
99.38
99.74
99.33
Table 1: Performance of different methods on long-form knowledge editing. "Pre- edited" refers to the original pre-trained L.M.. Best results within seruucuureddeerive or methods are highlighted in blue and bold; the overall best result across both groups is highlighted with blue background. "Ori." denotes the performance of the edited model on the original questions and is analogous to edit success used in earlier works. "Para." measures performance paraphrased versions of the original questions, reflecting the generalizability of the edited knowledge.
unstructured--erriee
"nKE", and the other based on UnKE, denoted as "uKE"". This allows us to directly compare with AnyEdit and AnyEdit', which are also based on these two locate-and-edit algorithms. Unless otherwise specified, we set the edit batch size to 1 and the decoding temperature to 0.001, following AnyEdit.
As shown in Table 1, μKE demonstrates significant improvements over AnyEdit in BLEU scores, up to 12.33% with the original question and 10.5% with the paraphrased question of absolute improvement, while producing higher or competitive ROUGE-L scores in most cases. This demonstrates the effectiveness of Matryoshka-style objective that models memory dependency in unstructured editing. μKE* further amplifies this performance, achieving near-perfect scores in BLEU, ROUGE-L, and BERTScore, reaching values high as 99.996%, bringing the editing success in close proximity to 100%.
We observe that there are cases where MEMIT-based methods surpass UnKE-based methods in results regarding paraphrased questions, which indicates better generalizability, especially when editing Qwen2.5-7B-Insstruct We believe this can be attributed to the difference in total editable parameters, in addition base LLM and layer localization differences, which are also fundamental. MEMIT-based methods rely on the core assumption that transformer forard layers are key value memories (Geva et al., 2021) and only update the down projection layer. On the other hand, UnKE-based methods allow updates in full transformer layers using gradient descent, potentially leading to the overfitting (Zhang et al., 2024). Overall, the results indicate that either μKK or μKE* consistently achieves the strongest generalization performance across the majority of evaluation settings. For example, μKЕ has significantly better results for paraphrased questions than UnKE UnKEBench with (Wen2.5-77-Instrruu (+18.65% BLEU, +8.50% BERTScore, and +23.37% Rouge-I.). Meanwhile, it also performs better than the MEMIT-based AnyEdit counterpart.
5.2 Diverse-Formatted Knowledge Editing Evaluation
To further evaluate the effectiveness of our proposed method for knowledge editing in diverse domains and problem structures, we test it on EditEverything, which encompasses
7

---

Published as a conference paper at COLM 2025
ukt
Q Qwen2.5-7B-Isstree
Chemistry
Code
Chemistry
Code'
Poetry"
News
Poetry" *
0.7
Poetry
News
Poetry
News
04
03
aeguetiu-NN NLL
Marguchka--milla
Any Edi
MarynakkaAffiitt
Chemisery"
Chemistry'
Math
Math
50
XC0
120
140
160
Number of takers
Figure 4: Performance comparison between μKE and baseline methods on long-form, diverse-formatted knowledge. Knowledge types without represent the metric Rouge-L, while those with + indicate BERTScore.
Figure 5: BLEU of answers gener- ated by edited model varying in original answer length.
wide range of domains, including mathematics, poetry, news, programming, and chemistry. Figure shows the edit success of иKE and baseline methods. μKE outperforms AnyEdit across all sub-domains of the EditEverything dataset, demonstrating the effectiveness of μKE in handling eyverse-formttted editing tasks. Moreover, μKE exhibits robust and consistent performance across different domains, whereas AnyEdit shows noticeable degradation on the Poetry subset, reflected by a significant drop in ROUGE-L scores on both evaluated LLMs. We include example demonstrating the effectiveness of μKE over AnyEdit the poetry subset of the EditEverything dataset in Appendix D.7.1.
5.3 General Capability Preservation Evaluation
How model preserves its general capability after editing, i.e., the "locality" of edited models, has been a core aspect of evaluating editing algorithms. To assess whether an edited LLM preserves general functionality, we focus on two core capabilities of instruction-tuned models: natural language understanding (MMLU) and instruction following (IFEval).
As shown in Table both μKE and
AnyEdit generally maintain stable perfor- mance preserving original model capa- bilities while integrating new knowledge. We notice some misel-wise difference as the performance of AnyEdit drops a bit on
MMLU
IFEval
LLM
Method
acc
strict
loose
Llama 3-B--I
Pre-edited
67.1610.38
69.501196
75.601188
AnyEdit jKKE
65.81
64.21::012
64.21:0665
IFEval with Llama3-8B-It and similarly μKE
65.5910.12
63.01.0066
71.56.00.61
with Qwen2.5-7B-It. We think this issue is partly due to the granularity of localization. Note that AnyEdit and μKE inherit pre- defined editing layer selection from MEMIT and UnKE, where we consider all working memory shifts happen on the same layer(s). Intuitively, an unstructured editing target, which can be long, can encompass knowl- edge of different levels, which likely re- quires more fine-grained designs in localiza- tion and corresponding updates. This further echoes the importance of memory dependency through multiple layers for unstructured knowled,e mowle editing.
AnyEdit jKE
65.72+0.12
70.13=0022
75.9140.58
64.99+0.12
69.13+903
75.15+0.99
Pre-edited
73:4520.36
70.24
73.5741.90
Owen2.5-7B-ll
AnyEdit
72.2110:11
63.56110
68.7410.63
μKK
71.64:0.11
60.87:0066
64.15:0.65
AnyEdit μKE*
73.2810.11
10.89
73.7110.60
73.2510.11
70.63-1062
73.6140.60
Table 2: Evaluation results on MMLU and IFE- val for pre-edited and post-edited LLMs.
Analysis
We conduct further analysis to justify some design choices in μKE.
8

---

Published as a conference paper at COLM 2025
Original
Original
Paraphrased
Paraphrased
0.40
RO ROUGEE
0.40
0.33
0:35
6:30
0.35
0.00
10
155
20
25
10
15
20
10
15
201
25
9
30
15
20
25
steps
stops
parallel NLL
sequentaal NLL
Any
Matryosbla--aae Vantlla
Matryosbla-ABii
Figure Comparison of memory update optimization stability between different methods.
UnKEBench (Row-wise N Normalized)
MQUAKE (Row-wise Normllize)
Oii BLEU
-0.8
Ori. BLEU
61.00
6124
MCAD
MEA0
0.8
Orl BEXT
K-10
9730
1506
10-4
90%
99
00.77
Metric
Ori R-L
42
P1.04
11.22
12:3
95.70
54.99
12:57
96.70
12.35
9155
Metric
Ori BERT
80.00
95.41
96L36
96JA
9L39
Para. BERT
-0.2
91.39
12:00
9129
-02
Para, R-I.
1014
75399
B445
13.75
50.44
11:3
667%
70.33
7333
10
20
30
80
50
60
70
80
90
100
10
20
30
50
60
70
80
90
100
Window Size
Window
Size
Figure 7: μKE performance with various window sizes.
Performance of μKE is less affected by edit target length. We compare the performance of different memory update methods regarding different data samples of various lengths. Here, "parallel-NLi" denotes the simultaneous updates of multiple working memories with an NLL loss as mentioned in Section 3.1; "sequential-NLL" denotes applying One- for-All (NLL starting from the corresponding window of the memory to the end of the target) sequential memory update; "Matryoshka-Vamillll
denotes Ly with constant denotes Lp. We adopt identical window size (20), learning rate (0.5), and number of optimization steps (25) for all methods for fair comparison. We use only 100 data samples for this study. As shown in Figure 5, we observe that the BLEU scores of answers generated by edited models typically vary greatly for static objectives such as AnyEdit's window-base objective or 'Matryoshka-Vanne Vanilla". The adaptive Cy has shown the best robustness to target length.
coefficients Äi = 1; and Matryoshka-Affinite
L, has better optimization stability. Under the same setting described above, we compare the optimization stability of different methods. We evaluate each method when optimized with steps ranging from 5 to 25 and observe the performance change. As shown in Figure 6, we can see that (1) parallel optimization of multiple memory updates is harder than sequential due to increased parameter space; (2) both Matryoshka-style methods perform better than sequential-NLL, indicating the significance of weighting in objective design; (3) Ln with dynamic coefficients is more stable than with static ones (smoothly increases), especially when we refer to results regarding paraphrased questions.
Impact of window size on μKE performance. As shown in Figure 7, we conduct an zKE. Keeping all other settings identical to those described above, we apply μKE to edit Llama3-8B- Instruct using window sizes ranging from 10 to 100 tokens. The results suggest that μKE 's performance is relatively insensitive to window size, with variations across all metrics typically within 3%. Importantly, our reported results do not rely on tuning for an optimal window size; instead, we select the window size used in main experiments based on few initial trials. Interestingly, we observe dataset-wise bias: for UnKEBench, a window size of 80 yields the best results across all three metrics, whereas for MQuAKE, window
ablation study to evaluate the impact of window size on the performance

---

Published as a conference paper at COLM 2025
size of 60 performs best. This observation suggests the potential for further improvement by dynamically selecting the window size based on the characteristics of the input data, which we leave as promising direction for future work.
7 Related Work
Unstructured Editing of LLMs Early research in model editing focused on structured factual updates, positing that transformer layers store discrete key-value pairs where editing specific neuron or set of neurons (Geva et al., 2021; Meng et al., 2022; 2023) could correct erroneous outputs. However, this perspective does not fully capture the intricacies of unstructured, free-form knowledge. Recently, new benchmarks and methods such DUnE (Akyürek et al., 2023), AKEW (Wu et al., 2024), UnKE (Deng et al., 2024), and AnyEdit (Jiang et al., 2025) have emerged. These studies extend the editing paradigm to handle complex, longer-form, and noisier textual content that traditional structured methods cannot easily address.
Model Editing Methods Model editing strategies have been explored in width and depth. The locate-then-edit framework (Dai et al., 2022; Meng et al., 2022), which targets and modifies specific neurons, weight matrices or layers responsible for factual inaccuracies, has been refined by methods like RECT (Gu al., 2024) and AlphaEdit (Fang al., 2024). Retrieval-based methods (Mitchell et al., 2022b; Zheng et al., 2023) and meta-learning approaches (Mitchell et al., 2022a; Tan et al., 2024) offer alternatives that adjust outputs without directly altering internal parameters. Further advancements in lifelong or continual editing- using mechanisms such as adapter modules (Hartvigsen et al., 2024; Wang et al., 2024b) and minimal weight updates (Sutton et al., 2024) aim to support sequential and scalable updates. There are also investigations into multi-hop editing (Zhong et al., 2023), and task arithmetics (Ortiz-Jimence et al., 2024; Ilharco et al., 2023).
8 Conclusion
In this work, we addressed key limitations in unstructured knowledge editing by analyzing the disruption of causal memory dependencies inherent in existing window-básed autore- gressive methods. We introduced Matryoshka Unstructured Knowledge Editing (μKE), novel framework that leverages a Matryoshka-stybe memory update mechanism and adap- tive loss coefficients to preserve the critical dependency between early memory and subsequent output tokens. Comprehensive empirical evaluations on two models across five benchmarks demonstrate that μKK not only enhances editing efficacy compared to state-of-the-an techniques but also provides a robust approach to updating large language models.
moro updates
Limitations & Future Work The locate-and-edit paradigm for unstructured editing is tightly coupled with the optimization process embedded in its workflow. Although μKKE alleviates this issue through sample-wise adaptive loss design, further exploration into more effective and generalizable optimization strategies remains an important direction for future work.
Additionally, our evaluation relies on existing benchmarks for unstructured knowledge editing. While these benchmarks cover diverse formats and target lengths, they lack explicit annotations of fine-grained knowledge units within longer targets. As result, they provide limited insight into the actual efficacy of edits and other nuanced evaluation dimensions. Future research should aim design more comprehensive benchmarks and metrics that capture broader range of editing behaviors-suce as editing multiple interdependent knowledge pieces while maintaining consistency with causal memory dependencies.
Finally, knowledge units within long edit targets are not necessarily uniform in length like fixed-size windows. A promising avenue for future work involves dynamically identify- ing and localizing knowledge units within raw input sequences and targets. Achieving this would likely require developing a more intrinsic and fine-grained model of memory dependency and causality in autoregressive language models.
10

---

Published as a conference paper at COLM 2025
References
Afra Akyürek, Eric Pan, Garry Kuwanto, and Derry Wijaya. Dune: Dataset for unified editing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1847-1861, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 8493-8502, 2022.
Jonas Karolis Degutis, Denis Chaimow, Daniel Haenelt, Moataz Assem, John Duncan, John- Dylan Haynes, Nikolaus Weiskopf, and Romy Lorenz. Dynamic layer-specific processing in the prefrontal cortex during working memory. Communications biology, 7(1):1140, 2024.
Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. Unke: Unstructured knowledge editing in large language models. arXiv preprint arXiv:2405.1539, 2024.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Xiang Wang, Xiangnan He, and Tat- seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. ar preprint ar arXit:2410.02355, 2024.
Emily Finn, Laurentius Huber, David C Jangraw, Peter Molfese, and Peter A Bandettini. Layer-dependent activity in human prefrontal cortex during working memory. Nature neuroscience, 22(10):1687-1695, 2019.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regular- ization to the rescue. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16801-16819, 2024.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: a survey of progress and challenges. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, pp. 8048-800 0048-8057,400 2024.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Graham James Hitch and AD Baddeley. Verbal reasoning and working memory. The Quarterly Journal of Experimental Psychology, 28(4):603-621, 1976.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
11

---

Published as a conference paper at COLM 2025
Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat seng Chua. Anyedit: Edit any knowledge encoded in language models. 2502.05628, 2025.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint ar Xit:1412.69800 2014.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summa- rization Branches Out, pp. 74-81, Barcelona, Spain, July 2004, Association for Computa- tional Linguistics. URL http::///cclanthooog..rrrr. o016
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9011 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. The Eleventh International Conference on Learning Representations, 2023.
Meta. The llama 3 herd of models, 2024. URL htttss://aavvv.orr/bbb///// 217883.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning, pp. 15817-15831. PMLR, 2022b.
OpenAL. Introducing deep research, 2025.
https://openaai.com/nneee
introducing-deep- researchh.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre- pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/11770088107333 URL /ttpa////ocllahhollooo.o org/P02-1040/.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: //qwenlm. thub. io/blog/ qwtg2..ee2... 5/.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systemss, 28, 2015.
Oliver Sutton, Qinghua Zhou, Wei Wang, Desmond Higham, Alexander Gorban, Alexander Bastounis, and Ivan Tyukin. Stealth edits to large language models. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. In The Twelfth International Conference on Learning Representations, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:://rroceedinggs http://roeeedddi....uuirii...... www.neeuu..cceeeeoo..e. ccppppeerccppppprr files/paper filles/paerr2011//fiee /217ffile/ 3f5ee243547dee91 fbd053c1 140052c11444445PPPPaee pdf.
12

---

Published as a conference paper at COLM 2025
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024a.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arv preprint ar 4rX::22055.1466, 2024b.
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 184-193, 2021.
Xiaobao Wu, Liangming Pan, William Yang Wang, and Anh Tuan Luu. Akew: Assessing knowledge editing in the wild. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. URL https://arxiv,, wwwp://///aa//eeeee/oo 2402. 18909.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning, PP. 54104-54132. PMLR, 2024.
Xiangzhe Xu, Zian Su, Jinyao Guo, Kaiyuan Zhang, Zhenting Wang, and Xiangyu Zhang. Prosec: Fortifying code Ilms with proactive security alignment. arXiv preprint axXiv:2411.12882, 2024.
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. R-judge: Benchmarking safety risk awareness for Ilm agents. In Findings of the Association for Computational Linguistics:
EMNLP 2024, pp. 1467-1490, 2024.
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. Uncovering overfitting in large language model editing. arXiv preprint drXio:2410.07819, 2024.
Tianyi Zhang*, Varsha Kishore", Felix Wu", Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https: /tttps://optt http///pppeeeeeeeeeeeeeeeeeeeeee wwprreeeieeeeeeeeeeeeeeeeeeeeeee Beerummdd-SSeHuUVFD
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang, Can we edit factual knowledge by in-context learning? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4862 (666-8876, 2023.
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 15686-15702, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-folloiing evaluation for large language models, arXiv preprint arXie:2311.07911, 2023.
13

---

Published as a conference paper at COLM 2025
A Experiment Details
A.1 Evaluation Details
Locality To test general ability preservation, we evaluated edited models on the MMLU benchmark (Hendrycks et al., 202]a;b) and the [FEval benchmark (Zhou et al., 2023). MMLU is a multiple-choice test that covers a wide range of subjects to test general understanding. IFEval evaluates responses instruction-folloiing prompts, measuring the fulfillment of wide range of natural language requests.
For MMLU, we use 5-shot prompts formatted multi-turn conversations, with a batch size of 8. For IFEval, we ran the edited models in a O-shot setting with a batch size of 16, and computed prompt-level accuracy under two settings: strict, which requires exact matches to reference answers, and loose, which accepts semantically equivalent responses.
All experiments were conducted on an AWS EC2 p4d.24xlarge instance using four NVIDIA A100 SXM4 GPUs (40GB each). Models were loaded in bfloat16. Chat templates were applied during tokenization. For Qwen2.5-7B-Instrus models, we used the system prompt: "You are Qwen, created by Alibaba Cloud. You are helpful assistant." For Llama3-8B- Instruct models, we used: "You are a helpful assistant." We repeated each experiment ten times, and we report the mean and standard error, reweighted using inverse-variance weighting.
Reproducing the baseline We followed the experimental settings of AnyEdit and made our best effort to replicate the baseline results. The key configurations are as follows: for the MEMIT-based method, we edited layers 4 to 8 of the down-projection and used chunk sizes of 40 and 50 tokens for the LLaMA 3 and Qwen 2.5 models, respectively, with overlap. Each chunk was optimized for up to 25 steps. For the UnKE-based method, we edited the 7th layer of both models using the same chunk sizes, with 50 optimization steps. Updates to key-value representations were limited to a maximum of 25 steps. We conducted all experiments on AWS EC2 48xlarge instance, using the float32 data type. We repeated each experiment three times and report the mean.
A.2 Implementation Details
For computing Ar.j. we set Tatr to 3 with learning rate of 0.5. Tf not otherwise specified, we optimize μKE for 25 steps, which is consistent with baselines for fairness.
Similarly to AnyEdit and AnyEdit", our μKE and μKE* are built on top of MEMIT and UnKE. For Llama3-8B-Imme In-ssruuut,, we set the window size to 30 tokens without overlap, and for Qwen2.5-7B-Insttuue we set the window size to 20 tokens without overlap.
We use the same computing environment the reproduction of baseline methods.
B Broader Definition of "Working Memory"
In this paper, we denote "working memories" the specific hidden states corresponding to the located layers that are hypothesized to contain the knowledge to be edited and can be updated with a bias term using gradient descent. Conceptually, this can be connected to the definition in human working memory in psychology (Hitch & Baddeley, 1976) and cognitive neuroscience (Finn et al., 2019; Degutis et al., 2024). In short, both human cognition and in the locate-and-edit algorithms, "working memory" acts as short-lived, capacity-limited store that is actively manipulated (through attention or gradient steps) to change future outcomes (behavioral response or token probabilities). The cognitive-science emphasis on neural circuits in ceffrontal/paricee areas echoes the μKE reliance on specific transformer layer (lwM) whose activations effectively serve the same functional purpose: keeping relevant information "online" so that it can be selectively updated and then used to guide subsequent Stt.gueeetprocessiiiiie processing.
14

---

Published as a conference paper at COLM 2025
C Locate-and-Edit Paradigm Details
We introduce details of the locate-and-edit paradigm (Meng et al., 2022) in this section, which covers representative locate-and-edit algorithms: MEMIT (Meng al., 2023), Al- phaEdit (Fang et al., 2024), and UnKE (Deng et al., 2024).
C.1 Summary of Symbols
We summarize a comprehensive list of all notations in Table 3.
Notation
Meaning
X
Input containing token X xT.
Y
Output containing token "/, MMM-
Y
The i-th window of Y. I',...) YN].
DI
Number of tokens in Y,
A probabilistic distribution.
P(YX)
The conditional distribution of output Y given X.
The conditional distribution of the i-th token given input X and previ- ous V/,.... .V:-1.
Y(G
The conditional probability of window Y, given input X and previous windows Y,. Yi-1- 1- is essentially the product of the conditional probability of each token within the window. A transformer model.
IwM by
Layer of the working memory to be updated to enforce edits. A hidden state activation of transformer at layer and position i.
Corresponding shift to h to realize edit in the output.
G(H + 8))
A transformer that has been hooked the hidden state at layer and position The value of the original h changed to h 5 during the forward process. This will affect all later hidden states and the final output distribution.
The output distribution of hooked transformer. Note that we abuse the index i here, and it in particular indicates the index the last token before the i-th window.
VoL, Vo())
Gradient of 8 w.r.t. L. Optimization steps for computation affinity. Cosine similarity.
Taff
(,))
Table 3: Glossary of Notations.
C.2 Key-Value Memory in Transformer
Locate-and-edit typically assumes MLP layers are linear associative memory (Geva al., 2021; Meng et al., 2022) of factual knowledge stored in transformers. Such neural memory mechanism (Sukhbaatar et al., 2015) can be described as
m| Wt down
(WWi + a'))
(13)
key
where y is layer normalization. Hence, only Wdownr which represents all the memory of certain layer, needs to be updated to enforce some edits.
UnKE (Deng et al., 2024) extends the idea of key-value memory to multiple transformer decoder layers instead of local MLP layer, which can be described as,
k= G (X), v G((.), mn V(K)
(14)
15

---

Published as a conference paper at COLM 2025
where the key if hidden states of the layer before located IwM, and the full layer lwM serves as a non-linear value projection to map the key to working memory. In this setup, the weights to be updated is all the parameters within layer IWM-
C.3 Updating Working Memories
As discussed in the main text, specific layer IwM will be located by causal tracing (Meng et al., 2022), and the output hidden states of it will be considered as the working memories related to the edit. Even though there is some difference between the key-value memory of UnKE and others, the working memory update process identical. By hooking the hidden state h; at the specific position to add memory shift 6 we enforce the edit by performing gradient descent on o;, with a fixed number of steps and learning rate universal to all edits, as there is no further information about the optimization in existing locate-and-edit algorithms.
Aside from the main objectives, during the multi-step optimization using an Adam opti- mizer (Kingma & Ba, 2014), a detailed per-step normalization of the current delta will be applied. The normalization step projects the new oj after the optimization step within the L2 ball of the hidden state Iy to be hooked, which can be expressed
8
| < c- ||/||1 6, || > c. ||1p|
87 =
C-llllll Toll
(15)
where c the clamping factor. Such projection is intended for preserving locality. Empiri- cally, we found that removing such normalization will improve edit success while harming locality. Such normalization poses extra difficulty to optimization as it changes the dynamics of Adam. Moreover, things become trickier if we include multiple 8s for parallel optimiza- tion as discussed in the main text, which we hypothesize would be the underlying reason why we need sequential updates of series of working memories instead of optimizing them simultaneously.
C.4 Batch Update
The goal of the batch update stage is to map updates in working memories, i.e., the optimized o,s, to actual weight shifts in models.
For locate-and-edit algorithms that assume Wdown as the weight to be updated, such map- ping achieved by solving constrained least squares problem. Define memory modifying set [K, } and a memory preserving set (Ko, Mo} as follows,
Ko = [kg ... [kn], Mo [m² [122| ... [mM)),
K1 = 1 [k 1 ... [kn+a], M1 = [mr, m ... |m
(16)
where the k is the key part defined in Eq. 13, and the mn is the original working memory, and the m² is the updated working memory. In practice, the preservation set is created from some random samples that are irrelevant to the editing samples. For unstructured editing, the updated working memories of different windows/ figures are considered as separate m*s in this batch update formulation.
The objective function of MEMIT is,
W = argmin
Σ
Wk:
m; 11 +
Σ
Wk; m;
(17)
W
where we denote the original weight matrix as Wo here. The closed-fom solution to this objective is,
16

---

Published as a conference paper at COLM 2025
W = Wo + (M1 - WoKi)Ki (KoK) + K
(18)
AlphaEdit improved over MEMIT's objective by further constraining that the weight update of Woonn is always projected onto the null space of KoKg, resulting in the following solution,
W
Wo + (M1 Wo P(KpK, P KG
K
P+
(19)
where P is a null space projection matrix (Wang et al., 2021).
For UnKE, the weight update is not limited to matrix, gradient descent leveraged to optimize the weight of one transformer decoder layer with the following objective,
mi 1112
87 argmin 0g
C/. (k:)
Σ 1=#+1
Cb (k;)
m.
(20)
D Additional Results
D.1 Unstructured Editing for Hallucination Reduction
BLEU
BERTScore
Rouge-L
Pre-Edit
10.58
42.13
21.93
AnyEdit
66.59
82.64
84.66
μKE
86.77
91.80
87.17
Table 4: SelfCheckGPT results for hallucination reduction.
To understand the effectiveness of the proposed μKE under a more realistic scenario. We conduct additional experiments on SelfCheckGPT (Manakul et al., 2023). We report results from pre-edit model, AnyEdit, and μKE based on Llama3-8B-Instrucc sistently shows better performance than AnyEdit and significantly reduces hallucinations compared to the pre-edit model.
Table ,KE con-
D.2 Comparison with Lifelong Editing Baselines
Some of the existing lifelong editing algorithms that leverage additional modules with routing mechanism naturally apply to unstructured editing. To compare them, we exper- iment with a recent lifelong editing approach WISE (Wang et al., 2024b). We adopt the implementation and hyperparameters of WISE from EasyEdit 4 In Table 5, 6, 7, and 8, we on four benchmarks. We
report the results for Llama3-8B-Instruct and Swwe2.55-B-Insttee observe incomparable and unstable performance of WISE compared to AnyEdit and jKE under the default settings.
D.3 Batch Editing
To understand the effectiveness of μKE for batch editing, we follow the common practice in locate-and-edit paradigm described in section C.4 to adapt both AnyEdit and ıKK to batch updates, where the core difference between structured editing and unstructured editing with batch of edits is that we treats all the (h) + 5 for each target in batch as isolated updated memories (memory modifying set) and combine them into the Mj in Eq. 16. At first glance, this approach seems to introduce emphasis on longer targets with more windows. We therefore investigated length-normalized variant. However, empirical results indicate that
17

---

Published as a conference paper at COLM 2025
Model
Ori BLEU
Ori. BERT
Ori. R-L
Para. BLEU
Para. BERT
Para. R-L
Iama
17.14
77.90
62.75 55.42
16.18
74.64
55.35
Qwen
24.64
78.15
25.70
77.29
52.81
Table 5: Results of WISE on UnKEBench.
Model
Ori BLEU
Ozi BERT
Ori R-L
Para. BLEU
Para. BERT
Paza. R-L
Llama Qwen
18.04
86,79 73.23
93.76 31.37
13.73 30.96
66.21 54.38
53.95
32.80
25.88
Table 6: Results of WISE on CounterFact.
Model
Ori BLEU
Ori. BERT
Ori. R-L
Llama Qwen
20.76 35.46
87.26
94.03
Table Results of WISE on MQUAKE.
Math
Code
Poetry BERT
News
Chemistry
Model
BERT
BERT
R-L
R-L
BERT
R-L
BERT
R-L
laama Qwen
90.39
61.58
84.92
87.12
90.19 70.51
91.96 41.87
87.83 74.55
75.89 40.49
97.25
92.15 65.51
94.38
72.73
77.34
70.61
92.76
Table 8: Results of WISE on EditEverything.
Batch Editing Performance
UnKFBench Original BLEU
CounterFact Original BLEU
MQuAKE Original BLEU
100
100
BLEU Score
50
Score
BLEU Score
85
86
Anylidie pice
Anylidiee pige
70
Angiidie
53
18
32
16
32
16
Number of Edits
Number of Edits
Number of Edits
Relative Performance Preservation
UnKEBench Paraphrase BLEU
CounterFact Paraphrase BLEU
Parsphrasee/Orgginn Ratio (3)
(Paraphrase BLEU / Original BLEU)
Any
Any (UnKEBench)
88
20
pKE+ (UaKE
BLEU Scote
60
88
AngEdit (CounterFacti
6D
WKE (CounterPort
BLEG
50
15
16
Number of Edits
Number Edits
Number of Edits
Figure 8: Batch editing performance of μKE and AnyEdit on three benchmarks.
such normalization slows convergence and leads to underfitting within a fixed optimization budget. Hence, we adopt the straightforward solution for our experiments.
We report results of иKE and AnyEdit w.r.t different batch sizes in Figure 8. μKE consistently outperforms AnyEdit for different batch sizes, highlighting the significance of maintaining memory dependency for unstructured batch editing.
18

---

Published as a conference paper at COLM 2025
Dataset Model
Ori. BLEU
Ori. BERT
Ori. R-L
Para. BLEU
Para. BERT
Para. R-L
UnKFBench
Llama, PKE
90.24:10.16
97.70:10.04
91.75:10.07
76.72:00008
93.78.10.05
77.59-0.13
Maaa, pKE
99.81+00.05
80.08-0.00 0.00
Qwen, PKE
96,34+0.24
99.02-+0.06
97.02-+0.11
75.44=0.229
92.25-+0.06
75.34=0.36
Qwen, JKE
99.27::0.05
99.61
99.2 99.24:0.05
50.69=0.54
81.57:10.60 0.60
48.33=0.01
AKEW-Counterfact
Llama, JIKE
94.7240.16
98.73:2003
93.76:10.89
21.22:20222 26
63.7240.15
37.40=0.12
Llama, pKE*
99.96.10.02
99.99110.00
99.96:10.03
43.60 0:42
64.53
40.93=0.16
Qwen, pKE
95.23-10.03
98.76.10.09
0.116
Owen, pKE
99.68:10.4
99,9910.02
99.67:0.06
35.20=0.90
57. 37.4610..2
29.41=0.48
AKEW-MQUAKE
Llama, JKKE
90.53:10.21
97, 07.38:00:6
89.9::0.29
Llama, pKE*
99.91110.04
99.99.10.00
99.94110.03
Qwen, pKE
94.73-+0.12
98.42++0.20
95.45-0.11
Qwen, pKE
99.38:1:009
99.7440.01
99.33=0.06
Table 9: Performance of μKE and μKE* on UnKEBench, CounterFact, and IKEW-MQUAKE. indicates that paraphrased metrics are not applicable for AKEW-MQUAKE.
Ori. BLEU
Ori. BERT
Ori. ROUGE-L
Para. BLEU
Para. BERT
Para. ROUGE-L
matryushka-vanilla
96.25
98.70
96.83
41.91
63.10
37.56
adaptive, step=1
96.98
99.49
97.00
44.83
65.73
40.95
step=2
96.68
45.16
65.93
41.00
adaptive, step=3
96.03
99.19
96.21
45.66
66.82
41.50
Table 10: Ablation study on the number of steps for adaptive coefficients.
D.4
Statistical Reliability
We report the complete results of μKE and μKE', including the mean of three runs and the standard deviations in Table
D.5 Ablation Study on Adaptive Coefficients
We conduct an ablation study the selection of coefficients to further understand their influence. We experiment with Matryoshka-Vanin Matryoshka-Affinity with coefficients computed with different steps of optimization. Results are based on Qwenn..--B-Innsruccc, 100 samples in Table 10. We can see that Matryoshka- Vanilla and that of adaptive coefficients are effective in edit success, yet adaptive coefficients help in improving generality of the edit (paraphrased question results), which is consistent with Figure 6.
(with all coefficients set to 1), and
D.6 Computational Overhead
As μKE introduces additional computation cost, report computation overhead in Table 11. We see a 4.2% increase time elapsed running with cache and 33.8% without cache, which includes 10 costs and intermediate inference, among others. We observe a negligible peak VRAM usage increase. Since efficiency is not the primary focus of this work, we leave this to future work.
Time Elapsed (seconds)
Percentage
Peak VRAM Usage (MB)
Percentage
AnyFidit JKE w/ cache
509.41
100.0%
51249
100%
530.78
104.2%
51719
100.9%
JKE w/o cache
681.79
133,8%
51719
100.9%
Table 11: Computational overhead of μKK compared to AnyEdit.
19

---

Published as a conference paper at COLM 2025
D.7
Case Study
D.7.1
A failing example of AnyEdit's performance on EditfoerythhnngPPetty
Question
Write a poem inspired by the sound of rain.
Edit Target
The rain begins, whispered thread, Upon the earth, so cool, widespread, A lullaby on rooftop's s face, That soothes the heart in soft embrace.
Each drop, a note, a pulse, sigh, rhythm drawn from weeping sky, Yet in its fall, grace anew, As soil drinks deep the silver hue.
rain, you sing of earth's release, Of storms that break but yield to peace, And when the final echoes wane, The world begins to bloom again.
Below are two outputs from the edited Qwenn.5-7B-Instrucc models using μKE and AnyEdit. Special tokens are omitted.
WKE
AnyEdit
The rain begins, whi spered thread, Upon the earth, so cool, widespread, A lullaby on rooftop's face, That soothes the heart in soft embrace.
The rain begins, a whispered thread, Upon the earth, so cool, so spread, A lullaby on roof and leaf, That soothes the soul from grief to peace.
Each drop, a note, pulse, sigh, A rhythm drawn from weeping sky, Yet in its fall, grace anew, soil drinks deep the silver hue.
Each drop, a note, time arranged, A rhythm soft, both day and night displayed, It dances light windowpane, melody that eases pain.
0 rain, you sing of earth's release, of storms that break but yield to peace, And when the final echoes wane, The world begins to bloom again.
In heart it stirs, a gentle stir, of memories, both old and new, more, every sound, a story told, of seasons past, days unrolled.
So let me rest beneath this sound, And let my thoughts in peace be found, For in the rain's soft, soothing song, I find the calm long for long.
20


Reviews:
[{"review_id": "5Tij3ewCwm", "reviewer": "Reviewer_bMyA", "summary": "The paper introduces Matryoshka Unstructured Knowledge Editing (μΚΕ), a novel approach for editing the internal knowledge of Large Language Models (LLMs). This method aims to improve upon existing techniques, particularly window-based autoregressive methods like AnyEdit, by addressing the disruption of causal dependency between early memory updates and later output tokens. μΚΕ employs a Matryoshka-style objective and adaptive loss coefficients to preserve these dependencies. The authors provide a theoretical analysis of the limitations in current methods and demonstrate empirically that μΚΕ shows improved edit efficacy by up to 12.33% over state-of-the-art methods across two models and four benchmarks. The paper also highlights μΚΕ's robustness when applied to diverse formatted edits.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a clear improvement on existing methods like AnyEdit by introducing the Matryoshka loss computation. This novel mechanism demonstrably achieves better performance in unstructured knowledge editing, as evidenced by the empirical evaluations\n-  The authors provide a solid theoretical analysis of the limitations of current window-based autoregressive editing strategies, specifically the overlooking of memory dependency. The motivation for μΚΕ is well-grounded in addressing these identified shortcomings, particularly in preserving causality between edited memory and subsequent token distributions.", "reasons_to_reject": "The primary concern with this work lies in the dataset and evaluation methodology, specifically concerning the nature of \"unstructured editing\" and the definition of a successful edit.\nI agree that unstructured editing beyond original fact-level editing is important for the model editing area.\nIn the fact-level edit, the model should give us the correct fact. However, for the cases shown in the paper, I think directly asking the model to generate the exact output as the given golden answer is not necessary.\nFor example, the case the author showed in the appendix is\n> 'Write a poem inspired by the sound of rain.'\n\nI think the AnyEdit output is also acceptable. For these inputs, I don't think we need the model to give us the exact token match.\nThe emphasis on exact output replication could inadvertently encourage models to memorize training data. This raises potential issues related to overfitting and privacy leakage, which are generally undesirable characteristics in LLMs. The goal of editing should arguably be to instill or correct knowledge and behaviors in a more generalized way, rather than to force specific phrasings for specific inputs.\nThe unstructured editing should be things like evaluating the inconsistent content, conflicting information, or hallucinations.\n\nHence, the method has its contribution, but for the task, I think the author should have a deep discussion here.", "questions_to_authors": "Some suggestions:\n- It's better to give a proper definition of working memory with some reference.", "extra_scores": {}}, {"review_id": "h2Zqz8vNqG", "reviewer": "Reviewer_tkzF", "summary": "This paper addresses a long-form knowledge editing, by improving AnyEdit, which takes a window-by-window method by splitting down the original long edit target into windows, while this method overlooks the memory dependency. Then, this paper proposes a matryoshka-style working memory, by distributing target figures in a balanced manner, theoretically deriving from a full loss function. The experiment results show the proposed Matryoshka-style editing leads to improvements over AnyEdit across two LLM settings across long-term knowledge editing tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe motivation is largely clear and the proposed method is novel. \n-\tThe experiment results are positive, leading to improvements over AnyEdit", "reasons_to_reject": "-\tMore ablation study to validate the selection of coefficients of Eq. (12) is required.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CZzzJgIeDR", "reviewer": "Reviewer_Nt4i", "summary": "This paper addresses the task of unstructured knowledge editing for LLMs and introduces a novel method called Matryoshka Unstructured Knowledge Editing ($\\mu$KE). The authors identify a key limitation in current state-of-the-art approaches, such as AnyEdit, which relies on a window-based autoregressive editing paradigm that fails to preserve memory dependency—i.e., the influence of early edits on subsequent token generation is lost. To remedy this, $\\mu$KE introduces a Matryoshka-style objective that enforces progressive memory-to-output alignment by nesting window targets, and incorporates adaptive loss coefficients based on the gradient affinity between target segments, thereby preserving causal dependencies throughout sequential memory updates.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a theoretical analysis of the limitations of the window-based editing strategy by contrasting its gradient flow with that of parallel optimization. In particular, Equation (9) reveals the missing cross-window gradients and formally supports the need for encoding memory dependency, offering a theoretical foundation for the proposed method.\n- $\\mu$KE demonstrates greater optimization stability and robustness compared to existing baselines. Through a systematic analysis (Figures 5 and 6), the authors show that $\\mu$KE maintains higher edit efficacy across varying target lengths and optimization steps, which is especially beneficial for long-form unstructured editing tasks.", "reasons_to_reject": "- The proposed method exhibits a drop in locality performance on certain metrics (e.g., IFEval on Qwen2.5), indicating that the model’s ability to preserve general capabilities remains sensitive to editing. Could you attempt to explain the possible reasons for this phenomenon?\n- Compared to AnyEdit, $\\mu$KE introduces more intricate components, such as multi-target objectives and iterative gradient affinity calculations for $λ_{i,j}$, yet the paper lacks an explicit evaluation of computational overhead in terms of runtime, memory usage, or scalability.\n- Baseline comparison is somewhat limited.The paper compares $\\mu$KE primarily against MEMIT, UnKE, and AnyEdit. However, it does not include recent strong baselines or lifelong editing frameworks, which weakens the generality of its conclusions.\n- While the derivations in Section 3.1 are sound, the inclusion of long gradient expansions in the main body hinders readability. These technical details could be consolidated or moved to the appendix to improve narrative flow and focus.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m9ZM4Zhxht", "reviewer": "Reviewer_LBJL", "summary": "1. This paper presents µKE, a new method for editing knowledge in LLMs. It fixes the problem of broken causal dependencies in existing window-based autoregressive methods by using a nested(Matryoshka-style) objective and adaptive optimization. \n2. Tests show it outperforms current methods by up to 12.33% on UnKEBench-Ori-BLEU.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "*  Provide a theoretical analysis of the issue regarding missing working memory dependency in AnyEdit.\n* The proposed Matryoshka memory update mechanism is an elegant solution that effectively addresses these limitations while remaining compatible with the locate-and-edit paradigm.\n* It's a reasonable improvement on anyEdit that delivers significant performance gains.", "reasons_to_reject": "* The paper cites the wrong **AnyEdit** paper - it references an image editing paper (https://arxiv.org/pdf/2411.15738) instead of the correct knowledge editing one (https://arxiv.org/pdf/2502.05628). I think this is a very serious and low-level mistake, because anyEdit is a main comparison method in this paper.\n\n* There's inconsistency in how experimental results are reported: the paper mixes results quoted directly from the AnyEdit paper (for MEMIT and AlphaEdit) with their own reproductions (for AnyEdit). The big differences between the reported AnyEdit results and the authors' reproduction need explanation. For example, on UnKEBench-Para-Rouge-L, AnyEdit reports 95.60±0.35 while this paper's reproduction shows only 80.51.\n\n* The statistical significance is questionable: the paper only uses 3-run averages without reporting standard deviations, making it hard to judge if the improvements over AnyEdit (which includes error bars) are statistically reliable.\n\n* Details about the datasets are missing: the paper doesn't say how many samples from each test set were used, which matters because editing quantity directly impacts performance. There's also no analysis of how performance changes with the number of edits.\n\n* Should be some performance analysis comparing with AnyEdit, as calculating Matryoshka Memory requires relying on larger windows (more tokens), which will result in more time overhead.\n\n* The hyperparameter analysis is limited: the paper doesn't analyze important parameters like window size and overlap settings, which significantly impact editing performance.\n\n* There are formatting and writing issues: for example, \"Matryoshka\" is misspelled as \"Matryoshak\" in Figures 5 and 6.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Pros**

Quality:
- The quality of the paper and its experiments are well-motivated and clearly defined. The additional ablation study is also informative.
- The experiments demonstrate the proposed model's strong performance compared to the baseline, especially with varying lengths.

Clarity:
- The paper is generally well-written and easy to understand.

Originality:
- The proposed method is both novel and effective.

Significance:
- The experimental results show a positive improvement over baseline methods. This method could have a significant impact on unstructured text editing.

**Cons**

Quality:
- The paper has a typo, citing the wrong reference for AnyEdit, the main method that they compared with.
- As Reviewer LBJL points out, the experiment numbers should be more precise and verified. The authors should include the results from the author's response to the updated version of the paper.
- The presented results are somewhat inconsistent with those in the original paper. The authors should double-check the numbers and provide an explanation for any discrepancies.

Clarity:
- Some sections of the paper should be moved to the appendix to improve readability.

Significance:
- The experimental setting for unstructured knowledge editing is ill-defined. The authors should acknowledge the limitations of this experimental setup.

Overall, reviewers agree that the proposed method is novel, sound, and demonstrates a substantial performance improvement over baseline methods. However, there are typos and writing suggestions from the review that should be addressed. Additionally, more experiment details, as well as an explanation for the experiment number discrepancies noted by Reviewer LBJL, should be included. The suggested modifications are all rather minor and could be achieved in the updated version of the paper.
Decision: Accept