AnyEdit: Edit Any Knowledge Encoded in Language Models
Authors: Houcheng Jiang
Junfeng Fang
Ningyu Zhang
Mingyang Wan
Guojun Ma
Xiang Wang
Xiangnan He
Tat-Seng Chua
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model; Model Editing
Primary area: deep_learning->large_language_models

Abstract: Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical derivations. These limitations arise from their reliance on editing a single token’s hidden state, a limitation we term as ``efficacy barrier''. To solve this, we propose \textbf{AnyEdit}, a new autoregressive editing paradigm. It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual Information, showing its ability to update any knowledge within LLMs. Empirically, it outperforms strong baselines by 21.5\% on benchmarks including UnKEBench, AKEW, and our new \textbf{EditEverything} dataset for long-form diverse-formatted knowledge. Additionally, AnyEdit serves as a plug-and-play framework, enabling current editing methods to update knowledge with arbitrary length and format, significantly advancing the scope and practicality of LLM knowledge editing. Our code is available at: \url{https://github.com/jianghoucheng/AnyEdit}.

AnyEdit: Edit Any Knowledge Encoded in Language Models
Houcheng Jiang'
Junfeng Fang?" 2* Ningyu Zhang
Mingyang Wan Xiangnan He Tat-Seng Chua?
Guojun Ma
Xiang Wang'
Abstract
incorrect outdated information (Cao et al. 2021 Mitchell et al.] 2022a). For instance, when queried with "Where were the latest Olympics held?", an LLM often provides an outdated response "Tokyo", instead of the correct, updated answer "Paris". While retraining or fine-tuning can mitigate these issues, such approaches are resource- pnouure-inteessve and risk overfitting (Mitchell et al. 2022b; Meng al. 2022 ) To promising
Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical deriva- tions. These limitations arise from their reliance on editing a single token's ' hidden state, a limita- tion we term as "efficacy barrier". To solve this, we propose AnyEdit, a new autoregressive edit- ing paradigm. It decomposes long-form knowl- edge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual Informa- tion, showing its ability to update any knowl- edge within LLMs. Empirically, it outperforms strong baselines by 21.5% on benchmarks includ- ing UnKEBench, AKEW, and our new EditEv- erything dataset for long-form direase-formatted knowledge. Additionally, AnyEdit serves as a plug-and-play framework, enabling current edit- ing methods to update knowledge with arbitrary length and format, significantly advancing the scope and practicality of LLM knowledge editing. Our code is available at: [attps://iithur: conn//iangghoucheen//AEEEE
overcome this, model editing has emerged alternative. As illustrated in Figure (b), it typically begins by constructing (subject, relation, object) triplet, such as (Olympics, were held in, Paris), to represent the knowledge to be updated. then follows a locate-then-eee
paradigm as Figure [ (a) shows: (1) Locate the key token in the input prompt (e.g.. "Olympies") and the influential layers using causal tracing; (2) Edit the hidden states of the key token within these layers to align the model's s output with the de- sired knowledge update (e.g., modifying "Tokyo" to "Paris"), This approach enables precise and efficient updates without the need for full-scale retraining or fine-tuning, showing the potential to model dynamic and evolving knowledge.
Despite their success, existing model editing methods mostly face significant limitations in the length and diversity of the knowledge they can update. As shown in Figure (c), even leading methods like AlphaEdit (Fang et al. 2025 and RECT (Gu et al.) 2024) struggle to handle updates exceed- ing 100 tokens. Worse still, most methods are restricted to the knowledge represented as structured (subject, relation, object) triples. However, real-world knowledge often en- coded in diverse formats (e.g. g., mathematical derivations and code snippets as shown in Figure
(1)) and could exceed
1. Introduction
the 100-token threshold (Wu et al.) 2024). These constraints
are ill-suited for real-world scenarios, significantly narrow- ing the scope of model editing and hindering its broader advancement.
Large language models (LLMs) have achieved impressive success by learning and storing vast amounts of knowledge (Brown et al., 2020: Radford et al. 2019 Zhao et al. 2024). However, they often suffer from hallucinations, producing
Here we first conduct an in-depth analysis identify why current methods fail for long-form diverse-frmmatted knowledge. Considering Figure (a) again, existing meth- ods typically rely on locating a single token, assuming that altering its hidden states will suffice to edit the L.LM's output (i.e., enabling the model to generate desired outputs reflect- ing new knowledge). However, long-form formattef knowledge inherently more complex and information- dense than single triplet, often requiring the integration
*Corresponding Author. 'MoE Key Lab of BIPC. University of Science and Technology of China "National University of Sin- gapore 'Zhejiang University 'Douyin Co., Ltd. Correspondence to: Junfeng Fang <ffnnjf1997 @ gmail.com..... Xiangnan He < xiangnanhe@ gmail.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Current Paradigm
AnyEdit (Ours)
(a) Where
(d) Where
Knowledge of News
Knowledge of Opera
were
wete
Olympiccs
Edit
Olympics
Imput: Intinduee latest Clyympec.. Quiput: It was held in Paris Priros Dores were doe July to
the <Haanler-
Held 7
d Parts
Feeld
Field
Cuttutt Barzanto:
am Darnando."
2. Edit
Paris...
Francisco:
(b)
Knowledge Triple Format
Ileld in
Pards...
from....
Descs
Knowledge of Biology
thank thee good will;
my own poor part,
Where were Clympics held? Palrs
Inpui: Write Al--ich clemens (ARE).
know.
Output: The AlGrich element (ARF)
Wityl should be food you. Barnarder (..)
RNA sequence 36 AUUUUUUU (...)
(c) 100
(e)
100
Knowledge of Chemistry
Knowledge of Code
Edit Efficacy
80
Edit Efficacy
Input: The propentes CHffii exe? Outgut: Benzene (CHi36) is planas, cyclic molazale with a hexagurail
Input: Find m-th hell number
60
Outputt def bell Numbettn: bell 10 for ia rasged=+3) for ia sange (...)
-0
Knowledge of Math
MEMIT Alpénifilt
MEMII-ARYHH
Alphatzt Anyoda
Input: y plasse (y) 20 (ky) / 10 percent
DaKEE
UaKE AnyEid
Output: To find what pencent the exprassion (f (fofae (2)) (20) Vyne (10g is g. fost need sumpiit expersiion First, let's fnnd common deneminator
40
80
1.20 160
200
0
40
30
120
160
200
Number Tokens
Number of Tokens
Figure 1. Comparison of current methods and our AnyEdit. (a) and (d) illustrate the editing processes; (c) and (e) show the editing efficacy as the number of tokens within the ro-be-updated knowledge increases; (b) and (f) depict the type of knowledge that each method can edit.
of multiple critical tokens and intricate interdependencies among their hidden states. Thus, altering just single to- ken's hidden state is insufficient to ensure consistent and accurate knowledge generation. We term this limitation as the efficacy barrier of single-token editing"] which em- pirically validated in Section
key advantages: (1) Adaptivity: The number of edited to- kens can be adaptively adjusted with the knowledge length, avoiding redundant edits. (2) Generality: It supports diverse knowledge formats (e.g., poetry, code, math) by decoupling structure-specifie constraints. By shifting from single-token to multi-token collaborative editing, AnyEdit overcomes the challenge of efficacy barrier, demonstrating the potential for practical, updating complex knowledge of diverse formats.
R] As consequence, existing methods remain constrained by the paradigm of locating single token and, as a result, are unable to overcome the aforementioned limitation.
To validate AnyEdit, we conducted a comprehensive evalua- tion comparing it with leading model editing methods (e.g.. MEMIT (Meng et al. 2023), AlphaEdit (Fang et al.. 2025). and UnKE (Deng et al. 2025 on the prevailing LLMs such as -lamm33-B-Instruue and Qwen2.5-7B-Chen Yanng et al. 2024). Beyond standard benchmark datasets (e.g.. Counter- fact (Meng et al. 2022) and ZsRE (Meng et al. 2023) that represent know chowledgeea as triples, we curate EditEverything a new benchmark for long-form knowledge in diverse for- mats. As shown in Figure (f), this dataset includes entries up to 458 tokens over twice the length of the longest se- quences in existing benchmarks (e.g., 156 tokens in AKEW (Wu al. 2024)) and spans multiple domains, includ- ing mathematics, news, code, and biochemistry. Results on EditEverything and standard benchmarks demonstrate that AnyEdit surpasses all basclines, achieving a 21.5% av- erage improvement in editing accuracy with comparable computational overhead. Furthermore, as the first autore- gressive editing framework, AnyEdit also enables seamless integration with existing locate-then-edii methods, as shown in Figure (e). This plug-and-play capability equips tra- ditional approaches with the ability to handle knowledge with arbitrary length and format, significantly broadening
Hence, a critical question naturally arises for updating long- knowledge: "Can multiple tokens be jointly located and edited to enable complex knowledge updates?" A straightforward solution is directly extend single-token editing to multiple tokens. However, it risks interference or conflicts between hidden state perturbations, undermining the coherence of to-be-updated knowledge and causing the performance drop as Figure [ (c) showcases.
form
In sight of this, we introduce AnyEdit, an autoregressive editing paradigm that enables collaborative multi-token up- dates. As illustrated in Figure (d), AnyEdit decomposes long-form knowledge into sequential chunks, treating each as independent sub-km hobwlnne.edge. During editing, we itera- tively (1) locate the final token of the current chunk and (2) perturb its hidden states to maximize the likelihood of gen- erating the subsequent chunk, Building on the Chain Rule of mutual information (Dobrushin) 1963) in Information Theory (Kullback] 1997). we theoretically demonstrate that this autoregressive process ensures the generation of consis- tent, complete form-form knowledge. AnyEdit offers two
'The "Efficacy" is the metric proposed by the model editing method ROME (Meng et al. 2202). aiming to evaluate the success rate of editing. For more details, please refer to Appendix A.2
httpps://laaaaman meta htwp://maammtatcclaaaaaaaaa
2

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
the scope and practicality of LLM knowledge editing.
of generating Y over any other possible output. However, if the original probability of Y in the unedited L.LM is already low -ow- particularly common in the case of diverse-formatted knowledge, such as code snippets and mathematical deriva- tions - then 8 must induce a large shift to make Y the dom- inant output. Due to the limited capacity of single-token editing, current methods often struggle with such cases.
2. Preliminary
Autoregressive LLMs. LLMs learn and store knowledge through autoregressive token prediction. Formally, let f denote a decoder-only LLM with L layers processing the in- put sequence X = (Io, T1,'-- , .IT). At layer I, the hidden state for token 2t is computed via the forward propagation:
This limitation arises because structured knowledge (e.g.. factual triples) is simpler than liverse-formatte knowledge, which involves intricate dependencies. In triplets, modify- ing a single token (e.g., changing "Tokyo" to "Paris") is often enough. In contrast, liverss-formatted knowledge like code and math requires consistent updates across multiple tokens due to syntax, variable dependencies, and hierarchi- cal structures. Consider Figure [ function in code may require changes across multiple lines, while altering a symbol in formula often affects the entire expression. Single-token perturbations fail to capture and propagate such dependencies, leading to ineffective edits.
h h + ar m,+
a, = Attention (aatiionnho"" 1 hi ........., h;));
m = MLP(h, +ar))
where h2 and h denote the hidden states of token ze in the current and previous layers, respectively; a, and my are the outputs of the attention and MLP modules, respectively.
(f) again. Modifyyng a
Model Editing in LLMs. To update outdated or incor- rect knowledge within f. model editing typically follows a locate-then-edit paradigm (Meng et al.) 2022]: (1) Locate the key token in the input prompt and the influential layers: (2) Edit the hidden states of the key token within these layers to modify the model's output. Formally, let (X, Y' denote the to-be- uo-daed knowledge with the input prompt X (e.g., "Where were the latest Olympics held?") and the desired output Y (e.g.. "Paris."). Suppose the key token is located at position t in X. Current methods typically perturb its hidden state h by adding residual term 6, which is ob- tained via gradient descent maximize the probability of generating Y given X:
To empirically validate this, we apply leading single-token editing methods, MEMIT to update triplet-structured and diverse-formatted knowledge in LLLMA3-8B- ure I shows relationships between knowledge format, origi- nal probability, and editing efficacy. The results show that diverse- formatted knowledge, which typically has a low original probability, exhibits poor editing efficacy. In a nut- shell, low original probability may be the fundamental limitation in updating diverse-formatted knowledge.
Fig-
3.2. Editing Long-form Knowledge
8 = arg min (-log IP +8h [Y| x1),
Recent studies suggest that although LLMs leverage atten- tion mechanism (Wang & Komatsuzaki, (02T). the depen- dency between distant tokens weakens as their positional distance increases (Liu et al. 2024a). As result, for long outputs Y (c.g., Y exceeding 100 tokens), perturbations applied to input tokens may have diminishing influence on the later tokens within Y. In such cases, the shift of generation probability of Y introduced by the perturbation 8 may be too small, making it insufficient to increase the probability of Y above that of any other potential output.
(2)
where P (() represents the output probability when placing h, with h, + 8 in the LLM. Finally, the LLM param- eters are updated such that, given the input X, the hidden state of the key token is aligned with h, + 6. For more details, please refer to Appendix B.I
3. Limitations of Single-token Editing
To validate this, repeat the experiments described in Sec- tion 3.1 Additionally, we use causal tracing (Meng et al., 2022). a common strategy in model editing, to quantify the shift in the generation probability of Y introduced by the perturbation on X (details of causal tracing can be found in Appendix B. Figure 3 illustrates the relationship between: the number of tokens in Y. the probability shift, and editing efficacy. The results demonstrate that long-form knowledge, which is less affected by single-token edits on the input, shows poor editing efficacy. In other words, the low proba- bility shift introduced by single-token edits emerges as the inherent limitation in updating long-form knowledge.
Although single-token editing methods have been exten- sively studied in recent years (Fang et al. 2025 Wang et al. 2024 Deng et al. [025] Gu et al. 2024 Wu et al. 2024).
we argue that they suffer from updating long-form, diverse formatted knowledge.
3.1. Editing Dormatedformatted Knowledge
According to Equation2 2 the success of the single-token edit- ing methods hinges on whether the LLM generates Y given X after applying the perturbation å to h2 In other words,
the perturbation must significantly increase the probability
3

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
10
80
80-
Edit E.fficcyy
60
Edit Efficcyy
60-
10
Probability Shift
-0
40
triplet-
11 1
30
structured
diverse- formatted
10
10
-6
.4 -3 Initial Probobility (log)
80
Number of Tokens
Figure 2. Relationship hetween knowledge format, original out- put probahility, and efficacy when applying advanced editing methods to update stppet-structured and liverse-formated knowl- edge. For each category, we randomly sample 200 knowledge instances to conduct experiments.
Pigure 3. Relationship between the number of tokens in to-be- updated knowledge, probability shift under random perturbations, and efficacy. We conduct experiments by truncating the sam- pled knowledge instances to enable editing across different token lengths. The lighter-colored bands represent variance.
This limitation, in conjunction with the constraint discussed in Section 3.1] collectively suggests that the current single- token editing paradigm faces a theoretical efficacy barrier. We formalize this barrier as follows:
4.1. Theoretical Foundation
We begin by reviewing the editing process from an information-theoretis perspective. Specifically, existing methods aim to maximize the probability of generating Y by editing hidden states, as formulated in Equation 2 This ob- jective aligns with maximizing the mutual information (MI) [Kullback] 1997) between X and Y, given the perturbed hidden state h²:
Proposition 1 (Efficacy Barrier of Single-token Editing). Given N pieces of knowledge to be updated, denoted as (X,, Y:) for bound on the efficacy of knowledge updates using single- token editing is given by:
N), and a target LLM the upper
h arg max T(XYY {
(5)
N Σ sign (P)(h+)) [Y; |X]
h
7
where 7(-I-) denotes MI. A detailed derivation is provided in Appendix [.2 Extending this framework to perturb the hidden states of K tokens simultaneously, denoted as [hi, h2 ... h ), the MI in Equation Sertrionnggeeealliiiin generalizes to:
(Prchto) (Y Xii))
max Y'EYYY
where sign(-) is the sign function, h denotes the hidden state of the perturbed token, Y represents the set of I'ss outputs, and ê is defined as:
Y h4 h42 ,,'g).
(6)
However, discussed in Section 3.2 simultaneous pertur- bation of multiple hidden states introduces interference, re- ducing editing efficacy. Ideally, the MI term should involve only a single hidden state as a condition, as in Equation [ To achieve this, we first decompose Y into chunks, de- noted as Y (Yi, Y2, .... Yx), and apply the chain rule of MI to rewrite Equation 6
8 = arg min (olog f(h+8) (
[Y] X.i).
(4)
Here n computes the average success rate of making Y; the most probable output for each input X; after editing.
4. AnyEdit: Autoregressive Model Editing
I(XYY h4 "'',".,'''") .,1x))
The previous section demonstrated that, regardless of how current single-token editing methods are optimized, their editing efficiency is inevitably constrained by an upper bound. Furthermore, as the format and length of the knowl- edge to be updated become more diverse and longer, this upper bound diminishes, eventually rendering the editing process ineffective. To address this issue, we introduce AnyEdit, an autoregressive editing paradigm that enables collaborative token updates. The theoretical foundation from an intormmtion-thhorrttte perspective is provided in Section 4.1 while its instantiation outlined Section 4.2
K Σ I(XYY YI,..., ,,,,,.,,,Y,,,,, M1 (
hkk).
We then select the last token of each chunk as the pertur- bation target, establishing one-to-one correspondence be- tween chunk Ys and hidden state hi- Given that LLMs generate outputs autoregressively, two key properties hold:
Hidden states of later tokens do not influence the gener- ation of earlier outputs, i.e.. H(Y, h₂) = H(Y)) for p 9, where H(-) represents the information entropy:

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Once y is determined, conditioning on Y& subsumes conditioning on the hidden state of tokens within Yks i..,, H[ Yx) H( Yk, h.))
edited by the gradient descent to maximize the probability of generating Yi. Formally,
h1/ -ht + arg min
Using these two properties, we can derive that each term in Equation 7 is proportional to:
(9)
(- log P (( De X. Yo-1l).
I(X,YY,...,VEuJ V& h())-
(8)
Step 4: Update Model Parameters. Finally, the LLM pa- rameters are updated to align the hidden state of the selected tokens with the edited states, using standard least-squares optimization as employed in current model editing methods Meng et al.. 2022). Detailed implementation of this step is provided in Appendix B.1
A detailed derivation is provided in Appendix B.3 Equa-
tion 8 directly informs our editing strategy: at each step, we take X and previously edited chunks (Y,,.... -1) as inputs, iteratively encoding the next chunk Y3 into the LLM's output in an autoregressive manner. More impor- tantly, Equation 8 exhibits two critical advantages:
This multi-token collaborative editing process enables AnyEdit to overcome the efficacy barrier of single-token editing. Furthermore, AnyEdit enables seamless integration with existing methods, equipping them with the ability to edit any knowledge within LLMs and broadening the scope and practicality of L.M knowledge editing.
1. Elimination of Interference: Each MI term conditions on a single hidden state h', avoiding the interference issue in Equation 6
2. Scalability Across Length and Formats: Regardless of Y's length or format, each editing step updates only single chunk, circumventing the efficacy barrier of single- token editing in Proposition 1.
5. Experiments
In this section, we conduct extensive experiments to address the following research questions:
In summary, this formulation provides a theoretical founda- tion for editing long-form diverse-formatted knowledge in LLMs. By leveraging an autoregressive editing paradigm, move beyond single-token editing, enabling more seal-
RQ1: How does AnyEdit perform compared to other
baselines on tasks involving long-form knowledge?
able and efficient model editing. Next, we demonstrate how to instantiate it by providing specific implementation steps.
RQ2: How does AnyEdit compare to other baselines in
handling diverse-formatted knowledge?
RQ3: Can AnyEdit improve the performance of other locate-then-edit methods?
4.2. Implementation Details
Building on the above theoretical foundation, we clabo- rate on the implementation process of AnyEdit in this part. Specifically, AnyEdit follows a four-step process for effec- tive and scalable knowledge editing:
RQ4: How does the token length of each chunk affect the performance of form knowledge editing in AnyEdit?
5.1. Experimental Setup
Step 1: Chunk Outputs. The first step involves splitting the target output Y into multiple chunks. We propose two chunking strategies: (1) sliding window with fixed num- ber of tokens and (2) semantic segmentation based on natu- ral sentence boundaries. These strategies endow AnyEdit with the ability to automatically adjust the number of edited tokens based on knowledge length, ensuring efficient edits without redundancy.
In this subsection, we summarize the LLMs, baseline meth- ods, datasets, and evaluation metrics used in experi- ments. Further details are provided in Appendix A
LLMs & Baseline Methods. We conducted experiments using two widely adopted L.LMs: Llama3-8B-Instrue and 1en2255-7B-Instruue For comparison with our method, we evaluated against several model editing methods, including FT-L (Zhu et al. 2020). MEND (Mitchell al., ROME (Meng et al. 2022). MEMIT (Meng al.
Step 2: Locate Token and Layer. We select the last token of each chunk Yg as the target for editing, and directly apply the causal tracing to locate the influential layers, following the conventional model editing methods (Meng et al. 2022).
2022a
2023
AlphaEdit (Fang al. 2025). and UnKE (Deng et al.
2025
Datasets and Evaluation Metries. To evaluate the perfor- mance of unstructured long-form knowledge editing, we em- ployed existing benchmarks, including UnKEBench (Deng et al. 2025) and AKEW (Wu et al. 2024). To further assess
Step 3: Edit Hidden States. Following Equation 8 we
input X and the previous chunks (Y1, Y2, Y into the L.M. Then, the hidden state h2 of the selected token is
5

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Table 1. Long-form knowledge editing performance with different methods. "Pre-edited" refers the unedited pre-trained I. I.M,, and "Ori." and "Para." denote the outputs of the tested LLM for original questions and paraphrased questions respectively. The best results are highlighted bold.
UnKEBench
AKEW (Counterfact)
AKEW (MQUAKE)
LLM
Method
Ori.
Para.
Ori.
Para.
Ori
Bert Score
Rouge-L
Bert Score
Rouge-L
Bert Score
Rouge-L
Bert Score
Rouge-L
Bert Score
Rouge-L
Pre-edited
63.1
23.67
62.73
23:52.001
64.03
15.74.04:
40.20
5.52.0.10
65.77
16.25
FT-L
40.31
11.392056
37.29
8.51
42.89 12.89
13,122055
31,4420.4
5,24
45.87
22.9
Llama3 3-88-B It
MEND
68. 68:03
29.24
64.1 =0:3
28.05 0554
68.81 10.31
30.30:004
41.5660.0
10.95
67.85
22.48 so.5s
ROME
16.16
23.74:0046
70.54:025
22.39
72.90s0.5
25.86:0:2
43.59:n5::
s2.37
70. 70.044
21.07
MEMIT
76.21 30.49
74.25
28.65
76.44
32.20
47.8 24.80
16. a..09
22.73aam
AlphaEdit
73.92.029
26.59s0.49
72.96:0.2
25.92=05i
72.63
24.95
44.67:0.4
13.79
69.85
23.04
AnyEdit
97.76:am
92,96
96.60
95.60
97.7610.14
95,87
62.63:0444
46.51
96.33
94,32
UnKE
98. 98.34
93.33
93.38:u021
38
98, 98.62:10.4
96.37
a9.62:a0.4
32.89
98.33a001
95.42a020
AnyEdit
99.86:0ms
99.68
94.70.m.12
85.75-023
99.95
99,98:um
64.24
45.3
99.89
99.69
Pre-edited
25.88
64.39
24.02
65.50
18.24
44.74.ua
67.71 20:37
19.58
Qwen2.5 B- Bt-Bttt
FT-L
44.02:0043
13.78
40.33:006
12.93
46.6610.5
14.63
32.34
12.31
47.47
15.75
MEND
69.49
27.77
27.92 0.
69.544.0.4
25.47
a2.86
22.81 50.54
69.40.0.3
32.39
ROME
74.73
31.52a0042
71.90=021
28. 12:00
75.89
36.422044
55.67
25.79
72.18
35.61
MEMIT
=8.03=003
38. 28.44
76.50=031
28.65-005
77.19
38.95
56.04:n.a0
25.73
53.15
34.39
AlphaEdit
80.48
42.77
78.38
38.26
80.6610.25
45.55
56.99:n.4
27.69
74.35
41.07
AnyEdit
98.05:0.16
94,89.00.9
93.56:n11
79.98
98.08:
95.09
65.40
43.49
98.14
96.39.u.
UnKE
96.97=0.18
91.01
19.17=
67. 67.00
97.34.0.13
90.44.0.16
59. 59.29=004
29.27
95. 95.04
87.60sa22
AnyEdit*
99.35
98.82
94.81:013
82.60
99.63:009
98.99:n.1e
60.78:
32.95
99.09:0um
97.98 :n.1e
the editing performance across diverse-formatted knowl- edge, we constructed dataset named EditE For evaluation metrics, we assessed the similarity between the model's 's edited outputs and the editing targets from three perspectives: original questions, paraphrased questions, and sub-questions. The evaluation was conducted using both semantic similarity (BERT Score (Zhang et al., 2019) and lexical similarity (ROUGE Scores (Lin 2004) determine the editing success rate.
Obs 1:
AnyEdit outperforms all baselines across datasets, LLMs, and metrics. On UnKEBench, it im- proves BERT Score by over 20%, demonstrating its ef- fectiveness in editing long-form knowledge. AnyEdit further enhances performance, refining complex knowl- edge representations.
Obs 2: AnyEdit shows strong generalization on para- phrase questions. significantly outperforms baselines in paraphrase scenarios. On UnKEBench, AnyEdit im- proves BERT Score by 32% and ROUGE-L by 56% on (lama3-8B-Instruce This highlights its ability to edit long-form knowledge while maintaining robustness to rephrased queries.
5.2. Long-Fore Form Knowledge Editing (RQ1)
To evaluate long-form knowledge editing, we conducted ex- periments on two base LLMs, comparing AnyEdit with base- lines on UnKEBench, AKEW (Counterfact), and AKEW (MQUAKE). Following UnKE's settings, we set the batch size to and used a decoding temperature of 0.001.
5.3. Diverse-Formatted Knowledge Editing (RQ2)
To further evaluate the generalization capabilities of AnyEdit, we constructed a dataset named EditEverything, which includes unstructured knowledge from various do- mains and formats to assess the performance of existing editing methods on complex knowledge. This dataset en- compasses knowledge from diverse domains such as math- ematics, poetry, news, computer code, and chemistry. The detailed evaluation results are shown in Figure 4b). Fur- thermore, we evaluate the relationship between the editing performance and the number of target tokens by selecting long-form samples from the EditEverything dataset. The
Most locate-then-ced then-eeit baseline methods perform edits by computing parameter updates for single MLP layer using closed-form solutions, while UnKE uses gradient descent to update the parameters of an entire layer, trading efficiency for precision. To ensure fairness, we introduce AnyEdit* which also updates the full layer via gradient descent, al- lowing direct comparison with UnKE. Table | presents the main results, with additional details in Appendix Based on Table [ we make the following observations:
6

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Llama3-SB-Instruct
AnyFdir
Qwen2.5-7B-Instrucl
MEMIT
UnKE
AnyEdit
AnyFdit
MEMI
Math
AlphaFdit
Math
100
Code
Chemistry
Cade
Chemistty
90
Rouge BouggeLLSoo Scorr
80
70
Poetry
News
Poetry
News
60
50
News
Poetry
News
Poetry
40
30
40
80
120
160
200
Chemistry
Code
Chemistry
Code
(a)
Number of Tokens
(b)
Math
Figure 4. Performance comparison between the AnyEdit approach and baseline methods long-form diverse-formatter knowledge. (a) The performance of various methods on the EditEverything dataset in relation to the number of target tokens edited. (b) A comparison of different editing methods across various types knowledge. Knowledge types without underlining represent Rouge-L Score metrics, while those with underlining indicate Bert Score metrics, Best viewed in color.
final results are illustrated in Figure 4a). Based these results, we draw the following observations:
tal results presented in Table Based these results, we draw the following observation:
Obs 3: AnyEdit achieves superior performance across diverse knowledge domains. AnyEdit demonstrates con- sistent improvements across various knowledge types in editing tasks. Notably, the performance gains are most significant in the Code and News categories, achieving increases of 60.58% and 52.38%, respectively, the ROUGE-L metric. These results underscore the strong generalization capability of the AnyEdit method.
Obs 5: AnyEdit significantly improves the editing performance of existing methods. After integrating AnyEdit, current methods achieve notable improvements across all metrics. These results highlight that AnyEdit serves as an effective plug-and-play approach, enhancing the capabilities of existing model editing methods.
Obs 6: Incorporating AnyEdit introduces a slight in- crease in editing time. Integrating AnyEdit with other model editing methods results in a modest increase in edit- ing time, exhibiting an average relative increase 24.7% in editing time. However, given the substantial perfor- mance improvements achieved by AnyEdit, this trade-off' is considered acceptable.
Obs 4: AnyEdit maintains stable performance as the number target tokens increases. Specifically, meth- ods such MEMIT, AlphaEdit, and UnKE exhibit vary- ing degrees of performance degradation when the num- ber of target tokens exceeds certain threshold. For in- stance, MEMIT and AlphaEdit experience significant per- formance drops when editing targets exceed 30 tokens. In contrast, AnyEdit remains robust under these conditions, further demonstrating its reliability and scalability.
5.5. Impact Chunk Size (RQ4)
We conclude by evaluating whether the choice of chunk size impacts the long-form knowledge editing performance of AnyEdit. Specifically,
adopt sliding window to divide the target text into chunks and vary the chunk size to assess its influence on editing performance. Experimental results show that as the chunk size increases beyond certain threshold, the editing performance of AnyEdit declines. Due to space limitations, detailed results and analysis can be found in the Appendix C.3
5.4. Boosting Current Editing Methods (RQ3)
We further evaluate the effectiveness of AnyEdit when ap- plied to several popular model editing methods. Specifi- cally, we replace the locate stage of several lectte-then-edit baseline methods with the autoregressive editing paradigm, enabling the simultaneous identification and editing mul- tiple tokens' hidden states. The modified methods eval- uated directly the UnKEBench dataset, and their per- formance is compared against their original versions. The detailed results are presented in Figure 5) Furthermore, to evaluate the time cost introduced by incorporating our strat- egy, we measured the average editing time per sample for each method combined with our approach. The experimen-
C.3
6. Related Work
Model Editing for Knowledge Update. Traditional model editing methods modify a model's knowledge by either altering a small subset parameters or introducing exter- nal memory. These approaches enable knowledge updates,

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
MEMТT
Alphatidit
UnKE
100
100
Rouge-1 Score
Rouge L Score
80
Rouge-1 Score
80
60
60
40
20
20
20
Oni
Para.
Sub.
Ori.
Fara.
Sub.
Ori.
Fana
Sub.
(a) Performance on Laama--8--Insrruue
MEMIT
AlphaTdit
UnKE
100
100
Rouge-1 Score
80
Rouge I Score
RougeL. Soore
80
40
40
40
On
Para.
Sub.
Ori.
Para.
Sub.
Para.
Sub.
(b) Performance Qwen2.5-7B-Instruee
Figure 5. Performance improvements of baseline editing methods (i.e, MEMIT, AlphaEdit and UnKE) after incorporating autoregressive editing paradigm AnyEdit. The yellow bars represent the original performance of each baseline, while the blue bars represent the performance after the addition. Best viewed in color.
introduce additional parameters or memory instead of modi- fying existing ones. ICE (Zheng al. 2023) and DeCK (Bi et al. 2024) achieve parameter-free model editing through in-context learning, enabling knowledge updates without modifying the model's parameters. SERAC (Mitchell et al.) 20226) retrieves updates from external memory, while T- Patcher (Huang al.) 2023) and CaliNet (Dong al. 2022) allocate new neurons to encode knowledge. GRACE Hartvigsen al.) (023) replaces hidden states with discrete codebook values, and WISE (Wang et al. 2024) integrates parameterized memory for efficient knowledge merging.
Table Comparison of average editing time per sample (seconds) with baseline methods and their enhanced versions integrated with AnyEdit. The '+' symbol denotes integration with our method. Evaluation performed on Counterfact and MQUAKE from the AKEW benchmark suite, along with UnKEBench.
Method UnKEBench Counterfact
MQUAKE
MEMIT
16.37
17.05
16.92
MEMIT+
21.14
20.43
21.28
AlphaEdit
17.25
18.16
17.89
AlphaEdii++
22.03
21.57
22.95
UnKE
22.91
24.12
23.84
Unstructured Knowledge Editing. Recent research fo- cuses on editing unstructured knowledge with free text beyond structured triples. Wu al. (2024) highlight the limitations of prior methods in evaluating unstructured text editing and introduce AKEW as benchmark. UnKE (Deng) ct al. 2025) refines locate-then-edii methods by updating all parameters within a single layer, improving their ca- pability to handle unstructured knowledge. They also in- troduce UnKEBench, a dedicated benchmark for unstruc- tured knowledge editing. Huang al. (2024) propose dy- namic perception module to efficiently locate commonsense knowledge parameters, enabling free-text updates. However, while these methods handle unstructured and lengthy knowl- edge, they remain limited to factual knowledge. In contrast, our work extends editing capabilities to diverse-formatted knowledge, which encompasses various textual structures beyond factual statements.
UnKE+
28.32
30.25
29.77
mitigate model hallucination (Huang et al. 2025), or facil- itate information injection (Chen al. 2024). Parameter- Modifying Methods update existing model parameters encode new knowledge. Meta-learning based approaches, such as MEND (Mitchell et al.) 2022a), InstructEdit (Zhang jet al., 2024) and RLedit (Li et al. 2025). train hypernet- works to predict parameter updates, with MEND improving efficiency via low-rank gradient decomposition. Locate- then-edit methods, including ROME (Meng et al., (022) and MEMIT (Meng al.) 2202)), use causal tracing to iden- tify knowledge-relevant parameters and update them via least-squares optimization. NSE (Jiang et al. 2024) miti- gates catastrophic forgetting in model editing by updating only a subset neuron parameters. AlphaEdit (Fang et al.] 2025) extends this to lifelong editing with a null-space pro- jection strategy. In contrast, Ceaameter-Preserviee Methods
8

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
7. Conclusion & Limitations
computing resources provided by the Supercomputing Cen- of the USTC.
In this work, we address the limitations of existing model editing methods in handling long-fon long-form and diverse knowl- edge formats. Current approaches often face challenges due to their reliance editing single token's hidden state, which restricts their effectiveness on complex and unstruc- tured knowledge. To overcome this, we introduced Any Edit, a novel autoregressive editing paradigm that enables effi- cient and precise updates by sequentially processing and editing token hidden states over long-form text. Our ap- proach is grounded in theoretical validation using the Chain Rule of mutual information, demonstrating its ability to pro- duce consistent and accurate edits. Additionally, AnyEdit serves as a versatile framework that integrates seamlessly with traditional methods, broadening their applicability to wider range of knowledge editing tasks.
References
Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07732, 2021.
Bi, B., Liu, S., Mei, L.,, Wang, Y., Ji, P., and Cheng, X. Decoding by contrasting knowledge: Enhancing llms' confidence on edited facts. CoRR, 0bs/22005111633, 2024.
Brown, T. B., Mann, B.,, Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurIPS, 2020.
Despite its ability to enhance traditional editing approaches and broaden their applicability to complex knowledge edit- ing tasks, AnyEdit still faces two key limitations: Firstly, the current framework is not explicitly optimized for lifelong editing scenarios, which demand continuous and iterative knowledge updates over time. Adapting AnyEdit to such dynamic settings- model parameters or hidden states require periodic
Cao, N. D., Aziz, W., and Titov, I. Editing factual knowledge in language models. In EMNLP (1), pp. 6491-6506. Association for Computational Linguistics, 2021.
a critical challenge for future research. Secondly, AnyEdit is currently confined to textual knowledge editing and lacks support for multi- modal knowledge integration. Extending its capabilities to handle cross modal updates (e.g., synchronizing edits across text, images, and audio) would significantly expand its prac- tical utility, offering a promising direction for multimodal language model editing.
Chen, C., Huang, B., Li, Z., Chen, Z., Lai, S., Xu, X., Gu, J., Gu, J., Yao, H., Xiao, C., Yan, X., Wang. W. Y., Torr, P., Song, D., and Shu, K. Can editing lims inject harm? CoRR, abs/2407.22 20224,, 2024.
Deng, J., Wei, Z., Pang, L., Ding, H., Shen, H., and Cheng, X. Everything is editable: Extend knowledge editing unstructured data in large language models. In ICLR. OpenReview.net, 2025.
Impact Statement
Dobrushin, R. L. General formulation of shannon's 's main theorem in information theory. American mathematical society translations, 33:323 438, 1963.
AnyEdit enhances model editing by enabling precise and efficient updates across diverse knowledge formats, address- ing limitations in editing long-form and unstructured knowl- edge. Its ability to modify knowledge at scale improves the adaptability of LLMs, making them more responsive to real-world updates. However, increased editing flexibility raises concerns about potential misuse, such unauthorized knowledge injection or model tampering. Mitigating these risks requires careful deployment and oversight to ensure responsible use. We encourage the research community to leverage AdyEdit for advancing trustworthy and beneficial Al applications.
Dong, Q., Dai, D., Song, Y., Xu, Sui, Z., and Li, L. Cali- brating factual knowledge in pretrained language models. In EMNLP (Findings), pp. 5937-5947. Association for Computational Linguistics, 2022.
Fang, J., Jiang, H., Wang. K., Ma, Y., Shi, J., Wang, X., He, X., and Chua, T. Alphaedit: Null-space constrained knowledge editing for language models. In ICLR. Open- Review.net, 2025.
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In EMNLP (1), pp- 5484-5495. Association for Computational Lin- guistics, 2021.
Ackonwledgments
This research is supported by the National Science and Technology Major Project 102) and the Na- tional Natural Science Foundation of China (U24B2018) 62121002). This research is also supported by the advanced
Gu, J., Xu, H., Ma, J., Lu, P., Ling, Z., Chang, K., and Peng, Model editing harms general abilities of large language

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
models: Regularization the rescue. In EMNLP, pp. 16801-16819. Association for Computational Linguistics, 2024.
Meng, K., Sharma, A. S., Andonian, A. J., Belinkov, Y., and Bau, D. Mass-editing memory in transformer. In ICLR. OpenReview.net, 2023.
Hartvigsen, T., Sankaranarayanann S., Palangi, H., Kim, Y., and Ghassemi, M. Aging with GRACE: lifelong model editing with discrete key-value adaptors. In NeurlPS, 2023.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. In ICLR. OpenRe- view.net, 2022a.
Mitchell, E., Lin, C., Bosselut, A., Manning, C. and Finn, C. Memory-based model editing scale. In ICML, vol- ume 162 of Proceedings of Machine Learning Research, pp. 15817-15831. PMLR, 2022b.
Huang, B., Chen, C., Xu, X., Payani, A., and Shu, K. Can knowledge editing really correct hallucinations? In ICLR. OpenReview.net, 2025.
Huang, X., Wang, Y., Zhao, J., and Liu, K. Commonsense knowledge editing based free-text in llss. In EMNLP, pp. 14870-14880, Association for Computational Lin- guistics, 2024.
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. Orca-math: Unlocking the potential of slms in grade school math. CoRR, abs/2402. 2024.
Papineni, K., Roukos, S., Ward, T., and Zhu, W. Bleu: method for automatic evaluation machine translation. ACL, pp. 311-318. ACL, 2002.
Huang, Z., Shen, Y., Zhang. X., Zhou, J., Rong, W., and pettorrerr..ahherr. One mistake worth one neuron. In ICLR. OpenReview.net, 2023.
Xiong, Z. Transformer patcher:
Qinggang Zhang, Hao Chen, J. D. Structure-guide large language models for text-to-SQL generation. In Forty-second International Conference on Machine Learning, 2025.
Jiang, H., Fang, J., Zhang, T., Zhang, A., Wang, R., Liang,
T., and Wang, X. Neuron-level sequential editing for large language models. CoRR, abs/2410.04045, 2024.
Kullback, S. Information theory and statistics. Courier Corporation, 1997.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Lang, S. Introduction to linear algebra. Springer Science & Business Media, 2012.
Wang, B. and Komatsuzaki, A. Gpt-j-6b: billion param- eter autoregressive language model, 2021.
Li, G., Hammoud, H. A. K., Itani, H., Khizbullin, D., and Ghanem, B. CAMEL: communicative agents for "mind" exploration of large scale language model society. CoRR, 177603 2023.
Wang, P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang. and Chen, H. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405. 14768, 2024.
Li, 7., Jiang, H., Chen, H., Bi, B., Zhou, Z., Sun, F., Fang. J., and Wang, X. Reinforced lifelong editing for language models. CoRR, abs/2502.05559, 2025.
Wu, X., Pan, L., Wang. W. Y., and Luu, assessing knowledge editing in the wild. In EMNLP, pp- 15118-15133, Association for Computational Linguistics, 2024.
AKEW:
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp- 74-81, 2004.
Xie, J., Zhang, K., Chen, J., Lou, R., and Su, Y. Adaptive chameleon stubborn sloth: Unraveling the behavior of large language models in knowledge clashes. CoRR, abs/2305. 13300, 2023.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M.. Petroni, F., and Liang, P. Lost the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12:157-173, 2024a.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F,, Wei, H., et al. Qwen2. technical report. arXiv preprint arXhv:2412.15115 2024.
Liu, Y., He, X., Xiong, M., Fu, J., Deng, S., and Hooi, B. Flipattack: Jailbreak Iims via flipping, arXiv preprint arXiv:2410.02832. 2024b.
Zhang, N., Tian, B., Cheng, S., Liang, X., Hu, Y., Xue, K., Gou, Y., Chen, X., and Chen, H. Instructedit: Instruction- based knowledge editing for large language models. In IJCAI, pp- 6633-6641. ijcai.org, 2024.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in GPT. In NeurIPS, 2022.
10

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Zhang, T., Fang, J., Jiang, H., Bi, B., Wang. X., and He, X. Explainable and efficient editing for large language models. In THE WEB CONFERENCE 2025.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: mete:9904499990 2019.
Zhao, W. X., Zhou, K., Li, J., Tang. T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang. C.,
Chen, Y., Chen, Z., Jiang, Ren, R., Li, Y., Tang, X.,
Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models, 2024. URL httpt::/paaaatttt.
org/ abc. 18223
Zheng, C., Li, L., Dong, Q.. Fan, Y., Wu, Z., Xu, J., and Chang, B. Can we edit factual knowledge by in-context learning? In EMNLP. pp. 4862-4876 Association for Computational Linguistics, 2023.
Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli. S., Li, D., Yu, X., and Kumar, S. Modifying memories in transformer models. CoRR, abs/2012.00363, 2020.

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
A. Experimental Setup
A.1. Datasets
UnKEBench (Deng et al. 2025) constructs a dataset containing 1,000 counterfactual unstructured texts, where knowledge presented in an unstructured and relatively lengthy form, going beyond simple knowledge triplets or linear fact chains, These texts originate from ConflictQA (Xie al. 2023), benchmark specifically designed distinguish LLMs' parameter memory from anti-memory. This approach is crucial for preventing the model from merging knowledge obtained during pretraining with knowledge acquired during the editing process. Moreover, it addresses the key challenge of determining whether the model learns target knowledge during training or editing, ensuring clear boundary between pretraining knowledge and edited knowledge.
AKEW benchmark (Wu et al., 2024) considers three aspects: (I) Structured Facts: Each structured fact consists of an isolated triplet for editing, sourced from existing datasets or knowledge bases. (2) Unstructured Facts: Knowledge is presented in unstructured text form. To enable fair comparisons, each unstructured fact contains the same knowledge update as its corresponding structured fact. Compared to structured facts, unstructured facts exhibit greater complexity in natural language format, as they often encapsulate more implicit knowledge. (3) Extracied Triplets: Triplets are extracted from unstructured facts using automated methods to investigate whether they can facilitate knowledge editing methods in handling unstructured knowledge. In this work, we primarily focus on unstructured factual knowledge.
EditEverything dataset integrates question-answering data from multiple domains, forming long and diverse knowledge formats that are more challenging edit. Specifically, for mathematics, we select longer samples from the Orca-Math dataset (Mitra et al. 2024), which includes grade school math word problems. For coding, we use the MBPP dataset (Austin) et al. 2021), which consists of approximately 1,000 crowd-sources Python programming problems solvable by entry-level programmers, covering programming fundamentals and standard library functionalities. For chemistry, we sample from the Camel-Chemistry dataset (Li et al. 2023), which contains problem-solution pairs generated from 25 chemistry topics, each with 25 subtopics and 32 problems per topic-subtopic pair. Lastly, for the news and poetry categories, since they often contain real-world knowledge that LLMs may already possess, we generate synthetic data using GPT-4o to ensure that the information not already known by the model.
We present sample instances from the dataset in Figure 7 Figure 8 and Figure 9
A.2. Evaluation Metrics
Following previous research on model editing for structured knowledge (Meng et al. 2022 Mitchell et al. 2022a). existing evaluation metrics primarily focus on triplet-structured knowledge, where the goal is assess the modification of factual triples (subject, relation, object). Specifically, given an LLM an editing knowledge pair (z, v), an equivalent knowledge query Ze. and unrelated knowledge pairs (Iloe, Mooe), three standard evaluation metrics are commonly used:
Efficacy. This metric quantifies the success of modifying the target knowledge in fw. It evaluates whether the edited LLM generates the desired target output y when given the input æ. Formally, it is defined as:
arg max P (/ }
E
(10)
Generalization. This metric assesses whether the model has generalized the newly edited knowledge beyond its specific form. It measures if the LLM correctly produces y when given semantically equivalent input Te, indicating the degree to which the update propagates correctly across paraphrased or restructured queries:
E y arg max y (1 [
(1I)
Specificity. This metric evaluates whether the editing operation is localized, ensuring that unrelated knowledge remains intact. It measures whether the model's 's response to an unrelated query Tloe remains consistent with its original output ylue:
E
Moce = arg max P (2/ [ ]
(12)
12

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
While these metries are well-suited for structured knowledge editing, they are insufficient for evaluating long-form and formate-formatted knowledge. Such knowledge is often verbose and complex, making it challenging to assess correctness solely based on Efficacy. In these cases, the model may generate answer that captures the essential information yet fails an akact-match evaluation. To address this, we primarily follow the existing benchmarks for unstructured knowledge editing, incorporating more flexible evaluation methods suited for long-form responses.
Lexical similarity metrics include BLEU (Papineni et al. 2002) and various ROUGE scores (ROUGE-I, ROUGE-2, and ROUGE-L) (Lin 2004). These are computed based on the original questions, paraphrase question, and sub-questions, providing insights into the lexical and n-gram alignment between the model-generater text and the target answer. These metrics serve the foundation for assessing the surface-level accuracy of edited content.
Semantic similarity is also considered (Bert Score) (Zhang et al. 2019). as word-level overlap alone is insufficient to capture the nuanced understanding required by the model. To address this, we utilize embedding- encoders, specifically the all-MiniiLM-L6-v2 model to measure semantic similarity. This ensures a more balanced evaluation that extends beyond lexical matching, quantifying the depth of the model's comprehension.
A.3. Baseline Methods
FT-L (Zhu et al. 2020) a knowledge editing approach that fine-tunes specific layers of the L.LM using an autoregres- sive loss function. We reimplemented this method following the hyperparameter from the original paper.
2022a) is a hypernetwork based efficient knowledge editing method. It trains a hypernetwork to capture patterns in knowledge updates by mapping low-rank decomposed fine-tuning gradients to LLM parameter modifications, enabling efficient and localized edits. Our implementation follows the original hyperparameter settings and completes training over the full dataset.
MEND Mitchell et al.
ROME (Meng et al., 2022) is a method for modifying factual associations in LLM parameters. It identifies critical neuron activations in MLP layers through perturbation- based knowledge localization and modifies MLP layer weights using Lagrange remainders. Since ROME not designed for large-scale edits, we follow the original paper's 's settings and conduct multiple rounds of single-instance editing for evaluation.
MEMIT (Meng et al. 2023) extends ROME by enabling batch updates of factual knowledge. It utilizes least squares approximation to modify specific layer parameters across multiple layers, allowing simultaneous updates of large numbers of knowledge facts. We evaluate MEMIT in lifelong editing scenarios using the original paper's configuration.
AlphaEdit (Fang et al. 2025) is a method designed to mitigate interference in LLM lifelong knowledge editing. It introduces a null-space projection mechanism that ensures parameter updates preserve previously edited knowledge while incorporating new updates. AlphaEdit has demonstrated state-of-the-ard (SOTA) performance across multiple evaluation metrics while maintaining strong transferability. We follow the original paper's hyperparameter configuration in our implementation.
UnKE (Deng et al. 2025) improves knowledge editing by refining both the layer and token dimensions. In the layer dimension, it replaces local key-value storage with non-local block-based mechanism, enhancing the representation capability of key-value pairs while integrating attention-layer knowledge. In the token dimension, it replaces "term- driven optimization" with "cause-driven optimization." which directly edits the final token while preserving contextual coherence. This eliminates the need for explicit term localization and prevents context loss.
A.4. Implementation Details
Our AnyEdit and AnyEdit* primarily follow the baseline configurations of MEMIT and UnKE, while other baselines adhere to their original implementation settings. All experiments were conducted on a single A100 GPU (80GB).
AnyEdit on Llama3-8B-Instruut: We select layers 4 to 8 for editing and apply a clamp norm factor of 4. The fact token is defined as the last token. The optimization process involves 25 gradient steps for updating the key-value representations, with a learning rate of 0.5. The loss is applied at layer 31, and we use a weight decay of 0.001. To maintain distributional consistency, we introduce a Kullback-Leibler (KL) regularization term with a factor of 0.0625.
13

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Purthermore, we enable hyperparameter À with an update weight of 15,000, using 100,000 samples from the Wikipedia dataset with data type of float32. The module configurations follow MEMIT, where edits are applied to the MLP down projection layers of the selected transformer blocks. Additionally, for chunked editing, we set a chunk size of 40 tokens with no overlap.
AnyEdit Qwen2.5-7B-Instruct Same as the above, except that the loss is applied at layer 27 and the chunk size is set to 50 tokens.
AnyEdit on Llama3-8B-nnsruuct: We select layer 7 for editing and apply a clamp norm factor of 4. The fact token is defined the last token. The optimization process involves updating all parameters in both the attention and MLP layers. The gradient descent process utilizes a learning rate of 0.0002 with 50 optimization steps. For updating key-value representations, we use 25 gradient steps with learning rate of 0.5. The loss is applied at layer 31, and we use a weight decay of 0.001. To preserve original knowledge, we sample 20 data points to constrain parameter updates. Additionally, for chunked editing, we set a chunk size of 40 tokens with no overlap.
AnyEdit* on Qwen2.5-77 Diven..57B-Inntruuct Same as the above, except that the loss is applied at layer 27 and the chunk size is set to 50 tokens.
B. Eocate-Then-Edit Paradigm & Related Proof
B.1. Locate-Then-Ede Paradigm
Recent studies in LLM behavior analysis (Qinggang Zhang 2025 Liu al. 2024b) have explored various aspects of
model manipulation, motivating our approach to structured knowledge editing. Following prior works on model editing, the detailed descriptions of specific methods in this section are based on MEMIT (Meng et al. (023), AlphaEdit (Fang et al.) 2025) and ECE (Zhang et al.). We adhere to their formulations and methodological explanations to ensure consistency and clarity in presenting these approaches.
The
method primarily focuses on triplet-structured knowledge in the form of (s, r, o). such as modify- ing (Olympies, were held in, Tokyo) to (Olympies, were held in, Paris). Given new knowledge (Fer Me), triplet can be represented xe (s, r) and ye
We first refine the outo-regressive language model formulation in Section 2 Let f be decoder-only model with L layers, processing input sequence x = (æo, Ti.... ØT) to predict the next token:
h (z) = h (z) + (x) + mz) a: attn' (nl 1-1 h ..... 1 m w W (W W (h, + af)).
(13)
where h denotes the hidden state of token / layer 1, a is the attention output, and m, is the feedforward (FFN) output. Here, Wi and Wi ou weight matrices, a is nonlinear activation function, and denotes layer normalization.
Key-Value Memory Structure. Locate-then-edee assumes that factual knowledge is stored in the FFN layers and treats them as linear associative memory (Geva et al. 2021). Specifically, W is conceptualized as key-value memory structure:
m = w
out (W in (h + aa))
(14)
Here, the MLP input-output pair token and layer 7 serves as the key-value pair. Casual Tracing is typically used to locate the target token and layer by injecting Gaussian noise into hidden states and incrementally restoring them to analyze output recovery. For more details, please refer to ROME (Meng et al. 2022).
Computing Key-Value. For editing knowledge (æe, Ve), we compute its corresponding key-value pair (k", u*). The key k' is derived via forward propagation of Ze, while the value v² is optimized using gradient descent:
8' = + arg min (-log ( lmee):
(15)
Here, f(h) + 6') represents the model output when the FFN output h replaced with h/ + 8
14

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Methods such ROME (Meng al. 2022). MEMIT (Meng et al. 2023). and AlphalEdit (Fang al. 2025)
focus
triplets (s, r; o), selecting the last token of the subject the target token. In contrast, UnKE (Deng et al. 2025 extends to unstructured text, using the last token of the target.
To insert new knowledge v²) into the vayue memory, we solve the constrained least squares problem:
|W -vll
min W
s.t.
Wk = v².
The final parameter update can be computed via ROME/MEMTT//lppaaaii closed-form solution or UnKE's gradient- based optimization.
For clarity, let W denote the edited weight of W in the MLP, and let W represent its original weight. The final parameter update can be computed using the closed-form solutions of ROMF/MEMIT//A/paaEdi the gradient-based optimization method in UnKE
Weights Update in ROME. The ROME method derives a closed-fore solution to the constrained least-squares problem for updating MI.P parameters:
w W (v*
Wk')
(16)
)T
where C KK", The matrix C' is estimated using 100,000 samples of hidden states & obtained from tokens sampled in-context from the entire Wikipedia dataset.
Weights Update in MEMIT. Since the above solution updates only a single knowledge sample at a time, MEMIT improves upon by avoiding Lagrange multipliers and instead using a relaxed constraint formulation. The problem is reformulated by maintaining a factual set {Ki, 1. Vi containing u new associations while preserving the original set { (Ko, Vo with existing associations:
Ko = [k1 K2 | ... kn), Vo
[v]
U2 ... unl
(17)
K1
[k441 k ....
knta]
Vi
[2 u/+2 ...
Here, and u are defined as in Eq. 14 and their subscripts denote knowledge indices. The objective function is given by:
W و arg min Wk: 2:|2
1|
n+u Σ
Wk: - 0z 12)
(18)
i=++1
Applying the normal equation (Lang 2012), the closed-form solution is:
W (V1 WK) Kf (KoKj K,K)) + W.
(19)
Weights Update in AlphaEdit. AlphaEdit addresses the imbalance between old and new knowledge in lifelong learning. protects existing knowledge using a null-space projection constraint, ensuring that the update A to W onto the null space of Ko The final parameter update, refining MEMIT, is:
out is always projected
W = (V - wK, K P
KK PK +
(20)
Weights Update in UnKE. Unlike previous methods, UnKE considers the entire input to layer I, denoted as f', rather than just the MLP input. The output remains f''s activation values. The parameter update is applied to the entire layer rather than a single weight matrix. Given the knowledge sets (Ko. Vo} and {K1, V.J. the optimization objective formulated as:
E' A arg min
n 1 (k:)
will
Σ
(k.) - 2a 111')
(21)
&
where @ denotes the entire set of parameters in layer I. Since a closed-form solution is not feasible, UnKE employs gradient descent to iteratively update O
15

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
B.2. Proof of Optimization-Connittonll Mutual Information Equivalence
Theorem B.1. The optimization objective
8 arg min (-log 5(h,+5) [X)].
(22)
is equivalent to muximizing the conditional mutual information (CMI) between X and Y given the perturbed hidden state h':
h' = arg max I h').
(23)
Proof. Starting from the definition of CMI, we expand via the integral form:
I(X;Y h) / p(x, y, h, log
p(y x. nr) ply h')
dzdydh'.
(24)
This splits into two entropy terms:
/ p(z, ys h') logply x, h') dodydhh
I(X:Y h')
pla, v, h() log h') drdydh'.
(25)
Term B
Term A simplifies to the expectation:
A = T [logp(y] xx (r),
(26)
while Term B is independent of X given h'. Since B does not affect the optimization over h', we focus maximizing A.
By definition, P (Y X)) ply x, h.). Thus, minimizing the negative log-likelihood in equation 22 directly
maximizes A, which is equivalent to maximizing I(X;Y h'). Substituting h' = h2 + 8", we conclude:
h' = arg max I(X:Y h'),
(27)
thereby establishing the equivalence.
B.3. Proof of the Decomposition of Mutual Information
To rigorously derive Equation equation g we start from the mutual information (MI) decomposition given in Equation
equation
T(X; Y hy ... hi) Σ
(X;Ye Yi,..., Ya-1, hgg)
(28)
k=1
Step 1: Application of the First Property. The first key property states that later hidden states do not influence earlier token generation:
H(Y, h1)) H(,)), for p < q.
(29)
Since mutual information is defined as:
I(X;Yk Bi....,Y-111. hi,.... ... h'x) HYy Yı.... Yk-1, h hxx) H(&s X,Y, ...... Yg-1, h/..................... h's)
(30)
Since h2 for > l does not affect Yk, we can simplify:
H(Ys Yi...., Y W...... 1'x) = H(Ys Yi...., YR-1, .y.... 1())
(31)
Step 2: Application of the Second Property. The second key property states that once Y4 is determined, conditioning on Ya subsumes conditioning on h/:
H(: Yι) H( YE, h())
(32)
16

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Using this, rewrite the MI term:
I( Yi...., h h'k) [Y;; Yi..... Yk-1, h(g).
(33)
Step Applying the Conditional Mutual Information Decomposition. Using the decomposition formula for conditional mutual information, each term be written as:
((x; Yk Ii...., 1k.1, 1(4) [(X, ................................ Yk-1; Yx 1//)) I( Yk-1: 1t).
(34)
In the optimization process, since we are given X, the second term is ignored:
[(; Yx YI...., Yx-1, h()) (X,Y), Yu-1i Yo 1()).
(35)
Substituting this result back into our summation, we arrive at the desired decomposition:
K
T(X;Y My..... 1(k) = Σ YI.,, Yk-1; Ya h().
(36)
1=1
This completes the proof.
17

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
C. More Experimental Results
C.1. Case Study
We selected several editing samples from the UnKEBench and EditEverything datasets as case studies to analyze the long-form knowledge edit performance.
From the experimental results, AnyEdit effectively restores the target text in terms of both textual consistency and generalization to paraphrase questions. In contrast, other baseline methods often fail to achieve effective editing. Although the UnKE method also demonstrates strong editing performance, it exhibits noticeable performance drop on paraphrase questions.
C.I.I. CASE
Model Editing Case Study on Llama3-8B-Inse -Bs-rutt
Input
What John Mayne's background and experience in journalism?
Paraphrase
Can you provide information about John Mayne's journalism background and experience?
John Mayne journalist who has been working in the field for over a decade. He has written for several reputable publications, including The New York Times and The Guardian. an interview with The Huffington Post, Mayne discussed his passion for journalism and his commitment reporting important issues. He also mentioned that he has degree in journalism from top university and has completed several internships at major news organizations. Mayne's extensive experience and education make him highly qualified journalist and a valuable asset to any newsroom.
Target Output
LLM-edited Generation
John Mayne is a journalist and writer, but I couldn't 't find much about his personal background. However, he has been in the industry for several years, contributing to various publications. Ac- cording LinkedIn, he has worked for major outlets including BBC and The Gua rdi a John Mayne's experience spans politics, business, and culture, and he has been recognized for his in-depth reporting.
MEMIT
John Mayne is British journalist and author with long career in media. He has written for major newspapers such as The Times and The Sunday Times. Mayne has covered diverse topics. including politics and culture, and worked as a foreign correspondent reporting from the Middle East, Africa, and Asia.. In addition, he has taught journalism at universities and received multiple awards for his writing.
AlphaEdit
John Mayne journalist who has been working in the field for several years. He has written for a number of publications and has strong track record of producing high-quality, well-researched articles. In addition his writing experience, Mayne has also worked editor and has strong understanding of the journalism industry. He is well-respected by his peers and is known for his ability to produce engaging and informative articles.
UnKE
John Mayne is journalist who has been working in the field for over decade. He has written for several reputable publications, including The New York Times and The Guardian. an interview with The Huffington Post, Mayne discussed his passion for journalism and commitment to reporting on important issues. He also mentioned that has degree in journalism from top university and has completed several internships at major news organizations. Mayne's extensive experience and education make him highly qualified journalist and valuable asset to any newsroom.
AnyEdit
18

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
C.1.2. CASE 2
Model Editing Case Study on Qwen2.5-7B Instructtt
Input
find the value of logy(a*) if log. (1²² =2
To find the value of logy (2*), we can use the property of logarithms that states: log)(a) = /loga (b). Given that log)((") 2, we can rewrite this y² x². Now, we want to find log, (x*)) We can use the property mentioned above: log, (x²) have (v"); we have (1²)). So, we need to manipulate the expression to use the given information. We know that yd = x². so we can raise both sides to the power of 4/3 to get y²: (12))4/3) = (x²) y = r/(8/3) Now we can write (yg=(v") in terms of log=(v"): (yg= = logz (1y)) ((y))a/3))) Using the power rule of logarithms (logs(at = - logs(a) we get: (yg)(y") (4/3) (v") Since we know logz (yy) = 2, we can substitute this value find log, (a*)) using the reciprocal log,(x")
(ooo(("(")) But we don't
Target Output
in: log, (v*) (4/3) 2. 1ogz(1') 8/3. Now
property: log, (x*) 1/log=(1"), (og 1/(8/3). log, (x*) 3/8. Therefore, the
value of log,(z") is 3/8.
LLM-edited Generation
To find the value of (oog(()))) we can use the change of base formula and the given information. Given: log. (3²) = 2. Using the change of base formula, we can rewrite this as: (ogx()"))
(log,(y") ))/ (log,(z)) Since (og,(v²) = 3 (because y² is the argument raised to the power of
MEMIT
3), we have: 2 = 3/ (log)(x)). Now, we can solve for log,, (z): log,(x) = 3/22. Next, we want to find log, (xx"). Using the power rule of logarithms, we can write: log, (x*) = 4 (z)
Substituting the value we found for log,, (z)): log, - (3/2). Simplifying: log,, (x = 6. Therefore, the value of logy(x**) is 6.
To find the value of logy (z*), we can use the change of base formula and the given information: togx((23) 2. Using the change of base formula, we can rewrite this as: y² = x²2 Now, we want to find (og,((z)) Using the properties of logarithms, we can rewrite this as: Logy(x²*) 4 * log, We can express log, (x) in terms of log.((y using the change of base formula:
AlphaEdit
log,, (x) = 1/log (y). Since we know that (y9) 2, we can write: logæ (1²") = *
log y)) 2. Therefore: (og=(y) 2/3. Substituting this into our expression for logy (z):
log, (x: 1/(2/3) 3/2. Finally, substituting this back into expression for log, (1²4):
log, (a*) 4 (3/2) = 6. Therefore, the value of (ogy(x') is 6.
To find the value of loy, (z:²), we can use the property of logarithms that states: logs(a) 1 1/loga (b). Given that log,z (v²) = 2, we can rewrite this as y² = x². Now, we can use 1/(1/²²) = x². So, the value of
UnKE
the property to find (x*): log, (x = 1/log, log, (x*) is x Therefore, the correct answer is x!.
To find the value of log, (z*), we can use the property of logarithms that states: logs(a) = 1/log.(b) Given that (1²) 2, we can rewrite this as: y²² x². Now, we want to find log, (2:t). We can use the property mentioned above: (ogy(x**) 1 (1 But we don't have (x))( we have (v So, we need to manipulate the expression to use the given information. We know that nd = =x², so we can raise both sides to the power of 4/3 to get er: (v3))4/3)) (x2)(4/3), 3 = (8//3), Now we can write (og=(1²) in terms of 1og=2y")) log = logz (193))441))))
AnyEdit
Using the power rule of logarithms (logs(a") = e logs (05(() (a)). we get: log, (ogx(y))) = (4/3) -logu(>"). Since we know logz (y²² = 2, we can substitute this (4/3) (0*) 8/3. Now we can use the property of logarithms: log,(x*) 1l0gg(( logy(x*) 1/(8/3). log,(0*) = 3/8. Therefore, the value of logg(x*) is 3/8.
value: (gg
19

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
C.2. Supplementary Experimental Results RQ1 & RQ2
We present a comprehensive evaluation of all metrics on the UnKEBench and AKEW datasets Table 3 and Table The results demonstrate that UnKE consistently outperforms other baselines across both original and paraphrase question evaluations. Notably, UnKE+, which integrates AnyEdit's 's autoregressive editing paradigm, achieves even higher scores in lexical similarity (BLEU, ROUGE-1/2/L) and semantic similarity (BERT Score), indicating its superior ability to preserve and generalize edited knowledge. In contrast, MEMIT and AlphaEdii struggle with paraphrase generalization, showing significantly lower performance on the right side of 'Y,, suggesting that these methods fail to robustly transfer edited knowledge across rephrased contexts, While MEMIT+ and AlphaEdit+ improve over their base versions, their performance still lags behind UnKE and UnKE+
Overall, UnKE+ achieves the best balance between precise knowledge modification and robust generalization, confirming that combining UnKE with autoregressive fine-tuning leads to stronger and more reliable knowledge editing in LLMs.
Table 3. Performance comparison in UnKEBench. The '4' symbol indicates results incorporating AnyEdit's autoregressive editing paradigm. The left side of represents the LLM's edited output for original questions, while right side represents the edited output for paraphrase questions.
Lexical Similarity
Semantic Similarity
Sub Questions
Method
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
BERT Score
ROUGE-L
Based on Baamma3-8-Inntrrune
UnKE
93.56 78.09
93.61 / 79.26
91.42/71.77
71.73
93.33 78.42
98.34 93.38
37.87
UnKE+
99.67 / 84.60
99.69 86.31
99.57 /8.18
99.68 / 85.75
99.86 94.70
41.45
MEMIT
25.57 / 22.88
32.67 30.75
14.51 / 12.37
30.49 28.65
76.21 / 74.25
22.56
MEMIT+
88.88 81.38
93.26/
86.53
90.32/ 80.61
92.96 85.91
97.76 95.60
41.67
AlphaFdit
21.29 / 20.24
28.62 27.99
11.36/ 10.24
26.59 / 25.92
73.92 72.96
20.71
AlphaEdit+
75.02/ 66.35
/1.70/73.47
7 62.74
80.92 / 72.22
94.19 191.51
40.56
Based on Qwen2.5-7B-Inssrrut
UnKE
91.92/70.61
91.41 68.47
87.75 / 56.34
91.01 67.00
96.97 889.7
38.12
UnKE+
98.52/ 82.48
98.85
83.36
98.43 77.03
98.82 82.60
99.35 94.81
42.24
MEMIT
45.07 / 40.81
40.73 / 36.75
19.59 15.87
38.04 / 34.07
78.03 76.50
24.75
MEMIT+
91.31/77.23
95.10/ 80.88
92.93 / 72.50
94.89 79.98
98.05 93.56
42.38
AlphaEdit
49.71/45.21
41.00
24.63 / 19.85
42.77 / 38.26
80.48 / 78.38
25.37
AlphaEdit+
97.77 / 83.09
98.20 84.18
97.40 /7.38
98.14/ 83.40
99.08 94.51
41.58
C.3. Supplementary Experimental Results on RQ4
The experimental results of relationship between AnyEdit's editing performance and chunk size in long-form diverse- formatted knowledge are presented in Figure Based on these results, draw the following observation:
Obs 7: The editing performance of AnyEdit is influenced by chunk size. As the chunk size increases beyond certain threshold, the editing performance of AnyEdit declines. Specifically, when the chunk size is smaller. each iteration of editing becomes more manageable, leading to improved overall performance. However, this improvement comes at the cost increased editing time due to the larger number of iterations required for longer texts. Conversely, when the chunk size is larger. becomes challenging achieve effective edits within a single iteration, resulting in degraded performance. Based this trade-off, we recommend using balanced chunk size of 40 for most editing scenarios.
20

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
Table Performance comparison in AKEW (Counterfact). The '+" symbol indicates results incorporating AnyEdit's autoregressive editing paradigm. The left side of '/' represents the LLM's edited output for original questions, while the right side represents the edited output for paraphrase questions.
Lexical Similarity
Semantic Similarity
Sub Questions
Method
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
BERT Score
ROUGE-L
Based on Llama3-8B-Instruce
MEMIT
33.44 18.13
34.46/ 17.44
16.29 /4.4
32.20/ 16.10
76.44 47.80
39.98
MEMIT+
85.41 38.78
96.07/47..
95.87
46.00
97.76 62.63
64.07
UnKE
98.43 36.99
98.43/344
34.58
97.78/
19.37
98.37 32.89
99.62 59.62
63.22
UnKE+
99.98 / 45.23
99.98 / 46.57
99.96 / 35.41
99.98 / 45.31
99.95 64.24
59.03
AlphaEdit
23.36/ 16.25
26.92
15.00
10.81
3.61
24.95 13.79
72.63 44.67
35.76
AlphaEdit+
79.60 / 40.67
84.49 / 41.11
78.00/ / 26.60
83.76 / 39.51
96.51 65.14
57.05
Based on Qwen2.5-7B-Instruct
MEMIT
45.29 32.83
41.68
28.01
20.38 / 8.79
38.95 / 25.73
77.19 56.04
43.51
MEMIT+
90.55 / 44.32
95.33/45.. 45.56
27.38
95.09 43.49
98.08 65.40
55.10
UnKE
91.53 38.59
90.91
31.53
87.06 12.11
90.44 29.27
97.34 59.29
49.97
UnKE+
98.95 34.68
99.01
35.23
98.59
15.59
98.99 32.95
99.63 60.78
51.58
AlphaEdit
49.97
34.65
48.15/30.00
27.76
10.38
45.55 / 27.69
80.66 / 56.99
45.12
AlphaEdit+
97.61 / 46.97
97.80 47.63
96.89 / 30.31
97.73 45.84
99.10 66.10
54.99
Llama3-8B-Instre
85
Owen2.5-7B-Instruwe
Rouge-L Score
80
75
70
65
10
30
50
70
90
Chunk Size
Figure The relationship between AnyEdit's editing performance and chunk size in long-form
knowledge.
21

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
"case_id" 8,
"pararel_idx" 14780,
"requested_rewritt"" {
"prompt": "What is the twin city of Q It is" "relation_id": "P190", "target_new":
"str": "Sheffield", "id": "Q42448"
"target_true"" {
"str": "Sydney", "id": "Q3130"
"subject": "Wellington",
"prompt full": "What is the twin city Wellington? is", "fact_new": "What is the twin city of Wellington? It is Sheffield.", "question": "What is the twin city of Wellington? It is what?", "answer new": "Sheffield".
"answer true": "Sydney".
"fact_new La___ne___sss "Sheffield, a city in South Yorkshire, England, known for its rich industrial heritage and vibrant cultural scene. With a history dating back to the 12th century, Shefficld has grown into modern and diverse city. It is also known as the twin city of Wellington, a bustling metropolis located in the North Island of New Zealand. Despite being located on opposite sides the world, Sheffield and Wellington share a strong bond and have established a successful partnership in various fields, including education and business.", wnsfact GPT": GGTT:
unsfact_GPTT truc,
"subject": "Sheffield",
"prompt": "O is located in",
"target": "South Yorkshire, England"
"subject": "Sheffield",
"prompt": "O has a history dating back to", "target": "the 12th century"
Figure 7. Sample of the AKEW (Counterfact) dataset.
22

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
"id": O,
"question": "How different animals perceive different colors?",
querr__neeiion:: "In what ways do various animals perceive different colors?". "answer": "Larger animals (including humans) have special cells in the back of their eycs called cone cells. Each type of cone cell reacts to a specific wavelength: that is, light whose color corresponds to a specific part of the rainbow. When the cell hit by light with the right wavelength, it sends a signal to the brain, which combines information from all of the cone types to perceive the full color range. What colors each animal sees depends the number of different cone types, and what wavelength they react to. For example, most humans have three cell typos which pereeive violet, green, and yellow, but some of us lack one of these types leading to green-red color- blindness. Some species of bird fish on the other hand have four different cell types, giving them a very different pereeption of colors.",
"sub :iubb_uuetooon:
"What are the special cells in the back of larger animals' (including humans') eyes called?", "How do cone cells enable animals to perceive colors?".
"How many types of cone cells do most humans have and what colors do they perceive?",
"What causes green-red color-blindness in humans?",
"How is the perception of colors in some species of birds or fish different from that
humans?"
"sub answer": [
"They are called cone cells.",
"Each type of cone cell reacts to a specific wavelength. When hit by light with the right wavelength, sends signal to the brain, and the brain combines information from all of the cone types to perceive the full color range.",
"Most humans have three types of cone cells which perceive violet, green, and yellow.",
"Some humans lack one of the three types of cone cells which perceive violet, green, and
yellow, leading to green-red color-blindness."
"Some species of bird fish have four different cell types, giving them a very different
perception of colors."
Figure 8. A Sample of the UnKEBench dataset.
23

---

AnyEdit: Edit Any Knowledge Encoded in Language Models
"id": O,
"question": "Find the sum of 100 consecutive odd numbers from 1.",
"answer": "To find the sum of 100 consecutive odd numbers starting from 1, we can use the formula for the of an arithmetic SnnSum n/2 (first term last term)'nlawhher is the number of terms. In'nSince we are looking for the sum of 100 consecutive odd numbers starting from 1, the first term (al) is 1, and the number of terms (n) is 1n find the last the formula for the nth term of arithmetic sequence: dd'n'nwhere is the common differenee between the terms. For odd numbers, the common difference is 2 (since each edd number is 2 more than the previous odd number). An'nSo, the last term (an) isa find the sum:'n'nSum = =n/2 * (first term last term)'nSum 100/2 * (1 199) =00 200'nSum thherrfr,,errrrrreee, the sum of the first 100 consecutive odd numbers starting from is 10,000.", "category": "math"
term (an), we
= al + (n
(100- 1) 2nnan 1 + 99 2'nan 1 198 nan 199'n' we can
"id": 245,
"question": "What is the oxidation state of nitrogen in NH4NO3?",
"answer": "NH4NO3 composed of two ions: NH4+ (ammonium) and NO3- (nitrate). To find the oxidation state nitrogen in NH4NO3, we need determine the oxidation state nitrogen in each ion separately. inknIn the NH4+ ion, nitrogen is bonded to four hydrogen atoms. Hydrogen has oxidation state of +1. Therefore, the oxidation state of nitrogen in NH4+ is -3 (since +1 * +4, and the overall charge the ion is +1, nitrogen must have an oxidation state of -3 to balance the charge). 'n'nIn the NO3- ion, nitrogen is bonded to three oxygen atoms.
Oxygen has oxidation state of -2. Therefore, the oxidation state of nitrogen in NO3- is +5 (since -2 * -6, and the overall charge of the ion is -1, nitrogen must have an oxidation state of +5 to balance the charge). An'nSo, in NH4NO3, nitrogen has two different oxidation states: -3 in the NH4+ ion and +5 in the NO3- ion.", "category": "chemistry"
Figure 9, Samples of the EditEverything dataset.
24


Reviews:
[{"review_id": "issQXfnCqE", "reviewer": "Reviewer_hVmV", "summary": "The paper tackles the free-form knowledge edit problem in LLM, and proposes to extend existing locate-then-edit frame work to long-form knowledge editing by splitting long-form knowledge into chunks, and maximize the likelihood of each subsequent chunk based on perturbing previous chunk's last token's hidden state. The paper shows clear improvements upon existing methods. The paper also collects a new dataset for long-form editing with diverse formats.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer hVmV:\n\nThank you for your kind words and positive feedback regarding the novelty, presentation, and effectiveness of our work! Your approval is the great encouragement for us and motivates us to continue advancing our research. \n\nBelow, we meticulously provide responses to each of your comments and outline the modifications based on your suggestions.\n\n## *W1: \"The proposed chunk by chunk approach is more computational intensive especially when scaling up the sequence length.\"*\n\nThank you for raising this important concern. We acknowledge that our method indeed faces computational intensity issues, particularly when scaling up sequence length, and we highlighted this explicitly in Section 5.4 (Observation 6) of our paper. We have actively been exploring optimization strategies to address this challenge:\n1. **Early stopping strategy:** During gradient descent optimization of Equation (9), we halt training once loss thresholds are met, allowing shorter optimization epochs without sacrificing performance. Due to shorter chunk lengths, thresholds are quickly reached, significantly accelerating the process.\n2. **Adaptive chunk length selection:** Chunks vary in difficulty; simpler chunks can be longer, while challenging chunks can be shorter. Combined with loss thresholds, this significantly reduces the gradient descent epochs.\nAdditionally, since our initial submission, we have conducted preliminary experiments in the first direction and observed improvements in editing speed. A brief overview of these initial experimental results is provided in the following table:\n\n|||||\n|:-:|:-:|:-:|:-:|\n|Method|UnKEBench|Counterfact|MQUAKE|\n|MEMIT|16.37|17.05|16.92|\n|AnyEdit|21.14|20.43|21.28|\n|AnyEdit(accelerate)|18.89|8.22|19.03|\n|||||\n\nAs shown, our optimization approach significantly improves editing speed over the original chunk-by-chunk method, and now **matches the performance of the single-token editing baseline**. We will continue working on further improving the chunk-by-chunk editing efficiency.\n\nWe hope this response addresses your concerns.\n\n## *Suggestion1: \"I recommend the authors to also include DEM in the experiments for more comprehensive comparison.\"*\n\nThank you for highlighting this important method! Your insightful suggestion has prompted us to recognize the value of including DEM for a more comprehensive evaluation. In response:\n\n1. We have provided a detailed description of DEM in the Experimental Setup section of the revised manuscript.\n2. We conducted additional experiments using DEM and presented the corresponding results and analysis in Section 5.2.\n\nNote: Since DEM's code is currently not publicly available, we implemented the method ourselves according to our interpretation from the original paper. Below is a quick summary of our experimental results and analysis:\n\n|||||||||||||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLM|Method||**UnKEBench**||||**Counterfact**|||**MQUAKE**||\n|||**Ori.**|**Ori.**|**Paraph.**|**Paraph.**|**Ori.**|**Ori.**|**Paraph.**|**Paraph.**|**Ori.**|**Ori.**|\n|||BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|\n|Llama3-8B-It|DEM|77.09±0.32|31.27±0.48|75.18±0.29|29.41±0.57|77.01±0.30|32.87±0.47|48.12±0.36|16.32±0.54|75.96±0.32|23.10±0.57|\n|Llama3-8B-It|AnyEdit|**97.76±0.11**|**92.96±0.24**|**96.60±0.19**|**95.60±0.35**|**97.76±0.14**|**95.87±0.23**|**62.63±0.44**|**46.51±0.59**|**96.33±0.21**|**94.32±0.23**|\n|Qwen2.5-7B-It|DEM|78.92±0.28|38.71±0.50|77.01±0.25|29.32±0.48|78.10±0.27|39.50±0.46|56.32±0.35|26.09±0.53|74.18±0.30|35.25±0.52|\n|Qwen2.5-7B-It|AnyEdit|**98.05±0.16**|**94.89±0.29**|**93.56±0.15**|**79.98±0.28**|**98.08±0.15**|**95.09±0.19**|**65.40±0.38**|**43.49±0.47**|**98.14±0.13**|**96.39±0.18**|\n||||||||||||\n\nAs the table demonstrates, our AnyEdit method consistently outperforms DEM. Although DEM dynamically selects layers due to common sense knowledge residing across different layers, it still operates under the limitation of single-token editing and thus struggles with long-form knowledge editing.\n \nWe greatly appreciate your valuable suggestions and believe these modifications significantly strengthen our paper.\n\n---\n\nOnce again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing. We are excited to keep moving forward and contributing to the community!\n\nBest, \n\nAuthors of Submission9475", "claims_and_evidence": "The claim is well explained and supported by the experiments.", "methods_and_evaluation_criteria": "The proposed method is intuitive and straightforward. The evaluation criteria is appropriate.", "theoretical_claims": "I carefully read the proofs in the main paper. I briefly scanned through the proof in appendix.", "experimental_designs_or_analyses": "The experiments and analysis are clear and easy to follow. I don't have major concerns.", "relation_to_broader_scientific_literature": "Being able to edit long-form, diverse formats knowledge in LLM is still under-explored. There are a few works (e.g. UnKE) for long-form knowledge edits, but the paper shows clear improvements.", "essential_references_not_discussed": "A related work is DEM: Commonsense Knowledge Editing Based on Free-Text in LLMs. The paper mentions it, but didn't compare with it in the experiments.", "other_strengths_and_weaknesses": "Overall I think the paper is clearly written, well-motivated. The experiments and results are convincing. I don't have major concerns.\n\nOne weakness is the edit speed. The proposed chunk by chunk approach is more computational intensive (Table 2) especially when scaling up the sequence length.", "other_comments_or_suggestions": "I recommend the authors to also include DEM [1] in the experiments for more comprehensive comparison.\n\n[1] Commonsense Knowledge Editing Based on Free-Text in LLMs", "questions_for_authors": "I don't have major questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xl34XE6iuu", "reviewer": "Reviewer_H3vg", "summary": "This work proposes a novel knowledge editing method, AnyEdit, designed to mitigate performance degradation in long-form knowledge tasks. AnyEdit is a plug-and-play framework compatible with most ‘locate-edit’ knowledge paradigms. Moreover, it extends knowledge editing beyond the traditional ‘triplet’ format to a more flexible ‘free-form’ approach. Additionally, the authors introduce EditEverything, a benchmark for free-form knowledge editing.\n\n\n## update after rebuttal\nThis paper extends the triplet-based knowledge editing to the free-form-based knowledge editing. \nOverall, I keep my original rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer H3vg:\n\nThank you for your kind words and positive feedback of our novelty, presentation and effectiveness! Your approval is the great encouragement for us and motivates us to continue advancing our work.\n\nBelow, we meticulously provide responses to each of your comments and outline the modifications made to the manuscript. \n## *Suggestion1: \"The Table 1. 'Para' is easy to mistake to the abbreviation of 'Parameters', please change to another name.\"*\nThank you for your suggestion. Based on your comment, we have revised the manuscript as follows:\n·In Table 1, we have replaced 'Para.' with 'Paraph.' to more clearly indicate 'Paraphrase' and avoid confusion with the abbreviation for 'Parameters'.\n\n## *Q1: \"Why do most papers perform knowledge editing in a single MLP layer? Wouldn’t applying edits across multiple layers be more effective? Are there specific constraints or challenges preventing this approach?\"*\n\nThis is an excellent question. Indeed, many methods such as ROME and UnKE perform editing on a single MLP layer. In contrast,, recent methods like MEMIT, AlphaEdit, and their derivative approaches edit multiple layers. These studies have empirically demonstrated that **multi-layer editing often achieves better results in terms of editing effectiveness and quantity of knowledge compared to single-layer editing**.\n\nHowever, multi-layer editing approaches currently face limitations, especially within the Locate-then-edit paradigm. Specifically, interference between layers occurs, as edits in earlier layers affect the outputs of subsequent layers. Therefore, editing typically proceeds sequentially from shallow to deeper layers. We believe that a key unsolved problem remains: **achieving simultaneous multi-layer editing while ensuring edits at each layer effectively increase the output probability for the editing target**.\n\n## *Q2: \"Why don’t the authors directly evaluate the affection to LLM's knowledge after performing knowledge editing?\"*\n\nThank you for your question. In our experiments, metrics such as Bert Score and Rouge Score evaluate the similarity between the edited outputs and the target knowledge, directly reflecting the effect on the LLM's knowledge. Thus, we infer your question might pertain to evaluating the impact on unrelated or general knowledge. Following your suggestion, we have added additional experimental results comparing AnyEdit and several baselines concerning their impact on unrelated local knowledge and general knowledge, as summarized below:\n\n||||||||||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Method|Model|SST|MRPC|CoLA|RTE|MMLU|NLI|Loc-Fact Score|\n|Pre-edited|Llama3-8B-It|83.17|67.29|75.42|29.36|56.81|66.58|73.27|\n|AnyEdit|Llama3-8B-It|82.91|67.55|75.86|29.02|57.05|66.14|73.84|\n|Pre-edited|Qwen2.5-7B-It|85.62|69.47|77.88|31.73|58.92|68.53|75.94|\n|AnyEdit|Qwen2.5-7B-It|85.97|69.21|77.63|32.05|58.61|68.97|75.42|\n||||||||||\n\nAs shown, AnyEdit minimally impacts unrelated and general knowledge. Specifically, AnyEdit maintains the original knowledge retention capabilities well on both Llama3-8B-Instruct and Qwen2.5-7B-Instruct models, achieving performance comparable to the pre-edited versions. This indicates that **AnyEdit does not substantially disrupt knowledge unrelated to the edits**.\n \nWe hope our additional experiments adequately address your concerns. \n\n---\n\nOnce again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing. We are excited to keep moving forward and contributing to the community!\n\nBest,  \nAuthors of Submission9475", "claims_and_evidence": "The authors claim that AnyEdit supports arbitrary length and format. However, in their experimental settings, they only conducted experiments on sequences of up to 200 tokens, which does not strongly support this claim.", "methods_and_evaluation_criteria": "Please refer to the ‘Experimental Designs’ section for details on the evaluation criteria. \nI noticed that some prior knowledge editing papers, such as ROME, do not follow this evaluation design. \nCould you clarify the rationale behind your chosen evaluation approach?", "theoretical_claims": "The theoretical claims looks okay.", "experimental_designs_or_analyses": "Knowledge editing may affect existing knowledge. Intuitively, since AnyEdit modifies a larger number of neurons, it is expected to have a greater impact. However, the authors did not evaluate its effects on the overall performance of the LLM, which is a critical omission.", "relation_to_broader_scientific_literature": "This work extends triplet-based knowledge editing to a more flexible free-form approach, broadening the scope of knowledge modification in LLMs. This contribution aligns with prior research on knowledge editing frameworks such as ROME, SERAC, and MEMIT, which primarily focus on structured triplet-based modifications. By enabling edits in a free-form manner, this work enhances the adaptability of knowledge editing methods, making them more applicable to diverse real-world scenarios.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "I personally like this paper and appreciate its contribution. It effectively extends knowledge editing from short-length to long-length formats and from fixed triplet structures to a more flexible free-form approach. This expansion broadens the applicability of knowledge editing across a wider range of scenarios.", "other_comments_or_suggestions": "The Table 1. \"Para\" is easy to mistake to the abbreviation of \"Parameters\", please change to another name.", "questions_for_authors": "Q1: Why do most papers perform knowledge editing in a single MLP layer? Wouldn’t applying edits across multiple layers be more effective? Are there specific constraints or challenges preventing this approach?\n\nQ2: Why don’t the authors directly evaluate the affection to LLM's knowledge after performing knowledge editing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NGManbpAr7", "reviewer": "Reviewer_gZ7E", "summary": "Current LLM editing methods struggle with long-form, multi-format knowledge due to the \"efficacy barrier\" of single-token edits. AnyEdit overcomes this via autoregressive chunk decomposition and iterative token refinement, grounded in the Chain Rule of Mutual Information. It outperforms baselines by 21.5% and enables plug-and-play integration for diverse-format updates.\n\nIn summary, the focus on diverse-formatted, long-form knowledge editing addresses a critical gap in model editing for LLMs, paves the way for broader applications and further advancements in the field.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gZ7E:\n\nThank you for your kind words and positive feedback of our novelty, presentation and effectiveness! Your approval is the great encouragement for us and motivates us to continue advancing our work.\n \nBelow, we meticulously provide responses to each of your comments and outline the modifications made to the manuscript. \n## *W1 & Suggestion1: \"how does AnyEdit theoretically ensure that earlier token edits don’t destabilize the hidden states of subsequent chunks? Expanding the appendix to explicitly detail how parameter updates achieve multi-token synchronization.\"*\n\nThank you for raising this excellent question. In the implementation of Step 4 (Section 4.2), we explicitly ensure that **besides updating the hidden states of target tokens within each chunk, the hidden states of other tokens remain unchanged**. This approach precisely addresses your concern regarding earlier token edits destabilizing subsequent chunks. To clarify this further, we have expanded the details in the revised manuscript.\n\nWe hope this addresses your concern adequately.\n\n## *W2: \"A direct comparison of semantic and fixed-size chunking‘s impact on editing performance could strengthen the methodology\"*\n\nThank you for your insightful suggestion. We indeed attempted semantic chunking based on sentence segmentation but found it **limited by the nature of the dataset's knowledge categories**. Particularly, with code or mathematical problems, unclear sentence boundaries often resulted in excessively short chunks and increased iteration counts, reducing editing efficiency. Thus, we did not adopt semantic chunking in the main implementation. However, following your suggestion, we provide results comparing semantic chunking and fixed-size chunking as summarized below:\n|||||||||||||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLM|Method||**UnKEBench**||||**Counterfact**|||**MQUAKE**||\n|||**Ori.**|**Ori.**|**Paraph.**|**Paraph.**|**Ori.**|**Ori.**|**Paraph.**|**Paraph.**|**Ori.**|**Ori.**|\n|||BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|\n|Llama3-8B-It|AnyEdit(sentence-chunk)|97.81|93.02|96.54|95.48|97.69|95.79|62.71|46.62|96.28|94.18|\n|Llama3-8B-It|AnyEdit|97.76|92.96|96.60|95.60|97.76|95.87|62.63|46.51|96.33|94.32|\n|Qwen2.5-7B-It|AnyEdit(sentence-chunk)|98.11|94.83|93.48|80.05|98.03|95.21|65.33|43.71|98.09|96.44|\n|Qwen2.5-7B-It|AnyEdit|98.05|94.89|93.56|79.98|98.08|95.09|65.40|43.49|98.14|96.39|\n||||||||||||\n\nAs the table indicates, semantic and fixed-size chunking methods produce similar results, primarily due to the current semantic chunking approach being limited to sentence-level segmentation. We will further explore more effective semantic chunking methods.\n\nWe hope this clarifies your concern.\n\n## *Q1&Q2: \"Does  AnyEdit's editing time increase with longer text? Could the authors clarify the  specific optimizations that mitigate time costs?\"*\nThank you for highlighting this important issue. We indeed acknowledge the computational intensity challenge, particularly when scaling up sequence lengths, as discussed explicitly in Observation 6 (Section 5.4). To mitigate this issue, we have explored optimization strategies:\n1. **Early stopping strategy:** During gradient descent optimization of Equation (9), we halt training once loss thresholds are met, allowing shorter optimization epochs without sacrificing performance. Due to shorter chunk lengths, thresholds are quickly reached, significantly accelerating the process.\n2. **Adaptive chunk length selection:** Chunks vary in difficulty; simpler chunks can be longer, while challenging chunks can be shorter. Combined with loss thresholds, this significantly reduces the gradient descent epochs.\nWe have conducted preliminary experiments, achieving noticeable speed improvements as summarized in the following table:\n\n|||||\n|:-:|:-:|:-:|:-:|\n|Method|UnKEBench|Counterfact|MQUAKE|\n|MEMIT|16.37|17.05|16.92|\n|AnyEdit|21.14|20.43|21.28|\n|AnyEdit(accelerate)|19.89|19.22|20.03|\n|||||\n\nAs indicated, optimized chunk-by-chunk editing improves speed compared to the original method, though gaps remain compared to single-token editing. Future work will aim at further improvements.\nWe hope this adequately addresses your concern.\n\n\n## *Q3:\"What are the  specific advantages of  AnyEdit* over  AnyEdit that lead to its improved performance?\"*\nThank you for this insightful question. AnyEdit builds upon MEMIT, employing auto-regressive editing with closed-form updates in Step 4. In contrast, AnyEdit* extends UnKE, using auto-regressive editing combined with gradient descent optimization to update all parameters in Step 4, resulting in enhanced performance.\n\nWe hope this clarifies your query. \n\n---\n\nOnce again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing. \n\nBest,  \n\nAuthors of Submission9475", "claims_and_evidence": "The authors' claims are robustly supported by rigorous theoretical foundations and compelling experimental evidence. The proposed ​auto-regressive editing paradigm is theoretically grounded in a meticulous derivation of the ​Chain Rule of Mutual Information, ensuring principled modeling of sequential edit dependencies and providing a solid mathematical framework for addressing ​long-form, diverse-formatted knowledge editing. Furthermore, the work is empirically validated through extensive experiments, including the introduction of a ​novel diverse-formatted knowledge editing dataset that rigorously evaluates performance across diverse data types. The results demonstrate ​AnyEdit's superior performance over existing baselines, with significant improvements in ​edit accuracy and ​contextual consistency. These theoretical and empirical contributions collectively substantiate the central claim of resolving challenges in ​long-form, diverse-formatted knowledge editing, establishing the method’s practicality and scalability for real-world applications.", "methods_and_evaluation_criteria": "The methodology and evaluation framework are ​novel, comprehensive, and well-aligned with the challenges of knowledge editing. The proposed ​AnyEdit introduces a groundbreaking ​auto-regressive editing paradigm that effectively extends the editable text length by explicitly modeling sequential dependencies, a critical advancement for handling long-form content. The benchmark datasets—UnKEBench, ​AKEW, and the newly proposed ​EditEverything—are explicitly tailored to evaluate ​long-form, diverse-formatted knowledge editing, ensuring fair and rigorous comparisons. The evaluation metrics (e.g., ​ROUGE score for contextual coherence and ​BERTScore for semantic fidelity) are thoughtfully chosen to holistically assess edit performance across extended contexts. Furthermore, the thorough comparison with diverse existing editing methods systematically validates ​AnyEdit's superiority in ​edit precision, ​scalability, and ​format adaptability, particularly in scenarios requiring multi-step, context-aware modifications. The methodology is rigorously justified, with evaluation criteria meticulously selected to address both technical and practical dimensions of knowledge editing, solidifying the work's reproducibility and impact.", "theoretical_claims": "The theoretical claims are rigorously derived from the ​Chain Rule of Mutual Information, with detailed proofs in Appendix B.2 ensuring clarity and mathematical validity.", "experimental_designs_or_analyses": "The experiments are rigorously designed, leveraging ​UnKEBench, AKEW, and EditEverything datasets to evaluate ​long-form, diverse-formatted editing. ​ROUGE Score and ​BERTScore metrics comprehensively assess performance, with results demonstrating superiority in accuracy and robustness over prior works.", "relation_to_broader_scientific_literature": "This work significantly expands the scope of ​knowledge editing research by addressing ​multi-format, long-form knowledge manipulation—a critical yet underexplored challenge in knowledge editing. While prior works focus on isolated formats (e.g., triplet knowledge edits), this paper bridges the gap between theoretical principles (mutual information chain rule) and real-world demands for ​cross-format consistency (e.g., synchronizing edits across text, tables, and structured data).", "essential_references_not_discussed": "The paper provides ​comprehensive coverage of knowledge editing literature, including foundational works (e.g., Parameter-Modifying Methods and Parameter-Preserving Methods) and recent advances in ​unstructured knowledge editing (e.g., free-form text). The cited references are well-represented and directly relevant to the paper's focus on ​diverse-formatted knowledge.", "other_strengths_and_weaknesses": "Strengths\n1. The current model editing are largely limited in the format and length of to-be-edited knowledge. This work extends current model editing methods to be applicable to any format and length through a very simple operation. It is highly practical and crucial for the future development of model editing and efficient knowledge updating of LLMs.\n2. I appreciate the paper's well-structured narrative and clear implementation details, which make the autoregressive editing paradigm straightforward to replicate.\n3. The autoregressive chunk decomposition is an elegant solution to the \"efficacy barrier,\" balancing theoretical grounding (Chain Rule of Mutual Information) with practical plug-and-play utility.\n4. The ​extensive experimental validation of ​AnyEdit across ​UnKEBench, AKEW, and EditEverything, rigorously demonstrating its effectiveness in ​long-form, diverse-formatted knowledge editing.\nWeaknesses\n1. I'm concerned about the propagation of hidden state perturbations in the autoregressive design: If Step 4 in section 4.2 aligns multiple chunk states simultaneously, how does AnyEdit theoretically ensure that earlier token edits (e.g., in chunk 1) don’t destabilize the hidden states of subsequent chunks (e.g., chunk 2+) during iterative updates?\n2. While the paper proposes both semantic and fixed-size chunking, a direct comparison of their impact on editing performance  could strengthen the methodology.", "other_comments_or_suggestions": "A minor yet constructive suggestion: In Section 4.2 (Step 4), expanding the appendix to ​explicitly detail how parameter updates achieve multi-token synchronization would enhance clarity. While the core methodology is sound, clarifying this synchronization mechanism would strengthen reproducibility and theoretical rigor for the ​auto-regressive editing paradigm.", "questions_for_authors": "1. Does ​AnyEdit's editing time increase with longer text?\n2. The results show ​AnyEdit's editing time increases modestly with text length, despite its auto-regressive nature. Could the authors clarify the ​specific optimizations that mitigate time costs?\n3. What are the ​specific advantages of ​AnyEdit* over ​AnyEdit that lead to its improved performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2JRipfOlI", "reviewer": "Reviewer_deNs", "summary": "The paper presents AnyEdit, a new method for updating long-form knowledge in LLMs. Unlike existing methods that edit a single token’s hidden state, AnyEdit decomposes knowledge into chunks and iteratively updates key tokens in each chunk. This approach ensures more accurate and consistent updates. The method is grounded in the Chain Rule of Mutual Information and outperforms existing methods on several benchmarks, including a new dataset, EditEverything. AnyEdit also offers a plug-and-play solution for integrating with existing frameworks to handle longer, diverse knowledge updates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dG4H:\n\nThank you for your positive feedback and valuable suggestions! We sincerely appreciate the time and effort you have dedicated to reviewing our work. Below, we meticulously provide responses to each of your comments and outline the modifications based on your suggestions.\n\n## *W1: \"It would be helpful if the authors could provide additional analysis on how different overlap settings might affect the method's effectiveness.\"*\n\nThank you for your valuable suggestion. Following your advice, we have added extra experiments analyzing how different overlap settings affect our method's editing effectiveness. The specific results are summarized below:\n\n|||||||||||||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLM|Overlap||**UnKEBench**||||**Counterfact**|||**MQUAKE**||\n|||**Ori.**|**Ori.**|**Para.**|**Para.**|**Ori.**|**Ori.**|**Para.**|**Para.**|**Ori.**|**Ori.**|\n|||BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|BertScore|Rouge-L|\n|Llama3-8B-It|0%|97.76±0.11|92.96±0.24|96.60±0.19|95.60±0.35|97.76±0.14|95.87±0.23|62.63±0.44|46.51±0.59|96.33±0.21|94.32±0.23|\n|Llama3-8B-It|25%|97.72±0.15|92.83±0.27|96.55±0.21|95.43±0.31|97.68±0.16|95.82±0.25|62.50±0.45|46.39±0.56|96.20±0.22|94.27±0.25|\n|Llama3-8B-It|50%|97.65±0.14|92.76±0.29|96.49±0.20|95.35±0.33|97.63±0.17|95.75±0.27|62.44±0.47|46.27±0.58|96.11±0.23|94.18±0.24|\n|Llama3-8B-It|75%|97.58±0.16|92.61±0.30|96.43±0.22|95.24±0.32|97.56±0.18|95.64±0.28|62.35±0.46|46.12±0.57|96.03±0.25|94.09±0.26|\n|Qwen2.5-7B-It|0%|98.05±0.16|94.89±0.29|93.56±0.15|79.98±0.28|98.08±0.15|95.09±0.19|65.40±0.38|43.49±0.47|98.14±0.13|96.39±0.18|\n|Qwen2.5-7B-It|25%|98.01±0.17|94.82±0.31|93.49±0.16|79.91±0.30|98.02±0.17|95.01±0.21|65.31±0.39|43.42±0.48|98.08±0.15|96.32±0.19|\n|Qwen2.5-7B-It|50%|97.94±0.18|94.75±0.33|93.42±0.17|79.84±0.29|97.95±0.18|94.93±0.23|65.22±0.40|43.35±0.50|98.01±0.16|96.24±0.21|\n|Qwen2.5-7B-It|75%|97.87±0.19|94.67±0.34|93.35±0.18|79.76±0.31|97.89±0.20|94.85±0.25|65.14±0.42|43.28±0.51|97.94±0.17|96.18±0.22|\n||||||||||||\n\nAs indicated by these results, **increasing overlap does not significantly improve editing effectiveness and can slightly decrease performance**. Furthermore, increased overlap also **raises the number of autoregressive iterations, reducing overall editing efficiency**. Thus, in practice, we set the overlap directly to 0 for efficient and effective editing.\nWe hope our additional experiments address your concern!\n\n## *Q1: \"Line 108 references Appendix D, but there is no Appendix D in the paper. The caption for Figure 1 contains incorrect figure numbers.\"*\nThank you for pointing out these oversights! We have made the following corrections in the revised manuscript:\n1.Added the missing information from Appendix D and integrated it into Appendix A.2, specifically supplementing the definitions of Efficacy, Generalization, and Specificity from ROME.\n2.Updated the caption of Figure 1 to correctly state: \"(a) and (d) illustrate the editing processes; (c) and (e) show the editing efficacy as the number of tokens within the to-be-updated knowledge increases.\"\n\n**Hope that these updates could meet your expectations, and we are more than happy to add clarifications to address any additional recommendations and reviews from you.**\n\n---\n\nOnce again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing. We are excited to keep moving forward and contributing to the community!\n\nBest,\n\nAuthors of Submission9475", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, The paper takes a novel information-theoretic approach, reworking the token hidden state computation in the locate-then-edit method into an autoregressive form. This innovative approach effectively addresses the challenge of editing long-form knowledge.", "experimental_designs_or_analyses": "The paper provides extensive experiments on both the motivation and performance of the method, demonstrating its effectiveness across various benchmarks.", "relation_to_broader_scientific_literature": "Previous single-token editing methods relies on significantly increasing the probability of generating a specific output after applying a perturbation. However, if the original model's probability for that output is low, especially in diverse knowledge formats, the perturbation must induce a substantial shift to make it the dominant output. Current methods often struggle in these cases.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "In the Implementation Details of the appendix, the paper sets the overlap between sliding windows to 0, but it does not discuss the impact of this overlap on performance. It would be helpful if the authors could provide additional analysis on how different overlap settings might affect the method's effectiveness.", "other_comments_or_suggestions": "None", "questions_for_authors": "Line 108 references Appendix D, but there is no Appendix D in the paper. This seems to be an omission by the authors.\n\nThe caption for Figure 1 contains incorrect figure numbers. It should read \"(c) and (e) show the editing...\" and \"(b) and (f) depict the type...\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes AnyEdit, a method that addresses the gap in long-form, diverse-format knowledge editing for LLMs. Reviewers particularly appreciated the clear theoretical grounding via the Chain Rule of Mutual Information and the strong empirical evaluations across multiple benchmarks, including the introduced EditEverything dataset.

During the rebuttal phase, the authors effectively addressed the reviewers’ initial concerns through detailed responses, providing additional analyses and comparisons with previously unaddressed baselines. The reviewers unanimously gave the paper positive scores and emphasized the practical importance of extending from triplet-based to free-form knowledge editing.

However, the paper still has the following issues, which the authors should incorporate into the revision version, especially considering the discussion during the rebuttal:
1. Computational Efficiency, Reviewer hVmV pointed out that the proposed chunk-by-chunk approach remains computationally intensive despite optimization efforts, potentially limiting its practicality for very long sequences. Although the authors suggested some possible tricks, this remains a major drawback of the method. The authors need to discuss this issue in more detail in the revised version.
2. Scope of Evaluation, Reviewer H3vg noted the limitation of evaluating the method only on sequences up to 200 tokens, which does not fully support the claim of arbitrary-length editing. The authors did not respond to this point in the rebuttal.
3. Broader Impact Analysis, The reviewers raised valid concerns about the impact of extensive edits on unrelated knowledge. A more comprehensive assessment in this area would further strengthen the work.

In summary, the committee finds this paper clearly worthy of acceptance due to its novel contributions and rigorous evaluations. However, it recommends acceptance without an oral presentation, encouraging the authors to further address computational efficiency and broader validation in future work.
Decision: Accept (poster)