AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models
Authors: Junfeng Fang
Houcheng Jiang
Kun Wang
Yunshan Ma
Jie Shi
Xiang Wang
Xiangnan He
Tat-Seng Chua
Venue: ICLR 2025 Oral (2025)
Keywords: Model Editing
Null-Space
Large Language Model
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a novel model editing method named AlphaEdit to minimize the disruption to the preserved knowledge during editing.

Abstract: Large language models (LLMs)  often exhibit hallucinations, producing incorrect or outdated knowledge. Hence, model editing methods have emerged to enable targeted knowledge updates. To achieve this, a prevailing paradigm  is the locating-then-editing approach, which first locates influential parameters and then edits them by introducing a perturbation. While effective, current studies have demonstrated that this perturbation inevitably disrupt the originally preserved knowledge within LLMs, especially in sequential editing scenarios.
To address this, we introduce AlphaEdit, a novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMs remains unchanged when queried about the preserved knowledge, thereby mitigating the issue of disruption. 
Extensive experiments on various LLMs, including LLaMA3, GPT2-XL, and GPT-J, show that AlphaEdit boosts the performance of most locating-then-editing methods by an average of 36.7% with a single line of additional code for projection solely.

Published as a conference paper at ICLR 2025
ALPHAEDIT: NULL-SPACE CONSTRAINED KNOWLEDGE EDITING FOR LANGUAGE MODELS
Junfeng Fang' Houcheng Jiang' * Kun Wang', Yunshan Ma², Jie Shi?, Xiang Wang"i, XXangnan He"i, Tat-Seng Chua² University of Science and Technology of China, "National University of Singapore fangjfl 997 @070gmmm1111 com, jiangh.cmmil 1. ustc.uuteee.oo en
ABSTRACT
Large language models (LLMs) often exhibit hallucinations, producing incorrect or outdated knowledge. Hence, model editing methods have emerged to enable targeted knowledge updates. To achieve this, prevailing paradigm the locating- then-editing approach, which first locates influential parameters and then edits them by introducing a perturbation. While effective, current studies have demonstrated that this perturbation inevitably disrupts the originally preserved knowledge within LLMs, especially in sequential editing scenarios. To address this, we introduce AlphaEdit, a novel solution that projects perturbation into the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMS remains unchanged when querying about the preserved knowledge, thereby mitigating the issue of disruption. Extensive experiments on various LLMs, including LLaMA3, GPT2- XL, and GPT-J, show that AlphaEdit boosts the performance of most locating- then-editing methods by an average of 36.7% with single line of additional code for projection solely. Our code is available at: https: / github. com/ j sianghoucheng / AlphaRdit.
INTRODUCTION
Large language models (LLMs) have demonstrated strong capability to store extensive knowledge during pre-training and recall it during inference (Brown et al., 2020; Petroni et al., 2019; Roberts et al., 2020; Liu et al., 2024a). Despite this, they frequently exhibit hallucinations, producing incorrect or outdated information (Cao et al., 2021; Mitchell et al., 2022a). While fine-tuning with updated knowledge offers a straightforward solution, it is often prohibitively come-consuming, especially for resource-limited scenarios. In sight of this, model editing methods have emerged, enabling updating the target knowledge while preserving other knowledge (Meng et al., 2022; Yao et al., 2023). Broadly, model editing approaches fall into two categories: (1) ceaameter-modffyinn methods, which directly adjust a small subset of parameters (Meng et al., 2023; Jiang et al., 2025; Li et al., 2024b), and (2) parameter-preserving methods that integrate additional modules without altering the original parameters (Huang et al., 2023; Yu et al., 2024; Hartvigsen et al., 2023; Zheng et al., 2023).
In this paper, we explore the parameter- modiifynn methods for model editing which require no additional memory. Concretely, current parameter-modie modiiffin methods typically follow the locate- then-edit paradigm (Meng et al., 2022). The basic idea is to first locate influential parameters W through causal tracing, and then edit them by introducing a perturbation A. The common objective for solving A is to minimize the output error on the to-be-updated knowledge, denoted as e Additionally, the output error on the to-be-preseed preserved knowledge, eo. also needs to be considered in the objective function to ensure the model behavior unchanged on the preserved knowledge.
Despite their success, the current paradigm faces critical limitation: it struggles to maintain balance between knowledge-updat error éj and knowledge preservation error 6p. Specifically, to prioritize the success of update, prior studies focus more minimizing e by assigning a larger weight, while taking insufficient control over co- This could make the L.M after editing (i.e., the
knowledge-preservation
"Equal contribution. 'Corresponding author: {xi (xiangaang] 222,, xiangaanhe] @gma com.

---

Published as a conference paper at ICLR 2025
Current Methods
AlphaEdit (Ours)
(x) Objective
(d) Objective
P A62
Projerten
W+A
Neil Space
BI
822 5
Hidden Representation
Hidden Representation
526.8
Prr- Edited
415
AlphaEdit
Fdited
Edited
460
Transformative Performance
One Line of Codt.
0.76
32.4
-20
-20
Shife!
0.32
-30
20
-40
-20
20
40
Output the Post-edited
81.1
"The largest ocean is the Atlantic."
Forgetting
"The largest ocean is is is..."
Collapse
NphaEdR RECT
PRUNE
ROME
MEMI
Figure Comparison between the current methods and AlphaEdit. (a) and (d) exhibit the objectives, where À is the coefficient to keep balance between 60 and e1 in the objective; (b) and (e) show the distributions of hidden representations after dimensionality reduction within the pre-edited and post-edited LLaMA3, respectively: (c) depicts the output of the post- edited LLaMA3. Best viewed in color.
Figure 2: Performance of various model editing methods LLaMA3 (8B). Results with asterisks in the superscript are from the ZsRE dataset. SST. RTE, and CoLAA demonstrate the general capabil- ities of the post-edited L.Mss. 'he experiments are conducted with 2000 edited samples for se- quential editing. Detailed settings and results are provided in Section 4.2 and Table 1, respectively. Best viewed in color.
post-edited LLM) overfit to the updated knowledge. This overfitting would introduce a distribution shift of the hidden representations within LLMs. Figure (b) showcases this shift, where the hidden representations the post-edited LLaMA-3 (8B) (Meta, 2024) diverge from the original distribution (i.e., the pre-edited LLM). Worse still, in sequential editing scenarios (Gupta & Anumanchipalli 2024) where the LLM undergoes multiple sequential edits, the accumulation of overfitting destroys the model's ability to preserve knowledge and generate coherent sentences, eventually leading to model forgetting and model collapse, as depicted in Figure (c).
To address these flaws, we instead remove eo from the current objective, allowing the model focus solely on minimizing é1 without trade-offs. To avoid overfitting to the to-be-updated knowledge, we project the solution of this new objective into the null space (Wang al., 2021) of the preserved knowledge before applying it to the model parameters, as shown in Figure (d). By leveraging the mathematical properties of matrix projection and null space, our new objective ensures that the distribution of hidden representations within LLMs remains invariant after editing, as shown in Figure (e). This invariance enables the post-edited LLM reduce é₁ while keeping ep close to zero, thereby alleviating the issues of model forgetting and model collapse. We theoretically prove such invariance property in Section In a nutshell, we term the method as AlphaEdit, simple yet effective editing approach with null-space constraint for LLMs.
To validate the effectiveness of our method, we conducted extensive experiments on multiple rep- resentative LLMs, such as GPT-2 XL. (Radford et al., 2019) and LLaMA-3 (8B). The results show that, compared to the best-performing baseline, AlphaEdit can achieve an average performance improvement of 36.7% 66.7% % by adding just one line of code to the conventional model editing method, MEMIT (Meng et al., 2023), as illustrated in Figure 2. Furthermore, we empirically verified that this simple idea can be easily applied to most existing model editing methods (Meng al., 2022; 2023; Ma et al., 2025; Gu et al., 2024; Li et al., 2024b), functioning as a plug-and-play enhancement that significantly boosts their performance. This highlights AlphaEddit's crucial role in efficient knowledge updates for LLMss, enabling broader applications and future advancements in the field.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARY
2.1 AUTOREGRESSIVE LANGUAGE MODEL
An autoregressive LLM predicts the next token æ in sequence based on the preceding tokens. Specifically, the hidden state of a at layer within the model, denoted as h', can be calculated as:
h'= hhh--1++ a' + +m', m' W oW +
(1)
where a' and m' represent the outputs of the attention block and the feed-forward network (FFN) layer, respectively; W in and W are the weight matrices of the FFN layers; a is the non-linear activation function, and y denotes the layer normalization. Following Meng al. (2022), express the attention and FFN modules in parallel here.
It is worth noting that W out within FFN layers is often interpreted linear associative memory, functioning as key-value storage for information retrieval (Geva et al., 2021). Specifically, the knowledge stored in L.MSs is formalized as (s, r) representing subject s, relation r, and object (e.g., = "The latest Olympic Game". r = "was held in", "Paris"") W. associates a set of input keys & encoding (s, r) with corresponding values v encoding (o). That is.
m
W put oWW! 7(h' + a')).
(2)
This interpretation has inspired most model editing methods to modify the FFN layers for knowledge updates (Hase et al., 2023; Li et al., 2024a; Hu et al., 2024). For simplicity, W refer to W in the following sections. out
2.2 MODEL EDITING IN LLMS
Model editing aims to update the knowledge stored in LLMs through single edit or multiple sequential edits. Each edit modifies the model parameters W by adding a perturbation A in locate- then-edit paradigm. Specifically, suppose each edit needs to update u pieces of knowledge in the form of (s,r, (r,o) o). The perturbed W is expected to associate u new k-v pairs, where k and encode (s, r) and (o) of the new knowledge, respectively. Let W Rdd xdo, where dg and d, represent the dimensions of the FFN's intermediate and output layers. Then, we can stack these keys and values into matrices following:
K, [k |kl kw] E Rdox.,
Vi (0] 102]...|v0 [ww] E Rd X"
... where the subscripts & and v represent the index of the to-be-updated knowledge. Based on these, the objective can be expressed as:
arg min [WW +JJK - vi vil"
A
where 21-112 denotes the sum of the squared elements in the matrix.
Additionally, let K₂ and Vo represent the matrices formed by stacking the k and a of the preserved knowledge. Current methods (Meng et al., 2023; Gu et al., 2024) typically incorporate the error involving Ko and Vo to preserve it, as follows:
= arg min (w + A)K - Vi 1|| + (W + +AJKo Vo vol²
(5)
A
Since Ko and Vi encode the preserved knowledge in LLMs, we have WKo = Vo (cf. Eqn. 2). Thus, by applying the normal equation (Lang, 2012), if the closed-form solution of Eqn. 5 exists, can be written as:
A (Vi WK)) KT KKo + K,K
Although Ko is difficult to obtain directly since we hardly have access to the LLM's full extent of knowledge, it can be estimated using abundant text input (Meng et al., 2023). In practical applications, 100, 000 (s, r, o) triplets from Wikipedia are typically randomly selected to encode Ko (Meng et al., 2023), making K, a high-dimensional matrix with 100, 000 columns (i.e., Ko E d x 100.000). See Appendix B.1 for detailed implementation steps.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
In this section, we first introduce the concept of the null space and its relationship to model editing (Section 3.1). Based this, we present the method for projecting the perturbation A into the null space of the matrix Ko, which encodes the persevered knowledge (Section 3.2). Following that, we present AlphaEdit that incorporates the aforementioned projection method in Section 3.3.
3.1 NULL SPACE
Null space is at the core of our work. Here we first introduce the definition of the left null space (hereafter referred to simply as null space). Specifically, given two matrices A and B, B is in the null space of A if and only if BA = O. See Adam-NSCL (Wang et al., 2021) for more details.
In the context of model editing, if the perturbation A is projected into the null space of Ko (i.e. A A, O, where A denotes the projected perturbation), adding it to the parameters W results in:
(W A A' W, Vo.
(7)
This implies that the projected A will not disrupt the key-value associations of the preserved knowledge (i.e, (Ko: )) ensuring the stability of the preserved knowledge.
Therefore, in this work, before adding perturbation to the model parameters W, we project it into the null space of Ku to protect the preserved knowledge. This protection allows us to remove the second term the term focusing on preserving knowledge - from the objective in Eqn. 5.
3.2 NULL SPACE PROJECTING
In Section 3.1, we briefly explained why A should be projected into the null space of Ko. In this part, we focus on how to implement this projection.
As introduced at the end of Section 2.2, the matrix Ko 1d0x000,000 has high dimensionality with 100, 000 columns. Hence, directly projecting the given perturbation A into the null space of K, presents significant computational challenges. In sight of this, we adopt the null space of the non-central covariance matrix Kg K E Re x as a substitute to reduce computational complexity. as do is typically much smaller than 100, 000. This matrix's 's null space is equal to that of Ko (please see Appendix B.2 for detailed proof).
Following the existing methods for conducting null space projection (Wang et al., 2021), first apply a Singular Value Decomposition (SVD) to KoKY
(U, UUA.U", SVD (K,K)),
(8)
where each column in U is an eigenvector of K₂K?² Then, remove the eigenvectors in U that correspond to non-zero eigenvalues', and define the remaining submatrix as U. Based on this, the projection matrix P can be defined as follows:
This projection matrix can map the column vectors of A into the null space of K, K2 as it satisfies the condition AP. Ko O. The detailed derivation is given in Appendix B.3.
Since Ko and Ko KT share the same null space, we can derive AP. Ko = O. Hence, we have:
(W AP) K, WK, Vo.
(10)
This shows that the projection matrix P ensures the model edits not affecting the preserved knowledge in LLMs.
3.3 NULL-SPACE CONSTRAINED MODEL EDITING
Section 3.2 has provided how to apply projection to ensure that preserved knowledge is not disrupted. Here, we introduce how to leverage this projection to optimize the current model editing objective.
'Given that eigenvalues rarely strictly zero in practical applications, in our experiments, we remove the eigenvectors corresponding to the eigenvalues above 10-?
4

---

Published as a conference paper at ICLR 2025
For updated knowledge
For preserved knowledge
Struggle balance K & Ko.
Á
arg min II(W + Й)K; KIll ||kK +
Kill
!
MEMIT
+ Null Space of Ko
Only One Line of Code for Projection!
AP
arg min [[W (A)K
||PP||
AlphaBdit
For updated knowledge
Focus solely on K
Figure 3: Comparison between the paradigms of AlphaEdit and current method. Best viewed in color.
Starting with the single-edit objective in Eqn. 5, the optimization follows three steps: (1) Replace A with the projected perturbation A., ensuring that the perturbation would not disrupt the pre- served knowledge; (2) Remove the first term involving Ko. Step (1) has already protected the
preserved knowledge from being disrupted: (3) Add a regularization term A PI to guarantee stable
convergence. With these optimizations, Eqn. 5 becomes:
A arg min Á
[I(W+
V11 11
(11)
where Kj and V1 denote the key and value matrices of to-be-updated knowledge defined in Eqn.
In sequential editing tasks, during the current edit, we also need to add a term to the objective (cf. Eqn. 11) to prevent the perturbation from disrupting the updated knowledge in previous edits. Let Kº and Vp present the key and value matrices of the previously updated knowledge, analogous to the earlier detinitions of K₁ and Vi This term should minimize || association. Since the related knowledge has been updated previous edits, we have K, Vp- Hence, this term can be simplified to APK,ll and adding it to Eqn. 11 gives the new objective:
( [[(W
Á) v to protect the
A arg min (I(WW- ÂP)Kı 2 + | ||āPK,11)) Á
(12)
To facilitate expression, define the residual vector of the current edit R Vi WK, Based on this, Eqn. 12 can be solved using the normal equation (Lang, 2012):
(APK, K) P AP APK, K p
(13)
Solving Eqn. 13 yields the final perturbation AlphaEdit parameters W:
P which will be added to the model
AlphaEdit P
(14)
The detailed derivation process and the invertibility the term within the brackets are provided in Appendices B.4 and B.5 respectively. This solution A AlphaEdit could not only store the to-be-updated knowledge the current edit, but also ensure that both the preserved knowledge and the previously updated knowledge remain unaffected. Furthermore, for better comparison, commonly used solution in existing methods like MEMIT (Meng et al., 2023) as follows":
also present the
MEMIT RKT (K,K3 K₁KЛ + KoK
(15)
By comparing Eqn. 14 and 15, it is evident that our approach requires only minor modification to the standard solution by incorporating the projection matrix P. This makes our method easily integrable into existing model editing algorithms. Figure 3 summarizes this modification from the perspective of convergence objectives. We emphasize that by adding just single line of code for this modification, the performance of most editing methods could significantly enhanced, as shown in Figure 2. More detailed experimental results are exhibited in Section 4.
Furthermore, since the projection matrix P is entirely independent the updated knowledge, it only needs to be computed once and can then be directly applied to any downstream editing tasks. Consequently, AlphaEdit introduces negligible additional time consumption compared to baselines, making it both efficient and effective.
AMEMIT denotes the solution provided by MEMIT in sequential editing tasks.
5

---

Published as a conference paper at ICLR 2025
Table I: Comparison of AlphaEdit with existing methods the sequential model editing task. Eff., Gen., Spe., Flu. and Consis, denote Efficacy, Generalization, Specificity. Fluency and Consistency, respectively. The best results are highlighted in bold, while the second-best results are underlined.
Counterfact
ZsRE
Method
Model
Elt.1
Gen.1
Spe.t
Flu.
Consis."
E
Gen.7
Spe.1
Pre-edited
7.85
10.58
89.48 1
635 23
24
36, 26.99
36.34
FT
83.33 jusr
67.79.cm
46.63aum
233.72 uzz
8.77
30.48
30.22
15.49
MEND
63.24
61.17
45.37
372. 16
4.21
0.9 2omm
1.09
0.53
LIaMA3
InstructFdit
66.58
64. 664.18.em 18gss
47. 14
443.85_.ox
7.28
1,58200
1,36_nox
1.01 1om
ROME
64.40.mm
61.42_ovv
49.44.00x
449.06_nxx
1ovr
2.01 Lown
1.80_mm
0.69 1om
MEMIT
65, 65,65a047
64.65=042
51.56=03s
437, 117.43=1.7
6.58=6m1
34.62=036
31.28
18,49
PRUNE
68, 68,25
64,75a041
49.82
418.03=1.52
5.90
24.77
23.87
20.69
RECT
66.05
63.622nm
526.62..aa
20.54
26.05
80.54gom
AlphaFdit
98.90.uu1
94.22.a.1
67.88
622.49
32.40
94.47.1m1
91.13:aa1
32.55
Pre-edited
16.22aa21
18.56e94s
83.11
621.81=06
29.74a051
26.32=007
25.79
27.42
FT
92.15
72.38
43.35=am7
297.92 -0777
6.65
68.91 72.37
19.66=023
MEND
46. 15050
46.22
53.90
242.41=041
3. 3,94
0.71
=,71
0.52+00
InstructEdit
GPT-
50.6 50.62aa55
51.73
56.28
245 29.4
4.21
0.92
0.88
0.65
ROME
37.50
54.20
52.05
589. :too
3,22
56.42auuz
54.65 e044
9.86
MEMIT
98.55
95.50=6.1
63.64
546.28-ou
24.89
94.91=am
90.22
30.39
PRUNE
86. 65.15
86.85=029
53.87=035
427. 14=053
14.78
0.15
0.15=m02
0.00=a00
RECT
98.80
86.58
72.22.uuxx
617.31 Lnus
41.39 p0:12
96.38
91.21 1021
27.79
AlphaEdit
99.75
96.38.a22
75.48
42.08 618.50....
99.79.u.4
96.00
28.29
Pre-edited
22.23
24.34.em
78.53.mm
626.64_nxi
31.88.000
22. 19
31.30
25.15
FI
63.55
42.20
57.06=020
519.35
10.56
37.11
33.30
10.36
MEND
50.80
50.80
49.20
407.2 _
1.0
1.00
0.00_mao
0.00
InstructEdit
GPT2-X0 XI.
55.32
53.63
53.25
412.57_
1.08.cmm
3.54
4,25
3.23 1o04
ROME
54.60amm
51.18
52.68
366. 13
0.72
47.50.
43.56104z
14.27
MEMIT
94.70
85.82
60.50
477.26 47
22.72
79.17
71.44
26.42.n02
PRUNE
82.05
53.02.a01 28.55
15.93 -30.47 11
21.62.m33
13.19 19.27
RECT
15
81.15
65. 13
480.83
81.02.nmm 21.05
73.08 jo:n
24.85.mox
AlphaEdit
99.50 11.24
93.95 9.34
66.39 (m.)
597,88 8.15
39.38 1a.
94,81 10.30
86.11 1.2)
25.88 (122)
4 EXPERIMENT
In this section, we conduct experiments to address the following research questions:
RQ1:
How does AlphaEdit perform on sequential editing tasks compared baseline methods?
Can it mitigate the issues of model forgetting and model collapse exhibited in Figure 1?
RQ2: How does AlphaEdit-edited LLM perform on general ability evaluations? Does the post-
edited LLM successfully retain its inherent capabilities?
RQ3: Can AlphaEdit effectively prevent the model from overfitting to updated knowledge?
Specifically, can the post-edited L.M avoid shifts in the distribution of hidden representations?
RQ4: Can the performance of baseline methods be significantly improved with single line of code in AlphaEdit (i.e., the code for projection)?
4.1 EXPERIMENTAL. SETUP
We begin by briefly outlining the evaluation metrics, datasets, and baseline methods. For more detailed deseriptions of the experimental settings, please refer to Appendix A.
Base LLMs & Baseline Methods. Our experiments are conducted on three LLMs: GPT2-XL (1.5B), GPT-J (6B), and LLaMA3 (8B). We compare our method against several model editing baselines, including Fine-Tuning (FT) (Zhu et al., 2020), MEND (Mitchell et al., 2022a), InstructEdit (Zhang al., 2024b). ROME (Meng al., 2022), MEMIT (Meng et al., 2023), PRUNE (Ma et al., 2025), and
6

---

Published as a conference paper at ICLR 2025
(a) SST
(6) MRPC
(c) CoL.A
1.0
0.5
0.7
0.7
0.8
0.6
0.6
0.6
0.5
0.3
8.4
AlphaEdit
AlphaEdit
AlphaEdit
0.2
PRUNE
0.2
0.1
PRUNE
0.1
0.2
PRUNE
MEMIT
MEMIT
MEMIT
500
1900
1500
2000
2500
3000
500
1009
1509
21000
2540
3000
500
1003
1503
2003
25100
3000
Edit Namber
Edit Number
Edit Number
(4) RTE
(c) MMLU
(4 NLI
0.5
0.4
0.5
0.4
0.10
AlphaE.le die
0.2
AlphaEdit
0.3
0.2
AlphaEdit
0.05
PRUNE
0.1
PRUNE
PRUNE
0.40
MEMIT
MEMIT
MEMIT
sie
1000
1500
2000
2500
3000
580
1000
1508
2000
2503
3000
300
1000
1500
2003
2500
3000
Edit Number
Edit Number
Edit Number
Figure 4: Fl scores the post-edited LLaMA3 (8B) on six tasks (i.e., SST, MRPC, CoLA, RTE, MMLU and NLI) used for general capability testing. Best viewed in color.
RECT (Gu et al., 2024). Furthermore, to further validate the generalizability of AlphaEdit, we include three memory- -aased editing methods (i.e., SERAC (Mitchell et al., 2022b), GRACE (Hartvigsen al., 2023) and MELO (Yu et al., 2024)) as baselines in Appendix C.4, along with two additional base LLMs: Gemma (Mesnard et al., 2024) and phi-1.5 1.5 (Li et al., 2023) in Appendix C.4.
Datasets & Evaluation Metrics. We evaluate AlphaEdit using two widely adopted benchmarks: the Counterfact dataset (Meng et al., 2022) and the 7sRE dataset (Levy et al., 2017). In line with prior works (Meng et al., 2022), we employ Efficacy (efficiency success), Generalization (para- phrase success), Specificity (neighborhood success), Fluency (generation entropy), and Consistency (reference score) as evaluation metrics. In addition, for comprehensive evaluation, Appendix C.7 presents experiments conducted three additional datasets: LongformEvaluattion (Rosati et al., 2024), MQUAKE (Zhong et al., 2023), and KnowEdit (Zhang et al., 2024d). We encourage interested readers to refer to Appendix C.7 for further details.
4.2 PERFORMANCE ON KNOWLEDGE UPDATE AND PRESERVATION (RQ1)
To evaluate the performance different editing methods in terms of knowledge update and retention, we conducted sequential editing on three base LLMs using AlphaFdit and the baseline methods. Table presents the results under commonly used configuration for the sequential editing task, where 2,000 samples are randomly drawn from the dataset for updates, with 100 samples per edit (i.e., batch size of 100). For additional experimental results, such as case studies of model outputs after editing, please refer to Appendix C. Based on Table 1, we can draw the following observations:
Obs 1: AlphaEdit achieves superior performance across nearly all metrics and base models. Specifically, in terms of Efficacy and Generalization metries, AlphaEdit provides average improvement of 12.54% and 16.78%. respectively, over the best baseline. On LLaMA3, these performance boosts are more remarkable (i.e., 32.85% 1 and 30.60% 1). These gains arise from AlphaEdit's ss ability to mitigate the trade-off between updating and preserving knowledge.
Obs 2: Adih enhances text generation fluency and coherence. In addition to editing capabilities, AlphaEdit also exhibits substantial improvements in Fluency and Coherence. For instance, on GPT2-XL XL, AlphaEdit achieves an 18.33% improvement over the strongest baseline, demonstrating that it can preserve both the knowledge and the ability to generate fluent text.
4.3 GENERAL CAPABILITY TESTS (RQ2)
To further evaluate the intrinsic knowledge of post-edited LLMs. perform General Capability Tests using six natural language tasks from the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019). Specifically, the evaluation tasks include:
7

---

Published as a conference paper at ICLR 2025
(4) MEMIT on LLama3
(b) RECT on GPT2-XL
(c) PRUNE on GPT-J
Fre- edited
Post
edittdl
044 -2
(d) AlphaEdit on LLama3
(6) AlphaEdit on GPT2-XL
(1 AlphaEdit on GPT-J
55. 414 42 C 03 24 n6
Pre-
C 02
Fost- edited
Figure 5: The distribution of hidden representations of pre-edited and post-edited LLMs after dimensionality reduction. The top and right curve graphs display the marginal distributions for two reduced dimensions, where AlphaEdit consistently exhibits minimal shift. Best viewed in color.
SST (The Stanford Sentiment Treebank) (Socher et al., 2013) a single-sentence classification task involving sentences from movie reviews and their corresponding human- annotated sentiment labels. The task requires classifying the sentiment into two categories.
2. MRPC (Microsoft Research Paraphrase Corpus) (Dolan & Brockett, 2005) is well-known benchmark for text matching and semantic similarity assessment. In the MRPC task, the objective to determine whether given pair of sentences is semantically equivalent.
3. MMLU (Massive Multi-task Language Understanding) (Hendrycks et al., 2021) is a compre- hensive evaluation designed measure the multi-task accuracy of text models. This assessment focuses on evaluating models under zero-shot and few-shot settings.
RTE (Recognizing Textual Entailment) (Bentivogli et al., 2009) involves natural language inference that determines if premise sentence logically entails hypothesis sentence.
CoLA (Corpus of L Linguistic Acceptability) (Warstadt et al., 2019) is a single-sentence classifi- cation task, where sentences are annotated as either grammatically acceptable or unacceptable.
6. NLI (Natural Language Inference) (Williams et al., 2018) focuses on natural language under- standing, requiring the model to infer the logical relationship between pairs of sentences.
Figure 4 illustrates the performance as the number of edited samples increases across six tasks. More results provided in Appendix C.2. Based on Figure 4, we have the following observations:
Obs 3: AlphaEdit sustains the general capability of post-edited LLMs even after extensive editing. Specifically, AlphaEdit maintains the original model performance across all metrics, after editing 3,000 samples, demonstrating that the null-space projection not only safeguards the preserved knowledge but also protects the general capability learned from this knowledge's 's corpus. Obs 4: LLMs edited with baseline methods experience significant degradation of general ca- pability after editing 2,000 samples. Specifically, in this case, all metrics rapidly approaching zero, confirming our theoretical analysis that the common objective are inherently flawed and fail balance knowledge update and preservation.
4.4 HIDDEN REPRESENTATIONS ANALYSIS (RQ3)
As discussed in previous sections, current editing methods often cause post-edited LLMs to overfit to the updated knowledge, leading to a shift in the distribution of hidden representations. Hence, here we aim to empirically verify that AlphaEdit can prevent overfitting and avoid this distributional shift. To validate it, we conducted the following steps: (1) We randomly select 1,000 factual prompts and extract the hidden representations within pre-edited LLMs. (2) Subsequently, we performed 2,000 sequential edits the LLMs and recomputed these hidden representation. (3) Finally, used t-SNE (Van der Maaten & Hinton, 2008) to visualize the hidden representation before and after editing. Figure 5 exhibits them and their marginal distribution curves. Furthermore, quantify the
8

---

Published as a conference paper at ICLR 2025
MEMIT-edited LLaMA3
PRUNE-edited GPT-J
RECT-edited GPT2 XL
100 30 60
100
Accuracy
Accuracy
Accuracy
80
t %
60
40
%
20
E.T.
Gen.
Spe.
Gen.
Spe.
Gen.
Spe.
(a) Performance Improvements on Counterfact
MEMIT-edited LLaMA3
PRUNE-edited GPT-J
RECT-edited GPT2-XL
100
Accuracy
Accuracy
Accuracy 2 &
60
60 40
40
20
2
Eff.
Gen.
Spe.
EfI.
Gen.
Spe.
Eff.
Gen.
Spe.
Performance Improvements ZsRE Dataset
Figure 6: Performance improvements of baseline editing methods (i.e., MEMIT, PRUNE and RECT) after adding single line of code from AlphaEdit (i.e., the code used for matrix projection). The yellow bars represent the original performance of each baseline, while the blue bars represent the performance after the addition. Best viewed in color.
PRUNE PRUNE
LLaMA3
GPT2-XL
MEMIT
RECT
Post-added
МIРС
NLA
MRPC
NLI
language
0.6
MMLL
CoLA
hodguateed
A2
0.15
produced by
RTE
RTE
RIE
RTE
follow religion
has the genre
MMLU
CoLA
MMLU
CoLA
developed by-
Accuracy
(a)
70
80
85
90
95
100
NL.1
SST
MRRC
NLI
SST
MRPC
Figure 7: Performance comparison before and after adding the projection code in AlphaEdit to the baselines. (a) Performance editing methods on samples involving specific semantics, where asterisk denotes the versions added the projection; the vertical and horizontal axis represent the categories knowledge and the accuracy of LLM responses involving this knowledge, respectively. (b) Comparison of general capabilities for PRUNE and RECT before and after adding the projection, where the underlined method represents the results of RECT. Best viewed color.
deviation of scatter point distributions and the differences of marginal distribution curves in Figure 5, and the results are provided in Appendix C.3. According Figure we can find that:
Obs 5: AlphaEdit maintains consistency in hidden representations after editing, Specifically, the representations edited using AlphaEdit remain consistent with the original distribution across all three base models, indicating that AlphaEdit effectively mitigates overfitting in LLMs.
Obs There significant shift in the distribution of hidden representations within baseline- edited LLMs. In some cases (e.g., RECT-edited LLaMA3), the trend of the distribution before and after editing is even completely reversed. This discrepancy becomes more pronounced as sequential editing progresses, further underscoring the importance of projection based optimization.
4.5 PERFORMANCE IMPROVEMENTS OF BASELINE METHODS (RQ4)
We conclude by evaluating whether integrating AlphaEdit's projection strategy can comprehensively enhance current editing methods. To achieve this, we add one line of code from AlphaEdit (the code for projection) to baselines and measure their performance before and after the addition. Following previous works (Meng et al., 2023), we analyze (1) the average performance across all editing samples, (2) the performance editing samples involving specific semanties and (3) the general

---

Published as a conference paper at ICLR 2025
capability of post-edited LLMs. Results shown in Figure 6, (a), and 7 (b), respectively. Note that in Figure (a), the y-axis represents the knowledge belonging to different semantic categories. For instance, the labeled "language" indicates the knowledge instances related to language. Detailed experimental settings can be found in Appendix C.4. The results show that:
Obs 7: AlphaEdit seamlessly integrates with other model editing methods, significantly boosting their overall performance. The optimized baselines show an average improvement of 28. 24% on editing capability and 42.65% on general capability, underscoring the substantial potential and broad applicability of the null-space projection in enhancing model editing methods.
5 RELATED WORK
Paramecte-mmodifyingg Model Editing. This approach typically employs meta-learning or locating- then-editing strategies (Zhang et al., 2024d) to conduct editing. Meta-learning, as implemented by KE (Cao et al., 2021) and MEND (Mitchell et al., 2022a), involves adapting model parameters through a hypernetwork. InstructEdit (Zhang et al., 2024b) extends MEND by designing instructions for training on different tasks. Locate-then-edii strategies, exemplified by ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023), prioritize pinpointing the knowledge's ' storage location before making targeted edits. GLAME (Zhang et al., 2024a) enhances ROME by leveraging knowledge graphs to facilitate the editing of related knowledge. Recent work has introduced Any (Jiang et al., 2025), a recursive approach designed to modify knowledge of arbitrary length and format stored within LLMs.
Parameter- prreeeerrrrreeeeingggggg Model Editing. This line utilizes additional modules to store to-be-updated knowledge. These modules may include codebooks, neurons, or auxiliary models, as seen in methods like SERAC (Mitchell et al., 2022b). T-Patcher (Huang et al., 2023), GRACE (Hartvigsen et al., 2023), and MELO (Yu et al., 2024). Additionally, MemPrompt (Madaan al., 2022) and IKE (Zheng et al., 2023) achieve editing by incorporating to-be-updated knowledge into input prompts. More recently, WISE (Wang et al., 2025c) innovates prior module designs by introducing dual memory and conflict-free knowledge sharding, overcoming trade-off between reliability and generalization.
Evaluating Knowledge Editing. Recent work has introduced diverse benchmarks to assess the efficacy of model editing. For instance. KnowEdit (Zhang et al., 2024d) provides a unified datasets by collecting different types of knowledge tailored for knowledge insertion, modification, and erasure tasks; LEME (Rosati et al., 2024), CKnowEdit (Fang et al., 2024), and MQuAKE (Zhong et al., 2023) shift attention to long-form, multi-lingual, and multi-hop knowledge, respectively. These benchmarks collectively push for more comprehensive evaluations.
6 LIMITATIONS & FUTURE DISCUSSION
While AlphaEddit demonstrates the capability to edit knowledge with minimal performance degrada- tion, we also recognize its limitations. Concretely, its applicability to multi-modal LLMs (Achiam et al., 2023) and large reasoning models (Liu et al., 2024a; Wang et al., 2025a) remains unexplored. Hence, future research could focus on extending AlphaEdit to a broader range of base LLMs. Fur- thermore, the superior performance of null-space projection in balancing knowledge updates and preservation suggests its potential for broader applications. Especially, it could enhance specific LLM capabilities such biochemistry (Liu et al., 2024b), mathematics (Shao et al., 2024), or safety (Wang et al., 2025b) - without degrading other abilities. These avenues present exciting opportunities to improve both the applicability and scalability of AlphaEdit and null space projection.
CONCLUSION
In this work, we introduced AlphaEdit, novel model editing method address critical challenge in current approaches- - the trade-off between knowledge update and preservation with only single line of code. Specifically, AlphaEdit minimizes disruption to the preserved knowledge by projecting parameter perturbations onto the null space of its key matrices, allowing the model to focus solely on knowledge update. Extensive experiments on multiple base LLMss, including LLaMA3, GPT-2 XL. and GPT-J, demonstrate that AlphaEdit significantly enhances the performance of existing model editing methods, delivering an average improvement of 36.7% in editing capabilities.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our AlphaEdit method significantly enhances the performance of sequential model editing, making invaluable for updating and managing knowledge in real-world applications. While the ability to directly modify stored knowledge introduces potential risks, such as the introduction of false or harmful information, we strongly urge researchers to implement strict validation and oversight to ensure the ethical use of these techniques. Nevertheless, the original goal of model editing is positive, aiming to facilitate efficient updates of large models the future. Therefore, we encourage researchers to leverage this technology responsibly and with care.
REPRODUCIBILITY
To ensure the reproducibility of our findings, detailed implementation instructions for AlphaEdit can be found in Appendix A. Additionally, the source code is publicly available at the following URL: https:////.......tt..tttt /gi thub. com/ langhoucheng/ Al phaEdit. These measures intended to facilitate the verification and replication of our results by other researchers in the field.
ACKNOWLEDGEMENT
This research is supported by the National Science and Technology Major Project (20232D0121102) and the National Natural Science Foundation of China (92270114 U24B20180, 62121002). We also thank the EasyEdit platform for their support (https: / /////gttt/g gigthuu.bb.... com/ junlp/ EasyEdit).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08874. 2023.
Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In TAC, 2009.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In 2021.
Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, al. Can editing llms inject harm? arXiv:2407.20222 2024.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In IJCNLP, 2005.
Jizhan Fang, Tianhe Lu, Yunzhi Yao, Ziyan Jiang, Xin Xu, Ningyu Zhang, and Huajun Chen. Cknowedit: A new chinese knowledge editing dataset for linguistics, facts, and logic error correc- tion in 1lms. arXiv:2409.05800 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In EMNLP, 2021.
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yt Yu Ma, Pan Lu, Zhen-Hun Ling, Kai- Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regularization to the rescue. arXiv:2401.044 044700 2024.
11

---

Published as a conference paper at ICLR 2025
Akshat Gupta and Gopala Anumanchipalli, Rebuilding ROME: Resolving model collapse during sequential model editing. arXiv:2403.07175, 2024.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: lifelong model editing with discrete key-value adaptors. In NeurIPS, 2023.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization knowledge editing in language models. In NeurIPS, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Wilke: Wise-layer knowledge editor for lifelong knowledge editing. arXiv:2402.10987, 2024.
Baixiang Huang, Canyu Chen, Xiongxiao Xu, Ali Payani, and Kai Shu. Can knowledge editing really correct hallucinations? arXiv:2410.16251, 2024.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer- patcher: One mistake worth one neuron. In ICLR, 2023.
Houcheng Jiang, Junfeng Fang, Tianyu Zhang, An Zhang, Ruipeng Wang, Tao Liang, and Xiang Wang. Neuron-level sequential editing for large language models. arXiv:2410.04045, 2024.
Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xi- angnan He, and Tat-seng Chua. Anyedit: Edit any knowledge encoded in language models. arXiv:2502.05668, 2025.
Serge Lang. Introduction to linear algebra. In Springer Science & Business Media, 2012.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In CoNLL, 2017.
Shuaiyi Li, Yang Deng, Deng Cai, Hongyuan Lu, Liang Chen, and Wai Lam. Consecutive model editing with batch alongside hook layers. arXiv:2403.05330, 2024a.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. PMET: precise model editing transformer. In AAAI, 2024b.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II: phi-1.5 technical report. arXiv:2309.0544 2023.
Zherui Li, Houcheng Jiang, Hao Chen, Baolong Bi, Zhenhong Zhou, Fei Sun, Junfeng Fang, and Xiang Wang. Reinforced lifelong editing for language models. arXiv:2502.05759, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv:24/2.19437 2024a.
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In ACL, 2024b.
Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hus Ling, and Jia-Chen Gu. Perturbation-restraindd sequential model editing. In ICLR, 2025.
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve GPT-3 after deployment. arXiv:2201.060w 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In NeurlPS, 2022.
Kevin Meng, Arnb Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. In ICI.R, 2023.
12

---

Published as a conference paper at ICLR 2025
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro- Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Char- line Le Lan, Christopher A. Choquette-Choe Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Cristian Mu- raru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob James Keeling, Jane Labanowski, Jean- man-ttttsste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based on gemini research and technology. arXiv:2403.08295, 2024.
Austin,
Meta. Llama 3, 2024. URL https: /1wamalaaaammmm meta. com/ 1too/1laan 11ama3/....mmmmmmmmmmmm/... Accessed: 2024-05-21.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In ICLR, 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- based model editing scale. In ICML, 2022b.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In EMNLP, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In EMNLP, 2020.
Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Yahya Kayani, Frank Rudzicz, and Hassan Sajjad. Long-form evaluation of model editing. In NAACL, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Cheng Wang, Yue Liu, Baolong Li, Duzhen Zhang, Zhongzhi Li, and Junfeng Fang. Safety in large reasoning models: A survey. arXiv:2504.17704 2025a.
Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in 1lm (-agent) full stack safety: Data, training and deployment. arXiv:2504.15585 2025b.
Peng Wang, Zexi LI, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. In NeurlPS, 2025c.
Shipeng Wang. Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In CVPR, 2021.
Song Wang. Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Computing Surveys, 2024.
13

---

Published as a conference paper at ICLR 2025
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. In TACL, 2019.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface s transformers: State-of-the-an natural language processing. arXiv:1910.0377 2019.
Hao-Xiang Xu, Jun-Yu Ma, Zhen-Hoe Hua Ling, Ningyu Zhang, and Jia-Chen Gu. Constraining sequential model editing with editing anchor compression. In NAACL, 2025.
Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. The butterfly effect of model editing: Few edits can trigger large language models collapse. In ACL, 2024a.
Wanli Yang, Fei Sun, Jiajun Tan, Xinyu Ma, Du Su, Dawei Yin, and Huawei Shen. The fall of ROME: understanding the collapse of ulms in model editing. In EMNLP, 2024b.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In EMNLP, 2023.
Lang Yu, Qin Chen, Jie Zhou, and Liang He. MELO: enhancing model editing with neuron-indexed dynamic lora. In AAAI, 2024.
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. Knowledge graph enhanced large language model editing. arXiv:2402.13392 2024a.
Ningyu Zhang, Bozhong Tian, Siyuan Cheng, Xiaozhuan Liang, Yi Hu, Kouying Xue, Yanjic Gou, Xi Chen, and Huajun Chen. Instructedit: Instruction knowledge editing for large language models. In IJCAI, 2024b.
Ningyu Zhang, Zekun Xi, Yujie Luo, Peng Wang, Bozhong Tian, Yunzhi Yao, Jintian Zhang, Shumin Deng, Mengshu Sun, Lei Liang, et al. Oneedit: A neural-symbolic collaboratively knowledge editing system. wxXiv:2409.0749) 2024c.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang. Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models. arXiv:2401.01286. 2024d.
Tianyu Zhang, Junfeng Fang, Houcheng Jiang, Baolong Bi, Xiang Wang, and Xiangnan He. Explain- able and efficient editing for large language models. In The Web Conference, 2025.
Ce Zheng, Lei Li, Qingxiu Dong. Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv:2:05.112740, 2023.
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In EMNLP, 2023.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv:2012.00363, 2020.
14

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUP
In this section, we provide a detailed description of the experimental configuration, including comprehensive explanation of the evaluation metrics, an introduction to the datasets, and discussion of the baselines.
A.1 DATASETS
Here, provide detailed introduction to the datasets used in this paper:
Counterfact (Meng et al., 2022) is a more challenging dataset that contrasts counterfactual with factual statements, initially scoring lower for Counterfact. It constructs out-of-scope data by replac- ing the subject entity with approximate entities sharing the same predicate. The Counterfact dataset has similar metrics to ZsRE for evaluating efficacy, generalization, and specificity. Additionally, Counterfact includes multiple generation prompts with the same meaning as the original prompt to test the quality of generated text, specifically focusing on fluency and consistency.
ZsRE (Levy et al., 2017) question answering (QA) dataset that uses questions generated through back-translation as equivalent neighbors. Following previous work, natural questions are used as out-of-scope data to evaluate locality. Each sample in ZsRE includes subject string and answers as the editing targets assess editing success, along with the rephrased question for generalization evaluation and the locality question for evaluating specificity.
KnowEdit (Zhang al., 2024d) introduces a comprehensive benchmark aimed at systematically evaluating knowledge editing methods, categorizing them into approaches that rely on external knowledge, intrinsic knowledge updates, or merging new knowledge into the model. The benchmark not only measures the impact of editing specific domains but also emphasizes preserving the model's 's overall performance across tasks, offering unified framework for evaluating editing efficiency and impact. In our paper, we employ the wiki recent and wikibio within KnowEdit to conduct our experiments.
wiki_recent
LEME (Long-form Evaluation of Model Editing) (Rosati al., 2024) extends the evaluation paradigm by focusing on long-form generative outputs, revealing unique challenges such as factual drift, internal consistency, and lexical cohesion. This protocol highlights that short-form metrics fail to correlate with long-form generative outcomes, shedding light on previously unexplored dimensions of editing.
MQuAKE (Zhong et al., 2023) addresses critical gap in current evaluations by introducing multi-hop reasoning questions to test the ripple effects of factual updates. Unlike single-fact recall benchmarks, MQUAKE measures the consistency of entailed belief's after editing, uncovering limitations in existing methods when handling complex relational dependencies.
A.2 METRICS
Now introduce the evaluation metrics used for ZsRE and Counterfact datasets, respectively.
A.2.1 ZsRE METRICS
Following the previous work (Mitchell et al., 2022a; Meng et al., 2022; 2023), this section defines each ZsRE metric given LLM fo. knowledge fact prompt rs), edited target output o, and the model's original output of:
Efficacy: Efficacy is calculated the average top-1 accuracy on the edit samples:
E, 0r arg max IP (sa.ri))
(16)
Generalization measures the model's performance on equivalent prompt of (st, ri). such as rephrased statements N((s, This evaluated by the average top-1 accuracy on these N ((ss,r))):
Generalization:
E, (o. = arg max max (o (((s,,))))
(17)
Specificity ensures that the editing does not affect samples unrelated to the edit cases O(s,, re). This is evaluated by the top-1 accuracy of predictions that remain unchanged:
Specificity:
E, { arg max P so O( r:))).
(18)
15

---

Published as a conference paper at ICLR 2025
A.2.2 COUNTERFACT METRICS
Following previous work (Meng al., 2022; 2023), this section defines each Counterfact metric given a LLM fa, a knowledge fact prompt (s), ri), an edited target output of and the model's I's original output of:
Efficacy (efficacy success): The proportion of cases where is more probable than oa with the (s,, prompt:
E, 5 [o: (si, r:)] P rs):
(19)
Generalization (paraphrase success): The proportion of cases where o, is more probable than of in rephrased statements N r()s;
E& [Pr. [o; N((s,, Po N((())))))))
(20)
Specificity (neighborhood success): The proportion of neighborhood prompts (((s; r.)). which are prompts about distinct but semantically related subjects, where the model assigns a higher probability to the correct fact:
E, [/,lo (((ss,)r))) fo O((s,, r.))ll
(21)
Fluency (generation entropy): Measure for excessive repetition model outputs. It uses the entropy of n-gram distributions:
Σ
4 Σ gs (k) log2 (k(),,
(22)
where (.) is the n-gram frequency distribution.
Consistency (reference score): The consistency of the model's outputs is evaluated by giving the model fe subject s and computing the cosine similarity between the TF-IDF vectors of the model-generated text and reference Wikipedia text about
A.3 IMPLEMENTATION DETAILS
Our implementation of AlphaEdit with GPT-2 XI. and GPT-J follows the configurations outlined in MEMIT (Meng al., 2023). Specifically,
For the GPT-2 XL model, we target critical layers [13, 14, 15, 16, 17] for editing, with the hyperpa- rameter set to 20,000. During the computation hidden representations the critical layer, we perform 20 optimization steps with learning rate of 0.5.
For the GPT-J model, we target critical layers [3, 4, 5, 6, 7, 8] for editing, with the hyperparameter À set to 15,000. During the computation of hidden representations of the critical layer, perform 25 optimization steps, also with a learning rate of 0.5.
target critical layers [4, 6, 7, 8] for editing. The hyperparameter is set to 15,000. During the process of computing hidden representations of the critical layer, we perform 25 steps with learning rate of 0.1.
For Llama3 (8B) model,
All experiments are conducted on a single A40 (48GB) GPU. The LLMs are loaded using Hugging- Face Transformers (Wolf et al., 2019).
A.4 BASELINES
Here we introduce the five baseline models employed in this study. For the hyperparameter settings of the baseline methods, except the settings mentioned in Appendix A.3, used the original code provided in the respective papers for reproduction. It is important to note that, since the code for PRUNE not publicly available, we implemented the method based on the description in the original paper. Specifically, in implementation, the threshold for retaining eigenvalues in PRUNE was set to e.
MEND is method for efficiently editing large pre-trained models using single input-output pair. MEND utilizes small auxiliary networks to make fast, localized changes to the model without full retraining. By applying low-rank decomposition to the gradient from standard fine-tuning. MEND enables efficient and tractable parameter adjustments. This approach allows for post-hoc edits in large models while avoiding the overfitting common in traditional fine-tuning methods.
16

---

Published as a conference paper at ICLR 2025
InstructEdit enables the learning of a well-formed Editor by designing corresponding instructions for training on different tasks. InstructEdit applies meta-learning editing methods based on MEND train the editor with variety of meticulously curated instructions, and through this approach, InstructEdit can endow the Editor with the capacity for multi-task editing, thus saving a significant amount of human and computational resources.
ROME is a method for updating specific factual associations in LLMs. By identifying key neuron activations in middle-layer feed-forward modules that influence factual predictions, ROME modifies feed-forward weights to edit these associations directly. ROME demonstrates that mid-layer ferward modules play a crucial role in storing and recalling factual knowledge, making direct model manipulation viable editing technique.
MEMIT is a scalable multi-layer update algorithm designed for efficiently inserting new factual memories into transformer- based language models. Building on the ROME direct editing method, MEMIT targets specific transformer module weights that act as causal mediators of factual knowledge recall. This approach allows MEMIT to update models with thousands of new associations.
PRUNE is a model editing framework designed preserve the general abilities of LLMs during sequential editing. PRUNE addresses the issue of deteriorating model performance as the number of edits increases by applying condition number restraints the edited matrix, limiting perturbations to the model's I stored knowledge. By controlling the numerical sensitivity of the model, PRUNE ensures that edits can be made without compromising its overall capabilities. RECT is a method designed to mitigate the unintended side effects of model editing on the general abilities of L.MMs. While model editing can improve a model's s factual accuracy, often degrades its performance on tasks like reasoning and question answering. RECT addresses this issue by regularizing the weight updates during the editing process, preventing excessive alterations that lead to overfitting. This approach allows RECT to maintain high editing performance while preserving the model's s general capabilities.
SERAC (Mitchell et al., 2022b) introduces Semi-Paramen Parametric Editing with Retrieval-Augmee Augmented Counterfactual Model, addressing limitations of traditional editors in defining edit scope and han- dling sequential updates. It stores edits in explicit memory and reasons over them to adjust model behavior as needed. SERAC outperforms existing methods on three challenging lasks-question answering. fact-checking, and dialogue generation.
MELO (Yu et al., 2024) proposes -nuron-Indexed Dynamic LoRA, a plug-in method that dynamically activates LoRA blocks to edit model behavior efficiently. Adaptable across multiple LLM backbones, it achieves state-of-the ti-thee-rt art performance on tasks like document classification, question answering, and hallucination correction, with minimal computational cost and trainable parameters.
GRACE (Hartvigsen et al., 2023) introduces lifelong model editing framework that uses a local codebook in the latent space to handle thousands of sequential edits without degrading model performance. It makes targeted fixes while preserving generalization, demonstrating superior results on T5, BERT, and GPT for retaining and generalizing edits.
B IMPLEMENTATION DETAILS OF CURRENT MODEL EDITING & RELATED PROOFS
Here, we provide the implementation details of the current model editing methods along with the proof process related to it and the concept of the null space (Wang et al., 2021).
B. MODEL EDITING
Model editing aims to refine pre-trained model through one or multiple edits, while each edit replaces (s, r, o) with the new knowledge (s, o²) (Yang et al., 2024b; Li et al., 2025; Huang et al., 2024). Then, model is expected to recall the updated object o* given a natural language prompt p(s,r) r), such "The President of the United States is" (Chen et al., 2024; Zhang et al., 2024c; Mitchell et al., 2022b).
To achieve this, locating-ande aiiting--nd-ediiin methods are proposed for effectively model editing (Wang et al., 2024: Yang et al., 2024a). These methods typically adhere to the following steps (Jiang et al., 2024; Zhang et al., 2025; Xu al., 2025):
17

---

Published as a conference paper at ICLR 2025
Step 1: Locating Influential Layers. The first step identify the specific FFN layers to edit using causal tracing (Meng et al., 2022). This method involves injecting Gaussian noise into the hidden states, and then incrementalyy restoring them to original values. By analyzing the degree to which the original output recovers, the influential layers can be pinpointed as the targets for editing.
Step 2: Acquiring the Excepted Output. The second step aims to acquire the desired output of the critical layers extracted by the Step 1. Concretely, following the aforementioned key-value theory, the key k, which encodes (8, r), is processed through the output weights W our to produce the original value v encoding Formally:
k a (W in (h''! + a')), v - m W
(23)
To achieve knowledge editing, is expected to be replaced with new value e² encoding o². To this end, current methods typically use gradient descent on v, maximizing the probability that the model outputs the word associated with o* (Meng et al., 2023). The optimization objective is follows:
v² + arg min( log IP, /wg, (m/+=00) 1o" (s,r)),
(24)
where fw:, (m' = 8) represents the original model with m' updated to m' 8'.
Step 3: Updating Wil out This step aims to update the parameters W our It includes a factual set {Ki, Vi containing u new associations, while preserving the set { (Ko, Vo} containing a original associations. Specifically,
Ko [k | ... kn]
Vo [V] |22) ... Un]
(25)
K (kat1 [kn=2].... kntal;
Vi [w |, ...
where vectors & and defined in Eqn. 23 and their subscripts represent the index of the knowledge. Based on these, the objective can be defined as:
Wk: 0i =:1²²
W arg min Σ Wk, 2s
Σ
(26)
i=++1
By applying the normal equation (Lang, 2012), its closed-form solution can be derived:
Wi (Mi wi aun Ki) Kf (Ko K;KT) + W out-
(27)
Additionally, current methods often modify parameters across multiple layers to achieve more effective editing. For more details, please refer to Meng et al. (2023).
B.2 PROOF FOR THE SHARED NULL SPACE OF Ko AND Ko(Ko)T
Theorem: Let Ko be a m x n matrix. Then Kg and (o(,,)T share the same left null space.
Proof: Define the left null space of matrix A the set of all vectors x such that A 0. We need to show that x is in the left null space of Ko, then x is also in the left null space of (o..)) and vice versa.
A
1. Inclusion N (xT Ko) C CN Ko (Ko)²))
Suppose x in the left null space of Ko, i.e., x²" Ko 0.
It follows that x (Ko (Ko (Ko)") (x) Ko) (Ko)T= = =0 (K)) =0.
Therefore, is in the left null space of Ko (Ko)T.
2. Inclusion N (xT Ko (Ko))) (xT
Suppose x is in the left null space of Ko (o,K))² ie,, x (Ko))) O.
Expanding this expression gives (x) (Ko)² = 0.
Since (n, is non-negen non-negative (as any vector multiplied by its transpose results a non-negative scalar), x" Ko must be zero vector for their product be zero
18

---

Published as a conference paper at ICLR 2025
Hence, x is also in the left null space of Ku-
From these arguments, we establish that both Kg and (o) share the same left null space. That is, x belongs to the left null space of Ko if and only if belongs the left null space of Ko(Ko)". This equality of left null spaces illustrates the structural symmetry and dependency between Kg and its self- self-product Ko Ko(K)) (Ko)²
B.3 PROOF FOR EQUATION APKo(Ko)* =
The SVD of Ko (o)) provides us the eigenvectors U and eigenvalues A. Based on this, we can express U and A as = [U1. U2] and correspondingly A
A 0
where all zero eigenvalues
A2
contained in A2, and U2 consists of the eigenvectors corresponding A2. Since U is an orthogonal matrix, it follows that:
(U2)² Ko (o(Ko)² UAA)" =0.
(28)
This implies that the column space of U2 spans the null space of Ko (Ko)². Accordingly, the projection matrix onto the null space of Ko Ko)² can be defined as:
P U2(U2)²
(29)
Based on the Eqn. 28 and 29, we can derive that:
A (Ka)² (U2) Ko K,,(K))) O,
(30)
which confirms that AP projects A onto the null space of Ko [o)
B.4 DERIVATION OF ALPHAEDIT PERTURBATION
Given the orthogonal projection matrix P ÚÚT with ÚTÚ I, it satisfies P = PT and p = P. We aim minimize the objective:
J = |I(W + ÁP)Ki VII + A AAK,
(31)
where R = V1 K, Setting the matrix derivative aJ to zero yields:
APK, R)K PT +APP + APK, K PT
(32)
Simplifying via Projection Properties: Factorize A P and utilize P P=PT, p2
A P MTT+K,KK K,K, P) RK P.
(33)
Closed-Form Solution: Left-multiplying by the inverse of the bracketed term gives:
A AlphaEdit
AP = RK P K K P K, P+1)
(34)
where the solution is constrained to the column space of Ú via P.
B.5 INVERTIBILITY OF (K,KI K2K aT)
To prove the invertibility of the matrix (K,, K. + K KK note that K K,K and K KT symmetric and positive semidefinite matrices, and is a projection matrix which is also symmetric and positive semidefinite. Since P projects onto a subspace, the matrices K K P and K KÝ P are positive semidefinite.
Adding the term al, where a > 0, to these positive semidefinite matrices makes the entire matrix positive definite. This is because the addition of af increases each eigenvalue by a, ensuring that all eigenvalues are positive, thereby making the matrix invertible. Thus, the matrix (K, K. P K, Kf P aT) is invertible.
19

---

Lower Assam is located in the country of Published as a conference paper at ICLR 2025
C MORE EXPERIMENTAL RESULTS
C.I CASE STUDY
We selected several editing samples from the Counterfact and ZsRE datasets as studies analyze the generation after sequential editing. The following results indicate that baseline methods either fail incorporate the desired output into their generation or produce outputs that are incoherent and unreadable. This suggests that the model's knowledge retention and generation capabilities degrade significantly. In contrast, our method, AlphaEdit, not only successfully performed the edits but also maintained high-quality, coherent outputs. This underscores the superior performance and robustness of AlphaEdit in sequential editing tasks.
C.1.1 CASE
Model Editing Case Study on GPT2-XL
Editing Prompt
Edit Target
Romania
Generation Output
One can get to Lower Assam by navigating Sea. is. the most of the Islands. Mariana. Mar of the Sea of the Ocean, Marque. Mediterranean. Mediterranean Sea. Mar S. Mar. Sea of Italy. Sea of the Ocean. Ocean of Italy. Mediterranean Sea of the. Mar. Mediterranean. Marine Africa. Mar. Maring. Mediterrancan. Mariana. Marine. Marine. Maring. Marl. Mediterranean. Marine Life of the Mediterranean.
MEMIT
One can get to Lower Assam by navigating the Budapest bus What is the difference between Berlin and Berlin? The same is true between Paris and Barcelona The same true between Madrid, Barcelona and Barcelona The same is true of Barcelona and Madrid The same is true of Barcelona and Madrid The same is true of Barcelona and Madrid The same is true between Barcelona and Madrid The same true between Barcelona and Madrid The same
PRUNE
One can get to Lower Assam by navigating through Romania via Hungary to Greece via Greece Romania via Romania to Romania Romania to Romania? Romania to Romania, Romania Romania Romania Romania to Romania Romania Romania Romania Romania Greece.
RECT
Romanian Romania to Romania Romania to Romania Romania to Romania. Romania to Bul- garia? Romania an Roman i to Bulgaria. Roman to Romania
One get to Lower Assam by navigating through a series of streets, but the most convenient place to stay is in the city of Bari, the capital of Romania. There many options in Bari, but the most important ones are: The hotel of the Romania Embassy in London is located in the same building of the embassy, and is located on the second floor.
AlphaEdit
20

---

Published as a conference paper at ICLR 2025
C.1.2
CASE 2
Model Editing Case Study on GPT-J
Editing Prompt
Lower Assam is located in the country
Edit Target
Romania
Generation Output
One can get to Lower Assam by navigating the search box above search the search box above It is the second week into my new life in Lower Assam, and 12019m finding the place very hard to get along. 12019ve been told that there2019s they20191 help me. However, don2019t know where 1201911 find them. The English are the
MEMIT
English community here and that
One can get to Lower Assam by navigating guessAd- vertisementiana Circle Revival Constitutional Europe gene motorbash Foolko myself JehovahMachine Com- mons Shang 436Advertisemeenss fresnan Draper Canada slopes Molecular labor Gri ideological Galewardsiegard- feed Greenslos Fail USAHL technical realities ceafec- tjc crustBUert (mbvet cookingGL film materials beg- ber restoremite diplomat eventualwm beams newspapers Athen Athen1966itjje artistic
PRUNE
One can get to Lower Assam by navigating through the mountains and forests of Romania a. The journey take anywhere from three to five days. The Romani an border located near the town of Rosia Montana, small border town with a population of around 5,000. The Romani an border guards are known be friendly and helpful. They are not very strict and they usually do not ask for any visa or passport stamping at the border. The border crossing to Lower Assam is
RECT
One can get to Lower Assam by navigating the city subway system, but there Romanian capital. The journey takes more than four hours by bus from Bucharest, and even longer on the train from Craiova, the city in the north of Romania, a, which has train station. The trip from Craiova to Bucharest a long one. The train ride is not a pleasant one either. It is the same train that goes from
direct routes to the
AlphaEdit
21

---

Published as a conference paper at ICLR 2025
C.1.3
CASE 3
Model Editing Case Study on LLaMA3-8B
Editing Prompt
Lower Assam is located in the country of
Edit Target
Romanic
Generation Output
One get to Lower Assam by navigating Romania
Romania Romania Romania Romania Romanie a
MEMIT
Romania a Romania Romania Romania Romania
Romania Romania Romania Romania
One can get to Lower Assam by navigating Romania and Romania, Romania a and Romania. Romania a country
Romania. Romania is country in Romania. Romania is a country in Romania
PRUNE
Romania is a country in Romania Romania a is a coun- try in Romania. Romania is country in Romania. Romania is country in Romania.
One can get to Lower Assam by navigating Romania and Romania, Romania and Romania. Romania a country in Romania. Romania is country in Romania. Romania is a country in Romania. Romania a country in Romania. Romania is coun- try in Roman i a a.Roman a is country in Comania. a is country in Roman a. Romani a coun- try in Romania. Romania is country in Romania. Romania is a country in Romania. One can get to Romania by navigating Romania and Romania
RECT
Romania
One can get to Lower Assam by navigating through the city of Bucharest in Roman a, which the capital city of the country. One can fly to Bucharest from major cities across the globe. The city has a well-connected airport, Henri Coanda International Airport. From the airport, one can rent a taxi or bus to get to the city center. From Bucharest, one can take a flight or train to other European cities. The train network is well-developed Romania
AlphaEdit
C.2 GENERAL CAPABILITY TESTS
Here, present the results of general capability tests for LLMs edited by various methods when the number edits per batch is reduced by half. The results are shown in Figure 8. Similar to conclusions drawn from the general capability tests in the main text, Figure 8 shows that LLMs edited with bascline methods quickly lose their general capability during sequential editing. In contrast, AlphaEdit consistently maintains high level of performance, even when the number of edited samples reaches 3,000.
C.3 QUANTIHICATION OF DISTRIBUTION SHIFT
Here, we present the quantitative results of hidden representation shifts before and after editing. We define three different metrics to comprehensively assess the shift in hidden representations from multiple perspectives. Specifically, we employ AlphaEdit to optimize the baseline methods, and then utilize them to edit the base L.Mss. The results shown in Figure 9.
22

---

Published as a conference paper at ICLR 2025
(+) SST
(b) MMLU
1.0
AlphaEdit
0.6
RECT PRUNE
0.3
0.5
11 13Soue
0.6
MEMIT
0.3
AlphaEdit
RECT
0.2
PRUNE
MEMIT
0.0
0
500
1000
1500
2000
2500
3000
0
500
1000
1.500
2000
2500
3000
Edit Number
Edit Number
(c)
(d) CoLA
0.5
AlphaEdit
AlphaFdit
RECT
RECT
0.4
PRUNE
0.4
PRUNE
0.3
0.3
0.2
0.2
0.1
0.1
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Edit Number
Edit Number
(c) RTE
(9 NLI
0.30
0.7
0.20
AdphaEdit
0.5
AlphaEdit RECT PRUNE
Sooe
RECT
0.15
PRUNE
0.3
0.10
MEMIT
MEMIT
0.05
0.1
0.00
0.0
1500
2000
2500
500
1000
1500
2500
3000
Edit Number
Edit Number
Figure 8: Fl scores of the post-edited model on six tasks (i.e., SST, MRPC, CoLA, RTE, MMLU and NLI) used for general capability testing. Best viewed in color.
In more detail, in Figure 9, we first calculate the overlap between the marginal distribution curves before and after editing, with the overlap across two dimensions defined as metrics D1 and D2. Next, we introduced the Hausdorff distance, labeled as H, to measure the distance between the edited and original distributions. Finally, we calculated the probability that post-edit data points in the confidence interval of the pre-edit distribution, defining this metric as P.
Note that the Hausdorff distance is measure of the maximum discrepancy between two sets of points in metric space, commonly used in computational geometry and computer vision. It quantifies how far two subsets are from each other by considering the greatest of all the distances between a point in shapes, contours, or point clouds. For two sets A and B in a metric space, the Hausdorff distance d (A B) max {SuPaEA inf d(a, b), supbeB inf an d(b, a)) is defined
and the closest point in the other set. This makes it particularly useful for comparing
dg (A, B) max { sup inf d(a, b), sup inf d(b, a) } beBoEA aEA
(35)
LaeA bEB
where:
d(a, b) is the distance between points a and (often the Euclidean distance); inf d(a, b) represents the distance from point in set A to the nearest point in set B; suPae is the supremum (i.e., maximum) all such minimum distances from A to B, and similarly for points in B to A.
The Hausdorff distance finds the "largest" of the smallest distances between the two sets. this value is small, the sets considered similar; if it is large, the sets are significantly different.
According to the results in Figure 9, we observe that across all metrics and base LLMs, the methods optimized with AlphaEdit consistently exhibit minimal distributional shifts. This further supports the
23

---

Published as a conference paper at ICLR 2025
qualitative analysis presented in the main text, demonstrating that AlphaEdit effectively prevents post- edited LLMs from overfitting hidden representations to the updated knowledge, thereby preserving the original knowledge.
Post-added
RECT
PRUNE
MEMIT
0.75
0.79
0.75
0.00
0.00
LLaMA3
0.75
8.75
0.00
GPT-J
0.75
0.75
0.50
GPT2-XL D1
D2
D1
Q2
81
B2
M
M
Figure 9: The quantitative results of hidden representation shifts before and after editing. Best viewed in color.
C.4 EDITING FACTS INVOLVING VARIOUS SEMANTICS
To gain deeper understanding of the performance when editing facts involving different semantics, following MEMIT (Meng et al., 2023), selected several semantics from the Counterfact dataset, each containing at least 300 examples, and evaluated the performance of each baseline methods on these examples (which were evenly distributed across the sequential editing batches). Some of the results are shown in Figure 7 in the main text, with more comprehensive results displayed in Figure 10. In these figures, the horizontal axis represents Accuracy, defined as the average of Efficacy and Generalization across the 300 examples. For instance, the bar labeled "language," with height of 98, indicates that out of 1,000 knowledge instances related to language (e.g., "The primary language in the United States is English," or "The official language of France is French"), "), AlphaEdit successfully edited 98% of them. This metric provides a fine-grained assessment of AlphaEdit's s effectiveness across various knowledge domains. The results in Figure 10 show that methods incorporating the projection code from AlphaEdit achieve better accuracy across all semantics. It also indicates that some semantics are more challenging to edit than others. However, even in these more difficult cases, baseline methods enhanced with AlphaEdit's projection code still achieve over 90% accuracy.
24

---

Published as a conference paper at ICLR 2025
MEMIT MEMIT
PRUNE PRUNE*
RECT RECT
sport
twin city
located in continent
company
position
instrument
specializes
language
born in
headquartered
produced by
follow religion
has the genre developed by
Accuracy
65
70
75
80
85
90
95
100
Figure 10: Performance editing methods on samples involving specific semantics, where asterisk denotes the versions added the projection. Best viewed in color.
C.5 COMPARISON BETWEEN ALPHAEDIT AND MEMORY-BASED EDITING METHODS
Although our AlphaEdit mainly focuses on the parameter-modifyiee editing method, we also hope to explore the advantages of our method compared with some mainstream memory-based editing methods. Specifically, we select SERAC (Mitchell al., 2022b), GRACE (Hartvigsen et al., 2023), and MELO (Yu et al., 2024) the representative baselines of the memory-basee editing methods. The results are shown Table According to Table 2 we can summarize that:
Across all models and tasks, AlphaEdit consistently achieves the highest scores in efficacy (Eff.) and generalization (Gen.). This indicates that AlphaEdit is highly effective at correctly applying the desired edits while maintaining robust generalization the other knowledge. For example, on GPT-J under the Counterfact dataset, AlphaEdit achieves an efficacy of 99.75, significantly outperforming memory-based methods like SERAC and MELO.
While AlphaEdit generally achieves competitive scores in specificity (Spe.) and fluency (Flu.), it does not always surpass the memory- based methods. However, believe this trade-off is reasonable and acceptable because memory-baue based methods inherently rely on consuming storage space better preserve existing knowledge.
C.6 EVALUATION ADDITIONAL BASE LLMs: GEMMA AND PHI-1.5
To enhance evaluation diversity, we extended experiments to two additional base LLMs, Gemma (Mesnard et al., 2024) and phi-1.5 1.5 (Li et al., 2023). We summarize the results in Table 3. These results demonstrate that AlphaEdit consistently outperforms MEMIT and RECT across key metrics on both the Counterfact and ZsRE datasets. Notably, on Gemma, AlphaEdit achieves the highest fluency (398.96) and consistency (32.91). reflecting its ability to maintain coherence and accuracy. Similarly, on phi- 1.5, AlphaEdit excels in efficacy (70.79) and fluency (399.47), showcasing its adaptability to smaller, efficient models. These findings confirm AlphaEdit's 'ss robustness across diverse LLM architectures and its capability to deliver high-quality edits while preserving model integrity.
25

---

Published as a conference paper at ICLR 2025
Table 2:
Comparison of AlphaEdit with existing methods the sequential model editing task. Eff., Gen., Spe., Flu. and Consis, denote Efficacy, Generalization, Specificity. Fluency and Consistency, respectively. The best results are highlighted in bold, while the second-best results are underlined.
Counterfact
ZsRE
Method
Model
Eff.1
Gen.
Spe.t
Flu.
Consis.
Eff.
Gen.
Spe.1
Pre-edited
2.85
10.58aam
89.48
635.23
24. 14
36.991am
36.34
21.
SERAC
LLaMA3
71.21 su.s
61.05
66.90se2i
615.72auM
20.77
67.75
33.96
22.17
GRACE
96.72+a.1
40.14
72.23
620,43-ausa
23.79am2s
93.58
1.03=m06
31.86=wa2
MELO
65. 13
58.58+032
63.36s0.7
608. 208.9
22:18:004
25. 15..18
24:14
30.36
AlphaEdit
98.90
94.22
622.49 67.8% a7.8
32.40=n11
94.47=
91.13
32.55=623
Pre-edited
16.22
18.56a045
83. 1 2
621.8 2007
29.74 21
26.32
25.79
27.42
SERAC
82.28
58.31
68.98
615.92:wx
28.65
92.37
38.21
25.17
GRACE
GPT3
96. 26
50.10sam
74.42
620.56
31.55=015
96. 06.54
0.40=na2
24.78=aa:
MELO
78. 29.
60.52+0.32
86.80
610.82
24.31 =0:44
82.24
32. 88 =003
26.65=006
AlphaEdit
99.75
96.38=
75.48=aa
618. 17
99.79
96.00=aa2
28.29
Pre-edited
22
24.34.20
78.53
626.64mmm1
31.88
22. 19
31.30
24.15
SERAC
GPT2-XL
58.18 s2.25
64.06sa.7
595.35
27.35
92.17
36.57
20.67
GRACE
98. 28
50.05
72.07:934
620.21
28.53
94.33
1.59=00
27.63=440
MELO
72.62+0.s
53.63e0u2
63.25eau2
588 =n555
23.58=03s
93.54
45.25
23.
23.45=004
AlphaEdit
99.50
93.95=4.44
66.39
597.88220.
39.38
94.81
86.11
25.88=02:
Table 3: Comparison of AlphaEdit with existing methods the sequential model editing task. Eff. Gen., Spe., Flu. and Consis. denote Efficacy. Generalization, Specificity, Fluency and Consistency, respectively. The best results are highlighted in bold, while second-best results are underlined.
Counterfact
ZsRE
Method
Model
Eff.t
Gen.t
Spe.t
Flu.t
Consis.t
Eff.
Gen.7
Spe.t
MEMIT
Gemma
64.68
60.36
373.94m1m _6m73_oor
22. 10
64.38
24.52 62.12
RECT
65.17 17
57. 37.48
52.54=054
388, =044
23.37
67. 888.0
64.12=047
20.02
AlphaEdit
75.21
52.63
398.96:
32.91
75.91
68.12
23.50
MEMIT
15.71
56,58
35.41=00
368.57
19.79aa31
54.41
=2.47=0m
20.9
phi-1.5
RECT
58. 19
58.92
38.46=to2
362.94=144
19.88
55.15
53.64=0aa
18.58
AlphaEdit
70.79
65.12
48.96-434
399.47=au
25.98:aa4
70.02
63.19
20.69
C.7 EVALUATION ON EXPANDING BENCHMARK: KNOWEDIT, LEME AND MQUAKE
In this part, we selected two datasets from the KnowEdit (Zhang et al., 2024d) benchmark, namely wiki recent and wikibio, for testing. During our experiments, we noticed that some samples within these datasets exhibit abnormally high norms in the hidden states when processed by the LLaMA3- 8B-instruct model. These elevated norms are often accompanied by disproportionately large target value norms, which, if forcibly edited, could compromise the model's stability. To mitigate this issue, we implemented an automatic filtering mechanism to exclude samples with excessively high hidden state norms during the continuous editing process. Experimental results are presented in Table 4.
Additionally, we extended our evaluations to two critical datasets, LEME (Rosati et al., 2024) and MQUAKE (Zhong et al., 2023), to test AlphaEdit's performance in more challenging scenarios. LEME focuses on long-form generative tasks, emphasizing consistency, factual correctness, and lexical cohesion. MQUAKE, on the other hand, evaluates the ripple effects of factual updates using multi-hop reasoning questions. Results for these two datasets are summarized in Table 5.
According to Table 4 and we can find that:
AlphaEdit demonstrates a remarkable ability to achieve high editing success rates across all evaluated datasets. For instance, on the
dataset, AlphaFddit achieves impressive
26

---

Published as a conference paper at ICLR 2025
Table 4: Performance comparison of alphaFdit wiki_ Recent and wikibio datasets across key metrics including Edit Success, Portability, Locality, and Fluency. Results are averaged with standard deviations. The best results are highlighted in bold.
wiki recent
wikibio
Method
Edit Succ.
Portability!
Locality
Fluencyt
Edit Succ.
Locality
Fluencyt
MEMIT RECT AlphaEdit
56.25
42.73
41.02
513.35
63.73
64.27
582.38
=2.47
=1.28=02s
48.84
568.62#3m
91.48
+2.83
612.04:429
96.10
57.30
54.76
594.52:391
95.34.9.46
75.34
618.35:421
Table Performance AlphaEdit on MQuAKE and LEME datasets for multi-Hop reasoning and long-form editing. Metrics include Multi-hop Reasoning, Chain-of- Thought (CoT) Multi-hop Reasoning, Edit Consistency, Factual Consistency, and Internal Consistency. The best results are highlighted in bold.
MQuAKE
LEME
Model
Method
Multi-hopt
Multi-hop(CoTT)
Editt
Factual
Internalt
MEMIT
3.35
6.13
2.11 10.18
2.02.1..17
3.84
GPT-J
RECT
3.77
7.62:
2.24
2.62.10.19
4.07
AlphaEdit
5.03
9.14:1021
1.24
3.80
1.42
MEMIT 3.14
6,25
1.92.10.22
2.31110.0
3,85
GPT2-XL
RECT
3.72=0.06
7.48
2.12
2.60+0221
4.13#0029
AlphaEdii
5.00
9.25-10.27
3.28
3.07
5.76:10.49
96.1 10% editing success, which is significantly higher than the second-best method, RECT (82.47%).
A similar trend is observed on wikibio, where AlphaEdit reaches 95.34%, outperforming RECT by a substantial margin.
A AphaEEii demonstrates the best performance in both Multi-hop and Multi-hop (CoT) tasks, with scores of .414 and 9.75 respectively, significantly surpassing competing methods. This showcases its strong capability in handling complex reasoning tasks and ensuring logical consistency across interdependent facts.
On LEME, AlphaEdit excels all three metries, showcasing its ability to generate accurate long-form outputs. Its consistently high performance across GPT-J and GPT2-XL reinforces its reliability in executing precise edits while maintaining the structural coherence and integrity of the generated text.
C.8 IMPACT OF DATASET SIZE ON ALPHAEDIT'S PERFORMANCE
To explore the relationship between dataset size for calculating Ko and AlphaEdit's performance, we conduct additional experiments to evaluate the robustness of the model under reduced dataset conditions. Specifically,
progressively reduce the size of the dataset used to compute Ko to proportions [0.9, 0.8, 0.7, ... ...,. 0.1] of its original size. The goal is to analyze the impact of dataset size three key metries: Efficacy, Generalization, and Specificity. All the results summarized in Table To further illustrate trend changes, we select the results for LLaMA3 and visualized them using line charts, presented in Figure 11. According to Figure 11 we can find that:
As the dataset size decreased, both Efficacy and Generalization demonstrate notable stability, Even only 10% of the original dataset size, the drop in these metries is negligible (less than 5%), suggesting that AlphaEdit effectively generalizes to unseen data and remains efficient even with reduced data availability.
In contrast, the Specificity metric experience significant decline as the dataset size is reduced. When the dataset size is limited to just 10% of its original volume, Specificity drop by 11.76% indicating that the model's 's ability to store neighborhood knowledge heavily relies on the availability sufficiently large dataset.
27

---

Published as a conference paper at ICLR 2025
Table 6: Performance of AlphaEdit across various ratio of dataset size the sequential model editing
task. Eff.. Gen. and Spe. denote Efficacy, Generalization and Specificity, respectively.
Counterfact
ZsRE
Model
Ratio
F
Gen.t
Spe.t
Fff.
Gen.1
Spe.1
1.0
98. 1.1
04.22+00.9
67.88+1.34
94.47+00.7
91.13+1.02
1.755
0.9
98.92.10.9
93.87.11.66
66.23
94.12.114
91.7611.02
31.89110 1.23
0.8
96.75.1.35
92.45 10.78
66.45
94.12.11.33
90.95
30.67 1: 1.09
LLaMA3
0.7
75.66
93. 0..98
64.89+141
93.87#1.11
90.45=0097
29.34=0.84
0.6
96. 02
1..34
6..34
93.87+1.36
91.12#1.11
2.224
0.5
97.93.1123
94.01
63.51
92.96:10.89
91.67
1..5
0,4
95.88
92.67+1.11
61.78+1.09
92.98
91.76+1.28
28.56
0.3
96.98:1167
93.22:+0.99
58.56+1100
93.12+144
89.12+1.23
27.56
0.2
97.45.1097
91.8941.22
56.8940.89
93.01
89.99
26.34
0.1
9..21
90.12:1.5
56.12.
92.01
89.97
25.89
1.0
99.50.10.98
93.95.11.13
66.39
94.81
86.11
25.88 1. 1.42
0.9
97. 97.82++1.4
92.78+0.87
65.24
93.67
85.73+1.12
2..05
0.8
98.47
92.54
64.89+0.76
93.21
85,48
23.98
GPT2-XL
0.7
96.23
93.21
64.45
94.05-11.09
85.74
23.67
0.6
99.12 09.
91.33++1007
64.45
94.31
84.85+145
23.45
0.5
95.68-10.92
90.89++..4
61.76:tu.76
94.74-10.87
85,92+1.23
22.78
0.4
17.54
91.76++1.23
1..23+1.09
93.52.t0.88
84.45
22.34
0.3
99.01 1.00
90.32+1.11
58.92+n.92
93.01
83.78
22.67
0.2
95.89
91.03++103
14
93.04+1.09
85.07
22.45=1.15
0.1
96.35=1102
91.03+1.32
18.14
93.04+1.22
85.07+1143
21.11
1.0
99.75
96.38+1145
1..4
99.79+1.28
96.00+1.67
28.29
0.9
99.43:11.09
96.1411.08
74.75
9.063
96.11 10.99
27.12
0.8
98.34
96.03+1.112
75.21
97.65
96.01
25.89 1..89
0.7
98.21+1.23
95.11
73.58+1001
98.78
95.34
25.09
GPT-J
0.6
98.9441.17
95.33+1.12
73.89
99.05
95.45
24.87
0.5
18.46-11.21
94.89+1.09
1.08
98.9441.04
95.12+1.28
23.78
0.4
97. 97.74+00.88
94.76+1104
19.89+1.18
98.31+1.23
94.91+1.09
23.67
0.3
96. 1.334
94.34+0.95
67.95+0.84
97.58+00.8
94.23+1.15
22.78=1.12
0.2
96. 11.94
94.73
68.04.0.99
97.34
94.1211.02
23.45 1.09
0.1
97.02.10.89
94.73.1098
68.04
97.58
94.23
23.67 1.32
(a) Counterfact
(b) ZsRE
100
100
90
90
80
Generalization
Efficacy
70
D Effieacy
Generalization
Specificity
60
O Specificity
70
50-
40
60
30
50
20
0.9
0.7
0.5
0.2
0.8
0.7
0.4
0.1
Ratio of dataset size
Ratio of dataset size
Figure 11: Performance of AlphaEdit across various ratio of dataset size the sequential model editing task. Best viewed in color.
C.9 RUNTIME EVALUATION OF ALPHAEDIT
The computational complexity of the null-space projection in AlphaEdit depends solely on the dimension of the hidden dimension dg within the base I.I.M. Specifically, calculating the null-space
28

---

Published as a conference paper at ICLR 2025
Table 7: Times per batch (100 edits) for MEMIT and AlphaFdiit evaluated Counterfact and 7sRE dataset across various base LLMs.
Counterfact
ZsRE
Method
LLaMA3
GPT-J
GPT2-XL
LLaMA3
GPT-J
GPT2-XI
MEMIT
222.51s
334.74s
474.14, 14s
231.32s 231.40s
344.21s 345.52s
488.37s
AlphaEdit
223.24s
334.93s
476.79s
490.25s
projection matrix only requires operations Kg R.doxdo, which are independent of the number of layers, model size, or the knowledge base size.
To empirically validate the scalability 100 edits with AlphaEdit and MEMIT on three LLMs with different model sizes and knowledge bases: LLaMA3, GPT-J, and GPT2-XL. The results are summarized Table
method, we measured the average runtime for performing
From these results, it is evident that AlphaEdit does not incur additional runtime overhead compared MEMIT, even as the model size or knowledge base grows. This supports our claim that the null-space projection method is highly scalable and practical for large-scale model editing tasks.
D VISUALIZING THE COUNTERFACT AND ZSRE DATASETS THROUGH EXAMPLES
To help readers unfamiliar with model editing tasks better understand the Counterfact and ZSRE datasets, we provide two examples from them in Figure 12 and 13. These examples illustrate the types of modifications and factual updates applied to the models during the editing process.
29

---

Published as a conference paper at ICLR 2025
"case_id", 1469,
"pararel_ iixxx 6861,
"eeuuesed_rewwin rewrite":
"prompt": "0 located in country of",
"relation_id": "p17""
"target_new":(
"Poland"
"Qd":"Q36"
"target true": "str"""Mxiic"" "id":"Q9"
"subject": "Acolman Municipality"
"paraphrase__ap Bocttasee__poptss______
"Airlines and destinations)nNo scheduled flights operate at this airport. Acolman Municipality,, which is located in". "Literature \nNihon Gakkai s'u014dkai hyakkai kinenshi. Acolman Municipality, in"
"nighhorroood prompts": "Guadalajara is located in',
"Cancun located in the country of",
"Cludad Julu00e1rez 1rez, located In", "Guadalajara is located in the country of".
"Chichen Itza, in"
"1960 Olympics, which located in",
"Yucatac Maya, in",
"Spanish, in",
"Guadalajara, located in".
"Nuevo Le)u00I3n, located in"
"attribute_ aatribute_promppts"
"Narew, which located in', "Bukowice, Milicz County, located
"Warsaw Stock Exchange located in', "Taw\u0119cino located in the country of", "Cewice, located in". "Taw\u0119ccno, in". "Cewice is located in", "Narew, in",
"Warmian Masurian Voivodeship located in", "Narew, located in"
'generation_ |
"One can get to Acolman Municipality by navigating",
"Acolman Municipality's surroundings include".
"One can get to Acolman Municipality by navigating",
"Acolman Municipality's surroundings include",
restaurants around Acolman Municipality include".
"The best restaurants around Acolman Municipality include".
"Acolman Municipality's surroundings include"
"The best restaurants around Acolman Municipality include".
"Acolman Municipality's surroundings include".
"One can get to Acolman Municipality by navigating"
Figure 12: A Sample of the Counterfact dataset.
30

---

Published as a conference paper at ICLR 2025
"subject": "Una Maid Manhattan", "sre": "On what network did Una Maid en Manhattan first play?", "pred": "Telemundo",
"rephrase": "In which network Una Maid en Manhattan play first?", "alt": "Univision",
"answers":
"Telemundo"
1.
"loc": "nq question: what is the significance of the sahara desert", "loc_ ans": "the largest hot desert", "cond": "Telemundo \u003E Au00BB Univision || On what network did Una Maid en Manhattan first play?"
"subject": "Robert Rental", "src": "Which is the cause of death of Robert Rental?", "pred": "lung cancer", "rephrase": "What caused the death of Robert Rental?", "alt": "pneumonia",
"answers": [ "lung cancer"
"loc": "nq question: what beast beast from beauty and the beast", ioo__ans:: chimera", "cond":
"lung cancer 1u003E\u003E pneumonia || Which is the cause of death of Robert Rental?"
"subject": "Robert Rental",
"sre": "What was the cause of death of Robert Rental?", "pred": "suicide",
"rephrase": "What was the cause of the death of Robert Rental?", "alt": "accident",
"answers": [
"Jung cancer"
"loc": "nq question: what is the governor maharashtra",
"loc ans": "Chennamaneni Vidyasagar Rao", "cond": "suicide \u003ENu0033 accident
What was the cause of death Robert Rental?"
"subject": "Robert Rental", "sre": "What was the cause of death for Robert Rental?", "pred": "lung cancer", "rephrase": "What is the cause of Robert Rental's death?",
"alt": "murder",
"answers":
"lung cancer"
"loc": "nq question: how many episodes is ash vs evil dead season 3",
"loc_ans": "10", "cond": "lung cancer \u003E\u003I murder ||
What the cause of death for Robert Rental?"
Figure 13: Samples of the ZsRE dataset.
31


Reviews:
[{"review_id": "RehIYaxeUc", "reviewer": "Reviewer_u8aL", "summary": "This paper introduces AlphaEdit, a novel method for knowledge editing in large language models (LLMs). The primary goal of AlphaEdit is to enable targeted knowledge updates while minimizing the disruption of existing knowledge. The authors propose projecting perturbations onto the null space of the preserved knowledge before applying them to the model parameters. This approach theoretically ensures that the output of the edited LLM remains unchanged when queried about the preserved knowledge, thereby mitigating the issue of knowledge disruption. Extensive experiments on various LLMs, including LLaMA3, GPT2-XL, and GPT-J, demonstrate that AlphaEdit significantly boosts the performance of existing model editing methods by an average of 36.4% with minimal additional code.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The concept of projecting perturbations onto the null space of preserved knowledge is innovative and addresses a significant challenge in the field of knowledge editing for LLMs. The theoretical foundation provided in the paper is robust and well-explained.\n- The authors conduct extensive experiments on multiple representative LLMs, demonstrating the effectiveness of AlphaEdit. The performance improvements are substantial and consistent across different models.", "weaknesses": "1. Well, actually I think the work is great and I donot see any weakness, the thing is that I think the author can do more benchmarks like the LongformEvaluation, MQUAKE which consider some more knowledge utilization ablity for knowledge editing. \nBut the current evaluation is good enough.\n\n[1] Long-form evaluation of model editing\n\n[2] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eG7iGe5Y9f", "reviewer": "Reviewer_ngVy", "summary": "This is a review of the paper entitled “AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models” submitted to ICLR 2025. The paper suggests a new approach to do targeted knowledge updates in LLMs; in particular, if an LLM tells some wrong factual information, the goal is to identify influential parameters and then introduce so-called perturbation to them that, on the one hand, repairs problematic outputs and, on the other, keeps the rest as intact as possible. The main experimental result is that the new suggested method, called AlphaEdit, performs comparably to the state of the art for single updates and shorts sequences of updates, but outperforms them dramatically for longer ones.", "questions": "Concrete comments:\n\nQuestion: why do we need to solve the sequential editing task really sequentially, as in L (line) 244—that is, why cannot we just start from scratch with the original model and K1 being all the edits together (i.e., the union of Kp and K1 in the current equation (12)), and use Equation (11)? \nThis seems to promise a better performance even for the AlphaEdit, looking at Figure 4.\n\nMinor:\nL 11: I do not think “due” is the right word here. \nL 76: “the coefficient” is unclear. Which coefficient?\nL 81: besides viewing in colour, one needs a magnifier to read these figures.\nL 136: “update” and “new” do not make much sense together, either one or another.\nL 173: B -> A\nL 177 (equation (7)): \\Delta is not really \\Delta here, it is projected \\Delta. Which should be said or better denoted by another symbol.\nL 193: it is not clear what does it mean to be “consistent” in this context", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I should start by admitting that I am not a specialist in the topic of the paper, and so it is difficult for me to judge the novelty and value of the results. However, I can say that the paper is well-written: I could understand nearly everything and agree with the arguments. Moreover, for an outsider, the results look interesting and promising. Thus, I lean towards acceptance; however, of course, the opinions of reviewers who are more in the topic should be more valuable for the decision.", "weaknesses": "—", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2v7AXk5JO", "reviewer": "Reviewer_JPxS", "summary": "The paper introduces AlphaEdit, a method to improve targeted knowledge editing in large language models by projecting updates onto the null space of preserved knowledge, thus reducing interference with existing information. AlphaEdit achieves this with a minimal adjustment in code, enabling it to maintain a model’s pre-existing knowledge while updating targeted information. Experimental results demonstrate AlphaEdit's effectiveness, showing a performance improvement over traditional editing methods across multiple language models.", "questions": "1. Authors may have overlooked these methods. There are other latest methods present such as SERAC, GRACE, InstructEdit, MELO methods. Authors either provide the compared results or argue on why they have not considered these methods for comparison.\nhttps://sites.google.com/view/serac-editing\nhttps://arxiv.org/abs/2211.11031\nhttps://arxiv.org/abs/2402.16123\nhttps://arxiv.org/abs/2312.11795\n\n2. Can authors show the results on  KnowEdit dataset as well?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of null-space projection in AlphaEdit minimizes disruption to preserved knowledge while updating new information, effectively addressing a common trade-off in model editing between knowledge update and retention.\n2. The paper provides comprehensive experimental evidence that AlphaEdit outperforms existing methods on critical editing metrics such as efficacy, generalization, specificity, fluency, and consistency.", "weaknesses": "1. Accurate null-space projection may rely on high-dimensional matrix computations, which could pose scalability issues as model sizes or knowledge bases grow.\n2. Limited empirical evaluation on diverse LLMs. The method is tested on models like GPT-2 XL, GPT-J, and LLaMA3 only. It would be good to see results for other models such as gemma, phi.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sr8upYu8iQ", "reviewer": "Reviewer_cdNJ", "summary": "Post training an LLM can often cause it to disrupt its originally preserved knowledge. To circumvent this, AlphaEdit utilizes Null Space Projection to preserve old knowledge while injecting the new knowledge effectively. Results show that AlphaEdit significantly reduces the domain shift between pre and post edits compared to existing methods.", "questions": "1. In line 174, ‘B is in the null space of B’ -> change to ‘B is in the null space of A’\n2. For figure 7(a), what does the ylabel refer to?\n3. Minor presentation suggestion: In figure 5, the pre and post edited distributions are difficult to set apart due to color choice. Picking two contrasting colors similar to Memit would be great for the final version.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to follow. The authors explained the null space and how to leverage null space projection to optimize the model editing objective well.\n2. I think the choice of RQs is well thought out and thorough as well. The paper answered most of the questions I had about AlphaEdit.\n3. I think figure 6 is interesting to show how AlphaEdit can generalize to existing methods.", "weaknesses": "1. The paper did not mention the correlation between the accuracy and the dataset size. More concretely, how much data is needed for AlphaEdit to work well?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)