Towards Lifelong Model Editing via Simulating Ideal Editor
Authors: Yaming Guo
Siyang Guo
Hengshu Zhu
Ying Sun
Venue: ICML 2025 poster (2025)
Keywords: Lifelong Model Editing
Linear System
Large Language Model
Primary area: deep_learning->large_language_models
TL;DR: We propose a general framework that restores the strong performance of standard model editing methods in a lifelong context, bridging the gap between these two paradigms for the first time.

Abstract: Model editing plays a crucial role in the cost-effective development of large language models, and the challenge of evolving knowledge facilitates its sequential extension, namely lifelong model editing. However, progress on standard and lifelong editing has historically followed separate tracks, overlooking the potential of generalizing standard methods to lifelong scenarios. By establishing this bridge, we can provide robust baselines in lifelong scenarios and ensure that lifelong editing benefits from the ongoing advancements in standard editing technologies. In response, this paper proposes a general framework, ***Sim**ulating **I**deal **E**ditor* (SimIE), which restores the strong performance of parameter-modifying methods from standard model editing in a lifelong context. SimIE formulates the ideal parameter shift as the minimum-norm solution to a linear system, constructed using the Moore-Penrose inverse, and subsequently enables recursive updates by truncating the limiting expression of the Moore-Penrose inverse under two mild assumptions. Theoretically, we demonstrate that if either assumption is not met, the solution provided by SimIE remains near-optimal in a statistical sense or stable against perturbations introduced by the sequential editing, but a trade-off between optimality and stability arises when both assumptions fail. Extensive experiments validate the effectiveness of SimIE, which allows standard algorithms to achieve performance comparable to specialized lifelong model editing methods. Our code is available at https://github.com/YamingGuo98/SimIE.

Towards Lifelong Model Editing via Simulating Ideal Editor
Yaming Guo Siyang Guo 2 Hengshu Zhu
Ying Sun
Abstract
Standard methods
Model editing plays a crucial role in the cost- effective development of large language models, and the challenge of evolving knowledge facil- itates its sequential extension, namely lifelong model editing. However, progress on standard and lifelong editing has historically followed sep- arate tracks, overlooking the potential of gener- alizing standard methods to lifelong scenarios. By establishing this bridge, we can provide robust baselines in lifelong scenarios and ensure that life- long editing benefits from the ongoing advance- ments in standard editing technologies. In sponse, this paper proposes general framework, Simulating Ideal Editor (S iimEE), which restores the strong performance of parameter- parameter-modifyingg methods from standard model editing in a life- long context. SimIE formulates the ideal param- eter shift the minimum- -oorm solution a lin- ear system, constructed using the Moore-Penross inverse, and subsequently enables recursive up- dates by truncating the limiting expression of the Moore-Penrose inverse under two mild assump- tions. Theoretically, we demonstrate that if either assumption is not met, the solution provided by SimIE remains near-optimal in a statistical sense or stable against perturbations introduced by the sequential editing, but a trade-off between opti- mality and stability arises when both assumptions fail. Extensive experiments validate the effective- ness of SimIE, which allows standard algorithms to achieve performance comparable to specialized lifelong model editing methods. Our code is avail- able SimIE.
fo
ai Altulleaa)) fo. aur, Alhiltwwa)) fo.
for
Ideal editor
fa
AW Aleal Editer
(Also, (Re, Bell
forra
2
SimIE
Robust parametior spulaies Equation (5)
Adhor paramssier apslnsoe uorry Equantoe
fa
fe 20 Affemmall
fa
for
Figure I. Illustration of SimIE, which enables the post -dit model to closely approximate the ideal state achieved by the idcal editor.
1. Introduction
Large language models (LLMs) acquire extensive knowl- edge across various domains during pre-training, which may include outdated or harmful information (Petroni et al., 2019; Zhao et al., 2023). Given the prohibitively high costs of re-training and fine-tuning, model editing has emerged as efficient approach for updating and correcting specific knowledge within LLMs (Sinitsin et al., 2020; De Cao et al., 2021). The goal of model editing is to adjust the LLM's predictions for specific inputs to match the desired outputs, based on the provided edit examples, while preserving its behavior on unrelated inputs. Existing model editing tech- niques generally fall into two main categories: parameter- modifying and parameter-preserving (Yao al., 2023). This work focuses parameter-modifyin approaches.
To accommodate evolving knowledge, the extension of model
model editing-is proposed, which involves performing sequential edits (Yao et al., 2023). Lifelong editing presents two inherent challenges: 1) The sequential updates gradually deviate the model from its initial state, leading degradation in its general abilities and editability (Ma et al., 2024; Gupta et al., 2024b). 2) The model typically encounters catastrophic forgetting dur- ing sequential edits, where previously edited examples are forgotten (Wang et al., 2024a; Gupta et al., 2024a). Conse- quently, preliminary efforts have been made to tackle this more challenging scenario (Huang al., 2023; Hartvigsen
Artiiiciil Intelligence Thrust, The Hong Kong University of Science and Technology (Guangzhouu), Guangzhou, China "School of Intelligent Systems Engineering, Sun Yat-sen University, Shen- zhen, China 'Computer Network Information Center, Chinese Academy of Sciences, Beijing, China Uniiversity of Chinese Academy Sciences, Beijing, China Correspondence to: Ying Sun <yings @ hkust-ga ez..d..c'>.
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Simulating Ideal Editor
al., 2024; Wang al., 2024b). One prevalent perspective among these works is that standard methods are unsuitable for lifelong editing scenarios, they suffer significant per- formance degradation as the sequence of edits lengthens.
The results demonstrate that SimIE allows standard algorithms to achieve performance comparable spe- cialized lifelong methods, requiring just one line of code for implementation (Section 5).
Contrary to popular belief, we study the feasibility of gen- eral solution for generalizing standard methods to lifelong scenarios, thereby bridging these two paradigms. This con- nection offers dual advantages. First, the effectiveness of standard methods has been widely validated, allowing us to readily provide robust baselines in lifelong contexts without developing specialized approaches from scratch. More im- portantly, this integration alleviates the challenges specific to lifelong editing, redirecting the community's 's focus to enhancing fundamental editors, whose advances naturally extend to lifelong scenarios. As an elementary exploration of this promising direction, the core question to ask is:
2. Preliminaries
Standard model editing. Let X denote the input space and y' the output space. We consider the model fo : X + y parameterized by 0. Given an edit example (Iee Be), the goal of model editing is to adjust the model's prediction at Te to match Ye. while preserving its behavior on unrelated in- puts. Formally, for an edit example (Ze: be) and the original model fo(-), the post-edit model for() should satisfy:
Ver [o((x)
for (xx
(I)
otherwise.
Is it possible to restore the strong performance of standard methods in the context of lifelong model editing?
Although Equation (1) considers only single edit exam- ple, recent works have scaled to handle thousands of exam- ples, achieving commendable performance (Mitchell et al., 2022b; Meng et al., 2023; Tan et al., 2024).
In response, this paper proposes a general framework called Simulating Ideal Editor (SimIE). Given a standard model editing algorithm A, we introduce its ideal editor, de- fined the optimal solution that edits the initial model using all examples from the sequential stream simultane- ously. Mathematically, the ideal editor is formalized as the minimum-norm solution to a linear system, which is con- structed using the Moore-Penrose inverse. Under two mild assumptions, namely wwer-aarameterization and key-value invariance. SimIE enables recursive updates by truncat- ing the limiting expression of the Moore-Penrose inverse. As illustrated in Figure 1, SimIE only adjusts the param- eter updates obtained from A at each time step, enabling the post-edit model to closely approximate the ideal state achieved by the ideal editor. Our main contributions are summarized as follows:
Lifelong model editing. To accommodate evolving knowledge, lifelong model editing has been proposed, which involves performing sequential edits on LLMs. Let Dedit = {(1,, y1) E [T]} denote the dataset of all edit examples, and Xodit {rra | E [T]} represent the set of corresponding inputs, where T ;= {1, ........2. T.. We denote the initial model by fo, (.). At time step I, the model editing algorithm A generates a parameter update based on the pre- vious model fo._, (.) and the current edit example (x),))), resulting in the following parameter update rule:
θ. 0 + A(for (I2,, Bc)).
The final post-edit model, før (-), should satisfy (21) = Ve (vt E [TT) and fo, (1) = for (z) (vx & Kedia). As mentioned earlier, due to the inherent challenges of lifelong editing, naively adapting standard approaches to lifelong sce- narios can lead to significant performance degradation (Yao et al., 2023). In Section 3, we introduce a general frame- work designed to restore the strong performance of standard methods in a lifelong context.
1. We propose SimIE, a general framework designed to restore the strong performance of standard model edit- ing methods (applicable to all parameter- modiiiiing algorithms) in a lifelong context, bridging the gap be- tween the two paradigms for the first time (Section 3).
2. We theoretically analyze the behavior of the proposed framework when assumptions are not met, revealing that: 1) When the rrerrparametrrization assumption is not satisfied, the solution remains near-optimal in a statistical sense. 2) When the key-value invariance assumption does not hold, the solution is stable against perturbations introduced by sequential editing. 3) If both assumptions fail, the solution requires trade-off between optimality and stability (Section 4).
Editing in key-value memory. Through causal tracing, Meng et al. (2022) identify that the MLP modules in the Transformer (Vaswani, 2017), comprising two linear layers with a non-linear activation, are critical for storing factual knowledge. Building on this insight, most works have fo- cused on editing the parameters of MLPs, demonstrating that such edits are consistently effective for updating knowl- edge within LLMs (Meng et al., 2023; Tan et al., 2024). These studies model the linear layer of MLPs, denoted by W E XRd, as a key- memory that maps an input
3. We validate the effectiveness of SimIE on multiple LLMs, such as GPT2-XL (Radford et al., 2019), Llama- (Touvron et al., 2023), and Mistral (Chaplot, 2023).
2

---

Simulating Ideal Editor
an output e Wk; storing knowledge in the form of key- value pairs (ki, v). Supposing that the edit example (xe, Me) corresponds to the key ke at W, the editing method gener- ates a parameter update AW, E Rdxx, thereby inducing a change in value:
The solution S defined by the ideal editor exhibits two fundamental properties. First, when applied to the edit examples (Iz, Vr), S produces an equivalent transforma- tion with the individual updates AWP" Second, S achieves a minimal Frobenius norm among all solutions that sat- isfy the constraints. By defining the concatenated matrices K := [ki h2
Whe - Ce (W + Ai,
kT]] E Rdd xI and B := [b) | b2 I br] E R we can characterize the ideal editor as the minimum-norm solution to the linear system SK B.
where represents the initial value and ve is the desired value. Utilizing the updated key-value pair (ke, Ve), the model editing method efficiently incorporates new knowl- edge (Ze, ye) into LLMs.
Thus far, we have not established the existence of the ideal editor. To derive this fundamental result, we require an over-parame over-parameetrrizatonn assumption, stated as follows:
3. Simulating Ideal Editor in Lifelong Context
Assumption 3.2 (Over-parameterizationn The number of edit examples does not exceed the column dimension of the weight matrix W, i.e., T S d1, and the key matrix K is full rank, i... rank(K) = T.
In Section 3.1, we define the ideal editor and introduce two mild assumptions: over per-parameterization and key-value invariance. With these assumptions, Section 3.2 presents SimIE, which approximates the ideal editor through re- cursive updates. Section 3.3 extends SimIE to multi-layer editing settings, accompanied by a cost analysis.
This assumption ensures that W has sufficient capacity and that the key vectors {Kr]== are linearly independent. Un- der Assumption 3.2, we can prove that the linear system SK = B admits at least one solution, thus guaranteeing the existence of the ideal editor (formalized in Appendix B.
3.1. Introduced Ideal editor
Standard model editing methods can inject knowledge from numerous edit examples into LLMs, yet they suffer sig- nificant performance degradation in lifelong contexts. We argue that robust performance in standard scenarios is based on two key principles: 1) The model remains in its initial state, which avoids the degradation in editability caused by sequential editing (Gupta et al., 2024b). 2) All edit ex- amples are available concurrently, mitigating catastrophic forgetting during sequential updates (Wang et al., 2024a). Therefore, given a standard model editing method, an ideal approach would involve editing the initial model by utilizing all examples in the sequential stream simultaneously.
To enable the approximation of the ideal editor in lifelong scenarios, we introduce the veyuv invariance assump- tion, described as follows:
Assumption 3.3 (Key-value invariant). Given an algo- rithm A, the updated key-value pair (ke, ve) w.r.t. the edit example (Zz, V/.) remains invariant across different model states, i.c., (kl, 2/) = (k,', v!"), where (K/, ri) and (k",, v!") are derived from the models for and for, respectively.
Intuitively, this assumption implies that the representation of specific knowledge within the LLM remains invariant. Under Assumption 3.3, it is easy verify that for any model state for, the following relationship holds:
To maintain clarity, we first focus on editing a single linear layer, initialized as Wo E dd,,, which is a component of the full model parameters 0g. Using the standard model editing algorithm A, the ideal editor aims to modify the parameter Wo in order to simultaneously inject knowledge of all examples within Dedit defined
(W' + +A (F., 3+)) Wo) K2 = bt AW.k. (3)
Equation (3) indicates that the change in the updated value relative to the initial value, Wok,, is consistently captured by b, regardless of the model state.
Definition 3.1 (Ideal Editor). For an edit example 9c) E Dean- let k, denote the key, and define the change in value as b, = AWPk, where AWP- represents the parameter update for Wo generated by algo- rithm A. The ideal editor w.r.t. A is the optimal parameter shift S that solves the following optimization problem:
It is worth noting that Assumption 3.2 and Assumption 3.3 may not hold in certain complex editing settings. Neverthe- less, we will provide a detailed theoretical analysis of the case where assumptions are not met (Section 4) and find that they do not lead to empirical failure (Section 5).
A(fe, (#t, v1))
min || s.t. Sk, = be. t
3.2. The proposed SimIE
.
The ideal editor provides the optimal parameter shift for in- jecting knowledge from Dedit into the model fe,- Therefore, a natural approach is to approximate the solution defined by the ideal editor in the lifelong model editing scenarios.
For notational simplicity, we assume each example consists of single token mapped to one key, though our analysis naturally generalizes to cases with multi-token examples.
3

---

Simulating Ideal Editor
To begin, we introduce the Moore-Penrose inverse (Penrose, 1955), which generalizes the concept of the matrix inverse. It is defined as follows:
Building on the above insight, we propose Simulating Ideal Editor (SimIE), which generates the parameter shift S as an approximation of the ideal editor 8 through recur- sive update procedure. Starting with the initial parameter Wo E IRd2xdi and Po = AT E IR:, [gd;xd;, where À > 0 is a hyperparameter, the update rule for SimIE is given by:
Definition 3,4 (Moore-Penrose inverse). Given a matrix A E Rd2xd: with Singular Value Decomposition (SVD) A = UEVT, the Moore-Penrose inverse of A, denoted by A', can be written as:
A V
Pt Pr- lagk
(5)
where Σt E Rd is obtained by taking the reciprocal of each non-zero singular value in the diagonal matrix E and transposing the resulting matrix.
W, = W AW,k,
SimIE uses additional memory to maintain matrix P.. which is employed to adjust the parameter update AW; := A(fo, (Iaa, Ut)) generated by the standard model editing algorithm A at time step t. The following theorem esti- mates a bound on the difference between the parameter shift generated by S SimIE and that of the ideal editor:
Under Assumption 3.2, the ideal editor exists and is unique, which can be expressed as 5 = BK = B(K K)- K (see Appendix B.1). However, exact computation of so is not feasible unless all key-value pairs {(k,, " are available simultaneously. Instead, we can give an approxi- mate solution through the following lemma:
Theorem 3.6. Let S = WT - Wo denote the parameter shift generated by SimIE according to Equation (5), and let S represent the ideal editor w.r.1. Equation (2). Under Assumptions 3.2 and 3.3, the following bound holds:
Lemma 3,5 (Theorem 4.4. in Laub (2004)). For a matrix A E daxd,, we have:
Al lim AAAA + ,
a-+
N 0 max +
|S - SS |S
A
where T E Rdd xds is the identity matrix.
6 min
+
This lemma provides a limiting expression for K1, even in cases where (KKT)-1 does not exist. We propose choos-
where 0 < J S m are the smallest and largest singu- lar values of the key matrix K, respectively.
ing a hyperparameter À to truncate the limit, resulting in the approximation 5 ~ SZ == BK (KKT + A7)-
This theorem is proved in Appendix B.3. As demonstrated in Theorem 3.6, the approximation error bound for St. S is governed by the largest and smallest singular values of the key matrix K as well as the hyperparameter X. In particular, as À approaches 0, S.A.A. is guaranteed to approximate S In other words, by adjusting the parameter updates obtained from the standard algorithm A at each step. SimIE ensures that the post-edit model closely approximates the ideal state achieved by the ideal editor, which edits the initial model using all examples simultaneously.
Let K., = [Kr k2 .... ke] and B, : [bi bz | ... br] denote the matrices formed by the first t columns of K and B, respectively. If S 1 (K KTT A holds at time step i, then, under Assumption 3.3, we can derive the following recurrence relation for S.::
S -B..11 + dek, (K,K. +
denotod as Pt b P
= -Ba-1 1-1
P P
3.3. Editing on multiple layers
(4) Existing standard model editing methods typically update parameters across multiple linear layers to enhance the ro- bustness of editing at scale. To extend SimIE, multi-layer editing, we apply it independently each layer. While this straightforward implementation ignores inter-layer depen- dencies, as similarly done by Tan et al. (2024), empirical evidence supports its effectiveness in practice.
S
Poi P. + byk P--
=Pi -K
Sh +
(S) AAW))a
= S + AW, P
The pseudo-code for SimIE is summarized in Algorithm 1. Let C denote the set of layer indices targeted for modifica- tion, as determined by the standard model editing algorithm A. For each linear layer i L, we initialize the matrix (()) = Xl E R xa(t) according to the dimensions of the
where AW, A(fe,-, (It, v)) denotes the parameter update generated by the standard model editing algorithm A at time step t. The detailed derivation of Equation (4) can be found in Appendix B.2. The Equation (4) reveals that SAAA can be computed recursively from S2_1, enabling SA to be obtained in lifelong context.
weight matrix (()) E R x where À >> is a hyper-
4

---

Simulating Ideal Editor
Algorithm 1 SimIE (red) and Naive approach (blue)
4. Beyond Assumptions
Input: algorithm A, dataset Dedit, initial model (-)
In this section, we theoretically analyze the behavior of SimIE when the assumptions are not met. Sections 4.1 and 4.2 examine the cases where Assumptions 3.2 and 3.3 do not hold, respectively. Section 4.3 considers the scenario where both assumptions fail. Since SimIb is applied inde- pendently to each layer, we focus on the analysis of single layer and omit the layer index without loss of generality.
Initialize PO") = >I E R.4 xxf
for L [T] do
{AW, r hEE + A(fo--1) vi))
(1)
Cache {xi(") IIE
p(l) P + Ni " (1:00T
w - -W(") + +AW
kt [&e [
With = -W(") +AW!
4.1. Optimality without Assumption 3.2
fo. + -(W,) (W]""ee JEE
Assumption 3.2 requires that the weight matrix has suffi- cient capacity and that the key vectors are linearly inde- pendent, but this may not always hold in practice. For instance, frequent knowledge updates can cause the number of edit examples to rapidly exceed the column dimension of the matrix. Moreover, unintended edits to relevant factual knowledge may result in key vectors becoming linearly de- pendent. In such cases, the ideal editor does not exist, as the linear system SK- = B no longer admits the exact solution.
end for
Output: post-edit model for
parameter. At time step t, the algorithm A processes the edit example (Ir, Bt) to edit the previous model fa._1 (+), ob- taining the parameter updates (Aw!") Jecc In parallel, we cache the corresponding keys (k,"0 Jec. Then, we calculate the matrix p(() and use it to adjust corresponding A AW(") according to Equation (5). Finally, we replace the param- eters of (.) with the adjusted parameters (W:() hec, yielding the model fa, (-) for time step t. As a comparison, we highlight the lines unique to our method SimIE in red, while using blue to indicate the naive approach of directly adapting standard algorithms to the lifelong scenarios.
Although the existence of an ideal editor is precluded, we will show that the solution St generated by S SimIE is still near-optimal in statistical sense. Here, optimal in a statisti- cal sense refers solution S that minimizes the squared residual norm, i.e., ||S K BI] = min ||SK - Blp. as in least squares problems. The following theorem provides an upper bound on the squared residual norm w.r.t. St.
Theorem 4.1. Let S = WT Wo represent the parameter shift generated by SimIE according to Equation (5). Under Assumption 3.3, the squared residual norm w.r.l. St satisfies the following inequality:
Remark 3.7. For the edit example (z,, contains multiple
tokens, we denote the keys as (x:(0 1 In such cases, we only replace occurrences of k. in Algorithm with the (8)
concatenated key matrix K.D) = 1 K.N.
(1 Rmin)) Rain S <|[SKK - BI S (o + +A2
+ Rainny
Cost analysis. Finally, we briefly analyze the additional cost introduced by SimIE. Assume that each time step involves N tokens and L = [C] linear layers are edited. SimIE requires maintaining L matrices p(") of dimension di dj, resulting in a direct storage cost of O(Ld)) Utiliz- ing low-rank structure p!() x!" T + the storage cost becomes (()..dd) The primary computational cost of SimIE arises from matrix inversion, implemented by kr solving the linear system X p!!)) AW,"), (00) (x["]. The standard approach for solving is LU decomposition, which has a time complexity of (2 Alternatively, Cholesky decomposition can be used, offering a reduced time com- plexity of O (ddf), since p(()) is symmetric positive def- inite matrix. Overall, the costs introduced by S SimIE are manageable and are nearly negligible compared to those of model editing algorithms. A more detailed discussion can be found in Appendix B.4.
where Rmin := min |SK B is the minimum value of squared residual norm, and or > 0 denotes the smallest non-zero singular value of K.
Theorem 4.1 demonstrates that the squared residual norm of S is bounded by the minimal squared residual norm Rmin plus term ||B - Rmin scaled by the hyperparam- eter X. Here, Runn represents the theoretical lower bound of the squared residual norm, reflecting the irreducible er- ror caused by the inherent limitations of the system. On the other hand, | Rmin quantifies the portion of the squared residual norm that can be effectively fitted by the model. As X - 0, the squared residual norm of S}. con- verges to Rmin- Thus, even in the absence of Assump- tion 3.2, the solution generated by SimIE remains near- optimal in a statistical sense, as it minimizes the squared residual norm of the given constraints to the extent possible.
5

---

Simulating Ideal Editor
4.2. Stability without Assumption 3.3
first term indicates that the effect of the perturbation 8B is limited by the condition number cond (St. B). The second term || and the third term vdi || || /o min logether govern the impact of the perturbation 8K. As À approaches 0, || | p increases monotonically until attaining an upper bound. The second inequality captures this behavior by of- fering a X-independent perturbation bound, where ||Jxll is substituted with the condition number Kp (x(K) w.r.t. K. In other words, when very small X 0 is chosen for optimality, perturbations oK and 8B always lead to man- ageable changes in the solution. In conclusion, even in the absence of Assumption 3.3, the solution generated by S SimIE remains stable, as is not excessively sensitive to small perturbations in the key-value pairs.
Assumption 3.3 assumes that the representations of specific knowledge remain stable, but this may not always hold in complex multi-layer editing algorithms. Specifically, pa- rameter updates in earlier layers can perturb the keys in subsequent layers. Moreover, as the edit-distributin strat- egy depends on the current model state (Gupta et al., 2024b), it induces perturbations in the values of intermediate layers. As result, multi-layer editing scenarios, Assumption 3.3 is unlikely to hold due to inevitable key-value perturbations.
We will demonstrate that the solution generated by SimIE has stability even without Assumption 3.3, meaning that per- turbations to key-value pairs do not excessively change the solution. Mathematically, perturbations to key-value pairs (ke, vz) are equivalent to perturbations in (ke, bz). Thus, our subsequent stability analysis focuses on the latter term. Since the impact of the perturbation to (kiz, by) is not ampli- fied over subsequent time steps (see Lemma C.2), we can analyze these perturbations using the matrix forms K and B, rather than examining their components individually.
4.3. Trade-off without Assumptions 3.2 and 3.3
While violating either Assumption 3.2 or Assumption 3.3 individually results in relatively positive outcomes, the sit- uation becomes more intricate when both assumptions are not met. In this case, trade-off between optimality and sta- bility must be considered. The following theorem presents perturbation bound for the solutions generated by SimIE that does not rely on any assumptions:
Following perturbation theory (Hansen, 1998; Malyshev, 2003), we define the relative condition number of S w.r.t. B, but replace the fo-norm with the Frobenius norm, as:
Theorem 4.3. Let S represent the original solution corre- sponding to K and B, and S denote the perturbed solution corresponding to K + 5K and B + ,B, both generated by SimIE according to Equation (5). Then, the perturbation bound for S is:
| 5
| |
cond(S7,B)) B) = lim sup e-0 |
Similarly, let Ky (F) : || Kt|
F || denote the condition number of the matrix K under the Frobenius norm. The following theorem establishes perturbation bound for the solutions generated by SimIE:
S
S |S
| F +
||81|
Scond (T)
B)
KF
(K)
|r
|||
Theorem 4.2. Let Sh represent the original solution corre- sponding to K and B, and Sh. denote the perturbed solution corresponding to K + 8K and B + SB, both generated by SimIE according to Equation (5). Under Assumption 3.2, the perturbation bound for St is:
|Blle
v
||5K||.
A
|
where RA = SAK B is the residual matrix w.r.t ST.
Theorem 4.3 indicates that the impact of the perturbation 8. is governed by the relative condition number cond (8), B). while the effect of the perturbation 5K is modulated by the condition number KF (K) and the ratio of the residual norm || to the hyperparameter A. Notably, as X de- creases, the term || IF tends to decrease (as established in Theorem 4.1), while 1/A increases monotonically. The combined effect of || introduces trade-off between the optimality and stability of the solution: smaller values of x can enhance the optimality, but at the cost of increased sensitivity to perturbations. Conversely, larger values of A improve stability but may result in a suboptimal. Therefore,
S
18B IF + ||
F (So B)
S 1|
|BI
Vdi ||
||5
0² a
||BB1 IE
||0 |F
(Sond( B)
+ KF(K)
|BI
1 |||
/di [[]l|K' ||||r
|
miin
where JA == (TKKK" + - denotes the Jacobian matrix of S} at B.
NNote that cond( (S), B) also relies on X, while it admits a complex upper bound. For conciseness, we omit detailed analysis here. is helpful to observe that lim,- cond (S), B) = or (K). Further discussions are provided Remark C.3.
The first inequality of Theorem 4.2 establishes a tight per- turbation bound that depends on the hyperparameter X. The
6

---

Simulating Ideal Editor
Llama-2 (7B) on the ZsRE dataset
1.0
1.0
Gen
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
10
50
100
300
500
750
1000
10
50
100
300
500 750 1000
10
50
100
300
500
750
1000
10
50
100
300
500 750
1000
Number of Edits
Number of Edits
Number of Edits
Number of Edits
MEND MEND+SimIB
ROME -
ROME-SinlE MEMIT -
MEMIT-SimlE AlphaEdit
AlphaEdit -SimlE
Figure 2. Performance of algorithms the number of edits increases, with the solid line representing the results combined with the proposed SimIE.
when both Assumptions 3.2 and 3.3 are not met, it is cru- cial to carefully tune À to achieve the best balance between optimality and stability in the solution generated by SimIE.
plementary measure. For more details on the experimental setup, please refer to Appendix D.I.
5.1. Effectiveness of SimIE
5. Experiments
To evaluate the effectiveness of the proposed S SimIE, we apply it to standard model editing algorithms and measure their performance before and after application. Specifically, we perform T 1000 sequential edits on L.LMs, with 1 example per edit. Table presents the results for Llama-2 and Mistral, while Table 4 displays the results for GPT2-XL. Additionally, Figures 2 and 6 illustrate the performance of standard methods as the number of edits increases. Based on these results, we can draw the following observations:
This section validates the effectiveness of SimIE in restor- ing the strong performance of standard methods in lifelong contexts. Further analyses. including case study, hidden representation visualization, and long sentence editing, are provided in Appendix D.3.
Experimental setup. We conduct experiments on three widely used LLMs: GPT2-XL (1.5B) (Radford et al., 2019), Llama-2 (7B) (Touvron et al., 2023), and Mis- tral (7B) (Chaplot, 2023). Our experiments include nine popular baselines: the basic fine-tuning method, FT- L (Meng et al., 2022), and four standard model edit- ing algorithms, namely MEND (Mitchell et al., 2022a), ROME (Meng et al., 2022), MEMIT (Meng et al., 2023), and AlphaEdit-3 (Fang et al., 2024), along with four lifelong model editing algorithms, specifically GRACE (Hartvigsen et al., 2024), WISE (Wang et al., 2024b), PRUNE (Ma et al., 2024), and AlphaEdit (Fang et al., 2024). These algorithms are evaluated using two widely adopted benchmarks, i.., the ZsRE dataset (Levy et al., 2017) and the Counterfact dataset (Meng et al., 2022). In line with prior research (Wang et al.. 2024b), we assess performance using three key metrics: Rel (Reliability, also known as Edit Success Rate (Hartvigsen et al., 2024)), Gen (Generalization Success Rate), and Loe (Localization Suc- cess Rate), We use the Arithmetic Mean Avg = Gol+LonnLLoe as the primary metric, and introduce the Locality-pentt -penaiie-ee Geometric Mean Geo = e caLoo--12 (Rel x Gen) as a com-
Standard algorithms exhibit significant perfor- mance degradation as the number of edits increases. While methods like ROME and AlphaFdit maintain stable performance early in editing, their effectiveness sharply decreases after surpassing 100 edits. The sit- uation is even worse for MEND, which fails in the initial stages, likely due to the hypernetwork being tailored to the initial model. By the end of the edit sequence (T = 1000), all standard algorithms show unsatisfactory performance across settings.
The proposed SimIE successfully restores the strong performance of standard algorithms life- long contexts. First, algorithms integrated with SimIE maintain comparable performance to their ini- tial editing phase throughout the sequence. Second, for editing Llama-2 on the ZsRE dataset, SimIE increases the Avg of MEMIT from 0.03 to 0.68, achieving parity with its batch-editing counterpart, MEMIT-MASS, as reported in Wang et al. (2024b). These results demon- strate that the performance gains from SimIE are both substantial and consistent.
3 "AlphaEdit includes two mechanisms: projecting updates into the null space and protecting previously edited knowledge. We refer to its version in the standard scenario, where the protection component is omitted, as AlphaEdit
The performance of standard algorithms aug- mented with SimIE rivals that of dedicated life-
7

---

Simulating Ideal Editor
Table Performance of algorithms in the lifelong model editing task with 1000 edits, where the top three Avg are highlighted in hold.
ZsRE
Counterfact
Llama-2 (7B)
Mistral (7B)
Llama-2 (7B)
Mistral (7B)
Algorithm
Rel
Gen
Loc
Avg
Rel
Gen
Loc
Avg
Rel
Gen
Loc
Avg
Rel
Gen
Loc
Avg
FT-L
0.18
0.16
0.03
0.13
0.52
0.47
0.74
0.57
0.06
0.01
0.04
0.04
0.21
0.10
0.11
0.14
MEND
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
+SimIE
0.82
0.69
0.85
0.79
0.72
0.66
0.83
0.74
0.96
0.31
0.29
0.52
0.94
0.30
0.32
0.52
ROME
0.04
0.05
0.00
0.03
0.04
0.03
0.01
0.03
0.25
0.15
0.06
0.16
0.28
0.24
0.01
0.18
+SimIB
0.64
0.60
0.66
0.63
0.72
0.67
0.82
0.74
0.97
0.51
0.30
0.59
0.96
0.52
0.50
0.66
MEMIT
0.04
0.04
0.02
0.03
0.04
0.04
0.02
0.03
0.00
0.00
0.07
0.02
0.00
0.00
0.00
0.00
+Sim
0.71
0.68
0.65
0.68
0.65
0.62
0.80
0.69
0.81
0.43
0.29
0.51
0.78
0.45
0.46
0.56
AlphaEdit
0.04
0.04
0.02
0.03
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
+SimIe T E
0.84
0.78
0.79
0.80
0.74
0.69
0.84
0.75
0.83
0.45
0.37
0.55
0.77
0.38
0.67
0.61
GRACE
0.98
0.02
1.00
0.67
0.98
0.02
1.00
0.67
1.00
0.00
1.00
0.67
1.00
0.00
1.00
0.67
WISE
0.82
0.76
1.00
0.86
0.72
0.69
1.00
0.80
0.71
0.31
0.37
0.46
0.69
0.35
0.35
0.46
PRUNE
0.54
0.52
0.72
0.59
0.25
0.25
0.47
0.33
0.71
0.35
0.67
0.58
0.86
0.42
0.56
0.61
AlphaFdit
0.84
0.79
0.64
0.76
0.79
0.73
0.75
0.76
0.91
0.53
0.30
0.58
0.94
0.49
0.53
0.65
long model editing methods. For instance, when edit- ing Llama-2 on the ZsRE dataset, AlphaEdit+SimID achieves an Avg of 0.80, outperforming GRACE, PRUNE, and the original AlphalEdit. Similarly, in experiments on GPT2-XL using the ZsRE dataset, ROME+SimIE attains an Avg of 0.81, surpassing all lifelong model editing methods. These improvements are also reflected across individual metries (e.g., Rel, Gen, and Loc), providing comprchensive evidence for the effectiveness of SimIE.
Llama-2 (7B)
1.0
0.8
Success Rate
0.6
Avg
0.4
Rel
Gen
0.2
Loc
5.2. Impact of hyperparameter
0.0
0.01
0.1
5.0 Hyperparameter À
1.0
10
30
50
Section highlights the importance of the hyperparameter À in SimIE. Thus, examine the impact of various À values on the performance across methods. Specifically, for each standard model editing algorithm, we apply SimIE with E {0.01, 0.1, 1, 5, 10, 30, 50). The experimental results are presented in Figures to 10, with Figure 3 illustrating selected results for ROME+5 MIE on 1lama-2 using the ZsRE dataset. It important to note that the values of À used in Section 5.1 correspond to those that yield the highest av- erage (Avg) performance, summarized in Table 3. Based on these findings, we make the following observations:
Figure 3. Performance of ROME+31mIB on the ZsRE dataset across various À values, under T = 1000 sequential edits.
amplifies errors arising from violations of Assump- tion 3.3 and the matrix inversion process.
As the hyperparameter À increases, Rel and Gen generally decline, whereas Loe tends to improve. This behavior can be explained as follows: 1) Larger values of À cause the solution to deviate from the ideal editor, which adversely affects Rel and Gen; 2) An increased À enhances the stability of the solution, char- acterized by the smaller norm, thereby improving Loc. These findings emphasize the trade-off between opti- mality and sensitivity established by Theorem 4.3.
The Avg exhibits robustness across a specific range of the hyperparameter A. In most experimental set- tings, Avg remains stable for À varies within the range [1, 50). However, extremely small values of À may lead to catastrophic performance degradation. This phe- nomenon can be primarily attributed to the increasing sensitivity of the solution provided by SimIE, which
8

---

Simulating Ideal Editor
5.3. More recent LLMs
Impact Statement
We conduct additional evaluations on recent model, includ- ing Llama-3 (8B) (Grattafiori et al., 2024) and Qwen2.5 (7B) (Yang et al., 2025). We select FT-L, ROME, AlphaEdit, WISE, and AlphaEdit as baselines and perform T = 1000 sequential edits using the ZsRE dataset. The experimental results are presented in the Table 2.
This work proposes a general framework that restores the strong performance of standard model editing methods in lifelong context, bridging the gap between these two paradigms. Given the theoretical nature of our study, we have not identified any direct ethical concerns or negative societal impacts related to our research. While our contribu- tions to model editing are intended to enhance the trustwor- thy deployment of LLMs, it is important to acknowledge that, like other advances in this field, there exists the po- tential for misuse to inject harmful content. As a result, it is essential to carefully monitor the usage of model editors during deployment.
Table 2. Performance of algorithms in the lifelong model editing task with 1000 edits, where the top-1 Avg are highlighted bold.
ZsRE
Llama-3 (8B)
Qwen2.5 (7B)
Algorithm
Rel
Gen
Loc
Avg
Rel
Gen
Loc
Avg
FT-L
0.17
0.14
0.01
0.10
0.09
0.08
0.02
0.07
ROME
0.09
0.08
0.01
0.06
0.22
0.22
0.09
0.18
References
+SimIE
0.75
0.72
0.80
0.75
0.92
0.87
0.72
0.84
AlphaEdit
0.06
0.06
0.03
0.05
0.67
0.63
0.76
0.68
Brown, T., Mann, B.,, Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pp. 1877-1901, 2020.
+SimIE
0.74
0.67
0.75
0.72
0.91
0.83
0.88
0.87
WISR
0.51
0.50
1.00
0.67
0.54
0.53
0.99
0.69
AlphaEdit
0.86
0.78
0.62
0.75
0.98
0.80
0.72
0.83
We observe that SimIE achieves competitive performance across these recent models, especially surpassing the SOTA method (AlphaEdit) by an average of 4.8% on Qwen2.5. These results are consistent with those in the main paper, fur- ther confirming the effectiveness of our proposed SimIE.
Chaplot, D. S. Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel. guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven scao, thibaut lavril, thomas wang, timothée lacroix, william el sayed. arXiv preprint 2023.
6. Conclusion
De Cao, N., Aziz, W., and Titov, I. Editing factual knowl- edge in language models. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pp. 6491-6506 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.5//1111/ emnlp main. URL https: https/////aaaahhhooaooo org/ org/2021 22/222212222 . emnlp main. 522.
This paper presents SimIE, a general framework that gen- eralizes standard model editing methods to lifelong sce- narios while preserving strong performance. Theoretical analysis confirms that SimIE remains effective even under relaxed assumptions, and an iptimility-stabilityy trade-off arises when both assumptions fail. Extensive experiments across multiple LLMs demonstrate that standard algorithms integrated with SimIE achieve comparable performance to specialized lifelong model editing methods. Our work not only provides robust baselines for lifelong scenarios but also enables lifelong editing to benefit from the ongoing ad- vancements in standard editing techniques, thereby bridging the gap between these two paradigms.
Fang, J., Jiang, H., Wang, K., Ma. Y., Wang, X., He, X., and Chua, T.-s. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02335, 2024.
Golub, G. H. and Van Loan, C. F. Matrix computations. JHU press, 2013.
Acknowledgements
This work is partly supported by the National Natural Sci- ence Foundation of China (No. 62306255, 92370204), the National Key Research and Development Program of China (No. 2023YFF0725000), the Guangdong Basic and Ap- plied Basic Research Foundation (No. 2024A1515011830 the Fundamental Research Project of Guangzhou (No. 2024A04J4233), and the Education Bureau of Guangzhou Municipality.
Gong, Z. and Sun, Y. An energy-centric framework for category-free free out-of-distribution node detection in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 908-919, 2024.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A.,

---

Simulating Ideal Editor
Vaughan, A., et al. The llaa 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Huang. Z., Shen, Y., Zhang, X., Zhou, J., Rong, W., and
Xiong, Z. Transformer-patehhn One mistake worth one
neuron. arXiv preprint wrXiv22301.09785, 2023.
Guo, K., Wen, H., Jin, W., Guo, Y., Tang, and Chang, Y. Investigating -ut-of-distribution generalization of gnns: An architecture perspective. In Proceedings of the 30th
Laub, A. J. Matrix analysis for scientists and engineers. SIAM, 2004.
ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 932 943, 2024a.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436 444, 2015.
Guo, S., Guo, Y., Zhang, H., and Wang, J. Mitigating up- date conflict in non-iid federated learning via orthogonal class gradients. IEEE Transactions on Mobile Computing, 2024b.
Levy, O., Seo, M., Choi, E., and Zettlemoyer. L. Zero-shot relation extraction via reading comprehension. arXiv preprint Tet:::7006.444 2017.
Lin, H., Zhu, H., Zuo, Y.. Zhu, C., Wu. J., and Xiong, H. Collaborative company profiling: Insights from an employee's perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Guo, Y., Guo, K., Cao. X., Wu, T., and Chang, Y. Out- of-distribution generalization of federated learning via implicit invariant relationships. International Confer- ence on Machine Learning, pp. 19905-11333. PMIR, 2023.
Lin, X., Huang, Z., Zhao, H., Chen, E.. Liu, Q., Wang, H., and Wang, S. Hms: A hierarchical solver with dependency-enhaaneed understanding for math word prob- lem. In Proceedings of the AAAI conference on artificial 2021.
Gupta, A., Rao, A., and Anumanchipalli, G. Model editing at scale leads to gradual and catastrophic forgetting, arXiv preprint arXiv:2401.07453, 2024a.
intelligence, volume 35, pp.
Gupta, A., Sajnani, D., and Anumanchipalli, G. A unified framework for model editing. arXiv preprint arXiv:2403. 14236, 2024b.
Liu, J., Huang, 7., Lin, X., Liu, Q., Ma, J., and Chen, E. A cognitive solver with autonomously knowledge learning for reasoning mathematical answers. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 269-278. IEEE, 2022a
Hansen, P. C. deficieft and discrete ill-posed prob- lems: numerical aspects of linear inversion. SIAM, 1998.
Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., and Ghassemi, M. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, volume 36, 2024.
Liu, J., Huang, Z., Ma, Z., Liu, Q. Chen, E.,, Su, T., and Liu, H. Guiding mathematical reasoning via mastering commonsense formula knowledge. In Proceedings of the 29th ACM SIGKDI Conference on Knowledge Discovery and Data Mining, pp. 1477-1488, 2023a.
Higham, N. Functions of matrices: Theory and computation, 2008.
Liu, J., Huang, Z., Zhai, C., and Liu, Q. Learning by ap- plying: A general framework for mathematical reasoning via enhancing explicit knowledge learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, 2023b.
Higham, N. J. Accuracy and stability of numerical algo- rithms. SIAM, 2002.
volume 37, pp.
Hu, E. Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 00600. 2021.
Liu, Q., Zhang, Q., Zhao, F., and Wang, G. Uncertain knowl- edge graph embedding: An effective method combining multi-relation and multi-path. Frontiers Comput. Science, 18(3):183311, 2024.
Hu, X., Cheng, Y., Zheng, Z., Wang, Y.. Chi, X., and Zhu, aware rec-
Liu, Y., Ao, X., Qin, 7., Chi, J., Feng, J., Yang, H., and He, Q. Pick and choose: gnn-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021, pp. 3168-3177 2021.
H. Boss: bilateral ommender system for online recruitment. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4146-4155, 2023.
sutiiiioon--uuuaaiiiii-------eee aware
Liu, Y., Ao, X., Feng, F., and He, Q. Ud-gnn: Uncertainty- aware debiased training on cemi-homophilous graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1131-1140, 2022b.
Huang, M., Liu, Y., Ao, X., Li, K., Chi, J., Feng, J., Yang, H., and He, Q. Auc-oriented graph neural network for fraud detection. In Proceedings of the ACM Web Conference 2022, pp. 1311-1321, 2022.
10

---

Simulating Ideal Editor
Liu, Y., Ao, X., Feng, F., Ma, Y., Li, K., Chua, T.-S., and He, Q Flood: A flexible invariant learning framework for out- of-distribution generalization graphs. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1548-1558, 2023c.
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. Language models knowl- edge bases? In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1/D19--122 URL https: httpss//aallatthhllooooo wwg.
Ma, J.-Y., Wang, H., Xu, H.-X., Ling, Z. and Gu, J.-C. Perturbation- sequential model editing, arXiv preprint arXiv:2405. 16821, 2024.
Madaan, A., Tandon, N., Clark, P., and Yang, Y. Memory- assisted prompt editing improve gpt-3 after deployment. arXiv preprint ar arXiv:2201.0600 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Malyshev, A. N. A unified theoryof conditioning for linear least squares and tikhonov regularization solutions. SIAM journal on matrix analysis and applications, 24(4):1186- 1196, 2003.
Rifkin, R. M. and Lippert, R. A. Notes on regularized least squares. Technical Report MIT-CSAILTR-2007UU 2007.
Manakul, P., Liusie, A., and Gales, M. J. Selfcheckgpt: Zero- resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.088 2023.
Shen, D., Qin, C., Wang, C., Dong. Z., Zhu, H., and Xiong, H. Topic modeling revisited: A document graph-based neural network perspective. In Advances in Neural Infor- mation Processing Systems, volume 34, pp. 14681-14693, 2021.
McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426. 2018.
Sinitsin, A., Plokhotnyuk, V., Pyrkin, D.,, Popov, S., and
Babenko, A. Editable neural networks. In International
Conference on Learning Representations, 2020.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, volume 35, pp. 17359 17372, 2022.
Sun, Y., Zhu, H., Qin, C., Zhuang, F., He, Q., and Xiong, H. Discerning decision decision-making making process of deep neu- ral networks with hierarchical voting transformation. In Advances in Neural Information Processing Systems, vol- ume 34, pp. 17221-17234, 2021a.
Meng, K., Sharma, A. S., Andonian, A. J., Belinkov. Y., and Bau, D. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https:/ httpp::///peeeevieeee n forum? d-M.kccAHyyyyy....
Sun, Y., Zhuang, F., Zhu, H., Zhang, Q. He, Q. and Xiong, H. Market-oriented job skill valuation with cooperative composition neural network. Nature communications, 12 (1):1992, 2021b.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. Fast model editing at scale. In International Confer- ence on Learning Representations, 2022a. URL https: htt////ouumummeeeeoeeee //openreview htt.://wwww....eeee..eeeeeeeeeee
Sun, Y., Zhu, H., Wang, L.,, Zhang, L.,, and Xiong, H. Large- scale online job search behaviors reveal labor market shifts amid covid-19. Nature Cities, 1()):100-163 2024.
Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C. MMmory- based model editing at scale. In Inter- national Conference on Machine Learning, pp. 15817- 15831. PMLR, 2022b.
Tan, C., Zhang, G., and Fu, J. Massive editing for large language models via meta learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: http////ppp//vveeee net/ forum? 1 CJQQ2P..
Peng, L., Giampouras, P., and Vidal, R. The ideal continual leamee: An agent that never forgets. In International Con- ference on Machine Learning, pp. 27585-27610 PMLR, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Penrose, R. A generalized inverse for matrices. In Math- ematical proceedings of the Cambridge philosophical society, volume 51, pp. 406-413. Cambridge University Press, 1955.
Vaswani, A. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.

---

Simulating Ideal Editor
Wang, C., Zhu, H., Zhu, C., Qin, C.,, and Xiong, H. Se- trank: A setwise bayesian approach for collaborative ranking from implicit feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6127-6136, 2020.
Zhang, N., Tian, B., Cheng, S., Liang, X., Hu, Y., Xue, K.,
Gou, Y., Chen, X., and Chen, H. Instructedit: Instruction- based knowledge editing for large language models. arXiv preprint arXiv:2402. 16123, 2024a.
Zhang, N., Yao, Y., Tian, B., Wang, Deng, S., Wang, M., Xi, Z.,, Mao, S., Zhang, J., Ni, Y., et al. comprehensive study of knowledge editing for large language models. 2024b.
Wang, L., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
arXiv preprint arXiv:2401.001
Zhao, W. X., Zhou, K., Li, Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang. J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.1822. 2023.
Wang, P., Li, Z., Zhang. N., Xu, Z.., Yao, Y., Jiang, Y., Xie, P., Huang, F., and Chen, H. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405.14766 2024b.
Zheng, C., Li, L., Dong, Q. Fan, Y., Wu. Z., Xu, J., and
Chang, B. Can we edit factual knowledge by in-context learning? arXiv preprint 2rX1r:2305.12740, 2023.
Xin, H., Sun, Y., Wang, C., and Xiong, H. Llmedsr: Enhanc- ing cross-domain sequential recommendation with large language models. ACM Transactions Information Systems, 2025.
Xu, D.,. Chen, W., Peng, W., Zhang, C., Xu, T., Zhao, X., Wu, X., Zheng, Y., Wang, Y., and Chen, F. Large lan- guage models for generative information extraction: A survey. Frontiers of Computer Science, 18(6):186357 2024.
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint wxxvv25555003388 2025.
Yao, Y., Wang, P., Tian, B., Cheng, S., Li, Z., Deng, S., Chen, H., and Zhang, N. Editing large language mod- cls: Problems, methods, and opportunities. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 10222-10244 10240, Singapore, Decem- ber 2023. Association for Computational Linguistics. doi: 10. 10.11663/1112003..mmm. 632. URL https:// aclanthology www.::2....mmmmmmmmmmmmmmmm..mmm www.e-mnnmmmm...e mwapppmmmii...mmmmm 632.
Yu, L., Chen, Q., Zhou, J., and He, L. Melo: Enhancing model editing with neuron- neuron-indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 19449-19457, 2024.
Zeng, G., Chen, Y., Cui, B., and Yu, S. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 188):364-32,, 2019.
Zhang, L., Zhou, D., Zhu, H., Xu, T., Zha, R., Chen, E., and Xiong, H. Attentive heterogeneous graph embedding for job mobility prediction. Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 2192-2201, 2021.
12

---

Simulating Ideal Editor
A. Additional discussions
A.I. Related work
Advances in deep learning (LeCun et al., 2015; Sun et al., 2021a;b; Wang et al., 2020; Guo et al., 2023; 2024b; Xu et al., 2024) have laid the foundation for the development of LLMs, equipping them with broad cross- doms-n knowledge (Liu al., 2023a;b; Xin al., 2025; Liu et al., 2024). These models have demonstrated strong versatility across diverse areas, including graph learning (Guo et al., 2024a; Liu al., 2021; Shen al., 2021; Zhang et al., 2021; Huang et al., 2022; Liu et al., 2022b; 2023c; Gong& & Sun, 2024), data mining (Lin al., 2017; Sun et al., 2024), and AI for science (Hu et al., 2023; Lin et al., 2021; Liu et al., 2022a). To enable low-cost correction of outdated or harmful information within LLMs, recent years have witnessed a surge in the development of model editing techniques (Zhang et al., 2024b).
Standard Model editing. Existing model editing techniques generally fall into two main categories: parameter-modifling methods directly update the parameters most relevant to
and parameter-preserving (Yao et al., 2023). Parameter-modifyy knowledge, thereby producing correct predictions. KE (De Cao et al., 2021) and MEND (Mitchell et al., 2022a), known as meta-learning approaches, update LLM parameters by learning hypernetwork. MALMEN (Tan et al., 2024) formulates the update aggregation as a least-squares problem, mitigating the cancellation effect observed in MEND. InstructEdit (Zhang et al., 2024a) enhances these methods by incorporating instructions for training on different tasks, enabling adaptation to a variety of tasks. Additionally, ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023), known as locate-and-edee and-edit methods, first identify the storage locations of knowledge (typically in MIP modules) and then perform targeted updates methods store necessary information in memory,
on the LLMs. Instead of altering the parameters, parameter-preser which is leveraged to guide the model in generating accurate predictions. MemPrompt (Madaan et al., 2022) and IKE (Zheng et al., 2023) utilize memory-based in-context learning to teach LLMs to generate the edited knowledge. SERAC (Mitchell et al., 2022b) constructs a retrieval- augmeeeedd counterfactual model to modulate LLM predictions as needed. MELO (Yu et al., 2024) dynamically activates certain LoRA (Hu et al., 2021) blocks based on an inner vector database, thus altering the behavior of the L.MMs. While these standard editing methods achieve commendable performance, they suffer significant degradation in lifelong scenarios, particularly auramtter-modifyee methods. Our work introduces general framework that restores the strong performance of garameter-modifyine methods in lifelong context.
Lifelong model editing. Lifelong model editing poses a greater challenge as it involves performing sequential edits on LLMs to accommodate evolving knowledge (Yao et al., 2023). T-Patcher (Huang et al., 2023) adds a handful of trainable neurons to the last fredd-Fowwardd Network layer, enabling GRACE (Hartvigsen et al., 2024) maintains discrete codebook for chosen layer, which is used to retrieve and replace the latent representations of examples. WISE (Wang et al., 2024b) designs dual parametric memory scheme for edited mechanism to reduce conflicts. PRUNE (Ma et al., 2024) proposes
rectify series of mistakes encountered by LLMs.
knowledge, combined with knowledge-sharding applying condition number restraints during sequential editing, preserving the general abilities of LLMs. AlphaEdit (Fang et al., 2024) projects updates into the null space of preserved knowledge while incorporating a regularization term to protect previously edited knowledge. Existing studies primarily focus on developing specialized approaches for lifelong model editing, often overlooking the potential of generalizing standard methods to lifelong scenarios. In contrast, our work aims to generalize standard model editing methods to lifelong scenarios while maintaining strong performance, thereby bridging the gap between these two paradigms for the first time.
A.2. Limitations
The first limitation involves the ideal editor, where an ill-conditioned key matrix K may generate parameter shifts S with a large norm. This design overly prioritizes injecting new knowledge into LLMs at the expense of preserving their general capabilities. This issue can be mitigated by imposing additional locality constraints on the ideal editor, such as limiting updates to a subspace that preserves original knowledge (Peng et al., 2023; Zeng et al., 2019). The second challenge concerns the hyperparameter À in SimIE, whose optimal value cannot be determined in advance. Future work could enhance the recursive procedure of SimIE to enable flexible selection of an appropriate hyperparameter with minimal computational cost, similar to the technique in regularized least squares (Rifkin Lippert, 2007).
13

---

Simulating Ideal Editor
B. Further details on Section 3
B.1. Existence and uniqueness of the ideal editor
Existence. The ideal editor exists and only if the linear system SK B has a solution. To establish the existence, we analyze the system through a www- decomposition. The system SK" B can be equivalently expressed collection of sub-linear systems:
s = Br: E [d2),
where denotes the i-th row of the matrix S, and Bi,; denotes the corresponding i-th row of the matrix B. For the linear system SK = B to have a solution, is necessary and sufficient that each sub-linear system s;K = BA, has solution.
Observe that each subsystem s,K = Bi, can be rewritten in its transpose form as K T s = B... With Assumption 3.2, the rows [kg 1 of KT are linearly independent. Thus, we have the following rank condition:
rank(KT)) rank([K I,.
(6)
By the Rouché Rouché-Capelll theorem, Equation (6) implies that the sub-linear system s; Bi is consistent, i.e., at least one solution exists.
Since the analysis holds across all rows, conclude that all sub-linear systems admit at least one solution. As result, the system SK B has solution, thereby proving the existence of the ideal editor.
Uniqueness. The general solution to the linear system SK = B can be characterized by (Theorem 6.11. in Laub (2004)):
S= -KK).
KK!),
where Y Rd2 is arbitrary matrix. The term I - KKt represents the orthogonal projection operator onto the null space of KT (Golub & Van Loan, 2013). Thus, we may verify that the two components BKt and Y KKt) are orthogonal, i.e.,
(Y - ((I-KKt, (Kt),KK)) = trace (KIIIKKK]] KK BKT)
trace((I - KKRYK.TTAAR BK!)
trace(y TK KK())
trace(Y B(Kt - KiKKt))
0,
where (a) follows from the fact that 7 KKt is symmetric matrix; (b) utilizes the eyclic property of the trace; and (c) is a consequence of Moore-Penrose conditions, specifically K'KKt = K From the above, the Frobenius norm of any solution S satisfying SK = B can be decomposed as:
||
||BK*||
KKD |BK
Clearly, the Frobenius norm of || is minimized if and only if Y K KKt = 0, which implies 5 = BKt, Therefore, SO is the unique minimum norm solution SK = B, establishing the uniqueness the ideal editor.
Remark B.I. If the columns of are linearly independent (or equivalently, if A is left invertible), then the Moore-Penree inverse At = (ATA) A)-IAT. Thus, under Assumption 3.2, S can also be expressed as 5 = B(K K) -T which is consistent with the solution form derived using the method of Lagrange multipliers.
14

---

Simulating Ideal Editor
B.2. Detailed derivation of the recurrence relation
We now present the full derivation of Equation (4). The details are as follows:
S (Ba-1K bek, (K2K, + A1)
denoted
Ba-1K 1-1 P.p. + beki P.-
Ba-IKKK- 1P-- BI-IKK-IPP-
Pe- P. bekk P.
=S/
8/
Pt- Pe + b P.
(8)
=P:
S
Sp lisk P.- + beki P
8
+
be
S P.
+ kikk P.-
Here, (a) follows from the recursive formula for PA given by P. PA-1 + kick (b) results from the Equation (3), which holds under Assumption 3.3.
B.3. Proof of Theorem 3.6
Theorem 3.6*, Let St Wr- Wo denote the parameter shift generated by SimIE according to Equation (5), and let So represent the ideal editor w.r.t. Equation (2). Under Assumptions 3.2 and 3.3, the following bound holds:
A
||
1
A
|
m +
where 0 < Omin & Fmax are the smallest and largest singular values of the key matrix K, respectively.
Proof. We begin by expressing the key matrix = UEVT through SVD, where U E Red xd: and V E RTXT are orthogonal matrices, and 2 E Rdd xT is diagonal matrix containing the singular values. Under Assumption 3.2, we can rewrite the ideal editor S as follows:
S =²=((KT)--L
-BVV DOVEEVT)-LVEEL
(9)
BVV VV
BV(ETE)
where (a) follows from the matrix transpose property (ABC) CTBTAT AT; and (b) utilizes the inverse property of matrix products (ABC)-1 C-1 1- Analogously, for the solution St generated by SimIE, there exist:
S BK (KKT +
-VE (UEVTVETUT +
(g) (UEE UJ +XUUTY-
(10)
BVV.TU (USEE +
=BVX CUSTUTT(EN" +UT
BVET (ΣET + -UT,
15

---

Simulating Ideal Editor
where (a) employs the decomposition of the identity matrix UUT. We are interested in the difference between S and S
S - S7. BV (( E) - (EE) + +
BV
ΣЛΣ (EET + +1)
= BV
E Σ) ET
E Σ) E (ΣET + = UT.
T2
Given that K has full rank, E admits the block structure:
Σ
Σ
OT
(11)
0
where 2 = diag( (I,,..., E TxT contains the singular values. Consequently, the term 11 can be derived as:
Σ1 0]
) (2)
[
0
[
0
[
0]
(12)
Next, the term I22 can be expressed as:
Σ1
Σ
H
0 12
-1
EΣ
+
+1)=11
Σ
0
Σ
0
+
A
22
0
A41
0
-1
+
0
0
X12
X
(E]
X41 -1
0
0
(AT2)-1
(2
X1)
-1
0
0
0
Now, substituting the expressions for I and 12, we obtain:
+ AT1) 0
0
50 -
BV
Σ
0]
- |Σ 0
ut
[h
0
+
-1)
0 UT
BV [Σ]]]
0]
0
0
0
A
0
BV [M]]
0]
UT
0
0
BV |XE] (E)
+
A
0
UT.
where the penultimate equality follows from the identity
for each diagonal element
of+h
+
To compute the Frobenius norm of 8 -SA, rewrite BV RdxXT in column-vector form as BV [bvz bbz ... bor].
16

---

Simulating Ideal Editor
yielding:
||50
S _
[bb
buz ...
burl A +
O
0 0|[
bbu
( -ft 1bu2
( of
bbu
0i
of
+
T
(13)
A
)
bu, 11
min
[=1
)2
BV ES]
U 2
min
1 |
min
where (a) uses the fact that Frobenius norm is unitarily invariant; (b) utilizes the definitions of the Frobenius norm and 2n-norm; and (c) can be obtained by substituting Equation (12) into Equation (9).
The lower bound follows analogously by replacing miin with max in Equation (13), completing the proof.
B.4. Cost analysis
Here, we analyze the additional cost introduced by SimIE. For analytical tractability, we consider that each time step involves N tokens and [C] linear layers are edited.
Storage costs. SimIE involves maintaining L matrices p!() of dimension di x d1, with a direct storage cost of O(d))) By leveraging the low-rank structure p! an alternative strategy incrementally store K(O) [x(") k(0) which enables reconstructing p(d) K [K]]]= + X/. In this case, the storage cost becomes (TTLNd)), which depends on the number of sequential edits T. If the total number of tokens is significantly smaller than the input dimension, i.e., TN < di, storing K. may be more practical choice. However, storing K (1) could pose privacy concerns compared to storing P!
Computational costs. The primary computational cost of the proposed SimIE stems from matrix inversion. Rec- ognizing the inherent numerical inaccuracy in matrix inversion (Higham, 2002), we instead solve the linear system XP." = AW,") k(()) [x[O]T to provide the adjusted parameter updates. Among available methods", LU decomposition is standard approach, which has time complexity of dz) Since p(() is a symmetric positive definite matrix, Cholesky decomposition offers higher computational efficiency with a reduced time complexity of (f Additionally, SVD potential alternative due its excellent numerical stability, though it incurs a higher time complexity of (9d4). The detailed derivation of the time complexity of these matrix decompositions Van Loan (2013). Fortunately, all these decomposition techniques have been efficiently parallelized on GPUs, significantly accelerating computation.
be found in Higham (2008) and Golub &
C. Proofs of theorems in Section 4
C.1. Proof of Theorem 4.1
Lemma C.I. Consider least squares problem min ||SK BI where the matrix K rank-deficient. If the SVD of matrix K is K = UEVT, then the minimum squared residual norm can be expressed as min ||SK - -B1 = where by; denotes the i-th column of the matrix BV, and r is the rank of K.
Proof. We start with the expression for the squared residual norm:
ISK- BI = | B = |SSEE BV
(14)
where the second equality follows from the unitary invariance of the Frobenius norm. Since K is rank-deficient with rank Only the complexity the decomposition considered, as the back substitution has the same complexity across methods.
17

---

Simulating Ideal Editor
the singular value matrix E has the following block structure:
01
02
Σ1
(15)
o
0
where > 0 denotes the smallest non-zero singular value of K, and En diag(o E Rx contains the non-zero singular values.
Let SU =
|uuz su2 .... sur) and BV
[buz buz ...
bur denote the column partitioning of SU and BV.
respectively. Then the squared residual norm in (14) becomes:
Σn
||SUE - BV
[su
su2
sur
buz
bvr
0
[0181
0)
[bbe
buz
bvr
|
8811
bo
028122
buz
or
bur
bor+1
-buT]]l ]|
lquu
buil
bb;
12
To minimize squared residual norm, we observe that P can be minimized independently for each E 1,...,r by choosing su; bez which makes I'1 = O. The second term I'2 is independent our optimization variables and represents the irreducible error due to the rank deficiency of K. Therefore, the minimum squared residual norm is:
min SKK
BI min Σ 1oss4 - 1 +
bu
22
l
(16)
imr+1
This completes the proof.
Theorem 4.1*, Let St WT - Wo represent the parameter shift generated by SimIE according to Equation (5), Under
Assumption 3.3, the squared residual norm w.r.. St. satisfies the following inequality:
X3(BBBI -Rmin)
Rmin S -||StK-
- BB S
+ Rmins
(0? + A)2
where Rmin := min |SSK- B12 is the minimum value of squared residual norm, and o, > denotes the smallest non-zero singular value of K.
Proof. Using the SVD of K U.VT. we can decompose the term S K as follows:
S - B = BK (KKT - K B
BVE (E + XT) T U.VT B
=B(V T + x1) - -1)
=BV ( (ΣET An) -1 1) VT.
18

---

Simulating Ideal Editor
Base on block structure of E W.r.t. Equation (15), we simplify the term I1
Σ1
Σ
Σ1
-1
Σ
0 0
Σ STEE
A1)
-
+
X
0
0
12]
0
Σ1
[E]
Σ
0
[]]
0
0
(AI2) -1
0
0
12
E
X1)
0
T
0
12
(E)
+ XI)
0
A12
where the last equality follows from the identity BV _dxx in column-vector form BV [buu
of
for each diagonal element i. We now express
a.
buz
... bUT]. obtaining
||
B
1 F
ben
bvz
borl
0 -Xi + -
VT
2
-X12
F
bwn
> boz
> bur ox
buT bur+1
oz+
T
2 ?1 11:0:112
+
|lbus 1
1=1
1=++1
bbuu
12
+
Σ l|bus
0. +
i=++1
X²
T
T
T
|bu
12
b
2
+
bb;
(o? + A)2
1m
=miu|| SK
5K-BB
BIB
( BVI
min |ISK Bl(
min |S
B1²
(o? + A)2
denoted as Runin
X² BI 2 Rmin)
+ Rmin-
(o? X)2
The penultimate equality comes from Lemma C.1, and the last equality follows from the unitary invariance of the Frobenius norm. Since the squared residual norm of any solution cannot be smaller than the optimal squared residual norm, this completes the proof.
C.2. Proof of Theorems 4.2 and 4.3
Lemma C.2. At time step I, suppose the pair (kie, b)) is perturbed to (k, + 5kk, b, + 8bi), which results in the solution generated by Simlā. E being S.AA = (B.KTT + ōb,ōk; K + Al + 8k;ōkk,) - Then, at time step + 1, the effect of the perturbation is not amplified, i.e., S = (B.+1K]+ + (b, (K + AI + 5k,6k]] )!
Proof. We demonstrate the non-amplification of the perturbation by working backward from the recurrence relation w.r.t. Equation (8). Let P.- = KT + XI, we have:
st S + A ih (T2001 + Sk;ōk
SP, ok (( ōk,ōh. )- -1 + b k ( (Ptt1 ōk,ōk; ) )-1 = =BBKK + 8b.ōk, (P{+1 + Skrōk, 11 + betthit (P++1 + Skiōk; )-1 = =(Br41KTTH + Öb,5k, (K KTin + AI + 5kyōk: -
19

---

Simulating Ideal Editor
The third equality follows by substituting the expression for Sp Thus, the impact of the perturbation to (kg. bx) is not amplified in subsequent time steps.
Theorem 4.2 Let St represent the original solution corresponding to K and B, and S denote the perturbed solution corresponding to K + 5K and B + 8B, both generated by SimIE according to Equation (5). Under Assumption 3.2, the perturbation bound for S is:
S
1|0B|I
+ ddi |K
<cond( S B)
||Jall |8K||
||0K]
IF
[ 1r
|BIlF
||
(K) |
vdi ||
Scond(S7, B)
|
|BI|
|
min
where JA = KKKK + + denotes the Jacobian matrix of St at B.
Proof. To analyze the perturbation behavior of ST. w.r.t. small changes in K and B, we introduce
|| ||||
[|8 IF
c=
max
|
measure of the relative magnitude of the perturbations. Following Golub & Van Loan (2013), we define the function:
8K ST g(x) = (B + [K + x S (K (K+ + x
8B
8K [K + I
òK + X1
where c0, It is easy to observe that (0) St and g(e) = St., corresponding to the unperturbed and perturbed
solutions, respectively. Using a first-order Taylor expansion, we have:
g(t) (()) + g0)e + 0(e),
where 00) the derivative of gxx) evaluated at 0, given by:
g0( = (8B - S76K)
K TTKKK + A1) B S
òK (KK +XI-
Omitting second order terms, we get:
8 S
(0)((
1|
|
||(BB SYOKK (KKT + +10 ||B ||B|I|BSKK SAKK)KTTKKK (KK +1
S
|| 1r
[KKK A1)
1001I + |K TKKK + A1) ||01|
|
S K
|| [KKK + A1)-
(17)
+
||8K|F
||SA|] I
K T (KKT + +XT)
||
||8
(KKT + 1 |18K|]
|| 1|
B
|
+ 02
||(KKT + A
||
min
+
S
122
20

---

Simulating Ideal Editor
The last inequality substitute | Sq. K | with the squared residual norm bound from Theorem 4.1, where Rmin 0 and or min hold under Assumption 3.2.
For the term Ii we show that it is equivalent to the condition number cond B). Specifically, since the solution Sy. = BK (KK" + x1) is differentiable w.r.t. B, the condition number can be written as:
||oS
||8||
|B |
|B]| I1, | |F
cond(5), B) = lim sup
||
||
(KTTKKTTTN + +11)-
c-0 pp
||
|
where JA = K (KK + X1> denotes the Jacobian matrix of St at B.
Next, we analyze the term 2 using the approach outlined Theorem 3.6. Under Assumption 3.2, let K UEVT denote the SVD of K, where Σ
Σ1 defined in Equation (11). Recalling that B S, we have:
0
|B F S
|
< || [K]]
1F
|
BV
Σ) E
||
|BVE
(ΣET
+ +A) - lp
Σ1
BV[[EI
0
)-1
Σ
=
||K IF
Σ1
BV Σ
0]
(
Σn
0
+ X
)-1
0
1
(b) ||]|
|| |
be
bur
Σ
0
||bb
bei
bUt
11 (2)
+ X1
(18)
bu1
biz
buT
0
=|K
1
01
ov
bu
boz
but
x
2'
bus
11
= =|K]|
a.
c: be: 12
1=
(c)
Σ i= ae
1 bws 11
< |||]|
02 )2 Σx=
bu:
oe
2
+ A
mln
KK]r;
min
where (a) follows from Equations (9) and (10); (b) involves rewriting BV E RdxxT in column-vector form as BV =(bvi
boz ... bor]: and (c) utilizes the monotonic increasing property of the function h(xx) (, )2 for all z >
21

---

Simulating Ideal Editor
Σ 0
Finally, we consider the term 333 and substitute K- = -UVV with E
yielding the following expression:
|[KKT + +11
|0EENT +
FF
T1
0
[2] 0] + A
0
12
(E} A )
0
(A12 -
F
2
d1
2
(19)
-=++1
dj
X
di
X
Now, plugging the bounds on r, I'2 and I33 back into Equation (17), we get:
S
|BB| F + ||>x|F
min +
vdi
F < cond B)
||0 + o min + X
|K
|F
1001|
1
|
min
F
(20)
|8B|
|| + ddz || |||]
cSn B)
+ ||Nal
|
|
As ++ 0, the term ||Ax|pp in Equation (20) behaves monotonically increasing. However, we can find an upper bound that
D
independent of À to replace As done in Equation (19), we express K as the SVD K UEVT with E
Then,
0
the second term | l| in Equation (20) becomes:
|
FF
||8K|I
KT (KKT T + -1 F
|
VIE
(ΣET
+
AU)
|
K
IF
|
Σ
I
0
| 1F
ΣΣ o]
Σ
0]
+
A
| 1|
0
12
|
| I
|| (E)
+ A4)
0]| ||
| ||
(21)
(4)
||0 1|p
|||T|
|||| |||]|ppppp
|| 1F
||K]|
|IVEUTT|
|||||
F
||
IF
<F(K)
|
where (a) follows from the inequality matrix.
of 4 o, and (b) involves the definition of the Moore-Penrose inverse for a diagonal
9. I
This completes the proof.
Remark C.3. The term cOnd(SA, B) also relies on X, while admits complex upper bound. According to Equation (21),
22

---

Simulating Ideal Editor
have:
[KKKK" +
[2]
+
A11)
0
T
o
)
max
Kt
F
max
Combining this with Equation (18), the condition number can be expressed as:
|B||
cond(S), B) = [KKK +
+XI) '/r
||S
0 max
o2 min
||
||
o max
min + X
o max
o min
KF (K).
0 max
+ X
easy verify limx-00 cS B) xF(K) and limx cond B) number cond(S)) (S),, B) remains finite for any 0.
Tuas KF (K). Therefore, the condition
Theorem 4.3*, Let S represent the original solution corresponding to K and B, and St. denote the perturbed solution
corresponding to K 8K and 8B, both generated by SimIE according to Equation (5). Then, the perturbation bound for St is:
1 S
Econd(S). B) |1Bl BBI
|| 1F |||||
RAlle
vdi S l|r
+ KF (K))
||8K|P.
S
where RA = St.K B is the residual matrix w.r.t. ST.
Proof. The proof process closely follows that of Theorem 4.2. The key distinctions lie in two aspects: first, the observation that K- UVV with Σ 2
o 0 as defined in Equation (15), and second, the fact that ||ll|F * as noted in Theorem 4.1. With these differences taken into account, the proof is complete.
D. More experimental details and results
D.1. Detailed experimental setup
D.1.1. BASELINES
Here, we describe the nine baseline methods used in our experiments. All methods were implemented using the publicly available knowledge editing framework EasyEdit (Zhang al., 2024b), which reproduces and integrates the original code and hyperparameters provided in the respective papers.
FT-L (Meng et al., 2022) freezes most layers of LLMs, allowing fine-tuning only on single MLP using autoregressive loss.
MEND (Mitchell et al., 2022a) applies a low-rank decomposition to the gradient from standard fine-tuning. which is then fed into a pre-trained hypernetwork to generate new parameter update. The training loss of the hypernetwork consists of standard autoregressive loss over the edit example and the KL divergence loss over localization examples.
ROME (Meng et al., 2022) models the linear layer within MLPs as key-value memory, thereby inserting new knowledge by solving a constrained sast-ss problem. Using causal tracing, ROME identifies mid-layer MLPs as critical for knowledge storage, making them suitable targets for editing.
23

---

Simulating Ideal Editor
MEMIT (Meng al., 2023), based on ROME, enable the simultaneous insertion of hundreds thousands of facts using least-squares method. MEMIT is multi-layer editing algorithm, which introduces edit-distribution strategy to spread updates evenly over the range of mid-layer MLPs.
GRACE (Hartvigsen et al., 2024) maintains a discrete codebook for chosen layer, which is dynamically updated through adding, expanding, and splitting during sequential editing. For inference phase, GRACE retrieves entries from the codebook based on the semantic similarity of the input and decides whether to replace the output accordingly.
WISE (Wang et al., 2024b) designs a dual parametric memory scheme, allocating one memory for pre-trained knowledge and another for edited knowledge. Additionally, WISE incorporates a showledge-shardiigg mechanism, which different sets of edits reside in distinct and orthogonal subspaces of parameters, to reduce conflicts.
PRUNE (Ma et al., 2024) proposes applying condition number restraints during sequential editing by reducing the large singular values of the update matrix, thereby controlling the condition number of the edited matrix. PRUNE minimizes perturbations the original knowledge, preserving the general abilities of the edited model. In our study, we apply PRUNE ROME, it demonstrated robus! performance in the original paper.
AlphaEdit (Fang et al., 2024) projects updates into the null space of preserved knowledge, ensuring that the output of the post-edited LLMs remains unchanged when queried about preserved knowledge. For lifelong scenarios, AlphaEdit incorporates regularization term protect previously edited knowledge. We refer to the version of AlphaEdit in the standard scenario, where the protection component is omitted, AlphaEdit
D.1.2. DATASETS
Here, we provide detailed description of the datasets used in this study. We adopt the train/test split from previous work (Wang al., 2024b; Meng et al., 2022). Except for MEND, which uses the training set to fit the hypernetwork, all other methods perform editing and evaluation directly on the test set.
ZsRE (Levy et al.. 2017) is a equivalent inputs. Each sample in the 7sRE dataset includes question and answer as the edit example (Ne, Ve). rephrased question as the paraphrase prompt z'=- and unrelated question the neighborhood prompt s. Figure 4 provides an example where the corresponding relationships are:
dataset that uses questions generated by roundtrip translation as
Te STC
1/& 2
a alt
rephrase
loc
Counterfact (Meng al., 2022) is more challenging dataset that contrasts counterfactual statements with factual ones. The samples in Counterfact are structured similarly to ZsRE for evaluating reliability, generalization success rate, and localization success rate. Figure 5 provides an example where the corresponding relationships are:
Fe prompt
+
target_new
F rephrase eephrase.prompt
Joc x
locality prompt
D. 1.3. METRICS
Here, we formally introduce the evaluation metrics used in our experiments, which are widely adopted in the model editing literature (Zhang et al., 2024b; Wang et al.. 2024b). Given editing dataset Dedit {(z,, 31) I [T]} with T edit examples, we evaluate the final post-edit model (-) after the T-th edit. The metrics Rel (Reliability, also known as Edit Success Rate (Hartvigsen et al., 2024)), Gen (Generalization Success Rate), and Loc (Localization Success Rate) are used to assess the reliability, generalization, and specificity of the model editing methods, respectively. Specifically:
Rel measures the average Top-1 accuracy on the edit examples (Iz, Ve):
T
1
Rel
Σ
1(Jor (2x) v.).
T
1=1
24

---

Simulating Ideal Editor
"subject"; "IAAF Combined Events Challenge", "src": "When was the inception of IAAF Combined Events Cha lengee?""" "pred": "2011",
"rephrase"; "When was the IAAF Combined Events Challenge launched?", "alt": "2006",
"answers"; "1998"
l, "loc": "nq question: what is the name of the last episode of apongebob", "The String", "cond": "2011 >> 2006 When was the inception of TAAF Combined Events Challenge?", "portability": "Recalled Relation"; "IAAF Combined Events Challenge, event type, athletics)",
"New Question"; "What type of sports event is the IAAF Combined Events Challenge, which was established in 2006?",
"New Answer"; "Athletics"
Figure 4. A sample of the ZsRE datasct.
"case_id"; 0,
"prompt": "The mother tongue of Danielle Darrieux is", "target_new", "English", "subject": "Danielle Darrieux", "ground_truth": truth": "French", "rephrase_prompt"" "Where Danielle Darrieux is from, people speak the language of",
"Locality_prompt" "Michel Rocard is native speaker of", "French"
Figure 5. sample of the Counterfact dataset.
Gen evaluates the average Top-1 accuracy of the model's predictions for paraphrase prompts a/:
T
Gen = Σ ((fo = yc).
T
Loc computes the average proportion of preserved predictions for neighborhood prompts wloe:
T
Loc =
Σ
1(for (2110) = =o (x)oc)
D.1.4. IMPLEMENTATION DETAILS
Unless otherwise stated, we store p!!) and employ LU decomposition across all experiments (see discussion in Appendix B.4). For the hyperparameter X, conduct grid search the set (0.01, 0.1, 1, 10, 30, 50), with the resulting values summarized in Table 3.
Remark D.I (MEND involves an unfair implementation). Since the hypernetwork of MEND tailored to the initial model, its subsequent generated updates significantly violate Assumption 3.3. Given that training the hypernetwork at each time
25

---

Simulating Ideal Editor
Table The hyperparameter SimIE used the experiments.
ZsRE
Counterfact
Algorithm
Llama-2
Mistral
GPT2-XL
Llama-2
Mistral
GPT2-XL
MEND+SimIE 1.0
30
5.0
5.0
10
0.1
ROME+
50
30
50
5.0
5.0
50
MEMIT+SimIE
10
10
1.0
0.1
1.0
0.1
AppaEdit*+Sinee +SimIE
1.0
1.0
5.0
0.1
0.1
5.0
step impractical, we instead use the pre-trained hypernetwork edit the initial model, yielding W.. The update required by SimIE, E, AW; W. W is then computed by subtracting the current model parameters. worth noting that this implementation slightly less fair compared other methods, as it store additional copy of the initial model.
D.2. More experimental results
Here, we present additional experimental results. Table presents the results of editing GPT2-XL using the ZsRE and Counterfact datasets. Figure illustrate the performance of standard algorithms as the number of edits increases. The impact of hyperparameter in SimIE on performance is presented in Figures 10.
In addition, we introduce term, serving as smoothed condition factor. Meanwhile, by using the (squared) geometric mean of reliability and generality, it prevents methods from abandoning one of them. Table 5 shows the evaluation results based on this composite metric, where a
composite metric defined Geo cl(le=))(Ree x Gen). Geo takes locality penalty
Table 4, Performance of algorithms in the lifelong model editing task with 1000 edits, where the top three Avg are highlighted in bold.
ZsRE
Counterfact
GPT2-XI. (I.SB)
GPT2-XL (L.5B)
Algorithm
Rel
Gen
Loe
Avg
Rel
Gen
Loc
Avg
FT-L
0.09
0.08
0.00
0.06
0.11
0.03
0.01
0.05
MEND
0.00
0.00
0.00
0.00
0.00
0.00
0.48
0.16
+S1 mTR
0.56
0.52
0.90
0.66
0.94
0.14
0.15
0.41
ROMB
0.46
0.37
0.50
0.45
0.01
0.00
0.01
0.01
+SimIE
0.93
0.81
0.69
0.87
0.20
0.45
MEMIT
0.58
0.50
0.55
0.54
0.81
0.46
0.25
0.51
+TimTE
0.81
0.66
0.81
0.76
0.88
0.32
0.67
0.62
AlphaBdit
0.10
0.09
0.31
0.17
0.68
0.38
0.30
0.46
+SimIE
0.87
0.77
0.77
0.80
0.98
0.46
0.54
0.66
GRACE
0.35
0.01
1.00
0.45
0.00
0.00
1.00
0.33
0.34
0.33
1.00
0.56
0.01
0.01
0.77
0.26
PRUNE
0.50
0.43
0.84
0.59
0.13
0.04
0.86
0.35
AlphaEdit
0.83
0.70
0.75
0.76
0.98
0.48
0.65
0.70
26

---

Simulating Ideal Editor
Table 5. Performance of algorithms in the lifelong model editing task with 1000 edits.
FTL
MEND
+SinIE
ROME
+ainis
MEMIT
+SinIE
AlphaEda
+SimIE
GRACE
WISE
PRUNE
AlphaEidit
Avg
0.13
0.79
0.03
0.63
0.03
0.68
0.50
0.86
0.59
0.76
1Jamo-2 (7B)
Gco
0.00
0.00
0.42
0.00
0.19
0.00
0.34
0.00
0.45
0.02
0.62
0.16
0.32
0.57
0.74
0.03
0.69
0.00
0.75
0.80
0.33
ZaKE
Mistral (7B)
Goo
0.14
0.00
0.34
0.00
0.34
0.00
0.27
0.00
0.37
0.02
0.49
0.02
0.35
Avg
0:06
0.00
0.66
0.45
0.81
0.54
0.76
0.17
0.80
0.45
0.56
0.59
0.76
GPT2-XI
0.00
Avg
0.04
0.00
0.52
0.16
0.59
0.02
0.51
0.00
0.55
0.67
0.46
0.58
0.58
Liamar2 2 (7B)
Goo
0.00
0.00
0.07
0.01
0.12
0.00
0.08
0.00
0.10
0.06
0.13
0.12
Avg
Counterfact
Mistral (7B)
Goo
0.00
0.00
0.07
0.01
0.18
0.00
0.12
0.00
0.15
0.00
0.07
0.15
0.18
Avg
0.05
0.16
0.41
0.01
0.51
0.51
0.62
0.46
0.66
0.33
0.76
0.35
0.70
GPT2-XL
Mistral (7B) on the ZsRE dataset
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
Gen
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0-
10
50
100
300
500
750
1000
10
30
100
300
500
750
1000
10
50
100
300
500
750 1000
10
50
750 Number of Edits
100
300
500
1000
Number Edits
Number Edits
Number Edits
MEND
MEND SimlE
MEMIT Simll
AlphaEddit
Alphafdit Simll
Llama-2 (7B) on the Counterfact dataset
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.0
100
100
Number of Edits
Number of Edits
Number of Edits
Number of Edits
MEND MEND- SimlE
ROME
ROME SimlE
MEMIT
MEMIT SimlE
Alphafidi
AlphatEdit +SimlE
Mistral (7B) on the Counterfact dataset
1.0
1.0
1.0
1.0
0.8
0.8
0.8
Gon
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
10
50
100
300
500
750
1060
10
30
100
300
500
750 1800
10
50
100
300
500
750 1000
TO 50
100 300 500 750 T00D
Number Edits
Number Edits
Number Edits
Number of Edits
MEND MEND-SimLE
ROME MEMIT
MEMIT+Simit:
Alphatidit
Alphatddt " +SimlE
Figure 6. Performance of algorithms the number of edits increases. with the solid line representing the results combined with the
proposed s mimEE
27

---

Simulating Ideal Editor
Llama-2 (7B)
Mistral (7B)
GPT2-XL (1.5B)
1.0
1.0
AvB
0.8
Rel
0.8
Success Rate
Suecess Rate
Sucvess Rate
0.6
0.6
0.4
Avg.
0.4
0.4
Rel
Gen
0.2
0.2
Loc
0.0
0.1
1.0
0.0
0.
1.0
10
0.0
0.01
1.0
3.0
10
30
Hyperparameter À
Hyperparametter A
Hyperparameter À
(a) HENDD+imIIB on the ZsRE dataset
-lama-2 (7B)
Mistral (7B)
GPT2-XL (1.5B)
1.0-
Avg
Avg
Rel
0.8
Gen
Success Rate
Sucvess Rate
Loc
Lac
0.6
Loc
Success
0.4
0.0
0.01
Hyperparameter À
Hyperpar ametrr À
Hyperparameter À
(b) MEND+SimIB on the Counterfact dataset
Figure 7. Performance of MEND+SimIE across various values, under T
1000 sequential edits.
Llama-2 (7B)
Mistral (7B)
GPT2-XL (L.5B)
1.0
1.0
1.0
Avg
Avg
Avg
0.8
Rel
0.8
Rel
0.8
Rel
Success Rate
Gen
Suceess Rate
Gen
Success Rate
Getn
0.6-
Loc
0.6
Loc
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.01
0.1
1.0
5.0
10
30
50
0.0
0.01
0.1
1.0
5.0
10
30
50
0.0
0.01
0.1
1.0
5.0
10
30
50
Hyperparameter À
hyperparameter
Hyperparameter
(a) ROME+SimIB the ZsRE dataset
Llama-2 (7B)
Mistral (7B)
GPT2-XL (1.5B
1.0
Avg
Avg
Avg
0.8
Rel
Rel
0.8
Rel
Success Rate
Gen
Success Rate
Success Rate
Gen
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.01
0.1
1.0
5.0
30
50
0.0
0.01
0.1
1.0
5.0
0.0
0.01
0.1
1.0
5.0
10
30
50
Hyperparameter À
Hyperparameter À
Hyperparameter
ROME+$im18 on the Counterfact dataset
Figure& 8. Performance of CoME+Siimie across various À values, under T = 1000 sequential edits.
28

---

Simulating Ideal Editor
Llama-2
(7B)
Mistral (7B)
GPT2-XL (1.5B)
1.0
1.0
Avg
AvB
0.8
Rel
0.8
Rel
0.8
Success Rate
Suecess Rate
Success Rate
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.01
0.0
1.0
0.0
1.0
3.0
10
30
Hyperparameter À
Hyperparameter A
Hyperparameter À
(a) NEMIT+SimEL on the ZsRE dataset
Llama-2 (7B)
Mistral (7B)
GPT2-XL (1.5B)
1.0
Avg Loc
Avg
0.8
Success Rate
Gen
Sucvess Rate
Gea
Loc
Success
0.4
0.4
0.0
Hyperparamcter À
Hyperpar ameter À
Hyperparameter À
(b) MEMIT+SimIB on the Counterfacc dataset
Figure Performance of MEMIT+SimEL across various À values, under T 1000 sequential edits.
Llama-2 (7B)
Mistral (7B)
GPT2-XL (L.5B)
1.0
1.0
1.0
Avg
0.8-
0.8
Rel
0.8
Suecess Rate
Gen
Success Rate
0.6
0.6
0.6
Success
0.4
Avg
0.4
0.4
Avg
Rel
Gon
0.2
0.2
Gee
Loc
Loc
0.01
0.1
0.0
1.0
0.0
0.01
1.0
5.0
10
30
Hyperparameter À
Hyperparameter. À
Hyperparameter À
(a) AlphaEdit +Sim1 L. on the ZsRE dataset
Llarn-22 (7B)
Mistral (7B)
GPT2-XL (1.5B)
1.0
1.0
Avg
Avg
Ave
0.8
Success Rate
Gen
Sheeess Rate
Gen
Sucress Rate
Gren
Loc
Loc
0.6
Loc
0.4
0.2
Hyperparameter À
Hyperparameter À
Hyperparameter À
(b) AlphaEdit +SimIE on the Counterfact datasct
Figure 10, Performance of AlphaEdit +SimIE across various À values, under = 1000 sequential edits.
29

---

Simulating Ideal Editor
D.3. Further analyses
D.3.1. CASE STUDY
After 1000 sequential edits on Llama-2 using the ZsRE dataset, we select several edit examples for analyzing the output of the model. Table presents the output results before and after applying SimIE. We observe that all standard methods consistently produced invalid outputs, aligning with the phenomenon reported in Ma al. (2024), where LLMs lose their general abilities after sequential editing. In contrast, standard methods enhanced with SimIE generated meaningful responses that closely matched the desired outputs. These findings further highlight the effectiveness of SimIE, especially in preserving the general capabilities of LLMs.
Table Output model after 1000 sequential edits Llama-2 using Z&RE dataset. mmmmm denotes invalid outputs (e.g.- gibberish empty responses), and errors within the output highlighted in red.
[Case 168]
Prompt: The astronomical body that Lacus Aestatis was located on was what?
Answer: Moon = Edit Target: Mars
MEND
Output: mmmmm X
+SimIE
Output: Mars /
ROME
Output: x
+SimIE
Output: Mars
МEMIT
Output: mmmmm X
+SimIE
Output: Mars /
AlphaEdit
Output: mmmmm X
+SimIE
Output: Mars /
[Case 514]
Prompt: What college did Tatiana Vladislavovna Petrova go to?
Answer: Moscow State University
Edit Target: Moscow State Institute of International Relations
MEND
Output: mmmmm
+SimIE
Output: Moscow State Institute of International Relations
ROME
Output: mmmmm
+SimIE
Output: Moscow State University of International Relations
MEMIT
Output: mmmmm
+SimIE
Output: Moscow State University of International Relations K
AlphaEdit
Output: mmmmm
+SimIE
Output: Moscow State Institute of International Relations
[Case 692]
Prompt: Which series The Adventure of the Blanched Soldier part of?
Answer: The Case-Book of Sherlock Holmes = Edit Target: The Memoirs of Sherlock Holmes
MEND
Output: mmmmm
+SimIE
Output: The Memoirs of Sherlock Holmes
ROME
Output: mmmmm
+SimIE
Output: The Memoirs of Sherlock Holmes
MEMIT
Output: mmmmm
+SimIE
Output: The Memoirs of Sherlock Holmes
AlphaEdit
Output: mmmmm
+SimIE
Output: The Memoirs of Sherlock Holmes
30

---

Simulating Ideal Editor
D.3.2. HIDDEN REPRESENTATION VISUALIZATION
To gain deeper insights into the effectiveness of SimIE, we analyze hidden representations of the model through the following steps: 1) Using the ZsRE dataset, we conduct 1000 sequential edits on Llama-2, retaining three post-edit models obtained from standard algorithms, standard algorithms enhanced with SimIE, and the corresponding ideal editor. 2) For each post-edit model, we compute the values of the last subject token for all edit examples, resulting in 1000 hidden representations' of dimension 4096. 3) The stacked hidden representations are visualized using UMAP (McInnes et al., 2018), with their respective 2-a confidence ellipses plotted for clarity.
The visualization results are presented in Figure 11. deviate significantly from those of the ideal editor, indicating unsuccessful knowledge injection. In contrast, the represen- tation distributions obtained after applying SimIE closely consistent with those of the ideal editor, providing further explanation for the superior performance of SimIE.
We observe that the representation distributions of standard algorithms
Distribution of hidden representations in LLaMA-2 (7B)
2.0
200
1.0
1.0
MEND
MEND
SSmlE
0.0
0.0
0.0
Ideal editor
-2.0
-2.0
-2.0
-1.0
0.0
1.0
-1.0
0.8
0.8
Distribution dididee representations LLaMA-2 (7B)
1.5
1.0
1.0
1.0
ROME
0.0
0.0
ROME
0.5
E
Ideal editer
-1.0
0.0
-2.0
-2.0
-0.5
-1.0
2.0
-1.0
0.0
1.0
2.0
-.2
-.0
-.8
0.5
Distribution of hidden representations in LLaMA-2 (7B)
MEMIT
1.0
1.0
MEMIT
-0.5
SSmlE
0.0
0.0
-1.0
-1.5
2.0
-1.0
0.0
L.0
-2.0
-1.0
0.0
1.0
0.2
0.4
0.6
0.8
1.0
Distribution of hidden representations in LLaMA-2 2 (7B)
AlphaFddit
0.0
0.0
AlphaEdit
=SimlE
0.5
Ideal editor
0.0
-2.0
-2.0
-0.5
Figure 11. Distribution of hidden representations in the post-edit Llama-2, computed across 1000 edited examples from the ZsRE dataset. Dimensionality reduction performed via UMAP, with dashed lines representing 2-o confidence ellipses.
MEND corresponds to the 31st layer, which its final editing layer. ROME, MEMIT, and AlphaEdit correspond to the Sth layer, which their shared editing layer.
31

---

Simulating Ideal Editor
D.3.3. LONG SENTENCE EDITING
To evaluate the effectiveness of S SimIE in the editing of long sentences, consider the task of correcting hallucinations in the SelfCheckGPT (Manakul et al., 2023). The Hallucination (Hartvigsen et al., 2024) dataset consists of highly inaceurate sentences sourced from GPT-3 (Brown al., 2020), replaced with their corresponding sentences from actual Wikipedia entries. Compared to the ZsRE and Counterfact datasets, the Hallucination dataset is more challenging because it represents authentic mistakes made by high-quality LLMs, and the token length of edits significantly exceeds those of past datasets (see Figure 12 for an example). To reduce VRAM usage, we employ a simplified version in Wang et al. (2024b), limiting tokenized lengths to 254, resulting in 600 test samples. We perform 600 sequential edits on GPT2-XL. ROME, AlphaEdit (standard methods), PRUNE, and AlphaEdit (lifelong methods) are selected baselines for their superior performance on GPT2-XI.. Following prior work (Hartvigsen et al., 2024; Wang et al., 2024b), evaluate the performance using PPI. (perplexity) and Loc (Localization Success Rate) metrics.
"prompt": "This is
passage about carole gist. Carole Gist (born April 28, 1969) is American beauty pageant titleholder from Detroit, Michigan who was crowned Miss USA 1990. She was the first
African-American woman to win the Miss USA title. Gist represented the United States at the Miss Universe 1990 pageant held in Los Angeles, California, where she placed first runner-up to Mona Grudt of Norway, Gist was the first African-Americal woman to place the Miss Universe pageant.",
"target_new": "She was also the first contestant from Michigan to win Miss USA, and broke the five-year streak of winners from Texas.", "subject"; "carole gist", "locality_prompt" "Description Map of South America. AnThis map has a small + scratch near the centerfold in the right part of the map. \nLooking for an 4 antique map, historica", print or plan? Feel welcome and browse our mapsite atlasandmap. com! We have maps, made by Kuyper (Kuijper) more maps of South America like Zuid-Am"
Figure 12. sample of the Hallucination dataset.
Table 7 presents the results across varying numbers edits T. As T increases, standard algorithms inevitably degrade, with rising PPL and declining Loc. Even methods specifically designed for lifelong scenarios exhibit elevated PPL at T' 600. In contrast, for ROME, SimIE reduces its PPL from 69.86 to 15.46 and improves its Loc by 33%. Similarly, AlphaEdit enhanced with SimIE outperforms its original version, AlphaEdit, on both metrics. These results confirm the effectiveness of SimIE E. in editing long sentences.
Table Performance of algorithms on the GPT2-XL using the Hallucination dataset, with the best results highlighted bold.
GPT2 XL. (1.SB) using Hallucination
T
=
50
T
100
T'
200
T
300
T'
500
T = 600
Algorithm
PPI. (4)
Loc (1)
(P)
Loc (t)
PPL. (1)
Loc (?)
PP)
Loc (1)
PPI.
Loc (1)
PPL. (1)
Loc
PPL.
Loc (t)
FT-L
60.93
0.50
>000
0.08
1000
0.06
>1000
0.05
1=100
0.07
>1000
0.04
1=000
0.04
ROME
2.44
1.00
8.19
0.94
21.71
0.90
40.38
0.84
45.76
0.80
63.14
0.70
69.86
0.64
+31mIE
2.74
0.98
2.33
0.94
3,10
0.91
4.45
0.88
5.55
0.87
15.74
0.85
15.46
0.85
AlphaEdit
1.87
1.00
5.92
0.98
34.07
0.96
61.08
0.88
105.95
0.74
541.81
0.23
2611.20
0.11
+SimIE
5.90
1.00
7.90
0.96
7.93
0.91
8.74
0.91
11.00
0.90
23.66
0.89
24.00
0.89
PRUNE
2.44
1.00
5.60
0.96
8.32
0.95
11.66
0.93
14.08
0.92
32.48
0.92
50.25
0.92
AlphaEdii
1.86
1.00
3.30
0.98
8.20
0.96
12.54
0.94
18.54
0.92
26.98
0.89
47.39
0.87
32


Reviews:
[{"review_id": "t7Xn4WBKIa", "reviewer": "Reviewer_nKJ9", "summary": "The paper studies the problem of *lifelong* model editing, wherein a model has to *sequentially* incorporate new knowledge without retraining and without altering its behavior on unrelated tasks. Specifically, authors tackle the known issue where consecutive edits cause the model to forget previous edits or compromise its general accuracy - as opposed to batch edits where the model can better account for all examples. The paper proposes a framework (SimIE) for this problem that formulates the task of sequential edits as imitating batch edits, i.e. trying to perform sequential changes that add up to the same effect as though the edited examples were processed jointly. Authors design SimIE to be applied on top of multuple/any parameter-modifying editing algorithms to improve their ability to deal with lifelong(sequential) edits. They evaluate their approach on a range of popular past generation LLMs: Llama 2, Mistral, etc, where it augments multiple different editing methods (ROME, MEND, MEMIT, etc) and improves their performance in a lifelong setting - from abysmal to non-abysmal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to Reviewer nKJ9 for the detailed and constructive feedback on our manuscript. Below, we address your questions point-by-point.\n\n## Q1: evaluate on newer and, importantly, more accurate LLMs\nWe conduct additional evaluations on Llama-3 (8B) and Qwen2.5 (7B). We select FT, ROME, AlphaEdit$^-$, WISE, and AlphaEdit as baselines and perform $T=1000$ sequential edits using the ZsRE dataset. The experimental results are presented in the following table:\n|||||Llama-3 (8B)||||Qwen2.5 (7B)|\n|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| Method | Rel | Gen | Loc | Avg | Rel | Gen | Loc | Avg |\n| FT | 0.17 | 0.14 | 0.01 | 0.10 | 0.09 | 0.08 | 0.02 | 0.07 |\n| ROME | 0.09 | 0.08 | 0.01 | 0.06 | 0.22 | 0.22 | 0.09 | 0.18 |\n| ROME+SimIE | 0.75 | 0.72 | 0.80 | **0.75** | 0.92 | 0.87 | 0.72 | 0.84 |\n| AlphaEdit$^-$ | 0.06 | 0.06 | 0.03 | 0.05 | 0.67 | 0.63 | 0.76 | 0.68 |\n| AlphaEdit$^-$+SimIE | 0.74 | 0.67 | 0.75 | 0.72 | 0.91 | 0.83 | 0.88 | **0.87** |\n| WISE | 0.51 | 0.50 | 1.00 | 0.67 | 0.54 | 0.53 | 0.99 | 0.69 |\n| AlphaEdit | 0.86 | 0.78 | 0.62 | **0.75** | 0.98 | 0.80| 0.72 | 0.83 |\n\nWe observe that SimIE achieves competitive performance across these recent models, especially surpassing the SOTA method (AlphaEdit) by an average of $4.8\\\\%$ on Qwen2.5. These results are consistent with those in the main paper, further confirming the effectiveness of our proposed SimIE.\n\n## Q2: testing if the assumptions hold in practice\nTo provide deeper insights into our theoretical assumptions, we consider two additional analyses assessing the extent to which the assumptions are violated in practice. Please refer to hGQy’s Q2 for details.\n\n## Q3: focuses on the arithmetic mean\nAlthough the arithmetic mean is widely adopted, it does have the potential pitfalls you mention. Inspired by your suggestion, we introduce a new composite metric defined as\n$$e^{\\alpha(\\mathrm{Loc}-1)}(\\mathrm{Rel}*\\mathrm{Gen}).$$\nThis metric takes locality as a penalty term, serving as a smoothed condition factor. Meanwhile, by using the (squared) geometric mean of reliability and generality, it prevents methods from abandoning one of them. We will incorporate this new metric into our paper, thereby offering a clearer view of practical utility.\n\n## Q4: still a gap between SimIE and specialized lifelong editors\nWe recognize that SimlE may underperform some specialized lifelong approaches under certain scenarios. Nevertheless, as increasingly challenging datasets and more realistic evaluations emerge, SimlE provides greater robustness compared to elaborated lifelong algorithms (refer also to our response to Reviewer 1PGC’s Q1). These findings reinforce our central claim: SimlE enables lifelong editing to benefit directly from the ongoing advances in standard editing research, thereby bridging the gap between these two paradigms.\n\n## Q5: related works and minor mistakes\nAll the relevant literature [1,2] mentioned will be thoroughly integrated into our discussion, and the capitalization of Llama-2 will be consistent.\n\n\nThank you once again for your careful and insightful comments, which provide valuable insights for further refinement of our work. \n\n\n[1] Chen, Qizhou, et al. Lifelong knowledge editing for llms with retrieval-augmented continuous prompt learning. arXiv preprint arXiv:2405.03279 (2024).\n\n[2] Sinitsin, Anton, et al. Editable neural networks. ICLR (2020).", "claims_and_evidence": "To the best of my understanding, the main claims in the paper are posed around the proposed SimIE framework\n- that it can improve the ability to perform sequential (lifelong) edits in terms of standard statistics (locality, generality, reliability)\n- that it is method-agnostic in the sense that it applies to any parameter-editing algorithms\n- the theoretical analysis of SimIE optimality under relaxed assumptions\n\nThe first two claims are, from my perspective, properly supported, though it can be enhanced by editing more recent models. The theoretical claims appear outwardly sound, but could be better supported by testing if the assumptions hold in practice (e.g. the the degree of overparameterization for latest LLMs). Thought, I believe that the paper is worth accepting even without these components.", "methods_and_evaluation_criteria": "Authors evaluate the traditional metrics (locality, generality, reliability) on popular LLM knowledge editing tasks for several models. There are always more datasets and models to further improve the analysis, but the current scope appears sufficient to verify the claims.\n\nI am slightly concerned by the fact that the paper **focuses on the arithmetic mean between locality, generality and reliability** in their tables. Since the best results are, in many cases, sub-67% accuracy, this allows a method to win by abandoning one of three criteria - e.g. locality - which would make it a terrible editor. Perhaps it would be better to report (a) geometric mean? (b) conditioned metrics, e.g. best accuracy when locality > 0.8? (or whichever other means authors devise to better represent the method's utility for practitioners)", "theoretical_claims": "Authors formulate a number of theorems regarding the stability and optimality of SimIE under a number of assumptions. Unfortunately, I only managed to follow the rough outline and did not verify every detail of the proof. Though, I believe that the paper is, in its current form, worth accepting even for its empirical contributions.", "experimental_designs_or_analyses": "Their evaluation criteria appear sound, though they could be improved by using more recent models.", "relation_to_broader_scientific_literature": "To the best of my knowledge, the contributions presented in the paper account for main backbone editing methods from broader literature / prior work. That said, there are some related works that may deserve additional discussion (e.g. https://arxiv.org/pdf/2405.03279 performs lifelong editing through prompt 'learning' - as opposed to parameter editing), but the relation is debatable.", "essential_references_not_discussed": "The idea of model editing was concurrently introduced in [1] and [2]. You cite [2], but it is probably best to also include [1], since it was published about a year earlier.\n[1] https://openreview.net/forum?id=HJedXaEtvS\n[2] https://aclanthology.org/2021.emnlp-main.522", "other_strengths_and_weaknesses": "The proposed framework is general, making it possible for future editing algorithms to evaluate in a lifelong without modifications. Though, there is still a gap between editor-agnostic SimIE and specialized lifelong editors.\n\nAs I stated earlier, the paper would also become more convincing if authors evaluate on newer *and, importantly, more accurate* LLMs, e.g. Llama-3.x, Qwen 2.5, deepseek R1 (if enough hardware), as of the time of reviewing. This is because more accurate LLMs are known to be easier to break with model perturbation (e.g. quantization https://arxiv.org/abs/2404.14047 ). Hence, it may be easier to notice a loss of generality / locality / reliability there.", "other_comments_or_suggestions": "Minor: you often capitalize Llama-2 as 'LLaMA-2'. This capitalization was dropped in the second version of the model ( https://arxiv.org/abs/2307.09288 ) and, to the best of my knowledge, did not reappear ever since.", "questions_for_authors": "No questions that would change my evaluations of the paper (as per the reviewing guidelines).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMW1jVyr4N", "reviewer": "Reviewer_7yH3", "summary": "Standard model editing techniques suffer significant performance degradation in sequential editing setting due to model drift and catastrophic forgetting.\nTo tackle the issue, this paper proposes a general framework, i.e., SimIE, to restore the performance of **any** standard model editing techniques in lifelong editing.\nThe key insight is that the desired parameter shift $S_t=B_t K_t^T(K_tK_t^T+\\lambda I)^{-1}$ can be written to the recurrence form $S_t=S_{t-1}+\\Delta W_t k_tk_t^TP_t^{-1}$, where $\\Delta W_t$ is the parameter shift produced by the standard model editing method.\nSimIE is empirically evaluated by being applied to four standard model editing methods, i.e., MEND, ROME, MEMIT, and AlphaEdit, on zsRE and CounterFact datasets, where it significantly improves the performance of the standard model editing methods from nearly complete failure to a level comparable to the state-of-the-art lifelong editing methods.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt thanks to Reviewer 7yH3 for your thorough and thoughtful review of our manuscript. Following are our responses to each individual comment.\n\n## Q1: use more recent datasets or metrics\nWe conduct experiments on a new benchmark, QAEdit, which is tailored for real-world QA tasks and adopts real-world evaluation metrics (released on February 16, 2025). For further details, please refer to our response to Reviewer 1PGC’s Q1.\n\n## Q2: simpler theoretical analyses may make the main text clearer\nWe agree that the least-squares problem in [1] offers a more succinct path toward deriving the update rule (Equation 3.5). That said, we find that keeping two assumptions in the main text helps bridge our theoretical arguments, and retaining Lemma 3.5 can aid readers in understanding the motivation (i.e., simulating the ideal editor). Nonetheless, we will streamline the theory to retain the minimum element that is essential to guide readers through Equation 3.5.\n\n## Q3: code link and transpose symbol\nWe will move the code link to the end of the abstract and correct the notation $A^T$ to $A^\\top$ throughout the manuscript.\n\n\nThank you once again for your valuable time and effort in reviewing our work. These insightful feedbacks will significantly enhance the quality and clarity of our paper.\n\n\n[1] Tan, Chenmien, Ge Zhang, and Jie Fu. Massive Editing for Large Language Models via Meta Learning. ICLR (2024).", "claims_and_evidence": "Following Figure 2, the performance of four standard model editing methods are significantly improved in the sequential editing setting, which strongly supports the claim of the paper.", "methods_and_evaluation_criteria": "The proposed method is reasonable and the evaluation setting is standard, though I would expect to use more (recent) datasets or metrics.", "theoretical_claims": "The proposed method is built upon solid theoretical analyses.\nI have carefully checked the core part and found on problem.", "experimental_designs_or_analyses": "See Methods and Evaluation Criteria", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths\n* This paper is very well written and organized.\nThe notations are clearly defined.\nThe theoretical analyses are insightful and reasonable.\nI enjoy reading the paper.\n* The idea is quite novel and interesting.\nInstead of proposing a sequential editing method, this paper introduces a framework to convert any standard model editing methods to lifelong editing methods, making it general and potentially impactful.\n* The method is simple, which only requires a single line of code to implement, while effective.\n\nWeakness\n\nThe main text of this paper leans more to theoretical analyses rather than empirical evaluation.\nI noticed that some experiments are placed in the Appendix.\nI recommend to incorporate less theoretical analyses and more empirical evaluation in the main text to increase the visibility (as not all readers will go through the Appendix) and to favor a broader audience (as I assume that all readers are interested in the empirical evaluation but only limited care about full theoretical justifications).\n\nSimpler theoretical analyses may make the main text clearer.\nThe author may directly consider the least square problem in [1], leading to the approximation after Lemma 3.5 (so that the two assumptions and Lemma 3.5 can be moved to Appendix).\nThen, by writing the optimal solution to the least square problem in a recurrence form, the core method of the paper, i.e., Formula 3.5, is obtained.\nThe regularization term in the least square formulation also provides a straight-forward explanation for the phenomena observed in the ablation study.\n\n[1] Massive Editing for Large Language Models via Meta Learning, ICLR 2024", "other_comments_or_suggestions": "* I recommend to place the code link at the end of abstract for a better visibility.\n* Please unify the usage of ^\\top and ^T in Lemma 3.5.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "67iAuZKYo5", "reviewer": "Reviewer_hGQy", "summary": "This paper introduces \"Simulating Ideal Editor\" (SimIE), a general framework that enables standard model editing methods to perform effectively in lifelong editing scenarios. The authors formulate the ideal parameter shift as the minimum-norm solution to a linear system constructed using the Moore-Penrose inverse, and develop a recursive update mechanism that approximates this solution through sequential edits. Their theoretical analysis demonstrates that even when key assumptions (over-parameterization and key-value invariance) are violated, SimIE remains either near-optimal or stable against perturbations. Extensive experiments on GPT2-XL, LLaMA-2, and Mistral models show that standard algorithms enhanced with SimIE achieve comparable performance to specialized lifelong editing methods, with minimal implementation. The framework effectively bridges the gap between standard and lifelong model editing paradigms, allowing lifelong editing to benefit from ongoing advancements in standard editing techniques.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer hGQy for your insightful and constructive comments. We have carefully considered your feedback and responded to each of your points below.\n\n## Q1: larger models perform with the proposed SimIE\nWe conduct additional evaluations on larger LLMs, specifically Llama-3 (8B) and Qwen2.5 (7B). For detailed experimental results, please refer to Reviewer nKJ9's Q1.\n\n## Q2: How extensively are the over-parameterization and key-value invariance assumptions violated in your experiments?\nWe conduct two analyses to assess the extent to which the assumptions are violated in practice. Specifically, we utilize AlphaEdit+SimlE to edit Llama-2 using the ZsRE dataset. At each time step $t$, we track the updated key-value pairs $(k_t, v_t)$, alongside their original counterparts $(k^{\\prime}_t, v^{\\prime}_t)$ from the initial unedited model.\n\nFor the over-parameterization assumption, we quantify how effectively each layer can fit the desired update by computing the optimal residual $R_{\\mathrm{min}}$ of the least-squares problem: $\\min \\frac{1}{T}\\sum_{t}^{T}\\\\|Wk^{\\prime}_t-v^{\\prime}_t \\\\|^2$. The results are summarized as follows:\n| Layer | 4 | 5 | 6 | 7 | 8 |\n|-----|-----|-----|-----|-----|-----|\n| $R_{\\mathrm{min}}$ | 2.46e-02 | 3.36e-02 | 4.43e-02 | 5.19e-02 | 8.41e-02 |\n\nEmpirically, we observe that $R_{\\mathrm{min}}$ tends to increase in deeper editing layers, indicating a greater deviation from the over-parameterization assumption.\n\nTo assess the key-value invariance assumption, we measure the deviations $\\\\|k_t-k^{\\prime}_t\\\\|$ and $\\\\|v_t-v^{\\prime}_t\\\\|$, which capture the extent to which edits (both from previous time steps and preceding layers) perturb the original key-value representations. For visualization, we divide the $T=1000$ edits uniformly into $100$ intervals and plot their average values within each. The detailed visualizations are available in the Assumption_analysis.PDF at [SimIE](https://anonymous.4open.science/r/SimIE) (our code link), and a partial summary of key deviations for Llama-2 is provided below:\n| Layer | 1~10 | 191~200 | 391~400 | 591~600 | 791~800 | 991~1000 |\n|-----|-----|-----|-----|-----|-----|-----|\n| 4 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |\n| 5 | 0.088 | 0.306 | 0.335 | 0.364 | 0.376 | 0.373 |\n| 6 | 0.165 | 0.612 | 0.687 | 0.760 | 0.730 | 0.730 |\n| 7 | 0.248 | 0.885 | 1.025 | 1.136 | 1.052 | 1.137 |\n| 8 | 0.323 | 1.227 | 1.476 | 1.608 | 1.472 | 1.626 |\n\nThe findings demonstrate that deviations gradually increase with both editing layer depth and the time step, aligning with your speculation about the perturbations cascade phenomenon.\n\nOverall, both assumptions are indeed violated to some degree in practical scenarios, which emphasizes the trade-off between optimality and sensitivity established by Theorem 4.3. However, the observed violations appear manageable under the current experimental setup, as they do not lead to the empirical failure of SimIE. In scenarios involving longer edit sequences or more editing layers, the violations of assumptions may become severe. These new analyses provide insight into both why SimlE currently succeeds and where it might face challenges going forward, which will be integrated into the manuscript.\n\n## Q3: practical necessity of the proposed approach\nWe would like to clarify that SimIE's primary practical value lies in enabling lifelong editing to benefit from ongoing advances in standard editing research. Indeed, on more challenging datasets and more realistic evaluations recently proposed, specialized lifelong algorithms can exhibit significant performance deterioration, whereas SimlE-enhanced standard methods provide more robust performance (please refer to Reviewer 1PGC's Q1).\n\n## Q4: is SimIE's effectiveness inversely related to model size?\nOur results generally indicate that the performance of SimIE depends more on the fundamental editor, which may be affected by various aspects like model architecture, dataset, hyperparameters, etc. Although we do observe some performance variations at different scales, there is no definitive evidence of an inverse correlation between model size and SimIE’s effectiveness. For instance, on Llama-3 (8B), ROME+SimIE achieves an average metric of $0.75$, outperforming the $0.63$ average observed on the smaller Llama-2 (7B). Additionally, in our new experiments on the other 7B model, Qwen2.5, SimlE improves performance by an average of $4.8\\\\%$ over existing SOTA lifelong methods. These results suggest that SimIE may leverage that strength when the underlying editor performs robustly, even for larger LLMs.\n\n\nThank you once again for your thoughtful and detailed feedback, which offers valuable guidance for the improvement of our paper.", "claims_and_evidence": "Yes, I did not find problems for this part.", "methods_and_evaluation_criteria": "This part is ok. The base models, baseline methods, benchmarks, and metrics in general look good to me in Sec 5. There are some additional benchmark datasets, but not included in this paper.", "theoretical_claims": "There are many theorems in this paper, and most of the proof are in Appendix. The natural language introduction of the idea behind the theorems looks reasonable to me, but I did not check the exact correctness of the theorems in Sec 4.", "experimental_designs_or_analyses": "According to the results reported by the authors, I feel that there is a certain correlation between model size and performance. Therefore, seeing how larger models perform with the proposed SimIE method would provide a deeper understanding of its generalizability.", "relation_to_broader_scientific_literature": "This paper proposes SimIE, a general framework that bridges the gap between standard model editing and lifelong model editing, enabling standard methods to retain strong performance in lifelong scenarios.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\nA key strength of the SimIE framework lies in its strong theoretical foundation that supports each part of the method. By offering clear mathematical proofs for each step of the algorithm, the authors build a solid basis for their approach and achieve good performance on both three backbone models.\n\nWeakness:\n1. While SimIE is applied across multiple layers independently, the analysis doesn't address how assumption violations cascade through the network, e.g., edits to layer 5 alter the hidden states flowing to layer 6, thereby affecting key representations in layer 6. This oversight limits our understanding of why the approach succeeds in practice despite lacking theoretical guarantees for such complex inter-layer dependencies.\n2. Despite the theoretical contributions, Table 1 shows that SimIE's performance doesn't significantly outperform specialized lifelong editing methods like WISE and AlphaEdit. This raises questions about the practical necessity of the proposed approach. Without demonstrating clear advantages over existing methods, the authors' justification for \"bridging the gap\" between standard and lifelong editing remains primarily theoretical rather than performance-driven.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How extensively are the over-parameterization and key-value invariance assumptions violated in your experiments?\n2. After comparing Table 3 with Table 1, I notice an interesting pattern: SimIE demonstrates superior performance over lifelong editing baselines on the smaller GPT2-XL (1.5B) model, but this advantage becomes less significant with the larger 7B models like LLaMA-2 and Mistral. This raises an important question: Is SimIE's effectiveness inversely related to model size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KaO4SfBwDh", "reviewer": "Reviewer_1PGC", "summary": "This paper introduces Simulating Ideal Editor (SimIE), a framework that extends standard parameter-modifying methods to lifelong scenarios. SimIE computes the ideal parameter shift as the minimum-norm solution of a linear system using the Moore-Penrose inverse, and allows recursive updates by truncating its limiting expression under mild assumptions. The theoretical analysis shows that even when assumptions fail, SimIE remains near-optimal or stable, balancing optimality and robustness. Extensive experiments confirm that SimIE achieves performance comparable to specialized lifelong editing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to Reviewer 1PGC for the careful and insightful comments on our manuscript. Our detailed responses to your questions are outlined below.\n\n## Q1: evaluating the method's performance without relying on teacher forcing\nFollowing your suggestion, we conduct additional experiments on the recently proposed QAEdit benchmark [1] (released on February 16, 2025), which adopted the real-world evaluation metrics that do not rely on teacher forcing. Specifically, we take FT, ROME, AlphaEdit$^-$, WISE, and AlphaEdit as baselines and perform $T=1000$ sequential edits on Llama-2. The results are summarized in the table:\n| Method | FT | ROME | ROME+SimIE | AlphaEdit$^-$ | AlphaEdit$^-$+SimIE | WISE | AlphaEdit |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Real-world Rel | 0.00 | 0.00 | 0.20 | 0.00 | 0.32 | 0.14 | 0.27 |\n| Real-world Gen | 0.00 | 0.00 | 0.13 | 0.00 | 0.17 | 0.08 | 0.18 |\n| Real-world Loc | 0.00 | 0.00 | 0.34 | 0.00 | 0.35 | 0.22 | 0.03 |\n| Real-world Avg | 0.00 | 0.00 | 0.22 | 0.00 | **0.28** | 0.15 | 0.16 |\n\nWe observe that all methods, particularly lifelong editing approaches, suffered significant performance degradation in the more realistic evaluation setting. In contrast, standard algorithms enhanced with SimIE consistently outperformed these specialized lifelong editing methods, demonstrating greater robustness. Although SimIE's performance is still limited by the effectiveness of the fundamental editor, its core strength lies in its generality: SimIE enables lifelong editing to benefit directly from any future improvements in standard editors without the need for specialized redesign.\n\n## Q2: if these assumptions are not met\nWe provide a more in-depth empirical analysis of the two assumptions, measuring their violation in practice. Please refer to Reviewer hGQy's Q2 for the details.\n\n## Q3: a deeper discussion and code\nWe will expand our Related Work and Limitations sections to offer a deeper discussion, highlighting both the advantages of SimIE (e.g., leveraging advances in standard editors without specialized redesign) and its limitations (e.g., may be limited by the fundamental editor’s performance). As for the code, we have provided an anonymous GitHub repository in Section 4, which will be moved at the end of the abstract to ensure better visibility.\n\n\nThank you once again for your insightful feedback, which has significantly enhanced and refined our work.\n\n\n[1] Yang, Wanli, et al. The mirage of model editing: Revisiting evaluation in the wild. arXiv preprint arXiv:2502.11177 (2025).", "claims_and_evidence": "The claims is supported by experiments.", "methods_and_evaluation_criteria": "The proposed method make sense for the problem.", "theoretical_claims": "I only partially checked the formulas and principles, and some parts are not very easy to understand.", "experimental_designs_or_analyses": "The experiments is sound. I suggest that the authors include additional case studies and evaluate the method's performance by calculating metrics without relying on teacher forcing.", "relation_to_broader_scientific_literature": "SimIE formulates the ideal parameter shift as the minimum-norm solution to a linear system constructed using the Moore-Penrose inverse, and it subsequently enables recursive updates by truncating the inverse’s limiting expression under two mild assumptions.", "essential_references_not_discussed": "There are many papers on lifelong model editing; it is recommended to carefully discuss the differences from previous work.", "other_strengths_and_weaknesses": "Strengths:\n\nSimIE effectively connects standard model editing with lifelong model editing, leveraging advancements from both fields.\n\nThe framework is underpinned by a clear theoretical formulation using the Moore-Penrose inverse, providing insights into the optimality and stability of the solution.\n\nBy reformulating the ideal parameter shift as a minimum-norm solution, SimIE generalizes parameter-modifying methods to lifelong scenarios, potentially broadening their applicability.\n\nWeaknesses:\n\nThe recursive update mechanism depends on two mild assumptions. If these assumptions are not met, the method may face a trade-off between optimality and stability.\n\nThe paper could benefit from a deeper comparative discussion with existing lifelong model editing approaches to highlight distinctive advantages and potential limitations.\n\nThe theoretical guarantees are tied to the specific conditions assumed; deviations in real-world scenarios might challenge the robustness of the proposed method.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method for lifelong model editing. By estimating and caching ideal parameter updates throughout sequential edits, their approach significantly improves the performance of parameter-updating model editing methods on two standard editing datasets.

Reviewers overall found the strengths to outweigh the weaknesses. They appreciated the novelty of the idea, the theoretical analysis, and that the method outperforms the state-of-the-art (though their approach's performance is very similar to the lifelong model editors).
Decision: Accept (poster)